How to Use Selenium to Web-Scrape with Example | by Bryan Pfalzgraf | Towards Data Science

Selenium is a Python library and tool used for automating web browsers to do a number of tasks. One of such is web-scraping to extract useful data and information that may be otherwise unavailable. Here’s a step-by-step guide on how to use Selenium with the example being extracting NBA player salary data from the website https://hoopshype.com/salaries/players/.
pip install selenium
Once installed, you’re ready for the imports.
from selenium import webdriverfrom selenium.webdriver.common.keys import Keysimport pandas as pd
A webdriver is a vital ingredient to this process. It is what will actually be automatically opening up your browser to access your website of choice. This step is different based on which browser you use to explore the internet. I happen to use Google Chrome. Some say Chrome works best with Selenium, although it does also support Internet Explorer, Firefox, Safari, and Opera. For chrome you first need to download the webdriver at https://chromedriver.chromium.org/downloads. There are several different download options based on your version of Chrome. To locate what version of Chrome you have, click on the 3 vertical dots at the top right corner of your browser window, scroll down to help, and select “About Google Chrome”. There you will see your version. I have version 80.0.3987.149, shown in the screenshots below.
Now you need to know where you saved your webdriver download on your local computer. Mine is just saved in my default downloads folder. You now can create a driver variable using the direct path of the location of your downloaded webdriver.
driver = webdriver.Chrome('/Users/MyUsername/Downloads/chromedriver')
Very simple yet very important step. You need your code to actually open the website you’re attempting to scrape.
driver.get('https://hoopshype.com/salaries/players/')
When run, this code snippet will open the browser to your desired website.
In order to extract the information that you’re looking to scrape, you need to locate the element’s XPath. An XPath is a syntax used for finding any element on a webpage. To locate the element’s XPath, highlight the first in the list of what you’re looking for, right click, and select inspect; this opens up the developer tools. For my example, I first want to locate the NBA player names, so I first select Stephen Curry.
In the developer tools, we now see the element “Stephen Curry” appears as such.
<td class=”name”> <a href=”https://hoopshype.com/player/stephen-curry/salary/"> Stephen Curry </a> </td>
This element can easily be translated to its XPath, but first, we need to remember that we aren’t just trying to locate this element, but all player names. Using the same process, I located the next element in the list, Russell Westbrook.
<td class=”name”> <a href=”https://hoopshype.com/player/russell-westbrook/salary/"> Russell Westbrook </a> </td>
The commonality between these two (and all other player names) is <td class=”name”>, so that is what we will be using to create a list of all player names. That translated into an XPath looks like //td[@class=”name”]. Breaking that down, all XPaths are preceded by the double slash, which we want in a td tag, with each class in that td tag needing to correspond to “name”. We now can create the list of player names with this Selenium function.
players = driver.find_elements_by_xpath('//td[@class="name"]')
And now to get the text of each player name into a list, we write this function.
players_list = []for p in range(len(players)):    players_list.append(players[p].text)
Following this same process to acquire the player salaries...
Stephen Curry’s 2019/20 Salary
<td style=”color:black” class=”hh-salaries-sorted” data-value=”40231758"> $40,231,758 </td>
Russel Westbrook’s 2019/20 Salary
<td style=”color:black” class=”hh-salaries-sorted” data-value=”38506482"> $38,506,482 </td>
While inspecting these elements and translating to XPath, we can ignore style and data-value, only worrying about the class.
salaries = driver.find_elements_by_xpath('//td[@class="hh-salaries-sorted"]')
And the list of salaries text...
salaries_list = []for s in range(len(salaries)):    salaries_list.append(salaries[s].text)
Often, when using Selenium, you’ll be attempting to retrieve data that is located on multiple different pages from the same website. In my example, hoopshype.com has NBA salary data dating back to the 1990/91 season.
As you can see, the difference between the URL of each season is just a matter of the years being included at the end. So the URL for the 2018/19 season is https://hoopshype.com/salaries/players/2018-2019/ and the URL for the 1990/91 season is https://hoopshype.com/salaries/players/1990-1991/. With that, we can create a function that loops through each year and accesses each URL individually and then puts all of the steps we’ve previously shown together for each year individually. I also pair each player with their salary for that season together, place into a temporary dataframe, add the year onto that temporary dataframe, and then add this temporary dataframe to a master dataframe that includes all of the data we’ve acquired. The final code is below!