Web Scraping Using Selenium — Python | by Atindra Bandi | Towards Data Science

Before we delve into the topic of this article let us first understand what is web-scraping and how is it useful.
What is web-scraping?
What is web-scraping?
Web scraping is a technique for extracting information from the internet automatically using a software that simulates human web surfing.
2. How is web-scraping useful?
Web scraping helps us extract large volumes of data about customers, products, people, stock markets, etc. It is usually difficult to get this kind of information on a large scale using traditional data collection methods. We can utilize the data collected from a website such as e-commerce portal, social media channels to understand customer behaviors and sentiments, buying patterns, and brand attribute associations which are critical insights for any business.
Let’s now get our hands dirty!!
Since we have defined our purpose of scraping, let us delve into the nitty-gritty of how to actually do all the fun stuff! Before that below are some of the housekeeping instructions regarding installations of packages.
a. Python version: We will be using Python 3.0, however feel free to use Python 2.0 by making slight adjustments. We will be using jupyter notebook, so you don’t need any command line knowledge.
b. Selenium package: You can install selenium package using the following command
!pip install selenium
c. Chrome driver: Please install the latest version of chromedriver from here.
Please note you need Google Chrome installed on your machines to work through this illustration.
The first and foremost thing while scraping a website is to understand the structure of the website. We will be scraping Edmunds.com, a car forum. This website aids people in their car buying decisions. People can post their reviews about different cars in the discussion forums (very similar to how one posts reviews on Amazon). We will be scraping the discussion about entry level luxury car brands.
We will scrape ~5000 comments from different users across multiple pages. We will scrape user id, date of comment and comments and export it into a csv file for any further analysis.
Let’s begin writing our scraper!
We will first import important packages in our Notebook —
#Importing packagesfrom selenium import webdriverimport pandas as pd
Let’s now create a new instance of google chrome. This will help our program open an url in google chrome.
driver = webdriver.Chrome('Path in your computer where you have installed chromedriver')
Let’s now access google chrome and open our website. By the way, chrome knows that you are accessing it through an automated software!
driver.get('https://forums.edmunds.com/discussion/2864/general/x/entry-level-luxury-performance-sedans/p702')
So, how does our web page look like?
We will inspect 3 items (user id, date and comment) on our web page and understand how we can extract them.
User id: Inspecting the userid, we can see the highlighted text represents the XML code for user id.
User id: Inspecting the userid, we can see the highlighted text represents the XML code for user id.
The XML path (XPath)for the userid is shown below. There is an interesting thing to note here that the XML path contains a comment id, which uniquely denotes each comment on the website. This will be very helpful as we try to recursively scrape multiple comments .
//*[@id=”Comment_5561090"]/div/div[2]/div[1]/span[1]/a[2]
If we see the XPath in the picture, we will observe that it contains the user id ‘dino001’.
How do we extract the values inside a XPath?
Selenium has a function called “find_elements_by_xpath”. We will pass our XPath into this function and get a selenium element. Once we have the element, we can extract the text inside our XPath using the ‘text’ function. In our case the text is basically the user id (‘dino001’).
userid_element = driver.find_elements_by_xpath('//*[@id="Comment_5561090"]/div/div[2]/div[1]/span[1]/a[2]')[0]userid = userid_element.text
2. Comment Date: Similar to the user id, we will now inspect the date when the comment was posted.
Let’s also see the XPath for the comment date. Again note the unique comment id in the XPath.
//*[@id="Comment_5561090"]/div/div[2]/div[2]/span[1]/a/time
So, how do we extract date from the above XPath?
We will again use the function “find_elements_by_xpath” to get the selenium element. Now, if we carefully observe the highlighted text in the picture, we will see that the date is stored inside the ‘title’ attribute. We can access the values inside attributes using the function ‘get_attribute’. We will pass the tag name in this function to get the value inside the same.
user_date = driver.find_elements_by_xpath('//*[@id="Comment_5561090"]/div/div[2]/div[2]/span[1]/a/time')[0]date = user_date.get_attribute('title')
3. Comments: Lastly, let’s explore how to extract the comments of each user.
Below is the XPath for the user comment —
//*[@id="Comment_5561090"]/div/div[3]/div/div[1]
Once again, we have the comment id in our XPath. Similar to the userid we will extract the comment from the above XPath
user_message = driver.find_elements_by_xpath('//*[@id="Comment_5561090"]/div/div[3]/div/div[1]')[0]comment = user_message.text
We just learnt how to scrape different elements from a web page. Now how to recursively extract these items for 5000 users?
As discussed above, we will use the comment ids, which are unique for a comment to extract different users data. If we see the XPath for the entire comment block, we will see that it has a comment id associated with it.
//*[@id="Comment_5561090"]
The following code snippet will help us extract all the comment ids on a particular web page. We will again use the function ‘find_elements_by_xpath’ on the above XPath and extract the ids from the ‘id’ attribute.
ids = driver.find_elements_by_xpath("//*[contains(@id,'Comment_')]")        comment_ids = []for i in ids:    comment_ids.append(i.get_attribute('id'))
The above code gives us a list of all the comment ids from a particular web page.
How to bring all this together?
Now we will bring all the things we have seen so far into one big code, which will recursively help us extract 5000 comments. We can extract user ids, date and comments for each user on a particular web page by looping through all the comment ids we found in the previous code.
Below is the code snippet to extract all comments from a particular web page.
Lastly, if you check our url has page numbers, starting from 702. So, we can recursively go to previous pages by simply changing the page numbers in the url to extract more comments until we get the desired number of comments.
This process will take some time depending on the computational power of your computer. So, chill, have a coffee, talk to your friends and family and let Selenium do its job!
Summary: We learnt how to scrape a website using Selenium in Python and get large amounts of data. You can carry out multiple unstructured data analytics and find interesting trends, sentiments, etc. using this data. If anyone is interested in looking at the complete code, here is the link to my Github.
Let me know if this was helpful. Enjoy Scraping BUT BE CAREFUL!
If you liked reading this, I would recommend reading another article about scraping Reddit data using Reddit API and Google BigQuery written by a fellow classmate (Akhilesh Narapareddy) at the University of Texas, Austin.