Web Scraping Using Selenium with Python🐍……….👀
By: Dr. Niraj Kumari
Tags: Selenium, Web Crawling, Web Scraping, Selenium Webdriver, Python3

Before we take dive into the topic of this article, let us first understand what web-scraping/ web-crawling is and how is it useful for any business analysis and development.

1. What is web-scraping?

Web scraping refers to the extraction of data from a website. Web scraping is a technique for extracting information from the internet automatically using a software that simulates human web surfing. Web Scrapping also called “Crawling” or “Spidering” is the technique to gather data automatically from an online source usually from a website. While Web Scrapping is an easy way to get a large volume of data in a relatively short time frame, it adds stress to the server where the source is hosted.

This is also one of the main reasons why many websites do not allow scraping all on their website. However, as long as it does not disrupt the primary function of the online source, it is fairly acceptable.

2. Uses Cases of Web Scraping:

There are many uses for Web Scraping but I will mention just a few:

Contact Scraping

Data Mining

Online Price Change Monitoring & Price Comparison

Product Review Scraping: to watch your competition.

Gathering Real Estate Listings

Weather Data Monitoring

Website Change Detection

Research

Tracking Online Presence and Reputation

Web Data Integration

3. How is web-scraping useful?

Web scraping is a technique used for extracting data from websites about customers, products, people, stock markets, etc. It is usually difficult to get this kind of information on a large scale using traditional data collection methods. We can utilize the data collected from a website such as e-commerce portal, social media channels to understand customer behaviors and sentiments, buying patterns, and brand attribute associations which are critical insights for any business.

Let’s do some hands on!!

Since we have defined our purpose of scraping, let us dive into that and explore that how to actually do all the stuff! Before that below are some of the housekeeping instructions regarding installations of packages.

a. Python version: We will be using Python 3.8, however feel free to use Python 2.0 by making slight adjustments. We will be using Spyder notebook, so you don’t need any command line knowledge.

What is Selenium: –

Selenium is an open-source web-based automation tool. Selenium primarily used for testing in the industry but it can also be used for web scraping. We’ll use the Chrome browser but you can try on any browser, it’s almost the same.

Note: It is important to know that Web scraping is against most websites’ terms of service. Your IP address may be banned from a website if you scrape too frequently or maliciously.

b. Selenium package: You can install selenium package using the following command.

1. Installation:

o Install selenium using pip

pip install selenium

o Install selenium using conda

conda install -c conda-forge selenium

c. Chrome driver: Please install the latest version of chromedriver from here.

Please note you need Google Chrome installed on your machines to work through this illustration.

­­­­­

The first thing while crawling a website is to understand the structure of the website. We will be scraping www.chrono24.com, a luxury watch website. Chrono24 is the world’s leading marketplace for luxury watches. People can buy and sell their watches on this website. We will be scraping the current price of the models either new or pre-owned one.

Download Chrome Driver:

To download web drivers, you can choose any of below methods-

You can either directly download chrome driver from the below link-

https://chromedriver.chromium.org/downloads

Or you can download it directly using below line of code-

driver = webdriver.Chrome(ChromeDriverManager().install())

Following methods will help us to find elements in a Web-page (these methods will return a list):

find_elements_by_name

find_elements_by_xpath

find_elements_by_link_text

find_elements_by_partial_link_text

find_elements_by_tag_name

find_elements_by_class_name

find_elements_by_css_selector

Now let’s write one Python code to scrape required data from web.

Implementation of Web Scrapping using Selenium with Python: –

Step1: — Import libraries

import sys

import csv

from selenium import webdriver

from datetime import datetime

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys

from selenium.common.exceptions import NoSuchElementException

from time import time

from time import sleep

Step 2: — Install Driver and create new instance of Chrome to open in Incognito mode

#Install Driver driver = webdriver.Chrome(ChromeDriverManager().install()) # we will be using below incognito argument to our webdriver option = webdriver.ChromeOptions()

option.add_argument(“ — incognito”) driver = webdriver.Chrome(executable_path=’/Library/Application Support/Google/chromedriver’, chrome_options=option)

Note: executable_path is the path that points to your downloaded ChromeDriver.

Step 3: — Pass the desired website URL

Here we are using Chrono24 luxury home page as the desired website URL.

driver.get('https://chrono24.com')

wooaahhh, we opened our website in Google Chrome Incognito mode from python notebook 😊

Step 4: — We need to first Accept the Data Privacy Settings

Once we open the Chrono24 website we must accept the data privacy setting to go further and its need to be clicked every time through python script.

Step 5: — Locate the Accept button to be clicked from the page.

Once we got the data privacy pop up on our browser window, we will locate and inspect the element “Accept” in web page.

We have to copy the XPath from XML because Selenium has a function called “find_element_by_xpath” and we will be using the same. We will pass our XPath into this function and get a selenium element. Once we have the selenium element we can click it’s XPath using “Click” function ‘Accept_ button.click()’ shown below.

Accept_button = WebDriverWait(driver,10).until(lambda driver: driver.find_element_by_xpath('//*[@id="jq-colorbox-container"]/a')) Accept_button.click()

Step 6: — Find out the Search Box Location for Watch Model search.

We need to locate search box and copy the XPath of that location. We will store model name in a variable then send that model name into search box using “Send keys” function in selenium.

#Search for Rolex Oyster Perpetual model on website R=”Rolex Oyster Perpetual” username = WebDriverWait(driver,10).until(lambda driver: driver.find_element_by_xpath('//*[@id="query"]')) username.send_keys(R)

After sending the model’s name in search box next step is to click on “FIND A WATCH NOW”. For that we will use same selenium “Click” function which is already mentioned above and to fetch the conditions of watch we need to change the search view layout to vertical alignment.

#Click on ‘FIND A WATCH NOW’ button Find_button = WebDriverWait(driver,10).until(lambda driver: driver.find_element_by_xpath('//*[@id="header-search"]/div/div[1]/span/button')) Find_button.click() #switch the search result view layout to vertical alignment wait.until(lambda driver: driver.find_element_by_xpath('//*[@id="jq-result-view-switch"]/a')).click()

Step 7: — Inspecting required data from the website

We will inspect 3 items on our web page (Model Name, Price and Watch Condition) and understand how we can extract them.

Model Name: Inspecting the Model Name, we can see the highlighted text represents the XML code for Model Name.

How do we fetch the text inside a XPath?

The XML path (XPath)for the Model location contains a watch model name, with Selenium “find_elements_by_xpath” function We will get list of selenium elements. Once we have elements, we can extract the text inside our XPath using the ‘text’ function. In our case the text is basically the watch brand name “Rolex Oyster Perpetual 41/ Silver Grey Rhodium Dial”.

Models = driver.find_elements(By.XPATH,"//div[contains(@class,'article-title')]")

2. Price: Similarly, like Model name, we will now inspect the price i.e. what is the price of each model.

We will again do the same process and will use the same selenium function “find_elements_by_xpath” to get the models Price details.

Prices = driver.find_elements(By.XPATH,"//div[contains(@class,'article-price-container')]")

3. Condition: Lastly, let’s extract condition of each model, whether it is good, very good or unworn.

Similar to the Model name as we did earlier, we will extract the condition from the below XPath.

Conditions = driver.find_elements(By.XPATH,'//*[@id="wt-watches"]/div/a/div/div[2]/div[1]/div[2]/div[4]/div[2]')

Step 8: — Extract all the details of search result (Models, Price, Condition) and store it in CSV.

We will need to apply python iteration (for loop) to get all the values or text from the selenium element which we got from XPath for all the records.

Store output in CSV with Python

# Creating an output CSV file

chrono_results = 'chrono_results.csv' with open('chrono_results.csv','w', newline='') as f:

thewriter1 = csv.writer(f)

thewriter1.writerow(['BRAND','PRICE', 'CONDITION'])

i= 0

for model in Models:

data=[]

data.append(str(model))

data.append(Prices[i])

data.append(Conditions[i])

i = i+1

thewriter1.writerow(data)

f.close()

Summary: You now have the power to scrap 😊

We have learnt how to scrape a website using Selenium with Python and get required and large amounts of data. You can do multiple unstructured data analytics and find interesting trends, etc. using this data.

Hope this was helpful. Enjoy Scraping BUT BE CAREFUL!!!

************** Thank you for reading ******************