Web Scraping Using Selenium. A simplified example of scraping an... | by Kiprono Elijah Koech | Towards Data Science

In this article we will go through a web scraping process of an E-Commerce website. I have designed this particular post to be beginner friendly. So, if you have no prior knowledge about web scraping or Selenium you can still follow along.
To understand web scraping, we need to understand HTML code basics. We will cover that as well.
There are a lot of things to talk about concerning HTML basics but we will focus on the things that will be helpful (at least most of the times) in web scraping.
HTML element (Fig 1 RIGHT) — an HTML element is the collection of start tag, its attributes, an end tag and everything in between.
Attributes — are special words used inside a start tag to control the element’s behavior. Attribute and its value are together used in referencing a tag and its content for styling. The most important attributes we will use in web scraping includes class, id and name.
class and id attributes — HTML elements can have one or more classes, separated by spaces (see Fig 1 LEFT above). On the other hand, HTML elements must have unique id attributes, that is, an id cannot be used to reference more than one HTML element.
Before we go into the actual scraping of an E-Commerce site let us scrape the site shown in the Figure below (from the HTML code in Fig 1 LEFT)
From the Figure (Fig 2) above note the following:
This is the Uniform Resource Locator (URL). For this particular case, the locator leads to HTML code stored locally.The button labelled 2 is very important when you are hovering through the page to identify the elements of your interest. Once your object of interest is highlighted the tag element will also be highlighted.This is the page source code. It is just the HTML code like in Fig 1 LEFT. You can view this page source by clicking Ctrl+Shift+I to inspect the page or right click on site and choose Inspect Element or Inspect whichever is available on the options.
This is the Uniform Resource Locator (URL). For this particular case, the locator leads to HTML code stored locally.
The button labelled 2 is very important when you are hovering through the page to identify the elements of your interest. Once your object of interest is highlighted the tag element will also be highlighted.
This is the page source code. It is just the HTML code like in Fig 1 LEFT. You can view this page source by clicking Ctrl+Shift+I to inspect the page or right click on site and choose Inspect Element or Inspect whichever is available on the options.
To conduct web scraping, we need selenium Python package (If you don’t have the package install it using pip) and browser webdriver. For selenium to work, it must have access to the driver. Download web drivers matching your browser from here: Chrome, Firefox, Edge and Safari. Once the web driver is downloaded, save it and note the path. By default, selenium will look for the driver on the current working directory and as such you may want to save the drive on the same directory as the Python script. You are however not obliged to do this. You can save it anyway and provide a full path to the executable on Line 5 below
Line 1 and 2 import necessary libraries.
Line 4 an 5— Define the path to the web driver you downloaded and instantiate a Chrome driver. I am using Chrome web driver but you can as well use Firefox, Microsoft Edge or Safari.
Line 6 — The driver launches a Chrome session in 5 and get the url source in 6.
Line 7 and 8— This line pauses Python execution for 5 seconds before closing the browser in 8. Pausing is important so that you have a glance of what is happening on the browser and closing ensures that the browsing session is ended otherwise we will end up with so many windows of Chrome sessions. Sleeping time may also be very important when waiting for the page load. However, there is another proper way of initiating a wait.
This is the most important part of web scraping. In this section we need to learn how to get HTML elements by using different attributes.
Recall: Elements of a web page can be identified by using a class, id, tag, name or/and xpath. Ids are unique but classes are not. This means that a given class can identify more than one web element whereas one id identifies one and only one element.
One HTML element can be identified using any of the following methods
driver.find_element_by_id
driver.find_element_by_name
driver.find_element_by_xpath
driver.find_element_by_tag_name
driver.find_element_by_class_name
Multiple HTML elements can be identified using any of the following (the result is a list of elements found)
driver.find_elements_by_name
driver.find_elements_by_xpath
driver.find_elements_by_tag_name
driver.find_elements_by_class_name
Note: id cannot be used to identify multiple elements because id can only identify one element.
Output:
Element identified by id: Content 2 hereElement identified by class: Content hereElement identified by class: Content 3 hereElement identified by tag name: My First HeadingElement identified by xpath: My first paragraph.
Line 8 — Note that container1 is a class attribute value identifying two elements and drive.find_element_by_class_name returns the first element found.
To extract the text from HTML element we use .text function as shown in the code snippet above.
Line 11 — To locate an element by xpath inspect the site elements, right click on the source code matching the element of interest and copy the XPath as show in the Figure (Fig 3) below
We can identify and loop through both elements identified by container1 class as shown below
Output:
Content hereContent 3 here
Now that you got the feel of how selenium work let us go ahead and scrape the actual site we are supposed to scrape. We will be scraping online book store [link].
We will proceed as follows.
Scrape details for each book on the page. Each page as 20 books. The details of each book can be found by using the URL on each card. So, to get the book details we need this links.
Scrape books in each and every page. This means that we will have a loop to scrape each book in the page and another one to iterate through pages.
Moving from one page to another involves a modification of the URL in a way that it is trivial to predict a link to any page.
Here are the pages:
Page 1 URL : http://books.toscrape.com/ . The following link also works for page 1 : http://books.toscrape.com/catalogue/page-1.html
Page 2 URL : http://books.toscrape.com/catalogue/page-2.html
Page 3 URL : http://books.toscrape.com/catalogue/page-3.html
Page 4 URL : http://books.toscrape.com/catalogue/page-4.html
and so on
Clearly, we can notice a pattern implying that looping through the pages will be simple because we can generate these URL as we move along the loop.
On inspecting the site here is the HTML code for the highlighted region (representing one book)
<article class="product_pod"> <div class="image_container">  <a href="the-nameless-city-the-nameless-city-1_940/index.html">   <img src="../media/cache/f4/79/f479de5f305c2ac0512702cf7155bb74.jpg" alt="The Nameless City (The Nameless City #1)" class="thumbnail">   </a>  </div>  <p class="star-rating Four">   <i class="icon-star"></i>   <i class="icon-star"></i>   <i class="icon-star"></i>   <i class="icon-star"></i>   <i class="icon-star"></i>  </p>  <h3>   <a href="the-nameless-city-the-nameless-city-1_940/index.html" title="The Nameless City (The Nameless City #1)">The Nameless City (The ...</a>  </h3>  <div class="product_price">   <p class="price_color">£38.16</p>   <p class="instock availability">    <i class="icon-ok"></i>            In stock</p>   <form>    <button type="submit" class="btn btn-primary btn-block" data-loading-text="Adding...">Add to basket</button>   </form>  </div> </article>
Before we go into coding lets make some observations
The book in question is inside article tag. The tag as a class attribute with the value product_prod
What we need in this card is to get the URL, that is, href in a tag. to get into href we need to move down the hierarchy as follows: class= “product_prod” > h3 tag > a tag and the get value of href attribute.
In fact all books in all pages belong to the same class product_prod and with article tag.
Lets go through some lines so that you understand what the code is actually doing
Line 16 through 18 — We are moving down the the HTML code to get the URL for the book. Once we get the link we open in 19. Note that 16 locates all books in the page because all the books belongs to the same class product_prod that is why we index (index 0) it to get only one book.
It is also important to note that the numbers of stars a book has comes as a property of p tag. Here is where the star is located:
<p class="star-rating Four">   <i class="icon-star"></i>   <i class="icon-star"></i>   <i class="icon-star"></i>   <i class="icon-star"></i>   <i class="icon-star"></i>  </p>
This book is rated 4-star but this fact is hidden as a value to the class attribute. You use get_attribute(class) to access such an information. On extracting content on line 24 by using text function, you will get such a string
star-rating Four
Therefore we had to splitting the string before using the function in line 28–38 to get the actual star as a number.
If you extract the text content in 23 you will end up with such as string
In stock (22 available)
What we need is just the number 22 to mean that we have 22 copies available. We achieve that with line 25 where we are using regular expressions to extract the number.
re.findall("\d+","In stock (22 available)")Output:['22']
In regular expression \d means the values [0–9] and + means one or more occurrence of a character in that class, that is, in our case, we want to capture any number (irrespective of the number of the digits) in the string.
Recall that line 16 above locates all the books in the page. Therefore, in order to scrape all the books in one page we need to loop through the list generated in that line as shown below
Once you have understood the previous example of scraping one book then this one should be easy to follow because the only difference is that we want to loop through all the books extracting the links (lines 33 through 38) and then loop through those links to extract the information we needed (line 42–84).
In this snippet also we have introduced try-except blocks to catch cases where the information we want is not available. Specifically, some books misses description section.
I am sure you will also enjoy to see selenium open the pages as you watch. Enjoy!
The key concept to understand here is that we need to loop through each book in each page, that is, two loops are involved here. As stated earlier we know that pages URL follows some pattern, for example, in our case we have
Page 1 : http://books.toscrape.com/index.html or http://books.toscrape.com/catalogue/page-1.html
Page 2: http://books.toscrape.com/catalogue/page-2.html
Page 3: http://books.toscrape.com/catalogue/page-3.html
and so on until page 50 (we have 50 pages on site).
We can easily create a Python for-loop to generate such URLs. Lets see how we can generate the first 10
Output:
http://books.toscrape.com/catalogue/category/books_1/page-1.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-2.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-3.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-4.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-5.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-6.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-7.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-8.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-9.htmlhttp://books.toscrape.com/catalogue/category/books_1/page-10.html
Therefore, we can scrape all books in all pages simply as below
The only difference between this snippet and the previous one is the fact that we are looping through pages with the loop starting in line 34 and the fact we also write all the information scraped into a CSV file named all_pages.csv .
We are also using try-expect to handle exceptions that may arise in the process of scraping. In case an exception is raise we just exit and close the browser (line 94)
Web scraping is an important process of collecting data from the internet. Different websites have different designs and therefore there’s no particular one scraper that can be used in any particular site. The most essential skill is to understand web scraping on a high level: knowing how to locate web elements and being able to identify and handle errors when they arise. This kind of understanding comes up when we practice web scraping on different sites.
Here are more web scraping examples:
towardsdatascience.com
medium.com
Lastly, some websites do not permit their data to be scraped especially scraping and publishing the result. It is, therefore, important to check the site policies before scraping.
Join medium on https://medium.com/@kiprono_65591/membership to get full access to every story on Medium.
You can also get the articles into your email inbox whenever I post using this link: https://medium.com/subscribe/@kiprono_65591
As always, thank you for reading :-)