Build a scalable web crawler with Selenium and Python | by Philipp Postels | Towards Data Science

Disclaimer: Since scraping of Medium.com Services is prohibited by the terms of use, I would like to point out that we immediately processed the underlying data within the project with NLP and no storing of the pure texts took place. The approach illustrated in this article is therefore for demonstration purposes only and can be used for other websites that allow web scraping.
This article is part of a larger project. If you are also interested in performing Natural Language Processing on the results to extract technology names by using PySpark and Kubernetes or building highly scalable Dashboards in Python, you will find corresponding links at the end of the article.
IntroductionProject Idea and approachSource Inspection and PackagesImplementation StepsResults
Introduction
Project Idea and approach
Source Inspection and Packages
Implementation Steps
Results
Life as a Data Scientist can be tough. It is not only the acquisition and quality of data and its interpretability that poses challenges. The rapid development of technologies, as well as constantly rising expectations from business (keyword rocket science), also make the work more difficult. However, in my experience, the acquisition and application of new technologies, in particular, is a source of enthusiasm for most data scientists. For this reason, I built a scalable web crawler with common technologies to improve my skillset.
All files and code snippets that are referenced in this article can be found in my GitHub Repository.
Towards Data Science (TWDS) is one of the best known and most instructive places to go for data science. It is a medium.com publication on which a large number of authors have published various articles. Recurrently used technologies are referenced and their use is often presented in case studies.Therefore I decided to build a web crawler that extracts the content of TWDS and stores it inside the NoSQL database “Google Datastore”. To make the web crawler scalable, I used Docker for containerizing my application and Kubernetes for the orchestration.
The approach was to develop the web crawler in a Jupyter Notebook on my local machine and to constantly professionalize and increase the project (see Fig 2). For instance, I built a Python application with a dedicated crawler class and all necessary methods based on the Jupyter Notebook scripts. But let us have a more detailed look at the implementation steps.
To develop a properly operating web crawler, it is important to familiarize yourself in advance with the site structure, available content, and irregularities.
TWDS is a classic publication with many authors and a lot of articles. Thanks to an archive page it was easy to understand the page structure in detail (see Fig. 3). Fortunately, the authors were not only listed there but also provided with links that led to overview pages for these authors.
The used HTML class was constantly used so that the links could easily be identified (see Fig. 4).
For the overview pages of the authors, I figured out that at first only the author’s articles published on TWDS were listed. Other articles published on Medium.com by the author were not displayed. It was therefore not necessary to check whether the specific article belonged to the TWDS publication. Unfortunately, the HTML class for these links was empty and the links could not be identified. However, the links contained the complete URL and thus the word “towards”. Therefore, the identification of these links was just as unambiguous. However, another challenge occurred when examining the page. Not all of the author’s articles were displayed directly, but when the website was scrolled down further content was dynamically reloaded using Javascript. To ensure completeness, this had to be taken into account for the development of the web crawler.
Finally, I had to examine the structure of the individual articles for similarities and patterns to extract the relevant data fields. The required properties were author, URL, title, text, reading time, publishing date, tags, claps and the number of responses. As can be seen in Figure 5, the HTML source code has some challenges. For example, the class names are seemingly dynamically generated and have only minor matches across articles. But there are also rays of hope, e.g. reading time, title, URL, and publishing date are standardized in the page header. The remaining content was reasonably easy to access.
At first, during development in Jupyter Notebooks, I was looking for Python packages I could use to fulfill all requirements. I quickly realized for Scrapy, one of the most commonly used packages for web scraping, that dynamic content reloading would be difficult. After focusing on this requirement, I became aware of Selenium. Selenium is a framework for automated software testing of web applications and can interact with browsers, e. g. to scroll down pages to load the dynamic javascript content and receive the full HTML source code.
To work with the extracted HTML source code, I found the Python package BeautifulSoup4, which provides various methods to systematically search the HTML tree structure for relevant content. With these packages selected, I could fulfill all the requirements to develop a web crawler.
During the development, I now worked along with the page structure shown in figure 3. So I started with the extraction of the author list.
I defined the URL “https://towardsdatascience.com/archive” to be crawled and used to start the Selenium Webdriver. In the following, I extracted all required parts of the code to run the Selenium Webdriver.
# Importfrom selenium import webdriverfrom selenium.webdriver.chrome.options import Options# Define Browser Optionschrome_options = Options()chrome_options.add_argument("--headless") # Hides the browser window# Reference the local Chromedriver instancechrome_path = r'/usr/local/bin/chromedriver'driver = webdriver.Chrome(executable_path=chrome_path, options=chrome_options)# Run the Webdriver, save page an quit browserdriver.get("https:/towardsdatascience.com/archive")<scroll page snippet goes here>htmltext = driver.page_sourcedriver.quit()
Since the command “driver.get()” only opens the browser and loads the referenced page, I further used a code snippet that automatically scrolled the page down to the end and thus allowed saving the complete HTML source code (“driver.page_source”).
#importsimport time# Scroll page to load whole contentlast_height = driver.execute_script("return document.body.scrollHeight")while True:    # Scroll down to the bottom.    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")    # Wait to load the page    time.sleep(2)    # Calculate new scroll height and compare with last height.    new_height = driver.execute_script("return document.body.scrollHeight")    if new_height == last_height:        break    last_height = new_height
This snippet is completely independent of any website specific structure and can be easily reused in another web crawling context as well.
Since the output is still only the HTML source code and I was looking for a list of all authors, I wrote a “for loop” to extract the links to the authors’ profiles by using my knowledge from source inspection (see chapter 3.1).
# Parse HTML structuresoup = BeautifulSoup(htmltext, "lxml")# Extract links to profiles from TWDS Authorsauthors = []for link in soup.find_all("a",                          class_="link link--darker link--darken u-accentColor--textDarken u-baseColor--link u-fontSize14 u-flex1"):    authors.append(link.get('href'))
The result was now a list with links to the respective authors that could be easily further exploited (see fig. 6). I used the list as the input for my next iteration to receive the articles for each author. As a result, I stored the links of the articles and the link to the authors’ profile page as a key-value pair inside a dictionary (see fig 7).
With the links to the articles in access, I iterated over the different articles, extracted the relevant field contents and stored them inside a dictionary (tempdic). In some cases, this was done simply by specifying the location in the HTML structure.
# Extract field values and store them in jsontempdic = {}tempdic['Article_ID'] = soup.find("meta", attrs={"name": "parsely-post-id"})["content"]tempdic['Title'] = soup.title.stringtempdic['Author'] = soup.find("meta", attrs={"name": "author"})["content"]
In other cases, the use of loops or regular expressions was necessary, e. g. by extracting the tags.
# Loop to extract tagsli = soup.select("ul > li > a")tags = []for link in li:    tags.append(link.string)tempdic['Tags'] = tags
Since I could now store the data of an article systematically in a dictionary, I had to find a suitable way to store the data.
As I already had a perfectly filled dictionary per article and my focus was not supposed to prepare a fitting SQL-Database, I was choosing the Google Datastore to store my data. The Google Datastore is a managed, NoSQL, schemaless database for storing non-relational data — just perfect for this use case.
To use Google Datastore, it is necessary to set up a project at Google Cloud Platform (How to set up a Google Cloud Project; Of course other cloud providers can be used instead). To access Google Datastore by using Python, it is likely to set up a service account with access rights to the Datastore (Role: Cloud Datastore-Owner) inside the project. This can be done in the menu path “API & Services > Credentials” by generating an access-key.
The usage of the generated connection data is easiest when calling the data from a JSON-file. How this can be generated can be seen from the following link.
In the web crawler source code, the connection has to be initialized first. The JSON-file is hereby referenced (“sa.json”).
def initializeGDS(self):    global credentials    global client    print("Setup Database Connection")    credentials = compute_engine.Credentials()        # Service account    client = datastore.Client.from_service_account_json('sa.json')
After adding all relevant information, the entity can finally be stored in Datastore.
Article = datastore.Entity(client.key('Article_ID', str_articlenumber), exclude_from_indexes=['Text'])Article.update({    "URL": str_URL,    "Title": str_title,    "Author": str_author,    "PublishingDate": str_pubdate,    "Text": str_text,    "Claps": int_claps,    "Tags": Tag_list,    "No_Responses": int_responses,    "Reading_time": int_reading_time})client.put(Article)
The functionality of the web crawlers is now completed. As the implementation is still running inside Jupyter Notebook, it is now time for refactoring the code and using a crawler class with specified methods (see TWDS_Crawler.py)
As Docker is the most relevant container platform in software development and part of many implementations, I will not explain any further background within this article. Nevertheless, this was my first use of Docker and I had a look for a convenient step-by-step tutorial to containerize my Python application that is likely to use.
www.wintellect.com
To build the first container image, I only used these four files (GitHub-Repository):
Application-file: In our case TWDS_Crawler.py
sa.json: The JSON-file you generated in the section above with the connection details to your GCP-project
Dockerfile: This file contains all the commands a user could call on the command line to assemble an image
Requirements.txt: Specifies the used Python packages
To build the container image, it necessary to enter the directory folder with the referenced files inside the shell and write the following command:
docker build -t twds-crawler .
This just specified the name of the container image to “twds-crawler” and placed the image in the current directory folder (“.”). To run the container the following command should be used:
docker run twds-crawler
Due to the pre-configured Dockerfile, the Python application inside the container starts automatically after the container is running. The output should look somehow like:
The web crawler application started (“Start Crawler”) and opened the getAuthors method (“Get Authors”) but crashed afterward due to the missing browser instance. For now, this can be ignored as the goal is to run this container inside a Kubernetes cluster.
Kubernetes is an open-source system for automating the deployment, scaling, and management of (docker-)container applications. As it was developed by Google, the Google Cloud Platform delivers a nice implementation so that you can build a cluster only by using the Google Cloud Shell inside the browser and the following script. Just replace <Your Project Name> with the name of your Google Cloud Platform project.
Note: I would recommend using the editor modus to show all stored files.
# Define project variableexport PROJECT_ID=<Your Project Name># Start Clustergcloud beta container — project ${PROJECT_ID} clusters create “twdscrawler” — zone “us-central1-a” — no-enable-basic-auth — cluster-version “1.13.11-gke.14” — machine-type “n1-standard-1” — image-type “COS” — disk-type “pd-standard” — disk-size “100” — metadata disable-legacy-endpoints=true — scopes “https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" — num-nodes “2” — enable-cloud-logging — enable-cloud-monitoring — enable-ip-alias — network “projects/${PROJECT_ID}/global/networks/default” — subnetwork “projects/${PROJECT_ID}/regions/us-central1/subnetworks/default” — default-max-pods-per-node “110” — enable-autoscaling — min-nodes “2” — max-nodes “8” — addons HorizontalPodAutoscaling,HttpLoadBalancing — no-enable-autoupgrade — enable-autorepair
To access the cluster from the shell after the deployment finished you simply use the following command:
gcloud container clusters get-credentials twdscrawler — zone us-central1-a — project <Your Project Name>
The created Kubernetes Cluster has auto-scaling and uses a minimum of 2 nodes and a maximum of 8 nodes (Note: To save some money, make sure to delete the cluster after using it, see main menu point “Kubernetes Engine”).
We are now ready to deploy the selenium grid and our containerized web crawler.
The Selenium Grid is a hub/nodes construction of Selenium with potentially heterogeneous browser versions (nodes) and a control unit (hub) that distributes or parallelizes the work items e. g. unit tests or crawling jobs. To connect both objects there is also a Hub-Service. For a more detailed description check this link.
To make the deployment process as easy as possible and reduce the necessary code to a minimum, I used YAML-Files and bash scripts. YAML-Files describe Kubernetes objects, e. g. in case of the nodes the number of different Selenium nodes to be deployed or the specific browser version. The bash scripts call the different YAML-Files in the right order.
To work inside the Google Cloud shell it necessary to upload the different files. This can easily be done by drag and drop. The following files have to be in there (sa.json needs to be added individually, the rest can be found in my GitHub-Repository).
By using the following command, a complete Selenium Grid with one Firefox-node will be deployed on the Kubernetes Cluster:
bash StartSelenium.sh
To check if everything is working following command could be used:
kubectl get pods
Since the Selenium Grid with a Firefox node is already running on the Kubernetes Cluster, it is time to go on with the web crawler. Due to the local development of the web crawler as well as the use of the local web browser, it is necessary to adjust the Webdriver to the Selenium Grid:
# Define Remote Webdriverdriver = webdriver.Remote(command_executor=’http://selenium-hub:4444/wd/hub',desired_capabilities=getattr(DesiredCapabilities, “FIREFOX”))
Note: The adjusted version can be found in my GitHub-Repository. Just replace the code of the TWDS_Crawler.py or change the referenced file inside the Dockerfile to “TWDS_Crawler_Cluster.py”.
After this change, a new Docker image can be built inside the Google Cloud Shell and published into the Google Cloud Container Registry (comparable to a repository). This can be done with the following commands:
export PROJECT_ID=<Your Project Name>docker build -t gcr.io/${PROJECT_ID}/twds-crawler .docker push gcr.io/${PROJECT_ID}/twds-crawler
If everything worked fine, the web crawler can finally be deployed inside the Kubernetes Cluster with
bash StartCrawler.sh
To check if the crawler runs and see the logs (e. g. the printed lines) you can use the following commands inside the Google Cloud Shell:
kubectl get pods
kubectl logs <crawler-pod-name>
The web crawler is now running. To increase the number of nodes, the YAML File for the Firefox-node has to be edited upfront, or during run time with the following command:
kubectl scale deployment selenium-node-firefox --replicas=10
The Selenium Grid will automatically use the deployed Firefox-node instances during the web crawling process.
If everything worked fine, the results should be visible inside the Google Cloud Datastore just moments later as I chose an incremental approach to write the article details inside the database.
Hope you enjoyed reading my article and good luck with your implementation.
If you have any problems by setting up the project, please also have a look at the troubleshooting area in my GitHub-Repository.
Related articles:
To see how to perform Natural Language Processing on the results and extract technology names by using PySpark and Kubernetes, please have a look at the project of Jürgen Schmidl.
To see how to build a highly scalable Python Dashboard that runs on Kubernetes as well, please have a look at the project of Arnold Lutsch.