Build a scalable web crawler with Selenium and Python
By: Philipp Postels
Tags: Selenium, Docker, Python, Data Science, Kubernetes

The approach was to develop the web crawler in a Jupyter Notebook on my local machine and to constantly professionalize and increase the project (see Fig 2). For instance, I built a Python application with a dedicated crawler class and all necessary methods based on the Jupyter Notebook scripts. But let us have a more detailed look at the implementation steps.

3. Source Inspection and Packages

3.1 Source inspection

To develop a properly operating web crawler, it is important to familiarize yourself in advance with the site structure, available content, and irregularities.

Fig. 3 — Connection of the relevant entities

TWDS is a classic publication with many authors and a lot of articles. Thanks to an archive page it was easy to understand the page structure in detail (see Fig. 3). Fortunately, the authors were not only listed there but also provided with links that led to overview pages for these authors.

Fig. 4 — Page source-code for authors list on TWDS-Archive

The used HTML class was constantly used so that the links could easily be identified (see Fig. 4).

For the overview pages of the authors, I figured out that at first

only the author’s articles published on TWDS were listed. Other articles published on Medium.com by the author were not displayed. It was therefore not necessary to check whether the specific article belonged to the TWDS publication. Unfortunately, the HTML class for these links was empty and the links could not be identified. However, the links contained the complete URL and thus the word “towards”. Therefore, the identification of these links was just as unambiguous. However, another challenge occurred when examining the page. Not all of the author’s articles were displayed directly, but when the website was scrolled down further content was dynamically reloaded using Javascript. To ensure completeness, this had to be taken into account for the development of the web crawler.

Fig. 5 — Example of the HTML source code from a TWDS-Article

Finally, I had to examine the structure of the individual articles for similarities and patterns to extract the relevant data fields. The required properties were author, URL, title, text, reading time, publishing date, tags, claps and the number of responses. As can be seen in Figure 5, the HTML source code has some challenges. For example, the class names are seemingly dynamically generated and have only minor matches across articles. But there are also rays of hope, e.g. reading time, title, URL, and publishing date are standardized in the page header. The remaining content was reasonably easy to access.

3.2 Package Selection

At first, during development in Jupyter Notebooks, I was looking for Python packages I could use to fulfill all requirements. I quickly realized for Scrapy, one of the most commonly used packages for web scraping, that dynamic content reloading would be difficult. After focusing on this requirement, I became aware of Selenium. Selenium is a framework for automated software testing of web applications and can interact with browsers, e. g. to scroll down pages to load the dynamic javascript content and receive the full HTML source code.

To work with the extracted HTML source code, I found the Python package BeautifulSoup4, which provides various methods to systematically search the HTML tree structure for relevant content. With these packages selected, I could fulfill all the requirements to develop a web crawler.