Scraping the web with Selenium on Google Cloud Composer (Airflow) | by Julian Smidek | Towards Data Science

There are already a lot of different resources available on creating web-scrapers using Python which are usually based on either a combination of the well known Python packages urllib+beautifulsoup4 or Selenium. When you are faced with the challenge to scrape a javascript-heavy web page or a level of interaction with the content is required that can not be achieved by simply sending URL requests, then Selenium is very likely your preferred choice. I don’t want to go into the details here on how you can set-up your scraping script and the best practices on how to run it in a reliable way. I just want to refer to this and this resources that I found are particularly helpful.
The problem that we want to solve in this post is: How can I, as a Data Analyst/Data Scientist, set up an orchestrated and fully managed process to facilitate a Selenium scraper with a minimum of dev-ops required? The main use case for such a set up is a managed and scheduled solution to run all your scraping jobs in the cloud.
The tools we are going to use are:
Google Cloud Composer to schedule jobs and orchestrate workflows
Selenium as a framework to scrape websites
Google Kubernetes Engine to deploy a Selenium remote driver as containerized application in the cloud
At HousingAnywhere we were already using Google Cloud Composer for a number of different tasks. Cloud Composer is quite an amazing tool to easily manage, schedule and monitor workflows as directed acyclic graphs (DAGs). It is based on the open-source framework Apache Airflow and using pure Python, which makes it ideal for everyone working in the data field. The entry barrier to deploy Airflow on your own is relatively high if you are not coming from DevOps which led to some cloud providers to provide managed deployments of Airflow — Google’s Cloud Composer being one of them.
When deploying Selenium for webscraping, we’re actually using the so-called Selenium Webdriver. This WebDriver is a framework that allows you to control a browser using code (Java, .Net, PHP, Python, Perl, Ruby). For most use-cases you would simply download a browser that can directly interact with the WebDriver framework, for example Mozilla Geckodriver or ChromeDriver. The scraping script will initiate a browser instance on your local and execute all actions as specified. In our use case things are a bit more complicated because we want to run the script on a recurring schedule without using any local resources. To be able to deploy and run web scraping scripts in the cloud we need to use a Selenium Remote WebDriver (a.k.a Selenium Grid) instead of the Selenium WebDriver.
The idea behind Selenium Grid is to provide a framework that allows you to run parallel scraping instances by running web browsers on a single or multiple machines . In this case, we can make use of the provided standalone browsers (keep in mind that each of the available browsers, Firefox, Chrome and Opera are a different image) which are already wrapped up as a Docker image.
Cloud Composer runs Apache Airflow on top of a Google Kubernetes Engine (GKE) cluster. Furthermore, it is fully integrated with other Google Cloud products. The creation of a new Cloud Composer environment also comes along with a functional UI and a Cloud Storage bucket. All DAGs, plugins, logs and other required files are stored in this bucket.
You can deploy a docker image for the Firefox standalone browser using the selenium-firefox.yaml file below and apply the specified configuration on your resource by running:
kubectl apply -f selenium-firefox.yaml
The configuration file describes what kind of object you want to create, it’s metadata as well as specs.
We can create new connection in the Admin UI of Airflow and access the connection details later in our Plugin. The connection details are either specified in the yaml file or can be found on your Kubernetes cluster.
After setting up the connections we can access the connection in our scraping script (Airflow Plugin) where we connect to the remote browser.
Thank you Massimo Belloni for technical consultancy and advice in realizing the project and this article.