How to create a Selenium web scraper in Azure Functions | by René Bremer | Towards Data Science

Selenium is the standard tool for automated web browser testing. On top of that, Selenium is a popular tool for web scraping. When creating a web scraper in Azure, Azure Functions is a logical candidate to run your code in. However, the default Azure Functions image does not contain the dependencies that Selenium requires. In this blog, a web scraper in Azure Functions is created as follows:
Create and deploy docker image as Azure Function with Selenium
Scrape websites periodically and store results
Update 2021–07–27: Code deployed successfully, git repo and blog are up to date.
The architecture of web scraper is depicted below.
In the remaining the steps are discussed to deploy and run your web scraper in Azure Functions. For details how to secure your Azure Functions, see this blog. For details how to create a custom docker image with OpenCV in Azure Functions, see here and DockerFile here.
The base Azure Function image does not contain the necessary chromium packages to run selenium webdriver. This project creates a custom docker image with the required libraries such that it can be run as Azure Function. The following steps are executed:
B01. Install prerequisites
B02. Clone project from GIT
B03. Create docker image using docker desktop
B04. Create Azure Function and deploy docker image
See also architecture below.
The following prerequisites need to be installed:
Docker desktop
Azure CLI
Azure Core Tools version 2.x
(optional) Visual Studio Code
(optional) Azure Container Registry (docker hub can also be used)
Run the command below to clone the project from git. In case you did not install git, the zip file can also be downloaded and extracted.
git clone https://github.com/rebremer/azure-function-selenium.git
In this project, the following files can be found:
TimeTrigger/__init__.py: Python file that contains all code to scrape websites. This Azure Function is time triggered
HttpTrigger/__init__.py: Same as previous bullet, however, this is function HTTP triggered and can be run from a browser..
DockerFile: File that contains all commands to create Docker image that will be used in the next step
Run the following commands that installs chromium, chrome driver and selenium on top of the Azure Function base image. Commands can be run in Visual Studio terminal or PowerShell terminal. For the <<username>>, <<password>>, you need to take the credentials of your container registry that can be found in the access key tabs.
# Variables$acr_id = "<<your acr>>.azurecr.io"# Create docker image using docker desktopdocker login $acr_id -u <<your username>> -p <<your password>>docker build --tag $acr_id/selenium .# Push docker image to Azure Container Registrydocker push $acr_id/selenium:latest
Run the following commands to create an Azure Function and deploy the docker image from Azure Container Registry.
# Variables$rg = "<<your resource group name>>"$loc = "<<your location>>"$plan = "<<your azure function plan P1v2>>"$stor = "<<your storage account adhering to function>>"$fun = "<<your azure function name>>"$acr_id = "<<your acr>>.azurecr.io"# Create resource group, storage account and app service planaz group create -n $rg -l $locaz storage account create -n $stor -g $rg --sku Standard_LRSaz appservice plan create --name $plan --resource-group $rg --sku P1v2 --is-linux# Create Azure Function using docker imageaz functionapp create --resource-group $rg --os-type Linux --plan  $plan --deployment-container-image-name $acr_id/selenium:latest --name  $fun --storage-account $stor --docker-registry-server-user <<your acr user>> --docker-registry-server-password <<your acr password>>
The Azure Function that was deployed in the previous step contains a time triggered function and an HTTP trigger function. In this part, the function will be triggered, scrape websites and store results to a data lake account. The following steps are executed:
B11. Create data lake account
B12. Run HTTP trigger Functions
See also architecture below.
Execute the following commands to create a data lake account in Azure and update the settings of the functions.
# Variables$rg = "<<your resource group name>>"$fun = "<<your azure function name>>"$adls = "<<your storage account>>"$sub_id = "<<your subscription id>>"$container_name = "scraperesults"# Create adlsgen2az storage account create --name $adls --resource-group $rg --location $loc --sku Standard_RAGRS --kind StorageV2 --enable-hierarchical-namespace trueaz storage container create --account-name $adls -n $container_name# Assign identity to function and set paramsaz webapp identity assign --name $fun --resource-group $rgaz functionapp config appsettings set --name $fun --resource-group $rg --settings par_storage_account_name=$adls par_storage_container_name=$container_name# Give fun MI RBAC role to ADLS gen 2 account$fun_object_id = az functionapp identity show --name $fun --resource-group $rg --query 'principalId' -o tsvNew-AzRoleAssignment -ObjectId $fun_object_id -RoleDefinitionName "Storage Blob Data Contributor" -Scope  "/subscriptions/$sub_id/resourceGroups/$rg/providers/Microsoft.Storage/storageAccounts/$adls/blobServices/default"
The time triggered function will run periodically to scrape website. However, there is also a HTTP triggered function. When the URL is taken, it can be copied in the browser and run, see also below.
Once the function is run, the results are stored in the data lake account, see also below.
Selenium is a popular tool for web scraping. However, the default Azure Functions image does not contain the depencencies that Selenium requires. In this blog, a web scraper in Azure Functions is created that installs these dependencies as follows:
Create and deploy docker image as Azure Function with Selenium
Scrape websites periodically and store results
Architecture of web scraper is depicted below.