Web scraping @edge, AWS Kinesis Data Firehose & Glue
By: Vittorio Nardone
Tags: Aws Glue, Web Scraping, Amazon Kinesis Firehose, Jupyter Notebook, Raspberry Pi

Photo by Hybrid on Unsplash

What is “Web scraping @edge” or “Intranet web scraping”? The ability to extract data from a private website, i.e. from an IoT resource or device that is not publicly reachable. Why should we care? To collect data available only privately and to be able to process them (on the cloud at scale) in order to produce information and insights.

The purpose of this project is the data acquisition from motion detection sensors usually installed for home alarm system. It’s a closed and proprietary system, for obvious security reasons. A Raspberry PI is used in the local network to scrape the UI of Paradox alarm control unit and to send collected data in (near) realtime to AWS Kinesis Data Firehose for subsequent processing. After few weeks of data collected, I played on a Notebook to identify the most used rooms or the most frequent movement vectors.

Let’s see the general scheme.

Image by Author

Project repository is available on Github.

Web scraping with Raspberry PI and Selenium

Having already experience using Selenium, I decided to adopt the same framework in this project as well. Compared to the past, the challenge was to use it on Raspberry PI, an option that led to the adoption of Firefox instead of Chromium, no longer supported on ARM architecture.

The web page of the Paradox control panel (actually its IP150 lan module) looks like the following image.

Image by Author

Our purpose is to capture the status of each zone — or motion sensor — in order to collect historical data on the cloud. Since the Paradox system is a closed system that does not foresee the possibility of integration with other solutions, web scraping in polling mode is my only available way.

How to login? As mentioned, we will use Selenium and Firefox in a Python script to automate the operation. Here is an example of the code used.

from selenium.webdriver.common.keys import Keys

from selenium.webdriver.firefox.options import Options

from selenium import webdriver

import os

from time import sleep

from pyvirtualdisplay import Display



def paradox_login():

# Set screen resolution

display = Display(visible=0, size=(1366, 768))

display.start()



# Open Firefox

options = Options()

options.headless = True

driver = webdriver.Firefox(

executable_path='/usr/local/bin/geckodriver',

options=options)



# Open homepage

ipaddress = os.environ['PARADOX_IPADDRESS']

print('Opening Paradox at address: {}'.format(ipaddress))

driver.get('http://{}'.format(ipaddress))

print('Waiting a moment..')

sleep(5)



#Login

usercode, password = os.environ['PARADOX_USERCODE'],

os.environ['PARADOX_PASSWORD']

print('Login..')

user_elem = driver.find_element_by_id("user")

pass_elem = driver.find_element_by_id("pass")

user_elem.clear()

user_elem.send_keys(usercode)

pass_elem.clear()

pass_elem.send_keys(password)

pass_elem.send_keys(Keys.RETURN)



if __name__ == "__main__":

paradox_login()

Once logged in, we can capture the status of the sensors.

driver.get('http://{}/statuslive.html'.format(ipaddress))

Periodically making HTTP / GET requests like the one shown, we will be able to get the HTML page to be parsed to extract the status information. I don’t dwell on the implementation of parsing, a simple Python string extraction exercise. The entire source code is available in the project repository.

Assuming we have a Python dictionary (firehose_record) with name and activation status of each sensor, we will be able to send it in JSON format to an AWS Kinesis Data Firehose stream previously configured to simply store the contents in an S3 bucket.

import boto3

import json



producer = boto3.client('firehose')



response = producer.put_record(

DeliveryStreamName=os.environ['KINESIS_STREAM'],

Record={

'Data': json.dumps(firehose_record) + '

'

}

)

Run in a Docker container

To conclude, I have chosen to run the scraping script in a docker container: starting from the Python 3.8 image for ARM32 architecture, it’s required to install Firefox, Selenium and the related Gecko driver, as well as other dependencies.

The dockerfile used in the project is shown below.