Build a Data Collection App on the Cloud for Your Next Time Series Data Science Project
By: Kevin C Lee
Tags: Heroku, Scheduler, Time Series Analysis, Sqlalchemy, Python

Build a Data Collection App on the Cloud for Your Next Time Series Data Science Project

Note: The code for this post can be found here

Scheduled Data Collection Application Workflow (image by Author)

In spite of the Data Proliferation observed in recent years, too often do we find ourselves being roadblocked by the lack of data for a project. Throughout a typical day of web browsing, we come across countless pieces of information (data) that hold potential for great Data Science ideas such as current weather, traffic, disaster warning, stock prices, vote counts, covid counts, view counts, social media comments — the list goes on. We know that the data is maintained and made available for public consumption, but a convenient compilation is rarely available. In this article, we will walk through step-by-step the development and deployment of a data collection application with RDMS (Relational Database Management System) in Python. Realizing that individual scripts can be drastically different depending on the data being collected, we take on the approach of walking through a simple example while providing adequate reasoning behind each step:

Development Steps

This demonstration application will consist of a script that collects a snapshot of publicly available data and a scheduler that calls the collection script periodically. The application can be optionally deployed on the cloud, which provides reliability and availability (and other benefits). We chose Heroku for deployment, which is a platform as a service (PaaS) that enables developers to build, run, and operation application entirely in the cloud.

Define Goals and Objectives

There are many different ways to acquire data: API requests, web scraping, and everything in between. The first step in the project is to identify a reliable data source. Our inspiration comes from Open Data, which is the idea that some data should be freely available for everyone to use as they wish. Its goals are similar to those of other “open(-source)” movements such as open-source software. Interestingly, the Public Sector at the federal and local levels in the U.S. have made tremendous progress towards data transparency. We demonstrate using data from 511.org. They provide a convenient API for current traffic events (see documentation). Registration is required for an API Key and requests are subject to a rate limit by the issuer. Once received, the response comes in JSON format, which we can use to extract useful data. We are interested in capturing individual traffic events with only the below attributes. By developing an automated script, we hope to collect a compilation of all traffic events.

one_record={'id': '511.org/721616'

'headline': 'Caltrans Bridge work...',

'event_type': 'CONSTRUCTION',

'severity': 'Unknown',

'lat': -122.241176,

'lon': 38.483586}

Create a GitHub Repository and Set Up Virtual Environment

Git is a free and open source distributed version control system. It tracks all changes made to files in the repository where the project lives. GitHub is a Git repository hosting platform that allows developers to build software for open-source as well as private projects. Many platforms, including Heroku, enable integration with GitHub to make it easy to deploy code.

More information about Git and basic usage can be found here. We’ll create a remote repository for this project on GitHub and clone it to our local machine using $ git clone "https://github.com/[github_name]/[repo_name].git” . Change the directory with $ cd [repo_name] to ensure commands are done with respect to the new repository as the root.

Next we will create a virtual environment for the project. The main purpose of virtual environments is to create isolated environments for projects. This means that each project can have its own dependencies, regardless of what dependencies other projects have. By the same token, virtual environments are important when developing applications that have specific requirements.

Virtual Environments

Create a new environment with $ conda create --name env_name python=3.6 and activate it with $ conda activate env_name . Note that we specified the new environment with python=3.6 to include pip (package installer for Python added in 3.4) and f strings (formatted string literal added in 3.6). This environment should be activated when working in this project directory/repository for application installations and running commands. Since this is a new environment, all of the dependencies will need to be installed via $ pip install before they can be imported. Use $ conda deactivate to deactivate the virtual environment.

We also need to set up environment variables for the application. Environment variables are considered part of the operating system that exist outside of any particular script. We can use environment variables to hide sensitive credentials such as the 511 API Key. They can be set at the host when we deploy, but also need to be available locally for development. These variables should be limited to the scope of the project so what we really need is are “temporary” environment variables. Fortunately, we can use python-dotenv to read in key-value pairs from a .env file and set them as environment variables at runtime. This allows us to develop our scripts as if the environment variables has been set. The .env file should be added to .gitignore so it’s never included in git commits locally or remotely.

$ echo "FIVEONEONE_TOKEN=[api_key]" > .env

$ echo ".env" >> .gitignore

Environment Variables are More Secure Than Scripts

Develop Data Collection Script and Database

In this step, we will create a script to fetch data once from the source. In this demonstration we make a request to the designated endpoint with the assigned api_key.

fetch.py

We implement a conditional check on the successfulness of the request. Once received, the response contains the data in JSON format. Specifically we’re interested in the list of active traffic events in the events key of the JSON. We can test this script with $ python fetch.py after installing the necessary dependencies:

$ pip install requests

$ pip install python-dotenv

Next we want to save the data in a database, but we have to do a bit of database engineering first. During this step we want to give considerable thought into the size of the data and how the data will be used. Generally there are two types of databases: SQL and NoSQL. In comparison, SQL databases require more effort to set up but are suitable when the data has a defined schema. As it relates to our data — given that we’ve already identified the subset of attributes we want to collect, it makes sense to use an SQL database. We’re going for an “Events” table where each Row represents one event and each Column represents an attribute of interest.

Sample Events Table on an SQL Database

In other words, starting with the list of traffic events in the JSON we receive, we’ll iterate through each elements and parse out the attributes we’re interested in. We can work with SQL in Python using the SQLAlchemy library. SQLAlchemy is a Python SQL toolkit and Object Relational Mapper that enables the full power of SQL.

Before we can store the records into a database, we have to define the table schema. This schema is mapped to a Python class using Declarative so each records will be represented as its instance.

models.py

We lay out the schema of the database table(s) in a separate script — models.py , which we can call at a later point for the purpose of creating the table(s) and records. Important to note that SQLAlchemy ORM requires a primary key for each table.

The fetch.py script has been modified as fetch_insert.py to 1) connect to the database, 2) create table(s) based on models.py , 3) iterate through each element from the response/parse out attributes of interest, and 4) create an instance of the mapped Python class to be inserted into the database.

fetch_insert.py

We make models.py a dependency (as in from models import * ) for loading the schema(s) to create tables and records.

Working with SQLAlchemy starts with establishing a connection at the database url. We wrote the script to connect to a single file SQLite database when there is no DATABASE_URL environment variable present. We’re using SQLite for development and backup purposes since it’s self-contained and full-featured. A DB server such as Postgres is often preferred as it supports availability and reliability. Once we add the production database url as an environment variable, it takes precedent for making a database connection. We should treat the database url as an environment variable since it includes username, password, hostname, and database name.

The typical usage of sqlalchemy.create_engine() and Base.metadata.create_all() is once per particular database URL, held globally for the lifetime of a single application process. A single Engine manages many individual DBAPI connections on behalf of the process and is intended to be called upon in a concurrent fashion. If the table(s) already exist, SQLAlchemy will not attempt to recreate them.

Once the response is returned from the request, we iterate through the list of events to gather the data for the table records. Records are added and manipulated in SQLAlchemy as instances of the Python class that is mapped to the SQL Table. The ** operator allows us to take a dictionary of key-value pairs and unpack it into keyword arguments for the Events() constructor.

Once the records are added and committed, we use session.close() to give back the resources to the connection pool of Engine. This should not be confused with engine.dispose() , which closes all connections of the connection pool.

At this point we’ve built a script to collect data and store records into an SQL database. We can test the application with $ python fetch_insert.py (after installing the dependencies), which will create a new SQLite file as the database if it doesn’t exist already.

$ pip install sqlalchemy

$ pip install psycopg2

Adding a Scheduler

We can schedule the data fetching process to run on intervals — enabling us to continuously collect data. The APScheduler (Advanced Python Scheduler) is a Python library that lets users schedule Python code to be executed later, either just once or periodically. It’s easy to use and integrated with other useful frameworks.

scheduler

Conveniently, Heroku uses APScheduler in their implementation of Custom Clock Process. In their documentation, they explain that there are mainly four components that facilitate the scheduling of jobs. In the main program, users chooses a scheduler and a job store. The scheduler is responsible for adding jobs to the job store based on the trigger and the executor(s) handles running the jobs.

clock.py

Scheduling jobs is easy with APScheduler — our basic implementation shown in clock.py starts by defining a scheduler and jobs store ( BlockingScheduler() and MemoryJobStore() as default) which act as the main procedure running the application. Next the job(s) are added with the specific trigger that contains the scheduling logic. We did this by decorating a function with scheduled_job() and the interval trigger that calls the data collection script. Technically, the process should be described as re-execution after a calculated delay. To manage the connections with the database, we create the engine connection in the main program and pass it to the collection script. We have to refactor the fetch.py script once again to fetch_job.py to accommodate this implementation.

fetch_job.py

The data collection script is wrapped around a function fetch_data so it can be called easily and accepts the database connection as a parameter. For each scheduled “fetch” job, a session is made with the database until all insertions are completed before closing. We can test this application with $ python clock.py after installing the new dependency.

$ pip install apscheduler

Set Up Heroku Application and Add Configuration Files

At this point, the data collection application “automatically” executes at fixed intervals, but only for as long as the application is active locally. We can deploy the application to the Heroku to remove this dependency.

This demonstration avoids using the Heroku Command Line Interface (CLI) to keep deployment simple. To start, create a new application on Heroku. Once logged in(#1), move on to “Create new app” (#3) and give the application a unique name to complete the creation. Heroku offers convenient integration with GitHub to seamlessly load code for deployment. Once authenticated (#5), a developer can search by name (#7) and connect the application’s source code to the corresponding repository (#8).

Setting Up Heroku App

We have to update the source code on GitHub before we deploy, but there are a couple of setting we have to configure first: Postgres Database, Config Vars, requirements.txt, and Procfile.

Postgres Database Add-on for Heroku Application

Heroku offers a Postgres Add-on for free (with limitations) — a separate database would be better than the SQLite local database used for development as it enables access outside of the application.

On Heroku, go to the Resources section of the application’s homepage (#1). Under Add-ons, search for “Heroku Postgres” (#2 & #3). Next select the appropriate plan (the Hobby Dev — Free plan offers up to 10k rows of data) and Provision.

Heroku-Postgres Add-on

Heroku Configuration

We have to set the API Key and Database URL as environment variables to prevent them from being exposed. From the Dashboard page, navigate to setting (#1) and click on “Reveal Config Vars” (#2). The connection string to the database is automatically added under DATABASE_URL (#3). This can be used to connect to the Database (outside of this project). We have to manually add the api_key (#4) as an environment variable (#5).

Setting Up Config Vars

To deploy the application to Heroku, there are two more files needed: 1) requirements.txt and 2) Procfile . These will help Heroku configure the server that is hosting the application. We have to manually create these files in the repository and they must live in the root so Heroku can find them during the deployment process.

requirements.txt —lists all the dependencies and packages required to run the application. To create the requirements.txt file, run the $ pip freeze > requirements.txt command in the project directory. This will write a list of all libraries installed in the active environment to the requirements.txt file. This helps Heroku install the same dependencies for the application. Procfile — tells Heroku how to run the application and declares what commands should be executed to start the application. To create the Procfile file, run $ echo "clock: python clock.py" > Procfile in the project directory. Notice that Procfile is a simple text file without a file extension.

The Procfile declares the applications process types on individual lines, each with the format <process type>: <command> where the process type is the prototype from which a dyno is instantiated. Dynos are containers that provide computer, memory, an operating system, and an ephemeral filesystem designed to execute code based on a user-specified command.

Given the simplicity of the demonstration application, a single clock dyno is sufficient. Note that either a clock or worker process type declaration would be fine in the Procfile for our use case. The choice of using clock process type is to signify clock.py as a scheduler application, which is a singleton process and should never be be scaled to more than one to prevent jobs from duplicated.

P.S. In our implementation, the Dyno does both the scheduling and the execution of the jobs. Depending on the complexity of the data collection script and frequency of jobs, separate worker dynos might need to be used (out of scope for this project).

A Single Clock Dyno Is Sufficient for Simple Jobs

Correct Postgres Database URL (Edited in March)

Since SQLAlchemy deprecated the postgres dialect in March’21, we have to manually update Heroku’s DATABASE_URL environment variable. Unfortunately, this cannot be updated at the Heroku operating system level since it was generated by the Add-on. Instead, we can add a simple String.replace() statement to clock.py accommodate this change.

SQLALCHEMY_DATABASE_URI=SQLALCHEMY_DATABASE_URI.replace('postgres://', 'postgresql://')

Deploy to Heroku

Once both configuration files (Procfile and requirements.txt) have been created, save all working files in the repository then add/commit/push them to the remote repository.

$ git add .

$ git commit -m 'ready to deploy'

$ git push

Once the code has been pushed to GitHub, we can deploy it in Heroku. Within the Deploy page (#1), choose the branch (#2) to deploy (#3). Heroku will then begin to build the application.

Deploy

The last thing we have to do is to “scale the dyno” in Heroku. Back at the Resources page (#1), the clock dyno has been recognized now that the application is deployed. It needs to be and remain scaled up in order to stay running and determine suitable times to schedule a job (#2, #3, and #4).

Scale Clock Dyno to 1

Summary

In this article we walked through step-by-step the creation of a data collection application. We also deployed it to the cloud using Heroku. I hope this was helpful to you. My hope is to motivate more people to share their ideas (and data) that they’ve worked hard to curate for others to collaborate.