Automating a Machine Learning Workflow using Google BigQuery and Amazon Managed Apache Airflow
By: Yi Ai
Tags: Big Data, Machine Learning, Airflow, AWS, Google Big Query

Automating a Machine Learning Workflow using Google BigQuery and Amazon Managed Apache Airflow

Using BigQuery, Airflow, and Amazon Personalize to Build a machine learning workflow

Amazon announced the availability of Amazon Managed Workflows for Apache Airflow (MWAA), a fully managed service that makes it easy to run Apache Airflow on AWS and to build data processing workflows in the cloud.

Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows”.

This article shows how we can build and manage an ML workflow using Google BigQuery, Amazon MWAA, and Amazon Personalize. We’ll build a session-based recommender system to predict the most popular items for an e-commerce website based on the traffic data of the product pages tracked by Google Analytics.

Amazon Personalize enables developers to build applications with the same machine learning (ML) technology used by Amazon.com for real-time personalized recommendations — no ML expertise required. BigQuery is an enterprise data warehouse that solves this problem by enabling super-fast SQL queries using the processing power of Google’s infrastructure.

High-level solution

We’ll start by extracting the data, transforming the data, building, training, deploying a solution version (a trained Amazon Personalize recommendation model), and deploying a campaign. These tasks will be plugged into a workflow that can be orchestrated and automated through Apache Airflow integration with Amazon Personalize and Google BigQuery.

The diagram below represents the workflow we’ll implement for building the recommender system:

Architecture diagram by Yi Ai

The workflow consists of the following tasks:

Data preparation

Export session and hit data from a Google Analytics 360 account to BigQuery, use SQL query analytics data into Pandas data frame with Personalize format, and then write data frame to CSV file directly to S3.

Amazon Personalize solution

Create a Personalize dataset group if it doesn’t exist.

Create an Interaction schema for our data if the schema doesn’t exist.

Create an ‘Interactions’ dataset type if it doesn’t exist.

Attach an Amazon S3 policy to your Amazon Personalize role if it doesn’t exist.

Create a Personalize role that has the right permissions if it doesn’t exist.

Create your Dataset import jobs.

Create / Update Solution.

Create/ Update Campaign.

Before implementing the solution, you have to create an Airflow Environment Using Amazon MWAA;

“extra packages” should be included while creating an environment; please don’t include BigQuery Client in the below requirements.txt, we will install the BigQuery Client in the next step:

boto >= 2.49.0 httplib2 awswrangler google-api-python-client

When the new Airflow environment is ready to be used, attach the Personalize policies to the IAM role of your environment, run CLI as below:

$aws iam put-role-policy — role-name AmazonMWAA-MyAirflowEnvironment-em53Wv — policy-name AirflowPersonalizePolicy — policy-document file://airflowPersonalizePolicy.json

Installing Google Bigquery Client

MWAA currently doesn't support Google Cloud Bigquery client ( google-cloud-bigquery ) and pandas-gbq with grpc > 1.20 , We are not able to install Bigquery Client through requirements.txt . if you put the above dependencies into requiremnts.txt, pip installation won’t install any dependencies in requirements.txt , and you will meet error No module named “httplib2” when running DAG.

To resolve this issue, we can 1)package the required Google libraries in local computer and upload to S3, 2)and then download them to Airflow workers when the bigquery export task started, 3)after that we can dynamically import required modules given the full file path.

I created a bash file and requirements.txt for the above steps; run the following command:

$bash setup.sh

setup.sh

requirements.txt

then, copy the following code to the DAG task to import google modules dynamically :

Next, We will create Google Cloud Connection in Airflow Airflow UI.

Now, we will use Google BigQuery in Amazon Managed Airflow workers; let’s begin to create workflow tasks.

Data preparation

First, export session and hit data from a Google Analytics 360 account to BigQuery, use SQL to query Analytics data into Pandas data frame with Personalize format.

To prepare an interaction dataset for Personalize, we need to extract the following data from BigQuery and Google Analytics:

USER_ID In this example, we don’t have user data of the e-commerce website, and there is no user interaction data from the website database. However, we can use the client id provided by Google Analytics. The client id ( cid ) is a unique identifier for a browser–device pair that helps Google Analytics link user actions on a site. By default, Google Analytics determines unique users using this parameter. The client ID format is a randomly generated 31-bit integer followed by a dot ( “.” ) followed by the current time in seconds. Hence we only need the 31-bit integer before the dot. BigQuery provides regular expression support, which we can put REGEXP_EXTRACT(USER_ID, r’(\d+)\.’) AS USER_ID in Big Query to extract the session id as session-based User Id.

In this example, we don’t have user data of the e-commerce website, and there is no user interaction data from the website database. However, we can use the client id provided by Google Analytics. The client id ( ) is a unique identifier for a browser–device pair that helps Google Analytics link user actions on a site. By default, Google Analytics determines unique users using this parameter. The client ID format is a randomly generated 31-bit integer followed by a dot ( ) followed by the current time in seconds. Hence we only need the 31-bit integer before the dot. BigQuery provides regular expression support, which we can put in Big Query to extract the session id as session-based User Id. ITEM_ID Google Analytics provides page location ( page_location ), so we can extract product pages (product slug) by WHERE Clause page_location LIKE ‘%/product/%’ and make it as Item Id.

Google Analytics provides page location ( ), so we can extract product pages (product slug) by WHERE Clause and make it as Item Id. TIMESTAMP , timestamp data must be in UNIX epoch time format, use TIMESTAMP_TRUNC(TIMESTAMP_MICROS(event_timestamp) to convert Analytics event_timestamp to correct format.

, timestamp data must be in UNIX epoch time format, use to convert Analytics to correct format. device.category AS DEVICE .

AS . geo.country AS LOCATION .

AS . event_name AS EVENT_NAME .

SQL query in BigQuery as below:

Next, write the data frame to the CSV file directory to S3 using AWS DataWragler.

The following PythonOperator snippet in the DAG defines BigQuery to S3 task.

Creating a recommendation model with Amazon Personalize

In this section, we will build a Personalize solution to identify the most popular items for an e-commerce website integrated with Google Analytics.

We will use Popularity-Count Recipe for training our model. Although Personalize supports importing interactions incrementally, we will retrain the model base on daily interaction data to get more relevant recommendations.

What we’ll cover:

check_s3_for_key ( S3KeySensor ): check if the dataset CSV file exists.

( ): check if the dataset CSV file exists. t_check_dataset_group ( BranchPythonOperator ): check if the Personalize dataset group exists. If Yes, trigger t_init_personalize , else trigger t_skip_init_personalize .

( ): check if the Personalize dataset group exists. If Yes, trigger , else trigger . t_init_personalize ( DummyOperator ): trigger parallels tasks if dataset group doesn't exist( t_create_dataset_group,t_create_schema,t_put_bucket_policies,t_create_iam_role) .

( ): trigger parallels tasks if dataset group doesn't exist( . t_create_dataset_group ( PythonOperator ): Create a Personalize dataset group if it doesn’t exist.

( ): Create a Personalize dataset group if it doesn’t exist. t_create_schema ( PythonOperator ): Create an Interaction schema for our data if the schema doesn’t exist.

( ): Create an Interaction schema for our data if the schema doesn’t exist. t_put_bucket_policies ( PythonOperator ): Attach an Amazon S3 policy to your Amazon Personalize role if it doesn’t exist.

( ): Attach an Amazon S3 policy to your Amazon Personalize role if it doesn’t exist. t_create_iam_role ( PythonOperator ): Create a Personalize role that has the right permissions if it doesn’t exist.

( ): Create a Personalize role that has the right permissions if it doesn’t exist. t_create_dataset_type ( PythonOperator ): Create an ‘Interactions’ dataset type if it doesn’t exist.

( ): Create an ‘Interactions’ dataset type if it doesn’t exist. t_skip_init_personalize ( DummyOperator ): Downstream task of BranchOperator task.

( ): Downstream task of BranchOperator task. t_create_import_dataset_job ( PythonOperator ): Create your Dataset import jobs.

( ): Create your Dataset import jobs. t_update_solution ( PythonOperator ): Create / Update Solution.

( ): Create / Update Solution. t_update_campagin (PythonOperator): Create/ Update Campaign.

DAG tasks for Personalize workflow

In the next section, we’ll see how all these tasks are stitched together to form a workflow in an Airflow DAG.

Defining DAG

Different tasks are created in the above sections using operators like PythonOperator for generic Python code to run on-demand or at a scheduled interval.

Now let’s set DAG with parameters; a DAG is simply a Python script that contains a set of tasks and their dependencies.

Next, specify task dependencies:

After triggering the DAG on-demand or on a schedule, we can monitor DAGs and task executions and directly interact with them through Airflow UI.

In the Airflow UI, we can see a graph view of the DAG to have a clear representation of how tasks are executed: