Serverless Data Pipelines and Recommendation Engines— Snowplow on AWS
By: Dharmil Gandhi
Tags: AWS, Snowplow, Data Pipeline, Kinesis, Recommendation System

In the e-commerce business you want to advertise smart. It also needs powerful analytics tool that can record customer’s interaction with their Website, App or Server and visualize the behavior at customer level. To advertise products smartly implies that it should be real time. In one of our projects we had to feed a recommendation engine by realtime clickstream data.

We decided to use snowplow for data collection. It is an event data pipeline, enabling you to track all events across all channels and action the data in real-time. Snowplow can collect raw clickstream data from the front-end of an application. It has tremendous power to track and record events on Cloud infrastructure.

Primary question that comes to mind considering many data tools available in market - most notably, Google Analytics: Why bother collecting your data on your own when you can use tools like Google Analytics? Isn’t GA easier? Do you need more than that?

Ownership of data : snowplow runs on end users stack under AWS account of the user. Hence, you have complete ownership of data.

Cost effective: snowplow is an open-source tool. Thus you have to only pay for the AWS infrastructure cost.

Highly customizable : you can easily add custom trackers, webhooks, enrichments, data models or metrics that suit your business.

How to set up snowplow:

Snowplow data pipeline has a modular architecture, allowing you to choose what parts you want implement. The main concept of it is an event, which is not limited to web-page views or clickstream data only, but can be pretty much anything.

Each event is processed through multiple stages pipeline of the pipeline — see diagram above.

Trackers are pieces of software that captures/triggers events when they occur and send them to the collector. I decided to implement javascript tracker. See list of available trackers here. Collector are used to collect data from trackers i.e. receiving events from trackers. It is basically a backend code running on AWS infrastructure. I decided to go with Scala stream collector due to it’s capability to stream events using AWS kinesis. See list of available collectors here. Enrichment stage is used to clean up data and to add some extra information to our raw data eg: infer the location of the visitor from his / her IP address. In original snowplow architecture enrichment can be done using this applications. However I tried to make it more customisable and cheaper by enriching data using serverless AWS Lambda functions.

This all looks good but how to implement this data pipeline?

Event Collector:

Launch an EC2 instance to run the collector. Moving forward we will containerise our code and push it to ECS fargate.

Install all the dependencies.

Install Java

$ apt-get update && apt-get install default-jdk

Install Scala



$ dpkg -i scala-2.11.8.deb

$ scala -version $ wget www.scala-lang.org/files/archive/scala-2.11.8.deb $ dpkg -i scala-2.11.8.deb$ scala -version

Install Sbt

$ echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list

$ apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823

$ apt-get -y update && apt-get install -y sbt

2. Compile Scala stream collector from source



$ cd 2-collectors/scala-stream-collector

$ sbt "project *targeted platform*" assembly $ git clone https://github.com/snowplow/snowplow.git $ cd 2-collectors/scala-stream-collector$ sbt "project *targeted platform*" assembly

where targeted platform can be: kinesis, google-pubsub, kafka, nsq, stdout. I have set up collector on AWS infrastructure hence targeted platform is kinesis in my case.

The jar file will be saved as snowplow-scala-collector-[targeted platform]-[version].jar in the [targeted platform]/target/scala-2.11 subdirectory - it is now ready to be deployed.

3. Configuration

Download a template configuration file from GitHub: config.hocon.sample and create a .conf file using this template. Follow this link to configure different options available with scala stream collector.

4. Run collector

Scala stream collector is a jar file. Run following command and provide configuration file as parameter.

$ java -jar snowplow-stream-collector-[targeted platform]-[version].jar --config [config-file].conf

Now, collector should be running and listening to events on public-ip address of EC2 instance. Pinging the collector on the /health path should return a 200 OK response.

$ curl [public-ip of EC2]/health

Trackers

We can install javascript directly into our website. Add following script in head section of your website template.

<!-- Snowplow starts plowing -->

<script type="text/javascript">

;(function(p,l,o,w,i,n,g){if(!p[i]){p.GlobalSnowplowNamespace=p.GlobalSnowplowNamespace||[];

p.GlobalSnowplowNamespace.push(i);p[i]=function(){(p[i].q=p[i].q||[]).push(arguments)

};p[i].q=p[i].q||[];n=l.createElement(o);g=l.getElementsByTagName(o)[0];n.async=1;

n.src=w;g.parentNode.insertBefore(n,g)}}(window,document,"script","//d1fc8wv8zag5ca.cloudfront.net/2.10.2/sp.js","snowplow"));



window.snowplow('newTracker', 'TrackerName', '{{MY-COLLECTOR-URI}}', { // Initialise a tracker

appId: '{{MY-SITE-ID}}',

cookieDomain: '{{MY-COOKIE-DOMAIN}}'

});



window.snowplow('trackPageView');

</script>

<!-- Snowplow stops plowing -->

where {{MY-COLLECTOR_URL}} represents collector endpoint. This ensures that any data generated by the JavaScript Tracker is sent to the Collector. Also, window.snowplow('trackPageView') captures page view event and sends it to collector.

Let’s test if trackers are firing :

Test that the JavaScript tracker is working correctly using Snowplow Inspector, a Chrome extension that exposes a new tab for snowplow beacons inside the developer tools. Follow this link for more information on testing if javascript tracker is firing.

See list of available custom trackers in snowplow here. Snowplow tracker protocol is list of parameters that tracker can use when firing events. See snowplow tracker protocol here.

Enrichment

Event data is sent to kinesis after collection stage. We can send streaming data from kinesis data stream to s3 using kinesis firehose. We can transform our data before push it in S3 using data transformation lambda function.

I have done enrichment using a lambda function to infer location of user from IP address available in raw data. The function is available here on Github. I am using Maxmind Lite database of GeoIP data.

Deploy collector on ECS-fargate

ECS fargate allows us to run Docker containers without having to manage servers or clusters. Also, cost for running collector application would be lowered down as we would not be paying for unused EC2 instance.

Firstly, we need to make a dockerfile with installation of all dependencies and command for running collector jar file.

FROM ubuntu:16.04

MAINTAINER Dharmil Gandhi RUN apt-get update -y

RUN apt-get install -y default-jdk && \

java -version

RUN apt-get install -y wget

RUN wget www.scala-lang.org/files/archive/scala-2.11.8.deb && \

dpkg -i scala-2.11.8.deb && \

scala -version

RUN apt-get update -y && apt-get -y install curl apt-transport-https ca-certificates

RUN echo "deb https://dl.bintray.com/sbt/debian /" | tee -a /etc/apt/sources.list.d/sbt.list && \

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823 && \

apt-get -y update && apt-get install -y sbt

COPY . /app

WORKDIR /app ENTRYPOINT ["java"]

CMD ["-jar", "snowplow-stream-collector-kinesis-0.15.0.jar","--config","collector.conf"]

Build and run docker image

$ docker build -t [container-filename]

$ docker run -d -p 80:80 [container-filename] /bin/bash

Next step is to push docker image to ECR(EC2 container registry). Create a repository in AWS ECR. Tag docker image with the repository URL

$ docker tag [container-name] [repository-URL]

Run the following command to get docker login authentication

$ aws ecr get-login

Remove “-e none” from the output and copy that output command and run it. Now Push tagged docker image to ECR

$ docker push [repository url]:[Tag]

Now, it is very easy to deploy docker image on ECS fargate from ECR. You need to create fargate cluster, create task definition and add container details, create service and follow the steps. The collector will be running and listening to public IP address of ECS fargate. This link helped me to deploy docker image to ECS fargate.