Apache Beam-Python on Google Cloud Platform (GCP) -Building and Running data pipelines-A step by step Guide
By: Annapoorani K
Tags: Data Pipeline, Data Engineering, Python, Google Cloud Platform, Apache Beam

This article focusses on:

A) Creating a project in GCP

B) Setting up the project environment

C) Build a data pipeline using Apache Beam and Python

D) Run the pipeline

E) Additional Notes

A) Create a project in GCP

1. Log in into the GCP Console with your gmail account. You can use the Google Cloud 90 day free trial and $300 credit for new customers

2. Create a new project. Enter project name and click Create. Note down the project ID(you can edit the project id by clicking on edit button below project name), under the Project name. The ID is used to identify your project to the Google Cloud tools so it is better to make it readable.

As you can see, my project id is gcpproject-id

3. Enable Billing account

4. Click the Activate Cloud Shell button at the top of the Google Cloud Console. Refer https://cloud.google.com/shell/docs/using-cloud-shell, if needed. Make sure the selected project is the one you want to work on.

Use the drop down button to select your project

5. Create a service account. From the Service account list, select New service account. In the Service account name field, enter a name. From the Role list, select Project > Owner. Download json file and save it, we will need it later. I have renamed it as gcpdataflowkey.json

6. In the Cloud Storage Browser page. Click Create bucket. Specify the following attributes: Name, Default storage class (standard) and location and click on Create

B) Setting up the project environment

To launch a Cloud Shell session from the Cloud Console, use the Activate Cloud Shell button in your Console. In the cloud shell, assign a PROJECT variable to your project ID

export PROJECT=gcpproject-id

2. From the Cloud Shell terminal you can run the following command to launch the Cloud Shell Editor :

cloudshell edit .

You can also click on the Open Editor button with the pencil icon on the toolbar of the Cloud Shell window to launch the editor.

More about it is here : https://cloud.google.com/shell/docs/launching-cloud-shell-editor

3. Create a directory (folder) for your project (gcpdataflow, in my case) and a python file (bestsellers.py in my case) inside that directory, either from the editor by accessing File menu or from the cloud shell terminal using the following commands

mkdir gcpdataflow

cd gcpdataflow/

touch bestsellers.py

4. Virtual Environment:

Virtualenv lets you create and manage virtual Python environments. More about it is here: https://dont-be-afraid-to-commit.readthedocs.io/en/latest/virtualenv.html

Run the following commands inside your project directory(gcpdataflow, in my case, to set up virtual environment) in the cloud shell terminal

python3 -m pip install — user venv

virtualenv -p python3 venv

source venv/bin/activate

pip install 'apache-beam[gcp]'

Note: venv is the name I have given to my virtual environment. You can substitute it with yours.

5. Navigate to Editor and upload the service account key (created in 5th point under Create a project in GCP) to your project folder.

Below is my folder structure for comparison:

C) Build a data pipeline using Apache Beam and Python

In this article, I am using kaggle dataset: https://www.kaggle.com/sootersaalu/amazon-top-50-bestselling-books-2009-2019 .This is a dataset on Amazon’s Top 50 bestselling books from 2009 to 2019., it contains 550 books, and has been categorized into fiction and non-fiction using Goodreads.

Download this csv and do data cleaning (I have replaced the comma’s in the Name column with semicolons) . Upload this csv in the bucket (created in the 6th point of Create a project in GCP). You can find the cleaned csv in my github (link given in additional notes)

The snapshot of the csv is here:

PROBLEM STATEMENT:

Find the average user rating for Fiction books, Non fiction books and for all books together and write the result in separate files Sort the books in alphabetical order and the last 5 books and print the output

Import Libraries and JSON Key

Set GOOGLE_APPLICATION_CREDENTIALS environment variable in Python code to the path key.json file

from __future__ import absolute_import

import argparse

import logging

import re

import apache_beam as beam

from apache_beam.options.pipeline_options import PipelineOptions

import os # set GOOGLE_APPLICATION_CREDENTIALS environment variable in Python code to the path key.json file os.environ[‘GOOGLE_APPLICATION_CREDENTIALS’] = “/home/annapooraniks/gcpdataflow/gcpdataflowkey.json”

Create/Configure a pipeline

A pipeline represents the entire data processing task, from start to finish. Pipeline = Reading input + Transforming Data + Writing Output

Create an instance of the Beam SDK class Pipeline. When you construct your PipelineOptions object, set configuration options to read options from the command-line as follows:

# instantiate the pipeline

options = PipelineOptions()

with beam.Pipeline(options=options) as p:

To build own custom options in addition to the standard PipelineOptions, use the add_argument() method

# defining custom arguments - To add your own options, use the add_argument() method # specify a description which appears when a user passes --help as a command-line argument, and a default value. class MyOptions(PipelineOptions):

@classmethod def _add_argparse_args(cls, parser): parser.add_argument('--input',

help='Input for the pipeline',

dest='input',

required=False,

default='gs://gcpdataflow/bestsellers.csv')

parser.add_argument('--output',

help='Output for the pipeline',

required=False,

default='gs://gcpdataflow/computedresult.txt')

PCollection

A PCollection is an immutable distributed data set that Beam pipeline functions on, it acts as inputs and outputs for each step in pipeline.

Pipeline I/O

Read transforms read data from an external source and return a PCollection representation of the data. Here I read the contents of the file — bestsellers.csv in bestsellers PCollection

# reading the csv and splitting lines by elements we want to retain

bestsellers = (

p | beam.io.ReadFromText(input_filename, skip_header_lines=1)

| beam.ParDo(Split())

)

Note that I have set skipped header to TRUE (1). If you need the headers you can set it to 0

Write transforms write the data in a PCollection to an external data source

"Write results to Fiction_Results file" >>beam.io.WriteToText('gs://gcpdataflow/Fiction_Result')

Transforms

Transforms are the operations in your pipeline. It takes PCollection as input and returns a PCollection as output. Multiple transforms are invoked using | pipe operator.

Generic Example:

[Final Output PCollection] = ([Initial Input PCollection]

| [First Transform]

| [Second Transform]

| [Third Transform])

For example, the bestsellers PCollection contains list of all bestsellers. I am creating a new pipeline to filter only the “Fiction” books using “Genre” column and then calculating the average user rating for all fiction books using MyTransform() and then printing the result in a file

Fiction_pipeline = ( bestsellers

| beam.Filter(lambda record: FilterBasedonGenre('Fiction',record))

| "Composite Transformation for Fiction elements" >> MyTransform()

| "Write results to Fiction_Results file" >>beam.io.WriteToText('gs://gcpdataflow/Fiction_Result') )

Composite Transforms

When you need to perform multiple simple transforms repeatedly, then you can put many simple transforms inside a “composite” transform.

Example, Here MyTransform() is a composite transform. I need to perform 1. extract rating 2.calculate average rating 3.format the results for fiction books, non fiction books and all books. So I have nested all these 3 simple transforms into a composite transform.

GroupByKey : used to aggregate data that has something in common. It takes key,value pairs as input. When we have data, that contains multiple pairs with the same key, but different values, then to collect all of the values associated with each unique key, you can use GroupByKey()

ExtractRating: Returns a list of tuples containing 1 (key) and Rating value. This form the input to GroupByKey, which takes (key,value) pair as input

class ExtractRating(beam.DoFn):

def process(self, element):

result = [(1, element['Rating'])]

return result

ParDo and DoFn

The ParDo transform processes each element in the input PCollection, independently and possibly in parallel, and produces zero or more elements to an output PCollection. DoFn is the argument to ParDo to process elements of the input PCollection.

The Split() is invoked as follows:

beam.ParDo(Split())

D) Run the pipeline

Run the program: