Comparing BigQuery Processing and Spark Dataproc
By: Vignesh Raj K
Tags: Spark, Dataproc, PayPal, Gcp, Bigquery

Introduction

Currently, PayPal is in the process of migrating its analytical work loads to Google Cloud. In this post, I will be covering the different approaches we evaluated for migrating our processes from on-prem to GCP.

We run a number of spark jobs for processing analytical data in PayPal. These jobs collect behavioural data from the website and provide them in a consumable format for Data Analysts and Data Scientists. As a part of GCP migration we evaluated different approaches like streaming the data to BigQuery, moving the entire processing into BigQuery, running our existing spark job in a DataProc cluster using BigQuery data and using GCS Data. Let’s dig deep into different approaches evaluated for one such process, Sessionization.

What is Sessionization process

Let’s say a user visits PayPal website, logs into their account and transfers some money to a friend. They check their account balance and log out. These are some of the user actions (events) that occur within a session and an event is fired for each one of them. Sessionization process combines all the events from a single user session and provides a consolidated view of them. It comprises of information like session start time, session end time, number of events in the session etc.

How is it handled on-prem ?

All the processed behavioural data are available in HDFS. These events are batched at an hourly level based on the event time. A spark job reads the batched client-side data and performs a group by operation based on user id and session id. User id uniquely identifies the user and session id is used to identify each session.

Based on the session end time, the grouped events are segregated as completed sessions and in-progress sessions. All the completed sessions are written to the final output and the in-progress session are written to a temporary location. At the start of each run, a union is performed for the in-progress dataset and the new batch of behavioural data. SQL database is used for state management of this job.

Options considered for GCP

Running a Dataproc Job on BigQuery Data Running a Dataproc job on GCS Data Running a BigQuery SQL Job

Running a Dataproc job on BigQuery data

All the user data are available in BigQuery tables. This data can be pumped into a Dataproc cluster using the Spark-BigQuery connector library (Github). The same spark job which was running on-prem can be repurposed to run on a Dataproc cluster. The same connector library can be used to write data back to BigQuery.

The advantage of this option is that we can reuse the existing spark job. But there is an additional IO step and cost involved to move data out of BigQuery for processing. Since the Dataproc cluster is ephemeral, additional time is taken to provision a cluster for every run.

Running a Dataproc job on GCS data

In addition to BigQuery, the user data are available in GCS as well. A spark job can be run on this GCS data in a Dataproc cluster using Dataproc Hadoop connectors (Github). The processed data can be written back to BigQuery.

This option is very similar to the last approach and shares the same advantages. The only cost involved in this approach will be running a Dataproc job.

Running a BigQuery SQL Job

In this approach, a BQ SQL job is run on the user data available in BigQuery. The data is not moved out of BigQuery and can be processed within BigQuery using BQSQL.

Existing Spark SQL job has to be rewritten using BQ SQL for this option. BQ SQL takes advantage of the existing BigQuery infrastructure to do the processing. If BQ jobs are billed with on-demand pricing, this approach would be costlier than above options.

Performance Comparison

We ran the three options discussed above on different batches of data. Each batch of data contains ~75 GB of user data stored either in BigQuery or GCS. Time taken by each approach can be seen in the graph below.

Performance comparison of the three options

Running a BQ SQL job has the best performance among the three options. On an average it takes ~2 mins to process one batch of data using BQ SQL. Dataproc on GCS has better performance than Dataproc on BigQuery.

Cost Comparison

The cost of each of the three options to process the same data can be seen in the image below. The cost for Dataproc-BQ comprises of cost associated with both running a Dataproc job and extracting data out of BigQuery. BQ SQL cost is calculated as per on demand pricing. In case dedicated slots are provided in BigQuery, the cost for running BQ SQL would be lesser.

Cost comparison of the three options

Conclusion

BQ SQL is much faster than Dataproc in terms of performance and would be best if dedicated slots are provisioned already. Dataproc job with GCS read can be considered as a viable option if cost is a concern.