PayPal's Teradata Migration to BigQuery for Product Analytics
By: Romit Mehta
Tags: Public Cloud, Programming, Google Cloud Platform, Google, Bigquery

Approach Used

Once we chose which cloud and warehouse to explore, we identified the following tracks and started getting to the next phase.

Customer Contact

We reached out to users of the warehouse based on their usage stats over the past 12 months, as well as the data providers in that cluster. We set up time, walked them over the decision and sought their buy-in for this migration. This stakeholder buy-in was important for us to have a successful migration. We explained the rationale and how we planned to approach the problem. Some users were excited and wanted to be closely involved with the migration effort. We identified one team within one business unit as an early adopter and focused our migration effort on their use cases and data requirements.

Secure Infrastructure Buildout

We built a secure infrastructure to move data to the cloud. We kept data within BigQuery as US, multi-region to be accessed from other regions in the US. We implemented a secure private interconnect between our data center and the region in Google Cloud Platform nearest the analytics warehouse. Since we expected to operate in a hybrid mode (other connected systems remain on-premises for the foreseeable future), a private interconnect with no egress costs was a better option.

We decided to secure our data using PayPal-provided private keys in BigQuery, within a service perimeter offered by Google Cloud Platform. This ensured that data was secure and within a perimeter which cannot be accessed from outside. We deployed automation to prevent accidental creations of data sets that lack encryption keys. This way, we had encryption enabled by default for all data stored in Google Cloud Platform in a way that was compliant with our internal policies and external regulations.

We have used this infrastructure to copy more than 15 petabytes for BigQuery and 80+ petabytes into Google Cloud Services for various use cases. We use the same network infrastructure for users to access BigQuery through Jupyter Notebooks, Tableau, or from their scheduled jobs.

Regulatory Compliance and Pen-Testing

As a fintech organization that deals with PCI and PII data elements in our datasets, we worked with various regulators to file our intent to move data to the cloud. PayPal’s InfoSec, regional business units and legal teams worked overtime with our teams to prepare paperwork for regulators. We followed up with rounds of pen-tests to ensure there were no gaps. This helped us validate what we had designed into the infrastructure to secure the data and access to the data.

DDL (Data Definition Language) and SQL Conversion

Given that we were taking our data users to the cloud with a new technology, we wanted to ease the pain of transitioning from Teradata to BigQuery. To achieve this, we evaluated various options and selected a tool from CompilerWorks. Its transpiler allowed us to create DDLs in BigQuery, and use that schema to convert DMLs and user SQLs from Teradata flavor to BigQuery. PayPal worked to harden the transpiler configurations to generate performant, clean BigQuery-compatible SQLs.

This automated code conversion was a very critical step for us to get right as we wanted to ease the migration for our users. In addition to the code conversion, we also extracted valuable lineage data from CompilerWorks’s tool. We created an automation framework as well as a portal for interactive use and self-service code conversion. The automation kept polling for changes in on-premises infrastructure and create the equivalent in BigQuery as new artifacts got created. We requested users to use the portal to convert their existing or known SQLs to BigQuery-compatible SQLs for their testing and validation. We leveraged the same framework to convert users’ jobs, Tableau dashboards, and Notebooks for testing and validation. This automation helped us convert more than 10K SQLs.

Workloads, Schemas and Tables Identification

To scope the workloads, the team went through all Notebooks in our repository, Tableau dashboards and UC4 logs. Based on the tables we identified, we created a lineage graph to come up with a list of tables and schemas used, active scheduled jobs, notebooks, and dashboards. We validated the scope of work with users and confirmed that it was a true representation of the workloads on the cluster. This helped the team greatly reduce the number of workloads that we needed to migrate. Here is a breakdown of what was deprecated from the overall inventory.

Figure 3: Deprecation of workloads as part of the migration journey

Spending time on automation helped us separate out the used ones from unused ones and gain validation from users at the last step. Having users manually find this out would be tedious and error prone.

Data Movement, Loading and Validation

As we worked through this project, it became clear that data movement is very contextual to our setup, and off-the-shelf tools had limitations in being able to seamlessly copy data to Google Cloud Platform. This was the hardest part of the whole project. It was not the volume but accidental complexities that made it harder. Here is a quick list of problems we encountered:

Resource Availability and Usage: Since we were extracting data from an on-premises warehouse, our extraction speeds were bounded by the capacity available on the source. Our warehouse usage was seasonal, and running data extractions became extremely hard during those peak times. This became a manual process at various times of the day or weekends when we sped up data transfer by allocating more capacity to the extraction process.

Since we were extracting data from an on-premises warehouse, our extraction speeds were bounded by the capacity available on the source. Our warehouse usage was seasonal, and running data extractions became extremely hard during those peak times. This became a manual process at various times of the day or weekends when we sped up data transfer by allocating more capacity to the extraction process. Data Manipulations at the Source: Since we were running on-premises systems while we were extracting data, we had to copy all incremental changes continuously to the target in BigQuery. For small tables, we could simply copy the whole table repeatedly. For larger tables that had rows added every day and did not have updates or deletes, we could track incremental changes and replicate it to target. For tables that updated rows at source or where rows were deleted and recreated, replicating actions became a bit harder.

Since we were running on-premises systems while we were extracting data, we had to copy all incremental changes continuously to the target in BigQuery. For small tables, we could simply copy the whole table repeatedly. For larger tables that had rows added every day and did not have updates or deletes, we could track incremental changes and replicate it to target. For tables that updated rows at source or where rows were deleted and recreated, replicating actions became a bit harder. DDL Changes in Source : Alterations to source tables to support business use cases were inevitable. Since DDL changes were already limited to batch processes, we instrumented the batch processing platform to detect and synchronize data copy operations with changes.

: Alterations to source tables to support business use cases were inevitable. Since DDL changes were already limited to batch processes, we instrumented the batch processing platform to detect and synchronize data copy operations with changes. Data Types: While the mapping between Teradata and compatible BigQuery datatypes was straightforward, decisions had to be made about handling implicit behaviors. For example, we had timestamps with an implicit timezone in our source data that applications relied on and had to be converted to Datetime instead of Timestamp for loading into BigQuery. Similarly, string values in the source system had to be trimmed before copying into BigQuery so that queries that used the equal operator would return the same results as Teradata.

While the mapping between Teradata and compatible BigQuery datatypes was straightforward, decisions had to be made about handling implicit behaviors. For example, we had timestamps with an implicit timezone in our source data that applications relied on and had to be converted to Datetime instead of Timestamp for loading into BigQuery. Similarly, string values in the source system had to be trimmed before copying into BigQuery so that queries that used the equal operator would return the same results as Teradata. Data Loading: Doing a one-time load into BigQuery is fairly straightforward. The process starts getting complicated when replicating changes at source on a periodic basis to BigQuery. This involves keeping track of the changes from the source, and replaying them in BigQuery. Handling a large backlog of automated data loads for these extreme cases was challenging. Since we were cutting over users progressively, we had to be aware that the tables in BigQuery needed to be in production quality.

Doing a one-time load into BigQuery is fairly straightforward. The process starts getting complicated when replicating changes at source on a periodic basis to BigQuery. This involves keeping track of the changes from the source, and replaying them in BigQuery. Handling a large backlog of automated data loads for these extreme cases was challenging. Since we were cutting over users progressively, we had to be aware that the tables in BigQuery needed to be in production quality. Data Validations: Multiple types of data validations were performed on the data before it was released to data users. This included row counts, partition counts, column aggregations and sampling checks .

Multiple types of data validations were performed on the data before it was released to data users. This included row counts, partition counts, column aggregations and sampling checks BigQuery Nuances: Limits on the number of partitions a single query can touch meant splitting up data load statements based on partitions and tuning the splits when we ran against the limit.

Dry Runs and Wet Runs

Dry runs, an execution with no data, ensured that there were no syntax errors with the queries that were transformed. If the dry run was successful, we loaded data into the tables and requested users to conduct a wet run. Wet runs were one-off executions to test whether the result sets were all correct. We created test data sets for users to do their wet run, before they qualified their workload for production. All of this were enabled for users using our application lifecycle management portal which our users were accustomed to for deploying applications. We laid lots of emphasis on fitting our testing into an ecosystem that our users were used to.

Visibility into Progress

A number of the above activities were happening in parallel. This required coordination, which is harder for humans or coordinated spreadsheets. We kept track of all the data in BigQuery which was automatically updated as executions happened. We created dashboards to track sequences of activities and to report consistently to our execs and stakeholders on progress. These dashboards tracked the data copy progress for multiple milestones, workload rationalization and progress on readiness of notebooks, scheduled jobs and BI dashboards for dry and wet runs. A sample report looked like this. Users were able to search by DB name and table names for checking status.