Beam Summit 2022 - Apache Beam backend for open source Scalding
Channel: Apache Beam
Date: 03/08/2022

Session presented by Navin Viswanath at Beam Summit 2022.

The batch processing data pipelines at Twitter process hundreds of petabytes of data on a daily basis. These pipelines run on on-premise Hadoop clusters and use Scalding as the framework for data processing. Twitter is currently in the midst of migrating its batch processing pipelines to the cloud. With approximately 5000 production pipelines, it becomes necessary to automate the migration of these pipelines to the cloud.
 
Scalding is an abstract API for expressing computations on data and allows backend implementations to be plugged in. It currently has both a Cascading and Spark backend. In this talk, Navin, Staff Software Engineer at Twitter, describes a new Apache Beam backend for Scalding that is in the process of being implemented at Twitter. This backend expresses the various computations in Scalding as PTransforms in Beam. With this backend, they can run Scalding pipelines on Beam in the cloud with near zero code changes, thus automating the migration of these pipelines to the cloud.