Beam Summit 2022 - Log Ingestion and Data Replication at Twitter
Channel: Apache Beam
Date: 03/08/2022

Session presented by Praveen Killamsetti, and Zhenzhao Wang at Beam Summit 2022

Data Analytics at Twitter relies on petabytes of data across data lakes and analytics databases. Data could come from log events generated by Twitter microservices based on user action (in the range of trillions of events per day) or data generated by processing jobs that process the log events. The Data Lifecycle Team at Twitter manages large-scale data ingestion and replication of data across Twitter data centers and the public cloud. Delivering the data either in streaming or batch fashion to data lakes (HDFS, GCS) and data warehouses (Google BigQuery) in a reliable and scalable way at the lowest possible latency is a complex problem. In this talk, Praveen, Staff Engineer at Twitter, and Zhenzhao, Staff SWE  at Twitter, explain their log ingestion architecture and data replication architecture across storage systems and explain how they use beam-based ingestion/replication pipelines for both batch and streaming use cases to achieve their goal.