Aydan Rende - Migrate Data, Mesh in mind
Channel: Plain Schwarz
Date: 2023-06-19

For quite some time, Hadoop served as the data warehouse for Kleinanzeigen. However, the central teams eventually decided to say goodbye to this old friend due to its outdated nature and high costs. This migration presented us with a valuable opportunity to embrace the Data Mesh strategy and establish a new data pipeline. In this presentation, our objective is to provide an overview of our approach, which involves implementing a cloud-based data pipeline with the help of dbt and Airflow. Furthermore, we will delve into the challenges we faced during the process, including the debugging of legacy data flows, the complexities of copying data to s3, and dealing with the domain ownership issues. By sharing these experiences, we aim to provide valuable insights into our journey.

Speaker: Aydan Rende

More: https://2023.berlinbuzzwords.de/sessions/?id=ZNJLXG

Web: https://2023.berlinbuzzwords.de/
Fediverse: https://floss.social/@berlinbuzzwords
Linkedin: https://www.linkedin.com/showcase/13978964/
Twitter: https://twitter.com/berlinbuzzwords