Julien Tournay - Better, faster, stronger Coder. Improving data processing @Spotify
Channel: ScalaIO FR
Date: 2019-01-07

Here at Spotify, we use Scio, an open-source Scala framework, to develop data pipelines and deploy them on Google Dataflow.

Since the very beginning of “Big Data”, serialization and deserialization of classes have always been one of the hardest problem. In Scio 0.7.0 we’ve worked hard to improve the serialization of classes shuffled between workers.

We’ll see in this talk how we managed to significantly improve serialization predictability and efficiency while preserving developer productivity by leveraging Scala macros and Jon Pretty “Magnolia”. We’ll also see how those tools can be used not only to scrap boilerplate, but also to provide clear and helpful insights to developers on how they can improve serialization and save resources.