Beam Summit 2022 - Relational Beam: Process columns, not rows!
Channel: Apache Beam
Date: 2022-08-03

Session presented by Andrew Pilloud and Brian Hulette at Beam Summit 2022

Last year, Andrew, Apache Beam Committer and Software Engineer at Google, and Brian, Software Engineer at Google as well as Apache Beam Committer, kicked off relational work with a vision of automatically optimizing your pipeline. Now they have a panel of contributors who are working towards this goal! They do their demo of the new optimizer in Java core, showing how we can automatically prune columns from IOs. Then, they discuss their upcoming work to make vectorized execution a reality through native columnar support in Python and Java. Finally, they discuss usage best practices around using Schemas, Dataframes, and SQL to ensure you can benefit from these changes.