Project: Beam (App Definition and Development / Streaming & Messaging)
File: beam-2.29.0.md

title: "Apache Beam 2.29.0" date: 2021-04-29 9:00:00 -0700 categories: blog release authors: klk <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> We are happy to present the new 2.29.0 release of Beam. This release includes both improvements and new functionality. See the for this release. <!--more--> For more information on changes in 2.29.0, check out the . Spark Classic and Portable runners officially support Spark 3 (). Official Java 11 support for most runners (Dataflow, Flink, Spark) (). DataFrame API now supports GroupBy.apply (). Added support for S3 filesystem on AWS SDK V2 (Java) () GCP BigQuery sink (file loads) uses runner determined sharding for unbounded data () KafkaIO now recognizes the `partition` property in writing records () Support for Hadoop configuration on ParquetIO () DataFrame API now supports pandas 1.2.x (). Multiple DataFrame API bugfixes (, ) DDL supported in SQL transforms () Upgrade Flink runner to Flink version 1.12.2 () Deterministic coding enforced for GroupByKey and Stateful DoFns. Previously non-deterministic coding was allowed, resulting in keys not properly being grouped in some cases. () To restore the old behavior, one can register `FakeDeterministicFastPrimitivesCoder` with `beam.coders.registry.registerfallbackcoder(beam.coders.coders.FakeDeterministicFastPrimitivesCoder())` or use the `allownondeterministickeycoders` pipeline option. Support for Flink 1.8 and 1.9 will be removed in the next release (2.30.0) (). See a full list of open this version. According to `git shortlog`, the following people contributed to the 2.29.0 release. Thank you to all contributors! Ahmet Altay, Alan Myrvold, Alex Amato, Alexander Chermenin, Alexey Romanenko, Allen Pradeep Xavier, Amy Wu, Anant Damle, Andreas Bergmeier, Andrei Balici, Andrew Pilloud, Andy Xu, Ankur Goenka, Bashir Sadjad, Benjamin Gonzalez, Boyuan Zhang, Brian Hulette, Chamikara Jayalath, Chinmoy Mandayam, Chuck Yang, dandy10, Daniel Collins, Daniel Oliveira, David Cavazos, David Huntsperger, David Moravek, Dmytro Kozhevin, Emily Ye, Esun Kim, Evgeniy Belousov, Filip Popi, Fokko Driesprong, Gris Cuevas, Heejong Lee, Ihor Indyk, Ismal Meja, Jakub-Sadowski, Jan Lukavsk, John Edmonds, Juan Sandoval, , Kenneth Jung, Kenneth Knowles, KevinGG, Kiley Sok, Kyle Weaver, MabelYC, Mackenzie Clark, Masato Nakamura, Milena Bukal, Miltos, Minbo Bae, Mira Vuslat Baaran, mynameborat, Nahian-Al Hasan, Nam Bui, Niel Markwick, Niels Basjes, Ning Kang, Nir Gazit, Pablo Estrada, Ramazan Yapparov, Raphael Sanamyan, Reuven Lax, Rion Williams, Robert Bradshaw, Robert Burke, Rui Wang, Sam Rohde, Sam Whittle, Shehzaad Nakhoda, Shehzaad Nakhoda, Siyuan Chen, Sonam Ramchand, Steve Niemitz, sychen, Sylvain Veyri, Tim Robertson, Tobias Kaymak, Tomasz Szerszen, Tomasz Szersze, Tomo Suzuki, Tyson Hamilton, Udi Meiri, Valentyn Tymofieiev, Yichi Zhang, Yifan Mai, Yixing Zhang, Yoshiki Obata