Project: Beam (App Definition and Development / Streaming & Messaging)
File: beam-2.38.0.md

title: "Apache Beam 2.38.0" date: 2022-04-20 9:00:00 -0700 categories: blog release authors: danoliveira <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> We are happy to present the new 2.38.0 release of Beam. This release includes both improvements and new functionality. See the for this release. <!--more--> For more information on changes in 2.38.0 check out the . Introduce projection pushdown optimizer to the Java SDK (). The optimizer currently only works on the , but more I/Os will be added in future releases. If you encounter a bug with the optimizer, please file a JIRA and disable the optimizer using pipeline option `--experiments=disableprojectionpushdown`. A new IO for Neo4j graph databases was added. () It has the ability to update nodes and relationships using UNWIND statements and to read data using cypher statements with parameters. `amazon-web-services2` has reached feature parity and is finally recommended over the earlier `amazon-web-services` and `kinesis` modules (Java). These will be deprecated in one of the next releases (). Long outstanding write support for `Kinesis` was added (). Configuration was simplified and made consistent across all IOs, including the usage of `AwsOptions` (, , ). Additionally, there's a long list of recent improvements and fixes to `S3` Filesystem (, , , , ), `DynamoDB` IO (, ), `SQS` IO (, ) and others. Pipeline dependencies supplied through `--requirementsfile` will now be staged to the runner using binary distributions (wheels) of the PyPI packages for linuxx8664 platform (). To restore the behavior to use source distributions, set pipeline option

To skip staging the packages at submission time, set pipeline option `--requirements_cache=skip` (Python). The Flink runner now supports Flink 1.14.x (). Interactive Beam now supports remotely executing Flink pipelines on Dataproc (Python) (). (Python) Previously `DoFn.inferoutputtypes` was expected to return `Iterable. (`amazon-web-services2`) The types of `awsRegion` / `endpoint` in `AwsOptions` changed from String to `Region` / `URI` (). Beam 2.38.0 will be the last minor release to support Flink 1.11. (`amazon-web-services2`) Client providers (`withXYZClientProvider()`) as well as IO specific `RetryConfiguration`s are deprecated, instead use `withClientConfiguration()` or `AwsOptions` to configure AWS IOs / clients. Custom implementations of client providers shall be replaced with a respective `ClientBuilderFactory` and configured through `AwsOptions` (). Fix S3 copy for large objects (Java) () Fix quadratic behavior of pipeline canonicalization (Go) () This caused unnecessarily long pre-processing times before job submission for large complex pipelines. Fix `pyarrow` version parsing (Python)() See a full list of open this version. According to git shortlog, the following people contributed to the 2.38.0 release. Thank you to all contributors! abhijeet-lele Ahmet Altay akustov Alexander Alexander Zhuravlev Alexey Romanenko AlikRodriguez Anand Inguva andoni-guzman andreukus Andy Ye Ankur Goenka ansh0l Artur Khanin Aydar Farrakhov Aydar Zainutdinov Benjamin Gonzalez Brian Hulette brucearctor bulat safiullin bullet03 Carl Mastrangelo Chamikara Jayalath Chun Yang Daniela Martn Daniel Oliveira Danny McCormick daria.malkova David Cavazos David Huntsperger dmitryor Dmytro Sadovnychyi dpcollins-google egalpin Elias Segundo Antonio emily Etienne Chauchot Hengfeng Li Ismal Meja Israel Herraiz Jack McCluskey Jakub Kukul Janek Bevendorff Jeff Klukas Johan Sternby Kamil Bregua Kenneth Knowles Ke Wu Kiley Kyle Weaver laraschmidt Lara Schmidt LE QUELLEC Olivier Luka Kalinovcic Luke Cwik Marcin Kuthan masahitojp Masato Nakamura Matt Casters Melissa Pashniak Michael Li Miguel Hernandez Moritz Mack mosche nancyxu123 Nathan J Mehl Niel Markwick Ning Kang Pablo Estrada paul-tlh Pavel Avilov Rahul Iyer Reuven Lax Ritesh Ghorse Robert Bradshaw Robert Burke Ryan Skraba Ryan Thompson Sam Whittle Seth Vargo sp029619 Steven Niemitz Thiago Nunes Udi Meiri Valentyn Tymofieiev Victor vitaly.terentyev Yichi Zhang Yi Hu yirutang Zachary Houfek Zoe