Project: Beam (App Definition and Development / Streaming & Messaging)
File: beam-2.52.0.md

title: "Apache Beam 2.52.0" date: 2023-11-17 09:00:00 -0400 categories: blog release authors: damccorm <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> We are happy to present the new 2.52.0 release of Beam. This release includes both improvements and new functionality. See the for this release. <!--more--> For more information on changes in 2.52.0, check out the . Previously deprecated Avro-dependent code (Beam Release 2.46.0) has been finally removed from Java SDK "core" package. Please, use `beam-sdks-java-extensions-avro` instead. This will allow to easily update Avro version in user code without potential breaking changes in Beam "core" since the Beam Avro extension already supports the latest Avro versions and should handle this. (). Publishing Java 21 SDK container images now supported as part of Apache Beam release process. () Direct Runner and Dataflow Runner support running pipelines on Java21 (experimental until tests fully setup). For other runners (Flink, Spark, Samza, etc) support status depend on runner projects. Add `UseDataStreamForBatch` pipeline option to the Flink runner. When it is set to true, Flink runner will run batch jobs using the DataStream API. By default the option is set to false, so the batch jobs are still executed using the DataSet API. `upload_graph` as one of the Experiments options for DataflowRunner is no longer required when the graph is larger than 10MB for Java SDK (). state amd side input cache has been enabled to a default of 100 MB. Use `--maxcachememoryusagemb=X` to provide cache size for the user state API and side inputs. (Python) (). Beam YAML stable

Beam pipelines can now be written using YAML and leverage the Beam YAML framework which includes a preliminary set of IO's and turnkey transforms. More information can be found in the YAML root folder and in the . `org.apache.beam.sdk.io.CountingSource.CounterMark` uses custom `CounterMarkCoder` as a default coder since all Avro-dependent classes finally moved to `extensions/avro`. In case if it's still required to use `AvroCoder` for `CounterMark`, then, as a workaround, a copy of "old" `CountingSource` class should be placed into a project code and used directly (). Renamed `host` to `firestoreHost` in `FirestoreOptions` to avoid potential conflict of command line arguments (Java) (). Fixed "Desired bundle size 0 bytes must be greater than 0" in Java SDK's BigtableIO.BigtableSource when you have more cores than bytes to read (Java) . `watchfilepattern` arg of the arg had no effect prior to 2.52.0. To use the behavior of arg `watchfilepattern` prior to 2.52.0, follow the documentation at https://beam.apache.org/documentation/ml/side-input-updates/ and use `WatchFilePattern` PTransform as a SideInput. () `MLTransform` doesn't output artifacts such as min, max and quantiles. Instead, `MLTransform` will add a feature to output these artifacts as human readable format - . For now, to use the artifacts such as min and max that were produced by the eariler `MLTransform`, use `readartifactlocation` of `MLTransform`, which reads artifacts that were produced earlier in a different `MLTransform` () Fixed a memory leak, which affected some long-running Python pipelines: . Fixed (Java/Python/Go) (). Mitigated (Python) . According to git shortlog, the following people contributed to the 2.52.0 release. Thank you to all contributors! Ahmed Abualsaud Ahmet Altay Aleksandr Dudko Alexey Romanenko Anand Inguva Andrei Gurau Andrey Devyatkin BjornPrime Bruno Volpato Bulat Chamikara Jayalath Damon Danny McCormick Devansh Modi Dominik Dbowczyk Ferran Fernndez Garrido Hai Joey Tran Israel Herraiz Jack McCluskey Jan Lukavsk JayajP Jeff Kinard Jeffrey Kinard Jiangjie Qin Jing Joar Wandborg Johanna jeling Julien Tournay Kanishk Karanawat Kenneth Knowles Kerry Donny-Clark Lus Bianchin Minbo Bae Pranav Bhandari Rebecca Szper Reuven Lax Ritesh Ghorse Robert Bradshaw Robert Burke RyuSA Shunping Huang Steven van Rossum Svetak Sundhar Tony Tang Vitaly Terentyev Vivek Sumanth Vlado Djerek Yi Hu aku019 brucearctor caneff damccorm ddebowczyk92 dependabot[bot] dpcollins-google edman124 gabry.wu illoise johnjcasey jonathan-lemos kennknowles liferoad magicgoody martin trieu nancyxu123 pablo rodriguez defino tvalentyn