Project: Beam (App Definition and Development / Streaming & Messaging)
File: beam-2.53.0.md

title: "Apache Beam 2.53.0" date: 2024-01-04 09:00:00 -0400 categories: blog release authors: jrmccluskey <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> We are happy to present the new 2.53.0 release of Beam. This release includes both improvements and new functionality. See the for this release. <!--more--> For more information on changes in 2.53.0, check out the . Python streaming users that use 2.47.0 and newer versions of Beam should update to version 2.53.0, which fixes a known issue: (). TextIO now supports skipping multiple header lines (Java) (). Python GCSIO is now implemented with GCP GCS Client instead of apitools () Adding support for LowCardinality DataType in ClickHouse (Java) (). Added support for handling bad records to KafkaIO (Java) () Add support for generating text embeddings in MLTransform for Vertex AI and Hugging Face Hub models.() NATS IO connector added (Go) (). The Python SDK now type checks `collections.abc.Collections` types properly. Some type hints that were erroneously allowed by the SDK may now fail. () Running multi-language pipelines locally no longer requires Docker. Instead, the same (generally auto-started) subprocess used to perform the expansion can also be used as the cross-language worker. Framework for adding Error Handlers to composite transforms added in Java (). Python 3.11 images now include google-cloud-profiler (). Euphoria DSL is deprecated and will be removed in a future release (not before 2.56.0) () (Python) Fixed sporadic crashes in streaming pipelines that affected some users of 2.47.0 and newer SDKs (). (Python) Fixed a bug that caused MLTransform to drop identical elements in the output PCollection (). Upgraded to go 1.21.5 to build, fixing and Potential race condition causing NPE in DataflowExecutionStateSampler in Dataflow Java Streaming pipelines (). Some Python pipelines that run with 2.52.0-2.54.0 SDKs and use large materialized side inputs might be affected by a performance regression. To restore the prior behavior on these SDK versions, supply the `--maxcachememoryusagemb=0` pipeline option. (). Python pipelines that run with 2.53.0-2.54.0 SDKs and perform file operations on GCS might be affected by excess HTTP requests. This could lead to a performance regression or a permission issue. () In Python pipelines, when shutting down inactive bundle processors, shutdown logic can overaggressively hold the lock, blocking acceptance of new work. Symptoms of this issue include slowness or stuckness in long-running jobs. Fixed in 2.56.0 (). For the most up to date list of known issues, see https://github.com/apache/beam/blob/master/CHANGES.md According to git shortlog, the following people contributed to the 2.53.0 release. Thank you to all contributors! Ahmed Abualsaud Ahmet Altay Alexey Romanenko Anand Inguva Arun Pandian Balzs Nmeth Bruno Volpato Byron Ellis Calvin Swenson Jr Chamikara Jayalath Clay Johnson Damon Danny McCormick Ferran Fernndez Garrido Georgii Zemlianyi Israel Herraiz Jack McCluskey Jacob Tomlinson Jan Lukavsk JayajP Jeffrey Kinard Johanna jeling Julian Braha Julien Tournay Kenneth Knowles Lawrence Qiu Mark Zitnik Mattie Fu Michel Davit Mike Williamson Naireen Naireen Hussain Niel Markwick Pablo Estrada Radosaw Stankiewicz Rebecca Szper Reuven Lax Ritesh Ghorse Robert Bradshaw Robert Burke Sam Rohde Sam Whittle Shunping Huang Svetak Sundhar Talat UYARER Tom Stepp Tony Tang Vlado Djerek Yi Hu Zechen Jiang clmccart damccorm darshan-sj gabry.wu johnjcasey liferoad lrakla martin trieu tvalentyn