Project: Beam (App Definition and Development / Streaming & Messaging)
File: beam-2.19.0.md

title: "Apache Beam 2.19.0" date: 2020-02-04 00:00:01 -0800 categories: blog release aliases: /blog/2020/02/04/beam-2.19.0.html authors: boyuanzz <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> We are happy to present the new 2.19.0 release of Beam. This release includes both improvements and new functionality. See the for this release.<!--more--> For more information on changes in 2.19.0, check out the . Multiple improvements made into Python SDK harness: , , , , Create a BigQuery source (that implements iobase.BoundedSource) for Python SDK Implement an S3 filesystem for Python SDK Support Elasticsearch 7.x More fine-grained controls for the size of a BigQuery Load job PubsubMessageToRow should not check useFlatSchema() in processElement Extend ParquetIO.Read/ReadFiles.Builder to support Avro GenericData model Report collection size from MongoDBIOIT Report saved data size from HadoopFormatIOIT Improve error reporting in Java/Python PortableRunner Containerize the Spark job server Allow the local job service to work from inside docker PCollectionVisualizationTest: possible bug Execute portable Spark application jar Improve Spark Encoders (wrappers of beam coders) Improve error message when unable to get the correct filesystem for specified path in Python SDK) Improve error message when unable to get the correct filesystem for specified path in Python SDK Unify the config names of Fn Data API across languages Issue with GroupByKey in BeamSql using SparkRunner [SQL] MongoDb should use predicate push-down [SQL] Create performance tests for BigQueryTable Upgrade to ZetaSQL 2019.12.1 Backwards incompatible change in

(caught by failure when running Apache Nemo quickstart) Backwards incompatible change related to how Environments are represented in Python `DirectRunner`. Template staging broken on Beam 2.18.0 Beam Dependency Update Request: com.google.protobuf:protobuf-java Beam Dependency Update Request: commons-io:commons-io Beam Dependency Update Request: org.apache.commons:commons-csv Beam Dependency Update Request: org.apache.commons:commons-lang3 Beam Dependency Update Request: com.datastax.cassandra:cassandra-driver-mapping Beam Dependency Update Request: commons-codec:commons-codec HadoopResourceId returns wrong directory name FlinkMetricContainer causes churn in the JobManager and lets the web frontend malfunction PipelineResources algorithm is not working in most environments Cassandra IO classMethod test is flaky FileSystems may have not be initialized during ResourceId deserialization Python SDK emits duplicate records for Default and AfterWatermark triggers SDK harness servers don't shut down properly when SDK harness environment cleanup fails apachebeam.io.gcp.bigqueryreadittest failing on Py3.5 PC with: TypeError: the JSON object must be str, not 'bytes' PGBKCVOperation does not respect timestamp combiners Beam pickler doesn't pickle classes that have module set to None. Various bug fixes and performance improvements. According to git shortlog, the following people contributed to the 2.19.0 release. Thank you to all contributors! Ahmet Altay, Alex Amato, Alexey Romanenko, Andrew Pilloud, Ankur Goenka, Anton Kedin, Boyuan Zhang, Brian Hulette, Brian Martin, Chamikara Jayalath, Charles Chen, Craig Chambers, Daniel Oliveira, David Moravek, David Rieber, Dustin Rhodes, Etienne Chauchot, Gleb Kanterov, Hai Lu, Heejong Lee, Ismal Meja, Jan Lukavsk, Jason Kuster, Jean-Baptiste Onofr, Jeff Klukas, Joo Cabrita, J Ross Thomson, Juan Rael, Juta, Kasia Kucharczyk, Kengo Seki, Kenneth Jung, Kenneth Knowles, Kyle Weaver, Kyle Winkelman, Lukas Drbal, ukasz Gajowy, Marek Simunek, Mark Liu, Maximilian Michels, Melissa Pashniak, Michael Luckey, Michal Walenia, Mike Pedersen, Mikhail Gryzykhin, Niel Markwick, Pablo Estrada, Pascal Gula, Reuven Lax, Rob, Robbe Sneyders, Robert Bradshaw, Robert Burke, Rui Wang, Ruoyun Huang, Ryan Williams, Sam Rohde, Sam Whittle, Scott Wegner, Thomas Weise, Tianyang Hu, ttanay, tvalentyn, Tyler Akidau, Udi Meiri, Valentyn Tymofieiev, Xinyu Liu, XuMingmin