Project: Beam (App Definition and Development / Streaming & Messaging)
File: splittable-do-fn-is-available.md

title: "Splittable DoFn in Apache Beam is Ready to Use" date: 2020-12-14 00:00:01 -0800 categories: blog aliases: /blog/2020/12/14/splittable-do-fn-is-available.html authors: boyuanzz <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> We are pleased to announce that Splittable DoFn (SDF) is ready for use in the Beam Python, Java, and Go SDKs for versions 2.25.0 and later. In 2017, proposed to build APIs as the new recommended way of building I/O connectors. Splittable DoFn is a generalization of `DoFn` that gives it the core capabilities of `Source` while retaining `DoFn`'s syntax, flexibility, modularity, and ease of coding. Thus, it becomes much easier to develop complex I/O connectors with simpler and reusable code. SDF has three advantages over the existing `UnboundedSource` and `BoundedSource`: SDF provides a unified set of APIs to handle both unbounded and bounded cases. SDF enables reading from source descriptors dynamically. Taking KafkaIO as an example, within `UnboundedSource`/`BoundedSource` API, you must specify the topic and partition you want to read from during pipeline construction time. There is no way for `UnboundedSource`/`BoundedSource` to accept topics and partitions as inputs during execution time. But it's built-in to SDF. SDF fits in as any node on a pipeline freely with the ability of splitting. `UnboundedSource`/`BoundedSource` has to be the root node of the pipeline to gain performance benefits from splitting strategies, which limits many real-world usages. This is no longer a limit for an SDF. As SDF is now ready to use with all the mentioned improvements, it is the recommended way to build the new I/O connectors. Try out building your own Splittable DoFn by following the . We have provided tonnes of common utility classes such as common types of `RestrictionTracker` and `WatermarkEstimator` in Beam SDK, which will help you onboard easily. As for the existing I/O connectors, we have wrapped `UnboundedSource` and `BoundedSource` implementations into Splittable DoFns, yet we still encourage developers to convert `UnboundedSource`/`BoundedSource` into actual Splittable DoFn implementation to gain more performance benefits. Many thanks to every contributor who brought this highly anticipated design into the data processing world. We are really excited to see that users benefit from SDF. Below are some real-world SDF examples for you to explore. Java Examples : An I/O connector for (an open-source distributed event streaming platform). : Uses a polling function producing a growing set of outputs for each input until a per-input termination condition is met. : An I/O connector for (an open-source columnar storage format). : An I/O connector for HL7v2 messages (a clinical messaging format that provides data about events that occur inside an organization) part of . : A wrapper which converts an existing implementation to a splittable DoFn. : A wrapper which converts an existing implementation to a splittable DoFn. Python Examples : A wrapper which converts an existing implementation to a splittable DoFn. Go Examples implements reading from text files using a splittable DoFn.