Project: Beam (App Definition and Development / Streaming & Messaging)
File: combineperkey.md

title: "CombinePerKey" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> {{< localstorage language language-py >}} {{< button-pydoc path="apache_beam.transforms.core" class="CombinePerKey" >}} Combines all elements for each key in a collection. See more information in the . In the following examples, we create a pipeline with a `PCollection` of produce. Then, we apply `CombinePerKey` in multiple ways to combine all the elements in the `PCollection`. `CombinePerKey` accepts a function that takes a list of values as an input, and combines them for each key. We use the function which takes an `iterable` of numbers and adds them together. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONCombinePerKeySimple" show="combineperkeysimple" >}} {{< /playground >}} We define a function `saturated_sum` which takes an `iterable` of numbers and adds them together, up to a predefined maximum number. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONCombinePerKeyFunction" show="combineperkeyfunction" >}} {{< /playground >}} We can also use lambda functions to simplify Example 2. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONCombinePerKeyLambda" show="combineperkeylambda" >}} {{< /playground >}} You can pass functions with multiple arguments to `CombinePerKey`. They are passed as additional positional arguments or keyword arguments to the function. In this example, the lambda function takes `values` and `max_value` as arguments. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONCombinePerKeyMultipleArguments" show="combineperkeymultiple_arguments" >}} {{< /playground >}} The more general way to combine elements, and the most flexible, is with a class that inherits from `CombineFn`. : This creates an empty accumulator. For example, an empty accumulator for a sum would be `0`, while an empty accumulator for a product (multiplication) would be `1`. : Called once per element. Takes an accumulator and an input element, combines them and returns the updated accumulator. : Multiple accumulators could be processed in parallel, so this function helps merging them into a single accumulator. : It allows to do additional calculations before extracting a result. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONCombinePerKeyCombineFn" show="combineperkeycombinefn" >}} {{< /playground >}} You can use the following combiner transforms: See also which allows you to combine more than one field at once. {{< button-pydoc path="apache_beam.transforms.core" class="CombinePerKey" >}}