Project: Beam (App Definition and Development / Streaming & Messaging)
File: groupby.md

title: "GroupBy" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> {{< localstorage language language-py >}} {{< button-pydoc path="apache_beam.transforms.core" class="GroupBy" >}} Takes a collection of elements and produces a collection grouped, by properties of those elements. Unlike `GroupByKey`, the key is dynamically created from the elements themselves. In the following example, we create a pipeline with a `PCollection` of fruits. We use `GroupBy` to group all fruits by the first letter of their name. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONGroupByExpr" show="groupbyexpr" >}} {{< /playground >}} We can group by a composite key consisting of multiple properties if desired. The resulting key is a named tuple with the two requested attributes, and the values are grouped accordingly. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONGroupByTwoExprs" show="groupbytwo_exprs" >}} {{< /playground >}} In the case that the property one wishes to group by is an attribute, a string may be passed to `GroupBy` in the place of a callable expression. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONGroupByAttr" show="groupbyattr" >}} {{< /playground >}} It is possible to mix and match attributes and expressions, for example {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONGroupByAttrExpr" show="groupbyattr_expr" >}} {{< /playground >}} Grouping is often used in conjunction with aggregation, and the `aggregate_field` method of the `GroupBy` transform can be used to accomplish this easily. This method takes three parameters: the field (or expression) which to aggregate, the `CombineFn` (or associative `callable`) with which to aggregate by, and finally a field name in which to store the result. For example, suppose one wanted to compute the amount of each fruit to buy. One could write {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONGroupBySimpleAggregate" show="simpleaggregate" >}} {{< /playground >}} Similar to the parameters in `GroupBy`, one can also aggregate multiple fields and by expressions. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONGroupByExprAggregate" show="expraggregate" >}} {{< /playground >}} One can, of course, aggregate the same field multiple times as well. This example also illustrates a global grouping, as the grouping key is empty. {{< playground height="700px" >}} {{< playgroundsnippet language="py" path="SDKPYTHONGroupByGlobalAggregate" show="globalaggregate" >}} {{< /playground >}} for combining with a single CombineFn. for grouping with a known key. for multiple input collections. {{< button-pydoc path="apache_beam.transforms.core" class="GroupByKey" >}}