Project: Flink (App Definition and Development / Streaming & Messaging)
File: python_udfs.md

title: "General User-defined Functions" weight: 5 type: docs aliases: /dev/python/table-api-users-guide/udfs/python_udfs.html <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> User-defined functions are important features, because they significantly extend the expressiveness of Python Table API programs. NOTE: Python UDF execution requires Python version (3.8, 3.9 or 3.10) with PyFlink installed. It's required on both the client side and the cluster side. It supports to use Python scalar functions in Python Table API programs. In order to define a Python scalar function, one can extend the base class `ScalarFunction` in `pyflink.table.udf` and implement an evaluation method. The behavior of a Python scalar function is defined by the evaluation method which is named `eval`. The evaluation method can support variable arguments, such as `eval(*args)`. The following example shows how to define your own Python hash code function, register it in the TableEnvironment, and call it in a query. Note that you can configure your scalar function via a constructor before it is registered: ```python from pyflink.table.expressions import call, col from pyflink.table import DataTypes, TableEnvironment, EnvironmentSettings from pyflink.table.udf import ScalarFunction, udf class HashCode(ScalarFunction): def init(self): self.factor = 12 def eval(self, s): return hash(s) * self.factor settings = EnvironmentSettings.inbatchmode() table_env = TableEnvironment.create(settings) hashcode = udf(HashCode(), resulttype='BIGINT') mytable.select(col("string"), col("bigint"), hashcode(col("bigint")), call(hash_code, col("bigint"))) tableenv.createtemporaryfunction("hashcode", udf(HashCode(), result_type='BIGINT')) tableenv.sqlquery("SELECT string, bigint, hash_code(bigint) FROM MyTable") ``` It also supports to use Java/Scala scalar functions in Python Table API programs. ```python ''' Java code: // The Java class must have a public no-argument constructor and can be founded in current Java classloader. public class HashCode extends ScalarFunction { private int factor = 12; public int eval(String s) { return s.hashCode() * factor; } } ''' from pyflink.table.expressions import call, col from pyflink.table import TableEnvironment, EnvironmentSettings settings = EnvironmentSettings.inbatchmode() table_env = TableEnvironment.create(settings) tableenv.createjavatemporaryfunction("hash_code", "my.java.function.HashCode") mytable.select(call('hashcode', col("string"))) tableenv.sqlquery("SELECT string, bigint, hash_code(string) FROM MyTable") ``` There are many ways to define a Python scalar function besides extending the base class `ScalarFunction`. The following examples show the different ways to define a Python scalar function which takes two columns of bigint as the input parameters and returns the sum of them as the result. ```python class Add(ScalarFunction): def eval(self, i, j): return i + j add = udf(Add(),

@udf(result_type='BIGINT') def add(i, j): return i + j add = udf(lambda i, j: i + j, result_type='BIGINT') class CallableAdd(object): def call(self, i, j): return i + j add = udf(CallableAdd(), result_type='BIGINT') def partial_add(i, j, k): return i + j + k add = udf(functools.partial(partialadd, k=1), resulttype='BIGINT') tableenv.createtemporary_function("add", add) my_table.select(call('add', col('a'), col('b'))) my_table.select(add(col('a'), col('b'))) ``` Similar to a Python user-defined scalar function, a user-defined table function takes zero, one, or multiple scalar values as input parameters. However in contrast to a scalar function, it can return an arbitrary number of rows as output instead of a single value. The return type of a Python UDTF could be of types Iterable, Iterator or generator. The following example shows how to define your own Python multi emit function, register it in the TableEnvironment, and call it in a query. ```python from pyflink.table.expressions import col from pyflink.table import DataTypes, TableEnvironment, EnvironmentSettings from pyflink.table.udf import TableFunction, udtf class Split(TableFunction): def eval(self, string): for s in string.split(" "): yield s, len(s) envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) my_table = ... # type: Table, table schema: [a: String] split = udtf(Split(), result_types=['STRING', 'INT']) mytable.joinlateral(split(col("a")).alias("word", "length")) mytable.leftouterjoinlateral(split(col("a")).alias("word", "length")) tableenv.createtemporaryfunction("split", udtf(Split(), resulttypes=['STRING', 'INT'])) tableenv.sqlquery("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)") tableenv.sqlquery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") ``` It also supports to use Java/Scala table functions in Python Table API programs. ```python ''' Java code: // The generic type "Tuple2<String, Integer>" determines the schema of the returned table as (String, Integer). // The java class must have a public no-argument constructor and can be founded in current java classloader. public class Split extends TableFunction<Tuple2<String, Integer>> { private String separator = " "; public void eval(String str) { for (String s : str.split(separator)) { // use collect(...) to emit a row collect(new Tuple2<String, Integer>(s, s.length())); } } } ''' from pyflink.table.expressions import call, col from pyflink.table import TableEnvironment, EnvironmentSettings envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) my_table = ... # type: Table, table schema: [a: String] tableenv.createjavatemporaryfunction("split", "my.java.function.Split") mytable.joinlateral(call('split', col('a')).alias("word", "length")).select(col('a'), col('word'), col('length')) mytable.leftouterjoinlateral(call('split', col('a')).alias("word", "length")).select(col('a'), col('word'), col('length')) tableenv.sqlquery("SELECT a, word, length FROM MyTable, LATERAL TABLE(split(a)) as T(word, length)") tableenv.sqlquery("SELECT a, word, length FROM MyTable LEFT JOIN LATERAL TABLE(split(a)) as T(word, length) ON TRUE") ``` Like Python scalar functions, you can use the above five ways to define Python TableFunctions. <span class="label label-info">Note</span> The only difference is that the return type of Python Table Functions needs to be an iterable, iterator or generator. ```python @udtf(result_types='BIGINT') def generator_func(x): yield 1 yield 2 @udtf(result_types='BIGINT') def iterator_func(x): return range(5) @udtf(result_types='BIGINT') def iterable_func(x): result = [1, 2, 3] return result ``` A user-defined aggregate function (UDAGG) maps scalar values of multiple rows to a new scalar value. NOTE: Currently the general user-defined aggregate function is only supported in the GroupBy aggregation and Group Window Aggregation in streaming mode. For batch mode, it's currently not supported and it is recommended to use the . The behavior of an aggregate function is centered around the concept of an

The accumulator is an intermediate data structure that stores the aggregated values until a final aggregation result is computed. For each set of rows that need to be aggregated, the runtime will create an empty accumulator by calling `create_accumulator()`. Subsequently, the `accumulate(...)` method of the aggregate function will be called for each input row to update the accumulator. Currently after each row has been processed, the `get_value(...)` method of the aggregate function will be called to compute the aggregated result. The following example illustrates the aggregation process: {{< img alt="UDAGG mechanism" src="/fig/udagg-mechanism-python.png" width="80%" >}} In the above example, we assume a table that contains data about beverages. The table consists of three columns (`id`, `name`, and `price`) and 5 rows. We would like to find the highest price of all beverages in the table, i.e., perform a `max()` aggregation. In order to define an aggregate function, one has to extend the base class `AggregateFunction` in `pyflink.table` and implement the evaluation method named `accumulate(...)`. The result type and accumulator type of the aggregate function can be specified by one of the following two approaches: Implement the method named `getresulttype()` and `getaccumulatortype()`. Wrap the function instance with the decorator `udaf` in `pyflink.table.udf` and specify the parameters `resulttype` and `accumulatortype`. The following example shows how to define your own aggregate function and call it in a query. ```python from pyflink.common import Row from pyflink.table import AggregateFunction, DataTypes, TableEnvironment, EnvironmentSettings from pyflink.table.expressions import call from pyflink.table.udf import udaf from pyflink.table.expressions import col, lit from pyflink.table.window import Tumble class WeightedAvg(AggregateFunction): def create_accumulator(self): return Row(0, 0) def get_value(self, accumulator): if accumulator[1] == 0: return None else: return accumulator[0] / accumulator[1] def accumulate(self, accumulator, value, weight): accumulator[0] += value * weight accumulator[1] += weight def retract(self, accumulator, value, weight): accumulator[0] -= value * weight accumulator[1] -= weight def getresulttype(self): return 'BIGINT' def getaccumulatortype(self): return 'ROW<f0 BIGINT, f1 BIGINT>' envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) weighted_avg = udaf(WeightedAvg()) t = tableenv.fromelements([(1, 2, "Lee"), (3, 4, "Jay"), (5, 6, "Jay"), (7, 8, "Lee")]).alias("value", "count", "name") result = t.groupby(col("name")).select(weightedavg(col("value"), col("count")).alias("avg")).execute() result.print() tableenv.createtemporaryfunction("weightedavg", WeightedAvg()) result = t.groupby(col("name")).select(call("weightedavg", col("value"), col("count")).alias("avg")).execute() result.print() tableenv.createtemporary_view("source", t) result = tableenv.sqlquery( "SELECT weighted_avg(`value`, `count`) AS avg FROM source GROUP BY name").execute() result.print() tumble_window = Tumble.over(lit(1).hours) \ .on(col("rowtime")) \ .alias("w") result = t.window(tumble_window) \ .group_by(col('w'), col('name')) \ .select(col('w').start, col('w').end, weighted_avg(col('value'), col('count'))) \ .execute() result.print() ``` The `accumulate(...)` method of our `WeightedAvg` class takes three input arguments. The first one is the accumulator and the other two are user-defined inputs. In order to calculate a weighted average value, the accumulator needs to store the weighted sum and count of all the data that have already been accumulated. In our example, we use a `Row` object as the accumulator. Accumulators will be managed by Flink's checkpointing mechanism and are restored in case of failover to ensure exactly-once semantics. The following methods are mandatory for each `AggregateFunction`: `create_accumulator()` `accumulate(...)` `get_value(...)` The following methods of `AggregateFunction` are required depending on the use case:

is required when there are operations that could generate retraction messages before the current aggregation operation, e.g. group aggregate, outer join. \ This method is optional, but it is strongly recommended to be implemented to ensure the UDAF can be used in any use case. `merge(...)` is required for session window ang hop window aggregations. `getresulttype()` and `getaccumulatortype()` is required if the result type and accumulator type would not be specified in the `udaf` decorator. If an accumulator needs to store large amounts of data, `pyflink.table.ListView` and `pyflink.table.MapView` could be used instead of list and dict. These two data structures provide the similar functionalities as list and dict, however usually having better performance by leveraging Flink's state backend to eliminate unnecessary state access. You can use them by declaring `DataTypes.LISTVIEW(...)` and `DataTypes.MAPVIEW(...)` in the accumulator type, e.g.: ```python from pyflink.table import ListView class ListViewConcatAggregateFunction(AggregateFunction): def get_value(self, accumulator): return accumulator[1].join(accumulator[0]) def create_accumulator(self): return Row(ListView(), '') def accumulate(self, accumulator, *args): accumulator[1] = args[1] accumulator[0].add(args[0]) def getaccumulatortype(self): return DataTypes.ROW([ DataTypes.FIELD("f0", DataTypes.LIST_VIEW(DataTypes.STRING())), DataTypes.FIELD("f1", DataTypes.BIGINT())]) def getresulttype(self): return DataTypes.STRING() ``` Currently there are 2 limitations to use the ListView and MapView: The accumulator must be a `Row`. The `ListView` and `MapView` must be the first level children of the `Row` accumulator. Please refer to the {{< pythondoc file="pyflink.table.html#pyflink.table.ListView" name="documentation of the corresponding classes">}} for more information about this advanced feature. NOTE: For reducing the data transmission cost between Python UDF worker and Java process caused by accessing the data in Flink states(e.g. accumulators and data views), there is a cached layer between the raw state handler and the Python state backend. You can adjust the values of these configuration options to change the behavior of the cache layer for best performance: `python.state.cache-size`, `python.map-state.read-cache-size`, `python.map-state.write-cache-size`, `python.map-state.iterate-response-batch-size`. For more details please refer to the . A user-defined table aggregate function (UDTAGG) maps scalar values of multiple rows to zero, one, or multiple rows (or structured types). The returned record may consist of one or more fields. If an output record consists of only a single field, the structured record can be omitted, and a scalar value can be emitted that will be implicitly wrapped into a row by the runtime. NOTE: Currently the general user-defined table aggregate function is only supported in the GroupBy aggregation in streaming mode. Similar to an , the behavior of a table aggregate is centered around the concept of an accumulator. The accumulator is an intermediate data structure that stores the aggregated values until a final aggregation result is computed. For each set of rows that needs to be aggregated, the runtime will create an empty accumulator by calling `create_accumulator()`. Subsequently, the `accumulate(...)` method of the function is called for each input row to update the accumulator. Once all rows have been processed, the `emit_value(...)` method of the function is called to compute and return the final result. The following example illustrates the aggregation process: {{<img alt="UDTAGG mechanism"

width="80%">}} In the example, we assume a table that contains data about beverages. The table consists of three columns (`id`, `name`, and `price`) and 5 rows. We would like to find the 2 highest prices of all beverages in the table, i.e., perform a `TOP2()` table aggregation. We need to consider each of the 5 rows. The result is a table with the top 2 values. In order to define a table aggregate function, one has to extend the base class `TableAggregateFunction` in `pyflink.table` and implement one or more evaluation methods named `accumulate(...)`. The result type and accumulator type of the aggregate function can be specified by one of the following two approaches: Implement the method named `getresulttype()` and `getaccumulatortype()`. Wrap the function instance with the decorator `udtaf` in `pyflink.table.udf` and specify the parameters `resulttype` and `accumulatortype`. The following example shows how to define your own aggregate function and call it in a query. ```python from pyflink.common import Row from pyflink.table import DataTypes, TableEnvironment, EnvironmentSettings from pyflink.table.expressions import col from pyflink.table.udf import udtaf, TableAggregateFunction class Top2(TableAggregateFunction): def emit_value(self, accumulator): yield Row(accumulator[0]) yield Row(accumulator[1]) def create_accumulator(self): return [None, None] def accumulate(self, accumulator, row): if row[0] is not None: if accumulator[0] is None or row[0] > accumulator[0]: accumulator[1] = accumulator[0] accumulator[0] = row[0] elif accumulator[1] is None or row[0] > accumulator[1]: accumulator[1] = row[0] def getaccumulatortype(self): return 'ARRAY<BIGINT>' def getresulttype(self): return 'ROW<a BIGINT>' envsettings = EnvironmentSettings.instreaming_mode() tableenv = TableEnvironment.create(envsettings) top2 = udtaf(Top2()) t = tableenv.fromelements([(1, 'Hi', 'Hello'), (3, 'Hi', 'hi'), (5, 'Hi2', 'hi'), (7, 'Hi', 'Hello'), (2, 'Hi', 'Hello')], ['a', 'b', 'c']) t.groupby(col('b')).flataggregate(top2).select(col('*')).execute().print() ``` The `accumulate(...)` method of our `Top2` class takes two inputs. The first one is the accumulator and the second one is the user-defined input. In order to calculate a result, the accumulator needs to store the 2 highest values of all the data that has been accumulated. Accumulators are automatically managed by Flink's checkpointing mechanism and are restored in case of a failure to ensure exactly-once semantics. The result values are emitted together with a ranking index. The following methods are mandatory for each `TableAggregateFunction`: `create_accumulator()` `accumulate(...)` `emit_value(...)` The following methods of `TableAggregateFunction` are required depending on the use case: `retract(...)` is required when there are operations that could generate retraction messages before the current aggregation operation, e.g. group aggregate, outer join. \ This method is optional, but it is strongly recommended to be implemented to ensure the UDTAF can be used in any use case. `getresulttype()` and `getaccumulatortype()` is required if the result type and accumulator type would not be specified in the `udtaf` decorator. Similar to , we can also use ListView and MapView in Table Aggregate Function. ```python from pyflink.common import Row from pyflink.table import ListView from pyflink.table.types import DataTypes from pyflink.table.udf import TableAggregateFunction class ListViewConcatTableAggregateFunction(TableAggregateFunction): def emit_value(self, accumulator): result = accumulator[1].join(accumulator[0]) yield Row(result) yield Row(result) def create_accumulator(self): return Row(ListView(), '') def accumulate(self, accumulator, *args): accumulator[1] = args[1] accumulator[0].add(args[0]) def getaccumulatortype(self): return DataTypes.ROW([ DataTypes.FIELD("f0", DataTypes.LIST_VIEW(DataTypes.STRING())), DataTypes.FIELD("f1", DataTypes.BIGINT())]) def getresulttype(self): return DataTypes.ROW([DataTypes.FIELD("a", DataTypes.STRING())]) ```