Project: StarRocks (App Definition and Development / Database)
File: JAVA_UDF.md

displayed_sidebar: "English" From v2.2.0 onwards, you can compile user-defined functions (UDFs) to suit your specific business needs by using the Java programming language. From v3.0 onwards, StarRocks supports global UDFs, and you only need to include the `GLOBAL` keyword in the related SQL statements (CREATE/SHOW/DROP). This topic how to develop and use various UDFs. Currently, StarRocks supports scalar UDFs, user-defined aggregate functions (UDAFs), user-defined window functions (UDWFs), and user-defined table functions (UDTFs). You have installed , so you can create and compile Java projects. You have installed JDK 1.8 on your servers. The Java UDF feature is enabled. You can set the FE configuration item `enable_udf` to `true` in the FE configuration file fe/conf/fe.conf to enable this feature, and then restart the FE nodes to make the settings take effect. For more information, see . You need to create a Maven project and compile the UDF you need by using the Java programming language. Create a Maven project, whose basic directory structure is as follows: ```Plain project |--pom.xml |--src | |--main | | |--java | | |--resources | |--test |--target ``` Add the following dependencies to the pom.xml file: ```XML <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.example</groupId> <artifactId>udf</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>8</maven.compiler.source> <maven.compiler.target>8</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.2.76</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> <version>2.10</version> <executions> <execution> <id>copy-dependencies</id> <phase>package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <outputDirectory>${project.build.directory}/lib</outputDirectory> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-assembly-plugin</artifactId> <version>3.3.0</version> <executions> <execution> <id>make-assembly</id> <phase>package</phase> <goals> <goal>single</goal> </goals> </execution> </executions> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> </descriptorRefs> </configuration> </plugin> </plugins> </build> </project> ``` Use the Java programming language to compile a UDF. A scalar UDF operates on a single row of data and returns a single value. When you use a scalar UDF in a query, each row corresponds to a single value in the result set. Typical scalar functions include `UPPER`, `LOWER`, `ROUND`, and `ABS`. Suppose that the values of a field in your JSON data are JSON strings rather than JSON objects. When you use an SQL statement to extract JSON strings, you need to run `GETJSONSTRING` twice, for example, `GETJSONSTRING(GETJSONSTRING('{"key":"{\\"k0\\":\\"v0\\"}"}', "$.key"), "$.k0")`. To simplify the SQL statement, you can compile a scalar UDF that can directly extract JSON strings, for example, `MYUDFJSON_GET('{"key":"{\\"k0\\":\\"v0\\"}"}', "$.key.k0")`. ```Java package com.starrocks.udf.sample; import com.alibaba.fastjson.JSONPath; public class UDFJsonGet { public final String evaluate(String obj, String key) { if (obj == null || key == null) return null; try { // The JSONPath library can be fully expanded even if the values of a field are JSON strings. return JSONPath.read(obj, key).toString(); } catch (Exception e) { return null; } } } ``` The user-defined class must implement the method described in the following table. NOTE The data types of the request parameters and return parameters in the method must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in and conform to the mapping that is provided in the "" section of this topic. | Method | Description | | -- | | | TYPE1 evaluate(TYPE2, ...) | Runs the

The evaluate() method requires the public member access level. | A UDAF operates on multiple rows of data and returns a single value. Typical aggregate functions include `SUM`, `COUNT`, `MAX`, and `MIN`, which aggregate multiple rows of data specified in each GROUP BY clause and return a single value. Suppose that you want to compile a UDAF named `MYSUMINT`. Unlike the built-in aggregate function `SUM`, which returns BIGINT-type values, the `MYSUMINT` function supports only request parameters and returns parameters of the INT data type. ```Java package com.starrocks.udf.sample; public class SumInt { public static class State { int counter = 0; public int serializeLength() { return 4; } } public State create() { return new State(); } public void destroy(State state) { } public final void update(State state, Integer val) { if (val != null) { state.counter+= val; } } public void serialize(State state, java.nio.ByteBuffer buff) { buff.putInt(state.counter); } public void merge(State state, java.nio.ByteBuffer buffer) { int val = buffer.getInt(); state.counter += val; } public Integer finalize(State state) { return state.counter; } } ``` The user-defined class must implement the methods described in the following table. NOTE The data types of the request parameters and return parameters in the methods must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in and conform to the mapping that is provided in the "" section of this topic. | Method | Description | | | | | State create() | Creates a state. | | void destroy(State) | Destroys a state. | | void update(State, ...) | Updates a state. In addition to the first parameter `State`, you can also specify one or more request parameters in the UDF declaration. | | void serialize(State, ByteBuffer) | Serializes a state into the byte buffer. | | void merge(State, ByteBuffer) | Deserializes a state from the byte buffer, and merges the byte buffer into the state as the first parameter. | | TYPE finalize(State) | Obtains the final result of the UDF from a state. | During compilation, you must also use the buffer class `java.nio.ByteBuffer` and the local variable `serializeLength`, which are described in the following table. | Class and local variable | Description | | | | | java.nio.ByteBuffer() | The buffer class, which stores intermediate results. Intermediate results may be serialized or deserialized when they are transmitted between nodes for execution. Therefore, you must also use the `serializeLength` variable to specify the length that is allowed for the deserialization of intermediate results. | | serializeLength() | The length that is allowed for the deserialization of intermediate results. Unit: bytes. Set this local variable to an INT-type value. For example, `State { int counter = 0; public int serializeLength() { return 4; }}` specifies that intermediate results are of the INT data type and the length for deserialization is 4 bytes. You can adjust these settings based on your business

For example, if you want to specify the data type of intermediate results as LONG and the length for deserialization as 8 bytes, pass `State { long counter = 0; public int serializeLength() { return 8; }}`. | Take note of the following points for the deserialization of intermediate results stored in the `java.nio.ByteBuffer` class: The remaining() method that is dependent on the `ByteBuffer` class cannot be called to deserialize a state. The clear() method cannot be called on the `ByteBuffer` class. The value of `serializeLength` must be the same as the length of the written-in data. Otherwise, incorrect results are generated during serialization and deserialization. Unlike regular aggregate functions, a UDWF operates on a set of multiple rows, which are collectively called a window, and returns a value for each row. A typical window function includes an `OVER` clause that divides rows into multiple sets. It performs a calculation across each set of rows and returns a value for each row. Suppose that you want to compile a UDWF named `MYWINDOWSUMINT`. Unlike the built-in aggregate function `SUM`, which returns BIGINT-type values, the `MYWINDOWSUMINT` function supports only request parameters and returns parameters of the INT data type. ```Java package com.starrocks.udf.sample; public class WindowSumInt { public static class State { int counter = 0; public int serializeLength() { return 4; } @Override public String toString() { return "State{" + "counter=" + counter + '}'; } } public State create() { return new State(); } public void destroy(State state) { } public void update(State state, Integer val) { if (val != null) { state.counter+=val; } } public void serialize(State state, java.nio.ByteBuffer buff) { buff.putInt(state.counter); } public void merge(State state, java.nio.ByteBuffer buffer) { int val = buffer.getInt(); state.counter += val; } public Integer finalize(State state) { return state.counter; } public void reset(State state) { state.counter = 0; } public void windowUpdate(State state, int peergroupstart, int peergroupend, int framestart, int frameend, Integer[] inputs) { for (int i = (int)framestart; i < (int)frameend; ++i) { state.counter += inputs[i]; } } } ``` The user-defined class must implement the method required by UDAFs (because a UDWF is a special aggregate function) and the windowUpdate() method described in the following table. NOTE The data types of the request parameters and return parameters in the method must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in and conform to the mapping that is provided in the "" section of this topic. | Method | Description | | -- | | | void windowUpdate(State state, int, int, int , int, ...) | Updates the data of the window. For more information about UDWFs, see . Every time when you enter a row as input, this method obtains the window information and updates intermediate results accordingly.<ul><li>`peergroupstart`: the start position of the current partition. `PARTITION BY` is used in the OVER clause to specify a partition column. Rows with the same values in the partition column are considered to be in the same partition.</li><li>`peergroupend`: the end position of the current partition.</li><li>`framestart`: the start position of the current window

The window frame clause specifies a calculation range, which covers the current row and the rows that are within a specified distance to the current row. For example, `ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING` specifies a calculation range that covers the current row, the previous row before the current row, and the following row after the current row.</li><li>`frameend`: the end position of the current window frame.</li><li>`inputs`: the data that is entered as the input to a window. The data is an array package that supports only specific data types. In this example, INT values are entered as input, and the array package is Integer[].</li></ul> | A UDTF reads one row of data and returns multiple values that can be considered to be a table. Table-valued functions are typically used to transform rows into columns. NOTE StarRocks allows a UDTF to return a table that consists of multiple rows and one column. Suppose that you want to compile a UDTF named `MYUDFSPLIT`. The `MYUDFSPLIT` function allows you to use spaces as delimiters and supports request parameters and return parameters of the STRING data type. ```Java package com.starrocks.udf.sample; public class UDFSplit{ public String[] process(String in) { if (in == null) return null; return in.split(" "); } } ``` The method defined by the user-defined class must meet the following requirements: NOTE The data types of the request parameters and return parameters in the method must be the same as those declared in the `CREATE FUNCTION` statement that is to be executed in and conform to the mapping that is provided in the "" section of this topic. | Method | Description | | - | -- | | TYPE[] process() | Runs the UDTF and returns an array. | Run the following command to package the Java project: ```Bash mvn package ``` The following JAR files are generated in the target folder: udf-1.0-SNAPSHOT.jar and udf-1.0-SNAPSHOT-jar-with-dependencies.jar. Upload the JAR file udf-1.0-SNAPSHOT-jar-with-dependencies.jar to an HTTP server that keeps up and running and is accessible to all FEs and BEs in your StarRocks cluster. Then, run the following command to deploy the file: ```Bash mvn deploy ``` You can set up a simple HTTP server by using Python and upload the JAR file to that HTTP server. NOTE In , the FEs will check the JAR file that contains the code for the UDF and calculate the checksum, and the BEs will download and execute the JAR file. StarRocks allows you to create UDFs in two types of namespaces: database namespaces and global namespaces. If you do not have visibility or isolation requirements for a UDF, you can create it as a global UDF. Then, you can reference the global UDF by using the function name without including the catalog and database names as prefixes to the function name. If you have visibility or isolation requirements for a UDF, or if you need to create the same UDF in different databases, you can create it in each individual

As such, if your session is connected to the target database, you can reference the UDF by using the function name. If your session is connected to a different catalog or database other than the target database, you need to reference the UDF by including the catalog and database names as prefixes to the function name, for example, `catalog.database.function`. NOTICE Before you create and use a global UDF, you must contact the system administrator to grant you the required permissions. For more information, see . After you upload the JAR package, you can create UDFs in StarRocks. For a global UDF, you must include the `GLOBAL` keyword in the creation statement. ```sql CREATE FUNCTION function_name (arg_type [, ...]) RETURNS return_type PROPERTIES ("key" = "value" [, ...]) ``` | Parameter | Required | Description | | - | -- | | | GLOBAL | No | Whether to create a global UDF, supported from v3.0. | | AGGREGATE | No | Whether to create a UDAF or UDWF. | | TABLE | No | Whether to create a UDTF. If both `AGGREGATE` and `TABLE` are not specified, a Scalar function is created. | | functionname | Yes | The name of the function you want to create. You can include the name of the database in this parameter, for example,`db1.myfunc`. If `function_name` includes the database name, the UDF is created in that database. Otherwise, the UDF is created in the current database. The name of the new function and its parameters cannot be the same as an existing name in the destination database. Otherwise, the function cannot be created. The creation succeeds if the function name is the same but the parameters are different. | | arg_type | Yes | Argument type of the function. The added argument can be represented by `, ...`. For the supported data types, see .| | return_type | Yes | The return type of the function. For the supported data types, see . | | PROPERTIES | Yes | Properties of the function, which vary depending on the type of the UDF to create. | Run the following command to create the scalar UDF you have compiled in the preceding example: ```SQL CREATE [GLOBAL] FUNCTION MYUDFJSON_GET(string, string) RETURNS string PROPERTIES ( "symbol" = "com.starrocks.udf.sample.UDFJsonGet", "type" = "StarrocksJar", "file" = "http://httphost:httpport/udf-1.0-SNAPSHOT-jar-with-dependencies.jar" ); ``` | Parameter | Description | | | | | symbol | The name of the class for the Maven project to which the UDF belongs. The value of this parameter is in the `<packagename>.<classname>` format. | | type | The type of the UDF. Set the value to `StarrocksJar`, which specifies that the UDF is a Java-based function. | | file | The HTTP URL from which you can download the JAR file that contains the code for the UDF. The value of this parameter is in the `http://<httpserverip>:<httpserverport>/<jarpackagename>` format. | Run the following command to create the UDAF you have compiled in the preceding example: ```SQL CREATE [GLOBAL] AGGREGATE FUNCTION MYSUMINT(INT) RETURNS INT PROPERTIES ( "symbol" = "com.starrocks.udf.sample.SumInt", "type" = "StarrocksJar", "file" =

); ``` The descriptions of the parameters in PROPERTIES are the same as those in . Run the following command to create the UDWF you have compiled in the preceding example: ```SQL CREATE [GLOBAL] AGGREGATE FUNCTION MYWINDOWSUM_INT(Int) RETURNS Int properties ( "analytic" = "true", "symbol" = "com.starrocks.udf.sample.WindowSumInt", "type" = "StarrocksJar", "file" = "http://httphost:httpport/udf-1.0-SNAPSHOT-jar-with-dependencies.jar" ); ``` `analytic`: Whether the UDF is a window function. Set the value to `true`. The descriptions of other properties are the same as those in . Run the following command to create the UDTF you have compiled in the preceding example: ```SQL CREATE [GLOBAL] TABLE FUNCTION MYUDFSPLIT(string) RETURNS string properties ( "symbol" = "com.starrocks.udf.sample.UDFSplit", "type" = "StarrocksJar", "file" = "http://httphost:httpport/udf-1.0-SNAPSHOT-jar-with-dependencies.jar" ); ``` The descriptions of the parameters in PROPERTIES are the same as those in . After you create the UDF, you can test and use it based on your business needs. Run the following command to use the scalar UDF you have created in the preceding example: ```SQL SELECT MYUDFJSON_GET('{"key":"{\\"in\\":2}"}', '$.key.in'); ``` Run the following command to use the UDAF you have created in the preceding example: ```SQL SELECT MYSUMINT(col1); ``` Run the following command to use the UDWF you have created in the preceding example: ```SQL SELECT MYWINDOWSUM_INT(intcol) OVER (PARTITION BY intcol2 ORDER BY intcol3 ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) FROM test_basic; ``` Run the following command to use the UDTF you have created in the preceding example: ```Plain -- Suppose that you have a table named t1, and the information about its columns a, b, and c1 is as follows: SELECT t1.a,t1.b,t1.c1 FROM t1; output: 1,2.1,"hello world" 2,2.2,"hello UDTF." -- Run the MYUDFSPLIT() function. SELECT t1.a,t1.b, MYUDFSPLIT FROM t1, MYUDFSPLIT(t1.c1); output: 1,2.1,"hello" 1,2.1,"world" 2,2.2,"hello" 2,2.2,"UDTF." ``` NOTE The first `MYUDFSPLIT` in the preceding code snippet is the alias of the column that is returned by the second `MYUDFSPLIT`, which is a function. You cannot use `AS t2(f1)` to specify the aliases of the table and its columns that are to be returned. Run the following command to query UDFs: ```SQL SHOW [GLOBAL] FUNCTIONS; ``` For more information, see . Run the following command to drop a UDF: ```SQL DROP [GLOBAL] FUNCTION <functionname>(argtype [, ...]); ``` For more information, see . | SQL TYPE | Java TYPE | | -- | -- | | BOOLEAN | java.lang.Boolean | | TINYINT | java.lang.Byte | | SMALLINT | java.lang.Short | | INT | java.lang.Integer | | BIGINT | java.lang.Long | | FLOAT | java.lang.Float | | DOUBLE | java.lang.Double | | STRING/VARCHAR | java.lang.String | Configure the following environment variable in the be/conf/be.conf file of each Java virtual machine (JVM) in your StarRocks cluster to control memory usage. If you use JDK 8, configure `JAVAOPTS`. If you use JDK 9 or later, configure `JAVAOPTSFORJDK9AND_LATER`. ```Bash JAVA_OPTS="-Xmx12G" JAVAOPTSFORJDK9ANDLATER="-Xmx12G" ``` Can I use static variables when I create UDFs? Do the static variables of different UDFs have mutual impacts on each other? Yes, you can use static variables when you compile UDFs. The static variables of different UDFs are isolated from each other and do not affect each other even if the UDFs have classes with identical names.