Package: databend-udf
Usage1. Define your functions in a Python filefromdatabend_udfimport*# Define a function@udf(input_types=["VARCHAR","VARCHAR","VARCHAR"],result_type="VARCHAR")defsplit_and_join(s:str,split_s:str,join_s:str)->str:returnjoin_s.join(s.split(split_s))# Define a function that accepts nullable values, and set skip_null to True to enable it returns NULL if any argument is NULL.@udf(input_types=["INT","INT"],result_type="INT",skip_null=True,)defgcd(x:int,y:int)->int:whiley!=0:(x,y)=(y,x%y)returnx# Define a function that accepts nullable values, and set skip_null to False to enable it handles NULL values inside the function.@udf(input_types=["ARRAY(INT64 NULL)","INT64"],result_type="INT NOT NULL",skip_null=False,)defarray_index_of(array:List[int],item:int):ifarrayisNone:return0try:returnarray.index(item)+1exceptValueError:return0# Define a function which is IO bound, and set io_threads to enable it can be executed concurrently.@udf(input_types=["INT"],result_type="INT",io_threads=32)defwait_concurrent(x):# assume an IO operation cost 2stime.sleep(2)returnxif__name__=='__main__':# create a UDF server listening at '0.0.0.0:8815'server=UDFServer("0.0.0.0:8815")# add defined functionsserver.add_function(split_and_join)server.add_function(gcd)server.add_function(array_index_of)server.add_function(wait_concurrent)# start the UDF serverserver.serve()@udfis an annotation for creating a UDF. It supports following parameters:input_types: A list of strings or Arrow data types that specifies the input data types.result_type: A string or an Arrow data type that specifies the return value type.name: An optional string specifying the function name. If not provided, the original name will be used.io_threads: Number of I/O threads used per data chunk for I/O bound functions.skip_null: A boolean value specifying whether to skip NULL value. If it is set to True, NULL values will not be passed to the function, and the corresponding return value is set to NULL. Default to False.2. Start the UDF ServerThen we can Start the UDF Server by running:python3udf_server.py3. Update Databend query node configNow, udf server is disabled by default in databend. You can enable it by setting 'enable_udf_server = true' in query node config.In addition, for security reasons, only the address specified in the config can be accessed by databend. The list of allowed udf server addresses are specified through theudf_server_allowlistvariable in the query node config.Here is an example config:[query]
...
enable_udf_server = true
udf_server_allow_list = [ "http://0.0.0.0:8815", "http://example.com" ]4. Add the functions to DatabendWe can use theCREATE FUNCTIONcommand to add the functions you defined to Databend:CREATE FUNCTION [IF NOT EXISTS] <udf_name> (<arg_type>, ...) RETURNS <return_type> LANGUAGE <language> HANDLER=<handler> ADDRESS=<udf_server_address>Theudf_nameis the name of UDF you declared in Databend. Thehandleris the function name you defined in the python UDF server.For example:CREATEFUNCTIONsplit_and_join(VARCHAR,VARCHAR,VARCHAR)RETURNSVARCHARLANGUAGEpythonHANDLER='split_and_join'ADDRESS='http://0.0.0.0:8815';NOTE: The udf_server_address you specify must appear inudf_server_allow_listexplained in the previous step.In step 2, when you starting the UDF server, the corresponding sql statement of each function will be printed out. You can use them directly.5. Use the functions in Databendmysql> select split_and_join('3,5,7', ',', ':');
+-----------------------------------+
| split_and_join('3,5,7', ',', ':') |
+-----------------------------------+
| 3:5:7                             |
+-----------------------------------+Data TypesThe data types supported by the Python UDF API and their corresponding python types are as follows :SQL TypePython TypeBOOLEANboolTINYINT (UNSIGNED)intSMALLINT (UNSIGNED)intINT (UNSIGNED)intBIGINT (UNSIGNED)intFLOATfloatDOUBLEfloatDECIMALdecimal.DecimalDATEdatetime.dateTIMESTAMPdatetime.datetimeVARCHARstrBINARYbytesVARIANTanyMAP(K,V)dictARRAY(T)list[T]TUPLE(T...)tuple(T...)The NULL in sql is represented by None in Python.Databend UDF Server Tests# start UDF serverpython3examples/server.py./target/debug/databend-sqllogictests--run_dirudf_serverAcknowledgementDatabend Python UDF Server API is inspired byRisingWave Python API.