Package: openeo-r-udf
openeo-udf-python-to-r / openeo-r-udfThis is an experimental engine for openEO to run R UDFs from an R environment.This currently is limited to R UDFs that are running without any other processes in the following processes:applyapply_dimensionreduce_dimensionThis repository contains the following content:The scripts to run for testing:tests/test.py(single core) andtests/test_parallel.py(parallelized).The foldertests/udfscontains UDF examples as users could provide them.udf_lib.pyis a Python library with the Python code required to run R UDFs from Pythonexecutor.Ris the R script that is run from R and executes the R UDF in the Python environment.The following image shows how the implementation roughly works:UDF integrationThis section is for back-end developers who want to add R UDF capabilities to their back-end
or for end-users who want to test their UDFs locally.Install from pypiYou may want to install all dependencies as a new conda environment first:conda env create -f environment.ymlYou can install this library from pypi then:pip install openeo-r-udfRun UDFsIn the folloowing chapter we'll give examples on how to use the UDF library from a Python environment.The following variables should be defined:udf(string - The content of the parameterudffromrun_udf, i.e. UDF code or a path/URL to a UDF)udf_folder(string - The folder where the UDFs reside or should be written to)process(string - The parent process, i.e.apply,apply_dimensionorreduce_dimension)data(xarray.DataArray - The data to process)dimension(string, defaults toNone- The dimension to work on if applicable, doesn't apply forapply)context(Any, defaults toNone- The data that has been passed in thecontextparameter)Additionally, if your back-end keeps track of it, you can passspatial_dimsandtemporal_dimstoexecute_udfwhere each is a list of dimension names (as strings) for the corresponding dimension types spatial (x,y,z) and temporal.Without ParallelizationorWith Parallelization through DaskIf your back-end parallelizes already, you can directly run the following code:# import the UDF libraryfromopeneo_r_udf.udf_libimportprepare_udf,execute_udf# Define variables as documented above# Load UDF file (this should not be paralelized)udf_path=prepare_udf(udf,udf_folder)# Execute UDF file (this can be parallelized)result=execute_udf(process,udf_path,data,dimension=dimension,context=context)If you parallelize with Dask, the xarray.DataArray must consist of Dask arrays, i.e. thechunksattribute of the DataArray MUST NOT beNone.With Parallelization through joblibIf your back-end is not parallelizing at all, you can run the following:# import the UDF library - make sure to install joblib beforefromopeneo_r_udf.udf_libimportprepare_udf,execute_udf,chunk_cube,combine_cubesfromjoblibimportParallel,delayedasjoblibDelayed# Parallelization configchunk_size=1000num_jobs=-1# Define variables as documented above# Load UDF file (this should not be paralelized)udf_path=prepare_udf(udf,udf_folder)# Define callback functiondefcompute_udf(data):returnexecute_udf(process,udf_path,data.compute(),dimension=dimension,context=context)# Run UDF in parallelinput_data_chunked=chunk_cube(data,size=chunk_size)results=Parallel(n_jobs=num_jobs,verbose=51)(joblibDelayed(compute_udf)(data)fordataininput_data_chunked)result=combine_cubes(results)Theresultvariable holds the processed data as anxarray.DataArrayagain.Writing a UDFThis is for end-usersA UDF must be written differently depending on where it is executed.
The underlying library used for data handling is alwaysstars.applyA UDF that is executed inside the processapplymanipulates the values on a per-pixel basis.
Youcan'tadd or remove labels or dimensions.The UDF function must be namedudfand receives two parameters:xis a multi-dimensional stars object and you can run vectorized functions on a per pixel basis, e.g.abs.contextpasses through the data that has been passed to thecontextparameter of the parent process (here:apply). If nothing has been provided explicitly, the parameter is set toNULL.This can be used to pass through configurable options, parameters or some additional data.
For example, if you would executeapply(process = run_udf(...), context = list(m = -1, max = -100))then you could access the corresponding values in the UDF below ascontext$mandcontext$max(see example below).The UDF must return a stars object with exactly the same shape.Example:udf=function(x,context){max(abs(x)*context$a,context$max)}apply_dimensionA UDF that is executed inside the processapply_dimensiontakes all values along a dimension and computes new values based on them.
This could for example compute a moving average over a timeseries.There are two different variants of UDFs that can be used as processes forapply_dimension.
A reducer can be executed either "vectorized" or "per chunk".
This is the same behavior as defined forreduce_dimension.
Please see below for more details.reduce_dimensionA UDF that is executed inside the processreduce_dimensiontakes all values along a dimension and computes a single value for it.
This could for example compute an average for a timeseries.There are two different forms of UDFs that can be used as reducers
forreduce_dimension: a reducer can be executed either "vectorized"
or "per chunk".vectorizedThe vectorized variant is usually the more efficient variant as it's executed once on a larger chunk of the data cube.The UDF function must be namedudfand receives two parameters:datais a matrix. Each row contains the values for a "pixel" and the columns are the values along the given dimension.
So if you reduce along the temporal dimension, the columns are the individual timestamps.context-> see the description ofcontextforapply.The UDF must return a list of values.Example:Please note that you may need to usedo.callto execute functions in a vectorized way. We also need to usepmaxfor this, instead ofmax.udf=function(data,context){# To get the labels for the values once:# labels = names(data)do.call(pmax,data)*context}The input data may look like this if you reduce along a band dimension with three bandsr,gandb:datacould belist(r = c(1, 2, 6), g = c(3, 4, 5), b = c(7, 1, 0))names(data)would bec("r", "g", "b")Executing the example above would returnc(7, 4, 6)per chunkThis variant is usually slower, but might be required for certain use cases. It is executed multiple times on the smallest chunk possible for the dimension given, e.g. a single time series.The UDF function must be namedudf_chunkedand receives two parameters:datais a list of values, e.g. a single time series which you could pass tomaxormean.context-> see the description ofcontextforapply.The UDF must return a single value.Example:udf_chunked=function(data,context){# To get the labels for the values:# labels = names(data)max(data)}The input data may look like this if you reduce along a band dimension with three bandsr,gandb:datacould bec(1, 2, 3)names(data)would bec("r", "g", "b")Exeucting the example above would return3Setup and TeardownAsudf_chunkedis usually executed many times in a row, you can optionally define two additional functions that are executed once before and once after the execution.
These functions must be namedudf_setupandudf_teardownand be placed in the same file asudf_chunked.udf_setupcould be useful to initially load some data, e.g. a machine learning (ML) model.udf_teardowncould be used to clean-up stuff that has been opened inudf_setup.Note:udf_setupandudf_teardownare only available if you implementudf_chunked.
If you implementudf, the two additional functions are not available as you can execute them directly in theudffunction, which is only executed once (for each worker).Both functions receive a single parameter, which is thecontextparameter explained above.
Here the context parameter could contain the path to a ML model file, for example.
By using the context parameter, you can avoid hard-coding information, which helps to make UDFs reusable.Example:udf_setup=function(context){# e.g. load a ML model from a file}udf_chunked=function(data,context){max(data)}udf_teardown=function(context){# e.g. clean-up tasks}Note:udf_teardownis only executed if none of theudf_chunkedcalls has resulted in an error.If you'd like to make some data available inudf_chunkedand/orudf_teardownthat you have prepared inudf_setup(orudf_chunked), you can use a global variable
and thespecial assignment operator<<-to assign to variables in the outer environments.Example:This loads a trained ML model object from an URL inudf_setupand makes it available for prediction inudf_chunked.
This is important as loading the ML model in udf_chunked may download the model very often, usually thousands of times and as such the computation gets very slow.model<-NULLudf_setup=function(context){model<<-load_model("https://example.com/model")}udf_chunked=function(data,context){return(predict(data,model))}ExamplesDockerimage for running on a backendHere's an example of an Dockerimage that is used to run the R-UDF service on an openEO platform backend.https://github.com/Open-EO/r4openeo-usecases/tree/main/vito-dockerImplementation at EuracHere is an example how the R-UDF service is integrated in the Eurac openEO backend based on Open Data Cube.https://github.com/SARScripts/openeo_odc_driver/blob/f34cd35107e4fb137fc1d23cae246ed362517c75/openeo_odc_driver.py#L289R4openEO use casesHere are use cases that use the R-UDF service.https://github.com/Open-EO/r4openeo-usecasesDevelopmentClone this repository and switch into the corresponding folder.Install environment via conda:conda env create -f environment.ymlInstall package for development:pip install -e .Now you can run one of the tests for example:python3 tests/test.py