Project: Flink (App Definition and Development / Streaming & Messaging)
File: dependency_management.md

title: "Dependency Management" weight: 30 type: docs aliases: /dev/python/table-api-users-guide/dependency_management.html /dev/python/datastream-api-users-guide/dependency_management.html /dev/python/dependency_management.html <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> There are requirements to use dependencies inside the Python API programs. For example, users may need to use third-party Python libraries in Python user-defined functions. In addition, in scenarios such as machine learning prediction, users may want to load a machine learning model inside the Python user-defined functions. When the PyFlink job is executed locally, users could install the third-party Python libraries into the local Python environment, download the machine learning model to local, etc. However, this approach doesn't work well when users want to submit the PyFlink jobs to remote clusters. In the following sections, we will introduce the options provided in PyFlink for these requirements. <span class="label label-info">Note</span> Both Python DataStream API and Python Table API have provided APIs for each kind of dependency. If you are mixing use of Python DataStream API and Python Table API in a single job, you should specify the dependencies via Python DataStream API to make them work for both the Python DataStream API and Python Table API. If third-party JARs are used, you can specify the JARs in the Python Table API as following: ```python tableenv.getconfig().set("pipeline.jars", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") tableenv.getconfig().set("pipeline.jars", "file:///E:/my/jar/path/connector.jar;file:///E:/my/jar/path/udf.jar") tableenv.getconfig().set("pipeline.classpaths", "file:///my/jar/path/connector.jar;file:///my/jar/path/udf.jar") ``` or in the Python DataStream API as following: ```python streamexecutionenvironment.add_jars("file:///my/jar/path/connector1.jar", "file:///my/jar/path/connector2.jar") streamexecutionenvironment.add_jars("file:///E:/my/jar/path/connector1.jar", "file:///E:/my/jar/path/connector2.jar") streamexecutionenvironment.add_classpaths("file:///my/jar/path/connector1.jar", "file:///my/jar/path/connector2.jar") ``` or through the `--jarfile` when submitting the job. <span class="label label-info">Note</span> It only supports to specify one jar file with the command line argument `--jarfile` and so you need to build a fat jar if there are multiple jar files. You may want to use third-part Python libraries in Python user-defined functions. There are multiple ways to specify the Python libraries. You could specify them inside the code using Python Table API as following: ```python tableenv.addpythonfile(filepath) ``` or using Python DataStream API as following: ```python

``` You could also specify the Python libraries using configuration or via `-pyfs` or `--pyFiles` when submitting the job. <span class="label label-info">Note</span> The Python libraries could be local files or local directories. They will be added to the PYTHONPATH of the Python UDF worker. It also allows to specify a `requirements.txt` file which defines the third-party Python dependencies. These Python dependencies will be installed into the working directory and added to the PYTHONPATH of the Python UDF worker. You could prepare the `requirements.txt` manually as following: ```shell echo numpy==1.16.5 >> requirements.txt echo pandas==1.0.0 >> requirements.txt ``` or using `pip freeze` which lists all the packages installed in the current Python environment: ```shell pip freeze > requirements.txt ``` The content of the requirements.txt file may look like the following: ```shell numpy==1.16.5 pandas==1.0.0 ``` You could manually edit it by removing unnecessary entries or adding extra entries, etc. The `requirements.txt` file could then be specified inside the code using Python Table API as following: ```python tableenv.setpython_requirements( requirementsfilepath="/path/to/requirements.txt", requirementscachedir="cached_dir") ``` or using Python DataStream API as following: ```python streamexecutionenvironment.setpythonrequirements( requirementsfilepath="/path/to/requirements.txt", requirementscachedir="cached_dir") ``` <span class="label label-info">Note</span> For the dependencies which could not be accessed in the cluster, a directory which contains the installation packages of these dependencies could be specified using the parameter `requirementscacheddir`. It will be uploaded to the cluster to support offline installation. You could prepare the `requirementscachedir` as following: ```shell pip download -d cached_dir -r requirements.txt --no-binary :all: ``` <span class="label label-info">Note</span> Please make sure that the prepared packages match the platform of the cluster, and the Python version used. You could also specify the `requirements.txt` file using configuration or via `-pyreq` or `--pyRequirements` when submitting the job. <span class="label label-info">Note</span> It will install the packages specified in the `requirements.txt` file using pip, so please make sure that pip (version >= 20.3) and setuptools (version >= 37.0.0) are available. You may also want to specify archive files. The archive files could be used to specify custom Python virtual environments, data files, etc. You could specify the archive files inside the code using Python Table API as following: ```python tableenv.addpythonarchive(archivepath="/path/to/archivefile", targetdir=None) ``` or using Python DataStream API as following: ```python streamexecutionenvironment.addpythonarchive(archivepath="/path/to/archivefile", target_dir=None) ``` <span class="label label-info">Note</span> The parameter `target_dir` is optional. If specified, the archive file will be extracted to a directory with the specified name of `target_dir` during execution. Otherwise, the archive file will be extracted to a directory with the same name as the archive file. Suppose you have specified the archive file as following: ```python tableenv.addpythonarchive("/path/to/pyenv.zip", "myenv") ``` Then, you could access the content of the archive file in Python user-defined functions as following: ```python def my_udf(): with

as f: ... ``` If you have not specified the parameter `target_dir`: ```python tableenv.addpythonarchive("/path/to/pyenv.zip") ``` You could then access the content of the archive file in Python user-defined functions as following: ```python def my_udf(): with open("pyenv.zip/pyenv/data/data.txt") as f: ... ``` <span class="label label-info">Note</span> The archive file will be extracted to the working directory of Python UDF worker and so you could access the files inside the archive file using relative path. You could also specify the archive files using configuration or via `-pyarch` or `--pyArchives` when submitting the job. <span class="label label-info">Note</span> If the archive file contains a Python virtual environment, please make sure that the Python virtual environment matches the platform that the cluster is running on. <span class="label label-info">Note</span> Currently, only zip files (i.e., zip, jar, whl, egg, etc) and tar files (i.e., tar, tar.gz, tgz) are supported. It supports to specify the path of the Python interpreter to execute Python worker. You could specify the Python interpreter inside the code using Python Table API as following: ```python tableenv.getconfig().setpythonexecutable("/path/to/python") ``` or using Python DataStream API as following: ```python streamexecutionenvironment.setpythonexecutable("/path/to/python") ``` It also supports to use the Python interpreter inside an archive file. ```python tableenv.addpythonarchive("/path/to/pyenv.zip", "venv") tableenv.getconfig().setpythonexecutable("venv/py_env/bin/python") streamexecutionenvironment.addpythonarchive("/path/to/py_env.zip", "venv") streamexecutionenvironment.setpythonexecutable("venv/py_env/bin/python") ``` You could also specify the Python interpreter using configuration or via `-pyexec` or `--pyExecutable` when submitting the job. <span class="label label-info">Note</span> If the path of the Python interpreter refers to the Python archive file, relative path should be used instead of absolute path. Python is needed at the client side to parse the Python user-defined functions during compiling the job. You could specify the custom Python interpreter used at the client side by activating it in the current session. ```shell source my_env/bin/activate ``` or specify it using configuration , `-pyclientexec` or `--pyClientExecutable`, environment variable It also supports to use Python user-defined functions in the Java Table API programs or pure SQL programs. The following code shows a simple example on how to use the Python user-defined functions in a Java Table API program: ```java import org.apache.flink.configuration.CoreOptions; import org.apache.flink.table.api.EnvironmentSettings; import org.apache.flink.table.api.TableEnvironment; TableEnvironment tEnv = TableEnvironment.create( EnvironmentSettings.inBatchMode()); tEnv.getConfig().set(CoreOptions.DEFAULT_PARALLELISM, 1); // register the Python UDF tEnv.executeSql("create temporary system function addone as 'addone.add_one' language python"); tEnv.createTemporaryView("source", tEnv.fromValues(1L, 2L, 3L).as("a")); // use Python UDF in the Java Table API program tEnv.executeSql("select add_one(a) as a from source").collect(); ``` You can refer to the SQL statement about for more details on how to create Python user-defined functions using SQL statements. The Python dependencies could then be specified via the Python , such as python.archives, python.files, python.requirements, python.client.executable, python.executable. etc or through when submitting the job.