Package: kfp-tekton
Kubeflow Pipelines SDK for TektonThe Kubeflow PipelinesSDKallows data scientists to define end-to-end machine learning and data pipelines.
The output of the Kubeflow Pipelines SDK compiler is YAML forArgo.Thekfp-tektonSDK is extending theCompilerand theClientof the Kubeflow
Pipelines SDK to generateTektonYAML
and to subsequently upload and run the pipeline with the Kubeflow Pipelines engine
backed by Tekton.Table of ContentsSDK Packages OverviewProject PrerequisitesInstallationCompiling a Kubeflow Pipelines DSL ScriptBig data passing workspace configurationRunning the Compiled Pipeline on a Tekton ClusterList of Available FeaturesList of Helper Functions for Python Kubernetes ClientTested PipelinesTroubleshootingSDK Packages OverviewThekfp-tektonSDK is an extension to theKubeflow Pipelines SDKadding theTektonCompilerand theTektonClient:kfp_tekton.compilerincludes classes and methods for compiling pipeline Python DSL into a Tekton PipelineRun YAML spec. The methods in this package
include, but are not limited to, the following:kfp_tekton.compiler.TektonCompiler.compilecompiles your Python DSL code
into a single static configuration (in YAML format) that the Kubeflow Pipelines service
can process. The Kubeflow Pipelines service converts the static
configuration into a set of Kubernetes resources for execution.kfp_tekton.TektonClientcontains the Python client libraries for theKubeflow Pipelines API.
Methods in this package include, but are not limited to, the following:kfp_tekton.TektonClient.upload_pipelineuploads a local file to create a new pipeline in Kubeflow Pipelines.kfp_tekton.TektonClient.create_experimentcreates a pipelineexperimentand returns an
experiment object.kfp_tekton.TektonClient.run_pipelineruns a pipeline and returns a run object.kfp_tekton.TektonClient.create_run_from_pipeline_funccompiles a pipeline
function and submits it for execution on Kubeflow Pipelines.kfp_tekton.TektonClient.create_run_from_pipeline_packageruns a local
pipeline package on Kubeflow Pipelines.Project PrerequisitesPython:3.8or later. For Python 3.12, make sure to not have theSETUPTOOLS_USE_DISTUTILSflag because it's alreadydeprecated.Tekton:v0.53.2orlaterTekton CLI:0.30.1Kubeflow Pipelines:KFP with Tekton backendFollow the instructions forinstalling project prerequisitesand take note of some important caveats.InstallationYou can install the latest release of thekfp-tektoncompiler fromPyPi. We recommend to create a Python
virtual environment first:python3 -m venv .venv
source .venv/bin/activate

pip install kfp-tektonAlternatively you can install the latest version of thekfp-tektoncompiler
from the source by cloning the repositoryhttps://github.com/kubeflow/kfp-tekton:Clone thekfp-tektonrepo:git clone https://github.com/kubeflow/kfp-tekton.git
cd kfp-tektonSetup Python environment with Conda or a Python virtual environment:python3 -m venv .venv
source .venv/bin/activateBuild the compiler:pip install -e sdk/pythonRun the compiler tests (optional):pip install pytest
make testCompiling a Kubeflow Pipelines DSL ScriptThekfp-tektonPython package comes with thedsl-compile-tektoncommand line
executable, which should be available in your terminal shell environment after
installing thekfp-tektonPython package.If you cloned thekfp-tektonproject, you can find example pipelines in thesamplesfolder or undersdk/python/tests/compiler/testdatafolder.dsl-compile-tekton \
    --py sdk/python/tests/compiler/testdata/parallel_join.py \
    --output pipeline.yamlNote: If the KFP DSL script contains a__main__method calling thekfp_tekton.compiler.TektonCompiler.compile()function:if__name__=="__main__":fromkfp_tekton.compilerimportTektonCompilerTektonCompiler().compile(pipeline_func,"pipeline.yaml")... then the pipeline can be compiled by running the DSL script withpython3executable from a command line shell, producing a Tekton YAML filepipeline.yamlin the same directory:python3 pipeline.pyBig data passing workspace configurationWhenbig data filesare defined in KFP. Tekton will create a workspace to share these big data files
among tasks that run in the same pipeline. By default, the workspace is a
Read Write Many PVC with 2Gi storage using the kfp-csi-s3 storage class to push artifacts to S3.
But you can change these configuration using the environment variables below:exportDEFAULT_ACCESSMODES=ReadWriteManyexportDEFAULT_STORAGE_SIZE=2GiexportDEFAULT_STORAGE_CLASS=kfp-csi-s3To pass big data using cloud provider volumes, it's recommended to use thevolume_based_data_passing_methodfor both Tekton and Argo runtime.If you want to change the input and output copy artifact images, please modify the following environment variables:exportTEKTON_BASH_STEP_IMAGE=busybox# input and output copy artifact imagesexportTEKTON_COPY_RESULTS_STEP_IMAGE=library/bash# output copy results imagesexportCONDITION_IMAGE_NAME=python:3.9.17-alpine3.18# condition task default image nameRunning the Compiled Pipeline on a Tekton ClusterAfter compiling thesdk/python/tests/compiler/testdata/parallel_join.pyDSL script
in the step above, we need to deploy the generated Tekton YAML to Kubeflow Pipeline engine.You can run the pipeline directly using a pre-compiled file and KFP-Tekton SDK. For more details, please look at theKFP-Tekton user guide SDK documentationexperiment=kfp_tekton.TektonClient.create_experiment(name=EXPERIMENT_NAME,namespace=KUBEFLOW_PROFILE_NAME)run=client.run_pipeline(experiment.id,'parallal-join-pipeline','pipeline.yaml')You can also deploy directly on Tekton cluster withkubectl. The Tekton server will automatically start a pipeline run.
We can then follow the logs using thetknCLI.kubectl apply -f pipeline.yaml

tkn pipelinerun logs --last --followOnce the Tekton Pipeline is running, the logs should start streaming:Waiting for logs to be available...

[gcs-download : main] With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate

[gcs-download-2 : main] I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath

[echo : main] Text 1: With which he yoketh your rebellious necks Razeth your cities and subverts your towns And in a moment makes them desolate
[echo : main]
[echo : main] Text 2: I find thou art no less than fame hath bruited And more than may be gatherd by thy shape Let my presumption not provoke thy wrath
[echo : main]List of Available FeaturesTo understand how each feature is implemented and its current status, please visit
theFEATURESdoc.List of Helper Functions for Python Kubernetes ClientKFP Tekton provides a list of common Kubernetes client helper functions to simplify
the process of creating certain Kubernetes resources. please visit theK8S_CLIENT_HELPERdoc for more details.Tested PipelinesWe aretesting the compileron more than 80 pipelines
found in the Kubeflow Pipelines repository, specifically the pipelines in KFP compilertestdatafolder, the KFP core samples and the samples contributed by third parties.A report card of Kubeflow Pipelines samples that are currently supported by thekfp-tektoncompiler can be foundhere.
If you work on a PR that enables another of the missing features please ensure that
your code changes are improving the number of successfully compiled KFP pipeline samples.TroubleshootingWhen you encounter ServiceAccount related permission issues, refer to the"Service Account and RBAC" docIf you run into the errorbad interpreter: No such file or directorwhen trying
to use Python's venv, remove the current virtual environment in the.venvdirectory
and create a new one usingvirtualenv .venv