Package: kfx
kfxkfxis a python package with the namespacekfx. Currently, it provides the
following sub-packageskfx.lib.dsl- Extensions to the kubeflow pipeline dsl.kfx.lib.vis- Data models and helpers to help generate themlpipeline-metrics.jsonandmlpipeline-ui-metadata.jsonrequired to render visualization in the kubeflow pipeline UI. See alsohttps://www.kubeflow.org/docs/pipelines/sdk/pipelines-metrics/andhttps://www.kubeflow.org/docs/pipelines/sdk/output-viewer/Documentation:https://kfx.readthedocs.io.Repo:https://github.com/e2fyi/kfxNOTE this is currently alphaThere will likely to have breaking changes, and feel free to do a feature requestKnown issueskfx.vis.vega.vega_web_appandKfpArtifactdoes not work well together (see example) because of CORs - the web app is hosted inside an iFrame which prevents it from accessing theml-pipeline-uiAPI server.kfx.vis.vega.vega_web_appis only supported in the latest kubeflow pipeline UI (as inline is only supported after0.2.5)ChangelogRefer toCHANGELOG.md.Quick startInstallationpipinstallkfxUsageExample: UsingContainerOpTransformto configure the internal k8s properties
of kubeflow pipelines tasks.kfx.dsl.ContainerOpTransformis a helper to modify the interal k8s properties
(e.g. resources, environment variables, etc) of kubeflow pipeline tasks.importkfp.componentsimportkfp.dslimportkfx.dsltransforms=(kfx.dsl.ContainerOpTransform().set_resources(cpu="500m",memory=("1G","4G")).set_image_pull_policy("Always").set_env_vars({"ENV":"production"}).set_env_var_from_secret("AWS_ACCESS_KEY",secret_name="aws",secret_key="access_key").set_annotations({"iam.amazonaws.com/role":"some-arn"}))@kfp.dsl.components.func_to_container_opdefecho(text:str)->str:print(text)returntext@kfp.dsl.pipeline(name="demo")defpipeline(text:str):op1=echo(text)op2=echo("%s-%s"%text)# u can apply the transform on op1 only# op1.apply(transforms)# or apply on all ops in the pipelinekfp.dsl.get_pipeline_conf().add_op_transformer(transforms)Example: UsingArtifactLocationHelperandKfpArtifactto determine the
uri of your data artifact generated by the kubeflow pipeline task.kfx.dsl.ArtifactLocationHelperis a helper to modify the kubeflow pipeline task
so that you can usekfx.dsl.KfpArtifactto represent the artifact generated
inside the task.importkfp.componentsimportkfp.dslimportkfx.dsl# creates the helper that has the argo configs (tells you how artifacts will be stored)# see https://github.com/argoproj/argo/blob/master/docs/workflow-controller-configmap.yamlhelper=kfx.dsl.ArtifactLocationHelper(scheme="minio",bucket="mlpipeline",key_prefix="artifacts/")@kfp.components.func_to_container_opdeftest_op(mlpipeline_ui_metadata:OutputTextFile(str),markdown_data_file:OutputTextFile(str)):"A test kubeflow pipeline task."importjsonimportkfx.dslimportkfx.visimportkfx.vis.vega# `KfpArtifact` provides the reference to data artifact created# inside this taskspec={"$schema":"https://vega.github.io/schema/vega-lite/v4.json","description":"A simple bar chart","data":{"values":[{"a":"A","b":28},{"a":"B","b":55},{"a":"C","b":43},{"a":"D","b":91},{"a":"E","b":81},{"a":"F","b":53},{"a":"G","b":19},{"a":"H","b":87},{"a":"I","b":52},]},"mark":"bar","encoding":{"x":{"field":"a","type":"ordinal"},"y":{"field":"b","type":"quantitative"},},}# write the markdown to the `markdown-data` artifactmarkdown_data_file.write("### hello world")# creates an ui metadata objectui_metadata=kfx.vis.kfp_ui_metadata(# Describes the vis to generate in the kubeflow pipeline UI.[# markdown vis from a markdown artifact.# `KfpArtifact` provides the reference to data artifact created# inside this taskkfx.vis.markdown(kfx.dsl.KfpArtifact("markdown_data_file")),# a vega web app from the vega data artifact.kfx.vis.vega.vega_web_app(spec),])# writes the ui metadata object as the `mlpipeline-ui-metadata` artifactmlpipeline_ui_metadata.write(kfx.vis.asjson(ui_metadata))# prints the uri to the markdown artifactprint(ui_metadata.outputs[0].source)@kfp.dsl.pipeline()deftest_pipeline():"A test kubeflow pipeline"op:kfp.dsl.ContainerOp=test_op()# modify kfp operator with artifact location metadata through env varsop.apply(helper.set_envs())Example: Usingpydanticdata models to generatemlpipeline-metrics.jsonandmlpipeline-ui-metadata.json.(See alsohttps://www.kubeflow.org/docs/pipelines/sdk/output-viewer/andhttps://www.kubeflow.org/docs/pipelines/sdk/pipelines-metrics/).kfx.vishas helper functions (with corresponding hints) to describe and
createmlpipeline-metrics.jsonandmlpipeline-ui-metadata.jsonfiles
(required by kubeflow pipeline UI to render any metrics or visualizations).importfunctoolsimportkfp.components# install kfxkfx_component=functools.partial(kfp.components.func_to_container_op,packages_to_install=["kfx"])@kfx_componentdefsome_op(# mlpipeline_metrics is a path - i.e. open(mlpipeline_metrics, "w")mlpipeline_metrics:kfp.components.OutputPath(str),# mlpipeline_ui_metadata is a FileLike obj - i.e. mlpipeline_ui_metadata.write("something")mlpipeline_ui_metadata:kfp.components.OutputTextFile(str),):"kfp operator that provides metrics and metadata for visualizations."# import inside kfp taskimportkfx.vis# output metrics to mlpipeline_metrics pathkfx.vis.kfp_metrics([# render as percentkfx.vis.kfp_metric("recall-score",0.9,percent=true),# override metric format with custom valuekfx.vis.kfp_metric(name="percision-score",value=0.8,metric_format="PERCENTAGE"),# render raw scorekfx.vis.kfp_metric("raw-score",123.45),]).write_to(mlpipeline_metrics)# output visualization metadata to mlpipeline_ui_metadata objkfx.vis.kfp_ui_metadata([# creates a confusion matrix viskfx.vis.confusion_matrix(source="gs://your_project/your_bucket/your_cm_file",labels=["True","False"],),# creates a markdown with inline sourcekfx.vis.markdown("# Inline Markdown: [A link](https://www.kubeflow.org/)",storage="inline",),# creates a markdown with a remote sourcekfx.vis.markdown("gs://your_project/your_bucket/your_markdown_file",),# creates a ROC curve with a remote sourcekfx.vis.roc("gs://your_project/your_bucket/your_roc_file",),# creates a Table with a remote sourcekfx.vis.table("gs://your_project/your_bucket/your_csv_file",header=["col1","col2"],),# creates a tensorboard viewerkfx.vis.tensorboard("gs://your_project/your_bucket/logs/*",),# creates a custom web app from a remote html filekfx.vis.web_app("gs://your_project/your_bucket/your_html_file",),# creates a Vega-Lite vis as a web appkfx.vis.vega.vega_web_app(spec={"$schema":"https://vega.github.io/schema/vega-lite/v4.json","description":"A simple bar chart with embedded data.","data":{"values":[{"a":"A","b":28},{"a":"B","b":55},{"a":"C","b":43},{"a":"D","b":91},{"a":"E","b":81},{"a":"F","b":53},{"a":"G","b":19},{"a":"H","b":87},{"a":"I","b":52}]},"mark":"bar","encoding":{"x":{"field":"a","type":"ordinal"},"y":{"field":"b","type":"quantitative"}}})]).write_to(mlpipeline_ui_metadata)Developer guideThis project used:isort: to manage import orderpylint: to manage general coding best practicesflake8: to manage code complexity and coding best practicesblack: to manage formats and stylespydocstyle: to manage docstr style/formatpytest/coverage: to manage unit tests and code coveragebandit: to find common security issuespyenv: to manage dev env: python version (3.6)pipenv: to manage dev env: python packagesConvention for unit tests are to suffix with_testand colocate with the actual
python module - i.e.<module_name>_test.py.The version of the package is read fromversion.txt- i.e. please update the
appropriate semantic version (major -> breaking changes, minor -> new features, patch -> bug fix, postfix -> pre-release/post-release).Makefile:# autoformat codes with docformatter, isort, and blackmakeformat# check style, formats, and code complexitymakecheck# check style, formats, code complexity, and run unit testsmaketest# test everything including building the package and check the sdistmaketest-all# run unit test onlymaketest-only# generate and update the requirements.txt and requirements-dev.txtmakerequirements# generate the docs with sphinx and autoapi extensionmakedocs# generate distributionsmakedists# publish to pypi with twine (twine must be configured)makepublish