Project: Kubeflow (Orchestration & Management / Scheduling & Orchestration)
File: ksonnet_packages.md

<!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> Table of Contents generated with - - - <!-- END doctoc generated TOC please keep comment here to allow auto update --> Here are some instructions for creating new Kubeflow packages. Decide whether the component belongs in a new ksonnet Packages correspond to subdirectories of Here's a simple decision tree If all Kubeflow deployments should include this package put it in core Optional packages probably don't belong in core Otherwise, if there's an existing package that it belongs with put it in that package If no existing package is a good fit create a new one If you are creating a new package follow these steps Copy new-package-stub to a new directory giving it an appropriate name modify parts.yaml setting the following fields name description modify registry.yaml adding an entry for the new package Create a libsonnet file see to define the manifests for your component If you have an existing YAML manifest you can just convert that to json and use that as a starting point for your parts If you have a helm chart you can use the template command to output a YAML file ``` helm template path/to/chart ``` You can use to easily convert a file containing YAML manifests to jsonnet Typically you will want to make the following changes Use params for any variables that should be easily overwritable For helm packages substitute `params.name` for the name of the release Create one or more prototype files in the prototypes directory for your package Use as a template. The easiest way to test ksonnet components is to follow the normal instructions for setting up a ksonnet app to deploy kubeflow. Then replace the directory `vendor/kubeflow` with a symbolic link to `${GIT_KUBEFLOW}/kubeflow` ``` ln -sf ${GITKUBEFLOW}/kubeflow ${APPDIR}/vendor/kubeflow ``` this way changes to your .libsonnet files will automatically be reflected in your components. If you make changes to prototypes you need to regenerate the prototype. You can just delete the `.jsonnet` file in your app's component directory and then regenerate the component using `ks generate`. ``` rm -rf ${APP_DIR}/components/jupyterhub.jsonnet ks generate jupyterhub jupyterhub INFO Writing component at 'components/jupyterhub' ERROR Component parameters for 'jupyterhub' already exists ``` If you use the same component name you will preserve the values of any parameters you have set. ksonnet will print a warning but it works. However, the parameters for the component is not changed. So if you added new @optionalParam, it will not be set in the `component/params.libsonnet` and you will see errors like `RUNTIME ERROR: Field does not exist: NEWOPTIONALPARAM`. In this case, use a new component name. If you are dealing with a new package you need to modify `app.yaml` in your ksonnet application and add an entry in the libraries section for your new package. There is what appears to be a problem when trying to introduce a new package for Kubeflow: the tests are of the master branch but the PR is its own separate git ref. However: the Argo workflow checks out the source at the commit/PR being tested a subsequent step sets up the ksonnet directory and does `ks registry add` for the new package the test then replaces `vendor/kubeflow` with a symlink to the source checked out in step 1 This ensures that we are testing the components at the SHA for the commit being tested.