Package: jupyterlab-pachyderm
jupyterlab-pachydermA JupyterLab extension for integrations with Pachyderm.This extension is composed of a Python package namedjupyterlab_pachydermfor the server extension and a NPM package namedjupyterlab-pachydermfor the frontend extension.RequirementsJupyterLab >= 3.0Python >=3.8,<4pyenvis a great way to manage and install different versions of python. You can check which version you are using by runningpyenv versions. Our Python extension is built to be compatible with Python versions 3.8 to 3.10. Therefore, it is best to run the lowest version (3.8.x) for highest compatibility.NodeIf you are usingnvmfirst runnvm install. This will install and switch the version of node to the one defined in the.nvmrc. If you are upgrading the version of node used in the project, please check and make sure that the versions defined in the.nvmrc.Development EnvironmentThere are two ways you can work on this project. One is setting up a local python virtual environment and the other is using a docker image developing on a virtual machine.Dev Container WorkflowBuilding and running the extension in a Docker container allows us to use mount, which is not possible in recent versions of macOS.Assuming that you are running a Pachyderm instance somewhere, and that you can port-forwardpachdkubectl port-forward service/pachd 30650:30650Start a bash session in thepachyderm/notebooks-usercontainer from the top-leveljupyterlab-pachydermrepo.docker run --name jupyterlab_pachyderm_frontend_dev \
  -p 8888:8888 \
  -it -e GRANT_SUDO=yes --user root \
  --device /dev/fuse --privileged \
  -v $(pwd):/home/jovyan/extension-wd \
  -w /home/jovyan/extension-wd \
  pachyderm/notebooks-user:<latest master SHA> \
  bashIf you are running the frontend container on Linux, and want to be able to talk to pachd in minikube ongrpc://localhost:30650, use--net=hostat the start of the thedocker runcommand as shown above. If you are on macOS, you will want to remove--net=hostas it is not supported. On macOS you can specifygrpc://host.docker.internal:30650to get the same effect.Install the project in editable mode, and start JupyterLabpip install -e ".[dev]"
jupyter labextension develop --overwrite

# Server extension must be manually installed in develop mode, for example
jupyter server extension enable jupyterlab_pachyderm

jupyter lab --allow-rootOpen another bash inside the same container:docker exec -it <container-id> bashWithin container run:npm run watchNote: Once run above, you can start the dev container again in future sessions usingdocker start <container id>and then runningdocker exec -it <container id> bashIterating on the mount server, from inside apachydermcheckout:CGO_ENABLED=0 make install
docker cp /home/luke/gocode/bin/pachctl jupyterlab_pachyderm_frontend_dev:/usr/local/bin/pachctl
docker exec -ti jupyterlab_pachyderm_frontend_dev pkill -f pachctlLocal Virtual Environment SetupWhen developing in python, it is good practice to set up a virtual environment. A simple guid to set up a virtual environment is as follows:
create a virtual environment using venvpython -m venv venvActivate the environmentsource venv/bin/activateWhen you are done using the environment you can close your shell or deactivate the environment:deactivateDevelopment installNote: You will need NodeJS to build the extension package.# Clone the repo to your local environment# Change directory to the jupyterlab-pachyderm directory (top level of repo)# Make sure you are using a virtual environment# Install package in development modepipinstall-e".[dev]"# Link your development version of the extension with JupyterLabjupyterlabextensiondevelop.--overwrite# Server extension must be manually installed in develop modejupyterserverextensionenablejupyterlab_pachyderm# Rebuild extension Typescript source after making changesnpmrunbuildYou can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension.# Watch the source directory in one terminal, automatically rebuilding when needednpmrunwatch# Run JupyterLab in another terminaljupyterlab--allow-root--ip=0.0.0.0With the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt).By default, thenpm run buildcommand generates the source maps for this extension to make it easier to debug using the browser dev tools. To also generate source maps for the JupyterLab core extensions, you can run the following command:jupyterlabbuild--minimize=FalseNote that to enable debug logging forjupyter(which also enables it for our extension backend, a python plugin to the JupyterLab server), you can run:jupyterlab--debugDevelopment uninstall# Server extension must be manually disabled in develop modejupyterserverextensiondisablejupyterlab_pachyderm
pipuninstalljupyterlab_pachydermIn development mode, you will also need to remove the symlink created byjupyter labextension developcommand. To find its location, you can runjupyter labextension listto figure out where thelabextensionsfolder is located. Then you can remove the symlink namedjupyterlab-pachydermwithin that folder.Locally building the docker imageUseful if iterating on the Dockerfile locally or iterating on changes to a version of mount-server.Create & activate venv:python3 -m venv venv
source venv/bin/activateBuilddistdirectory:python -m pip install --upgrade pip
python -m pip install -r ci-requirements.txt
python -m buildBuild docker image:export PACHCTL_VERSION=aaa7434c714fab6130c3982ebdaa8f279bd850c2 # or whichever version you want
docker build --build-arg PACHCTL_VERSION=$PACHCTL_VERSION -t pachyderm/notebooks-user:dev .
docker run -ti -p 8888:8888 pachyderm/notebooks-user:devNavigate to the URL that's printed out by the docker container, then changetreetolabin your browser's address bar.InstallTo install the extension, execute:pipinstalljupyterlab_pachydermUninstallTo remove the extension, execute:pipuninstalljupyterlab_pachydermServer extensionFirst make sure the server extension is enabled:jupyter server extension list 2>&1 | grep -ie "jupyterlab_pachyderm.*OK"Install Jupyterhub locally (Sidecar mode testing)helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/
helm repo update
helm install jhub jupyterhub/jupyterhub --version 2.0.0 --values scripts/jhub-dev-helm-values.ymlNote: Use any username and no password to loginAPI endpointsGET /repos # returns a list of all repos/branches
GET /mounts # returns a list of all active mounts and unmounted repos/branches
PUT /_mount # mounts a single repo
PUT /_unmount # unmounts a single repo
PUT /_commit # commits any changes to the repo
PUT /_unmount_all # unmounts all repos
PUT /datums/_mount # mounts first datum of given input spec
PUT /datums/_next # mounts next datum
PUT /datums/_prev # mounts prev datum
GET /datums # returns info on mounted datumThe servers-side extension extends jupyter server, so it automatically starts as part ofjupyter lab.API endpoints can be accessed vialocalhost:8888/pachyderm/v2The frontend can access the endpoints via/v2, for example:requestAPI<any>('/v2/repos').then(data=>{console.log(data);}).catch(reason=>{console.error(reason);});You can also access it vialocalhost:8888/v2SVG ImagesWe are leveragingsvgrto simplify the use of non icon svgs in in the project. Svg images that are to be converted live in thesvg-imagesfolder and get output to thesrc/utils/components/Svgsfolder. If you want to add a new image to the project simply add the svg to thesvg-imagesfolder and runnpm run build:svg. We have spent some time trying to get svgr to work through the@svgr/webpackplugin but have not been successful as of yet.Project StructureJupyter extensions are composed of several plugins. These plugins can be selectively enabled or disabled. Because of this we have decided separate the functionality in the extension using plugins. Plugins exported in this extension are as follows.HubThis plugin contains custom styling and other features only used by hub. By default this extension is disabled.MountThis plugin contains the mount feature currently under development.Plugin settingsYou can disable certain plugins by specifying the following config data in the<jupyter_config_path>/labconfig/page_config.json:{
  "disabledExtensions": {
    "jupyterlab-pachyderm:examples": true
  }
}You can check your config paths by runningjupyter --paths. Setting this config file is not part of the built extension and needs to be done by the user.Adding the following to the package.json in thejupyterlabobject will disable the examples plugin by default."disabledExtensions": ["jupyterlab-pachyderm:examples"]So we can build the extension with hub features turned off and override the setting for hub.TroubleshootIf you are seeing the frontend extension, but it is not working, check
that the server extension is enabled:jupyterserverextensionlistIf the server extension is installed and enabled, but you are not seeing
the frontend extension, check the frontend extension is installed:jupyterlabextensionlist