Package: jupyterhub-multicluster-kubespawner
jupyterhub-multicluster-kubespawnerLaunch user pods into many different kubernetes clusters from the same JupyterHub!Why?A single JupyterHub as an 'entrypoint' to compute in a variety of clusters
can be extremely useful. Users can dynamically decide to launch their notebooks
(and dask clusters) dynamically based on a variety of factors - closer to their
data, on different cloud providers, paid for by different billing accounts, etc.
It also makes life much easier for JupyterHub operators.You can check out an early demo of the spawner in actionhere.Installationjupyterhub-multicluster-kubespawneris available from PyPI:pipinstalljupyterhub-multicluster-kubespawnerYou'll also need to installkubectlas well as any tools needed toauthenticateto your target clusters.Cloud ProviderToolGoogle CloudgcloudAWSawsAzureazDigitalOceandoctlConfigurationYou can ask JupyterHub to useMultiClusterKubeSpawnerwith the following config
snippet in yourjupyterhub_config.pyfile, although more configuration is
needed to connect the hub to different clusters.Configuration philosophyMultiClusterKubeSpawnertries to be as kubernetes-native as possible, unlike
the venerablekubespawner. It doesn't
try to provide a layer of abstraction over what kubernetes offers, as we have
found that is often a very leaky abstraction. This makes it difficult for JupyterHub
operators to take advantage of all the powerful features Kubernetes offers, and
increases maintenance burden for the maintainers.MultiClusterKubeSpawneruses the popularkubectlunder the hood, making the configuration familiar for anyone who has a basic
understanding of working with Kubernetes clusters. The flip side is thatsomefamiliarity with Kubernetes is required to successfully configure this spawner,
but the tradeoff seems beneficial for everyone.Setting upKUBECONFIGSince multicluster-kubespawner talks to multiple Kubernetes clusters, it uses akubeconfigfile connect to the kubernetes clusters. It looks for the file in~/.kube/config-
in production environments, your file probably exists elsewhere - you can set theKUBECONFIGenvironment variable to point to the location of the file.Each cluster is represented by acontext,
which is a combination of a pointer to where the cluster's kubernetes API endpoint
is as well as what credentials to use to authenticate to it. More detailshere.The easiest way to construct akubeconfigthat will work with all the clusters
you want to use is tocarefullyconstruct it locally on your laptop and then
copy that file to your deployment.Start by setting yourKUBECONFIGenv var locally to a file that you can then copy
over.exportKUBECONFIG=jupyterhub-mcks-kubeconfigOn Google CloudCreate aGoogle Cloud Service Account,
and give it enough permissions to access your Kubernetes cluster.roles/container.developershouldbe enough permission.Create a JSONService Account Keyfor your service account. This is whatkubectlwill eventually use to authenticate to
the kubernetes cluster. You'll need to put this service account key in your production
JupyterHub environment as well.Set theGOOGLE_APPLICATION_CREDENTIALSenvironment variable to point to the location of
this JSON Service Account key.exportGOOGLE_APPLICATION_CREDENTIALS=<path-to-json-service-account-key>Generate an appropriate entry in your customkubeconfigfile.gcloudcontainerclustersget-credentials<cluster-name>--zone=<zone>When you deploy your JupyterHub, make sure that you set the environment variableGOOGLE_APPLICATION_CREDENTIALSto point to the service account key in a place where
JupyterHub can find it.On AWS with EKSAWS has a plethora of interesting options to authenticate with it, here we will
specify the simplest (albeit maybe not the most secure or 'best practice').Create an AWSIAM Userfor use bykubectlto authenticate to AWS. This user will need a access key
and access secret, but no console access.Create anaccess keyfor this user. JupyterHub will need these while running to make requests to the kubernetes
API, set asenvironment variables.Grant the user access to theeks:DescribeClusterpermission, either directly or
via a group you create specifically for this purpose.Grant the user access to the Kubernetes API by editing theaws-authconfigmap
asdescribed in this document.Generate an appropriate entry in yourKUBECONFIGfile:exportAWS_ACCESS_KEY_ID=<access-key-id>exportAWS_SECRET_ACCESS_KEY=<access-key-secret>
awseksupdate-kubeconfig--name=<cluster-name>--region=<aws-region>When you deploy your JupyterHub, you need to set the environment variablesAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYon the JupyterHub process itself
so it can talk to the Kubernetes API properly.On DigitalOceanInstalldoctlCreate apersonal access tokento access your kubernetes cluster. Unlike the other cloud providers, this
can not be scoped - it grants full access to your entire DO account! So
use with care.Generate an appropriate entry in yourKUBECONFIGfile:exportDIGITALOCEAN_ACCESS_TOKEN=<your-digitalocean-access-token>
doctlkubernetesclusterkubeconfigsave<cluster-name>When you deploy your JupyterHub, you need to set the environment variableDIGITALOCEAN_ACCESS_TOKENon the JupyterHub process itself so it can talk
to the Kubernetes API properly.Setting up target clustersEach target cluster needs to have anIngress Controllerinstalled that the spawner can talk to. This provides apublic IPthat
JupyterHub can use to proxy traffic to the user pods on that cluster.Any ingress provider will do, although the current suggested one is
to useProject Contour,
as it's faster than the more popularnginx-ingressat picking up routing changes.A 'production' install might usehelmand thecontour helm chart.
But to quickly get started, you can also just configure yourkubectlto point to the correct kubernes cluster and runkubectl apply --wait -f https://projectcontour.io/quickstart/contour.yaml. After
it succeeds, you can get the public IP of the ingress controller withkubectl -n projectcontour get svc envoy. TheEXTERNAL-IPvalue here
can be passed to theingress_public_urlconfiguration option for your
cluster.Setupprofile_listAfter login, each user will be provided with a list ofprofiles to choose from.
Each profile can point to a different kubernetes cluster, as well as other
customizations such as image to use, amount of RAM / CPU, GPU use, etc.Each item in the list is a python dictionary, with the following keys recognized:display_name: Name to display to the user in the profile selection screendescription: Description to display to the user in the profile selection screenspawner_override: Dictionary of spawner options for this profile, determining
where the user pod is spawned and other properties of it.The following properties are supported underspawner_override.kubernetes_context: Name of the kubernetes context to use when connecting to
this cluster. You can usekubectl config get-contextsto get a list of contexts
available in yourKUBECONFIGfile.ingress_public_url: URL to the public endpoint of the Ingress controller you
setup earlier. This should be formatted as a URL, so don't forget thehttp://.
For production systems, you should also setup HTTPS for your ingress controller,
and provide thehttps://<domain-name>here.patches: A list of patches (as passed tokubectl patch).
Described in more detail below. This is the primary method of customizing the
user pod, although some convenience methods are also offered (detailed below).environment: A dictionary of extra environment variables to set for the
user pod.image: The image to use for the user pod. Defaults topangeo/pangeo-notebook:latest.mem_limitandmem_guarantee, asunderstood by JupyterHubcpu_limitandcpu_guarantee, asunderstood by JupyterHubHere is a simple example:c.MultiClusterKubeSpawner.profile_list=[{"display_name":"Google Cloud in us-central1","description":"Compute paid for by funder A, closest to dataset X","spawner_override":{"kubernetes_context":"<kubernetes-context-name-for-this-cluster">,"ingress_public_url":"http://<ingress-public-ip-for-this-cluster>"}},{"display_name":"AWS on us-east-1","description":"Compute paid for by funder B, closest to dataset Y","spawner_override":{"kubernetes_context":"<kubernetes-context-name-for-this-cluster">,"ingress_public_url":"http://<ingress-public-ip-for-this-cluster>","patches":{"01-memory":"""kind: Podmetadata:name: {{key}}spec:containers:- name: notebookresources:requests:memory: 16Mi""",}}},]Customizations withpatchesTo try and be as kubernetes-native as possible, we usestrategic merge patchas implemented by kubectl to allow JupyterHub operators to customize per-user resources.
This lets operators have fine grained control over what gets spawned per-user, without
requiring a lot of effort by the maintainers of this spawner to support each possible
customization.Behind the scenes,kubectl patchis used to merge the initial list of generated
kubernetes resources for each user with some customizations before they are passed tokubectl apply. Operators set these by customizing thepatchestraitlet. It can
be either set for all profiles by settingc.MultiClusterKubeSpawner.patchesor just
for a particular set of profiles by settingpatchesunderspawner_overridefor that
particular profile.patchesis a dictionary, where the key is used just for sorting and the value is
a string that should be a valid YAML object when parsed after template substitution.
Resources are merged based on the value forkindandmetadata.namekeys in the YAML.kubectlknows when to add items to a list or merge their properties on appropriate
attributes.To patch the user pod to add some extra annotations to the pod and request a GPU,
you could set the following:c.MultiClusterKubernetesSpawner.patches={"01-annotations":"""kind: Podmetadata:name: {{key}}annotations:something-else: hey""","02-gpu":"""kind: Podmetadata:name: {{key}}spec:containers:- name: notebookresources:limits:nvidia.com/gpu: 1""",}The values are first expanded viajinja2 templatesbefore being passed tokubectl patch.{{key}}expands to the name of the resource
created, and you should use it for all your modifications. In the02-gpupatch,kubectlknows to merge this with the existing notebook container instead of create a new container
or replace all the existing values, because it knows there already exists a container with
thenameproperty set tonotebook. Hence it merges values provides in this patch with
the existing configuration for the container.Please read thekubectldocumentationto understand how strategic merge patch works.Additional per-user kubernetes resources withresourcesYou can also create arbitrary additional kubernetes resources for each user
by setting theresourcesconfiguration. It's a dictionary where the key is used
for sorting, and the value should be valid YAML after expansion via jinja2 template.For example, the following config creates a KubernetesRole and RoleBindingfor each user to allow them to (insecurely!) run adask-kubernetescluster.c.MultiClusterKubernetesSpawner.resources={"10-dask-role":"""apiVersion: rbac.authorization.k8s.io/v1kind: Rolemetadata:name: {{key}}-daskrules:- apiGroups:- ""resources:- podsverbs:- list- create- delete""","11-dask-rolebinding":"""apiVersion: rbac.authorization.k8s.io/v1kind: RoleBindingmetadata:name: {{key}}-daskroleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: {{key}}-dasksubjects:- apiGroup: ""kind: ServiceAccountname: {{key}}""",)This takes advantage of the fact that by default a KubernetesService Accountis already created for each pod byMultiClusterUserSpawner, and gives it just
enough rights to create, list and delete pods.