Package: slurm-jupyter-kernel
Slurm Jupyter KernelManage (create, list, modify and delete) and starting jupyter slurm kernels usingsbatch.slurmkernel is able to connect to a kernel started on a compute node using SSH port forwarding.
You can specify a SSH proxy jump, if you have to jump over two hosts (e.g. a loadbalancer)Slurm job 3251854 is in state "RUNNING"
Slurm job is in state running on compute node cn213
Starting SSH tunnel to forward kernel ports to localhost
Your started kernel is now ready to use on compute node cn213Features & Use-CasesStart Remote Jupyter kernel using sbatch (Slurm)Access to your local filesystem with remote code executionManage existing Slurm Jupyter kernelUse the template module to use pre-defined script templates for remote installation and local kernel creationCustom environment variables supported (e.g. JULIA_NUM_THREADS)Table of ContentsSlurm Jupyter KernelFeatures & Use-CasesTable of ContentsInstallationInstall using pipRequirements for usageCreate a new kernelTemplate module (Script templates)ExampleIPython ExampleRemote HostLocalhostSet kernel-specific environmentUsing the kernel with ApplicationsQuarto ExampleTroubleshootingKernel exceptionsDebuggingGet helpInstallationslurm_jupyter_kernelmust be installed locally where the Jupyter notebooks will run.Install using pippython3-mpipinstallslurm_jupyter_kernelRequirements for usageSSH-Key based authenticationYou need a running SSH agent with the loaded key file to access the loginnode without a password.Create a new kernelWe assume to install the Jupyter kernel tools into your$HOMEdirectory on your cluster.Template module (Script templates)With$ slurmkernel template {list, use, add, edit}you can use pre-defined script templates to initialize your remote environment (IJulia, IPython, ...), add new script templates or edit existing templates.If you want to create your own script templates, see here:Create Script TemplatesExampleNote:Add the parameter--dry-runto check the commands that will be executed!$slurmkerneltemplateuse--proxyjumplb.hpc.pc2.de--loginnodelogin001--userhpcuser1--templateipythonYou will be interactively asked for the required information if you do not pass any arguments when callingslurmkernel template useIPython ExampleRemote Hostload required software (if necessary)Create a Python virtual environmentInstall the IPython package (ipython, ipykernel)Create a wrapper script and mark it as executableremotehost~$moduleloadlangPython
remotehost~$python3-mvenvremotekernel/
remotehost~$sourceremotekernel/bin/activate(remotekernel)remotehost~$python3-mpipinstallipythonipykernel;deactivate
remotehost~$echo-e'#!/bin/bash\nmodule load lang Python\n\nsource remotekernel/bin/activate\n"$@"'>remotekernel/ipy_wrapper.sh&&chmod+xremotekernel/ipy_wrapper.shLocalhostKernel Remote Slurm kernel with commandslurmkernelnotebook~$slurmkernelcreate--displayname"Python 3.8.2"\--slurm-parameter="account=slurmaccount,time=00:30:00,partition=normal"\--kernel-cmd="\$HOME/remotekernel/ipy_wrapper.sh ipython kernel -f {connection_file}"\--proxyjump="lb.n1.pc2.uni-paderborn.de"\--loginnode="login-0001"\--language="python"Set kernel-specific environmentIf you want to set kernel specific environment variables (e.g.JULIA_NUM_THREADSfor the number of threads) just extend the jupyter kernelspec file withenv.Parameter forslurmkernel:--environment="JULIA_NUM_THREADS=4"More information here:https://jupyter-client.readthedocs.io/en/stable/kernels.htmlUsing the kernel with ApplicationsInstall kernel as shown aboveMake sure that you pass the--languageflag as well (e.g.pythonorjulia)Quarto ExampleTroubleshootingKernel exceptionsWhen you start a Jupyter slurm kernel, it throws an excpetion, depending on the error case. You can read the exception from the graphical user interface like JupyterLab.
If you start a kernel in the classic notebook view, you can click on "Error" on the left of the kernel status. There you can also find the exception.DebuggingIf your Slurm jupyter kernel does not start, it can have many causes.
Before we turn on the debug mode, check following things:SSH-Agent is active/running and my key is loadedIf you can log in in to the loginnode of the HPC system passwordless (Shell) that should workCorrect Proxyjump (Loadbalancer), LoginnodeGet help$slurmkernel--help

usage:Tooltomanage(create,list,modifyanddelete)andstartingjupyterslurmkernelsusingsrun[-h][--version]{create,list,edit,delete,template}...

positionalarguments:{create,list,edit,delete,template}createcreateanewslurmkernellistlistavailableslurmkernelediteditanexistingslurmkerneldeletedeleteanexistingslurmkerneltemplatemanagescripttemplates(list,use,add,edit)optionalarguments:-h,--helpshowthishelpmessageandexit--versionshowprogram'sversionnumberandexit