Package: jh-hpc-interface
JupyterHub + High-Performance ComputingHigh performance Jupyter NotebooksThe aim of this project is to connect JupyterHub to a high-performance computer (HPC). By automatically offloading the computations in a Jupyter notebook to the HPC system, even complex calculations are possible. While JupyterHub is deployed on a regular server, the notebooks themselves are spawned and run on the remote HPC system using a workload manager, such as Slurm.MotivationThe technical core of this project is the transparent integration of digital worksheets (Jupyter notebooks), in which learning content and programs can be displayed, edited and executed on the students' own laptops, with current cloud and high-performance computing (HPC) technologies. This provides the conditions for innovative, digital teaching that encourages independent and interactive development of, for example, data science applications, without imposing the complexity of using a high-performance computer system on the students. Instead, particularly computationally and data-intensive calculations are automatically offloaded to a high-performance computer, enabling even sophisticated analyses to be performed that would otherwise not be feasible on students' laptops.Features and use casesStarting a jupyter notebook server on a remote HPC system in a pre-defined singularity containerQuick config setup when using the Slurm configuration wizardAutomatically create a singularity overlay so that user changes are persistentGreat for managing courses with external participantsPossibility to include files in the notebook directory using WebDAVSuitable for HPC users who have their own JupyterHub instance running and want to use HPC resourcesTable of ContentsJupyterHub + High-Performance ComputingTable of ContentsInstallation of JupyterHub ServerJupyterHub and BatchSpawnerSSH tunnel userNode mappingInstallation on HPC SystemRequirementsInstall using pipSingularity ContainerBuild Singularity ContainerComputeGPU (Tensorflow)The configuration fileSlurm configuration wizardExamplesDebug modeShibboleth IntegrationNBGrader IntegrationInstallationChanging the Student ID to the JupyterHub logged in user nameCreate nbgrader_config.pySecurity PrecautionsSingularity Host FilesystemsJupyterHub API (HTTPS)HTTPStunnelbot userTroubleshootingInstallation of JupyterHub ServerThis section describes the required installations and configurations on the JupyterHub server.JupyterHub and BatchSpawnerThe first thing you should do is install JupyterHub and BatchSpawner. For this purpose we provide an Ansible playbook which can be found in/jupyterhub-deployment/. See the README for details. Alternatively, you can follow the official installation instructions.If you decide to do the installations yourself, please proceed as follows:installJupyterHubinstallBatchSpawnerinstallWrapSpawner(make sure to install it in the right environment:/opt/jupyterhub/bin/pip3 install git+https://github.com/jupyterhub/wrapspawner)copy the JupyterHub configuration file/jupyterhub-deployment/config_files/jupyterhub_config.pyto/opt/jupyterhub/etc/jupyterhub/(you will most likely have to edit this file afterwards to make it fit your needs)restart the JupyterHub serviceSSH tunnel userA user calledtunnelbotis needed on the JupyterHub server. This user is responsible for starting an SSH tunnel between the compute node and the JupyterHub server. An SSH key pair for the above mentioned purpose must be generated. See/examples/jupyterhub_config.pyfor more information.Node mappingJupyterHub extracts the execution host name of the HPC system (e.g.node01-002). When a notebook server is started, an SSH tunnel is established using the notebook port.In order for JupyterHub to be able to resolve the compute nodes host name, the/etc/hostsfile must be edited. An example entry might look like the following:127.0.0.1 node01-001
127.0.0.1 node01-002
127.0.0.1 node01-003
...
127.0.0.1 node12-048The actual node names depend on your HPC system of course.Installation on HPC SystemThis section describes the required installations and configurations of the HPC system to enable the interaction with the JuypterHub server.RequirementsYou need a user who is allowed to allocate resources on the HPC systemWith a SSH key pair. The public part must be deposited on the JupyterHub serer (tunnelbotuser)The public key part of thetunnelbot-user created on the JupyterHub (->~/.ssh/authorized_keys)Singularity (> v.3.7.0)mkfs/e2fsprogs with following option:https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=217c0bdf17899c0f79b73f76feeadd6d55863180Install using pipYou can download and install the required files with pip.You may want to build a small Python environment, or install the tools with--user.python3-mpipinstall--userjh-hpc-interfaceSingularity ContainerSingularity recipe examples are in the directory singularity/.If you do not want to use singularity, then change the value ofuse_singularityin jh_config.ini to false.Build Singularity ContainerTo build the container with the recipe files in singularity/ you have to clone this repository.The following commands replace USER_ID in the recipes to the output ofid -u, create a new hidden file and build the singularity container with the new created file.ComputeUSER_ID=$(id-u)&&sed"s/USER_ID/$USER_ID/"<singularity/Singularity>singularity/.recipefile_compute&&singularitybuild--remotesingularity/compute_jupyter.sifsingularity/.recipefile_computeGPU (Tensorflow)USER_ID=$(id-u)&&sed"s/USER_ID/$USER_ID/"<singularity/Singularity_Tensorflow>singularity/.recipefile_gpu&&singularitybuild--remotesingularity/gpu_jupyter.sifsingularity/.recipefile_gpusingularity build help section:-r, --remotebuild image remotely (does not require root)Please refer to the official docs on how to use the remote build feature:https://sylabs.io/docs/The configuration fileIn the directorybin/is a script, which is deposited after the installation on the system.With the following call you can display the location of the configuration file:$jh_wrappergetconfigTo learn more about the configuration file, seedocs/jh_config.ini.mdSlurm configuration wizardWith the configuration wizard you can prepare your HPC environment.The script interactively goes through the configuration file and creates a temporary file which can be copied with a simplecp.To start the wizard type the following:$jh_slurm_wizardExamplesYou will find examples for the configuration filesjh_config.iniandjupyterhub_config.pyin the directoryexamples/.Debug modeBy default the logs contain only information such as warnings or error messages.
It is also possible to switch on the debug mode, which writes extended information into the log files.Just setlog_levelin the configuration file to 'DEBUG'.Shibboleth IntegrationShibboleth authentication was set up for a JupyterHub server in a test environment. See./shibboleth/for an example configuration.NBGrader IntegrationInstallationInstallation instructions:https://nbgrader.readthedocs.io/en/latest/configuration/jupyterhub_config.htmlTo create an exchange directory for every user, just create an empty directory in$scratch_dirand mount it into the container with$singularity_bind_extra.Changing the Student ID to the JupyterHub logged in user nameSince the containers run as userjovyan, the value from the$JUPYTERHUB_USERvariable is automatically used.See here for more information:https://jupyter.readthedocs.io/en/latest/community/content-community.html#what-is-a-jovyanCreate nbgrader_config.pySee here:https://nbgrader.readthedocs.io/en/stable/configuration/nbgrader_config.html#use-case-3-nbgrader-and-jupyterhubTo makenbgrader_config.pyavailable in the container, just append the file in$singularity_bind_extra.Security PrecautionsSingularity Host FilesystemsIn case you are using Singularity, the host file system may be automatically mounted into the container when you start a Singularity Container.A possible cause is the optionmount hostfsinsingularity.confSee here:https://sylabs.io/guides/3.5/admin-guide/configfiles.html#singularity-confJupyterHub API (HTTPS)HTTPSSee here for more information:https://jupyterhub.readthedocs.io/en/stable/reference/websecurity.htmltunnelbot userYou can increase the security by deactivating shell access for this user.Just type:usermod-s/bin/falsetunnelbotTroubleshootingWhen problems occur with the JupyterHub, some information can be obtained from the logs when debug mode is enabled:https://github.com/jupyterhub/jupyterhub/wiki/Debug-Jupyterhub