Package: hpc05
hpc05🖥ipyparallel.Clientpackage for a PBS or SLURM cluster with a headnode.Script that connects to PBS or SLURM cluster with headnode over ssh. Sinceipyparalleldoesn't cull enginges when inactive and people are lazy (because they forget toqdeltheir jobs), it automatically kills theipenginesafter the set timeout (default=15 min). Note that this package doesn't only work for thehpc05cluster on the TU Delft but also other clusters.InstallationFirst install this package onbothyour machine and the cluster.condaconfig--addchannelsconda-forge
condainstallhpc05or usingpippip install hpc05Make sure you can connect oversshpasswordless by copying your ssh key:ssh-copy-idhpc05Setup profileYou need a parallel profile on your cluster, which can be created by the following command on your local machine:importhpc05# for PBS usehpc05.create_remote_pbs_profile(profile='pbs',hostname='hpc05')# on the remote machine# orhpc05.create_local_pbs_profile(profile='pbs')# on the cluster# for SLURM usehpc05.create_remote_slurm_profile(profile='slurm',hostname='hpc05')# on the remote machine# orhpc05.create_local_slurm_profile(profile='slurm')# on the clusterStartipclusterand connect (viassh)To startandconnect to anipclusterjust do (and read the error messages if any, for instructions):client,dview,lview=hpc05.start_remote_and_connect(n=100,profile='pbs',hostname='hpc05',folder='~/your_folder_on_the_cluster/')This is equivent to the following three commmands:# 0. Killing and removing files of an old ipcluster (this is optional with#    the `start_remote_and_connect` function, use the `kill_old_ipcluster` argument)hpc05.kill_remote_ipcluster(hostname='hpc05')# 1. starting an `ipcluster`, similar to running#    `ipcluster start --n=100 --profile=pbs` on the cluster headnode.hpc05.start_remote_ipcluster(n=100,profile='pbs',hostname='hpc05')# 2. Connecting to the started ipcluster and adding a folder to the cluster's `PATH`client,dview,lview=hpc05.connect_ipcluster(n=200,profile='pbs',hostname='hpc05',folder='~/your_folder_on_the_cluster/')Startipclusterand connect (on cluster headnode)To startandconnect to anipclusterjust do (and read the error messages if any, for instructions):client,dview,lview=hpc05.start_and_connect(n=100,profile='pbs',folder='~/your_folder_on_the_cluster/')This is equivent to the following three commmands:# 0. Killing and removing files of an old ipcluster (this is optional with#    the `start_remote_and_connect` function, use the `kill_old_ipcluster` argument)hpc05.kill_ipcluster()# 1. starting an `ipcluster`, similar to `ipcluster start --n=200 --profile=pbs`hpc05.start_ipcluster(n=200,profile='pbs')# 2. Connecting to the started ipcluster and adding a folder to the cluster's `PATH`client,dview,lview=hpc05.connect_ipcluster(n=200,profile='pbs',folder='~/your_folder_on_the_cluster/')Monitor resourcesThis package will monitor your resources if you start it withhpc05_monitor.start(client), see the following example use:importtimeimporthpc05_monitorhpc05_monitor.start(client,interval=5)# update hpc05_monitor.MAX_USAGE every 'interval' seconds.whilenothpc05_monitor.LATEST_DATA:time.sleep(1)hpc05_monitor.print_usage()# uses hpc05_monitor.LATEST_DATA by defaulthpc05_monitor.print_max_usage()# uses hpc05_monitor.MAX_USAGEWith output:id hostname             date                             CPU% MEM%
 15 node29.q1cluster     2018-09-10T14:25:05.350499       190%   3%
 19 node29.q1cluster     2018-09-10T14:25:04.860693       200%   3%
 26 node29.q1cluster     2018-09-10T14:25:05.324466       200%   3%
 28 node29.q1cluster     2018-09-10T14:25:05.148623       190%   2%
 29 node29.q1cluster     2018-09-10T14:25:04.737664       190%   3%
 ...DevelopmentWe usepre-commitfor linting of the code, sopip install pre_commitand runpre-commit installin the repository.