Package: simple-gpu-scheduler
simple_gpu_schedulerA simple scheduler to run your commands on individual GPUs.  Following theKISS principle, this script
simply accepts commands viastdinand executes them on a specific GPU by
setting theCUDA_VISIBLE_DEVICESvariable.The commands read are executed using the login shell, thus redirections>pipes|and all other kinds of bash magic can be used.InstallationThe package can simply be installed frompypi$pipinstallsimple_gpu_schedulerExampleTo show how this generally works, we will create jobs that simply outputs
a job id and the value ofCUDA_VISIBLE_DEVICES:foriin{0..10};doecho"echo job_id=$idevice=\$CUDA_VISIBLE_DEVICES && sleep 3";done|simple_gpu_scheduler--gpus0,1,2which results in the following output:Processing `command echo job_id=0 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing `command echo job_id=1 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1
Processing `command echo job_id=2 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=0 device=2
job_id=1 device=1
job_id=2 device=0
--- 3 seconds no output ---
Processing command `echo job_id=3 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing command `echo job_id=4 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1
Processing command `echo job_id=5 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=3 device=2
job_id=4 device=1
job_id=5 device=0
--- 3 seconds no output ---
Processing command `echo job_id=6 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing command `echo job_id=7 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 1
Processing command `echo job_id=8 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=6 device=2
job_id=7 device=1
job_id=8 device=0
--- 3 seconds no output ---
Processing command `echo job_id=9 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 2
Processing command `echo job_id=10 device=$CUDA_VISIBLE_DEVICES && sleep 3` on gpu 0
job_id=9 device=2
job_id=10 device=0This is equivalent to creating a filecommands.txtwith the following content:echojob_id=0device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=1device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=2device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=3device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=4device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=5device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=6device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=7device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=8device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=9device=$CUDA_VISIBLE_DEVICES&&sleep3echojob_id=10device=$CUDA_VISIBLE_DEVICES&&sleep3and runningsimple_gpu_scheduler--gpus0,1,2<commands.txtSimple scheduler for jobsCombined with some basic command line tools, one can set up a very basic
scheduler which waits for new jobs to be "submitted" and executes them in order
of submission.Setup and start scheduler in background or in a separate permanent session
(using for exampletmux):touchgpu.queue
tail-f-n0gpu.queue|simple_gpu_scheduler--gpus0,1,2the commandtail -f -n 0follows the end of the gpu.queue file. Thus if there
was anything written intogpu.queueprior to the execution of the command it
will not be passed tosimple_gpu_scheduler.Then submitting commands boils down to appending text to thegpu.queuefile:echo"my_command_with | and stuff > logfile">>gpu.queueHyperparameter searchIn order to allow user friendly utilization of the scheduler in the common
scenario of hyperparameter search, a convenience scriptsimple_hypersearchis
included in the package.simple_hypersearch-husage:simple_hypersearch[-h][--sampling-mode{shuffled_grid,grid}][--n-samplesN_SAMPLES][--seedSEED][-pNAME[VALUES...]]command_pattern

Conveniencetooltogeneratehyperparametersearchcommandsfromacommandpatternandparameterranges.

positionalarguments:command_patternCommandpatternwhereplaceholderswith{parameter_name}shouldbereplaced.

optionalarguments:-h,--helpshowthishelpmessageandexit--sampling-mode{shuffled_grid,grid}Determinehowtosamplecommands.Eitherinthegridorder[grid]orinashuffledorder[shuffled_grid,default].--n-samplesN_SAMPLESNumberofsamplestodraw.Ifnotprovideduseallpossiblecombinations.--seedSEEDRandomseedtoensurereproducabilitywhenusingrandomizedorderofthegrid.-pNAME[VALUES...],--parameterNAME[VALUES...]Nameofparameterfollowedbyvaluesthatshouldbeconsideredforhyperparametersearch.Example:`-plr0.010.0010.0001`Usageexample:simple_hypersearch"my_program --param1 {param1} --param2 {param2}"-pparam101-pparam223willgeneratetheoutput:my_program--param10--param22my_program--param10--param23my_program--param11--param22my_program--param11--param23This allows to easily perform hyperparameter searches over a grid of values or
uniform samples of the grid (dependent on the setting ofsampling-mode).
The output can directly be piped intosimple_gpu_scheduleror appended to the
"queue file" (seeSimple scheduler for jobs).Here some more concrete examples:Grid of all possible parameter configurations in random order:simple_hypersearch"python3 train_dnn.py --lr {lr} --batch_size {bs}"-plr0.0010.00050.0001-pbs3264128|simple_gpu_scheduler--gpus0,1,25 uniformly sampled parameter configurations:simple_hypersearch"python3 train_dnn.py --lr {lr} --batch_size {bs}"--n-samples5-plr0.0010.00050.0001-pbs3264128|simple_gpu_scheduler--gpus0,1,2TODOMulti line jobs (evtl. we would then need a submission script after all)Stop, but let commands finish when receiving a defined signalTests would be nice, until now the project is stillvery smallbut if it
grows tests should be added