Package: shopty
shoptySimple Hyperparameter OPTimization in pYthonInstall from source (recommended)gitclonehttps://github.com/colligant/shopty# optional: pip install flitcdshopty&&flitinstallInstall via pippipinstallshoptyhyperband on a slurm clustershoptyhyperband--config_filemy_config.yaml--supervisorslurmrun 20 random hyperparameter configs each for 100 iterationsshoptyrandom--config_filemy_config.yaml--supervisorslurm--max_iter100--n_experiments20A non-cli example ishere.What is the purpose of this tool?Lots of other hyperparameter tuning libraries (at least the ones I've found, anyways)
require modifying a bunch of source code and make assumptions about your deployment environment.shoptyis a simple library to tune hyperparameters either on your personal computer or a slurm-managed
cluster that requires minimal code changes and uses a simple config file to do hyperparameter sweeps.DesignTheSupervisorclasses inshoptyspawn (if on CPU) or submit (if on slurm) different experiments, each
with their own set of hyperparameters. Submissions are done within python by creating a bash or sbatch file and
submitting it viasubprocess.call.Each experiment writes a "results.txt" after its finished to a unique directory. TheSupervisorclass detects when each
experiment is done and reads the "results.txt" file for the outcome of the experiment that wrote it.Source code modificationsSee a simple examplehere. A neural network example ishere.Supervisors communicate with experiments via environment variables. Your custom training code must know how to deal with
some shopty-specific use cases. In particular, it must a) run the code formax_iteriterations, b) reload the training
state from a checkpoint file, and c) write the result post-training to a results file. Themax_itervariable is an
experiment-specific environment variable, as is the checkpoint file's name and the results file's name.I've already written the code for this forpytorch lightning(PTL).
I highly recommend using PTL, as it does a lot of useful things for you under the hood.How to define hyperparameters and slurm directivesWe use a .yaml file to define hyperparameters for training models as well as other commands you want to run to set up
the training environment.
The .yaml file must have the following structure:project_name:'your_project_name'run_command:"python3my_cool_script.py"project_dir:"~/deep_thought/"monitor:"max"poll_interval:10hparams:learning_rate:begin:-10end:-1random:Truelog:Trueyour_custom_hparam:begin:1end:5step:1another_custom_hparam:begin:1end:5random:Truestatics:a_static_hparam:1e-10slurm_directives:-"--partition=gpu"-"--gres=gpu:1"environment_commands:-"condaactivatemy_env"run_commandTherun_commandis how shopty runs your program. Generated hyperparameters are passed in to therun_commandvia the
command line in no particular order. For example, if you want to tune the learning rate of the model
inmy_cool_script.py,my_cool_script.pymust accept a--learning_rate <learning_rate>argument.Notice how thehparamsheader has two levels of indentation: one for the name of hyperparameter, and the next for the
beginning and end of the range over which to sample from. There are three required elements for each hparam:begin, end, and <random or step>. The hyperparameter can either be sampled randomly between the interval[begin, end)or iterated over frombegintoendwith stepstep. Binary variables can be added to the project withhparams:binary_indicator:begin:0end:2step:1Static variables can be added viastatics:my_static_var:10# or, if you need to specify a type:my_other_static_var:val:100.0type:'float'Slurm directivesSlurm scripts have headers that specify what resources a program will use (#SBATCHstatements). Add these
to each experiment by editing theslurm_directivessection of the yaml file. They will be added as#SBATCHstatements
in each slurm submission script.Environment commandsThese are arbitrary commands that you want to run before therun_commandis called in the generated script.