Package: hyperparameters
HyperparametersAre you tired of repeating hyperparameters in code,argparsedefinitions, and hyperparameter tunning libraries?Hyperparameterslets you define your hyperparameters once and use everywhere! Moreover, you get type linting and spelling checks for free!QuickstartInstallHyperparameters:pipinstallhyperparametersYou need to install theray.tunepackage separately if you want to use theray.tunehypertunning integration:pipinstall-U"ray[tune]"Define your parameters once using theHyperparamsclass:fromargparseimportArgumentParserfromtypingimportOptionalfromhyperparametersimportHP,HyperparamsclassMyHyperparams(Hyperparams):epochs:int=HP("Number of epochs to train for",default=5,)lr:float=HP("Learning rate",default=1e-3,)tokenizer:str=HP("HF tokenizer to use",default="BPE",choices=["BPE","WordPiece"],)train_data_path:str=HP("Path to the training dataset",)use_dropout:bool=HP("Whether the dropout layers should be activated",default=True,)pretrained_weights:Optional[str]=HP("Path to the pretrained model weights, if any",default=None,)parser=ArgumentParser()MyHyperparams.add_arguments(parser)params=MyHyperparams.from_arguments(parser.parse_args())# access parameters as typed attributesparams.train_data_path# convert to a dictionaryprint(params.dict())Let's see the registered arguments by running the code above with--help:usage: example.py [-h] [--epochs EPOCHS] [--lr LR] [--tokenizer {BPE,WordPiece}] --train-data-path TRAIN_DATA_PATH [--use-dropout | --no-use-dropout]
                  [--pretrained-weights PRETRAINED_WEIGHTS]

options:
  -h, --help            show this help message and exit
  --epochs EPOCHS       Number of epochs to train for
  --lr LR               Learning rate
  --tokenizer {BPE,WordPiece}
                        HF tokenizer to use
  --train-data-path TRAIN_DATA_PATH
                        Path to the training dataset
  --use-dropout         Whether the dropout layers should be activated
  --no-use-dropout      Disable: Whether the dropout layers should be activated
  --pretrained-weights PRETRAINED_WEIGHTS
                        Path to the pretrained model weights, if anyAs can be seen from the above,Hyperparameterstakes care of low level details for you:The--train-data-pathparameter is required because we didn't provide a default value for it. All other parameters are optional.Even though the default value for--pretrained-weightsisNone, this parameter is optional. However, we had to use theOptional[str]type hint.You can provide the--use-dropoutor the--no-use-dropoutflag, but not both at the same time. Neither flag is required, as theuse_dropoutparameter has a default value.The--tokenizerparameter can only beBPEorWordPiece. Providing any other value results in an error.The data types of the parameters are parsed from strings and validated according to the type hints in theMyHyperparamsclass.The default values are used whenever an argument is ommitted. Let's check that by running with--train-data-path mydata/: the script prints:{'epochs': 5, 'lr': 0.001, 'tokenizer': 'BPE', 'train_data_path': 'mydata/', 'use_dropout': True, 'pretrained_weights': None}.Default values and required parametersIf a default value is omitted completely, the parameter will be required.Otherwise, if any default value is specified, evenNone, the parameter will be optional.You must use theOptionaltype hint if the default value isNone.You can't haveboolparameters withNoneas default. Flags that can beNonejust don't make any sense.You can't have choice parameters withNoneas default. If you need this, just add a "null" value to the list of choices.HypertunningDifferent hypertunning libraries provide different APIs for defining search spaces.Hyperparameterscan be easily extended to support any hypertunning library. You can do it yourself following the steps discussed below - it's easy! Theray.tunelibrary is supported out of the box.When defining hypertunable parameters withHyperparameters, make sure to inherit from both theHyperparamsclass and the mixin class specific to your hypertunning library. Here is an example forray.tune:fromhyperparametersimportHP,Hyperparamsfromhyperparameters.ray_tune_hyperparamsimportRayTuneHyperparamsMixinclassMyHyperparams(Hyperparams,RayTuneHyperparamsMixin):...Next, you can specify the search spaces:use the hypertunning library API to describe the search space and provide this object to theHP(..., search_space=)parameter;if you already specify theHP(..., choices=[])parameter you can treat the possible choices as a search space by setting theHP(..., tunable=True)parameter.forboolfields just set theHP(..., tunable=True)parameter, which is equivallent toHP(..., tunable=True, choices=[False, True]).Note that when using thesearch_spaceparameter, you don't need to use thetunableparameter.Finally, you are ready to use the mixin-specific methods to extract information about your search spaces. Forray.tunethese are:.ray_tune_param_space()method that returns a dictionary describing the search space..ray_tune_best_values()method that returns a dictionary of the best values to start hypertunning from.Here is a complete example of defining parameters hypertunable withray.tune:fromrayimporttunefromhyperparametersimportHP,Hyperparamsfromhyperparameters.ray_tune_hyperparamsimportRayTuneHyperparamsMixinclassMyHyperparams(Hyperparams,RayTuneHyperparamsMixin):lr:float=HP("Learning rate",default=1e-3,search_space=tune.loguniform(1e-5,1e-2),)layers_num:int=HP("Number of model layers",default=8,tunable=True,choices=[4,8,16,24],)use_dropout:bool=HP("Whether the dropout layers should be activated",default=True,tunable=True,)params=MyHyperparams()# Search space config that ray.tune understandsparams.ray_tune_param_space()# Best values to start the hypertunning fromparams.ray_tune_best_values()Supporting other hypertunning librariesInHyperparameters, the logic specific to hypertunning libraries is implemented with mixin classes. This means that you can add many mixins to your parameters and support several hypertunning libraries at once.The best place to start is to review the implementation of theRayTuneHyperparamsMixinclass inhyperparameters/ray_tune_hyperparams.py.The class implementing a new hypertunning library must:Inherit fromhyperparameters.hyperparams.HyperparamsProtocol.Provide class methods for converting the params data stored inhyperparameters.hyperparams.HyperparamInfostructure into a format that the hypertunning library understands.Use thecls._tunable_params()to get the info about the parameters.Wrapinfo.choicesin a proper type for the hypertunning library.Provide a method for returning the best parameter values to start the hypertunning from.