Package: echo-opt
EarthComputingHyperparameterOptimization (ECHO): A distributed hyperparameter optimization package build with OptunaInstallTo install a stable version of ECHO from PyPI, use the following command:pipinstallecho-optInstall the latest version of ECHO directly from github with the following command:pipinstallgit+https://github.com/NCAR/echo-opt.gitSeveral commands will be placed onto the PATH:echo-opt, echo-report, echo-runUsageLaunch a new optimization study:echo-opthyperparameters.ymlmodel_config.ymlProduce a report about the results saved in the study:echo-reporthyperparameters.yml[-pplot_config.yml][-mmodel_config.yml]Run one trial:echo-runhyperparameters.ymlmodel_config.ymlDependenciesThere are three files that must be supplied to use the optimize script:A custom objective class that trains your model and returns the metric to be optimized.A configuration file specifying the hyperparameter optimization settings.A model configuration file that contains the information needed to train your model (see examples in the holodec and gecko projects).Custom objective classThe customObjectiveclass (objective.py) must inherit aBaseObjectiveclass (which lives in base_objective.py), and must contain a method namedtrainthat returns the value of the optimization metric (in a dictionary, see below). There are example objective scripts for both torch and Keras in the examples directory. Your custom Objective class will inherit all of the methods and attributes from the BaseObjective. The Objective's train does not depend on the machine learning library used! For example, a simple template has the following structure:fromecho.src.base_objectiveimport*fromecho.src.prunersimportKerasPruningCallbackclassObjective(BaseObjective):def__init__(self,config,metric="val_loss"):# Initialize the base classBaseObjective.__init__(self,config,metric)deftrain(self,trial,conf):# Make any custom edits to the model conf before using it to train a model.conf=custom_updates(trial,conf)# ... (load data sets, build model, etc)callbacks=[KerasPruningCallback(trial,self.metric,interval=1)]result=Model.fit(...,callbacks=callbacks)results_dictionary={"val_loss":result["val_loss"],"loss":result["loss"],#..."val_accuracy":result["val_accuracy"]}returnresults_dictionaryYou can have as many inputs to your custom Objective as needed, as long as those that are required to initialize the base class are included. The Objective class will call the train method from the inherited thundercallmethod, and will finish up by calling the inherited save method that writes the metric(s) details to disk. Note that, due to the inheritance of the one class on the other, you do not have to supply these two methods, as they are in pre-coded in the base class. You can customize them at your leisure using overriding methods in your custom Objective. Check out the scripts base_objective.py and run.py to see how things are structured and called.As noted, the metric used to toggle the model's training performance must be in the results dictionary. Other metrics that the user may want to track will be saved to disk if they are included in the results dictionary (the keys of the dictionary are used to name the columns in a pandas dataframe). See the example above where several metrics are being returned.Note that the first line in the train method states that any custom changes to the model configuration (conf) must be done here. If custom changes are required, the user may supply a method namedcustom_updatesin addition to the Objective class (you may save both in the same script, or import the method from somewhere else in your custom Objective script). See also the sectionCustom model configuration updatesbelow for an example.Finally, if using Keras, you need to include the (customized) KerasPruningCallback that will allow optuna to terminate unpromising trials. We do something similar when using torch -- see the examples directory.Hyperparameter optimizer configurationThere are several fields: log, slurm, pbs, optuna, and variable subfields within each field. The log field allows us to save a file for printing messages and warnings that are placed in areas throughout the package. The slurm/pbs fields allows the user to specify how many GPU nodes should be used, and supports any slurm setting. The optuna field allows the user to configure the optimization procedure, including specifying which parameters will be used, as well as the performance metric. For example, consider the configuration settings:log:Truesave_path:"/glade/work/schreck/repos/echo-opt/echo/examples/torch/fmc"pbs:jobs:10kernel:"ncar_pylib/glade/work/schreck/py37"bash:["moduleloadncarenv/1.3gnu/8.3.0openmpi/3.1.4python/3.7.5cuda/10.1"]batch:l:["select=1:ncpus=8:ngpus=1:mem=128GB","walltime=12:00:00"]A:"NAML0001"q:"casper"N:"echo_trial"o:"echo_trial.out"e:"echo_trial.err"slurm:jobs:15kernel:"ncar_pylib/glade/work/schreck/py37"bash:["moduleloadncarenv/1.3gnu/8.3.0openmpi/3.1.4python/3.7.5cuda/10.1"]batch:account:"NAML0001"gres:"gpu:v100:1"mem:"128G"n:8t:"12:00:00"J:"echo_trial"o:"echo_trial.out"e:"echo_trial.err"optuna:storage:"mlp.db"study_name:"mlp"storage_type:"sqlite"objective:"examples/torch/objective.py"metric:"val_loss"direction:"minimize"n_trials:500gpu:Truesampler:type:"TPESampler"n_startup_trials:30parameters:num_dense:type:"int"settings:name:"num_dense"low:0high:10dropout:type:"float"settings:name:"dr"low:0.0high:0.5optimizer:learning_rate:type:"loguniform"settings:name:"lr"low:0.0000001high:0.01model:activation:type:"categorical"settings:name:"activation"choices:["relu","linear","leaky","elu","prelu"]The save_path field sets the location where all generated data will be saved.save_path: Directory path where data will be saved.The log field allows you to save the logging details to file to save_path; they will always be printed to stdout. If this field is removed, logging details will only be printed to stdout.log: boolean to save log.txt in save_path.The subfields within "pbs" and slurm" should mostly be familiar to you. In this example there would be 10 jobs submitted to pbs queue and 15 jobs to the slurm queue. Most HPCs just use one or the other, so make sure to only speficy what your system supports. The kernel field is optional and can be any call(s) to activate a conda/python/ncar_pylib/etc environment. Additional snippets that you might need in your launch script can be added to the list in the "bash" field. For example, as in the example above, loading modules before training a model is required. Note that the bash options will be run in order, and before the kernel field. Remove or leave the kernel field blank if you do not need it.The subfields within the "optuna" field have the following functionality:storage: sqlite or mysql destination.study_name: The name of the study.storage_type: Choose "sqlite" or "maria" if a MariaDB is setup.If "sqlite", the storage field will automatically be appended to the save_path field (e.g. sql:///{save_path}/mlp.db)If "maria", specify the full path including username:password in the storage field (for example, mysql://user:pw@someserver.ucar.edu/optuna).objective: The path to the user-supplied objective classmetric: The metric to be used to determine the model performance.direction: Indicates which direction the metric must go to represent improvement (pick from maximimize or minimize)n_trials: The number of trials in the study.gpu: Set to true to obtain the GPUs and their IDssamplertype: Choose how optuna will do parameter estimation. The default choice both here and in optuna is theTree-structured Parzen Estimator Approach,e.g. TPESampler. See the optuna documentation for the different options. For some samplers (e.g. GridSearch) additional fields may be included (e.g. search_space).parameterstype: Option to select an optuna trial setting. See theoptuna Trial documentationfor what is available. Currently, this package supports the available options from optuna: "categorical", "discrete_uniform", "float", "int", "loguniform", and "uniform".settings: This dictionary field allows you to specify any settings that accompany the optuna trial type. In the example above, the named num_dense parameter is stated to be an integer with values ranging from 0 to 10. To see all the available options, consolt theoptuna Trial documentationLastly, the "log" field allows you to save the logging details to file; they will always be printed to stdout. If this field is removed, logging details will only be printed to stdout.Model configurationThe model configuration file can be what you had been using up to this point to train your model, in other words no changes are necessary. This package will take the suggested hyperparameters from an optuna trial and make changes to the model configuration on the fly. This can either be done automatically with this package, or the user may supply an additional method for making custom changes. For example, consider the (truncated) configuration for training a model to predict properties with a dataset containing images:model:image_channels:1hidden_dims:[3,94,141,471,425,1122]z_dim:1277dense_hidden_dims:[1000]dense_dropouts:[0.0]tasks:["x","y","z","d","binary"]activation:"relu"optimizer:type:"lookahead-diffgrad"learning_rate:0.000631weight_decay:0.0trainer:start_epoch:0epochs:1clip:1.0alpha:1.0beta:0.1path_save:"test"The model configuration will be automatically updated if and only if the name of the parameter specified in the hyperparameter configuration, optuna.parameters can be used as a nested lookup key in the model configuration file. For example, observe in the hyperparameter configuration file above that the named parameteroptimizer:learning_ratecontains a colon, that is downstream used to split the named parameter into multiple keys that allow us to, starting from the top of the nested tree in the model configuration file, work our way down until the relevant field is located and the trial-suggested value is substituted in. In this example, the split keys are ["optimizer", "learning_rate"].This scheme will work in general as long as the named parameter in optuna.parameters uses : as the separator, and once split, the resulting list can be used to locate the relevant field in the model configuration.Note that optuna has a limited range of trial parameters types; all but one them being numerical in one form or another. If you wanted to optimize the activation layer(s) in your neural network as in the example above, you could go about that by utilizing the "categorical" trial suggestor. For example, the following list of activation layer names could be specified: ["relu", "linear", "leaky", "elu", "prelu"].Custom model configuration updatesYou may additionally supply rules for updating the model configuration file, by including a method namedcustom_updates, which will make the desired changes to the configuration file with optuna trail parameter guesses.In the example configurations described above, the hyperparameter configuration contained an optuna.parameters field "num_dense," but this field is not present in the model configuration. There is however a "dense_hiddden_dims" field in the model configuration that contains a list of the layer sizes in the model, where the number of layers is the length of the list. In our example just one layer specified but we want to vary that number.To use the "num_dense" hyperparameter from the hyperparameter configuration file, we can create the following method:defcustom_updates(trial,conf):# Get list of hyperparameters from the confighyperparameters=conf["optuna"]["parameters"]# Now update some via custom rulesnum_dense=trial.suggest_discrete_uniform(**hyperparameters["num_dense"])# Update the config based on optuna's suggestionconf["model"]["dense_hidden_dims"]=[1000forkinrange(num_dense)]returnconfThe method should be called first thing in the custom Objective.train method (see the example Objective above). You may have noticed that the configuration (named conf) contains both hyperparameter and model fields. This package will copy the hyperparameter optuna field to the model configuration for convenience, so that we can reduce the total number of class and method dependencies (which helps me keep the code generalized). This occurs in the run.py script.Custom plot settings for echo-reportThe script report.py will load the current study, identify the best trial in the study, and will compute the relative importantance of each parameter using both fanova and MDI (seeherefor details).Additionally, the script will create two figures, an optimization history plot and an intermediate values plot. If your metric returns two values to be optimized, only the pareto front plot will be generated. See thedocumentationfor details on the plots.Note that ECHO only supports thematplotlibgenerated plots from Optuna, for now. Optuna's default is to use plot.ly, however not all LTS Jupyter-lab environments support that backend.The user may customize the plots to a degree, by additionally supplying a plot configuration yaml file (named plot_config.yml above, and called as an optional argument using the parser -p or --plot). Currently, the user may only adjust the rcParam backend variables (seeherefor a comprehensive list) plus a limited set of other variables (see below),optimization_history:save_path:'/glade/work/schreck/repos/holodec-ml/scripts/schreck/decoder/results/opt_multi_particle'set_xlim:[0,100]set_ylim:[3e4,1e6]set_xscale:"log"set_yscale:"log"rcparams:'backend':'ps''lines.markersize':4'axes.labelsize':10'legend.fontsize':10'xtick.labelsize':10'ytick.labelsize':10'xtick.top':True'xtick.bottom':True'ytick.right':True'ytick.left':True'xtick.direction':'in''ytick.direction':'in''font.serif':'Helvetica''figure.dpi':600'figure.autolayout':True'legend.numpoints':1'legend.handlelength':1.0'legend.columnspacing':1.0For the other supported plots, simply add or change "optimization_history" to "intermediate_values", or if optimizing more than one metric, "pareto_front".