Package: allennlp-optuna
allennlp-optuna: Hyperparameter Optimization Library for AllenNLPallennlp-optunaisAllenNLPplugin for
hyperparameter optimization usingOptuna.Supported environmentsMachine \ DeviceSingle GPUMulti GPUsSingle Node:white_check_mark:PartialMulti Nodes:white_check_mark:PartialAllenNLP provides a way of distributed training (https://medium.com/ai2-blog/c4d7c17eb6d6).
Unfortunately,allennlp-optunadoesn't fully support this feature.
With multiple GPUs, you can run hyperparameter optimization.
But you cannot enable a pruning feature.
(For more detail, please seehimkt/allennlp-optuna#20andoptuna/optuna#1990)Alternatively,allennlp-optunasupports distributed optimization with multiple machines.
Please read thetutorialabout
distributed optimization inallennlp-optuna.
You can also learn about a mechanism of Optuna in thepaperordocumentation.DocumentationYou can read the documentation onreadthedocs.1. Installationpipinstallallennlp_optuna# Create .allennlp_plugins at the top of your repository or $HOME/.allennlp/plugins# For more information, please see https://github.com/allenai/allennlp#pluginsecho'allennlp_optuna'>>.allennlp_plugins2. Optimization2.1. AllenNLP configModel configuration written in Jsonnet.You have to replace values of hyperparameters with jsonnet functionstd.extVar.
Remember casting external variables to desired types bystd.parseInt,std.parseJson.locallr=0.1;// before↓↓↓locallr=std.parseJson(std.extVar('lr'));// afterFor more information, please refer toAllenNLP Guide.2.2. Define hyperparameter search speacesYou can define search space in Json.Each hyperparameter config must havetypeandkeyword.
You can see what parameters are available for each hyperparameter inOptuna API reference.[{"type":"int","attributes":{"name":"embedding_dim","low":64,"high":128}},{"type":"int","attributes":{"name":"max_filter_size","low":2,"high":5}},{"type":"int","attributes":{"name":"num_filters","low":64,"high":256}},{"type":"int","attributes":{"name":"output_dim","low":64,"high":256}},{"type":"float","attributes":{"name":"dropout","low":0.0,"high":0.5}},{"type":"float","attributes":{"name":"lr","low":5e-3,"high":5e-1,"log":true}}]Parameters forsuggest_#{type}are available for config oftype=#{type}. (e.g. whentype=float,
you can see the available parameters insuggest_floatPlease see theexamplein detail.2.3. Optimize hyperparameters by allennlp cliallennlptune\config/imdb_optuna.jsonnet\config/hparams.json\--serialization-dirresult/hpo\--study-nametestOptionally, you can specify the metrics and direction you are optimizing for:allennlptune\config/imdb_optuna.jsonnet\config/hparams.json\--serialization-dirresult/hpo\--study-nametest\--metricsbest_validation_accuracy\--directionmaximize2.4. [Optional] Specify Optuna configurationsYou can choose a pruner/sample implemented in Optuna.
To specify a pruner/sampler, create a JSON config fileThe example ofoptuna.jsonlooks like:{"pruner":{"type":"HyperbandPruner","attributes":{"min_resource":1,"reduction_factor":5}},"sampler":{"type":"TPESampler","attributes":{"n_startup_trials":5}}}And add a epoch callback to your configuration.
(https://guide.allennlp.org/hyperparameter-optimization#6)callbacks: [
    {
      type: 'optuna_pruner',
    }
  ],config/imdb_optuna.jsonnetis a simple configuration for allennlp-optunaconfig/imdb_optuna_with_pruning.jsonnetis a configuration using Optuna pruner (and TPEsampler)$diffconfig/imdb_optuna.jsonnetconfig/imdb_optuna_with_pruning.jsonnet
32d31
<datasets_for_vocab_creation:['train'],
58a58,62
>callbacks:[>{>type:'optuna_pruner',
>}>],Then, you can use a pruning callback by running following:allennlptune\config/imdb_optuna_with_pruning.jsonnet\config/hparams.json\--optuna-param-pathconfig/optuna.json\--serialization-dirresult/hpo_with_optuna_config\--study-nametest_with_pruning3. Get best hyperparametersallennlpbest-params\--study-nametest4. Retrain a model with optimized hyperparametersallennlpretrain\config/imdb_optuna.jsonnet\--serialization-dirretrain_result\--study-nametest5. Hyperparameter optimization at scale!you can run optimizations in parallel.
You can easily run distributed optimization by adding an option--skip-if-existstoallennlp tunecommand.allennlp tune \
    config/imdb_optuna.jsonnet \
    config/hparams.json \
    --optuna-param-path config/optuna.json \
    --serialization-dir result \
    --study-name test \
    --skip-if-existsallennlp-optuna uses SQLite as a default storage for storing results.
You can easily run distributed optimizationover machinesby using MySQL or PostgreSQL as a storage.For example, if you want to use MySQL as a storage,
the command should be like following:allennlp tune \
    config/imdb_optuna.jsonnet \
    config/hparams.json \
    --optuna-param-path config/optuna.json \
    --serialization-dir result \
    --study-name test \
    --storage mysql://<user_name>:<passwd>@<db_host>/<db_name> \
    --skip-if-existsYou can run the above command on each machine to
run multi-node distributed optimization.If you want to know about a mechanism of Optuna distributed optimization,
please see the official documentation:https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/004_distributed.htmlReferenceCookpad Techlife (in Japanese):https://techlife.cookpad.com/entry/2020/11/06/110000allennlp-optunais used for optimizing hyperparameter of NER model