Package: asreview-hyperopt
ASReview-hyperoptHyper parameter optimization extension forASReview. It uses thehyperoptpackage to quickly optimize parameters
of the different models. The hyper parameters and their sample space are defined in theASReviewpackage, and
automatically used for hyper parameter optimization.InstallationThe easiest way to install the hyper parameter optimization package is to use the command line:pipinstallasreview-hyperoptAfter installation of the visualization package, asreview should automatically detect it.
Test this by:asreview--helpIt should list three new entry points:hyper-active,hyper-passiveandhyper-cluster.Basic usageThe three entry-points are used in a roughly similar fashion. The main difference between them is
the types of models that have to be supplied:hyper-cluster: feature_extractionhyper-passive: model, balance_strategy, feature_extractionhyper-active: model, balance_strategy, query_strategy, feature_extractionTo get help for entry points type:asreviewhyper-active--helpWhich results in the following options:usage:hyper-active[-h][-nN_ITER][-rN_RUN][-dDATASETS][--mpi][--data_dirDATA_DIR][--output_dirOUTPUT_DIR][--server_job][-mMODEL][-qQUERY_STRATEGY][-bBALANCE_STRATEGY][-eFEATURE_EXTRACTION]optionalarguments:-h,--helpshowthishelpmessageandexit-nN_ITER,--n_iterN_ITERNumberofiterationsofBayesianOptimization.-rN_RUN,--n_runN_RUNNumberofrunsperdataset.-dDATASETS,--datasetsDATASETSDatasetstouseinthehyperparameteroptimizationSeparatebycommastousemultipleatthesametime[default:all].--mpiUsethempiimplementation.--data_dirDATA_DIRBasedirectorywithdatafiles.--output_dirOUTPUT_DIROutputdirectoryfortrials.--server_jobRunjobontheserver.ItwillincurlessoverheadofusedCPUs,butmorelatencyofworkerswaitingfortheservertofinishitsownjob.Onlymakessenseincombinationwiththeflag--mpi.-mMODEL,--modelMODELPredictionmodelforactivelearning.-qQUERY_STRATEGY,--query_strategyQUERY_STRATEGYQuerystrategyforactivelearning.-bBALANCE_STRATEGY,--balance_strategyBALANCE_STRATEGYBalancestrategyforactivelearning.-eFEATURE_EXTRACTION,--feature_extractionFEATURE_EXTRACTIONFeatureextractionmethod.Data structureThe extension will by default search for datasets in thedatadirectory, relative to the current
working directory. Either put your datasets there, or specify and data directory.The output of the runs will by default be stored in theoutputdirectory, relative to
the current path.An example of a structure that has been created:output/
├──active_learning
│├──nb_max_double_tfidf
││└──depression_hall_ace_ptsd_nagtegaal
││├──best
│││├──ace
│││├──depression
│││├──hall
│││├──nagtegaal
│││└──ptsd
││├──current
│││├──ace
│││├──depression
│││├──hall
│││├──nagtegaal
│││└──ptsd
││└──trials.pkl
│└──nb_max_random_double_tfidf
│└──nagtegaal
│├──best
││└──nagtegaal
│├──current
││└──nagtegaal
│└──trials.pkl
├──cluster
│└──doc2vec
│├──ace
││├──best
│││└──ace
││├──current
│││└──ace
││└──trials.pkl
│├──depression_hall_ace_ptsd_nagtegaal
││└──current
││├──ace
││├──depression
││├──hall
││├──nagtegaal
││└──ptsd
│└──nagtegaal
│└──current
│└──nagtegaal
└──passive└──nb_double_tfidf└──depression├──best│└──depression├──current│└──depression└──trials.pklThe files with nametrials.pklare special files that contain data on which trials were run.To list these trials, use the following command:asreviewshow$SOME_DIRECTORY/trials.pklIt should give a list of trials sorted by the loss (lower is better). The column names (apart
from the loss) are prefixed with the kind of parameter it is:mdl: Model parameterbal: Balance strategy parameterqry: Query strategy parameterfex: Feature extraction parameterOptionsThe default number of iterations is 1, which you'll probably want to increase. It depends on the
number of hyper-parameters that need to be optimized, but several hundred iterations is probably
a good estimate for most combinations to get reasonably close to the optimum. In all cases,
use good common sense; if the loss is still going down at a quick pace, do a few more iterations.The hyperopt extension has built-in support for MPI. MPI is used for parallelization of runs. On
a local PC with an MPI-implementation (like OpenMPI) installed, one could run with 4 cores:mpirun-n4asreviewhyper-active--mpiIf you want to be slightly more efficient on a machine with a low number of cores, you can run
jobs on the MPI server as well:mpirun-n4asreviewhyper-active--mpi--server_jobOn super computers one should sometimes replacempirunwithsrun.