Package: hyperspace-explorer
Hyperspace explorerCollection of tools meant to enable faster progress and reproducibility of results in ML/DL
projects and competitions, without assuming too much about the projects themselves and used tools.IntroductionWhat is it useful for?GoalsGlossarySetupInstallationProject structureAdding new parameters, default valuesUsageRunning a workerBrowsing experiment resultsPossible access points, usage modesCLIRun queue + workersInteractive prototyping in JupyterRunning testsIntroductionWhat is it useful for?Meant to support a process where we have:a dataset, possibly pre-processed to some degree, that can be deterministically loaded (multiple versions of data?
loading code should take an argument specifying which version to load)an objective - we only consider supervised ML problems here, and we need some quality metric (be it
accuracy or AUROC of classification, RMSE of regression, or something else). The metric can be changed, multiple metrics also supported.A common, useful approach is to then to quickly develop a simple version of the full pipeline:loading the datapre-processing, data augmentation, etcthe model logic (if not a ready-made solution - perhaps a custom neural architecture?)model training procedurescoring the model on the validation setThen the real work begins - multiple iterations of:developing and testing new versions of parts of the pipeline- adding dimensions to the hyper-parameter
space (e.g. different neural architecture, different kind of model altogether, different pre-trained network, different
data pre-processing logic),optimizing hyper-parameters,inspecting how/when the model fails, and how it works - to guide further development,(in some cases) periodically pushing new versions to production.This package is meant to provide common utilities supporting workflows like those, regardless
of the frameworks/libraries in use or the structure of the problem to solve. It was born from
common parts of multiple projects, often having a "weird" structure - e.g.:fitting multiple models with identical hyper-parameters to time-series from corresponding machines,periodic re-training of the model (also for time-series)testing the same approach on multiple subsets of the training setThe strength of the package lies in its modularity, minimal assumptions about what you need to do
and how you need to do it. It is meant to get out of your way - your code should still work without it,
ran the way you need - be it CLI, a notebook, etc.GoalsIt should be possible to:store and analyze results of all past experiments,reproduce past results (-> store results together with hyper-parameters, data, versions of code, ...),easily re-run past experiments with some hyper-parameters changed,develop or inspect parts of the pipeline in a notebook, easily run all of it from there (TODO: instructions),extend the code's functionality and the hyper-parameter space; new hyper-params should always be added
with default values matching previous behaviour. Replacing pieces of logic with different ones, with different
hyper-parameters should be handled as well.queue runs from a notebook (e.g. generate a grid of HPs in some dimensions) and have them
executed by worker processes (possibly distributed among multiple machines). Then close the notebook, or
add some more runs, never again waiting for computations to finish.install core parts of the project's package and use it in production code (if the code got ugly with too many
options, write a new version of it, and compare results with the old version, all in the same environment)use automatic hyper-parameter tuning algorithms, informed by all past experiments during development.GlossaryRun- single execution of aTaskwith a specific config (set of parameters), concluded by calculation
of a quality metric.Example: training a classifier with the hyper-parameters provided in the config, calculating accuracy
on the validation set.Each run gets its entry in a selected MongoDB database, in therunscollection. It includes
all parameters, code versions, result metrics, captured output, and possibly custom diagnostic information.Task- fully specified problem to solve, and then compare different solutions to.Consists of a chosenScenarioclass and parameter values for its constructor.
Each task should be specified as a .json file and include keys:Scenario(= dictionary of its settings, starting withclassName) and optionallyseed.Scenario- template of aTaskwithout its parameters. When parameterized byTaskparameters
and aRunconfig it can be executed, and should return a quality metric. EachScenariois a class,
inheriting fromscenario_base.Scenario. Its.single_run()method typically contains dataset construction,
model construction, model training and evaluation.Study- a project in which we define one or moreScenariosto study
solutions for, with a shared codebase, database, and virtual environment.
Corresponds to a Python package, which has a dependency onhyperspace_explorerand includes ascenariosmodule.SetupInstallationInstall with pip, as a normal python package.Other than all the modules being made available for import, it will also makehyperspace_worker.pyavailable in your system's PATH (or a specific virtual environment).To use most of this package's functions a running instance of MongoDB will be needed.Project structureTODOAdding new parameters, default valuesAs we add new parameters to our code (our classes), we must provide default
values that ensure behaviour consistent with the earlier versions of the code.To achieve that, default values (they do not have to be optimal/recommended
ones!) live inside the classes. They must not be set in function  - instead,
they should be provided by the.get_default_values()method of eachConfigurable. If using dataclasses, that method is provided automatically.When a worker process starts to process a Run, it fills in all default valuesthey end up stored in the database. It is redundant, but makes result analysis
much easier - they do not have to be aware of the Study-specific codebase.If expanding a Scenario to fit more Tasks, and adding parameters to it,
similarly default values should be provided for all of them.A tool for back-filling new default values to past runs (while keeping original
config in a different field as backup?) would be useful - future development.UsageRunning a workerRun a commandhyperspace_worker.py [path to tasks dir] [mongo db name].Important:it has to be ran from a directory containing ascenarios.pymodule, which defines
experiment scenarios allowed to be ran within the given project. Thehyperspace_worker.pyfile
should not be present in the folder.Arguments:path to tasks dir- path to a directory containing.jsonfiles, describing each allowedtask-
a parameterization of ascenariomongo db name- name of the mongoDB database to store results inoptional params: mongoDB URI (if not localhost, or if password is required), interval to query for new tasksBrowsing experiment resultsThis project (ab)usesSacredto collect and store information about each run.One of the benefits: we can use many ready-made dashboards for Sacred,
e.g.Omniboard- highly recommended, works out of the box,
many impressive features.Possible access points, usage modesCLITODORun queue + workersStart workers on 1 or more nodes, set them up to use the same database (which
also serves as a task queue). Workers are specific to one Study (project) -
will only process tasks for the Study they were started for.Example code, usually ran from a notebook, to submit one task. From here, it is
easy to e.g. submit a grid of hyper-parameters for the workers to test.fromhyperspace_explorer.queueimportRunQueuefrompathlibimportPathtasks_dir=Path.cwd().resolve().parent/'tasks'# just an example - relative to the notebookdb_name='ulmfit_attention'mongo_uri='localhost:27017'q=RunQueue(mongo_uri,db_name,tasks_dir)task_name='imdb_1k_sample_single'conf={'aggregation':{# different additional parameters are available depending on `className`'className':'BranchingAttentionAggregation','agg_layers':[50,10]},'classifier':{# this dict is passed to a specific function within a scenario, but polymorphism is not needed'lin_ftrs':[],'drop_mult':0.5,},'training_schedule':{# even if we do not want to change any default parameters, className is required'className':'DefaultSchedule',}}q.submit(task_name,conf)The code above works with the project:https://github.com/tpietruszka/ulmfit_attention.
In this case workers should be ran from within the innerulmfit_attentiondirectory.Interactive prototyping in JupyterTODORunning testsTODO