Package: experitur
experiturexperiturautomates machine learning and other computer experiments. Includes grid search and resuming aborted experiments. No lock-in, all your data is easily accessible in the text-based, machine-readable YAML format.Experiment descriptionEvery experiment is described in a regular python file. The@experimentdecorator is used to mark experiment entry-points.fromexperiturimportexperiment@experiment(parameter_grid={"parameter_1":[1,2,3],"parameter_2":["a","b","c"],})defexample(trial):"""This is an example experiment."""...Parameter gridThe core of an experiment is itsparameter grid. It works likesklearn.model_selection.ParameterGrid. Each parameter has a list of values that it can take. A number oftrialsis generated from the cross product of the values of each parameter.Entry pointAn experiment is a regular function that is decorated with@experiment(unless it isabstractorderived). Upon execution, the function gets called with the current trial. It may return a result dictionary.Signature:(trial) -> dictfromexperiturimportexperiment@experiment(parameter_grid={"parameter_1":[1,2,3],"parameter_2":["a","b","c"],})defexample(trial):"""This is an example experiment."""print("parameters:",pformat(parameters))return{}Now, you can run the experiment:$ experitur run example.py
...As you can see,runwas called four times with every combination of [1,2] x [a,b].Multiple experimentsThe Python file can contain multiple experiments:fromexperiturimportexperiment@experiment(...)defexample1(trial):...@experiment(...)defexample2(trial):...Experiment inheritanceOne experiment may inherit the settings of another, using theparentparameter.fromexperiturimportexperiment@experiment(...)defexample1(trial):...# Derived  with own entry point:@experiment(parent=example1)defexample2(trial):...# Derived  with inherited entry point:example3=experiment("example3",parent=example2)Parameter substitutionexperiturincludes a recursive parameter substitution engine. Each value string is treated as arecursive format stringand is resolved using the whole parameter set of a trial.@experiment(parameter_grid={"a1":[1],"a2":[2],"b":[1,2],"a":["{a_{b}}"],})defexample(trial):...$ experitur run parsub
...This way, you can easily run complicated setups with settings that depend on other settings.Recursive format strings work likestring.Formatterwith two exceptions:Recursive field names:The field name itself may be a format string:format("{foo_{bar}}", bar="baz", foo_baz="foo") -> "foo"Literal output:If the format string consist solely of a replacement field and does not contain a format specification, no to-string conversion is performed:format("{}", 1) -> 1This allows the use of format strings for non-string values.ApplicationThis feature is especially useful if you want to run your experiments for different datasets but need slightly different settings for each dataset.Let's assume we have two datasets, "bees" and "flowers".@experiment(parameter_grid={"dataset":["bees","flowers"],"dataset_fn":["/data/{dataset}/index.csv"],"bees-crop":[10],"flowers-crop":[0],"crop":["{{dataset}-crop}"]})defexample(trial):...The experiment will be executed once for each dataset, withtrial["crop"]==10for the "bees" dataset andtrial["crop"]==0for the "flowers" dataset.ThetrialobjectEvery experiment receives atrialobject that allows access to the parameters and meta-data of the trial.Parameters are accessed with the[]operator (e.g.trial["a"]), meta-data is accessed with the.operator (e.g.trial.wdir).Access of parent data...FilesWhenexperiturexecutes a script, it creates the following file structure in the directory where the DOX file is located:/
+- script.py
+- script/
|  +- experiment_id/
|  |  +- trial_id/
|  |  |  +- experitur.yaml
|  |  ...
|  ...<script>/<experiment_id>/<trial_id>/experitur.yamlcontains the parameters and the results from a trial, e.g.:callable:example.experiment1experiment:experiment1id:experiment1/a-1_b-3parameters:{a:1,b:3}parent_experiment:nullresult:nullsuccess:truetime_end:2019-06-07 14:22:41.697925time_start:2019-06-07 14:22:41.697837wdir:examples/example/experiment1/a-1_b-3Most items should be self-explanatory.parametersare the parameters passed to the entry point.idis derived from the parameters that are varied in the parameter grid. This way, you can easily interpret the file structure.Collecting resultsUseexperitur collect script.pyto collect all the results (including parameters and metadata) of all trials of a lab book into a single CSV file located atscript/results.csv.Calling functions and default parametersYour experiment function might call other functions that have default parameters.experiturgives you some utility functions that extract these default parameters adds them to the list of parameters.For the following examples, let's assumetrial["p_a"]=1andtrial["p_b"]=2.trial.without_prefix(prefix: str, parameters: dict) -> dict: Extract all parameters that start withprefix.trial.without_prefix("p_")=={"a":1,"b":2}trial.apply(prefix: str, callable_: callable, *args, **kwargs): Callcallable_with the parameters starting withprefix.trial.apply("p_",fun,10,c=20)# is the same asfun(10,a=1,b=2,c=20)trial.record_defaults(prefix, [callable_,] **defaults): Set default values for parameters that were not set previously. Values indefaultsoverride default parameters ofcallable_.deffoo(a=1,b=2,c=3):passset_default_parameters("foo_",parameters,foo,c=4)# is the same asparameters.setdefault("foo_a",1)parameters.setdefault("foo_b",2)parameters.setdefault("foo_c",4)It is a good idea to make use ofset_default_parametersandapply_parametersexcessively. This way, your result files always contain the full set of parameters.For a simple example, seeexamples/example.py.Installationexperituris packaged onPyPI.pipinstallexperiturBe warned that this package is currently under heavy development and anything might change any time!Examplesexamples/example.py: A very basic example showing the workings ofset_default_parametersandapply_parameters.examples/classifier.py: Try different parameters ofsklearn.svm.SVCto classify handwritten digits (theMNISTtest set). Run the example, add more parameter values and see howexperiturskips already existing configurations during the next run.Contributionsexperituris under active development, so any user feedback, bug reports, comments, suggestions, or pull requests are highly appreciated. Please use thebug trackerandforkthe repository.Compatibilityexperituristestedwith Python 3.5, 3.6 and 3.7.Similar softwareSacred