Package: digitallab
Digital Lab (digitallab)Introductiondigitallabis a python package for conducting large-scale computational experiments. The underlying framework
is based on the modulesacred. It extends its functionality by
allowing batches of experiments, repetitions of experiments with different seeds, and parallel execution
of experiments. Furthermore, it provides tools to evaluate the experiments via plots or tables.DependenciesPython packages:numpytqdmsacredpandasseabornmatplotlibpytablespickOptional dependencies:For using MongoDB:pymongoMongoDB databaseFor using TinyDB:tinydb (~=3.15.2)tinydb-serialization (~=1.0.4)hashsfInstallationVia pipRunpip install --user digitallabFrom sourceClone the project to your hard drive and run the commandpython3 setup.py install --userin the project folder.ExampleConducting experimentsAssume we want to compare the run times and quality of three methods (fast,slow,special).fastandsloware taking the same arguments whilespecialrequires an extra parameter.
We want to compare two instances "A" and "B". The three methods are defined as follows:import numpy as np

def slow(config):
    np.random.seed(config["seed"])
    return_dict = dict()
    if config["instance"] == "A":
        return_dict["runtime"] = np.max(np.random.normal(1000, scale=300), 0)
        return_dict["value"] = np.random.normal(1, scale=0.5)
    else:
        return_dict["runtime"] = np.max(np.random.normal(10000, scale=300), 0)
        return_dict["value"] = np.random.normal(10, scale=0.5)
    return return_dict


def fast(config):
    np.random.seed(config["seed"])
    return_dict = dict()
    if config["instance"] == "A":
        return_dict["runtime"] = np.max(np.random.normal(200, scale=100), 0)
        return_dict["value"] = np.random.normal(2, scale=0.7)
    else:
        return_dict["runtime"] = np.max(np.random.normal(2000, scale=100), 0)
        return_dict["value"] = np.random.normal(20, scale=0.7)
    return return_dict


def special(config):
    np.random.seed(config["seed"])
    return_dict = dict()
    if config["instance"] == "A":
        return_dict["runtime"] = np.max(np.random.normal(500, scale=100), 0)
        return_dict["value"] = np.random.normal(1.5, scale=config["scale"])
    else:
        return_dict["runtime"] = np.max(np.random.normal(5000, scale=100), 0)
        return_dict["value"] = np.random.normal(15, scale=config["scale"])
    return return_dictThen we can run the experiments. For the purpose of this example we will be using TinyDB, however
MongoDB is highly recommended and should be the preferred database for storing experimental results.from dlab.lab import Lab

# create the lab
lab = Lab("example").add_tinydb_storage("example_db")Then we assign two dictonaries which define our experiments.digitallabwill provide every
possible combination of parameters to our experiment function. Additionally, every
parameter combination will be submitted as often as specified by the fieldnumber_of_repetitions(each time with a different seed). By the way, a fieldseedis added for each config
with the specific seed. The results of the experiments can be deleted and the experiments
repeated and the given seeds will be identical.Mandatory keys in a settings file areexperiment,sub_experiment,version, andnumber_of_repetitions.standard_setting = {
    "experiment": "test",
    "sub_experiment": "standard",
    "version": "1",
    "number_of_repetitions": 10,
    "method": ["slow", "fast"],
    "instance": ["A", "B"]
}

special_setting = {
    "experiment": "test",
    "sub_experiment": "special",
    "version": "1",
    "number_of_repetitions": 10,
    "method": "special",
    "scale": [0.1, 0.5, 1],
    "instance": ["A", "B"]
}Finally we can define our experiment function and run the experiments:@lab.experiment
def main(_config):
    if _config["method"] == "fast":
        return fast(_config)
    elif _config["method"] == "slow":
        return slow(_config)
    elif _config["method"] == "special":
        return special(_config)Evaluating experimentsTo be done...ToDosThe project is work in progress and there are still some tasks to be done:DocumentationExamplesAdd support for SQLFaster caching!Experiments should not run if they do not have a matching experiment nameUI (perhaps)