Package: scientific-experiment-manager
SEM: Scientific Experiment ManagerStreamline IO operations, storage, and retrieval of your scientific resultsSEM helps streamline IO operations and organization of scientific results
in Python.At its core, SEM is based onregular expressionsand simply creates, parses and manages intricate folder structures containing
experimental results.Minimal exampleConsider the results organized in theexample/example_resultsfolder.These are different directories containing the results of the same experiment, where two
parameters are varied: the randomseedand a threshold valueeps. Every one of the
folders contains some output files fromexample_results
│
└───seed=111
│   └───eps_1.3
│   │   └───...
│   └───eps_7.4
│       └───...
│   
└───seed=222
│   └───...
│
└───seed=333
│   └───...
│   
└───useless_filesSEM does not take care of loading and/or saving files.Rather, it takes care of the folder structure, leaving the user the freedom to manage the result's
format.To retrieve the parameters relative to these results,ResultManagerparses the folders'
names and only returns the path relative to those that match.importrefrompathlibimportPathfromsem.managerimportResultManagerexample_res=Path("./example_results")parsers=[re.compile(r"seed=(?P<seed_value>\d+)"),re.compile(r"eps_(?P<eps>\d+.\d+)")]manager=ResultManager(root_dir=example_res,parsers=parsers)manager.parse_paths()In the case above, the parser forseed_valueexpects a positive integer, specified by
the regular expression"\d+", andepsa float format.The results are stored inmanager.df, a pandasDataFrame, which
contains the parsed parameter values, as well as the path to the deepest sub-directories__PATH__ seed_value  eps
0  example_results/seed=333/eps_1.1        333  1.1
1  example_results/seed=333/eps_0.3        333  0.3
2  example_results/seed=222/eps_7.4        222  7.4
3  example_results/seed=222/eps_2.7        222  2.7
4  example_results/seed=111/eps_1.3        111  1.3
5  example_results/seed=111/eps_7.4        111  7.4
...Directories whose names don't match the patterns are ignored, e.g.example_results/useless_files.Notice that, since they are the results of parsing, all the values in the data frame are
strings.The conversion to a different data type can be performed after parsing:manager.df["seed_value"]=manager.df["seed_value"].astype(int)manager.df["eps"]=manager.df["eps"].astype(float)Utilizing the parsed directoriesOnce the directory names have been parsed, the main utility of the manager is to have a
coupling between the parameters and the results.For example, one can read and insert the computational time of every experiment in the
data frame:defread_comp_time(res_dir):withopen(res_dir/"computational_time.txt","r")asfile:time=float(file.read())returntimemanager.df["time"]=manager.df["__PATH__"].map(read_comp_time)From there, conventional pandas operations can be used. For example, the average
computational time for seed111is given bydf=manager.dftimes=df["time"].loc[df["seed_value"]==111]times.mean()Loading more complex objectsPandas data frames can contain arbitrary objects.
For example, one can create a column of numpy arrays from a model:importnumpyasnpdefload_mat(path):returnnp.load(path/"result_params.npy")df["mat"]=df["__PATH__"].map(load_mat)Creating default pathsStandardizing result structure reduces the amount of code needed for
simple IO operations, and eases compatibility across machines, e.g. local vs cloud or
cluster results.To this end,SEMoffers a way to create saving paths
which only depend on the parameters specified by the user.For example, the paths of a repository with three levels and different parameters,
can be created as:root_dir=Path(".")/"tmp"forparam1in[True,False]:forparam2in["a","b"]:forparam3in[1,2,3]:values=[{"param1":param1,"param2":param2},"results_of_my_experiments",{"param3":param3},]new_path=ResultManager.create_default_path(root_dir,values,auto_sort=True)new_path.mkdir(parents=True)print(new_path)which producestmp/param1=True_param2=a/results_of_my_experiments/param3=1
tmp/param1=True_param2=a/results_of_my_experiments/param3=2
tmp/param1=True_param2=a/results_of_my_experiments/param3=3
tmp/param1=True_param2=b/results_of_my_experiments/param3=1
...
tmp/param1=False_param2=a/results_of_my_experiments/param3=1
...If desired, the argumentauto_sortimposes a uniform order at every directory level.For example, using{"param2": param2, "param1": param1}would produce the same paths a above ifauto_sort=True.Parsing directories with this structure is similarly easy:manager=ResultManager.from_arguments(root_dir,arguments=[{"param1":"True|False","param2":"a|b"},"results_of_my_experiments",{"param3":r"\d+"},],auto_sort=True)manager.parse_paths()which yields__PATH__ param1 param2 param3
0   tmp/param1=False_param2=b/results_of_my_experi...  False      b      1
1   tmp/param1=False_param2=b/results_of_my_experi...  False      b      3
2   tmp/param1=False_param2=b/results_of_my_experi...  False      b      2
3   tmp/param1=True_param2=b/results_of_my_experim...   True      b      1
...InitializationNotice that the advantage of using the default directory naming, as opposed to a custom
one, is that theResultManagercan be initialized as above, by only specifying the
arguments inResultManager.from_arguments.A more flexible initialization for custom paths, can be performed by giving as input
regular expression patterns. For example, an equivalent initialization to that above is
given by:parsers=[re.compile("param1=(?P<param1>True|False)_param2=(?P<param2>a|b)"),re.compile("results_of_my_experiments"),re.compile("param3=(?P<param3>\d+)"),]manager=ResultManager(root_dir,parsers)manager.parse_paths()Other utilities and tricksFiltering resultsAnother usefulResultManagermethod isResultManager.filter. This method filters therowsof the results' data frame. Results can be selected by specifying exact
column values or a list of possible values. For example, for a manager whose data frame
has columns__PATH__ param1 param2 param3
0   tmp/param1=False_param2=b/results_of_my_experi...  False      b      1
1   tmp/param1=False_param2=b/results_of_my_experi...  False      b      3
2   tmp/param1=False_param2=b/results_of_my_experi...  False      b      2
3   tmp/param1=True_param2=b/results_of_my_experim...   True      b      1
...the querymanager.filter_results(equal={"param1":True},contained={"param3":[1,3]})yields a filtered data frame__PATH__ param1 param2 param3
3   tmp/param1=True_param2=b/results_of_my_experim...   True      b      1
4   tmp/param1=True_param2=b/results_of_my_experim...   True      b      3
9   tmp/param1=True_param2=a/results_of_my_experim...   True      a      1
10  tmp/param1=True_param2=a/results_of_my_experim...   True      a      3Loading fewer resultsWhile results can be filtered a posteriori as just explained, one can also load fewer
results in the first place.This is done by specifying an appropriate regular expression
parser in the first place.For example, to select only configurations whereparam1is equal toTrue, one can writeparsers=[re.compile("param1=(?P<param1>True)_param2=(?P<param2>a|b)"),re.compile("results_of_my_experiments"),re.compile("param3=(?P<param3>\d+)"),]manager=ResultManager(root_dir,parsers)In general, any regular expression with named groups is considered valid, checkthe docsfor further details.Common parsing patternsSome common regular expression patterns are available atsem.re_patterns.These are strings that can be utilized for initializing parsersfromsem.re_patternsimportINT_PATTERNparsers=[re.compile("param1=(?P<param1>True|False)_param2=(?P<param2>a|b)"),re.compile("results_of_my_experiments"),re.compile(f"param3=(?P<param3>{INT_PATTERN})"),]manager=ResultManager(root_dir,parsers)orResultManagerargumentsmanager=ResultManager.from_arguments(root_dir,arguments=[{"param1":"True|False","param2":"a|b"},"results_of_my_experiments",{"param3":INT_PATTERN},],)Common type conversion from stringSome common type conversion functions from string are available atsem.str_to_type.These are useful in combination with theargparsepackage, for command line inputsfromargparseimportArgumentParserfromsem.str_to_typeimportbool_type,unit_float_or_positive_integer,none_or_typeparser=ArgumentParser()parser.add_argument("--flag",type=bool_type)parser.add_argument("--train_size",type=unit_float_or_positive_integer)parser.add_argument("--K",type=none_or_type(int))Importantly,bool_typecorrectly converts both string inputs"0"or"1", as well
as the case-insensitive strings"true","True","False", etc.Alternatively, these functions can also be used for type conversion inside pandas data
framesmanager=ResultManager(root_dir,parsers)manager.parse_paths()manager.df["flag"]=manager.df["flag"].map(bool_type)InstallationYou can install this package by downloading the GitHub repository and, from inside the
downloaded folder, runningpip install .