Package: ml-experiment-manager
Machine Learning Experiment ManagerSimply manage pytorch and skorch models, hparams, output charts, and results CSV filesContainer to hold all information about an experiment: tabular data
created during training and inference, hyper parameters, charts,
tensorboard entries, and (pytorch/skorch) model snapshots. These data are
stored on disk in standard formats that are accessible by other tools,
such as Excel, pytorch, and image viewers. Except for model snapshots, the
facility is ML platform agnostic. Only pytorch and skorch model instances are
currently handled for saving, though all other features remain fully
functional.The user associates each asset with a key (string) of their
choice. The manager accesses the asset when given a key.Selected Details:Persistent dict API for 'non-special' data, such as lists,
numbers, or dicts.Uniform API for special machine learning data types like
tables, pyplot figures, and model snapshots.Creates human-readable files under a single directory rootEntire experiment archives are movable/copyable with standard OS tools.Files on disk are .csv, for tables, Pandas Series and DataFrame
instances, pdf, png, etc. for pyplot figures, state_dict .pth
files for pytorch models, and .pkl save_params exports of skorch
models.Starting an Experiment and Saving DataExamples: an experiment instance is started to retain all data in
subdirectories of MyExperiment under the current directory. All
archived data will reside under this root.'''
pip install ml-experiment-manager
'''The primary method issave(), which accepts an arbitrary stringkey, and data of varying types. The method stores the data in
files appropriate for those types:from experiment_manager.experiment_manager import ExperimentManager, Datatype
    
    experiment_archive = os.path.join(os.path.dirname(__file__), 'MyExperiment')
    exp = ExperimentManager(experiment_archive)

    # Create a tabular entry called 'my_tbl'. An underlying .csv file
    # named my_tbl.csv will be created under experiment_archive:

    probabilities = pd.DataFrame([[.1,.2,.7],[.4,.5,.1]], columns=['foo', 'bar'])
    exp.save('my_tbl', probabilities)

    # Add a row to the same tabular content (i.e. to the same .csv file):
    exp.save('my_tbl, pd.Series([.7,.1,.2], index=['foo', 'bar']))

    # A bit more data still
    df_more = pd.DataFrame([[.3,.3,.4],[.5,.5.,0],[.6,.2,.2]],
                           columns=['foo', 'bar'])
    exp.save('my_tbl', df_more)

    # Save a pytorch or skorch model snapshot; the manager will save the state_dict,
    # using torch.save() or model.save_params() for skorch models. The type of
    # model is detected automatically:
    # 
    exp.save('model_snaphot1', model1)

    # Save another snapshot
    exp.save('model_snaphot2', model2)

    # Result chart figures in format indicated by
    # extension of the archive key:

    exp.save('pr_figure.pdf', pr_curve)
    exp.save('cm.png', conf_matrix)

    # A tensorboard location. Note that in this case
    # only the key is provided. This string will be a subdirectory
    # name under the experiment root. As usual for the
    # full path to that directory is returned, and can be
    # used when creating the tensorboard writer:
    
    exp.save('tb_data')
    
    # Saving any other, non-special type of data, as long as
    # json.dump() can handle it: use ExperimentManager instances like
    # a persistent dict. The new dict state is saved soon after being
    # updated

    exp['some_number'] = 10
    exp['some_other_number'] = 20Client's should callclose()on an experiment to close all open
writers.Opening Existing ExperimentsTo reopen an existing experiment:exp = ExperimentManager(experiment_archive_root)The data saved in the experiment can be retrieved as appropriate
Python data types through theread()method. Callers must provide
the data key and the type of data being requested.# Obtain a Pandas DataFrame instance:
   my_tbl_df = exp.read('my_tbl', Datatype.tabular)

   # A pyplot Figure instance:
   my_fig    = exp.read('pr_figure.pdf', Datatype.figure)

   # Pytorch and skorch model retrievals work by
   # creating an uninitialized model, i.e. a pytorch module,
   # or a skorch net. That model is passed to read, which
   # returns the model with state initialized.
   my_model = <create torch.nn.Module or skorch.classifier.NeuralNet>
   my_model_initialized  = exp.read('model_snaphot1', Datatype.model, initialized_net=my_model)

   # A [hyperparameter configuration][#hyperparameters]:
   config = exp.read('hparams')

   # The full tensorboard directory path:
   tb_path = exp.read('tb_data')HyperparametersHyperparameter values may be stored as dict key/value pairs
(exp['lr'] = 0.8). However a class NeuralNetConfig is available for
organizing hyperparameters. Given an instanceconfigof this class,exp.save('hparams', config)will create a copy of the configuration under the experiment root.NeuralNetConfigextends thestandard Pythonconfigparserpackage.  That
is, NeuralNetConfig instances read configuration files of the form[Paths]

# Root of the data/test files:
root_train_test_data = /Users/paepcke/EclipseWorkspacesNew/birds/src/birdsong/tests/data/birds

[Training]

net_type      = resnet18
batch_size    = 64
lr            = 0.8
pretrained    = True
class_names   = foo,bar,baz

       etc.wherePathandTrainingare calledsections. The NeuralNetConfig
class adds the following convenience methods:# Obtain specific data types, rather than strings:

   config.getint('Training', 'batch_size')
   config.getfloat('Training', 'lr')
   config.getboolean('Training', 'pretrained')
   config.getarray('Training', 'class_names')
   config.sections()

   config.copy()

   # Equality test:
   config1 == config2
   
   config.to_json()
   config.from_json()Miscellaneous Methods# Obtain full path to any saved data;
   # datatype is an element of the Datatype
   # enumeration:
   exp.abspath(<key>, datatype)