Package: malet
Malet: a tool for machine learning experiment🔨Malet(MachineLearningExperimentTool) is a tool for efficient machine learning experiment execution, logging, analysis, and plot making.The following features are provided:Simple YAML-based hyperparameter configuration w/ grid search syntaxExperiment logging and resuming systemUser-friendly command-line tool for flexible graphing and easy data extraction from experiment logsEfficient parallelization by splitting a sequence of experiments over GPU jobsInstallationYou can install Malet using pip,pipinstallmaletor from this repository.pipinstallgit+https://github.com/edong6768/Malet.gitDependenciesabsl-py 1.0.0numpy 1.22.0pandas 2.0.3matplotlib 3.7.0ml-collections 0.1.0seaborn 0.11.2Documentation(🚨 Plan to move to Sphinx-based Read the docs in near future)ContentsQuick startPrerequisiteRunning experimentsPlot makingAdvanced topicsAdvanced gridding in yamlAdvanced plot makingParallel friendly grid splittingSaving logs in intermediate epochsMerging multiple log filesQuick start1. PrerequisiteExperiment FolderUsing Malet starts with making a folder with a single yaml config file.
Various files resulting from some experiment is saved in this single folder.
We advise to create a folder for each experiment underexperimentsfolder.experiments/└── {experiment folder}/├── exp_config.yaml:experiment config yaml file            (User created)├── log.tsv:log file for saving experiment results (generated by malet.experiment)├── (log_splits):folder for splitted logs               (generated by malet.experiment)└── figure:folder for figures                     (generated by malet.plot)Pre-existing training pipelineSay you have some training pipeline that takes in a configuration (any object w/ dictionary-like interface).
We require you to return the result of the training so it gets logged.deftrain(config,...):...# training happens here...metric_dict={'train_accuracies':train_accuracies,'val_accuracies':val_accuracies,'train_losses':train_losses,'val_losses':val_losses,}returnmetric_dict2. Running experimentsExperiment config yamlYou can configure as you would do in the yaml file.
But we provide useful special keywordgrid, used as follows:# static configsmodel:LeNet5dataset:mnistnum_epochs:100batch_size:128optimizer:adam# grided fieldsgrid:seed:[1,2,3]lr:[0.0001,0.001,0.01,0.1]weight_decay:[0.0,0.00005,0.0001]Specifying list of config values undergridlets you run all possible combination (i.e.grid) of your configurations, with field least frequently changing in the order of declaration ingrid.Running experimentsThe following will run thetrain_fnon grid of configs based on{exp_folder_path}andtrain_fn.fromfunctoolsimportpartialfrommalet.experimentimportExperimenttrain_fn=partial(train,...{otherargumentsbesidesconfig}..)metric_fields=['train_accuracies','val_accuracies','train_losses','val_losses']experiment=Experiment({exp_folder_path},train_fn,metric_fields)experiment.run()Note that you need to partially apply your original function so that you pass in a function with onlyconfigas its argument.Experiment logsThe experiment log will be automatically saved in the{exp_folder_path}aslog.tsv, where the static configs and the experiment log are eached saved in yaml and tsv like structure respectively.
You can retrieve these data in python usingExperimentLoginmalet.experimentas follows:frommalet.experimentimportExperimentLoglog=ExperimentLog.from_tsv({tsv_file})static_configs=log.static_configsdf=log.dfExperiment logs also enables resuming to the most recently run config when a job is suddenly killed.
Note that this only enable you to resume from the begining of the training.
For resuming from intermediate log checkpoints, check outSaving logs in intermediate epochs.3. Plot makingRunningmalet.plotlets you make plots based onlog.tsvin the experiment folder.malet-plot\-exp_folder../experiments/{exp_folder}\-modecurve-epoch-train_accuracyThe key intuition for using this is toleave only two field in the dataframe for the x-axis and the y-axisbyspecifying a specific value(e.g.model, dataset, optimizer, etc.),averaging over(seed),orchoose value with best metric(other hyperparameters),which will leave only one value for each field.
This can be done using the following arguments.Data related arguments-mode: Mode consists of mode of the plot (currently only has 'curve' and 'bar'), the field for x-axis, and the metric to use for y-axis.-mode{plot_mode}-{x_field}-{metric}Any other field exceptx_fieldandseed(always averaged over) is automatically chosen value with best metric.
To specify a value of a field, you can use the following-filterargument.-filter: Use to explicitly choose only certain subset values of some field.-filter'{field1} {v1} {v2} / {field2} {v3} {v4} ...'Here,epoch- fromexplodeing list-type metricmetric- frommelting different metrics column name into a new columnare automatically generated.-multi_line_fields: Specify the fields to plot multiple lines over.-multi_line_field'{field1} {field2} ...'-best_at_max(Default: False): Specify whether chosen metric is best when largest (e.g. accuracy).-best_at_max
-nobest_at_maxStyling arguments-colors: There are two color mode'','cont', where one uses rainbow like colors and the other uses more continuous colors.-colors'cont'-annotate: Option to add annotation based on field specified inannotate_fields.-annotate-annotate_fields: Field to annotate.-annotate_fields'{field1} {field2} ...'-fig_size: Figure size.Square figure-fig_size7Rectangular figure (x, y)-fig_size108-style: Matplotlib style.-style'ggplot'-plot_config: The path for a yaml file to configure all aspects the plot.-plot_config{plot_config_path}In this yaml, you can specify theline_styleandax_styleunder each mode as follows:'curve-epoch-train_accuracy':annotate:falsestd_plot:fillline_style:linewidth:4marker:'D'markersize:10ax_style:frame_width:2.5fig_size:7legend:[{'fontsize':20}]grid:[true,{'linestyle':'--'}]tick_params:-axis:bothwhich:majorlabelsize:25direction:inlength:5line_style: Style of the plotted line (linewidth,marker,markersize,markevery)ax_style: Style of the figure.Most attribute ofmatplotlib.axes.Axesobjectcan be set as follows:yscale:[parg1,parg2,{'kwarg1':v1,'kwarg2':v2}]is equivalent to runningax.set_yscale(parg1,parg2,kwarg1=v1,kwarg2=v2)For more details, go toAdvanced plot makingsection.Advanced topicsAdvanced gridding in yaml1. List comprehensionThis serves similar functionality as list comprehensions in python, and used as follows:lr:[10**{-i};1:1:5]Syntax:[{expression};{start}:{step}:{end}]where expression should be any python-interpretable using symbolsi, +, -, *, /, [], ()and numbers.
This is equivalent to python expression[{expression}foriinrange({start},{end},{step})]2. Grid sequencesWe can execute sequence of grids by passing in a list of dictionary instead of a dictionary undergridkeyword as followsgrid:-optimizer:sgdlr:[0.001,0.01]seed:[1,2,3]-optimizer:adamlr:[0.005]seed:[1,2,3]3. GroupingGrouping lets you group two different fields so it gets treated as a single field in the grid.grid:group:optimizer:[[sgd],[adam]]lr:[[0.001,0.01],[0.005]]seed:[1,2,3]Syntax:grid:group:cfg1:[A1,B1]cfg2:[A2,B2]cfg3:[1,2,3]is syntactically equivalent togrid:-cfg1:A1cfg2:A2cfg3:[1,2,3]-cfg1:B1cfg2:B2cfg3:[1,2,3]Here the two config fieldscfg1andcfg2has grouped values(A1, A2)and(B1, B2)that acts like a single config field and arn't gridded seperately (A1-2, B1-2are lists of values.)You can also create several groupings with list of dictionary undergroupkeyword as follows.grid:group:-cfg1:[A1,B1]cfg2:[A2,B2]-cfg3:[C1,D1]cfg4:[C2,D2]cfg5:[1,2,3]Advanced Plot making1. Other arguments formalet.plot-best_ref_x_fields: On defualt, each point inx_fieldget its own optimal hyperparameter set, which is sometimes undesirable.
This argument lets you specify on which value ofx_fieldto choose the best hyperparamter.-best_ref_x_field{x_field_value}-best_ref_ml_fields: Likewise, we might want to use the same hyperparameter for all lines inmulti_line_fieldwith best hyperparameter chosen from a single value inmulti_line_field.-best_ref_ml_field{ml_field_value}-best_ref_metric_field: To plot one metric with the hyperparameter set chosen based on another, pass the name of the metric of reference inmetric_field_value.-best_ref_metric_field{metric_field_value}2. Advanced yaml plot configMore details on ax_style keywordUnlike other fields,frame_width, fig_size, tick_params, legend, gridare not attributes ofAxesbut are enabled for convinience.
From these,frame_widthandfig_sizeshould be set as a number, while others can be similarly used like the rest of the attributes inAxes.Default styleYou can change the default plot style by adding thedefault_stylekeyword in the yaml file.'default_style':annotate:falsestd_plot:fillline_style:linewidth:4marker:'D'markersize:10ax_style:frame_width:2.5fig_size:7legend:[{'fontsize':20}]grid:[true,{'linestyle':'--'}]tick_params:-axis:bothwhich:majorlabelsize:25direction:inlength:5Mode aliasesYou can specify a set of arguments formalet.plotin the yaml file and give it an alias you can pass in tomodeargument.'sam_rho':mode:curve-rho-val-accuracymulti_line_field:optimizerfilter:'optimizersgdsam'annotate:Truecolors:''std_plot:barax_style:title:['SGDvsSAM',{'size':27}]xlabel:['$\rho$',{'size':30}]ylabel:['ValAccuracy(%)',{'size':30}]malet-plot\-exp_folder../experiments/{exp_folder}\-plot_config{plot_config_path}\-modesam_rhoWhen using mode aliases, the conflicting argument passed within the shell will be ignored.Style hierarchyIf conflicting style is passed in, we use the specifications given in the highest priority, given as the following:default_style<{customstyle}<{modealias}3. Custom dataframe processingThelegendand thetickare automatically determined based on the processed dataframe withindraw_metricfunction.
You can pass in a function to thepreprcs_dfkeyword argument indraw_metricwith the following arguments and return values:defpreprcs_df(df,legend):...# Process df and legend...returnprocessed_df,processed_legendWe advise to assign a new mode for eachpreprcs_df.4. Custom plotting usingavgbest_dfandax_drawinplot_utils.metric_drawerMuch of whatmalet.plotdoes comes fromavgbest_dfandax_draw.avgbest_df(df, metric_field, avg_over=None,  best_over=tuple(),  best_of=dict(), best_at_max=True)Paramters:df(pandas.DataFrame) : Base dataframe to operate over. All hyperparameters should be set asMultiIndex.metric_field(str) : Column name of the metric. Used to evaluate best hyperparameter.avg_over(str) :MultiIndexlevel name to average over.best_over(List[str]) : List ofMultiIndexlevel names to find value yielding best values ofmetric_field.best_of(Dict[str, Any]) : Dictionary of pair{MultiIndex name}: {value in MultiIndex}to find best hyperparameter of. The other values in{MultiIndex name}will follow the best hyperparamter found for these values.best_at_max(bool) :Truewhen larger metric is better, andFalseotherwise.Returns: Processed DataFrame (pandas.DataFrame)ax_draw(ax, df, label, annotate=True, std_plot='fill', unif_xticks=False, plot_config = {'linewidth':4, 'color':'orange', 'marker':'D', 'markersize':10, 'markevery':1})Paramters:ax(matplotlib.axes.Axes) : Axes to plot in.df(pandas.DataFrame) : Dataframe used for the plot.
This dataframe should have one named index for the x-axis and one column for the y-axis.label(str) : label for drawn line to be used in the legend.std_plot(Literal['none','fill','bar']) : Style of standard error drawn in to the plot.unif_xticks(bool) : WhenTrue, the xticks will be uniformly distanced regardless of its value.plot_config(Dict[str, Any]) : Dictionary of configs to use when plotting the line (e.g. linewidth, color, marker, markersize, markevery).Returns: Axes (matplotlib.axes.Axes) with single line added based ondf.Parallel friendly grid splittingWhen using GPU resource allocation programs such as Slurm, you might want to split multiple hyperparameter configurations over different GPU jobs in parallel.
We provide two methods of spliting the grid as arguments ofExperiment.
We advise to use flags to pass these as argument of yourtrain.pyfile.fromabslimportapp,flagsfrommalet.experimentimportExperiment...FLAGS=flags.FLAGSdefmain(argv):...experiment=Experiment({exp_folder_path},train_fn,metric_fields,total_splits=FLAGS.total_splits,curr_splits=FLAGS.curr_splits,auto_update_tsv=FLAGS.auto_update_tsv,configs_save=FLAGS.configs_save)experiment.run()if__name__=='__main__':flags.DEFINE_string('total_splits','1')flags.DEFINE_string('curr_splits','0')flags.DEFINE_bool('auto_update_tsv',False)flags.DEFINE_bool('configs_save',False)app.run(main)1. PartitioningUniform Partitioning (Pass in number)This method of splits the experiments uniformally given the following argumentsnumber of total partition (total_splits),batch index to allocate to this script (curr_splits).Each sbatch script needs to be using differentcurr_splitsnumbers (=0~total-1).splits=4echo"run sbatch slurm_train$n"for((i=0;i<n;i++))dopythontrain.py./experiments/{exp_folder}\--workdir=./logdir\--total_splits=splits\--curr_splits=$idoneField Partitioning (Pass in field name)This method of splits the experiments based on some field given in the following argumentsname of the field to split over (total_splits),string of field values seperated by ' ' to allocate to this current split script (curr_splits).Each sbatch script needs different field values (whitespace seperated strings for multiple values) incurr_splits.pythonexperiment_util.py./experiments/{exp_folder}\--total_splits'optimizer'\--curr_splits='sgd'pythonexperiment_util.py./experiments/{exp_folder}\--total_splits'optimizer'\--curr_splits='rmsprop adam'Both of these split methods result in multiple.tsvfiles, which is saved in{exp_folder}/log_splits/split_{i}.tsvfolder.Comments onauto_update_tsvargument.auto_update_tsvis used for 'Current run checking' stated in the next section, but using it in 'Partitioning' doesn't cause problems.
However we advise to not use it by adding since additional read/writing adds unnessacery computation time, especially as thelog.tsvfile grows larger.2. QueueingWith this method, each jobs, once finished running its config, runs the next config in the queue of the unrun configs.
More precisly, it skips any configs that finished running or are currently running.
The key to doing this isconfigs_save=True, which saves the configs to the{exp_folder}/log.tsvfile before a config is run, enabling other jobs to know what config is currently running and skip it.pythonexperiment_util.py./experiments/{exp_folder}--workdir=./logdir\--auto_update_tsv\--configs_saveThis method requires the keywordauto_update_tsv=TrueinExperimentto automatically read/write tsv files after a job starts/finishes running a config.One adventage of 'Queueing' over 'Partitioning' is that you can freely allocate/deallocate new GPUs while running an experiment.3. Use Both (Partitioning + Queueing)However aslog.tsvgrows larger, read/write time gets larger which cause various conflicts across different GPU jobs. One workaround is to use 'Partitioning' to split experiments to be saved in seperatelog_splits/split_{i}.tsvto keep the.tsvfiles small, while using 'Queueing' in each splits to freely allocate GPU jobs to leverage the advantages of both methods.splits=4echo"run sbatch slurm_train$n"for((i=0;i<n;i++))dopythonexperiment_util.py./experiments/{exp_folder}\--workdir=./logdir\--total_splits=splits\--curr_splits=$i\--auto_update_tsv\--configs_savedoneSaving logs in intermediate epochsWe checkpoint training state so that we can resume training in the event of an unexpected termination.
We can also checkpoint the experiment log so that we don't have to retrain a certain config to re-evaluate the metrics.Training pipelineFor this, we need to addexp_logargument intrainfunction for checkpointing the experiment log, where you can use it to add the following code for retrieveing/saving intermediate metric dictionary from/to thetsvfile.importosdefget_ckpt_dir(config):...returnckpt_dirdefget_ckpt(ckpt_dir):...returnckptdeftrain(config,experiment,...):...# set up# retrieve model/trainstate checkpoint if there exists# these are just placeholders for the logicckpt_epoch=0ckpt_dir=get_ckpt_dir(config)ifos.path.exists(ckpt_dir)ckpt=get_ckpt(ckpt_dir)ckpt_epoch=ckpt.epoch############# retrieve log checkpoint if there exists #############metric_dict={'train_accuracies':[],'val_accuracies':[],'train_losses':[],'val_losses':[],}ifconfiginexperiment.log:metric_dict=experiment.get_log_checkpoint(config,metric_dict)###################################################################...# training happens hereforepochinrange(config.ckpt_epoch,config.epochs):...# train...# update metric_dictifnotepoch%config.ckpt_every:...# train state, model checkpoint####################### checkpoint log #######################experiment.update_log(metric_dict,configs=config)##############################################################...returnmetric_dictTheExperimentLog.get_log_checkpointmethod retrieves themetric_dictbased on thestatusfield in the dataframe.statusDescriptionBehavior when resumedRCurrently runningGet skippedCCompletedGet skippedFFailed while runningRerun andmetric_dictis retrievedNote that with some external halt (e.g. computer shut down, slurm job cancellation), malet won't be able to log the status asF(failed).
In these cases, you need tomanually find the row in thelog.tsvfile corresponding to the halted job and change thestatusfromR(running) toF(falied).Running experimentfromfunctoolsimportpartialfrommalet.experimentimportExperimenttrain_fn=partial(train,...{otherargumentsbesidesconfig&exp_log}..)metric_fields=['train_accuracies','val_accuracies','train_losses','val_losses']experiment=Experiment({exp_folder_path},train_fn,metric_fields,checkpoint=True,auto_update_tsv=True)experiment.run()You should addcheckpoint=True, auto_update_tsv=Truewhen instanciatingExperiment.Merging multiple log filesThere are two methods for merging multiple log files.1. Merge all logs in a folderfrommalet.experimentimportExperimentLogExperimentLog.merge_folder({log_folder_path})2. Merge a portion of the logs in a folderfrommalet.experimentimportExperimentLognames=["log1","log2",...,"logn"]ExperimentLog.merge_tsv(names,{log_folder_path})Both methods automatically merges and saves aslog_merged.tsvin the folder.
These methods are helpful after running splitted experiments, where merging is required for using plot tools.