Package: l2metrics
Lifelong Learning Metrics (L2Metrics)Table of ContentsIntroductionMetricsData PreprocessingRequirementsInstallationUsageCommand-Line ExecutionStoring Single-Task Expert DataClearing Single-Task Expert DataGenerating a Metrics ReportCustom MetricsChangelogLicenseAcknowledgementsIntroductionLifelong Learning Metrics (L2Metrics) is a Python library containing foundational code for the L2M Metrics Framework. This framework includes the following:Python libraries for processing performance logs generated by lifelong learning algorithmsSupport for extending the framework with custom metricsMetricsThe L2Metrics library supports the following lifelong learning metrics as defined in theLifelong Learning Metrics for L2M specification:Performance Recovery (PR)Mean Evaluation Performance (MEP)Mean Training Performance (MTP)Performance Maintenance (PM)Forward Transfer (FT)Backward Transfer (BT)Performance Relative to a Single-Task Expert (RP)Sample Efficiency (SE)Data PreprocessingRefer to theData Processing READMEfor details on the data preprocessing methods in this library.RequirementsL2Metrics is written in Python 3 and it is highly recommended to use at least version Python 3.6. The Metrics Framework has been tested on Windows 10 and Ubuntu 18.04/20.04. It should work on other platforms but has not been verified.Installation1. (Optional) Create a Python virtual environmentpython-mvenv<path_to_new_venv>Activate the virtual environment as follows:Linux:source<path_to_new_venv>/bin/activateWindows:<path_to_new_venv>/Scripts/Activate.ps12. Update pip and wheel in your environmentpipinstall-Upipwheel3. Clone the L2Logger and L2Metrics repositoriesgitclonehttps://github.com/lifelong-learning-systems/l2logger.git
gitclonehttps://github.com/lifelong-learning-systems/l2metrics.git4. Install the L2Logger and L2Metrics packagespipinstall-e<path_to_l2logger>
pipinstall-e<path_to_l2metrics>UsageTo calculate metrics on the performance of your system, you must first generate log files in accordance with the L2Logger format version 1.1. Please refer to theL2Logger documentationfor more details on how to generate compatible logs.Once these logs are generated, you'll need to store Single-Task Expert (STE) data and pass the log directories as command-line arguments in order to compute STE-related metrics. Several example files are included to get you started:Example STE and LL log directories:./examples/ste_logs/ste_task1_1_run1/./examples/ste_logs/ste_task2_1_run1/./examples/ste_logs/ste_task3_1_run1/./examples/ste_logs/ste_task3_1_run2/./examples/ll_logs/multi_task/Examplesettings.jsonfile for configuring command-line argumentsExampledata_range.jsonfile to show how the user can specify task normalization rangesCommand-Line ExecutionRefer to theCommand-Line READMEfor more information on how to run L2Metrics from the command line.Storing Single-Task Expert DataThe following commands are examples of how to store STE data from the provided logs, run from the root L2Metrics directory:python-ml2metrics-lexamples/ste_task1_1_run1-sw
python-ml2metrics-lexamples/ste_task2_1_run1-sw
python-ml2metrics-lexamples/ste_task3_1_run1-sw
python-ml2metrics-lexamples/ste_task3_1_run2-saThe specified log data will be stored in the$L2DATAdirectory under thetaskinfosubdirectory, where all single-task expert data is pickled and saved. The STE store mode specified in the first three example commands isw, which is "write" or "overwrite." This mode will create a new pickle file for the STE if one does not already exist; if there is already a file for the same task in thetaskinfolocation, it will be overwritten in this mode. The last example command used the append mode,a, which allows users to store multiple runs of STE data in the same pickle file. Then, the STE averaging method can be selected in thel2metricsmodule to modify how multiple STE runs are handled. Storing STE data assumes the provided log only contains data for a single task/variant.Replace the log directory argument with logs for other STE tasks and repeat until all STE data is stored.Clearing Single-Task Expert DataIf the user would like to clear thetaskinfosubdirectory of all previously-stored STE data,
run the following command:python-ml2metrics.clear_steGenerating Metrics ReportTo generate a metrics plot and report with default settings, run the following command from thel2metrics/examplesdirectory:python-ml2metrics-l./ll_logs/multi_task-pperformanceThe default output files are saved in the current working directory underresults/and defined below:multi_task_data.feather: The log data DataFrame containing raw and preprocessed data.multi_task_metrics.json: The lifetime and task-level metrics of the run.multi_task_settings.json: The settings used to generate the metrics report.multi_task_regime.tsv: The regime-level metrics of the run.plots/multi_task_evaluation.png: Evaluation blockpoint plotsgrouped by task labels (i.e., task variants appear on the same subplot).plots/multi_task_learning.png: Plot showing smoothed, normalized learning curves for the lifetime.plots/multi_task_raw.png: Plot showing raw training reward values with smoothed curve overlaid.plots/multi_task_ste.png: Plot showing concatenated learning curves compared to stored STE runs.If you wish to generate a metrics report with modified settings (e.g., disabling normalization or aggregating lifetime metrics with the mean operator), you can either modify the arguments on the command line or specify a JSON file containing the desired settings. The settings loaded from the JSON file will take precedence over any arguments specified on the command line.python-ml2metrics-csettings.jsonLastly, if you wish to compute metrics on multiple lifetimes at once, assert the recursive flag on the command line. When the recursive flag is set, L2Metrics will scan the subdirectories for valid LL logs, calculate metrics, then save out a TSV and JSON file containing lifetime/task-level metrics for each discovered lifetime.python-ml2metrics-l<path/to/directory/containing/multiple/runs>-RNote: If you do not wish to provide a fully qualified path to your log directory, you may copy it to your$L2DATA/logsdirectory. This is the default location for logs generated using the TEF.Log DataRefer to theLog Data READMEfor more information on how to interface with the raw and preprocessed log data from the scenario.Output Settings FileIf saving of L2Metrics settings is enabled, the framework will generate a JSON file containing the primary parameters used to calculate L2Metrics:{"log_dir":"ll_logs\\multi_task","perf_measure":"performance","variant_mode":"aware","ste_averaging_method":"metrics","aggregation_method":"mean","maintenance_method":"mrlep","transfer_method":"ratio","normalization_method":"task","smoothing_method":"flat","window_length":null,"clamp_outliers":false}Metrics and Metrics FileThe metrics module will print the lifetime metrics to the console when it has successfully completed execution. The following table shows an example of a metrics report output:perf_recoveryavg_train_perfavg_eval_perfperf_maintenance_mrlepforward_transfer_ratiobackward_transfer_ratioste_rel_perfsample_efficiency-2.0083.8278.523.8612.631.081.110.91If saving is enabled, the framework will also generate a JSON file containing lifetime and task-level metrics for the scenario. Please refer to theFile Description READMEfor more information on the format of this file.Evaluation PlotThe resulting evaluation plot from example run should look like this:This figure shows point plots for all tasks in the lifetime grouped by task label (i.e., task variants are shown on the same subplot). The points are the mean values in the evaluation blocks and the lines extending from each point show the 95% confidence intervals. The x-axis represents the block number in the lifetime.Performance PlotThe resulting learning plot from example run should look like this:This figure shows the pre-processed (smoothed, normalized, clamped, etc.) learning curves across the lifetime. The dashed lines in the plot show the slopes between each task's evaluation blocks.Raw Performance PlotThe resulting raw performance plot from example run should look like this:This figure shows the raw reward values from learning blocks with the smoothed curves overlaid in black. The values in this figure are not normalized, even if that option is enabled.Performance Relative to STE PlotThe framework should also produce a performance relative to STE plot shown below, where the task performance curves are generated by concatenating all the training data from the scenario:The vertical black dashed lines indicate the block boundaries where task performance was stitched together.Custom MetricsSee documentation in the examples folder atexamples/README.mdfor more details on how to implement custom metrics.ChangelogSeeCHANGELOG.mdfor a list of notable changes to the project.LicenseSeeLICENSEfor license information.AcknowledgementsPrimary development of Lifelong Learning Metrics (L2Metrics) was funded by the DARPA Lifelong Learning Machines (L2M) Program.Â© 2021-2022 The Johns Hopkins University Applied Physics Laboratory LLC