Repository: schuderer/mllaunchpad

Issue:
Improve logging and monitoring facilities (ml-ops)
There is a need for more detailed documentation of created models.

Ideally, in a way that is easily customizable by the user.

Current suggestion: extend the model metadata with the required metrics and information.

![image](https://user-images.githubusercontent.com/4430527/118503862-49871b80-b72b-11eb-9396-2ba8a129d373.png)

To this end, we will add the two sub-dicts `system` and `train_report` to the metadata. Three mechanisms will be used:

1. More hard-coded info will be added the metadata (Let's keep these as few as possible. See below for details)
2. All keys of the `model` config dict will be added to the metadata (so one can add "description", links to docs, hyper params etc. there and they all will be logged)
3. From code, one can call `mllaunchpad.report("my key name", "my value")`, which causes a key-value pair to be added to subdict `train_report`. This should work inside `create_trained_model` only (in `test_trained_model`, one can simply return the metrics that should be logged).

Elaborating on point 1 above, the things to hard-code are:
- add the `repr()` of the inner model to `train_report`
- add mllp version to `train_report` (and `system`?)
- add IP/name of training machine as `host` to subdict `system`
- add platform to `system`
- add python version to `system`
- add pip freeze result to `system`

Also sensible to provide some API functionality to access the metadata.

Discussion:

Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 61d1150..14bd34d 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -28,7 +28,7 @@ and this project adheres to `Semantic Versioning <https://semver.org/spec/v2.0.0
 Unreleased
 ------------------------------------------------------------------------------
 
-* |Fixed| require ``flask<2.0.0`` as temporary workaround for `flask-restful issue #915 <https://github.com/flask-restful/flask-restful/pull/915>`_,
+* |Feature| Add training metadata reporting and querying `issue #142 <https://github.com/schuderer/mllaunchpad/issues/142>`_,
   by `Andreas Schuderer <https://github.com/schuderer>`_.
 * |Feature| Add support for typed CSVs (option ``dtypes_path`` of
   :class:`FileDataSource <mllaunchpad.datasources.FileDataSource>` and
diff --git a/docs/about.rst b/docs/about.rst
index 5491762..5c50167 100644
--- a/docs/about.rst
+++ b/docs/about.rst
@@ -134,6 +134,10 @@ Besides through the command line, the main functionality is also made
 available through a Python convenience API.
 See :meth:`mllaunchpad.train_model`,
 :meth:`mllaunchpad.retest` and :meth:`mllaunchpad.predict`.
+To log additional data to the ``model_store`` when training,
+use :meth:`mllaunchpad.report` in your train and test code.
+To get all information on models available in the the model
+store, use :meth:`mllaunchpad.list_models`.
 
 .. _extending:
 
@@ -212,7 +216,8 @@ specialised on providing the missing features.
 
 Here's a by no means ideal, but relatively simple
 example scenario for an ML Launchpad production
-deployment of one model in a low to medium load setting:
+deployment of one model in a low to medium load setting
+(available as a template `here <https://github.com/schuderer/mllaunchpad-template>`_):
 
 * Deployment on a single Linux VM instance (for now),
   with its own user to run as,
diff --git a/docs/config.rst b/docs/config.rst
index c7a6f7e..cb7618d 100644
--- a/docs/config.rst
+++ b/docs/config.rst
@@ -13,7 +13,7 @@ implementation, to the things on the "outside", such as the data connection
 algorithm by using different configuration files for different purposes of your
 development life cycle. That way, you can cleanly separate different environments
 like development/testing/production, without having to touch your code (using the
-same build) when between these environments.
+same build) when switching between these environments.
 
 For ML Launchpad to know how to do its job, it *always* needs a configuration.
 To accommodate different ways of using ML Launchpad, you have different options of
diff --git a/docs/index.rst b/docs/index.rst
index 6e6f6d8..57ca97a 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,9 +1,12 @@
 ML Launchpad documentation
 ==============================================================================
-ML Launchpad is an extensible Python package which makes it easy to publish Machine Learning models as RESTful APIs.
+ML Launchpad is an extensible Python package which makes it easy to publish Machine Learning
+models as RESTful APIs.
 
 It separates the Machine Learning side (how to train, test and to predict) from the environment-specific aspects (local or cloud deployment of the API, database connections, format validation, model storage, etc.).
 
+ML Launchpad is open source and free to use by anyone, including commercially (:doc:`Apache 2.0 License <license>`).
+
 Contents:
 
 .. toctree::
@@ -20,6 +23,7 @@ Contents:
    Code of Conduct <conduct>
    authors
    changelog
+   license
 
 Indices and tables
 ==================
diff --git a/docs/license.rst b/docs/license.rst
new file mode 100644
index 0000000..61bf9c0
--- /dev/null
+++ b/docs/license.rst
@@ -0,0 +1,4 @@
+License
+==============================================================================
+.. include:: ../LICENSE
+   :literal:
diff --git a/docs/usage.rst b/docs/usage.rst
index 21e8872..af83d64 100644
--- a/docs/usage.rst
+++ b/docs/usage.rst
@@ -16,7 +16,7 @@ or Waitress) is used to run ``mllaunchpad.wsgi:application`` instead
 (the config file is then provided via an environment variable).
 
 All commands (``train``, ``retest``, ``predict``, ``api`` and ``generate-raml``) can
-be abbreviated, so you can use ``mllaunchpad t`` or ``mllaunchpad pred`` to save
+be abbreviated, so you can use e.g. ``mllaunchpad t`` or ``mllaunchpad pred`` to save
 some keystrokes.
 
 .. click:: mllaunchpad.cli:main
@@ -262,6 +262,12 @@ The three methods return the same things as our own functions:
   returns a prediction (usually a ``dict``, but
   can also contain ``lists``, numpy arrays or pandas DataFrames).
 
+**Sidenote**: To save additional information while training for traceability's sake,
+use :meth:`mllaunchpad.report` in your train and test code.
+The metadata thus saved resides in the model store together with the model. By default, it includes basic
+info such as the configuration (see below), some system info, and the test metrics.
+When done with training, you can retrieve metadata of all models in the model
+store from Python by using :meth:`mllaunchpad.list_models`.
 
 Next, we will configure some extra info about our model,
 as well as tell ML Launchpad where to find
diff --git a/examples/tree_model.py b/examples/tree_model.py
index ad75c4a..ecec6f6 100644
--- a/examples/tree_model.py
+++ b/examples/tree_model.py
@@ -1,7 +1,7 @@
 import logging
 
 # Third-party imports
-from mllaunchpad import ModelInterface, ModelMakerInterface, order_columns
+from mllaunchpad import ModelInterface, ModelMakerInterface, order_columns, report
 import pandas as pd
 from sklearn import tree
 from sklearn.metrics import accuracy_score, confusion_matrix
@@ -42,6 +42,10 @@ class MyExampleModelMaker(ModelMakerInterface):
         my_tree = tree.DecisionTreeClassifier()
         my_tree.fit(X_ordered, y)
 
+        # We can add info (numbers, strings, dicts, lists) to be saved in the training report (JSON metadata):
+        report("meaning_of_life", 42)
+        report("train_data_used", df)  # If we pass dataframes, their summary will be added to the training report
+
         return my_tree
 
     def test_trained_model(self, model_conf, data_sources, data_sinks, model):
@@ -49,6 +53,9 @@ class MyExampleModelMaker(ModelMakerInterface):
         X_test = order_columns(df.drop("variety", axis=1))
         y_test = df["variety"]
 
+        report("test_data_used", df)
+        report("awesomest_number", 300000)
+
         my_tree = model
 
         y_predict = my_tree.predict(X_test)
diff --git a/mllaunchpad/__init__.py b/mllaunchpad/__init__.py
index c109959..4750250 100644
--- a/mllaunchpad/__init__.py
+++ b/mllaunchpad/__init__.py
@@ -1,10 +1,21 @@
 """Top-level package for ML Launchpad."""
 
+# Stdlib imports
+from typing import Dict, Union
+
 # Third-party imports
 import pkg_resources
 
 # Project imports
+from mllaunchpad import (
+    api,
+    datasources,
+    model_actions,
+    model_interface,
+    resource,
+)
 from mllaunchpad.config import get_validated_config, get_validated_config_str
+from mllaunchpad.model_actions import _add_to_train_report as report
 from mllaunchpad.model_actions import predict, retest, train_model
 from mllaunchpad.model_interface import ModelInterface, ModelMakerInterface
 from mllaunchpad.resource import order_columns
@@ -13,7 +24,46 @@ from mllaunchpad.resource import order_columns
 __version__ = pkg_resources.get_distribution("mllaunchpad").version
 
 
-# Get rid of stupid pyflakes warning (be warned: redundancy warning):
+def list_models(model_store_location_or_config_dict: Union[Dict, str]):
+    """Get information on all available versions of trained models.
+
+    :param model_store_location_or_config_dict: Location of the model store. If you have a config dict available, use that instead.
+    :type model_store_location_or_config_dict: Union[Dict, str]
+
+    Side note: The return value includes backups of models that have been re-trained without changing
+    the version number (they reside in the subdirectory ``previous``).
+    Please note that these backed up models are just listed for information and are not available
+    for loading (one would have to restore them by moving them up a directory level from ``previous``.
+
+    Example::
+
+        import mllaunchpad as mllp
+        my_cfg = mllp.get_validated_config("./my_config_file.yml")
+        all_models = mllp.list_models(my_cfg)  # also accepts model store location string
+
+        # An example of what a ``list_models()``'s result would look like:
+        # {
+        #     iris: {
+        #         1.0.0: { ... complete metadata of this version number ... },
+        #         1.1.0: { ... },
+        #         latest: { ... duplicate of metadata of highest available version number, here 1.1.0 ... },
+        #         backups: [ {...}, {...}, ... ]
+        #     },
+        #     my_other_model: {
+        #         1.0.1: { ... },
+        #         2.0.0: { ... },
+        #         latest: { ... },
+        #         backups: []
+        #     }
+        # }
+
+    :returns: Dict with information on all available trained models.
+    """
+    ms = resource.ModelStore(model_store_location_or_config_dict)
+    return ms.list_models()
+
+
+# Make it possible for Sphinx to find the relevant imports
 __all__ = [
     "train_model",
     "retest",
@@ -23,4 +73,6 @@ __all__ = [
     "ModelInterface",
     "ModelMakerInterface",
     "order_columns",
+    "report",
+    "list_models",
 ]
diff --git a/mllaunchpad/api.py b/mllaunchpad/api.py
index 49ab068..c9360b3 100644
--- a/mllaunchpad/api.py
+++ b/mllaunchpad/api.py
@@ -23,6 +23,12 @@ logger = logging.getLogger(__name__)
 # TODO: factor out datasource/model preparation code -- use model_actions instead of low-level functionality
 
 
+def get_api_base_url(config):
+    api_name = config["api"]["name"]
+    api_version = _get_major_api_version(config)
+    return "/{}/{}".format(api_name, api_version)
+
+
 def _get_major_api_version(config):
     match = re.match(r"\d+", config["model"]["version"])
     if match is None:
@@ -267,9 +273,7 @@ class ModelApi:
         logger.debug("Initializing RESTful API")
         api = Api(application)
 
-        api_name = config["api"]["name"]
-        api_version = _get_major_api_version(config)
-        api_url = "/{}/{}".format(api_name, api_version)
+        api_url = get_api_base_url(config)
 
         raml = _load_raml(config)
         res_normal, res_with_id, res_file = _get_resources(raml)
diff --git a/mllaunchpad/model_actions.py b/mllaunchpad/model_actions.py
index 092e419..37dce13 100644
--- a/mllaunchpad/model_actions.py
+++ b/mllaunchpad/model_actions.py
@@ -3,7 +3,8 @@
 # Stdlib imports
 import logging
 import sys
-from typing import Dict, Optional
+from contextlib import contextmanager
+from typing import Any, Dict, Iterator, Optional
 
 # Project imports
 from mllaunchpad import resource
@@ -18,6 +19,7 @@ _cached_model_tuples: Dict = {}
 _cached_data_source_sink_tuples: Dict = {}
 _cached_model_makers: Dict = {}
 _cached_model_classes: Dict = {}
+_current_train_report: Optional[Dict[str, Any]] = None
 
 suppress_order_columns_not_used_warning = (
     "To adjust this warning, set model config's `order_columns_not_used_warning` "
@@ -80,45 +82,52 @@ def train_model(
         complete_conf, tags=["train"], cache=cache
     )
 
-    inner_model = user_mm.create_trained_model(
-        model_conf, dso_train, dsi_train, old_model=old_inner_model
-    )
-    m_cls = _get_model_class(complete_conf, cache=cache)
-    model_wrapper: ModelInterface = m_cls(contents=inner_model)
-
-    if resource._order_columns_called:
-        model_wrapper.have_columns_been_ordered = True
-    elif (
-        "order_columns_not_used_warning" not in model_conf
-        or model_conf["order_columns_not_used_warning"] == "always"
-    ):
-        logger.warning(
-            "Training code does not call function order_columns. "
-            + suppress_order_columns_not_used_warning
+    report_dict: Dict[str, Any]
+    with train_report() as report_dict:
+        inner_model = user_mm.create_trained_model(
+            model_conf, dso_train, dsi_train, old_model=old_inner_model
         )
+        m_cls = _get_model_class(complete_conf, cache=cache)
+        model_wrapper: ModelInterface = m_cls(contents=inner_model)
+
+        if resource._order_columns_called:
+            model_wrapper.have_columns_been_ordered = True
+        elif (
+            "order_columns_not_used_warning" not in model_conf
+            or model_conf["order_columns_not_used_warning"] == "always"
+        ):
+            logger.warning(
+                "Training code does not call function order_columns. "
+                + suppress_order_columns_not_used_warning
+            )
 
-    if not isinstance(model_wrapper, ModelInterface):
-        logger.warning(
-            "Model's class is not a subclass of ModelInterface: %s",
-            model_wrapper,
-        )
+        if not isinstance(model_wrapper, ModelInterface):
+            logger.warning(
+                "Model's class is not a subclass of ModelInterface: %s",
+                model_wrapper,
+            )
 
-    metrics = {}
-    if test:
-        logger.debug("Testing trained model...")
-        dso_test, dsi_test = _get_data_sources_and_sinks(
-            complete_conf, tags=["test"], cache=cache
-        )
-        metrics = user_mm.test_trained_model(
-            model_conf, dso_test, dsi_test, inner_model
-        )
-        _check_ordered_columns(
-            complete_conf, model_wrapper, "testing code", times=2
-        )
+        metrics = {}
+        if test:
+            logger.debug("Testing trained model...")
+            dso_test, dsi_test = _get_data_sources_and_sinks(
+                complete_conf, tags=["test"], cache=cache
+            )
+            metrics = user_mm.test_trained_model(
+                model_conf, dso_test, dsi_test, inner_model
+            )
+            _check_ordered_columns(
+                complete_conf, model_wrapper, "testing code", times=2
+            )
 
-    if persist:
-        model_store = _get_model_store(complete_conf, cache=cache)
-        model_store.dump_trained_model(complete_conf, model_wrapper, metrics)
+        if persist:
+            model_store = _get_model_store(complete_conf, cache=cache)
+            for name, val in report_dict.items():
+                model_store.add_to_train_report(name, val)
+            model_store.add_to_train_report("algorithm", repr(inner_model))
+            model_store.dump_trained_model(
+                complete_conf, model_wrapper, metrics
+            )
 
     logger.info(
         "Created%s trained model %s, version %s, metrics %s",
@@ -418,7 +427,10 @@ def _get_data_sources_and_sinks(complete_conf, tags=None, cache=True):
 def _check_ordered_columns(complete_conf, model_wrapper, what: str, times=1):
     if (
         "order_columns_not_used_warning" in complete_conf["model"]
-        and complete_conf["model"]["order_columns_not_used_warning"] == "never"
+        and str(
+            complete_conf["model"]["order_columns_not_used_warning"]
+        ).lower()
+        == "never"
     ):
         return
     if model_wrapper.have_columns_been_ordered:
@@ -429,3 +441,67 @@ def _check_ordered_columns(complete_conf, model_wrapper, what: str, times=1):
                     what, suppress_order_columns_not_used_warning
                 )
             )
+
+
+@contextmanager
+def train_report() -> Iterator[Dict[str, Any]]:
+    # initialize
+    global _current_train_report
+    if _current_train_report is not None:
+        raise RuntimeError(
+            "Cannot initialize a train_report context within another train_report context."
+        )
+    _current_train_report = {}
+    try:
+        yield _current_train_report
+    finally:
+        # release
+        _current_train_report = None
+
+
+def _add_to_train_report(name: str, value) -> None:
+    """Add a piece of information to the train report during training.
+
+    The train report is part of the model's metadata that is saved to the model store.
+    Use :func:`mllaunchpad.list_models` to query metadata from the model store.
+
+    This function is supposed to be called from your :func:`~mllaunchpad.ModelMakerInterface.create_trained_model` or
+    :func:`~mllaunchpad.ModelMakerInterface.test_trained_model` implementation. You can pass any values that are
+    JSON-able, same as with :func:`~mllaunchpad.ModelMakerInterface.test_trained_model`'s
+    returned metrics.
+
+    However, if the value is a DataFrame, it will be summarized (using `pd.describe()`). You can use this for example
+    to improve traceability of your trained models and for some basic sanity checks of training data distribution.
+
+    :param name: Key to save the information under (e.g. "meaning_of_life")
+    :type name: str
+    :param value: Value to save. Any JSON-able value or structure will work. Pandas DataFrames will be summarized instead of saved.
+    :type value: str, number, list, dict, Numpy Array or Pandas DataFrame
+    """
+    global _current_train_report
+    if _current_train_report is None:
+        logger.info(
+            'Ignoring attempt to add record "{}" to train report '
+            "(hint: 'mllaunchpad.report()' does nothing when re-testing).".format(
+                name
+            )
+        )
+    else:
+        import pandas as pd
+
+        if isinstance(value, pd.DataFrame):
+            data_dict = _current_train_report.get("data", {})
+            data_dict[name] = {
+                "nrows": value.shape[0],
+                "ncols": value.shape[1],
+                "colnames": list(value.columns),
+                "dtypes": [str(dt) for dt in value.dtypes],
+                "description": value.describe(),
+            }
+            logger.info(
+                "Train report data: {}={}".format(name, data_dict[name])
+            )
+            _current_train_report["data"] = data_dict
+        else:
+            logger.info("Train report: {}={}".format(name, value))
+            _current_train_report[name] = value
diff --git a/mllaunchpad/resource.py b/mllaunchpad/resource.py
index 723a4a7..52d5956 100644
--- a/mllaunchpad/resource.py
+++ b/mllaunchpad/resource.py
@@ -5,7 +5,10 @@ import glob
 import json
 import logging
 import os
+import platform
 import shutil
+import socket
+import subprocess  # nosec We are running a known process using its full path (python -m pip).
 import sys
 from collections import OrderedDict
 from datetime import datetime
@@ -30,6 +33,9 @@ import dill as pickle  # nosec
 import numpy as np
 import pandas as pd
 
+# Project imports
+import mllaunchpad as mllp
+
 
 DS = TypeVar("DS", "DataSource", "DataSink")
 Raw = Union[str, bytes]
@@ -49,13 +55,19 @@ class ModelStore:
     best metrics which serves a particular API.'
     """
 
-    def __init__(self, config):
+    def __init__(self, config: Union[Dict, str]):
         """Get a model store based on the config settings
 
-        Params:
-            config: configuration dict
+        :param config: configuration dict containing the model store location, or model store location as string
+        :type complete_conf:  Union[Dict, str]
         """
-        self.location = config["model_store"]["location"]
+        if isinstance(config, dict):
+            self.location = config["model_store"]["location"]
+        else:
+            self.location = str(config)
+        self.train_report: Dict[str, Any] = {}
+
+    def _ensure_location(self):
         if not os.path.exists(self.location):
             os.makedirs(self.location)
 
@@ -114,6 +126,7 @@ class ModelStore:
         """
         model_conf = complete_conf["model"]
         base_name = self._get_model_base_name(model_conf)
+        self._ensure_location()
 
         # Check if exists and backup if it does
         self._backup_old_model(base_name)
@@ -129,12 +142,28 @@ class ModelStore:
             "version": model_conf["version"],
             "created": datetime.now().strftime(DATE_FORMAT),
             "created_by": getpass.getuser(),
+            "system": {
+                "mllaunchpad_version": mllp.__version__,
+                "platform": platform.platform(),
+                "host": socket.getfqdn(),
+                "host_ip": socket.gethostbyname(socket.getfqdn()),
+                "python": sys.version,
+                "packages": subprocess.getoutput(
+                    "{} -m pip freeze".format(sys.executable)
+                ).splitlines(),
+            },
+            "train_report": self.train_report,
             "metrics": metrics,
             "metrics_history": {datetime.now().strftime(DATE_FORMAT): metrics},
-            "config_snapshot": model_conf,
+            "config_snapshot": complete_conf,
         }
         if "api" in complete_conf:  # API is optional
-            meta["api_name"] = complete_conf["api"]["name"]
+            meta["api_base_url"] = mllp.api.get_api_base_url(complete_conf)
+
+        # Add all extra and non-colliding keys in model: section
+        for key, val in model_conf.items():
+            if key not in meta and key != "module":
+                meta[key] = val
 
         self._dump_metadata(base_name, meta)
 
@@ -172,6 +201,78 @@ class ModelStore:
         meta["metrics_history"][datetime.now().strftime(DATE_FORMAT)] = metrics
         self._dump_metadata(base_name, meta)
 
+    def add_to_train_report(self, name: str, value):
+        plain_val = to_plain_python_obj(value)
+        self.train_report[name] = plain_val
+
+    def _get_infos_from_json_file(self, file):
+        base_name, _ = os.path.splitext(file)
+        fn = os.path.basename(base_name)
+        fn_parts = fn.split("_")  # there can be more than two
+        name = fn_parts[0]
+        version = fn_parts[1]
+        meta = self._load_metadata(base_name)
+        return name, version, meta
+
+    def list_models(self):
+        """Get information on all available versions of trained models.
+
+        Side note: This also includes backups of models that have been re-trained without changing
+        the version number (they reside in the subdirectory ``previous``).
+        Please note that these backed up models are just listed for information and are not available
+        for loading (one would have to restore them by moving them up a directory level from ``previous``.
+
+        Example::
+
+            ms = mllp.ModelStore("./model_store")
+            all_models = ms.list_models()
+
+            # An example of what a ``list_models()``'s result would look like:
+            {
+                iris: {
+                    1.0.0: { ... complete metadata of this version number ... },
+                    1.1.0: { ... },
+                    latest: { ... duplicate of metadata of highest available version number, here 1.1.0 ... },
+                    backups: [ {...}, {...}, ... ]
+                },
+                my_other_model: {
+                    1.0.1: { ... },
+                    2.0.0: { ... },
+                    latest: { ... },
+                    backups: []
+                }
+            }
+
+        :returns: Dict with information on all available trained models.
+        """
+
+        result = {}
+        store_pattern = os.path.join(self.location, "*.json")
+        for file in glob.glob(store_pattern):
+            name, version, meta = self._get_infos_from_json_file(file)
+            result[name] = result.get(name, {})
+            result[name][version] = meta
+
+        for name, versions in result.items():
+            latest_version_key = max(versions)
+            versions["latest"] = versions[latest_version_key]
+            result[name]["backups"] = []
+
+        backup_dir = os.path.join(self.location, "previous")
+        backup_pattern = os.path.join(backup_dir, "*.json")
+        for file in glob.glob(backup_pattern):
+            name, version, meta = self._get_infos_from_json_file(file)
+            if name in result:
+                result[name]["backups"].append(meta)
+            else:
+                logger.debug(
+                    "Ignoring backup for obsolete model %s_%s while collecting model directory",
+                    name,
+                    version,
+                )
+
+        return result
+
 
 def _tags_match(tags, other_tags) -> bool:
     tags = tags or []
diff --git a/setup.cfg b/setup.cfg
index eedc35d..e984e37 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -5,6 +5,7 @@ ignore_missing_imports = True
 
 [flake8]
 exclude = docs
+per-file-ignores = __init__.py:F401
 # Ignore black styles.
 ignore = E501, W503
 
@@ -21,7 +22,7 @@ import_heading_stdlib=Stdlib imports
 import_heading_thirdparty=Third-party imports
 import_heading_firstparty=Project imports
 known_first_party=mllaunchpad
-known_third_party=click,dill,flask,flask_restful,nox,numpy,pandas,pkg_resources,pyspark,pytest,ramlfications,rpy2,setuptools,sklearn,sqlalchemy,werkzeug,yaml
+known_third_party=click,dill,flask,flask_restful,nox,numpy,pandas,pkg_resources,pytest,ramlfications,rpy2,setuptools,sklearn,werkzeug,yaml
 # known_third_party=pandas,numpy,flask,flask-restful,ramlfications,werkzeug,sklearn,dill,pyyaml,pytest
 # known_third_party=pytest  # rest is installed so isort can detect it
 
@@ -67,7 +68,7 @@ setup_requires = setuptools >= 38.3.0
 packages = find:
 python_requires = >= 3.6
 install_requires =
-  flask<2.0.0  # Temporary workaround until release of https://github.com/flask-restful/flask-restful/pull/915
+  flask
   flask-restful
   werkzeug
   ramlfications


Test patch:
diff --git a/tests/test_model_actions.py b/tests/test_model_actions.py
index 9926775..2dc5f28 100644
--- a/tests/test_model_actions.py
+++ b/tests/test_model_actions.py
@@ -358,20 +358,53 @@ def test__get_model_caching(ms_class, config):
 def test__check_ordered_columns(caplog):
     from mllaunchpad.resource import order_columns
 
-    dummy_config = {"model": {}}
-    mock_wrapper = MockModelClass()
+    with caplog.at_level(logging.INFO):
+        dummy_config = {"model": {}}
+        mock_wrapper = MockModelClass()
 
-    ma._check_ordered_columns(dummy_config, mock_wrapper, "never_ordered")
-    assert "never_ordered".lower() not in caplog.text.lower()
+        ma._check_ordered_columns(dummy_config, mock_wrapper, "never_ordered")
+        assert "never_ordered".lower() not in caplog.text.lower()
 
-    mock_wrapper.have_columns_been_ordered = True
-    ma._check_ordered_columns(
-        dummy_config, mock_wrapper, "ordered_only_in_train"
-    )
-    assert "ordered_only_in_train does not call".lower() in caplog.text.lower()
+        mock_wrapper.have_columns_been_ordered = True
+        ma._check_ordered_columns(
+            dummy_config, mock_wrapper, "ordered_only_in_train"
+        )
+        assert (
+            "ordered_only_in_train does not call".lower()
+            in caplog.text.lower()
+        )
 
-    order_columns({"a": 1})
-    ma._check_ordered_columns(
-        dummy_config, mock_wrapper, "ordered_in_train_and_now"
-    )
-    assert "ordered_in_train_and_now".lower() not in caplog.text.lower()
+        order_columns({"a": 1})
+        ma._check_ordered_columns(
+            dummy_config, mock_wrapper, "ordered_in_train_and_now"
+        )
+        assert "ordered_in_train_and_now".lower() not in caplog.text.lower()
+
+
+def test_train_report_context(caplog):
+    with caplog.at_level(logging.INFO):
+        with ma.train_report() as tr:
+            ma._add_to_train_report("myname", "myvalue")
+            assert ma._current_train_report is tr
+
+        assert tr["myname"] == "myvalue"
+        assert "{}={}".format("myname", "myvalue") in caplog.text
+
+
+def test_train_report_uninitialized(caplog):
+    with caplog.at_level(logging.INFO):
+        ma._add_to_train_report("myname", "myvalue")
+        assert "Ignoring" in caplog.text
+
+        with ma.train_report():
+            pass
+
+        ma._add_to_train_report("myname", "myvalue")
+        assert "{}={}".format("myname", "myvalue") not in caplog.text
+
+
+def test_train_report_reentrant():
+    with ma.train_report():
+        with pytest.raises(RuntimeError):
+            with ma.train_report():
+                pass
diff --git a/tests/test_resource.py b/tests/test_resource.py
index d869304..345426a 100644
--- a/tests/test_resource.py
+++ b/tests/test_resource.py
@@ -2,8 +2,10 @@
 
 # Stdlib imports
 import json
+import logging
 import os
 from collections import OrderedDict
+from random import random
 from unittest import mock
 
 # Third-party imports
@@ -41,7 +43,8 @@ def modelstore_config():
 @mock.patch("{}.os.path.exists".format(r.__name__), return_value=False)
 @mock.patch("{}.os.makedirs".format(r.__name__))
 def test_modelstore_create(makedirs, path_exists, modelstore_config):
-    _ = r.ModelStore(modelstore_config)
+    ms = r.ModelStore(modelstore_config)
+    ms._ensure_location()
     makedirs.assert_called_once_with(
         modelstore_config["model_store"]["location"]
     )
@@ -49,7 +52,8 @@ def test_modelstore_create(makedirs, path_exists, modelstore_config):
     path_exists.reset_mock()
     makedirs.reset_mock()
     path_exists.return_value = True
-    _ = r.ModelStore(modelstore_config)
+    ms = r.ModelStore(modelstore_config)
+    ms._ensure_location()
     assert not makedirs.called
 
 
@@ -64,7 +68,9 @@ def test_modelstore_dump(glob, copy, makedirs, path_exists, modelstore_config):
         "{}.open".format(r.__name__), mock.mock_open(), create=True
     ) as mo:
         ms = r.ModelStore(modelstore_config)
-        ms.dump_trained_model(modelstore_config, {"hi": 1}, {"there": 2})
+        ms.dump_trained_model(
+            modelstore_config, {"pseudo_model": 1}, {"pseudo_metrics": 2}
+        )
 
     model_conf = modelstore_config["model"]
     base_name = os.path.join(
@@ -78,6 +84,135 @@ def test_modelstore_dump(glob, copy, makedirs, path_exists, modelstore_config):
     mo.assert_has_calls(calls, any_order=True)
 
 
+@mock.patch("{}.os.path.exists".format(r.__name__), return_value=True)
+@mock.patch("{}.pickle.dump".format(r.__name__))
+@mock.patch("{}.json.dump".format(r.__name__))
+def test_modelstore_dump_extra_model_keys(
+    jsond, pickled, path_exists, modelstore_config
+):
+    modelstore_config["model"]["extraparam"] = 42
+    modelstore_config["model"]["anotherparam"] = 23
+    modelstore_config["model"]["created"] = "colliding keys should not occur"
+    with mock.patch(
+        "{}.open".format(r.__name__), mock.mock_open(), create=True
+    ) as _:
+        ms = r.ModelStore(modelstore_config)
+        ms.dump_trained_model(
+            modelstore_config, {"pseudo_model": 1}, {"pseudo_metrics": 2}
+        )
+
+    dumped = jsond.call_args[0][0]
+    print(modelstore_config)
+    print(dumped)
+    assert "extraparam" in dumped
+    assert dumped["extraparam"] == 42
+    assert "anotherparam" in dumped
+    assert dumped["anotherparam"] == 23
+    assert dumped["created"] != "colliding keys should not occur"
+
+
+@mock.patch("{}.os.path.exists".format(r.__name__), return_value=True)
+@mock.patch("{}.pickle.dump".format(r.__name__))
+@mock.patch("{}.json.dump".format(r.__name__))
+def test_modelstore_train_report(
+    jsond, pickled, path_exists, modelstore_config
+):
+    with mock.patch(
+        "{}.open".format(r.__name__), mock.mock_open(), create=True
+    ) as _:
+        ms = r.ModelStore(modelstore_config)
+        ms.add_to_train_report("report_key", "report_val")
+        ms.dump_trained_model(
+            modelstore_config, {"pseudo_model": 1}, {"pseudo_metrics": 2}
+        )
+
+    dumped = jsond.call_args[0][0]
+    print(modelstore_config)
+    print(dumped)
+    assert "train_report" in dumped
+    assert "report_key" in dumped["train_report"]
+    assert dumped["train_report"]["report_key"] == "report_val"
+
+    assert "system" in dumped
+    assert "mllaunchpad_version" in dumped["system"]
+    assert "platform" in dumped["system"]
+    assert "packages" in dumped["system"]
+
+
+@mock.patch("{}.os.path.exists".format(r.__name__), return_value=True)
+@mock.patch(
+    "{}.ModelStore._load_metadata".format(r.__name__),
+    side_effect=lambda _: {"the_json": round(random() * 1000)},
+)
+def test_list_models(_load_metadata, path_exists, modelstore_config, caplog):
+    model_jsons = [
+        "mymodel_1.0.0.json",
+        "mymodel_1.1.0.json",
+        "anothermodel_0.0.1.json",
+    ]
+    backup_jsons = [
+        "mymodel_1.0.0_2021-08-01_18-00-00.json",
+        "mymodel_1.0.0_2021-07-31_12-00-00",
+    ]
+
+    def my_glob(pattern):
+        if "previous" in pattern.lower():
+            return backup_jsons
+        else:
+            return model_jsons
+
+    with mock.patch(
+        "{}.glob.glob".format(r.__name__), side_effect=my_glob,
+    ):
+        with caplog.at_level(logging.DEBUG):
+            ms = r.ModelStore(modelstore_config)
+            models = ms.list_models()
+
+    print(models)
+    assert (
+        len(models) == 2
+    )  # one model ID named "mymodel" and one named "anothermodel"
+    assert models["mymodel"]["latest"] == models["mymodel"]["1.1.0"]
+    assert len(models["mymodel"]["backups"]) == 2
+    assert models["anothermodel"]["backups"] == []
+    assert "ignoring" not in caplog.text.lower()
+
+
+@mock.patch("{}.os.path.exists".format(r.__name__), return_value=True)
+@mock.patch(
+    "{}.ModelStore._load_metadata".format(r.__name__),
+    side_effect=lambda _: {"the_json": round(random() * 1000)},
+)
+def test_list_models_ignore_obsolete_backups(
+    _load_metadata, path_exists, modelstore_config, caplog
+):
+    model_jsons = [
+        "mymodel_1.0.0.json",
+        "mymodel_1.1.0.json",
+        "anothermodel_0.0.1.json",
+    ]
+    backup_jsons = [
+        "mymodel_1.0.0_2021-08-01_18-00-00.json",
+        "OBSOLETE-IGNORED_1.0.0_2021-07-31_12-00-00",
+    ]
+
+    def my_glob(pattern):
+        if "previous" in pattern.lower():
+            return backup_jsons
+        else:
+            return model_jsons
+
+    with mock.patch(
+        "{}.glob.glob".format(r.__name__), side_effect=my_glob,
+    ):
+        with caplog.at_level(logging.DEBUG):
+            ms = r.ModelStore(modelstore_config)
+            models = ms.list_models()
+
+    print(models)
+    assert "ignoring" in caplog.text.lower()
+
+
 @mock.patch("{}.pickle.load".format(r.__name__), return_value="pickle")
 @mock.patch("{}.json.load".format(r.__name__), return_value={"json": 0})
 def test_modelstore_load(json, pkl, modelstore_config):