Repository: Galileo-Galilei/kedro-mlflow (language: python)

Issue:
Setup mlflow configuration when context is loaded
## Description

When I call load_context() interactively, the configuration of the `mlflow.yml` is:
- not accessible through a context attributes
- not set up (especially, the mlflow tracking uri is not set)

## Context

I always struggle when i want to use the configuration in the `mlflow.yml` interactively. This makes experimentation & debugging harder.

## Possible Implementation

For now, I can't see how it is possible to modify the context object to add attributes. We can eventually add some to the catalog with the `after_catalog_created` hook, but the configuration should not be tied to the catalog execution.



Discussion / hints:
I understand the point, but to my knowledge  the kedro context do not manage plugins properties, it just use them to supercharge his own properties (catalog, params, pipelines)

It isn't sufficient for you to get the config by calling the `get_mlflow_config` of kedro_mlflow ?
```
context = load_context(Path.cwd())
mlflow_config = get_mlflow_config(context)
```
The current implementation seems indeed satisfying enough (with ``kedro>=0.17`` and the a ``KedroSession`` activated, it can be simplified to 
```python
from kedro_mlflow.framework.context
mlflow_config=get_mlflow_config()
mlflow_config.setup()
```

However, we should setup automatically the configuration  for jupyter, through the ``%reload_kedro`` line magic.

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 0dce25b2..43ef4b7b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,10 @@
 
 ## [Unreleased]
 
+### Added
+
+- :sparkles: Add support for notebook use . When a notebook is opened via a kedro command (e.g. `kedro jupyter notebook`), you can call the `%reload_kedro_mlflow` line magic to setup mlflow configuration automatically. A ``mlflow_client`` to the database is also created available as a global variable ([#124](https://github.com/Galileo-Galilei/kedro-mlflow/issues/124)).
+
 ### Changed
 
 - :recycle: ``KedroMlflowConfig`` was refactored with pydantic for improved type checking when loading configuration, overall robustness and autocompletion. Its keys have changed, but it is not considered as a user facing changes since the public function ``get_mlflow_config()`` and ``KedroMlflowConfig().setup()`` are not modified.
diff --git a/docs/index.rst b/docs/index.rst
index 8a5e5de3..bb60b290 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -12,8 +12,9 @@ Welcome to kedro-mlflow's documentation!
    Introduction <source/01_introduction/index.rst>
    Installation <source/02_installation/index.rst>
    Getting Started <source/03_getting_started/index.rst>
-   Advanced machine learning versioning <source/04_experimentation_tracking/index.rst>
-   A mlops framework for efficient deployment <source/05_framework_ml/index.rst>
+   Experimentation tracking <source/04_experimentation_tracking/index.rst>
+   A mlops framework for continuous model serving <source/05_framework_ml/index.rst>
+   Interactive use <source/06_interactive_use/index.rst>
    Python objects <source/07_python_objects/index.rst>
 
 
diff --git a/docs/source/02_installation/index.rst b/docs/source/02_installation/index.rst
index 552dcb50..bbb824f5 100644
--- a/docs/source/02_installation/index.rst
+++ b/docs/source/02_installation/index.rst
@@ -6,4 +6,4 @@ Introduction
 
    Install the plugin <01_installation.md>
    Setup your kedro project <02_setup.md>
-   Migration guide <03_migration_guide.md>
+   Migration guide between versions <03_migration_guide.md>
diff --git a/docs/source/04_experimentation_tracking/index.rst b/docs/source/04_experimentation_tracking/index.rst
index cb5b73a5..05031f8c 100644
--- a/docs/source/04_experimentation_tracking/index.rst
+++ b/docs/source/04_experimentation_tracking/index.rst
@@ -9,4 +9,4 @@ Introduction
    Version datasets <03_version_datasets.md>
    Version models <04_version_models.md>
    Version metrics <05_version_metrics.md>
-   Opening the User Interface <06_mlflow_ui.md>
+   Open the User Interface <06_mlflow_ui.md>
diff --git a/docs/source/06_interactive_use/01_notebook_use.md b/docs/source/06_interactive_use/01_notebook_use.md
new file mode 100644
index 00000000..fea6f738
--- /dev/null
+++ b/docs/source/06_interactive_use/01_notebook_use.md
@@ -0,0 +1,70 @@
+# How to use `kedro-mlflow` in a notebook
+
+```eval_rst
+.. important:: You need to call ``pip install kedro_mlflow[extras]`` to access notebook functionalities.
+```
+
+## Reminder on mlflow's limitations with interactive use
+
+Data science project lifecycle are very iterative. Mlflow intends to track parameters changes to imporove reproducibility. However, one must be conscious that being able to **execute functions outside of a end to end pipeline** puts a strong burden on the user shoulders **because he is in charge to make the code execution coherent** by running the notebooks cells in the right order. Any back and forth during execution to change some parameters in a previous notebook cells and then retrain a model creates an operational risk that the recorded parameter stored in mlflow is different than the real parameter used for training the model.
+
+To make a long story short: **forget about efficient reproducibility** when using mlflow interactively.
+
+It may **still be useful to track some experiments results** especially if they are long to run and vary wildly with parameters, e.g. if you are performing hyperparameter tuning.
+
+These limitations are inherent to the data science process, not to mlflow itself or the plugin.
+
+## Setup mlflow configuration in your notebook
+
+Open your notebook / ipython session with the Kedro CLI:
+
+```bash
+kedro jupyter notebook
+```
+
+Kedro [creates a bunch of global variables](https://kedro.readthedocs.io/en/latest/11_tools_integration/02_ipython.html#kedro-and-jupyter), including a `session` which are automatically accessible. It also registers some line_magic from plugins, including `%=reload_kedro_mlflow` from `kedro-mlflow`.
+
+In your first cell, launch:
+```python
+%reload_kedro_mlflow
+```
+
+This automatically:
+- load and setup (create the tracking uri, export credentials...) the mlflow configuration of your `mlflow.yml`
+- import ``mlflow`` which is now accessible in your notebook
+- Create a `mlflow_client` object with your mlflow server settings, which is now accessible in your notebook
+
+If you change your ``mlflow.yml``, re-execute this cell for the changes to take effect.
+
+## Difference with running through the CLI
+
+- The DataSets `load` and `save` methods works as usual. You can call `catalog.save("my_artifact_dataset", data)` inside a cell, and your data will be logged in mlflow properly (assuming "my_artifact_dataset" is a `kedro_mlflow.io.MlflowArtifactDataSet`).
+- The `hooks` which setup configuration are only accessible if you run the session interactive, e.g.:
+```python
+session.run(
+    pipeline_name = "my_ml_pipeline",
+    tags = "training",
+    from_inputs = "data_2",
+    to_outputs = "data_7"
+    )
+```
+but it is not very likely in a notebook.
+
+## Guidelines and best practices suggestions
+
+During experimentation phase, you will likely not run entire pipelines (or sub pipelines filtered out between some inputs and outputs). Hence, you cannot benefit from Kedro's ``hooks`` (and hence from ``kedro-mlflow`` tracking). From this moment on, perfect reproducbility is impossible to achieve: there is no chance that you manage to maintain a perfectly linear workflow, as you will go back and forth modifying parameters and code to create your model.
+
+I suggest to :
+- **focus on versioning parameters and metrics**. The goal is to finetune your hyperparameters and to be able to remember later the best setup. It is not very important to this stage to version all parameters (e.g. preprocessing ones) nor models (after all you will need an entire pipeline to predict and it is very unlikely that you will need to reuse these experiment models one day.) It may be interesting to use ``mlflow.autolog()`` feature to have a easy basic setup.
+- **transition quickly to kedro pipelines**. For instance, when you preprocessing is roughly defined, try to put it in kedro pipelines. You can then use notebooks to experiment / perfom hyperparameter tuning while keeping preprocessing "fixed" to enhance reproducibility. You can run this pipeline interactively with :
+
+```python
+res=session.run(
+    pipeline_name = "my_preprocessing_pipeline",
+    tags = "training",
+    from_inputs = "data_2",
+    to_outputs = "data_7"
+    )
+```
+
+``res`` is a python dict with the outputs of your pipeline (e.g. a "preprocessed_data" ``pandas.DataFrame``), and you can use it interactively in your notebook.
diff --git a/docs/source/06_interactive_use/index.rst b/docs/source/06_interactive_use/index.rst
new file mode 100644
index 00000000..2c42da9d
--- /dev/null
+++ b/docs/source/06_interactive_use/index.rst
@@ -0,0 +1,7 @@
+Introduction
+============
+
+.. toctree::
+   :maxdepth: 4
+
+   How to use in a notebook <01_notebook_use.md>
diff --git a/docs/source/06_advanced_use/01_run_with_mlproject.md b/docs/source/08_advanced_use/01_run_with_mlproject.md
similarity index 100%
rename from docs/source/06_advanced_use/01_run_with_mlproject.md
rename to docs/source/08_advanced_use/01_run_with_mlproject.md
diff --git a/docs/source/06_advanced_use/02_compatbility_chart.md b/docs/source/08_advanced_use/02_compatbility_chart.md
similarity index 100%
rename from docs/source/06_advanced_use/02_compatbility_chart.md
rename to docs/source/08_advanced_use/02_compatbility_chart.md
diff --git a/docs/source/06_advanced_use/index.rst b/docs/source/08_advanced_use/index.rst
similarity index 100%
rename from docs/source/06_advanced_use/index.rst
rename to docs/source/08_advanced_use/index.rst
diff --git a/kedro_mlflow/extras/__init__.py b/kedro_mlflow/extras/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/kedro_mlflow/extras/extensions/__init__.py b/kedro_mlflow/extras/extensions/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/kedro_mlflow/extras/extensions/ipython.py b/kedro_mlflow/extras/extensions/ipython.py
new file mode 100644
index 00000000..51aad1ab
--- /dev/null
+++ b/kedro_mlflow/extras/extensions/ipython.py
@@ -0,0 +1,28 @@
+import logging.config
+
+from IPython import get_ipython
+from kedro.framework.startup import bootstrap_project
+from mlflow.tracking import MlflowClient
+
+from kedro_mlflow.config import get_mlflow_config
+
+
+def reload_kedro_mlflow(line=None, local_ns=None):
+    metadata = bootstrap_project(
+        local_ns["project_path"]
+    )  # project_path is a global variable defined by init_kedro
+    mlflow_config = get_mlflow_config()  # can only be called if a session exist
+
+    mlflow_client = MlflowClient(tracking_uri=mlflow_config.mlflow_tracking_uri)
+    global_variables = {"mlflow_client": mlflow_client}
+    get_ipython().push(variables=global_variables)
+    get_ipython().run_cell("import mlflow", "", "")
+    get_ipython().run_cell("mlflow_config.setup()")
+
+    global_variables_names = "\n- ".join(
+        [f"'{key}'" for key in global_variables.keys()]
+    )
+    logging.info("** Kedro project %s", str(metadata.project_name))
+    logging.info("Import mlflow in namespace")
+    logging.info("Setup mlflow configuration")
+    logging.info(f"Defined global variable(s):\n- {global_variables_names}")
diff --git a/setup.py b/setup.py
index 7be733c5..6582c737 100644
--- a/setup.py
+++ b/setup.py
@@ -59,6 +59,7 @@ def _parse_requirements(path, encoding="utf-8"):
             "pre-commit>=2.0.0,<3.0.0",
             "jupyter>=1.0.0,<2.0.0",
         ],
+        "extras": ["notebook>=6.0.0"],
     },
     author="Galileo-Galilei",
     entry_points={
@@ -72,6 +73,9 @@ def _parse_requirements(path, encoding="utf-8"):
             "mlflow_pipeline_hook = kedro_mlflow.framework.hooks.pipeline_hook:mlflow_pipeline_hook",
             "mlflow_node_hooks = kedro_mlflow.framework.hooks.node_hook:mlflow_node_hook",
         ],
+        "kedro.line_magic": [
+            "line_magic = kedro_mlflow.extras.extensions.ipython:reload_kedro_mlflow"
+        ],
     },
     zip_safe=False,
     keywords="kedro-plugin, mlflow, model versioning, model packaging, pipelines, machine learning, data pipelines, data science, data engineering",


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 2d9c781b..3127b27a 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -28,7 +28,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install -e .[test]
+        pip install -e .[test,extras]
     - name: Lint with flake8
       if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.7' # linting should occur only once in the loop
       run: |
diff --git a/tests/extras/__init__.py b/tests/extras/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/extras/extensions/__init__.py b/tests/extras/extensions/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/tests/extras/extensions/test_ipython.py b/tests/extras/extensions/test_ipython.py
new file mode 100644
index 00000000..8baa1387
--- /dev/null
+++ b/tests/extras/extensions/test_ipython.py
@@ -0,0 +1,32 @@
+import pytest
+from kedro.framework.session.session import KedroSession, _deactivate_session
+from kedro.framework.startup import bootstrap_project
+
+from kedro_mlflow.extras.extensions.ipython import reload_kedro_mlflow
+
+
+@pytest.fixture(autouse=True)
+def mocked_logging(mocker):
+    # Disable logging.config.dictConfig in KedroSession._setup_logging as
+    # it changes logging.config and affects other unit tests
+    return mocker.patch("logging.config.dictConfig")
+
+
+@pytest.fixture(autouse=True)
+def cleanup_session():
+    yield
+    _deactivate_session()
+
+
+def test_load_global_variables_in_ipython(mocker, kedro_project_with_mlflow_conf):
+
+    mock_ipython = mocker.patch("kedro_mlflow.extras.extensions.ipython.get_ipython")
+    mocker.patch("kedro.framework.session.session.KedroSession._setup_logging")
+
+    metadata = bootstrap_project(kedro_project_with_mlflow_conf)
+    with KedroSession.create(metadata.package_name, project_path=metadata.project_path):
+        print("with session project_path:", metadata.project_path)
+        reload_kedro_mlflow(line=None, local_ns={"project_path": metadata.project_path})
+
+    mock_ipython().push.assert_called_once_with(variables={"mlflow_client": mocker.ANY})
+    mock_ipython().run_cell.assert_called_with("mlflow_config.setup()")
diff --git a/tests/framework/hooks/test_all_hooks.py b/tests/framework/hooks/test_all_hooks.py
index a07bd7e9..87b2f7a1 100644
--- a/tests/framework/hooks/test_all_hooks.py
+++ b/tests/framework/hooks/test_all_hooks.py
@@ -256,8 +256,9 @@ def mock_session(
     mocker, mock_settings_with_mlflow_hooks, kedro_project_path
 ):  # pylint: disable=unused-argument
 
-    # see here why it is needed: https://github.com/quantumblacklabs/kedro/blob/956dbed2289397b69506a8d79cca831032e89163/tests/framework/session/conftest.py#L407
-    mocker.patch("kedro.framework.project._validate_module")
+    # we need to patch "kedro.framework.session.session.validate_settings" instead of
+    # "kedro.framework.project.validate_settings" because it is imported
+    mocker.patch("kedro.framework.session.session.validate_settings")
     mocker.patch(
         "kedro.framework.project._register_hooks_setuptools"
     )  # prevent registering the one of the plugins which are already installed