Repository: kedro-org/kedro

Issue:
Use soft-merge for certain configuration files for `OmegaConfLoader`
## Description
At the moment all configuration files are merged in a destructive way. This means all fields/keys are overwritten when two files are merged e.g.:

Base catalog file:
```yml
regressor:
  type: pickle.PickleDataSet
  filepath: data/06_models/regressor.pickle
  versioned: true
```

Overriding catalog file:
```yml
regressor:
  type: pandas.CSVDataSet
```

Will result in:
```yml
regressor:
  type: pandas.CSVDataSet
```

With catalog and logging files this behaviour is desirable because if you'd like to change a file type, you don't want any fields to remain that belonged to the other file type (e.g. `plotly_args:` for the `PlotlyDataSet`). However, with parameter or other config files (e.g. for mlflow) it might be better to merge them in a soft way so you can do something like: 

Base `mlflow.yml` file:
```yml
tracking:
  disable_tracking:
    pipelines:
    - on_exit_notification

  experiment:
    name: name-of-local-experiment

  params:
    long_params_strategy: tag
```

Overriding `mlflow.yml` file:
```yml
tracking:
  experiment:
    name: name-of-prod-experiment
```

Results in:
```yml
tracking:
  disable_tracking:
    pipelines:
    - on_exit_notification
  experiment:
    name: name-of-prod-experiment
  params:
    long_params_strategy: tag
```

This can be achieved my using the OmegaConf merge: https://omegaconf.readthedocs.io/en/2.2_branch/usage.html#merging-configurations 

## Context
https://github.com/kedro-org/kedro/issues/1868#issuecomment-1285020986

## Possible Implementation
Add another setting to `OmegaConfLoader` where the user can specify which files to merge in a destructive way and which one with the regular OmegaConf merge.

* Check if it's possible to remove keys with the `OmegaConf` merge.

## Possible Alternative
Merge `catalog` files in a destructive way and all other in a soft way. This would be a breaking change. 



Discussion:
After I explored the first iteration of the `OmegaConfLoader`, I decided to give this one a take of my own. I was able to already implement a soft-merge functionality as well. But for now, this only applies to params (hard-coded). I mostly drew inspiration from the current implementation as well as the older `ConfigLoader` implementations. You can find it on my GitHub: https://github.com/MatthiasRoels/kedro-examples/blob/main/custom-configloader/src/prototype/lib/configloader.py
Note that there are tests included too (mostly copy-paste from the existing test-script). 

To iterate further on this, we should come up with a way to expose to the user which patterns they would like to use in "merge mode" and which ones they would like to use in "overwrite mode" (.e.g merge params, but overwrite catalog entries).

As a final note (having used a `TemplatedConfigLoader` before combined with Yaml anchors and aliases to keep our catalog files leaner), we used to do something along the following lines to reduce complexity in our catalog files: 

``` 
_s3: &s3
  fs_args:
    s3_additional_kwargs:
      ServerSideEncryption: AES256

_parquet: &parquet
  type: kedro.extras.datasets.pandas.ParquetDataSet
  <<: *s3

example_dataset:
  <<: *parquet
  filepath: s3-file-uri
```
With OmegaConf, this is no longer possible and while replacing Yaml anchors with interpolations  is possible, this would also mean that the keys "s3" and "parquet" would end up in our catalog config files. So, ideally, we should come up with a way to filter these keys (or implement something similar to what we had in `TemplatedConfigLoader` with globals). 

Finally, one other disadvantage of the current implementations of `OmegaConfLoader` is that we can only use interpolations on a per pattern-directory basis (e.g. parameters-base). Ideally, we also come up with a way to use this globally (perhaps with something like a `globals` pattern that we load when we init our `OmegaConfLoader` class? 

Anyway, I found this a fun exercise to do and I hope this brings some additional ideas/inspiration to the table for future development. Happy to discuss this further as well (you can also find me on the Kedro Slack channel). 
This is amazing @MatthiasRoels , thanks so much for having a look at the `OmegaConfLoader` and giving your feedback! When it's officially released we'll get more people to do the same thing 😄 

> After I explored the first iteration of the `OmegaConfLoader`, I decided to give this one a take of my own. I was able to already implement a soft-merge functionality as well. But for now, this only applies to params (hard-coded). I mostly drew inspiration from the current implementation as well as the older `ConfigLoader` implementations. You can find it on my GitHub: https://github.com/MatthiasRoels/kedro-examples/blob/main/custom-configloader/src/prototype/lib/configloader.py Note that there are tests included too (mostly copy-paste from the existing test-script).

Nice one! This will definitely be useful when we implement the functionality. We'll likely get this in for the `0.19.x` series. 

> As a final note (having used a `TemplatedConfigLoader` before combined with Yaml anchors and aliases to keep our catalog files leaner), we used to do something along the following lines to reduce complexity in our catalog files:
> 
> ```
> _s3: &s3
>   fs_args:
>     s3_additional_kwargs:
>       ServerSideEncryption: AES256
> 
> _parquet: &parquet
>   type: kedro.extras.datasets.pandas.ParquetDataSet
>   <<: *s3
> 
> example_dataset:
>   <<: *parquet
>   filepath: s3-file-uri
> ```
> 
> With OmegaConf, this is no longer possible and while replacing Yaml anchors with interpolations is possible, this would also mean that the keys "s3" and "parquet" would end up in our catalog config files. So, ideally, we should come up with a way to filter these keys (or implement something similar to what we had in `TemplatedConfigLoader` with globals).

This is a very good point, and definitely something we need to explore further. If `OmegaConfLoader` is to successfully replace `TemplatedConfigLoader` it needs to at least offer similar functionality. Large config files is a huge issue for Kedro users, so we need to make sure that this new config loader doesn't make it worse. 

> Finally, one other disadvantage of the current implementations of `OmegaConfLoader` is that we can only use interpolations on a per pattern-directory basis (e.g. parameters-base). Ideally, we also come up with a way to use this globally (perhaps with something like a `globals` pattern that we load when we init our `OmegaConfLoader` class?

This is somewhat related to: https://github.com/kedro-org/kedro/issues/2175 You're absolutely right that we should come up with a way to use interpolations across config. 

> Anyway, I found this a fun exercise to do and I hope this brings some additional ideas/inspiration to the table for future development. Happy to discuss this further as well (you can also find me on the Kedro Slack channel).

Awesome! 😄 This is really insightful and I appreciate you taking the time to investigate and leave the feedback ⭐ I'll definitely reach out to you when I need more input!
Thanks! Related to https://github.com/kedro-org/kedro/issues/2175, I think I have a neat implementation for that one too, extending on what I already implemented (with just a tiny bit refactoring). Will code that up tomorrow and post a link to my code in that issue. 

Happy to create a PR to contribute my changes!
As another temporary workaround, at least for the `parameters.yml` I noticed that you can overload a single nested value by creating an entry in the non-base `parameters.yml` using the path to the value, while keeping remaining values intact. Using @MatthiasRoels example, it works more or less like this:

Base `parameters.yml` file:
```yml
tracking:
  disable_tracking:
    pipelines:
    - on_exit_notification

  experiment:
    name: name-of-local-experiment

  params:
    long_params_strategy: tag
```

Overriding `parameters.yml` file:

```yml
tracking.experiment.name: name-of-prod-experiment
```

Results in:

```yml
tracking:
  disable_tracking:
    pipelines:
    - on_exit_notification
  experiment:
    name: name-of-prod-experiment
  params:
    long_params_strategy: tag
```
Has anyone found a workaround for globals as well? The above mentioned does only work for parameters sadly.

For anyone interested, I solved it by adding this to our custom config loader. 

```

SOFT_MERGE_KEY = "~~soft_merge~~"
class ExtendedOmegaConfigLoader(OmegaConfigLoader):
    
    def __getitem__(self, key: str) -> dict[str, Any]:
        """Extension to allow for soft merging on certain parameters.

        Soft merging has to be explicit, by placing it behind a ~~soft_merge~~
        key.

        For example in a yaml file it could look like this.
        ~~soft_merge~~:
            general:
                schema: value_to_be_soft_merged
        """
        item = super().__getitem__(key)
        if key in self:
            return item
        soft_merge_values = item.pop(SOFT_MERGE_KEY, {})
        return OmegaConf.to_container(
            OmegaConf.merge(item, soft_merge_values), resolve=True
        )
```

For us it is especially useful when writing test cases. As you then often want to overwrite only a limited number of parameters, to check if the base config is still valid.
Hi @IngerMathilde , thanks for sharing your workaround! This was marked as a blocker for 0.19, so we will get to it very soon.

Resolving patch:
diff --git a/kedro/config/omegaconf_config.py b/kedro/config/omegaconf_config.py
--- a/kedro/config/omegaconf_config.py
+++ b/kedro/config/omegaconf_config.py
@@ -83,6 +83,7 @@ def __init__(  # noqa: too-many-arguments
         base_env: str = "base",
         default_run_env: str = "local",
         custom_resolvers: dict[str, Callable] = None,
+        merge_strategy: dict[str, str] = None,
     ):
         """Instantiates a ``OmegaConfigLoader``.
 
@@ -103,6 +104,7 @@ def __init__(  # noqa: too-many-arguments
         """
         self.base_env = base_env
         self.default_run_env = default_run_env
+        self.merge_strategy = merge_strategy or {}
 
         self.config_patterns = {
             "catalog": ["catalog*", "catalog*/**", "**/catalog*"],
@@ -224,17 +226,16 @@ def __getitem__(self, key) -> dict[str, Any]:  # noqa: PLR0912
             else:
                 raise exc
 
-        # Destructively merge the two env dirs. The chosen env will override base.
-        common_keys = config.keys() & env_config.keys()
-        if common_keys:
-            sorted_keys = ", ".join(sorted(common_keys))
-            msg = (
-                "Config from path '%s' will override the following "
-                "existing top-level config keys: %s"
+        merging_strategy = self.merge_strategy.get(key)
+        if merging_strategy == "soft":
+            resulting_config = self._soft_merge(config, env_config)
+        elif merging_strategy == "destructive" or not merging_strategy:
+            resulting_config = self._destructive_merge(config, env_config, env_path)
+        else:
+            raise ValueError(
+                f"Merging strategy {merging_strategy} not supported. The accepted merging "
+                f"strategies are `soft` and `destructive`."
             )
-            _config_logger.debug(msg, env_path, sorted_keys)
-
-        config.update(env_config)
 
         if not processed_files and key != "globals":
             raise MissingConfigException(
@@ -242,7 +243,7 @@ def __getitem__(self, key) -> dict[str, Any]:  # noqa: PLR0912
                 f" the glob pattern(s): {[*self.config_patterns[key]]}"
             )
 
-        return config
+        return resulting_config
 
     def __repr__(self):  # pragma: no cover
         return (
@@ -445,6 +446,26 @@ def _resolve_environment_variables(config: dict[str, Any]) -> None:
         else:
             OmegaConf.resolve(config)
 
+    @staticmethod
+    def _destructive_merge(config, env_config, env_path):
+        # Destructively merge the two env dirs. The chosen env will override base.
+        common_keys = config.keys() & env_config.keys()
+        if common_keys:
+            sorted_keys = ", ".join(sorted(common_keys))
+            msg = (
+                "Config from path '%s' will override the following "
+                "existing top-level config keys: %s"
+            )
+            _config_logger.debug(msg, env_path, sorted_keys)
+
+        config.update(env_config)
+        return config
+
+    @staticmethod
+    def _soft_merge(config, env_config):
+        # Soft merge the two env dirs. The chosen env will override base if keys clash.
+        return OmegaConf.to_container(OmegaConf.merge(config, env_config))
+
     def _is_hidden(self, path: str):
         """Check if path contains any hidden directory or is a hidden file"""
         path = Path(path)


Test patch:
diff --git a/tests/config/test_omegaconf_config.py b/tests/config/test_omegaconf_config.py
--- a/tests/config/test_omegaconf_config.py
+++ b/tests/config/test_omegaconf_config.py
@@ -121,6 +121,31 @@ def proj_credentials_env_variable(tmp_path):
     )
 
 
+@pytest.fixture
+def mlflow_config(tmp_path):
+    base_mlflow = tmp_path / _BASE_ENV / "mlflow.yml"
+    base_config = {
+        "tracking": {
+            "disable_tracking": {"pipelines": "[on_exit_notification]"},
+            "experiment": {
+                "name": "name-of-local-experiment",
+            },
+            "params": {"long_params_strategy": "tag"},
+        }
+    }
+    local_mlflow = tmp_path / _DEFAULT_RUN_ENV / "mlflow.yml"
+    local_config = {
+        "tracking": {
+            "experiment": {
+                "name": "name-of-prod-experiment",
+            },
+        }
+    }
+
+    _write_yaml(base_mlflow, base_config)
+    _write_yaml(local_mlflow, local_config)
+
+
 use_config_dir = pytest.mark.usefixtures("create_config_dir")
 use_proj_catalog = pytest.mark.usefixtures("proj_catalog")
 use_credentials_env_variable_yml = pytest.mark.usefixtures(
@@ -417,20 +442,29 @@ def test_customised_config_patterns(self, tmp_path):
             "**/params*",
         ]
 
-    def test_destructive_merging_strategy(self, tmp_path):
+    def test_default_destructive_merging_strategy(self, tmp_path, mlflow_config):
         mlflow_patterns = {"mlflow": ["mlflow*", "mlflow*/**", "**/mlflow*"]}
-        base_mlflow = tmp_path / _BASE_ENV / "mlflow.yml"
-        base_config = {
+        conf = OmegaConfigLoader(str(tmp_path), config_patterns=mlflow_patterns)[
+            "mlflow"
+        ]
+
+        assert conf == {
             "tracking": {
-                "disable_tracking": {"pipelines": "[on_exit_notification]"},
                 "experiment": {
-                    "name": "name-of-local-experiment",
+                    "name": "name-of-prod-experiment",
                 },
-                "params": {"long_params_strategy": "tag"},
             }
         }
-        local_mlflow = tmp_path / _DEFAULT_RUN_ENV / "mlflow.yml"
-        local_config = {
+
+    def test_destructive_merging_strategy(self, tmp_path, mlflow_config):
+        mlflow_patterns = {"mlflow": ["mlflow*", "mlflow*/**", "**/mlflow*"]}
+        conf = OmegaConfigLoader(
+            str(tmp_path),
+            config_patterns=mlflow_patterns,
+            merge_strategy={"mlflow": "destructive"},
+        )["mlflow"]
+
+        assert conf == {
             "tracking": {
                 "experiment": {
                     "name": "name-of-prod-experiment",
@@ -438,21 +472,33 @@ def test_destructive_merging_strategy(self, tmp_path):
             }
         }
 
-        _write_yaml(base_mlflow, base_config)
-        _write_yaml(local_mlflow, local_config)
-
-        conf = OmegaConfigLoader(str(tmp_path), config_patterns=mlflow_patterns)[
-            "mlflow"
-        ]
+    def test_soft_merging_strategy(self, tmp_path, mlflow_config):
+        mlflow_patterns = {"mlflow": ["mlflow*", "mlflow*/**", "**/mlflow*"]}
+        conf = OmegaConfigLoader(
+            str(tmp_path),
+            config_patterns=mlflow_patterns,
+            merge_strategy={"mlflow": "soft"},
+        )["mlflow"]
 
         assert conf == {
             "tracking": {
+                "disable_tracking": {"pipelines": "[on_exit_notification]"},
                 "experiment": {
                     "name": "name-of-prod-experiment",
                 },
+                "params": {"long_params_strategy": "tag"},
             }
         }
 
+    def test_unsupported_merge_strategy(self, tmp_path, mlflow_config):
+        mlflow_patterns = {"mlflow": ["mlflow*", "mlflow*/**", "**/mlflow*"]}
+        with pytest.raises(ValueError):
+            OmegaConfigLoader(
+                str(tmp_path),
+                config_patterns=mlflow_patterns,
+                merge_strategy={"mlflow": "hard"},
+            )["mlflow"]
+
     @use_config_dir
     def test_adding_extra_keys_to_confloader(self, tmp_path):
         """Make sure extra keys can be added directly to the config loader instance."""