Repository: kedro-org/kedro

Issue:
Enable overriding configuration via variable interpolation for all configurations other than just `parameters` for OmegaConfigLoader
## Description
Is your feature request related to a problem? A clear and concise description of what the problem is: "I'm always frustrated when ..."

With #2467 we resolves OCL earlier and parameters propogate correctly for `parameters`. We cannot do the same for `catalog` or other configuration yet due to the problem of "residual" configurations. The problem is specific to `OmegaConfigLoader`, because it requires merging the dictionaries before resolving.

Take this simple example

```yaml
# catalog.yml
my_dataset:
   path: ${base_path}/my_file.txt
```


## Expected Result
`kedro run --params base_path:my_bucket` this should work and replace the value correctly.

## Current Result
The value will not propagate for configurations other than `parameters`. This is because we need to merge `aggregated_configs` with `runtime_params`, which produce invalid catalog entries.

The compiled catalog may look like this
```yaml
# catalog.yml
my_dataset:
  path: ${base_path}/my_file.txt


base_path: my_bucket # <-- This is not a dataset, ideally we need to remove it or bypass it.
```

## Context
Why is this change important to you? How would you use it? How can it benefit other users?
This is essential to enable common use case like providing a bucket name for `catalog` via CLI.
`kedro run --params base_path:my_bucket` , with TemplatedConfigLoader it doesn't work either, but there are [workarounds](https://github.com/kedro-org/kedro/discussions/1782). In contrast, there is no workaround for OmegaConfigLoader.

## Possible Implementation
(Optional) Suggest an idea for implementing the addition or change.
1. Figure out a way to remove the residual configurations

The implementation should make sure above case will work correctly. It should also work for arbitrary configurations, i.e. `config_loader["spark"] or `config_loader["my_random_config"]` as long as it's registered with `config_patterns`

## Possible Alternatives
(Optional) Describe any alternative solutions or features you've considered.


## Detail context
> **Change 1:** Resolves configs eagerly in ConfigLoader - only enable for parameters for now
> config_loader[“parameters”] return Dict instead of DictConf
> interpolation will works correctly
> **Change 2**: Separate the runtime_params & meta_params - the name of “meta_params” need to be determined afterward. Some concerns around “runtime_params” works for “parameters” only, but “mete_params” works for everything and will confuse people
> 
> 1. Do Change 1 now - limited the scope to “parameters”
> 2. Change 2 should be done, but requires some extra thinking - potentially interfering with the solution of enabling “globals” for OCL
> 3. Change 2 should be done - would be great to do it in a non-breaking way. General agreement on the change, but unsure about what’s the correct name of this argument.
> Some altenative: “config_vars”, “meta_config”
> 
> Extended Discussion

> How does “global” fit in the picture? It will be clearer when we finished Change 1
> Antony thinks that with the validation change, it shouldn’t be matter even when the “residuals” are left in the config, Python config also don’t do anything special to extra configurations.
> Ivan think this doesn’t solve the problem in a generic way, config_lodaer[“global”] is purposed, but this doesn’t solve all the problem because OmegaConfigLoader requires to merge dictionary before interpolating. This shouldn’t be a limitation, but will requires more work to implement a solution
> Merel has some concern about the messaging of “meta_params”, will people really understand which argument to be used.
> kedro run --params=base_path:s3_bucket will not work for catalog yet, but agree that it would be nice to support.


Discussion:
Needs to be completed after https://github.com/kedro-org/kedro/issues/2507
There's two (potentially more) solutions to this ticket -
## Solution 1
I have a draft PR with this one - https://github.com/kedro-org/kedro/pull/2931
Suppose for runtime params (from the CLI) you get -
```python
runtime_params = {"_base_path": "<some_other_path>", "random_state":5}
```
Catalog.yml -
```yaml
cars:
  filepath: ${_base_path}
  type: pandas.CSVDataSet

_base_path: <some_path>
```
parameters.yml -
```yaml
random_state : 45
split: 0.2
```
* For parameters we merge with runtime params with `OmegaConf.merge()`. Doesn't really matter if extra unnecessary params make it to `config_loader["parameters"]` inside a session run.
* For catalog (+ other config) : 
    1. Keep track of the original keys in the config
    2. Merge config + runtime params with `OmegeConf.merge()`
    3. Resolve variable interpolation
    4. Filter out keys that start with '_' and keys that weren't in the original keys (might have been intended for parameters etc).
This makes sure that you can't add any top level keys to the config through CLI commands.

Things to note : 
* You could overwrite other non top level keys through CLI params like `cars.type` even if it's not an interpolated variable.
* No special syntax

## Solution 2
Use a similar custom resolver approach to the globals implementation.
* Introduce a "runtime_params" resolver
* Have users explicitly specify they want this interpolation overwritten by run time params when available
catalog.yml
```yaml
cars:
  filepath: "${runtime_params:path, <some_default_path>}"
  type: pandas.CSVDataSet
```
Comments from @noklam on Slack 

> I now think that using a custom resolver may be a elegant solution. It comes with a few advantage:
> 1. No need to escape from the _ for catalog
> the resolver can optionally takes a default, i.e. if it exists in runtime_params then override it, otherwise takes the default.
> 2. More explicit name
> 3. This way we can handle kedro run --params my_base_path=<some_path>   elegantly without caring about the _ and merging issue.
> 4. is align with [Separate ](https://github.com/kedro-org/kedro/issues/2530)params[ to ](https://github.com/kedro-org/kedro/issues/2530)runtime_params[ and ](https://github.com/kedro-org/kedro/issues/2530)meta_params ticket, we used to suffer from the problems that “params” is overloaded for everything. We are still overloading params with this solution but at least we make the usage of it

Things to note:
* How does this align with the merging of params we do for "parameters"? Users would not need to use the resolver in the `parameters.yml` but in other configs. Would this be hard to explain?

❓ If we go with this solution 
* Q: What should be the name of the resolver? `runtime_params:` or something else?

Tagging for feedback: @noklam @merelcht 
One thought - what if we inject whatever exist in `--params` to `global`? This way we don't need an extra resolver.

Pro:
- Easy implementation
- No extra resolver

Con:
- Mixed `runtime_params` and `meta_params`(but consistent with current approach)
I like the resolver solution, but also think it's important to have consistency between how runtime resolution happens for parameters and other config files. So if the decision is to go with the resolver I'd vouch for also implementing that for parameters. 
I **really like the resolver approach**, because it makes the approach less magic and **more declarative** with fine grained level. Eventually this will *facilitate maitenance** because we can create helpers to show which variables are available for overriding (e.g. a CLI ``kedro config runtime`` which will return the list of runtime parameters)


I have a question though. **How will the (potential) ``runtime_params`` resolver interact with the ``globals`` resolver**? I guess these will often be used together: the ``globals`` resolver enable overriding from a config file and the ``runtime_params`` one from the CLI but they are two ways to achieve the same goal, e.g. overriding some configuration which is expected to change. 

Should we expect users be able to do something like that: 

```yaml
cars:
  filepath: "${runtime_params:path, ${globals:path, <some_default_path>}}"
  type: pandas.CSVDataSet
```

The syntax feels a bit cluttered and hard to read for beginners. 

Should we : 
1. create an extra resolver ``runtime_params_or_globals`` (name completely made up) which implements basically above combination and end up with 3 resolvers?
2.  merge both resolvers to end up with a single one which suppports both globals and runtime params at the same time? 
3. Eventually suggest how to do 1. in the documentation, but not implement it on the framework side, like you do with the spark hook? 
@Galileo-Galilei I have the same thought before and I try to summarize @ankatiyar thought.

- `globals` has 1 and only 1 entry point, which is the yaml file.
- If it need to be override from CLI, use `runtime_params`

In earlier comment I have an idea that we can inject `runtime_param` to `global` so you can override if `runtime_param`. This is less explicit and bring us closer to the TemplatedConfigLoader where everything is called `runtime_params`.

I try to summarise the feature that we want (I think we need to improve the terminology too):
1. Support overriding via CLI (any config, at least for `params` and `catalog`). i.e. kedro run --params base_path=my_s3_bucket, learning_rate=0.1 (This ticket)
2. template variable that can be used across environment ( "globals" resolver)
3. template variable that can be used within environment & same catagory (i.e. params or catalog) - OmegaConf variable interpolation

I think the question here is do we want to support overriding `global` via `runtime_params`, can the same be achieve with `runtime_params` already? IMO it's reasonable, but it also make it less explicit and harder to reason. (one or two extra hops to figure out where is the true value). Assuming we want to support this, there are many entry points that we can assign a value. They should be resolving in this order (Highest to Lowest): 
1. --param CLI (a.k.a. runtime_params) 
2. global resolver - i.e. {$global: value}
3. variable interpolation
4. environment variable

Essentially the more specific scope should takes higher priority.

We had a separate ticket that we explore using an extra CLI argument to make things more explicit, but in practice users may not really care about the difference.
- https://github.com/kedro-org/kedro/issues/2530
Summary from discussion on 8/09/2023 with @merelcht @noklam @amandakys:
* Implement the `runtime_params:` resolver solution.
* Allow users to overwrite `parameters` the usual way through CLI but also allow using `runtime_params:` resolver
* For non breaking release in `0.18.x`
    * Don't allow nesting of `globals` and `runtime_params` - 
    ```
    cars:
   filepath: "${runtime_params:path, ${globals:path, <some_default_path>}}"
   type: pandas.CSVDataSet
    ```
    * Don't allow the use of `runtime_params:` resolver in `globals.yml`
 * For breaking release `0.19.0`
     * Consider lifting restriction on nested resolvers
     * Consider lifting restriction on using `runtime_params:` resolver in `globals.yml`

Resolving patch:
diff --git a/kedro/config/omegaconf_config.py b/kedro/config/omegaconf_config.py
--- a/kedro/config/omegaconf_config.py
+++ b/kedro/config/omegaconf_config.py
@@ -11,7 +11,7 @@
 
 import fsspec
 from omegaconf import OmegaConf
-from omegaconf.errors import InterpolationResolutionError
+from omegaconf.errors import InterpolationResolutionError, UnsupportedInterpolationType
 from omegaconf.resolvers import oc
 from yaml.parser import ParserError
 from yaml.scanner import ScannerError
@@ -141,8 +141,18 @@ def __init__(  # noqa: too-many-arguments
             env=env,
             runtime_params=runtime_params,
         )
-
-    def __getitem__(self, key) -> dict[str, Any]:
+        try:
+            self._globals = self["globals"]
+        except MissingConfigException:
+            self._globals = {}
+
+    def __setitem__(self, key, value):
+        if key == "globals":
+            # Update the cached value at self._globals since it is used by the globals resolver
+            self._globals = value
+        super().__setitem__(key, value)
+
+    def __getitem__(self, key) -> dict[str, Any]:  # noqa: PLR0912
         """Get configuration files by key, load and merge them, and
         return them in the form of a config dictionary.
 
@@ -161,6 +171,10 @@ def __getitem__(self, key) -> dict[str, Any]:
         """
         # Allow bypassing of loading config from patterns if a key and value have been set
         # explicitly on the ``OmegaConfigLoader`` instance.
+
+        # Re-register runtime params resolver incase it was previously deactivated
+        self._register_runtime_params_resolver()
+
         if key in self:
             return super().__getitem__(key)
 
@@ -170,6 +184,10 @@ def __getitem__(self, key) -> dict[str, Any]:
             )
         patterns = [*self.config_patterns[key]]
 
+        if key == "globals":
+            # "runtime_params" resolver is not allowed in globals.
+            OmegaConf.clear_resolver("runtime_params")
+
         read_environment_variables = key == "credentials"
 
         processed_files: set[Path] = set()
@@ -178,9 +196,18 @@ def __getitem__(self, key) -> dict[str, Any]:
             base_path = str(Path(self.conf_source) / self.base_env)
         else:
             base_path = str(Path(self._fs.ls("", detail=False)[-1]) / self.base_env)
-        base_config = self.load_and_merge_dir_config(
-            base_path, patterns, key, processed_files, read_environment_variables
-        )
+        try:
+            base_config = self.load_and_merge_dir_config(
+                base_path, patterns, key, processed_files, read_environment_variables
+            )
+        except UnsupportedInterpolationType as exc:
+            if "runtime_params" in str(exc):
+                raise UnsupportedInterpolationType(
+                    "The `runtime_params:` resolver is not supported for globals."
+                )
+            else:
+                raise exc
+
         config = base_config
 
         # Load chosen env config
@@ -189,9 +216,18 @@ def __getitem__(self, key) -> dict[str, Any]:
             env_path = str(Path(self.conf_source) / run_env)
         else:
             env_path = str(Path(self._fs.ls("", detail=False)[-1]) / run_env)
-        env_config = self.load_and_merge_dir_config(
-            env_path, patterns, key, processed_files, read_environment_variables
-        )
+        try:
+            env_config = self.load_and_merge_dir_config(
+                env_path, patterns, key, processed_files, read_environment_variables
+            )
+        except UnsupportedInterpolationType as exc:
+            if "runtime_params" in str(exc):
+                raise UnsupportedInterpolationType(
+                    "The `runtime_params:` resolver is not supported for globals."
+                )
+            else:
+                raise exc
+
         # Destructively merge the two env dirs. The chosen env will override base.
         common_keys = config.keys() & env_config.keys()
         if common_keys:
@@ -209,6 +245,7 @@ def __getitem__(self, key) -> dict[str, Any]:
                 f"No files of YAML or JSON format found in {base_path} or {env_path} matching"
                 f" the glob pattern(s): {[*self.config_patterns[key]]}"
             )
+
         return config
 
     def __repr__(self):  # pragma: no cover
@@ -297,6 +334,7 @@ def load_and_merge_dir_config(  # noqa: too-many-arguments
             return OmegaConf.to_container(
                 OmegaConf.merge(*aggregate_config, self.runtime_params), resolve=True
             )
+
         return {
             k: v
             for k, v in OmegaConf.to_container(
@@ -322,15 +360,22 @@ def _register_globals_resolver(self):
             replace=True,
         )
 
+    def _register_runtime_params_resolver(self):
+        OmegaConf.register_new_resolver(
+            "runtime_params",
+            self._get_runtime_value,
+            replace=True,
+        )
+
     def _get_globals_value(self, variable, default_value=_NO_VALUE):
         """Return the globals values to the resolver"""
         if variable.startswith("_"):
             raise InterpolationResolutionError(
                 "Keys starting with '_' are not supported for globals."
             )
-        global_omegaconf = OmegaConf.create(self["globals"])
+        globals_oc = OmegaConf.create(self._globals)
         interpolated_value = OmegaConf.select(
-            global_omegaconf, variable, default=default_value
+            globals_oc, variable, default=default_value
         )
         if interpolated_value != _NO_VALUE:
             return interpolated_value
@@ -339,6 +384,19 @@ def _get_globals_value(self, variable, default_value=_NO_VALUE):
                 f"Globals key '{variable}' not found and no default value provided."
             )
 
+    def _get_runtime_value(self, variable, default_value=_NO_VALUE):
+        """Return the runtime params values to the resolver"""
+        runtime_oc = OmegaConf.create(self.runtime_params)
+        interpolated_value = OmegaConf.select(
+            runtime_oc, variable, default=default_value
+        )
+        if interpolated_value != _NO_VALUE:
+            return interpolated_value
+        else:
+            raise InterpolationResolutionError(
+                f"Runtime parameter '{variable}' not found and no default value provided."
+            )
+
     @staticmethod
     def _register_new_resolvers(resolvers: dict[str, Callable]):
         """Register custom resolvers"""


Test patch:
diff --git a/tests/config/test_omegaconf_config.py b/tests/config/test_omegaconf_config.py
--- a/tests/config/test_omegaconf_config.py
+++ b/tests/config/test_omegaconf_config.py
@@ -12,7 +12,7 @@
 import pytest
 import yaml
 from omegaconf import OmegaConf, errors
-from omegaconf.errors import InterpolationResolutionError
+from omegaconf.errors import InterpolationResolutionError, UnsupportedInterpolationType
 from omegaconf.resolvers import oc
 from yaml.parser import ParserError
 
@@ -879,3 +879,184 @@ def test_ignore_ipynb_checkpoints(self, tmp_path, mocker):
         with pytest.raises(ValueError, match="Duplicate keys found in"):
             # fail because of reading the hidden files and get duplicate keys
             conf["parameters"]
+
+    def test_runtime_params_resolution(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        base_catalog = tmp_path / _BASE_ENV / "catalog.yml"
+        runtime_params = {
+            "x": 45,
+            "dataset": {
+                "type": "pandas.CSVDataSet",
+            },
+        }
+        param_config = {
+            "my_runtime_param": "${runtime_params:x}",
+            "my_param_default": "${runtime_params:y,34}",  # y does not exist in globals
+        }
+        catalog_config = {
+            "companies": {
+                "type": "${runtime_params:dataset.type}",
+                "filepath": "data/01_raw/companies.csv",
+            },
+        }
+        _write_yaml(base_params, param_config)
+        _write_yaml(base_catalog, catalog_config)
+        conf = OmegaConfigLoader(
+            tmp_path, default_run_env="", runtime_params=runtime_params
+        )
+        # runtime are resolved correctly in parameter
+        assert conf["parameters"]["my_runtime_param"] == runtime_params["x"]
+        # The default value is used if the key does not exist
+        assert conf["parameters"]["my_param_default"] == 34
+        # runtime params are resolved correctly in catalog
+        assert conf["catalog"]["companies"]["type"] == runtime_params["dataset"]["type"]
+
+    def test_runtime_params_missing_default(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        runtime_params = {
+            "x": 45,
+        }
+        param_config = {
+            "my_runtime_param": "${runtime_params:NOT_EXIST}",
+        }
+        _write_yaml(base_params, param_config)
+        conf = OmegaConfigLoader(
+            tmp_path, default_run_env="", runtime_params=runtime_params
+        )
+        with pytest.raises(
+            InterpolationResolutionError,
+            match="Runtime parameter 'NOT_EXIST' not found and no default value provided.",
+        ):
+            conf["parameters"]["my_runtime_param"]
+
+    def test_runtime_params_in_globals_not_allowed(self, tmp_path):
+        base_globals = tmp_path / _BASE_ENV / "globals.yml"
+        local_globals = tmp_path / _DEFAULT_RUN_ENV / "globals.yml"
+        runtime_params = {
+            "x": 45,
+        }
+        base_globals_config = {
+            "my_global_var": "${runtime_params:x}",
+        }
+        local_globals_config = {
+            "my_local_var": "${runtime_params:x}",  # x does exist but shouldn't be allowed in globals
+        }
+        _write_yaml(base_globals, base_globals_config)
+        _write_yaml(local_globals, local_globals_config)
+
+        with pytest.raises(
+            UnsupportedInterpolationType,
+            match=r"The `runtime_params:` resolver is not supported for globals.",
+        ):
+            OmegaConfigLoader(
+                tmp_path,
+                base_env="",
+                default_run_env="local",
+                runtime_params=runtime_params,
+            )
+        with pytest.raises(
+            UnsupportedInterpolationType,
+            match=r"The `runtime_params:` resolver is not supported for globals.",
+        ):
+            OmegaConfigLoader(tmp_path, runtime_params=runtime_params)
+
+    def test_runtime_params_default_global(self, tmp_path):
+        base_globals = tmp_path / _BASE_ENV / "globals.yml"
+        base_catalog = tmp_path / _BASE_ENV / "catalog.yml"
+        runtime_params = {
+            "x": 45,
+        }
+        globals_config = {
+            "dataset": {
+                "type": "pandas.CSVDataSet",
+            }
+        }
+        catalog_config = {
+            "companies": {
+                "type": "${runtime_params:type, ${globals:dataset.type, 'MemoryDataSet'}}",
+                "filepath": "data/01_raw/companies.csv",
+            },
+        }
+        _write_yaml(base_catalog, catalog_config)
+        _write_yaml(base_globals, globals_config)
+        conf = OmegaConfigLoader(
+            tmp_path, default_run_env="", runtime_params=runtime_params
+        )
+        # runtime params are resolved correctly in catalog using global default
+        assert conf["catalog"]["companies"]["type"] == globals_config["dataset"]["type"]
+
+    def test_runtime_params_default_none(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        base_param_config = {
+            "zero": "${runtime_params: x.NOT_EXIST, 0}",
+            "null": "${runtime_params: x.NOT_EXIST, null}",
+            "null2": "${runtime_params: x.y}",
+        }
+        runtime_params = {
+            "x": {
+                "z": 23,
+                "y": None,
+            },
+        }
+        _write_yaml(base_params, base_param_config)
+        conf = OmegaConfigLoader(
+            tmp_path, default_run_env="", runtime_params=runtime_params
+        )
+        # Default value can be 0 or null
+        assert conf["parameters"]["zero"] == 0
+        assert conf["parameters"]["null"] is None
+        # runtime param value is null
+        assert conf["parameters"]["null2"] is None
+
+    def test_unsupported_interpolation_globals(self, tmp_path):
+        base_globals = tmp_path / _BASE_ENV / "globals.yml"
+        local_globals = tmp_path / _DEFAULT_RUN_ENV / "globals.yml"
+        runtime_params = {
+            "x": 45,
+        }
+        base_globals_config = {
+            "my_global_var": "${non_existent_resolver:33}",
+        }
+        local_globals_config = {
+            "my_local_var": "${non_existent_resolver:x}",
+        }
+        _write_yaml(local_globals, local_globals_config)
+        _write_yaml(base_globals, base_globals_config)
+        # Test that the `runtime_params` error message is not shown for cases other than
+        # when `runtime_params` resolver is used in `globals`
+        with pytest.raises(
+            UnsupportedInterpolationType,
+            match=r"Unsupported interpolation type non_existent_resolver",
+        ):
+            OmegaConfigLoader(tmp_path, runtime_params=runtime_params)
+        with pytest.raises(
+            UnsupportedInterpolationType,
+            match=r"Unsupported interpolation type non_existent_resolver",
+        ):
+            OmegaConfigLoader(
+                tmp_path,
+                base_env="",
+                default_run_env="local",
+                runtime_params=runtime_params,
+            )
+
+    def test_override_globals(self, tmp_path):
+        """When globals are bypassed, make sure that the correct overwritten values are used"""
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        base_globals = tmp_path / _BASE_ENV / "globals.yml"
+        param_config = {
+            "my_global": "${globals:x}",
+            "my_second_global": "${globals:new_key}",
+        }
+        globals_config = {
+            "x": 45,
+        }
+        _write_yaml(base_params, param_config)
+        _write_yaml(base_globals, globals_config)
+        conf = OmegaConfigLoader(tmp_path, default_run_env="")
+        conf["globals"] = {
+            "x": 89,
+            "new_key": 24,
+        }
+        assert conf["parameters"]["my_global"] == 89
+        assert conf["parameters"]["my_second_global"] == 24