Repository: kedro-org/kedro

Issue:
Add globals functionality to `OmegaConfigLoader` using custom "globals" resolver
# Description
After the introduction of variable interpolation in `OmegaConfigLoader`, two issues still stand -
    * Sharing variables across different configs (catalog, parameters etc)
    * Sharing variables across different envs (base, local etc)

This will be resolved by allowing global variables for configs. 

# Possible Implementation
Proposed by @marrrcin 🥇 

To use globals, in your project -
* Update `settings.py`
```python
CONFIG_LOADER_ARGS = {
    "globals_pattern": "*globals.yml"
}
```
* Add `globals.yml` to `conf/base/` or `conf/local`
* To use global values in `parameters.yml` or `catalog.yml` 
`globals.yml`
```yaml
datasets: 
    csv: pandas.CSVDataSet

test_size: 0.3
```
`catalog.yml`
```yaml
reviews:
  type: ${globals:"datasets.csv"}
  filepath: <path_to_file>
```
`parameters.yml`
```yaml
model_options:
    test_size: ${globals:"test_size"}
```
Here "`globals`" is a custom resolver - https://omegaconf.readthedocs.io/en/latest/custom_resolvers.html#id9


> **Q: Configurable `globals_pattern`?**
> A: Allow users to pass `globals_pattern` through `settings.py` but have it be `globals_pattern: {"*globals.yml"}` by default.
> 
> **Q: Where would `globals.yml` go?**
> A: The `globals.yml` files would stay within `conf/base/` and `conf/local/` instead of a top level `conf/globals.yml`. The common keys in `local/globals.yml` will overwrite the ones in `base/globals.yml`
> ```
> conf/
>   |_ base/
>     |_ globals.yml
>     |_ catalog.yml
>   |_ local/
>     |_ globals.yml
>     |_ catalog.yml
> ```
> 
> **Q: Soft or hard merge common keys?**
> A: Overwrite keys from `base/globals.yml` with keys from `local/globals.yml` for now but allow soft-merging in Kedro 0.19
> 
> **Q: Allow `globals_dict` like `TemplatedConfigLoader`?`**
> A: No, this use case for dynamically providing global values can be solved using "custom resolvers"
> 
> _Originally posted by @ankatiyar in https://github.com/kedro-org/kedro/issues/2175#issuecomment-1634269590_
>             

### TBD
Q: Do global variables need to start with "_"?
Q: What should the name of the custom resolver be? "`oc.global`", "`global`" etc?


Discussion:
The first point:

> - Update settings.py
> ```
> CONFIG_LOADER_ARGS = {
>     "globals_pattern": "*globals.yml"
> }
> ```

should be optional, with `*globals.yml` being default, right?
That's right @marrrcin, thanks, I missed mentioning it!

Resolving patch:
diff --git a/kedro/config/omegaconf_config.py b/kedro/config/omegaconf_config.py
--- a/kedro/config/omegaconf_config.py
+++ b/kedro/config/omegaconf_config.py
@@ -11,6 +11,7 @@
 
 import fsspec
 from omegaconf import OmegaConf
+from omegaconf.errors import InterpolationResolutionError
 from omegaconf.resolvers import oc
 from yaml.parser import ParserError
 from yaml.scanner import ScannerError
@@ -109,6 +110,7 @@ def __init__(  # noqa: too-many-arguments
             "parameters": ["parameters*", "parameters*/**", "**/parameters*"],
             "credentials": ["credentials*", "credentials*/**", "**/credentials*"],
             "logging": ["logging*", "logging*/**", "**/logging*"],
+            "globals": ["globals.yml"],
         }
         self.config_patterns.update(config_patterns or {})
 
@@ -117,7 +119,8 @@ def __init__(  # noqa: too-many-arguments
         # Register user provided custom resolvers
         if custom_resolvers:
             self._register_new_resolvers(custom_resolvers)
-
+        # Register globals resolver
+        self._register_globals_resolver()
         file_mimetype, _ = mimetypes.guess_type(conf_source)
         if file_mimetype == "application/x-tar":
             self._protocol = "tar"
@@ -199,7 +202,7 @@ def __getitem__(self, key) -> dict[str, Any]:
 
         config.update(env_config)
 
-        if not processed_files:
+        if not processed_files and key != "globals":
             raise MissingConfigException(
                 f"No files of YAML or JSON format found in {base_path} or {env_path} matching"
                 f" the glob pattern(s): {[*self.config_patterns[key]]}"
@@ -308,6 +311,36 @@ def _is_valid_config_path(self, path):
             ".json",
         ]
 
+    def _register_globals_resolver(self):
+        """Register the globals resolver"""
+        OmegaConf.register_new_resolver(
+            "globals",
+            lambda variable, default_value=None: self._get_globals_value(
+                variable, default_value
+            ),
+            replace=True,
+        )
+
+    def _get_globals_value(self, variable, default_value):
+        """Return the globals values to the resolver"""
+        if variable.startswith("_"):
+            raise InterpolationResolutionError(
+                "Keys starting with '_' are not supported for globals."
+            )
+        keys = variable.split(".")
+        value = self["globals"]
+        for k in keys:
+            value = value.get(k)
+            if not value:
+                if default_value:
+                    _config_logger.debug(
+                        f"Using the default value for the global variable {variable}."
+                    )
+                    return default_value
+                msg = f"Globals key '{variable}' not found and no default value provided. "
+                raise InterpolationResolutionError(msg)
+        return value
+
     @staticmethod
     def _register_new_resolvers(resolvers: dict[str, Callable]):
         """Register custom resolvers"""


Test patch:
diff --git a/tests/config/test_omegaconf_config.py b/tests/config/test_omegaconf_config.py
--- a/tests/config/test_omegaconf_config.py
+++ b/tests/config/test_omegaconf_config.py
@@ -12,6 +12,7 @@
 import pytest
 import yaml
 from omegaconf import OmegaConf, errors
+from omegaconf.errors import InterpolationResolutionError
 from omegaconf.resolvers import oc
 from yaml.parser import ParserError
 
@@ -671,3 +672,128 @@ def test_custom_resolvers(self, tmp_path):
         assert conf["parameters"]["model_options"]["param1"] == 7
         assert conf["parameters"]["model_options"]["param2"] == 3
         assert conf["parameters"]["model_options"]["param3"] == "my_env_variable"
+
+    def test_globals(self, tmp_path):
+        globals_params = tmp_path / _BASE_ENV / "globals.yml"
+        globals_config = {
+            "x": 34,
+        }
+        _write_yaml(globals_params, globals_config)
+        conf = OmegaConfigLoader(tmp_path, default_run_env="")
+        # OmegaConfigLoader has globals resolver
+        assert OmegaConf.has_resolver("globals")
+        # Globals is readable in a dict way
+        assert conf["globals"] == globals_config
+
+    def test_globals_resolution(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        base_catalog = tmp_path / _BASE_ENV / "catalog.yml"
+        globals_params = tmp_path / _BASE_ENV / "globals.yml"
+        param_config = {
+            "my_param": "${globals:x}",
+            "my_param_default": "${globals:y,34}",  # y does not exist in globals
+        }
+        catalog_config = {
+            "companies": {
+                "type": "${globals:dataset_type}",
+                "filepath": "data/01_raw/companies.csv",
+            },
+        }
+        globals_config = {"x": 34, "dataset_type": "pandas.CSVDataSet"}
+        _write_yaml(base_params, param_config)
+        _write_yaml(globals_params, globals_config)
+        _write_yaml(base_catalog, catalog_config)
+        conf = OmegaConfigLoader(tmp_path, default_run_env="")
+        assert OmegaConf.has_resolver("globals")
+        # Globals are resolved correctly in parameter
+        assert conf["parameters"]["my_param"] == globals_config["x"]
+        # The default value is used if the key does not exist
+        assert conf["parameters"]["my_param_default"] == 34
+        # Globals are resolved correctly in catalog
+        assert conf["catalog"]["companies"]["type"] == globals_config["dataset_type"]
+
+    def test_globals_nested(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        globals_params = tmp_path / _BASE_ENV / "globals.yml"
+        param_config = {
+            "my_param": "${globals:x}",
+            "my_nested_param": "${globals:nested.y}",
+        }
+        globals_config = {
+            "x": 34,
+            "nested": {
+                "y": 42,
+            },
+        }
+        _write_yaml(base_params, param_config)
+        _write_yaml(globals_params, globals_config)
+        conf = OmegaConfigLoader(tmp_path, default_run_env="")
+        assert conf["parameters"]["my_param"] == globals_config["x"]
+        # Nested globals are accessible with dot notation
+        assert conf["parameters"]["my_nested_param"] == globals_config["nested"]["y"]
+
+    def test_globals_across_env(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        local_params = tmp_path / _DEFAULT_RUN_ENV / "parameters.yml"
+        base_globals = tmp_path / _BASE_ENV / "globals.yml"
+        local_globals = tmp_path / _DEFAULT_RUN_ENV / "globals.yml"
+        base_param_config = {
+            "param1": "${globals:y}",
+        }
+        local_param_config = {
+            "param2": "${globals:x}",
+        }
+        base_globals_config = {
+            "x": 34,
+            "y": 25,
+        }
+        local_globals_config = {
+            "y": 99,
+        }
+        _write_yaml(base_params, base_param_config)
+        _write_yaml(local_params, local_param_config)
+        _write_yaml(base_globals, base_globals_config)
+        _write_yaml(local_globals, local_globals_config)
+        conf = OmegaConfigLoader(tmp_path)
+        # Local global overwrites the base global value
+        assert conf["parameters"]["param1"] == local_globals_config["y"]
+        # Base global value is accessible to local params
+        assert conf["parameters"]["param2"] == base_globals_config["x"]
+
+    def test_bad_globals(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        base_globals = tmp_path / _BASE_ENV / "globals.yml"
+        base_param_config = {
+            "param1": "${globals:x.y}",
+        }
+        base_globals_config = {
+            "x": {
+                "z": 23,
+            },
+        }
+        _write_yaml(base_params, base_param_config)
+        _write_yaml(base_globals, base_globals_config)
+        conf = OmegaConfigLoader(tmp_path, default_run_env="")
+        with pytest.raises(
+            InterpolationResolutionError,
+            match=r"Globals key 'x.y' not found and no default value provided.",
+        ):
+            conf["parameters"]["param1"]
+
+    def test_bad_globals_underscore(self, tmp_path):
+        base_params = tmp_path / _BASE_ENV / "parameters.yml"
+        base_globals = tmp_path / _BASE_ENV / "globals.yml"
+        base_param_config = {
+            "param2": "${globals:_ignore}",
+        }
+        base_globals_config = {
+            "_ignore": 45,
+        }
+        _write_yaml(base_params, base_param_config)
+        _write_yaml(base_globals, base_globals_config)
+        conf = OmegaConfigLoader(tmp_path, default_run_env="")
+        with pytest.raises(
+            InterpolationResolutionError,
+            match=r"Keys starting with '_' are not supported for globals.",
+        ):
+            conf["parameters"]["param2"]