Repository: kedro-org/kedro
Issue category: Performance Issue

Issue:
Improve OmegaConfigLoader performance when global/variable interpolations are involved
## Description

Extending on the investigation in #3893 , OmegaConfigLoader lags in resolving catalog configurations when global/variable interpolations are involved.

## Context

**Some previous observations:**
https://github.com/kedro-org/kedro/issues/3893#issuecomment-2460629752
https://github.com/kedro-org/kedro/issues/3893#issuecomment-2465507178

## Steps to Reproduce

Run stress test which creates a single catalog with variable interpolations - https://github.com/kedro-org/kedro/blob/test/ocl-bottleneck/kedro_benchmarks/temp_investigate_ocl/ocl_plot_variables.py

## Expected Result

Reduce the time spent on below methods (when interpolations are involved, the bottleneck seems to be `OmegaConf.to_container`)

- [OmegaConf.load](https://omegaconf.readthedocs.io/en/2.3_branch/_modules/omegaconf/omegaconf.html#OmegaConf)
- [OmegaConf.to_container](https://omegaconf.readthedocs.io/en/2.3_branch/api_reference.html#omegaconf.OmegaConf.to_container)
- [OmegaConf.merge](https://omegaconf.readthedocs.io/en/2.3_branch/api_reference.html#omegaconf.OmegaConf.merge)

## Actual Result

https://github.com/kedro-org/kedro/issues/3893#issuecomment-2465507178

## Your Environment

<img width="245" alt="image" src="https://github.com/user-attachments/assets/07e3d00f-ad2b-41c6-863a-29e7d20d37c1">

* Kedro version used (`pip show kedro` or `kedro -V`): 0.19.9
* Python version used (`python -V`): 3.11
* Operating system and version: macOS and Sonoma 14.7.1



Hints:
During the investigation, I found that there was a slow bit in `_set_globals_value`. I didn't spent enough time to fix it, but with a quick fix it improves roughly from 1.5s -> 0.9s, but there are probably more.
Particularly, there is an obvious slow path `global_oc` get created and destroyed for every reference of `$globals`

https://github.com/kedro-org/kedro/blob/01c095b2537c1cfbafe6443a021e60ec94bfa1af/kedro/config/omegaconf_config.py#L433-L439

Patch:
diff --git a/RELEASE.md b/RELEASE.md
index 7af26f6ca4..c19418f0ca 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -2,6 +2,7 @@
 
 ## Major features and improvements
 * Implemented `KedroDataCatalog.to_config()` method that converts the catalog instance into a configuration format suitable for serialization.
+* Improve OmegaConfigLoader performance
 
 ## Bug fixes and other changes
 * Added validation to ensure dataset versions consistency across catalog.
diff --git a/kedro/config/omegaconf_config.py b/kedro/config/omegaconf_config.py
index 8d82ebf360..3f40624a4b 100644
--- a/kedro/config/omegaconf_config.py
+++ b/kedro/config/omegaconf_config.py
@@ -126,7 +126,8 @@ def __init__(  # noqa: PLR0913
         self.base_env = base_env or ""
         self.default_run_env = default_run_env or ""
         self.merge_strategy = merge_strategy or {}
-
+        self._globals_oc: DictConfig | None = None
+        self._runtime_params_oc: DictConfig | None = None
         self.config_patterns = {
             "catalog": ["catalog*", "catalog*/**", "**/catalog*"],
             "parameters": ["parameters*", "parameters*/**", "**/parameters*"],
@@ -346,12 +347,11 @@ def load_and_merge_dir_config(
                 OmegaConf.merge(*aggregate_config, self.runtime_params), resolve=True
             )
 
+        merged_config_container = OmegaConf.to_container(
+            OmegaConf.merge(*aggregate_config), resolve=True
+        )
         return {
-            k: v
-            for k, v in OmegaConf.to_container(
-                OmegaConf.merge(*aggregate_config), resolve=True
-            ).items()
-            if not k.startswith("_")
+            k: v for k, v in merged_config_container.items() if not k.startswith("_")
         }
 
     @staticmethod
@@ -436,9 +436,12 @@ def _get_globals_value(self, variable: str, default_value: Any = _NO_VALUE) -> A
             raise InterpolationResolutionError(
                 "Keys starting with '_' are not supported for globals."
             )
-        globals_oc = OmegaConf.create(self._globals)
+
+        if not self._globals_oc:
+            self._globals_oc = OmegaConf.create(self._globals)
+
         interpolated_value = OmegaConf.select(
-            globals_oc, variable, default=default_value
+            self._globals_oc, variable, default=default_value
         )
         if interpolated_value != _NO_VALUE:
             return interpolated_value
@@ -449,9 +452,11 @@ def _get_globals_value(self, variable: str, default_value: Any = _NO_VALUE) -> A
 
     def _get_runtime_value(self, variable: str, default_value: Any = _NO_VALUE) -> Any:
         """Return the runtime params values to the resolver"""
-        runtime_oc = OmegaConf.create(self.runtime_params)
+        if not self._runtime_params_oc:
+            self._runtime_params_oc = OmegaConf.create(self.runtime_params)
+
         interpolated_value = OmegaConf.select(
-            runtime_oc, variable, default=default_value
+            self._runtime_params_oc, variable, default=default_value
         )
         if interpolated_value != _NO_VALUE:
             return interpolated_value


Test patch: