Repository: kedro-org/kedro

Issue:
`OmegaConfigLoader` chokes when `.ipynb_checkpoints` contains checkpoint of `catalog.yml`
## Description
As per title.

## Context
I was migrating a project from `TemplatedConfigLoader` to `OmegaConfigLoader`, and got an error that was not previously happening.

Jupyter automatically creates checkpoints of files (both notebooks and plain text files) in an `.ipynb_checkpoints` directory:

```
$ find conf/base -name "catalog*.yml"
conf/base/.ipynb_checkpoints/catalog-checkpoint.yml
conf/base/catalog.yml
```

## Steps to Reproduce
1. Copy `conf/base/catalog.yml` to `conf/base/.ipynb_checkpoints/catalog-checkpoint.yml`
2. Try `kedro catalog list` while using `TemplatedConfigLoader`, everything works okay.
3. Switch to `OmegaConfigLoader` and try `kedro catalog list`, see error.

## Expected Result
`OmegaConfigLoader` behavior matches the other catalog files when there are duplicates.

```bash
$ kedro catalog list
DataSets in '__default__' pipeline:
  Datasets mentioned in pipeline:
    CSVDataSet:
    - openrepair-0_3-combined
    - openrepair-0_3-events-raw
    - openrepair-0_3-categories
    - openrepair-0_3
DataSets in 'data_processing' pipeline:
  Datasets mentioned in pipeline:
    CSVDataSet:
    - openrepair-0_3-combined
    - openrepair-0_3-events-raw
    - openrepair-0_3-categories
    - openrepair-0_3
```

## Actual Result

```bash
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/juan_cano/.micromamba/envs/kpolars310/bin/kedro:8 in <module>                             │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/cli/cl │
│ i.py:211 in main                                                                                 │
│                                                                                                  │
│   208 │   """                                                                                    │
│   209 │   _init_plugins()                                                                        │
│   210 │   cli_collection = KedroCLI(project_path=Path.cwd())                                     │
│ ❱ 211 │   cli_collection()                                                                       │
│   212                                                                                            │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1130 in  │
│ __call__                                                                                         │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/cli/cl │
│ i.py:139 in main                                                                                 │
│                                                                                                  │
│   136 │   │   )                                                                                  │
│   137 │   │                                                                                      │
│   138 │   │   try:                                                                               │
│ ❱ 139 │   │   │   super().main(                                                                  │
│   140 │   │   │   │   args=args,                                                                 │
│   141 │   │   │   │   prog_name=prog_name,                                                       │
│   142 │   │   │   │   complete_var=complete_var,                                                 │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1055 in  │
│ main                                                                                             │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1657 in  │
│ invoke                                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1657 in  │
│ invoke                                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:1404 in  │
│ invoke                                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/core.py:760 in   │
│ invoke                                                                                           │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/click/decorators.py:38 │
│ in new_func                                                                                      │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/cli/ca │
│ talog.py:56 in list_datasets                                                                     │
│                                                                                                  │
│    53 │                                                                                          │
│    54 │   session = _create_session(metadata.package_name, env=env)                              │
│    55 │   context = session.load_context()                                                       │
│ ❱  56 │   datasets_meta = context.catalog._data_sets  # pylint: disable=protected-access         │
│    57 │   catalog_ds = set(context.catalog.list())                                               │
│    58 │                                                                                          │
│    59 │   target_pipelines = pipeline or pipelines.keys()                                        │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/contex │
│ t/context.py:236 in catalog                                                                      │
│                                                                                                  │
│   233 │   │   │   KedroContextError: Incorrect ``DataCatalog`` registered for the project.       │
│   234 │   │                                                                                      │
│   235 │   │   """                                                                                │
│ ❱ 236 │   │   return self._get_catalog()                                                         │
│   237 │                                                                                          │
│   238 │   @property                                                                              │
│   239 │   def params(self) -> Dict[str, Any]:                                                    │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/framework/contex │
│ t/context.py:279 in _get_catalog                                                                 │
│                                                                                                  │
│   276 │   │                                                                                      │
│   277 │   │   """                                                                                │
│   278 │   │   # '**/catalog*' reads modular pipeline configs                                     │
│ ❱ 279 │   │   conf_catalog = self.config_loader["catalog"]                                       │
│   280 │   │   # turn relative paths in conf_catalog into absolute paths                          │
│   281 │   │   # before initializing the catalog                                                  │
│   282 │   │   conf_catalog = _convert_paths_to_absolute_posix(                                   │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/config/omegaconf │
│ _config.py:168 in __getitem__                                                                    │
│                                                                                                  │
│   165 │   │   │   base_path = str(Path(self.conf_source) / self.base_env)                        │
│   166 │   │   else:                                                                              │
│   167 │   │   │   base_path = str(Path(self._fs.ls("", detail=False)[-1]) / self.base_env)       │
│ ❱ 168 │   │   base_config = self.load_and_merge_dir_config(                                      │
│   169 │   │   │   base_path, patterns, read_environment_variables                                │
│   170 │   │   )                                                                                  │
│   171 │   │   config = base_config                                                               │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/config/omegaconf │
│ _config.py:272 in load_and_merge_dir_config                                                      │
│                                                                                                  │
│   269 │   │   │   file: set(config.keys()) for file, config in config_per_file.items()           │
│   270 │   │   }                                                                                  │
│   271 │   │   aggregate_config = config_per_file.values()                                        │
│ ❱ 272 │   │   self._check_duplicates(seen_file_to_keys)                                          │
│   273 │   │                                                                                      │
│   274 │   │   if not aggregate_config:                                                           │
│   275 │   │   │   return {}                                                                      │
│                                                                                                  │
│ /Users/juan_cano/.micromamba/envs/kpolars310/lib/python3.10/site-packages/kedro/config/omegaconf │
│ _config.py:311 in _check_duplicates                                                              │
│                                                                                                  │
│   308 │   │                                                                                      │
│   309 │   │   if duplicates:                                                                     │
│   310 │   │   │   dup_str = "\n".join(duplicates)                                                │
│ ❱ 311 │   │   │   raise ValueError(f"{dup_str}")                                                 │
│   312 │                                                                                          │
│   313 │   @staticmethod                                                                          │
│   314 │   def _resolve_environment_variables(config: Dict[str, Any]) -> None:                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Duplicate keys found in /Users/juan_cano/Projects/QuantumBlack 
Labs/talk-kedro-polars/conf/base/.ipynb_checkpoints/catalog-checkpoint.yml and /Users/juan_cano/Projects/QuantumBlack 
Labs/talk-kedro-polars/conf/base/catalog.yml: openrepair-0_3, openrepair-0_3-categories, openrepair-0_3-combined, 
openrepair-0_3-events-raw
```

## Workaround

Changing the file patterns is an effective workaround:

```python
# settings.py

CONFIG_LOADER_ARGS = {
    "config_patterns": {
        "catalog": ["catalog.yml", "**/catalog.yml"],
    }
}
```

## Your Environment
Include as many relevant details about the environment in which you experienced the bug:

* Kedro version used (`pip show kedro` or `kedro -V`): 0.18.7
* Python version used (`python -V`): 3.10.9
* Operating system and version: macOS Ventura



Discussion:

Resolving patch:
diff --git a/kedro/config/omegaconf_config.py b/kedro/config/omegaconf_config.py
--- a/kedro/config/omegaconf_config.py
+++ b/kedro/config/omegaconf_config.py
@@ -250,11 +250,12 @@ def load_and_merge_dir_config(  # noqa: too-many-arguments
                 f"or is not a valid directory: {conf_path}"
             )
 
-        paths = [
-            Path(each)
-            for pattern in patterns
-            for each in self._fs.glob(Path(f"{str(conf_path)}/{pattern}").as_posix())
-        ]
+        paths = []
+        for pattern in patterns:
+            for each in self._fs.glob(Path(f"{str(conf_path)}/{pattern}").as_posix()):
+                if not self._is_hidden(each):
+                    paths.append(Path(each))
+
         deduplicated_paths = set(paths)
         config_files_filtered = [
             path for path in deduplicated_paths if self._is_valid_config_path(path)
@@ -392,3 +393,11 @@ def _resolve_environment_variables(config: dict[str, Any]) -> None:
             OmegaConf.clear_resolver("oc.env")
         else:
             OmegaConf.resolve(config)
+
+    def _is_hidden(self, path: str):
+        """Check if path contains any hidden directory or is a hidden file"""
+        path = Path(path).resolve().as_posix()
+        parts = path.split(self._fs.sep)  # filesystem specific separator
+        HIDDEN = "."
+        # Check if any component (folder or file) starts with a dot (.)
+        return any(part.startswith(HIDDEN) for part in parts)


Test patch:
diff --git a/tests/config/test_omegaconf_config.py b/tests/config/test_omegaconf_config.py
--- a/tests/config/test_omegaconf_config.py
+++ b/tests/config/test_omegaconf_config.py
@@ -797,3 +797,43 @@ def test_bad_globals_underscore(self, tmp_path):
             match=r"Keys starting with '_' are not supported for globals.",
         ):
             conf["parameters"]["param2"]
+
+    @pytest.mark.parametrize(
+        "hidden_path", ["/User/.hidden/dummy.yml", "/User/dummy/.hidden.yml"]
+    )
+    def test_is_hidden_config(self, tmp_path, hidden_path):
+        conf = OmegaConfigLoader(str(tmp_path))
+        assert conf._is_hidden(hidden_path)
+
+    @pytest.mark.parametrize(
+        "hidden_path",
+        [
+            "/User/conf/base/catalog.yml",
+            "/User/conf/local/catalog/data_science.yml",
+            "/User/notebooks/../conf/base/catalog",
+        ],
+    )
+    def test_not_hidden_config(self, tmp_path, hidden_path):
+        conf = OmegaConfigLoader(str(tmp_path))
+        assert not conf._is_hidden(hidden_path)
+
+    def test_ignore_ipynb_checkpoints(self, tmp_path, mocker):
+        conf = OmegaConfigLoader(str(tmp_path), default_run_env=_BASE_ENV)
+        base_path = tmp_path / _BASE_ENV / "parameters.yml"
+        checkpoints_path = (
+            tmp_path / _BASE_ENV / ".ipynb_checkpoints" / "parameters.yml"
+        )
+
+        base_config = {"param1": "dummy"}
+        checkpoints_config = {"param1": "dummy"}
+
+        _write_yaml(base_path, base_config)
+        _write_yaml(checkpoints_path, checkpoints_config)
+
+        # read successfully
+        conf["parameters"]
+
+        mocker.patch.object(conf, "_is_hidden", return_value=False)  #
+        with pytest.raises(ValueError, match="Duplicate keys found in"):
+            # fail because of reading the hidden files and get duplicate keys
+            conf["parameters"]