Repository: kedro-org/kedro

Issue:
Fix Configuration Loader to Handle Paths Containing dots
## Description

When working within Databricks workflows, it's possible to create a job that utilizes a notebook hosted inside a repository. Each time the job is executed, the code is automatically cloned to a path that depends on the commit. In my case, I have hosted a notebook that runs the Kedro project within the `.azuredevops` directory.

To locate the current project root, we can employ a regular expression (regex) pattern:

```python
pattern = r"^/Workspace/Repos/.internal/(?!./.azuredevops)."
```

With this regex, we can effectively identify the project root. For example, one possible path is:

```
/Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/
```

However, there's an issue that arises when a period (dot) appears within the path. This problem is not limited to Databricks; it can also occur locally. It's important to note that the OmegaConfigLoader did not encounter any issues with this path until Kedro version 0.18.12.


## Context
Just running a normal kedro project. Which was working untill 0.18.12

## Steps to Reproduce
1. modify you project directory adding some `.`, for example if you a have a kedro project in `Users/yourname/kedro_project` you can create a new folder with some point `Users/yourname/.somefolder/kedro_project` 
2. [Second Step] Enable OmageConfigLoader
3. [And so on...]  Kedro run

## Expected Result
kedro should load the session

## Actual Result
The config loader is not be able to find any config file

```
`No files of YAML or JSON format found in /Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/conf/base or /Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/conf/databricks_dev matching the glob pattern(s): ['catalog*', 'catalog*/**', '**/catalog*']`
```



## Your Environment
<!-- Include as many relevant details about the environment in which you experienced the bug: -->

* Kedro version used (`pip show kedro` or `kedro -V`): 0.18.13
* Python version used (`python -V`): 3.10.12
* Operating system and version: mac



Discussion:
@erwinpaillacan Potential conflict requirements with https://github.com/kedro-org/kedro/pull/2977

Does it work with other ConfigLoader? And what will be the expected behavior? 

We can hard coded for. ipynb_checkpoints but doesn't seem ideal to me. 
I think we need to ignore internal folders but after project root, not before.
to allow to have project root with dots, which is useful in databricks workflows.

The expected behaviour would be to able to run the kedro project.

If I have my project_root in `~/Projects/Workspace/Repos/.internal/02476ba86f_commits/e3754f960754e93d5dadcd17a74901cf7fb65fda/` I would expect to ignore `.folders` inside this directory, but dont ignore `.internal` since is part of the project root


Just tested TemplatedConfigLoader and it works fine. 
@erwinpaillacan I agree that but we need something more generic. Conf is not necessary inside a project. While it may not be common, one may actually have their conf folder inside a hidden folder but outside project root.

I have some idea to fix this but will need further investigation. 
Potential root cause (not checked):

https://github.com/kedro-org/kedro/blob/955bc935d709ee85ecbb4c36e987162c2ae4f0ca/kedro/config/omegaconf_config.py#L258-L259

https://github.com/kedro-org/kedro/blob/955bc935d709ee85ecbb4c36e987162c2ae4f0ca/kedro/config/omegaconf_config.py#L394-L400
When addressing this we'd ideally find a solution for this problem as well as https://github.com/kedro-org/kedro/pull/2977. That fix seems to be the cause of this.
I fixed this locally as a quick bugfix like this. 

```
    def _is_hidden(self, path: str) -> bool:
        """Check if path contains any hidden directory or is a hidden file.

        Have to overwrite to fix a bug.
        """
        file_path = Path(path).resolve()
        conf_path = Path(self.conf_source).resolve()
        if file_path.is_relative_to(conf_path):
            relative_path = file_path.relative_to(conf_path).as_posix()
        else:
            # In the rare case the env points to a path.
            # We use if for unit testing
            env_path = Path(self.env).resolve()
            relative_path = file_path.relative_to(env_path).as_posix()
        # filesystem specific separator
        parts = relative_path.split(self._fs.sep)
        hidden = "."
        # Check if any component (folder or file) starts with a dot (.)
        return any(part.startswith(hidden) for part in parts)
```

Maybe a usefull starting point :). Believe it also still works for the original problem. Could make a PR if you would like.
@IngerMathilde Thank you for investigating this and sharing the solution! ⭐ 
Since this ticket is a part of the current sprint and a blocker for the next release, I'd like to take care of this PR!
If you do have the capacity, please check out [the Hacktober tickets](https://github.com/kedro-org/kedro/issues/3130)! We'd love to see contributions from you! :) 

Resolving patch:
diff --git a/kedro/config/omegaconf_config.py b/kedro/config/omegaconf_config.py
--- a/kedro/config/omegaconf_config.py
+++ b/kedro/config/omegaconf_config.py
@@ -451,7 +451,13 @@ def _resolve_environment_variables(config: dict[str, Any]) -> None:
 
     def _is_hidden(self, path: str):
         """Check if path contains any hidden directory or is a hidden file"""
-        path = Path(path).resolve().as_posix()
+        path = Path(path)
+        conf_path = Path(self.conf_source).resolve().as_posix()
+        if self._protocol == "file":
+            path = path.resolve()
+        path = path.as_posix()
+        if path.startswith(conf_path):
+            path = path.replace(conf_path, "")
         parts = path.split(self._fs.sep)  # filesystem specific separator
         HIDDEN = "."
         # Check if any component (folder or file) starts with a dot (.)


Test patch:
diff --git a/tests/config/test_omegaconf_config.py b/tests/config/test_omegaconf_config.py
--- a/tests/config/test_omegaconf_config.py
+++ b/tests/config/test_omegaconf_config.py
@@ -848,15 +848,19 @@ def test_is_hidden_config(self, tmp_path, hidden_path):
         assert conf._is_hidden(hidden_path)
 
     @pytest.mark.parametrize(
-        "hidden_path",
+        "hidden_path, conf_source",
         [
-            "/User/conf/base/catalog.yml",
-            "/User/conf/local/catalog/data_science.yml",
-            "/User/notebooks/../conf/base/catalog",
+            ("/User/project/conf/base/catalog.yml", "/User/project/conf"),
+            ("/User/project/conf/local/catalog/data_science.yml", "/User/project/conf"),
+            ("/User/project/notebooks/../conf/base/catalog", "/User/project/conf"),
+            (
+                "/User/.hidden/project/conf/base/catalog.yml",
+                "/User/.hidden/project/conf",
+            ),
         ],
     )
-    def test_not_hidden_config(self, tmp_path, hidden_path):
-        conf = OmegaConfigLoader(str(tmp_path))
+    def test_not_hidden_config(self, conf_source, hidden_path):
+        conf = OmegaConfigLoader(str(conf_source))
         assert not conf._is_hidden(hidden_path)
 
     def test_ignore_ipynb_checkpoints(self, tmp_path, mocker):