Repository: kedro-org/kedro
Pull request: Make `OmegaConfigLoader` accept paths containing dots outside of conf_source
> **NOTE:** Kedro datasets are moving from `kedro.extras.datasets` to a separate `kedro-datasets` package in
> [`kedro-plugins` repository](https://github.com/kedro-org/kedro-plugins). Any changes to the dataset implementations
> should be done by opening a pull request in that repository.
## Description
<!-- Why was this PR created? -->
This PR is to fix #3004 

- Changes in #2977 which was a bug fix to let `OmegaConfigLoader` ignore hidden files inside the `conf/` folder (or `conf_source`) created this issue with Kedro ignoring files entirely if any part of the path contains a dot.

## Development notes
<!-- What have you changed, and how has this been tested? -->
- Updated `_is_hidden` function to only return `True` when the hidden folder/file is inside the `conf_source`
- Added a test case

EDITED TO ADD:

- Turns out `pathlib.is_relative_to()` and `pathlib.relative_to()` are methods that do not work for python <3.9. And, they're already deprecated and will be removed for 3.12.
- `os.path.rel_path()` is an alternate for this functionality but it works slightly differently from `pathlib.Path.relative_to()`. So some of the relative paths contain stuff like `../../some/path` which doesn't pass the "is_hidden()" test

#### Implementation info
* I first tried to remove `resolve()` since the `path` for files that is passed to this function is already a relative path (and not do any of the resolving with `conf_path`. But this doesn't work if something like `"../user1/catalog*"` has been added to `config_patterns` (as is the case in one of the tests)
```
    def _is_hidden(self, path: str):
        """Check if path contains any hidden directory or is a hidden file"""
        path = Path(path).as_posix()
        parts = path.split(self._fs.sep)  # filesystem specific separator
        HIDDEN = "."
        # Check if any component (folder or file) starts with a dot (.)
        return any(part.startswith(HIDDEN) for part in parts)
```
* Relative path resolving with `conf_path` trips up incase the `conf_source` is a zip/tar file. So if the protocol is zip/tar, it simply checks the "relative" path passed on to this function without `resolve()`


## Developer Certificate of Origin
We need all contributions to comply with the [Developer Certificate of Origin (DCO)](https://developercertificate.org/). All commits must be signed off by including a `Signed-off-by` line in the commit message. [See our wiki for guidance](https://github.com/kedro-org/kedro/wiki/Guidelines-for-contributing-developers/).

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

## Checklist

- [ ] Read the [contributing](https://github.com/kedro-org/kedro/blob/main/CONTRIBUTING.md) guidelines
- [ ] Signed off each commit with a [Developer Certificate of Origin (DCO)](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/managing-the-commit-signoff-policy-for-your-repository)
- [ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
- [ ] Updated the documentation to reflect the code changes
- [ ] Added a description of this change in the [`RELEASE.md`](https://github.com/kedro-org/kedro/blob/main/RELEASE.md) file
- [ ] Added tests to cover my changes
- [ ] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team


Problem statement:
Fix Configuration Loader to Handle Paths Containing dots
## Description

When working within Databricks workflows, it's possible to create a job that utilizes a notebook hosted inside a repository. Each time the job is executed, the code is automatically cloned to a path that depends on the commit. In my case, I have hosted a notebook that runs the Kedro project within the `.azuredevops` directory.

To locate the current project root, we can employ a regular expression (regex) pattern:

```python
pattern = r"^/Workspace/Repos/.internal/(?!./.azuredevops)."
```

With this regex, we can effectively identify the project root. For example, one possible path is:

```
/Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/
```

However, there's an issue that arises when a period (dot) appears within the path. This problem is not limited to Databricks; it can also occur locally. It's important to note that the OmegaConfigLoader did not encounter any issues with this path until Kedro version 0.18.12.


## Context
Just running a normal kedro project. Which was working untill 0.18.12

## Steps to Reproduce
1. modify you project directory adding some `.`, for example if you a have a kedro project in `Users/yourname/kedro_project` you can create a new folder with some point `Users/yourname/.somefolder/kedro_project` 
2. [Second Step] Enable OmageConfigLoader
3. [And so on...]  Kedro run

## Expected Result
kedro should load the session

## Actual Result
The config loader is not be able to find any config file

```
`No files of YAML or JSON format found in /Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/conf/base or /Workspace/Repos/.internal/02476ba86f_commits/20adcb96c8bec99ac2fad8b78b25158e7d968fa4/conf/databricks_dev matching the glob pattern(s): ['catalog*', 'catalog*/**', '**/catalog*']`
```



## Your Environment
<!-- Include as many relevant details about the environment in which you experienced the bug: -->

* Kedro version used (`pip show kedro` or `kedro -V`): 0.18.13
* Python version used (`python -V`): 3.10.12
* Operating system and version: mac

Hints from issue discussion:
@erwinpaillacan Potential conflict requirements with https://github.com/kedro-org/kedro/pull/2977

Does it work with other ConfigLoader? And what will be the expected behavior? 

We can hard coded for. ipynb_checkpoints but doesn't seem ideal to me. 
I think we need to ignore internal folders but after project root, not before.
to allow to have project root with dots, which is useful in databricks workflows.

The expected behaviour would be to able to run the kedro project.

If I have my project_root in `~/Projects/Workspace/Repos/.internal/02476ba86f_commits/e3754f960754e93d5dadcd17a74901cf7fb65fda/` I would expect to ignore `.folders` inside this directory, but dont ignore `.internal` since is part of the project root


Just tested TemplatedConfigLoader and it works fine. 
@erwinpaillacan I agree that but we need something more generic. Conf is not necessary inside a project. While it may not be common, one may actually have their conf folder inside a hidden folder but outside project root.

I have some idea to fix this but will need further investigation. 
Potential root cause (not checked):

https://github.com/kedro-org/kedro/blob/955bc935d709ee85ecbb4c36e987162c2ae4f0ca/kedro/config/omegaconf_config.py#L258-L259

https://github.com/kedro-org/kedro/blob/955bc935d709ee85ecbb4c36e987162c2ae4f0ca/kedro/config/omegaconf_config.py#L394-L400
When addressing this we'd ideally find a solution for this problem as well as https://github.com/kedro-org/kedro/pull/2977. That fix seems to be the cause of this.
I fixed this locally as a quick bugfix like this. 

```
    def _is_hidden(self, path: str) -> bool:
        """Check if path contains any hidden directory or is a hidden file.

        Have to overwrite to fix a bug.
        """
        file_path = Path(path).resolve()
        conf_path = Path(self.conf_source).resolve()
        if file_path.is_relative_to(conf_path):
            relative_path = file_path.relative_to(conf_path).as_posix()
        else:
            # In the rare case the env points to a path.
            # We use if for unit testing
            env_path = Path(self.env).resolve()
            relative_path = file_path.relative_to(env_path).as_posix()
        # filesystem specific separator
        parts = relative_path.split(self._fs.sep)
        hidden = "."
        # Check if any component (folder or file) starts with a dot (.)
        return any(part.startswith(hidden) for part in parts)
```

Maybe a usefull starting point :). Believe it also still works for the original problem. Could make a PR if you would like.
@IngerMathilde Thank you for investigating this and sharing the solution! ⭐ 
Since this ticket is a part of the current sprint and a blocker for the next release, I'd like to take care of this PR!
If you do have the capacity, please check out [the Hacktober tickets](https://github.com/kedro-org/kedro/issues/3130)! We'd love to see contributions from you! :) 
@IngerMathilde I've opened a PR for this #3145, would you mind sharing your email id so we can add you as a co-author of that PR? :) 
Hi!
Thanks a lot! You can put it on: 42641352+IngerMathilde@users.noreply.github.com

Commit message: Merge branch 'main' into bug/config-hidden-filepaths

Patch to review:
diff --git a/RELEASE.md b/RELEASE.md
index c79395c06c..c77b871184 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -15,6 +15,7 @@
 
 ## Bug fixes and other changes
 * Updated dataset factories to resolve nested catalog config properly.
+* Updated `OmegaConfigLoader` to handle paths containing dots outside of `conf_source`.
 
 ## Documentation changes
 * Added documentation to clarify execution order of hooks.
@@ -24,6 +25,7 @@
 Many thanks to the following Kedroids for contributing PRs to this release:
 
 * [Jason Hite](https://github.com/jasonmhite)
+* [IngerMathilde](https://github.com/IngerMathilde)
 
 
 # Release 0.18.13
diff --git a/kedro/config/omegaconf_config.py b/kedro/config/omegaconf_config.py
index 89fb7d1458..645b3aa75c 100644
--- a/kedro/config/omegaconf_config.py
+++ b/kedro/config/omegaconf_config.py
@@ -6,6 +6,7 @@
 import io
 import logging
 import mimetypes
+import os
 from pathlib import Path
 from typing import Any, Callable, Iterable
 
@@ -451,8 +452,13 @@ def _resolve_environment_variables(config: dict[str, Any]) -> None:
 
     def _is_hidden(self, path: str):
         """Check if path contains any hidden directory or is a hidden file"""
-        path = Path(path).resolve().as_posix()
-        parts = path.split(self._fs.sep)  # filesystem specific separator
+        path = Path(path)
+        conf_path = Path(self.conf_source).resolve().as_posix()
+        if self._protocol == "zip" or self._protocol == "tar":
+            rel_path = path.as_posix()
+        else:
+            rel_path = os.path.relpath(path.resolve().as_posix(), conf_path)
+        parts = rel_path.split(self._fs.sep)  # filesystem specific separator
         HIDDEN = "."
         # Check if any component (folder or file) starts with a dot (.)
         return any(part.startswith(HIDDEN) for part in parts)
diff --git a/tests/config/test_omegaconf_config.py b/tests/config/test_omegaconf_config.py
index 5f2fa2e67f..782a8df5db 100644
--- a/tests/config/test_omegaconf_config.py
+++ b/tests/config/test_omegaconf_config.py
@@ -848,15 +848,19 @@ def test_is_hidden_config(self, tmp_path, hidden_path):
         assert conf._is_hidden(hidden_path)
 
     @pytest.mark.parametrize(
-        "hidden_path",
+        "hidden_path, conf_source",
         [
-            "/User/conf/base/catalog.yml",
-            "/User/conf/local/catalog/data_science.yml",
-            "/User/notebooks/../conf/base/catalog",
+            ("/User/project/conf/base/catalog.yml", "/User/project/conf"),
+            ("/User/project/conf/local/catalog/data_science.yml", "/User/project/conf"),
+            ("/User/project/notebooks/../conf/base/catalog", "/User/project/conf"),
+            (
+                "/User/.hidden/project/conf/base/catalog.yml",
+                "/User/.hidden/project/conf",
+            ),
         ],
     )
-    def test_not_hidden_config(self, tmp_path, hidden_path):
-        conf = OmegaConfigLoader(str(tmp_path))
+    def test_not_hidden_config(self, conf_source, hidden_path):
+        conf = OmegaConfigLoader(str(conf_source))
         assert not conf._is_hidden(hidden_path)
 
     def test_ignore_ipynb_checkpoints(self, tmp_path, mocker):


Code review comments:
diff_hunk: @@ -6,6 +6,7 @@
 import io
 import logging
 import mimetypes
+import os
path: kedro/config/omegaconf_config.py
text: @user1:
We actually try to not use `os` if it's possible.. Is there no other way to achieve this with using `pathlib`?

@author:
Could actually implement the checking of relative paths logic ourselves?