Repository: huggingface/datasets
Pull request: ignore dummy folder and dataset_infos.json
Fixes #2877

Added the `dataset_infos.json` to the ignored files list and also added check to ignore files which have parent directory as `dummy`.
Let me know if it is correct. Thanks :)

Problem statement:
Don't keep the dummy data folder or dataset_infos.json when resolving data files
When there's no dataset script, all the data files of a folder or a repository on the Hub are loaded as data files.

There are already a few exceptions:
- files starting with "." are ignored
- the dataset card "README.md" is ignored
- any file named "config.json" is ignored (currently it isn't used anywhere, but it could be used in the future to define splits or configs for example, but not 100% sure)

However any data files in a folder named "dummy" should be ignored as well as they should only be used to test the dataset.
Same for "dataset_infos.json" which should only be used to get the `dataset.info`

Hints from issue discussion:
Hi @lhoestq I am new to huggingface datasets, I would like to work on this issue!

Commit message: ignore dummy folder and dataset_infos.json

Patch to review:
diff --git a/src/datasets/load.py b/src/datasets/load.py
index e58e9e0737b..dd8c7125d7d 100644
--- a/src/datasets/load.py
+++ b/src/datasets/load.py
@@ -233,7 +233,7 @@ def _resolve_data_files_locally_or_by_urls(
     Return the absolute paths to all the files that match the given patterns.
     It also supports absolute paths in patterns.
     If an URL is passed, it is returned as is."""
-    data_files_ignore = ["README.md", "config.json"]
+    data_files_ignore = ["README.md", "config.json", "dataset_infos.json"]
     if isinstance(patterns, str):
         if is_remote_url(patterns):
             return [patterns]
@@ -245,7 +245,10 @@ def _resolve_data_files_locally_or_by_urls(
         matched_paths = [
             filepath.resolve()
             for filepath in glob_iter
-            if filepath.name not in data_files_ignore and not filepath.name.startswith(".") and filepath.is_file()
+            if filepath.name not in data_files_ignore
+            and not filepath.name.startswith(".")
+            and not str(filepath.parent).endswith("dummy")
+            and filepath.is_file()
         ]
         if allowed_extensions is not None:
             out = [
@@ -286,7 +289,7 @@ def _resolve_data_files_in_dataset_repository(
     patterns: Union[str, List[str], Dict],
     allowed_extensions: Optional[list] = None,
 ) -> Union[List[PurePath], Dict]:
-    data_files_ignore = ["README.md", "config.json"]
+    data_files_ignore = ["README.md", "config.json", "dataset_infos.json"]
     if isinstance(patterns, str):
         all_data_files = [
             PurePath("/" + dataset_file.rfilename) for dataset_file in dataset_info.siblings
@@ -296,6 +299,7 @@ def _resolve_data_files_in_dataset_repository(
             for filepath in all_data_files
             if filepath.name not in data_files_ignore
             and not filepath.name.startswith(".")
+            and not str(filepath.parent).endswith("dummy")
             and filepath.match(patterns)
         ]
         if allowed_extensions is not None:


Code review comments:
diff_hunk: @@ -245,7 +245,10 @@ def _resolve_data_files_locally_or_by_urls(
         matched_paths = [
             filepath.resolve()
             for filepath in glob_iter
-            if filepath.name not in data_files_ignore and not filepath.name.startswith(".") and filepath.is_file()
+            if filepath.name not in data_files_ignore
+            and not filepath.name.startswith(".")
+            and not str(filepath.parent).endswith("dummy")
+            and filepath.is_file()
path: src/datasets/load.py
text: @user1:
```suggestion
            if filepath.name not in data_files_ignore and not filepath.name.startswith(".") and filepath.is_file()
```

diff_hunk: @@ -233,7 +233,7 @@ def _resolve_data_files_locally_or_by_urls(
     Return the absolute paths to all the files that match the given patterns.
     It also supports absolute paths in patterns.
     If an URL is passed, it is returned as is."""
-    data_files_ignore = ["README.md", "config.json"]
+    data_files_ignore = ["README.md", "config.json", "dataset_infos.json"]
path: src/datasets/load.py
text: @user1:
```suggestion
    data_files_ignore = ["README.md", "config.json", "dataset_infos.json", "dummy_data.zip"]
```

diff_hunk: @@ -296,6 +299,7 @@ def _resolve_data_files_in_dataset_repository(
             for filepath in all_data_files
             if filepath.name not in data_files_ignore
             and not filepath.name.startswith(".")
+            and not str(filepath.parent).endswith("dummy")
path: src/datasets/load.py
text: @user1:
```suggestion
```

diff_hunk: @@ -286,7 +289,7 @@ def _resolve_data_files_in_dataset_repository(
     patterns: Union[str, List[str], Dict],
     allowed_extensions: Optional[list] = None,
 ) -> Union[List[PurePath], Dict]:
-    data_files_ignore = ["README.md", "config.json"]
+    data_files_ignore = ["README.md", "config.json", "dataset_infos.json"]
path: src/datasets/load.py
text: @user1:
```suggestion
    data_files_ignore = ["README.md", "config.json", "dataset_infos.json", "dummy_data.zip"]
```