Make packaged builders skip non-supported file formats
State: open

This PR makes packaged builders skip non-supported file formats:
- Csv builder skips non-CSV files
- Analogously for the other builders

Fix #5849.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5850). All of your documentation changes will be reflected on that endpoint.

Good idea. @mariosasko!!!

Please note that before this PR, the files are not evenly distributed for archives: `_generate_examples` gets a list of iterators, one for each archive (uncompressed to a directory).

This change could create silent problems when loading files with extensions that are not listed here. For example

```python
load_dataset("text", data_files=["20230515.log"])
```

wouldn't even log anything to say that the file was ignored.

Maybe it's possible to do this at data files patterns resolution ?

e.g. in get_data_patterns_in_dataset_repository / get_data_patterns_locally we could return patterns that include the most common extension

@lhoestq the issue you evoke (.log files skipped by text builder if .log is not added to .txt as supported extension) persists whether you perform the skip at the pattern resolution or in the builder itself.

The solution is to add the .log extension (besides the .txt) as supported by text, independently of where we perform the skip (at pattern resolution or in the builder itself).

Additionally, at the time we call for pattern resolution, we do not know the builder class yet, so that we cannot pass specific file extensions. First we call data files pattern resolution, and afterwards we call `infer_module_for_data_files` and then know the builder class.

> @lhoestq the issue you evoke (.log files skipped by text builder if .log is not added to .txt as supported extension) persists whether you perform the skip at the pattern resolution or in the builder itself.

No I simply think it's a bad breaking change to not support

```python
load_dataset("<builder_name>", data_files=["path/to/file_with_unknown_or_no_extension"])
# or
load_dataset("<builder_name>", data_files=["https://url.to/file_with_unknown_or_no_extension"])
```

Idk if it's the easiest solution, but maybe it's possible to do the change only when inferring the patterns of dataset repositories. This should avoid this breaking change.

For example it could do something like that in `get_data_patterns_locally`

```python
    Input:

        my_dataset_repository/
        ├── README.md
        ├── banner.png
        ├── data0.csv
        ├── data1.csv
        └── data2.csv

    Output:

        {"train": ["**.csv"]}
```

instead of 

```python
    Output:

        {"train": ["**"]}
```

I agree with @lhoestq - it should still be possible to request parsing a file with a specific builder even if the file's extension is "invalid" for the builder, and only ignore non-supported file formats when inferring the patterns.

Therefore, if I understand correctly, what you suggest is:
- if the user passes a packaged builder to `load_dataset` (e.g. `load_dataset("csv",...`), then the *passed* `data_files` should not be filtered to remove unsupported extensions. No breaking change in this case
- if the user passes a no-script repo/folder to `load_dataset` (e.g. `load_dataset("my_dataset_repository",...`), then the *inferred* data files should be filtered to remove the extensions that are not supported by the inferred module name builder
  - if the user passes `data_files` as well, then I guess these should not be filtered, to avoid any breaking change as in the first case above

Yes that would be ideal imo !

I think this now fulfills all the requirements.

I find it a bit confusing to still be able to pass data_files that are going to be silently ignored based on the value of `only_supported_extensions`. My suggestion was to have the right data files pattern, not to filter a posteriori (sorry if my last message was confusing).

Having the right data files pattern would also allow users to inspect what's actually being loaded with
```
load_dataset_builder(...).config.data_files
```
and it would list exactly what data files are used.