Skip hidden files/directories in data files resolution and `iter_files`
Opened by mariosasko (closed)

Fix #4115 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This PR (via new release) broke many transformers tests.

I will try to post a summary shortly.

cc: @ydshieh 

So now it can't handle a local path via: `--train_file tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/train.json` even though it's there. it works just fine if I change the path to not have `..`

You can reproduce the original problem with:

```
$ cd transformers 
$ python examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --train_file tests/fixtures/tests_samples/wmt_en_ro/train.json --validation_file tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json --output_dir /tmp/tmp5o5to4k0 --overwrite_output_dir --max_source_length 32 --max_target_length 32 --val_max_target_length 32 --warmup_steps 8 --predict_with_generate --save_steps 0 --eval_steps 1 --group_by_length --label_smoothing_factor 0.1 --source_lang en --target_lang ro --report_to none --source_prefix "translate English to Romanian: " --fp16 --do_train --num_train_epochs 1 --max_train_samples 16 --per_device_train_batch_size 2 --learning_rate 3e-3
[...]
Traceback (most recent call last):
  File "examples/pytorch/translation/run_translation.py", line 656, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 346, in main
    raw_datasets = load_dataset(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1656, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1439, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 1097, in dataset_module_factory
    return PackagedDatasetModuleFactory(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/load.py", line 743, in get_module
    data_files = DataFilesDict.from_local_or_remote(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 588, in from_local_or_remote
    DataFilesList.from_local_or_remote(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 556, in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 194, in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/data_files.py", line 144, in _resolve_single_pattern_locally
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find '/mnt/nvme0/code/huggingface/transformers-master/tests/deepspeed/../fixtures/tests_samples/wmt_en_ro/val.json' at /mnt/nvme0/code/huggingface/transformers-master
```

will apply a workaround to `transformers` tests here https://github.com/huggingface/transformers/pull/17721


This has been fixed with https://github.com/huggingface/datasets/pull/4505, will do a patch release tomorrow for `datasets` ;)

Thank you for the quick fix, @lhoestq