version2.3.2   load_dataset()data_files can't include .xxxx in path
Opened by BUAAChuanWang (closed)

### Describe the bug

First, I cd workdir.
Then, I just use load_dataset("json", data_file={"train":"/a/b/c/.d/train/train.json", "test":"/a/b/c/.d/train/test.json"})
that couldn't work and 
<FileNotFoundError: Unable to find 
'/a/b/c/.d/train/train.jsonl' at 
/a/b/c/.d/>

And I debug, it is fine in version2.1.2

So there maybe a bug in path join.

Here is the whole bug report:
/x/datasets/loa │
│ d.py:1656 in load_dataset                                                    │
│                                                                              │
│   1653 │   ignore_verifications = ignore_verifications or save_infos         │
│   1654 │                                                                     │
│   1655 │   # Create a dataset builder                                        │
│ ❱ 1656 │   builder_instance = load_dataset_builder(                          │
│   1657 │   │   path=path,                                                    │
│   1658 │   │   name=name,                                                    │
│   1659 │   │   data_dir=data_dir,                                            │
│                                                                              │
│ x/datasets/loa │
│ d.py:1439 in load_dataset_builder                                            │
│                                                                              │
│   1436 │   if use_auth_token is not None:                                    │
│   1437 │   │   download_config = download_config.copy() if download_config e │
│   1438 │   │   download_config.use_auth_token = use_auth_token               │
│ ❱ 1439 │   dataset_module = dataset_module_factory(                          │
│   1440 │   │   path,                                                         │
│   1441 │   │   revision=revision,                                            │
│   1442 │   │   download_config=download_config,                              │
│                                                                              │
│ x/datasets/loa │
│ d.py:1097 in dataset_module_factory                                          │
│                                                                              │
│   1094 │                                                                     │
│   1095 │   # Try packaged                                                    │
│   1096 │   if path in _PACKAGED_DATASETS_MODULES:                            │
│ ❱ 1097 │   │   return PackagedDatasetModuleFactory(                          │
│   1098 │   │   │   path,                                                     │
│   1099 │   │   │   data_dir=data_dir,                                        │
│   1100 │   │   │   data_files=data_files,                                    │
│                                                                              │
│x/datasets/loa │
│ d.py:743 in get_module                                                       │
│                                                                              │
│    740 │   │   │   if self.data_dir is not None                              │
│    741 │   │   │   else get_patterns_locally(str(Path().resolve()))          │
│    742 │   │   )                                                             │
│ ❱  743 │   │   data_files = DataFilesDict.from_local_or_remote(              │
│    744 │   │   │   patterns,                                                 │
│    745 │   │   │   use_auth_token=self.download_config.use_auth_token,       │
│    746 │   │   │   base_path=str(Path(self.data_dir).resolve()) if self.data │
│                                                                              │
│ x/datasets/dat │
│ a_files.py:590 in from_local_or_remote                                       │
│                                                                              │
│   587 │   │   out = cls()                                                    │
│   588 │   │   for key, patterns_for_key in patterns.items():                 │
│   589 │   │   │   out[key] = (                                               │
│ ❱ 590 │   │   │   │   DataFilesList.from_local_or_remote(                    │
│   591 │   │   │   │   │   patterns_for_key,                                  │
│   592 │   │   │   │   │   base_path=base_path,                               │
│   593 │   │   │   │   │   allowed_extensions=allowed_extensions,             │
│                                                                              │
│ /x/datasets/dat │
│ a_files.py:558 in from_local_or_remote                                       │
│                                                                              │
│   555 │   │   use_auth_token: Optional[Union[bool, str]] = None,             │
│   556 │   ) -> "DataFilesList":                                              │
│   557 │   │   base_path = base_path if base_path is not None else str(Path() │
│ ❱ 558 │   │   data_files = resolve_patterns_locally_or_by_urls(base_path, pa │
│   559 │   │   origin_metadata = _get_origin_metadata_locally_or_by_urls(data │
│   560 │   │   return cls(data_files, origin_metadata)                        │
│   561                                                                        │
│                                                                              │
│ /x/datasets/dat │
│ a_files.py:195 in resolve_patterns_locally_or_by_urls                        │
│                                                                              │
│   192 │   │   if is_remote_url(pattern):                                     │
│   193 │   │   │   data_files.append(Url(pattern))                            │
│   194 │   │   else:                                                          │
│ ❱ 195 │   │   │   for path in _resolve_single_pattern_locally(base_path, pat │
│   196 │   │   │   │   data_files.append(path)                                │
│   197 │                                                                      │
│   198 │   if not data_files:                                                 │
│                                                                              │
│ /x/datasets/dat │
│ a_files.py:145 in _resolve_single_pattern_locally                            │
│                                                                              │
│   142 │   │   error_msg = f"Unable to find '{pattern}' at {Path(base_path).r │
│   143 │   │   if allowed_extensions is not None:                             │
│   144 │   │   │   error_msg += f" with any supported extension {list(allowed │
│ ❱ 145 │   │   raise FileNotFoundError(error_msg)                             │
│   146 │   return sorted(out)                                                 │
│   147                         

### Steps to reproduce the bug

1. Version=2.3.2
2. In shell, cd workdir.(cd /a/b/c/.d/)
3. load_dataset("json", data_file={"train":"/a/b/c/.d/train/train.json", "test":"/a/b/c/.d/train/test.json"})

### Expected behavior

fix it please~

### Environment info

2.3.2

Comments:
Version 2.3.2 is over one year old, so please use the latest release (2.14.0) to get the expected behavior. Version 2.3.2 does not contain some fixes we made to fix resolving hidden files/directories (starting with a dot).