Private datasets do not load when passing token
Opened by albertvillanova (closed)

### Describe the bug

Since the release of `datasets` 2.14, private/gated datasets do not load when passing `token`: they raise  `EmptyDatasetError`.

This is a non-planned  backward incompatible breaking change.

Note that private datasets do load if instead `download_config` is passed:
```python
from datasets import DownloadConfig, load_dataset

ds = load_dataset("albertvillanova/tmp-private", split="train", download_config=DownloadConfig(token="<MY-TOKEN>"))
ds
```
gives
```
Dataset({
    features: ['text'],
    num_rows: 4
})
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("albertvillanova/tmp-private", split="train", token="<MY-TOKEN>")
```
gives
```
---------------------------------------------------------------------------
EmptyDatasetError                         Traceback (most recent call last)
[<ipython-input-2-25b48732107a>](https://localhost:8080/#) in <cell line: 3>()
      1 from datasets import load_dataset
      2 
----> 3 ds = load_dataset("albertvillanova/tmp-private", split="train", token="<MY-TOKEN>")

5 frames
[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, token, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   2107 
   2108     # Create a dataset builder
-> 2109     builder_instance = load_dataset_builder(
   2110         path=path,
   2111         name=name,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, token, use_auth_token, storage_options, **config_kwargs)
   1793         download_config = download_config.copy() if download_config else DownloadConfig()
   1794         download_config.storage_options.update(storage_options)
-> 1795     dataset_module = dataset_module_factory(
   1796         path,
   1797         revision=revision,

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1484                     raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
   1485                 if isinstance(e1, EmptyDatasetError):
-> 1486                     raise e1 from None
   1487                 if isinstance(e1, FileNotFoundError):
   1488                     raise FileNotFoundError(

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1474                     download_config=download_config,
   1475                     download_mode=download_mode,
-> 1476                 ).get_module()
   1477         except (
   1478             Exception

[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in get_module(self)
   1030             sanitize_patterns(self.data_files)
   1031             if self.data_files is not None
-> 1032             else get_data_patterns(base_path, download_config=self.download_config)
   1033         )
   1034         data_files = DataFilesDict.from_patterns(

[/usr/local/lib/python3.10/dist-packages/datasets/data_files.py](https://localhost:8080/#) in get_data_patterns(base_path, download_config)
    457         return _get_data_files_patterns(resolver)
    458     except FileNotFoundError:
--> 459         raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None
    460 
    461 

EmptyDatasetError: The directory at hf://datasets/albertvillanova/tmp-private@79b9e4fe79670a9a050d6ebc385464891915a71d doesn't contain any data files
```

### Expected behavior

The dataset should load.

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Our CI did not catch this issue because with current implementation, stored token in `HfFolder` (which always exists) is used by default.

I can confirm this and have the same problem (and just went almost crazy because I couldn't figure out the source of this problem because on another computer everything worked well even with `DownloadMode.FORCE_REDOWNLOAD`).

We are planning to do a patch release today, after the merge of the fix:
- #6127

In the meantime, the problem can be circumvented by passing `download_config` instead:
```python
from datasets import DownloadConfig, load_dataset

load_dataset("<DATASET-NAME>", split="train", download_config=DownloadConfig(token="<TOKEN>"))
``` 

> We are planning to do a patch release today, after the merge of the fix:
> 
>     * [Fix authentication issues #6127](https://github.com/huggingface/datasets/pull/6127)
> 
> 
> In the meantime, the problem can be circumvented by passing `download_config` instead:
> 
> ```python
> from datasets import DownloadConfig, load_dataset
> 
> load_dataset("<DATASET-NAME>", split="train", download_config=DownloadConfig(token="<TOKEN>"))
> ```

This did not work for me (there was some other error with the split being an unexpected size 0). Downgrading to 2.13 fixed it....