storage_options provided to load_dataset not fully piping through since datasets 2.14.0
Opened by exs-avianello (closed)

### Describe the bug

Since the latest release of `datasets` (`2.14.0`), custom filesystem `storage_options` passed to `load_dataset()` do not seem to propagate through all the way - leading to problems if loading data files that need those options to be set.

I think this is because of the new `_prepare_path_and_storage_options()` (https://github.com/huggingface/datasets/pull/6028), which returns the right `storage_options` to use given a path and a `DownloadConfig` - but which might not be taking into account the extra `storage_options` explicitly provided e.g. through `load_dataset()`

### Steps to reproduce the bug

```python
import fsspec

import pandas as pd
import datasets

# Generate mock parquet file
data_files = "demo.parquet"
pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}).to_parquet(data_files)

_storage_options = {"x": 1, "y": 2}
fs = fsspec.filesystem("file", **_storage_options)

dataset = datasets.load_dataset(
    "parquet",
    data_files=data_files,
    storage_options=fs.storage_options
)
```

Looking at the `storage_options` resolved here:

https://github.com/huggingface/datasets/blob/b0177910b32712f28d147879395e511207e39958/src/datasets/data_files.py#L331

 they end up being `{}`, instead of propagating through the `storage_options` that were provided to `load_dataset` (`fs.storage_options`). As these then get used for the filesystem operation a few lines below

https://github.com/huggingface/datasets/blob/b0177910b32712f28d147879395e511207e39958/src/datasets/data_files.py#L339

the call will fail if the user-provided `storage_options` were needed.

---

A temporary workaround that seemed to work locally to bypass the problem was to bundle a duplicate of the `storage_options` into the `download_config`, so that they make their way all the way to `_prepare_path_and_storage_options()` and get extracted correctly:

```python
dataset = datasets.load_dataset(
        "parquet",
        data_files=data_files,
        storage_options=fs.storage_options,
        download_config=datasets.DownloadConfig(storage_options={fs.protocol: fs.storage_options}),
    )
```

### Expected behavior

`storage_options` provided to `load_dataset` take effect in all backend filesystem operations.

### Environment info

datasets==2.14.0

Comments:
Hi ! Thanks for reporting, I opened a PR to fix this

What filesystem are you using ?

Hi @lhoestq ! Thank you so much 🙌 

It's a bit of a custom setup, but in practice I am using a [pyarrow.fs.S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html) (wrapped in a `fsspec.implementations.arrow.ArrowFSWrapper` [to make it](https://arrow.apache.org/docs/python/filesystems.html#using-arrow-filesystems-with-fsspec) `fsspec` compatible). I also register it as an entrypoint with `fsspec` so that it's the one that gets automatically resolved when looking for filesystems for the `s3` protocol

In my case the `storage_option` that seemed not getting piped through was the filesystem's `endpoint_override` that I use in some tests to point at a mock S3 bucket