Unexpected keyword argument 'hf' when downloading CSV dataset from S3
Opened by dguenms (closed)

### Describe the bug

I receive this error message when using `load_dataset` with "csv" path and `dataset_files=s3://...`:
```
TypeError: Session.__init__() got an unexpected keyword argument 'hf'
```

I found a similar issue here: https://stackoverflow.com/questions/77596258/aws-issue-load-dataset-from-s3-fails-with-unexpected-keyword-argument-error-in

Full stacktrace:
```
.../site-packages/datasets/load.py:2549: in load_dataset
    builder_instance.download_and_prepare(
.../site-packages/datasets/builder.py:1005: in download_and_prepare
    self._download_and_prepare(
.../site-packages/datasets/builder.py:1078: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
.../site-packages/datasets/packaged_modules/csv/csv.py:147: in _split_generators
    data_files = dl_manager.download_and_extract(self.config.data_files)
.../site-packages/datasets/download/download_manager.py:562: in download_and_extract
    return self.extract(self.download(url_or_urls))
.../site-packages/datasets/download/download_manager.py:426: in download
    downloaded_path_or_paths = map_nested(
.../site-packages/datasets/utils/py_utils.py:466: in map_nested
    mapped = [
.../site-packages/datasets/utils/py_utils.py:467: in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
.../site-packages/datasets/utils/py_utils.py:387: in _single_map_nested
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
.../site-packages/datasets/utils/py_utils.py:387: in <listcomp>
    mapped = [_single_map_nested((function, v, types, None, True, None)) for v in pbar]
.../site-packages/datasets/utils/py_utils.py:370: in _single_map_nested
    return function(data_struct)
.../site-packages/datasets/download/download_manager.py:451: in _download
    out = cached_path(url_or_filename, download_config=download_config)
.../site-packages/datasets/utils/file_utils.py:188: in cached_path
    output_path = get_from_cache(
...1/site-packages/datasets/utils/file_utils.py:511: in get_from_cache
    response = fsspec_head(url, storage_options=storage_options)
.../site-packages/datasets/utils/file_utils.py:316: in fsspec_head
    fs, _, paths = fsspec.get_fs_token_paths(url, storage_options=storage_options)
.../site-packages/fsspec/core.py:622: in get_fs_token_paths
    fs = filesystem(protocol, **inkwargs)
.../site-packages/fsspec/registry.py:290: in filesystem
    return cls(**storage_options)
.../site-packages/fsspec/spec.py:79: in __call__
    obj = super().__call__(*args, **kwargs)
.../site-packages/s3fs/core.py:187: in __init__
    self.s3 = self.connect()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <s3fs.core.S3FileSystem object at 0x1500a1310>, refresh = True

    def connect(self, refresh=True):
        """
        Establish S3 connection object.
    
        Parameters
        ----------
        refresh : bool
            Whether to create new session/client, even if a previous one with
            the same parameters already exists. If False (default), an
            existing one will be used if possible
        """
        if refresh is False:
            # back compat: we store whole FS instance now
            return self.s3
        anon, key, secret, kwargs, ckwargs, token, ssl = (
            self.anon, self.key, self.secret, self.kwargs,
            self.client_kwargs, self.token, self.use_ssl)
    
        if not self.passed_in_session:
>           self.session = botocore.session.Session(**self.kwargs)
E           TypeError: Session.__init__() got an unexpected keyword argument 'hf'
```

### Steps to reproduce the bug

1. Assuming a valid CSV file located at `s3://bucket/data.csv`
2. Run the below code:
```
storage_options = {
    "key": "...",
    "secret": "...",
    "client_kwargs": {
        "endpoint_url": "...",
    }
}
load_dataset("csv", data_files="s3://bucket/data.csv", storage_options=storage_options)
```

Encountered in version `2.16.1` but also reproduced in `2.16.0` and `2.15.0`.

Note: I encountered this in a unit test using a `moto` mock for S3, however since the error occurs before the session is instantiated, it should not be the issue.

### Expected behavior

No exception is raised, the boto3 session is created successfully, and the CSV file is downloaded successfully and returned as a dataset.

===

After some research I found that `DownloadConfig` has a `__post_init__` method that always forces this value to be set in its `storage_options`, even though in case of an S3 location the storage options get passed on to the S3 Session which does not expect this parameter. I assume this parameter is needed when reading from the huggingface hub and should not be set in this context.

Unfortunately there is nothing the user can do to work around it. Even if you manually do something like:
```
download_config = DownloadConfig()
del download_config.storage_options["hf"]
load_dataset("csv", data_files="s3://bucket/data.csv", download_config=download_config)
```
the library will still reinsert this parameter when `download_config = self.download_config.copy()` in line 418 of `download_manager.py` (`DownloadManager.download`).

Therefore `load_dataset` currently cannot be used to read a dataset in CSV format from an S3 location.

### Environment info

- `datasets` version: 2.16.1
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0


Comments:
I am facing similar issue while reading a csv file from s3. Wondering if somebody has found a workaround. 

same thing happened to other formats like parquet

I am facing similar issue while reading a parquet file from s3.
i try with every version between 2.14 to 2.16.1 but it dosen't  work 

Re-define the DownloadConfig might work:

```
class ReviseDownloadConfig(DownloadConfig):
    def __post_init__(self, use_auth_token):
        if use_auth_token != "deprecated":
            warnings.warn(
                "'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n"
                f"You can remove this warning by passing 'token={use_auth_token}' instead.",
                FutureWarning,
            )
            self.token = use_auth_token

    def copy(self):
        return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})

downloadconfig = ReviseDownloadConfig()
```


> Re-define the DownloadConfig might work:
> 
> ```
> class ReviseDownloadConfig(DownloadConfig):
>     def __post_init__(self, use_auth_token):
>         if use_auth_token != "deprecated":
>             warnings.warn(
>                 "'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.\n"
>                 f"You can remove this warning by passing 'token={use_auth_token}' instead.",
>                 FutureWarning,
>             )
>             self.token = use_auth_token
> ```
This seemed to work for me.


use pandas and then convert to `Dataset`

I am currently facing the same issue while using a custom loading script with files located in a remote S3 instance. I was using the `download_custom` functionality but now it is deprecated mentioning that I should use the native S3 loading, which is not working. 

As stated before, the library forces the existence of a `hf` key in the `storage_options` variable, which is **not** accepted by  `s3fs` : 

```python
.../site-packages/s3fs/core.py", line 516, in set_session
    self.session = aiobotocore.session.AioSession(**self.kwargs)
TypeError: __init__() got an unexpected keyword argument 'hf'.
````

Meanwhile, if my `storage_options` var stays like:
```python
{'key': '...',
 'secret': '...',
 'client_kwargs': {'endpoint_url': '...'}}
```
it works alright. 

Did anyone look into similar issues with model upload? setting s3 for checkpointing return `FileNotFoundError`