add support for storage_options for load_dataset API
Opened by janineguo (closed)

to solve the issue in #5880 

1.  add s3 support in the link check step, previous we only check `http` and `https`,

2. change the parameter of `use_auth_token` to `download_config` to support both `storage_options` and `use_auth_token` parameter when trying to handle(list, open, read, etc,.) the remote files.

3.  integrate the check part's duplicate code to make adding or deleting other sources easier.

Comments:
hi @lhoestq,
I saw some errors in my test and found all the failed reasons are `FileNotFoundError`  about `test_load_streaming_private_dataset_with_zipped_data` and `test_load_dataset_private_zipped_images` in `test_load.py `, I run pytest on my own Wins and Ubuntu system all the test in `test_load.py ` are succeed. could you help me to check the test environment of our server?

`2023-06-08T16:50:48.0828281Z FAILED tests/test_load.py::test_load_streaming_private_dataset_with_zipped_data - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_txt_data-16862429577813\repo_zipped_txt_data-16862429577813.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_txt_data-16862429577813`
`2023-06-08T16:50:48.0830602Z FAILED tests/test_load.py::test_load_dataset_private_zipped_images[False-False] - FileNotFoundError: Couldn't find a dataset script at D:\a\datasets\datasets\__DUMMY_TRANSFORMERS_USER__\repo_zipped_img_data-16862429594168\repo_zipped_img_data-16862429594168.py or any data file in the same directory. Couldn't find '__DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168' on the Hugging Face Hub either: FileNotFoundError: No (supported) data files or dataset script found in __DUMMY_TRANSFORMERS_USER__/repo_zipped_img_data-16862429594168`

I just re-ran the CI, hopefully it's fixed

_The documentation is not available anymore as the PR was closed or merged._

> I just re-ran the CI, hopefully it's fixed

I just checked, still has the same error, maybe need someone to fix it

I think the issue comes from this PR somehow, since the CI fail is related to loading private repositories and this PR touches authentication related code. Let me check what's the issue, and I'll also review your PR later (sorry I don't have a ton of bandwidth atm)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5919). All of your documentation changes will be reflected on that endpoint.

@lhoestq Hi sorry to bother you, the CI check_code_quality failed and it said `would reformat /home/runner/work/datasets/datasets/src/datasets/download/streaming_download_manager.py` but I cant see any changes when I run `python3 -m black --check tests src benchmarks metrics` and `python3 -m ruff tests src benchmarks metrics` on my own computer, is there any version requirements on the tools? I didn't specific the version.

I just ran `make style` and pushed the changes.
You can install the right versions of black and ruff using `pip install -e .[quality]` ;)

I am working on this issue right now https://github.com/huggingface/datasets/issues/6017 which is strongly connected to your PR, and I might end up cherry-picking some of your commits (keeping attribution of course !). Would you be ok with that ?

it's totally ok for me, I just wish the S3 File system could support streaming too.



I already adjust the code and test on my local Mac, you can check it now, and you can make any changes to it.

Closing this PR in favor of https://github.com/huggingface/datasets/pull/6028 which includes your contribution :)