Streaming Dataset behind Proxy - FileNotFoundError
Opened by Veluchs (closed)

### Describe the bug

When trying to stream a dataset i get the following error after a few minutes of waiting.

```
FileNotFoundError: https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json
If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

I have already set the proxy environment variables. Downloading a Dataset without streaming works as expected.
Still i suspect that this is connected to being behind a proxy.

Is there a way to set the proxy for streaming datasets? Possibly a keyword argument that gets passed to ffspec?

### Steps to reproduce the bug

This is the code i use.

```
import os
os.environ['http_proxy'] = "http://example.com:xxxx" 
os.environ['https_proxy'] = "http://example.com:xxxx" 


from datasets import load_dataset

ds = load_dataset("facebook/voxpopuli", name="de", streaming=True)
```

### Expected behavior

I would expect the streaming functionality to use the set proxy settings.

### Environment info


- `datasets` version: 2.13.0
- Platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.2


Comments:
Duplicate of #

Hi ! can you try to set the upper case environment variables `HTTP_PROXY` and `HTTPS_PROXY` ?

We use `aiohttp` for streaming and it uses case sensitive environment variables

Hi, thanks for the quick reply.

I set the uppercase env variables with

`
os.environ['HTTP_PROXY'] = "http://example.com:xxxx" 
os.environ['HTTPS_PROXY'] = "http://example.com:xxxx" 
`

However, I still get the same error.

One thing that could be helpfull: When downloading a dataset without streaming i get the following message:
_HF google storage unreachable. Downloading and preparing it from source_.
The download does however work as expected.


Are you able to use `aiohttp` to get the file at `https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json` using your proxy ?

It only works when passing trust_env=True when creating the ClientSession, as well as setting ssl=False.

Working Example:

```
import os

os.environ['HTTP_PROXY'] = "xyz"
os.environ['HTTPS_PROXY'] = "xyz"

import asyncio
import aiohttp

async def download_pep(url):
    async with aiohttp.ClientSession(trust_env=True) as session:
        print("1")
        async with session.get(url, ssl=False) as resp:
            print("2")
            content = await resp.text()
            print(content)
            return content

asyncio.run(download_pep("https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json"))
```



SSL Verification has been a problem with other packages as well. Usually I circumvent the problem by setting
```
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
```
(probably not the best idea for security), although here aiohttp does not seem to use this default context.

We do pass `trust_env` as well. Could you share the full stack trace you get when streaming using `datasets` ? That could help locate where we might have forgotten to pass `trust_env`

Is there a way to disable ssl verification when streaming a dataset. I suspect this might be the isssue with my proxy.


Here you go:

```
FileNotFoundError                         Traceback (most recent call last)
Cell In[8], line 3
      1 from datasets import load_dataset
----> 3 ds = load_dataset("facebook/voxpopuli", name="de", streaming=True)
      5 sample = next(iter(ds))

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/load.py:1790](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/load.py:1790), in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, verification_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, num_proc, storage_options, **config_kwargs)
   1788 # Return iterable dataset in case of streaming
   1789 if streaming:
-> 1790     return builder_instance.as_streaming_dataset(split=split)
   1792 # Some datasets are already processed on the HF google storage
   1793 # Don't try downloading from Google storage for the packaged datasets as text, json, csv or pandas
   1794 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/builder.py:1281](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/builder.py:1281), in DatasetBuilder.as_streaming_dataset(self, split, base_path)
   1274 dl_manager = StreamingDownloadManager(
   1275     base_path=base_path or self.base_path,
   1276     download_config=DownloadConfig(use_auth_token=self.use_auth_token, storage_options=self.storage_options),
   1277     dataset_name=self.name,
   1278     data_dir=self.config.data_dir,
   1279 )
   1280 self._check_manual_download(dl_manager)
-> 1281 splits_generators = {sg.name: sg for sg in self._split_generators(dl_manager)}
   1282 # By default, return all splits
   1283 if split is None:

File [~/.cache/huggingface/modules/datasets_modules/datasets/facebook--voxpopuli/b5ff837284f0778eefe0f642734e142d8c3f574eba8c9c8a4b13602297f73604/voxpopuli.py:120](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.cache/huggingface/modules/datasets_modules/datasets/facebook--voxpopuli/b5ff837284f0778eefe0f642734e142d8c3f574eba8c9c8a4b13602297f73604/voxpopuli.py:120), in Voxpopuli._split_generators(self, dl_manager)
    118 def _split_generators(self, dl_manager):
    119     n_shards_path = dl_manager.download_and_extract(_N_SHARDS_FILE)
--> 120     with open(n_shards_path) as f:
    121         n_shards = json.load(f)
    123     if self.config.name == "en_accented":

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/streaming.py:71](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/streaming.py:71), in extend_module_for_streaming..wrap_auth..wrapper(*args, **kwargs)
     69 @wraps(function)
     70 def wrapper(*args, **kwargs):
---> 71     return function(*args, use_auth_token=use_auth_token, **kwargs)

File [~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:517](https://vscode-remote+ssh-002dremote-002bml-002er-002dsoftware-002eat.vscode-resource.vscode-cdn.net/home/wrsbri/projects/audio_course/~/.conda/envs/audio_hf/lib/python3.10/site-packages/datasets/download/streaming_download_manager.py:517), in xopen(file, mode, use_auth_token, *args, **kwargs)
    515 except FileNotFoundError:
    516     if file.startswith(config.HF_ENDPOINT):
--> 517         raise FileNotFoundError(
    518             file + "\nIf the repo is private or gated, make sure to log in with `huggingface-cli login`."
    519         ) from None
    520     else:
    521         raise

FileNotFoundError: https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json
If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

> Is there a way to disable ssl verification when streaming a dataset.

I don't think so.

We use `fsspec` HTTPFileSystem implementation that is based on `aiohttp`. If you register a subclass of HTTPFileSystem that has SSL disabled by default it could work, but I wouldn't recommended it because it can raise security issues.

Okay thanks for your help! I guess I have to figure out how to improve the proxy environment / see if I can make it work with ssl connections.