connection issue 
Author: rabeehkarimimahabadi
State: closed

Hi
I am getting this connection issue, resulting in large failure on cloud, @lhoestq  I appreciate your help on this.

If I want to keep the codes the same, so not using save_to_disk, load_from_disk, but save the datastes in the way load_dataset reads from and copy the files in the same folder the datasets library reads from, could you assist me how this can be done, thanks

I tried to do read the data, save it to a path and then set HF_HOME, which does not work and this is still not reading from the old set path, could you assist me how to save the datasets in a path, and let dataset library read from this path to avoid connection issue. thanks

```
imdb = datasets.load_dataset("imdb")
imdb.save_to_disk("/idiap/temp/rkarimi/hf_datasets/imdb")
>>> os.environ["HF_HOME"]="/idiap/temp/rkarimi/hf_datasets/"
>>> imdb = datasets.load_dataset("imdb")
Reusing dataset imdb (/idiap/temp/rkarimi/cache_home_2/datasets/imdb/plain_text/1.0.0/90099cb476936b753383ba2ae6ab2eae419b2e87f71cd5189cb9c8e5814d12a3)
```

I tried afterwards to set HF_HOME in bash, this makes it read from it, but it cannot let dataset library load from the saved path and still  downloading data. could you tell me how to fix this issue @lhoestq  thanks 

Also this is on cloud, so I save them in a path, copy it to "another machine" to load the data

### Error stack

```
Traceback (most recent call last):
  File "./finetune_t5_trainer.py", line 344, in <module>
    main()
  File "./finetune_t5_trainer.py", line 232, in main
    for task in data_args.eval_tasks} if training_args.do_test else None
  File "./finetune_t5_trainer.py", line 232, in <dictcomp>
    for task in data_args.eval_tasks} if training_args.do_test else None
  File "/workdir/seq2seq/data/tasks.py", line 136, in get_dataset
    split = self.get_sampled_split(split, n_obs)
  File "/workdir/seq2seq/data/tasks.py", line 64, in get_sampled_split
    dataset = self.load_dataset(split)
  File "/workdir/seq2seq/data/tasks.py", line 454, in load_dataset
    split=split, script_version="master")
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/usr/local/lib/python3.6/dist-packages/datasets/load.py", line 263, in prepare_module
    head_hf_s3(path, filename=name, dataset=dataset)
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 200, in head_hf_s3
    return http_head(hf_bucket_url(identifier=identifier, filename=filename, use_cdn=use_cdn, dataset=dataset))
  File "/usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py", line 403, in http_head
    url, proxies=proxies, headers=headers, cookies=cookies, allow_redirects=allow_redirects, timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 104, in head
    return request('head', url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/glue/glue.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6d6c60a20>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))
```


Comments: Also, mayjor issue for me is the format issue, even if I go through changing the whole code to use load_from_disk, then if I do 

d = datasets.load_from_disk("imdb")
d = d["train"][:10] => the format of this is no more in datasets format
this is different from you call load_datasets("train[10]")

could you tell me how I can make the two datastes the same format @lhoestq 



> `
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/glue/glue.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7ff6d6c60a20>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))`

Do you have an internet connection on the machine ? Is there a proxy that might block requests to aws ?

> I tried to do read the data, save it to a path and then set HF_HOME, which does not work and this is still not reading from the old set path, could you assist me how to save the datasets in a path, and let dataset library read from this path to avoid connection issue. thanks

HF_HOME is used to specify the directory for the cache files of this library.
You can use save_to_disk and load_from_disk without changing the HF_HOME:
```python
imdb = datasets.load_dataset("imdb")
imdb.save_to_disk("/idiap/temp/rkarimi/hf_datasets/imdb")
imdb = datasets.load_from_disk("/idiap/temp/rkarimi/hf_datasets/imdb")
```

> could you tell me how I can make the two datastes the same format

Indeed they returns different things:
- `load_dataset` returns a `Dataset` object if the split is specified, or a `DatasetDict` if no split is given. Therefore `load_datasets("imdb", split="train[10]")` returns a `Dataset` object containing 10 elements.
- doing `d["train"][:10]` on a DatasetDict "d" gets the train split `d["train"]` as a `Dataset` object and then gets the first 10 elements as a dictionary