Cannot load TREC dataset: ConnectionError
State: closed | Author: kaletap (NONE)
Labels: name: dataset bug

## Problem
I cannot load "trec" dataset, it results with ConnectionError as shown below. I've tried on both Google Colab and locally. 
* `requests.head('http://cogcomp.org/Data/QA/QC/train_5500.label')` returns <Response [302]>. 
* `requests.head('http://cogcomp.org/Data/QA/QC/train_5500.label', allow_redirects=True)` raises `requests.exceptions.TooManyRedirects: Exceeded 30 redirects.`
* Opening `http://cogcomp.org/Data/QA/QC/train_5500.label' in a browser works, but opens a different address
* Increasing max_redirects to 100 doesn't help

Also, while debugging I've seen that requesting 'https://storage.googleapis.com/huggingface-nlp/cache/datasets/trec/default/1.1.0/dataset_info.json' returns <Response [404]> before, but it doesn't raise any errors. Not sure if that's relevant.

* datasets.__version__ == '1.1.2'
* requests.__version__ == '2.24.0'

## Error trace
```
>>> import datasets
>>> datasets.__version__
'1.1.2'
>>> dataset = load_dataset("trec", split="train")
Using custom data configuration default
Downloading and preparing dataset trec/default (download: 350.79 KiB, generated: 403.39 KiB, post-processed: Unknown size, total: 754.18 KiB) to /home/przemyslaw/.cache/huggingface/datasets/trec/default/1.1.0/ca4248481ad244f235f4cf277186cad2ee8769f975119a2bbfc41b8932b88bd7...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/przemyslaw/.cache/huggingface/modules/datasets_modules/datasets/trec/ca4248481ad244f235f4cf277186cad2ee8769f975119a2bbfc41b8932b88bd7/trec.py", line 140, in _split_generators
    dl_files = dl_manager.download_and_extract(_URLs)
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/utils/download_manager.py", line 179, in download
    num_proc=download_config.num_proc,
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 225, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 225, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/utils/py_utils.py", line 163, in _single_map_nested
    return function(data_struct)
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/home/przemyslaw/.local/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 475, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label
```

I would appreciate some suggestions here. 

Comments: Hi ! Indeed there's an issue with those links.
We should probably use the target urls of the redirections instead

Hi, the same issue here, could you tell me how to download it through datasets? thanks 

Same issue. 

Actually it's already fixed on the master branch since #740 
I'll do the 1.1.3 release soon

Hi
thanks, but I did tried to install from the pip install git+... and it does
not work for me,. thanks for the help. I have the same issue with wmt16,
"ro-en"
thanks.
Best
Rabeeh

On Mon, Nov 16, 2020 at 10:29 AM Quentin Lhoest <notifications@github.com>
wrote:

> Actually it's already fixed on the master branch since #740
> <https://github.com/huggingface/datasets/pull/740>
> I'll do the 1.1.3 release soon
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/798#issuecomment-727854736>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCEUBJKPOCLABXCKMPDSQDWH3ANCNFSM4TJBUKSA>
> .
>


I just tested on google colab using
```python
!pip install git+https://github.com/huggingface/datasets.git
from datasets import load_dataset
load_dataset("trec")
```
and it works.
Can you detail how you got the issue even when using the latest version on master ?

Also about wmt we'll look into it, thanks for reporting !

I think the new url with .edu is also broken:
```
ConnectionError: Couldn't reach https://cogcomp.seas.upenn.edu/Data/QA/QC/train_5500.label
```
Cant download the dataset anymore.

Hi ! The URL seems to work fine on my side, can you try again ?

Forgot to update, i wrote an email to the webmaster of seas.upenn.edu because i couldnt reach the url on any machine. This was the answer:
```
Thank you for your report. The server was offline for maintenance and is now available again.
```
Guess all back to normal now 🙂