Add VCTK dataset
Opened by jaketae (closed)

Fixes #1837. 

Comments:
Hello @patrickvonplaten, I hope it's okay to ping you with a (dumb) question!

I've been trying to get `dl_manager.download_and_extract(_DL_URL)` to work with no avail. I verified that this is a problem on two different machines (lab server, GCP), so I doubt it's an issue with network connectivity. Here is the full trace.

```
(venv) (base) jaketae@jake-gpu1:~/documents/datasets$ datasets-cli test datasets/vctk --save_infos --all_configs
Testing builder 'main' (1/1)
Downloading and preparing dataset vctk/main to /home/jaketae/.cache/huggingface/datasets/vctk/main/0.9.2/2bfa52a93469fa9d6d4b1831c6511db5442b9f4e48620aef2bc3890d7a5268a8...
Traceback (most recent call last):
  File "/home/jaketae/documents/datasets/venv/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/home/jaketae/documents/datasets/src/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/home/jaketae/documents/datasets/src/datasets/commands/test.py", line 146, in run
    builder.download_and_prepare(
  File "/home/jaketae/documents/datasets/src/datasets/builder.py", line 593, in download_and_prepare
    self._download_and_prepare(
  File "/home/jaketae/documents/datasets/src/datasets/builder.py", line 659, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/jaketae/.cache/huggingface/modules/datasets_modules/datasets/vctk/2bfa52a93469fa9d6d4b1831c6511db5442b9f4e48620aef2bc3890d7a5268a8/vctk.py", line 76, in _split_generators
    root_path = dl_manager.download_and_extract(_DL_URL)
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 283, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 195, in download
    downloaded_path_or_paths = map_nested(
  File "/home/jaketae/documents/datasets/src/datasets/utils/py_utils.py", line 234, in map_nested
    return function(data_struct)
  File "/home/jaketae/documents/datasets/src/datasets/utils/download_manager.py", line 216, in _download
    return cached_path(url_or_filename, download_config=download_config)
  File "/home/jaketae/documents/datasets/src/datasets/utils/file_utils.py", line 298, in cached_path
    output_path = get_from_cache(
  File "/home/jaketae/documents/datasets/src/datasets/utils/file_utils.py", line 608, in get_from_cache
    raise ConnectionError(f"Couldn't reach {url}")
ConnectionError: Couldn't reach https://datashare.is.ed.ac.uk/bitstream/handle/10283/3443/VCTK-Corpus-0.92.zip
```

On my local, however, the URL correctly points to the download zip file. My admittedly naive guess is that the website is web-crawler or scraper proof (requiring specific headers, etc.), but I also think I might have just missed a very basic step in the process.

Apologies for the delayed PR, and TIA for the help!

Hey @jaketae, 

Hmm, yeah I don't know really either - the link also works correctly for me when doing:

```
wget https://datashare.is.ed.ac.uk/bitstream/handle/10283/3443/VCTK-Corpus-0.92.zip
```

I think however that I had a similar problem previously with Edinburgh's (`.ed.ac.uk`) websites which I solved with the following hack. Not sure if this could be the same problem here...
https://github.com/huggingface/datasets/blob/e1104ad5d3e83f8b1571e0d6fef4fdabf0a1fde5/datasets/ami/ami.py#L364



The AMI dataset is stored under a different website though it seems: `"https://groups.inf.ed.ac.uk/ami/AMICorpusMirror//amicorpus/{}/audio/{}"`

so not 100p sure if this solves the problem

Hi @patrickvonplaten,

Thanks for the feedback! Sadly, disabling multi-processing didn't cut it for me. 

I've been looking at VCTK code in [`torchaudio`](https://pytorch.org/audio/stable/_modules/torchaudio/datasets/vctk.html) and [`tfds`](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/vctk.py). I don't think they're using a hack to accomplish this, so I'll try to look into it to see if I can pinpoint the cause. I'll keep you in the loop here. Thank you!

Hi @patrickvonplaten, 

After more investigation, I found that simply increasing `etag_timeout` in `get_from_cache` from 10 to 100 solved it. However, unless I'm missing something, an issue is that `etag_timeout` is basically hard-coded as a default parameter because `cached_path`, which calls `get_from_cache` has no way of modifying the default. 

https://github.com/huggingface/datasets/blob/b25ac1d62670e7b339ed552ecc37846d2abd30c7/src/datasets/utils/file_utils.py#L298-L310

https://github.com/huggingface/datasets/blob/b25ac1d62670e7b339ed552ecc37846d2abd30c7/src/datasets/utils/file_utils.py#L497-L510


I can think of two solutions.

* Simply increase the default to 100
* Allow `etag_timeout` to be modifiable on a per-dataset basis by integrating it to `download_config` (maybe this is already supported?)

Thank you!

I think in this case we can increase the `etag_timeout` - what do you think @lhoestq @albertvillanova ?

Yes let's increase it to 100 for the moment. Later we can see if it really needed to move it into `download_config` or not

Thanks for the feedback @patrickvonplaten @lhoestq, I'll continue working on this in that direction!

Hello @patrickvonplaten, VCTK is ready for review! 

```python
>>> from datasets import load_dataset
>>> ds = load_dataset("vctk")
Using the latest cached version of the module from /home/lily/jt856/.cache/huggingface/modules/datasets_modules/datasets/vctk/b7aa278182de3a7aa2897cbd12c1e19f1af9840a2ead69a6d710fdbc1d2df02a (last modified on Sat Dec 25 00:47:31 2021) since it couldn't be found locally at vctk., or remotely on the Hugging Face Hub.
Reusing dataset vctk (/home/lily/jt856/.cache/huggingface/datasets/vctk/main/0.9.2/b7aa278182de3a7aa2897cbd12c1e19f1af9840a2ead69a6d710fdbc1d2df02a)
100%|████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 198.35it/s]
>>> len(ds["train"])
88156
>>> ds["train"][0]
{'speaker_id': 'p225', 'audio': {'path': '/home/lily/jt856/.cache/huggingface/datasets/downloads/extracted/8ed7dad05dfffdb552a3699777442af8e8ed11e656feb277f35bf9aea448f49e/wav48_silence_trimmed/p225/p225_001_mic1.flac', 'array': array([0.00485229, 0.00689697, 0.00619507, ..., 0.00811768, 0.00836182,
       0.00854492], dtype=float32), 'sampling_rate': 48000}, 'file': '/home/lily/jt856/.cache/huggingface/datasets/downloads/extracted/8ed7dad05dfffdb552a3699777442af8e8ed11e656feb277f35bf9aea448f49e/wav48_silence_trimmed/p225/p225_001_mic1.flac', 'text': 'Please call Stella.', 'text_id': '001', 'age': '23', 'gender': 'F', 'accent': 'English', 'region': 'Southern England', 'comment': ''}
```
A number of tests are failing on CircleCI, but from my limited knowledge they appear to be complaining about `conda` and `pip`/`wheel`-related incompatibilities. But if I'm reading them wrong and it's an issue with this PR, please let me know and I'll try to fix them.

Belated merry Christmas and a happy new year!