SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Add TIMIT

Comment:
I am sorry! I created the PR [#1903](https://github.com/huggingface/datasets/pull/1903#). Requesting your comments! CircleCI tests are failing, will address them along with your comments!
Issue: Update Open Subtitles corpus with original sentence IDs

Comment:
Hi ! You're right this can can useful.
This should be easy to add, so feel free to give it a try if you want to contribute :)
I think we just need to add it to the _generate_examples method of the OpenSubtitles dataset builder [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L103)
Issue: Update Open Subtitles corpus with original sentence IDs

Comment:
Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format: 
(the following is line `22497315` of the `ids` file of the `de-en` dump)


`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)


Where filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)

That being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?

Going with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.
As for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?

**Note:** I did not find any official information about this encoding, but it appears to check out:
https://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.
Issue: Update Open Subtitles corpus with original sentence IDs

Comment:
I like the idea of having `year`, `imdbId` and `subtitleId` as columns for filtering for example.
And for the `sentenceIds` a list of integers is fine.
Issue: Update Open Subtitles corpus with original sentence IDs

Comment:
Something like this? (adapted from [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L114))

```python
result = (
  sentence_counter,
    {
      "id": str(sentence_counter),
      "meta": {
        "year": year,
        "imdbId": imdb_id,
        "subtitleId": {l1: l1_sub_id, l2: l2_sub_id},
        "sentenceIds": {l1: [... source_sids ...], l2: [... target_sids ...]},
        # or maybe src/tgt? I'd go with the first one for consistency with 'translation'
        "subtitleId": {"src": l1_sub_id, "tgt": l2_sub_id},
        "sentenceIds": {"src": [... source_sids ...], "tgt": [... target_sids ...]},
      },
      "translation": {l1: x, l2: y},
    },
  )
```
Or at top level, avoiding nesting into 'meta'?
Issue: MustC Speech Translation

Comment:
That's awesome! Actually, I just noticed that this dataset might become a bit too big!

MuST-C is the main dataset used for IWSLT19 and should probably be added as a standalone dataset. Would you be interested also in adding `datasets/MuST-C` instead?

Description: 
_MuST-C is a multilingual speech translation corpus whose size and quality facilitates the training of end-to-end systems for speech translation from English into several languages. For each target language, MuST-C comprises several hundred hours of audio recordings from English TED Talks, which are automatically aligned at the sentence level with their manual transcriptions and translations._

Paper: https://www.aclweb.org/anthology/N19-1202.pdf

Dataset: https://ict.fbk.eu/must-c/ (One needs to fill out a short from to download the data, but it's very easy).

It would be awesome if you're interested in adding this datates. I'm very happy to guide you through the PR! I think the easiest way to start would probably be to read [this README on how to add a dataset](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) and open a PR. Think you can copy & paste some code from:

- Librispeech_asr: https://github.com/huggingface/datasets/blob/master/datasets/librispeech_asr/librispeech_asr.py
- Flores Translation: https://github.com/huggingface/datasets/blob/master/datasets/flores/flores.py

Think all the rest can be handled on the PR :-)
Issue: MustC Speech Translation

Comment:
Hi @patrickvonplaten 
I have tried downloading this dataset, but the connection seems to reset all the time. I have tried it via the browser, wget, and using gdown . But it gives me an error message. _"The server is busy or down, pls try again"_ (rephrasing the message here)

I have completed adding 4 datasets in the previous data sprint (including the IWSLT dataset #1676 ) ...so just checking if you are able to download it at your end. Otherwise will write to the dataset authors to update the links.
Issue: MustC Speech Translation

Comment:
@skyprince999, I think I'm getting the same error you're getting :-/

```
Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
```

It would be great if you could write the authors to see whether they can fix it.
Also cc @lhoestq - do you think we could mirror the dataset?
Issue: MustC Speech Translation

Comment:
Also there are huge those datasets. Think downloading MuST-C v1.2 amounts to ~ 1000GB... because there are 14 possible configs each around 60-70GB. I think users mostly will only use one of the 14 configs so that they would only need, in theory, will have to download ~60GB which is ok. But I think this functionality doesn't exist yet in `datasets` no? cc @lhoestq
Issue: MustC Speech Translation

Comment:
> Also cc @lhoestq - do you think we could mirror the dataset?

Yes we can mirror it if the authors are fine with it. You can create a dataset repo on huggingface.co (possibly under the relevant org) and add the mirrored data files.

> I think users mostly will only use one of the 14 configs so that they would only need, in theory, will have to download ~60GB which is ok. But I think this functionality doesn't exist yet in datasets no? cc @lhoestq

If there are different download links for each configuration we can make the dataset builder download only the files related to the requested configuration.
Issue: MustC Speech Translation

Comment:
I have written to the dataset authors, highlighting this issue. Waiting for their response. 

Update on 25th Feb: 
The authors have replied back, they are updating the download link and will revert back shortly! 

```
first of all thanks a lot for being interested in MuST-C and for building the data-loader.

Before answering your request, I'd like to clarify that the creation, maintenance, and expansion of MuST-c are not supported by any funded project, so this means that we need to find economic support for all these activities. This also includes permanently moving all the data to AWS or GCP.  We are working at this with the goal of facilitating the use of MuST-C, but this is not something that can happen today. We hope to have some news ASAP and you will be among the first to be informed.

I hope you understand our situation.
```
Issue: MustC Speech Translation

Comment:
Awesome, actually @lhoestq let's just ask the authors if we should host the dataset no? They could just use our links then as well for their website - what do you think? Is it fine to use our AWS dataset storage also as external links?
Issue: MustC Speech Translation

Comment:
Yes definitely. Shall we suggest them to create a dataset repository under their org on huggingface.co ? @julien-c 
The dataset is around 1TB
Issue: MustC Speech Translation

Comment:
Sounds good! 

Order of magnitude is storage costs ~$20 per TB per month (not including bandwidth). 

Happy to provide this to the community as I feel this is an important dataset. Let us know what the authors want to do!
Issue: MustC Speech Translation

Comment:
Great! @skyprince999, do you think you could ping the authors here or link to this thread? I think it could be a cool idea to host the dataset on our side then
Issue: MustC Speech Translation

Comment:
Done. They replied back, and they want to have a call over a meet/ skype. Is that possible ? 
Btw @patrickvonplaten you are looped in that email (_pls check you gmail account_)
Issue: MustC Speech Translation

Comment:
@gegallego  there were some concerns regarding dataset usage & attribution by a for-profit company, so couldn't take it forward. Also the download links were unstable. 
But I guess if you want to test the fairseq benchmarks, you can connect with them directly for downloading the dataset.
Issue: MustC Speech Translation

Comment:
Yes, that dataset is not easy to download... I had to copy it to my Google Drive and use `rsync` to be able to download it.
However, we could add the dataset with a manual download, right?
Issue: MustC Speech Translation

Comment:
yes that is possible. I couldn't unfortunately complete this PR, If you would like to add it, please feel free to do it.
Issue: Add AMI Corpus

Comment:
@mariosasko actually the "official" AMI dataset can be found here: https://huggingface.co/datasets/edinburghcstr/ami -> the old one under `datasets/ami` doesn't work and should be deleted. 

The new one was tested by fine-tuning a Wav2Vec2 model on it + we uploaded all the processed audio directly into it
Issue: Add common voice

Comment:
Hey @BirgerMoell - awesome that you started working on Common Voice. Common Voice is a bit special since, there is no direct download link to download the data. In these cases we usually consider two options:

1) Find a hacky solution to extract the download link somehow from the XLM tree of the website 
2) If this doesn't work we force the user to download the data himself and add a `"data_dir"` as an input parameter. E.g. you can take a look at how it is done for [this](https://github.com/huggingface/datasets/blob/66f2a7eece98d2778bd22bb5034cb7c2376032d4/datasets/arxiv_dataset/arxiv_dataset.py#L66) 

Also the documentation here: https://huggingface.co/docs/datasets/add_dataset.html?highlight=data_dir#downloading-data-files-and-organizing-splits (especially the "note") might be helpful.
Issue: Add common voice

Comment:
I added a Work in Progress pull request (hope that is ok). I've made a card for the dataset and filled out the common_voice.py file with information about the datset (not completely).

I didn't manage to get the tagging tool working locally on my machine but will look into that later.

Left to do.

- Tag the dataset
- Add missing information and update common_voice.py

https://github.com/huggingface/datasets/pull/1886
Issue: Add common voice

Comment:
I saw that this current datasets package holds common voice version 6.1, how to add the new version 7.0 that is already available?
Issue: Add common voice

Comment:
Common voice still appears to be a 6.1. Is the plan still to upgrade to 7.0?
Issue: Add common voice

Comment:
We actually already have the code and everything ready to add Common Voice 7.0 to `datasets` but are still waiting for the common voice authors to give us the green light :-) 

Also gently pinging @phirework and @milupo here
Issue: Add tedlium

Comment:
Hi @patrickvonplaten 
I can have a look to this dataset later since I am trying to add the OpenSLR dataset https://github.com/huggingface/datasets/pull/2173
Hopefully I have enough space since the compressed file is 21GB. The release 3 is even bigger: 54GB :-0
Issue: Add VCTK

Comment:
@patrickvonplaten I'd like to take this, if nobody has already done it. I have added datasets before through the datasets sprint, but I feel rusty on the details, so I'll look at the guide as well as similar audio PRs (#1878 in particular comes to mind). If there is any detail I should be aware of please, let me know! Otherwise, I'll try to write up a PR in the coming days.
Issue: Add VCTK

Comment:
That sounds great @jaketae - let me know if you need any help i.e. feel free to ping me on a first PR :-)
Issue: Some question about raw dataset download info in the project .

Comment:
Hi ! The `dl_manager` is a `DownloadManager` object and is responsible for downloading the raw data files.
It is used by dataset builders in their `_split_generators` method to download the raw data files that are necessary to build the datasets splits.

The `Conll2003` class is a dataset builder, and so you can download all the raw data files by calling `_split_generators` with a download manager:
```python
from datasets import DownloadManager
from datasets.load import import_main_class

conll2003_builder = import_main_class(...)

dl_manager = DownloadManager()
splis_generators = conll2003_builder._split_generators(dl_manager)
```

Then you can see what files have been downloaded with
```python
dl_manager.get_recorded_sizes_checksums()
```
It returns a dictionary with the format {url: {num_bytes: int, checksum: str}}

Then you can get the actual location of the downloaded files with
```python
from datasets import cached_path

local_path_to_downloaded_file = cached_path(url)
```

------------------

Note that you can also get the urls from the Dataset object:
```python
from datasets import load_dataset

conll2003 = load_dataset("conll2003")
print(conll2003["train"].download_checksums)
```
It returns the same dictionary with the format {url: {num_bytes: int, checksum: str}}
Issue: Some question about raw dataset download info in the project .

Comment:
I am afraid that there is not a very straightforward way to get that location.

Another option, from _split_generators would be to use:
- `dl_manager._download_config.cache_dir` to get the directory where all the raw downloaded files are:
  ```python
  download_dir = dl_manager._download_config.cache_dir
  ```
- the function `datasets.utils.file_utils.hash_url_to_filename` to get the filenames of the raw downloaded files:
  ```python
  filenames = [hash_url_to_filename(url) for url in urls_to_download.values()]
  ```
Therefore the complete path to the raw downloaded files would be the join of both:
```python
downloaded_paths = [os.path.join(download_dir, filename) for filename in filenames]
```

Maybe it would be interesting to make these paths accessible more easily. I could work on this. What do you think, @lhoestq ?
Issue: Some question about raw dataset download info in the project .

Comment:
Sure it would be nice to have an easier access to these paths !
The dataset builder could have a method to return those, what do you think ?
Feel free to work on this @albertvillanova , it would be a nice addition :) 

Your suggestion does work as well @albertvillanova if you complete it by specifying `etag=` to `hash_url_to_filename`.

The ETag is obtained by a HEAD request and is used to know if the file on the remote host has changed. Therefore if a file is updated on the remote host, then the hash returned by `hash_url_to_filename` is different.
Issue: Some question about raw dataset download info in the project .

Comment:
Once #1846 will be merged, the paths to the raw downloaded files will be accessible as:
```python
builder_instance.dl_manager.downloaded_paths
```
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
Hi @wumpusman 
`datasets` has a caching mechanism that allows to cache the results of `.map` so that when you want to re-run it later it doesn't recompute it again.
So when you do `.map`, what actually happens is:
1. compute the hash used to identify your `map` for the cache
2. apply your function on every batch

This can explain the time difference between your different experiments.

The hash computation time depends of how complex your function is. For a tokenizer, the hash computation scans the lists of the words in the tokenizer to identify this tokenizer. Usually it takes 2-3 seconds.

Also note that you can disable caching though using
```python
import datasets

datasets.set_caching_enabled(False)
```
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
Hi @lhoestq ,

Thanks for the reply. It's entirely possible that is the issue. Since it's a side project I won't be looking at it till later this week, but, I'll verify it by disabling caching and hopefully I'll see the same runtime. 

Appreciate the reference,

Michael
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
I believe this is an actual issue, tokenizing a ~4GB txt file went from an hour and a half to ~10 minutes when I switched from my pre-trained tokenizer(on the same dataset) to the default gpt2 tokenizer.
Both were loaded using:
```
AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
```
I trained the tokenizer using ByteLevelBPETokenizer from the Tokenizers library and save it to a tokenizer.json file.

I have tested the caching ideas above, changing the number of process, the TOKENIZERS_PARALLELISM env variable, keep_in_memory=True and batching with different sizes.

Apologies I can't really upload much code, but wanted to back up the finding and hopefully a fix/the problem can be found.
I will comment back if I find a fix as well.
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
Hi @johncookds do you think this can come from one tokenizer being faster than the other one ? Can you try to compare their speed without using `datasets` just to make sure ?
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
Hi yes, I'm closing the loop here with some timings below. The issue seems to be at least somewhat/mainly with the tokenizer's themselves. Moreover legacy saves of the trainer tokenizer perform faster but differently than the new tokenizer.json saves(note nothing about the training process/adding of special tokens changed between the top two trained tokenizer tests, only the way it was saved). This is only a 3x slowdown vs like a 10x but I think the slowdown is most likely due to this.

```
trained tokenizer - tokenizer.json save (same results for AutoTokenizer legacy_format=False):
Tokenizer time(seconds): 0.32767510414123535
Tokenized avg. length: 323.01

trained tokenizer - AutoTokenizer legacy_format=True:
Tokenizer time(seconds): 0.09258866310119629
Tokenized avg. length: 301.01

GPT2 Tokenizer from huggingface
Tokenizer time(seconds): 0.1010282039642334
Tokenized avg. length: 461.21
```
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
@lhoestq ,

Hi, which version of datasets has datasets.set_caching_enabled(False)? I get 
module 'datasets' has no attribute 'set_caching_enabled'. To hopefully get around this, I reran my code on a new set of data, and did so only once.

@johncookds , thanks for chiming in, it looks this might be an issue of Tokenizer.

**Tokenizer**: The runtime of GPT2TokenizerFast.from_pretrained("gpt2") on 1000 chars is: **143 ms**
**SlowTokenizer**: The runtime of a locally saved and loaded Tokenizer using the same vocab  on 1000 chars is:  **4.43 s**

That being said, I compared performance on the map function:

Running Tokenizer versus using it in the map function for 1000 chars goes from **141 ms** to **356 ms** 
Running SlowTokenizer versus using it in the map function for 1000 chars with a single element goes from **4.43 s** to **9.76 s**

I'm trying to figure out why the overhead of map would increase the time by double (figured it would be a fixed increase in time)? Though maybe this is expected behavior.

@lhoestq, do you by chance know how I can redirect this issue to Tokenizer?

Regards,

Michael
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
Thanks for the experiments @johncookds and @wumpusman ! 

> Hi, which version of datasets has datasets.set_caching_enabled(False)?

Currently you have to install `datasets` from source to have this feature, but this will be available in the next release in a few days.

> I'm trying to figure out why the overhead of map would increase the time by double (figured it would be a fixed increase in time)? Though maybe this is expected behavior.

Could you also try with double the number of characters ? This should let us have an idea of the fixed cost (hashing) and the dynamic cost (actual tokenization, grows with the size of the input)

> @lhoestq, do you by chance know how I can redirect this issue to Tokenizer?

Feel free to post an issue on the `transformers` repo. Also I'm sure there should be related issues so you can also look for someone with the same concerns on the `transformers` repo.
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
@lhoestq,

I just checked that previous run time was actually  3000 chars. I increased it to 6k chars, again, roughly double.

SlowTokenizer **7.4 s** to **15.7 s**
Tokenizer: **276 ms** to **616 ms**

I'll post this issue on Tokenizer, seems it hasn't quite been raised (albeit I noticed a similar issue that might relate).

Regards,

Michael
Issue: using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Comment:
Hi, 
I'm following up here as I found my exact issue. It was with saving and re-loading the tokenizer. When I trained then processed the data without saving and reloading it, it was 10x-100x faster than when I saved and re-loaded it.
Both resulted in the exact same tokenized datasets as well. 
There is additionally a bug where the older legacy tokenizer save does not preserve a learned tokenizing behavior if trained from scratch.
Understand its not exactly Datasets related but hope it can help someone if they have the same issue.
Thanks!
Issue: Regarding On-the-fly Data Loading

Comment:
Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).
Issue: Regarding On-the-fly Data Loading

Comment:
Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?