SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Bug Report: timestamp[ns] not recognized

Comment:
Thanks for the clarification @lhoestq !

This may be a little bit of a stupid question, but I wanted to clarify one more thing before I took a stab at this:

When the features get inferred, I believe they already have a pyarrow schema (https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L234).

We then convert it to a string (https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L778) only to convert it back into the arrow type (https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L143, and https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L35).  Is there a reason for this round-trip?

I'll open a PR later to add `timestamp` support to `string_to_arrow`, but I'd be curious to understand since it feels like there may be some opportunities to simplify!
Issue: Bug Report: timestamp[ns] not recognized

Comment:
The objective in terms of design is to make it easy to create Features in a pythonic way. So for example we use a string to define a Value type.
That's why when inferring the Features from an arrow schema we have to find the right string definitions for Value types. I guess we could also have a constructor `Value.from_arrow_type` to avoid recreating the arrow type, but this could create silent errors if the pyarrow type doesn't have a valid mapping with the string definition. The "round-trip" is used to enforce that the ground truth is the string definition, not the pyarrow type, and also as a sanity check.

Let me know if that makes sense
Issue: Bug Report: timestamp[ns] not recognized

Comment:
OK I think I understand now:

Features are datasets' internal representation of a schema type, distinct from pyarrow's schema.
Value() corresponds to pyarrow's "primitive" types (e.g. `int` or `string`, but not things like `list` or `dict`).
`get_nested_type()` (https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L698) and `generate_from_arrow_type()` (https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L778) *should* be inverses of each other, and similarly, for the primitive values, `string_to_arrow()` and `Value.__call__` (https://github.com/huggingface/datasets/blob/master/src/datasets/features.py#L146) should be inverses of each other?

Thanks for taking the time to answer - I just wanted to make sure I understood before opening a PR so I'm not disrupting anything about how the codebase is expected to work!
Issue: benchmarking against MMapIndexedDataset

Comment:
Hi sam !
Indeed we can expect the performances to be very close since both MMapIndexedDataset and the `datasets` implem use memory mapping. With memory mapping what determines the I/O performance is the speed of your hard drive/SSD.

In terms of performance we're pretty close to the optimal speed for reading text, even though I found recently that we could still slightly improve speed for big datasets (see [here](https://github.com/huggingface/datasets/issues/1803)).

In terms of number of examples and example sizes, the only limit is the available disk space you have.

I haven't used `psrecord` yet but it seems to be a very interesting tool for benchmarking. Currently for benchmarks we only have github actions to avoid regressions in terms of speed. But it would be cool to have benchmarks with comparisons with other dataset tools ! This would be useful to many people
Issue: benchmarking against MMapIndexedDataset

Comment:
Also I would be interested to know what data types `MMapIndexedDataset` supports. Is there some documentation somewhere ?
Issue: benchmarking against MMapIndexedDataset

Comment:
no docs haha, it's written to support integer numpy arrays.

You can build one in fairseq with, roughly:
```bash

wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip
unzip wikitext-103-raw-v1.zip
export dd=$HOME/fairseq-py/wikitext-103-raw

export mm_dir=$HOME/mmap_wikitext2
mkdir -p gpt2_bpe
wget -O gpt2_bpe/encoder.json https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/encoder.json
wget -O gpt2_bpe/vocab.bpe https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/vocab.bpe
wget -O gpt2_bpe/dict.txt https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt
for SPLIT in train valid; do \
    python -m examples.roberta.multiprocessing_bpe_encoder \
        --encoder-json gpt2_bpe/encoder.json \
        --vocab-bpe gpt2_bpe/vocab.bpe \
        --inputs /scratch/stories_small/${SPLIT}.txt \
        --outputs /scratch/stories_small/${SPLIT}.bpe \
        --keep-empty \
        --workers 60; \
done

mkdir -p $mm_dir
fairseq-preprocess \
    --only-source \
    --srcdict gpt2_bpe/dict.txt \
    --trainpref $dd/wiki.train.bpe \
    --validpref $dd/wiki.valid.bpe \
    --destdir $mm_dir \
    --workers 60 \
    --dataset-impl mmap
```

I'm noticing in my benchmarking that it's much smaller on disk than arrow (200mb vs 900mb), and that both incur significant cost by increasing the number of data loader workers. 
This somewhat old [post](https://ray-project.github.io/2017/10/15/fast-python-serialization-with-ray-and-arrow.html) suggests there are some gains to be had from using `pyarrow.serialize(array).tobuffer()`. I haven't yet figured out how much of this stuff `pa.Table` does under the hood.

The `MMapIndexedDataset` bottlenecks we are working on improving (by using arrow) are:
1) `MMapIndexedDataset`'s index, which stores offsets, basically gets read in its entirety by each dataloading process.
2) we have separate, identical, `MMapIndexedDatasets` on each dataloading worker, so there's redundancy there; we wonder if there is a way that arrow can somehow dedupe these in shared memory.

It will take me a few hours to get `MMapIndexedDataset` benchmarks out of `fairseq`/onto a branch in this repo, but I'm happy to invest the time if you're interested in collaborating on some performance hacking.
Issue: wmt19 is broken

Comment:
This was also mentioned in https://github.com/huggingface/datasets/issues/488 

The bucket where is data was stored seems to be unavailable now. Maybe we can change the URL to the ones in https://conferences.unite.un.org/uncorpus/en/downloadoverview ?
Issue: request to mirror wmt datasets, as they are really slow to download

Comment:
Yes that would be awesome. Not only the download speeds are awful, but also some files are missing.
We list all the URLs in the datasets/wmt19/wmt_utils.py so we can make a script to download them all and host on S3.
Also I think most of the materials are under the CC BY-NC-SA 3.0 license (must double check) so it should be possible to redistribute the data with no issues.

cc @patrickvonplaten who knows more about the wmt scripts
Issue: request to mirror wmt datasets, as they are really slow to download

Comment:
Yeah, the scripts are pretty ugly! A big refactor would make sense here...and I also remember that the datasets were veeery slow to download
Issue: request to mirror wmt datasets, as they are really slow to download

Comment:
I'm downloading them.
I'm starting with the ones hosted on http://data.statmt.org which are the slowest ones
Issue: request to mirror wmt datasets, as they are really slow to download

Comment:
@lhoestq better to use our new git-based system than just raw S3, no? (that way we have built-in CDN etc.)
Issue: request to mirror wmt datasets, as they are really slow to download

Comment:
Hi there! What about mirroring other datasets like [CCAligned](http://www.statmt.org/cc-aligned/) as well? All of them are really slow to download...
Issue: suggestion to improve a missing dataset error

Comment:
This is the current error thrown for missing datasets:
```
FileNotFoundError: Couldn't find a dataset script at C:\Users\Mario\Desktop\projects\datasets\missing_dataset\missing_dataset.py or any data file in the same directory. Couldn't find 'missing_dataset' on the Hugging Face Hub either: FileNotFoundError: Dataset 'missing_dataset' doesn't exist on the Hub. If the repo is private, make sure you are authenticated with `use_auth_token=True` after logging in with `huggingface-cli login`.
```

Seems much more informative, so I think we can close this issue.
Issue: Allow concatenation of both in-memory and on-disk datasets

Comment:
I started working on this. My idea is to first add the pyarrow Table wrappers InMemoryTable and MemoryMappedTable that both implement what's necessary regarding copy/pickle. Then have another wrapper that takes the concatenation of InMemoryTable/MemoryMappedTable objects.

What's important here is that concatenating two tables into one doesn't double the memory used (`total_allocated_bytes()` stays the same).
Issue: Allow concatenation of both in-memory and on-disk datasets

Comment:
Hi @lhoestq @albertvillanova,

I checked the linked issues and PR, this seems like a great idea. Would you mind elaborating on the in-memory and memory-mapped datasets? 
Based on my understanding, it is something like this, please correct me if I am wrong:
1. For in-memory datasets, we don't have any dataset files so the entire dataset is pickled to the cache during loading, and then whenever required it is unpickled .
2. For on-disk/memory-mapped datasets, we have the data files provided, so they can be re-loaded from the paths, and only the file-paths are stored while pickling.

If this is correct, will the feature also handle pickling/unpickling of a concatenated dataset? Will this be cached?

This also leads me to ask whether datasets are chunked during pickling? 

Thanks,
Gunjan
Issue: Allow concatenation of both in-memory and on-disk datasets

Comment:
Hi ! Yes you're totally right about your two points :)

And in the case of a concatenated dataset, then we should reload each sub-table depending on whether it's in-memory or memory mapped. That means the dataset will be made of several blocks in order to keep track of what's from memory and what's memory mapped. This allows to pickle/unpickle concatenated datasets
Issue: Allow concatenation of both in-memory and on-disk datasets

Comment:
Hi @lhoestq

Thanks, that sounds nice. Can you explain where the issue of the double memory may arise? Also, why is the existing `concatenate_datasets` not sufficient for this purpose?
Issue: Allow concatenation of both in-memory and on-disk datasets

Comment:
Hi @lhoestq,

Will the `add_item` feature also help with lazy writing (or no caching) during `map`/`filter`?
Issue: Allow concatenation of both in-memory and on-disk datasets

Comment:
> Can you explain where the issue of the double memory may arise?

We have to keep each block (in-memory vs memory mapped) separated in order to be able to reload them with pickle.
On the other hand we also need to have the full table from mixed in-memory and memory mapped data in order to iterate or extract data conveniently. That means that each block is accessible twice: once in the full table, and once in the separated blocks. But since pyarrow tables concatenation doesn't double the memory, then building the full table doesn't cost memory which is what we want :)

> Also, why is the existing concatenate_datasets not sufficient for this purpose?

The existing `concatenate_datasets` doesn't support having both in-memory and memory mapped data together (there's no fancy block separation logic). It works for datasets fully in-memory or fully memory mapped but not a mix of the two.

> Will the add_item feature also help with lazy writing (or no caching) during map/filter?

It will enable the implementation of the fast, masked filter from this discussion: https://github.com/huggingface/datasets/issues/1949
However I don't think this will affect map.
Issue:  load_dataset("multi_woz_v22") NonMatchingChecksumError

Comment:
Thanks for reporting !
This is due to the changes made in the data files in the multiwoz repo: https://github.com/budzianowski/multiwoz/pull/59
I'm opening a PR to update the checksums of the data files.
Issue:  load_dataset("multi_woz_v22") NonMatchingChecksumError

Comment:
I just merged the fix. It will be available in the new release of `datasets` later today.
You'll be able to get the new version with
```
pip install --upgrade datasets
```
Issue:  load_dataset("multi_woz_v22") NonMatchingChecksumError

Comment:
Hi, I still meet the error when loading the datasets after upgradeing datasets.

raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/dialog_acts.json', 'https://github.com/budzianowski/multiwoz/raw/master/data/MultiWOZ_2.2/test/dialogues_001.json']
Issue:  load_dataset("multi_woz_v22") NonMatchingChecksumError

Comment:
This must be related to https://github.com/budzianowski/multiwoz/pull/72
Those files have changed, let me update the checksums for this dataset.

For now you can use `ignore_verifications=True` in `load_dataset` to skip the checksum verification.
Issue: Adding a new column to the dataset after set_format was called

Comment:
Hi ! Indeed if you add a column to a formatted dataset, then the new dataset gets a new formatting in which:
```
new formatted columns = (all columns - previously unformatted columns)
```
Therefore the new column is going to be formatted using the `torch` formatting.

If you want your new column to be unformatted you can re-run this line:
```python
data.set_format("torch", columns=["some_integer_column1", "some_integer_column2"], output_all_columns=True)
```
Issue: Adding a new column to the dataset after set_format was called

Comment:
Ok cool :) 
Also I just did a PR to mention this behavior in the documentation
Issue: ERROR WHEN USING SET_TRANSFORM() 

Comment:
Hi @alejandrocros it looks like an incompatibility with the current Trainer @sgugger 
Indeed currently the Trainer of `transformers` doesn't support a dataset with a transform

It looks like it comes from this line: https://github.com/huggingface/transformers/blob/f51188cbe74195c14c5b3e2e8f10c2f435f9751a/src/transformers/trainer.py#L442

This line sets the format to not return certain unused columns. But this has two issues:
1. it forgets to also set the format_kwargs (this causes the error you got):
```python
dataset.set_format(type=dataset.format["type"], columns=columns, format_kwargs=dataset.format["format_kwargs"])
```
2. the Trainer wants to keep only the fields that are used as input for a model. However for a dataset with a transform, the output fields are often different from the columns fields. For example from a column "text" in the dataset, the strings can be transformed on-the-fly into "input_ids". If you want your dataset to only output certain fields and not other you must change your transform function.
Issue: ERROR WHEN USING SET_TRANSFORM() 

Comment:
FYI that option can be removed with `remove_unused_columns = False` in your `TrainingArguments`, so there is a workaround @alexvaca0 while the fix in `Trainer` is underway.

@lhoestq I think I will just use the line you suggested and if someone is using the columns that are removed in their transform they will need to change `remove_unused_columns` to `False`. We might switch the default of that argument in the next version if that proves too bug-proof.
Issue: ERROR WHEN USING SET_TRANSFORM() 

Comment:
I've tried your solutions @sgugger @lhoestq and the good news is that it throws no error. However, TPU training is taking forever, in 1 hour it has only trained 1 batch of 8192 elements, which doesn't make much sense... Is it possible that "on the fly" tokenization of batches is slowing down TPU training to that extent?
Issue: ERROR WHEN USING SET_TRANSFORM() 

Comment:
I don't know what the value of `padding` is in your lines of code pasted above so I can't say for sure. The first batch will be very slow on TPU since it compiles everything, so that's normal (1 hour is long but 8192 elements is also large). Then if your batches are not of the same lengths, it will recompile everything at each step instead of using the same graph, which will be very slow, so you should double check you are using padding to make everything the exact same shape.
Issue: ERROR WHEN USING SET_TRANSFORM() 

Comment:
I have tried now on a GPU and it goes smooth! Amazing feature .set_transform() instead of .map()! Now I can pre-train my model without the hard disk limitation. Thanks for your work all HuggingFace team!! :clap:
Issue: ERROR WHEN USING SET_TRANSFORM() 

Comment:
In the end, to make it work I turned to A-100 gpus instead of TPUS, among other changes. Set_transform doesn't work as expected and slows down training very much even in GPUs, and applying map destroys the disk, as it multiplies by 100 the size of the data passed to it (due to inefficient implementation converting strings to int64 floats I guess). For that reason, I chose to use datasets to load the data as text, and then edit the Collator from Transformers to tokenize every batch it receives before processing it. That way, I'm being able to train fast, without memory breaks, without the disk being unnecessarily filled, while making use of GPUs almost all the time I'm paying for them (the map function over the whole dataset took ~15hrs, in which you're not training at all). I hope this info helps others that are looking for training a language model from scratch cheaply, I'm going to close the issue as the optimal solution I found after many experiments to the problem posted in it is explained above.
Issue: ERROR WHEN USING SET_TRANSFORM() 

Comment:
Great comment @alexvaca0 . I think that we could re-open the issue as a reformulation of why it takes so much space to save the arrow. Saving a 1% of oscar corpus takes more thank 600 GB (it breaks when it pass 600GB because it is the free memory that I have at this moment) when the full dataset is 1,3 TB. I have a 1TB M.2 NVMe disk that I can not train on because the saved .arrow files goes crazily big. If you can share your Collator I will be grateful.
Issue: Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)

Comment:
Hi @corticalstack ! Thanks for reporting. Indeed in the recent versions of Faiss we must use `getDevice` to check if the index in on GPU.

I'm opening a PR
Issue: Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)

Comment:
I fixed this issue. It should work fine now.
Feel free to try it out by installing `datasets` from source.
Otherwise you can wait for the next release of `datasets` (in a few days)
Issue: Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)

Comment:
Thanks for such a quick fix and merge to master, pip installed git master, tested all OK
Issue: Unable to upload "community provided" dataset - 400 Client Error

Comment:
Hi ! We're in the process of switching the community datasets to git repos, exactly like what we're doing for models.
You can find an example here:
https://huggingface.co/datasets/lhoestq/custom_squad/tree/main

We'll update the CLI in the coming days and do a new release :)

Also cc @julien-c maybe we can make improve the error message ?
Issue: load_dataset("amazon_polarity") NonMatchingChecksumError

Comment:
Hi ! This issue may be related to #996 
This comes probably from the Quota Exceeded error from Google Drive.
Can you try again tomorrow and see if you still have the error ?

On my side I didn't get any error today with `load_dataset("amazon_polarity")`
Issue: load_dataset("amazon_polarity") NonMatchingChecksumError

Comment:
@lhoestq Hi! I encounter the same error when loading `yelp_review_full`.

```
from datasets import load_dataset
dataset_yp = load_dataset("yelp_review_full")
```

When you say the "Quota Exceeded from Google drive". Is this a quota from the dataset owner? or the quota from our (the runner) Google Drive?
Issue: load_dataset("amazon_polarity") NonMatchingChecksumError

Comment:
> When you say the "Quota Exceeded from Google drive". Is this a quota from the dataset owner? or the quota from our (the runner) Google Drive?

Each file on Google Drive can be downloaded only a certain amount of times per day because of a quota. The quota is reset every day. So if too many people download the dataset the same day, then the quota is likely to exceed.
That's a really bad limitations of Google Drive and we should definitely find another host for these dataset than Google Drive.
For now I would suggest to wait and try again later..

So far the issue happened with CNN DailyMail, Amazon Polarity and Yelp Reviews. 
Are you experiencing the issue with other datasets ? @calebchiam @dtch1997
Issue: load_dataset("amazon_polarity") NonMatchingChecksumError

Comment:
@lhoestq Gotcha, that is quite problematic...for what it's worth, I've had no issues with the other datasets I tried, such as `yelp_reviews_full` and `amazon_reviews_multi`.
Issue: load_dataset("amazon_polarity") NonMatchingChecksumError

Comment:
This error still happens, but for a different reason now: Google Drive returns a warning instead of the dataset.
Issue: load_dataset("amazon_polarity") NonMatchingChecksumError

Comment:
Hi ! Thanks for reporting. Google Drive changed the way to bypass the warning message recently.

The latest release `1.18.4` fixes this for datasets loaded in a regular way.

We opened a PR to fix this recently for streaming mode at #3843 - we'll do a new release once the fix is merged :)
Issue: Feature Request: Dataset.add_item

Comment:
Hi @sshleifer.

I am not sure of understanding the need of the `add_item` approach...

By just reading your "Desired API" section, I would say you could (nearly) get it with a 1-column Dataset:
```python
data = {"input_ids": [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])]}
ds = Dataset.from_dict(data)
assert (ds["input_ids"][0] == np.array([4,4,2])).all()
```
Issue: Feature Request: Dataset.add_item

Comment:
Hi @sshleifer :) 

We don't have methods like `Dataset.add_batch` or `Dataset.add_entry/add_item` yet.
But that's something we'll add pretty soon. Would an API that looks roughly like this help ? Do you have suggestions ?
```python
import numpy as np
from datasets import Dataset

tokenized = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

# API suggestion (not available yet)
d = Dataset()
for input_ids in tokenized:
    d.add_item({"input_ids": input_ids})

print(d[0]["input_ids"])
# [4, 4, 2]
```

Currently you can define a dataset with what @albertvillanova suggest, or via a generator using dataset builders. It's also possible to [concatenate datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets).
Issue: Add TIMIT

Comment:
@patrickvonplaten Could you please help me with how the output text has to be represented in the data? TIMIT has Words, Phonemes and texts. Also has lot on info on the speaker and the dialect. Could you please help me? An example of how to arrange it would be super helpful!
Issue: Add TIMIT

Comment:
Hey @vrindaprabhu - sure I'll help you :-) Could you open a first PR for TIMIT where you copy-paste more or less the `librispeech_asr` script: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L93 (obviously replacing all the naming and links correctly...) and then you can list all possible outputs in the features dict: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L104 (words, phonemes should probably be of kind `datasets.Sequence(datasets.Value("string"))` and texts I think should be of type `"text": datasets.Value("string")`.

When you've opened a first PR, I think it'll be much easier for us to take a look together :-)