SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: how to get all the options of a property in datasets 

Comment:
I think the `features` attribute of the dataset object is what you are looking for:
```
>>> dataset.features
{'sentence1': Value(dtype='string', id=None),
 'sentence2': Value(dtype='string', id=None),
 'label': ClassLabel(num_classes=2, names=['not_equivalent', 'equivalent'], names_file=None, id=None),
 'idx': Value(dtype='int32', id=None)
}
>>> dataset.features["label"].names
['not_equivalent', 'equivalent']
```

For reference: https://huggingface.co/docs/datasets/exploring.html
Issue: Inconsistent argument names.

Comment:
Also for the `Accuracy` metric the `accuracy_score` method should have its args in the opposite order so `accuracy_score(predictions, references,,,)`.
Issue: Inconsistent argument names.

Comment:
Thanks for pointing this out ! 🕵🏻 
Predictions and references should indeed be swapped in the docstring.
However, the call to `accuracy_score` should not be changed, it [signature](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) being:
```
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)
```

Feel free to open a PR if you want to fix this :)
Issue: SNLI dataset contains labels with value -1

Comment:
I believe the `-1` label is used for missing/NULL data as per HuggingFace Dataset conventions. If I recall correctly SNLI has some entries with no (gold) labels in the dataset.
Issue: FileNotFound remotly, can't load a dataset

Comment:
This dataset will be available in version-2 of the library. If you want to use this dataset now, install datasets from `master` branch rather.

Command to install datasets from `master` branch:
`!pip install git+https://github.com/huggingface/datasets.git@master`
Issue: Can't map dataset (loaded from csv)

Comment:
Please could you post the whole script? I can't reproduce your issue. After updating the feature names/labels to match with the data, everything works fine for me. Try to update datasets/transformers to the newest version.
Issue: Can't map dataset (loaded from csv)

Comment:
Actually, the problem was how `tokenize` function was defined. This was completely my side mistake, so there are really no needs in this issue anymore
Issue: ❓ Sharing ElasticSearch indexed dataset 

Comment:
Hello @pietrolesci , I am not sure to understand what you are trying to do here.

If you're looking for ways to save a dataset on disk, you can you the `save_to_disk` method:
```python
>>> import datasets
>>> loaded_dataset = datasets.load("dataset_name")
>>> loaded_dataset.save_to_disk("/path/on/your/disk")
```

The saved dataset can later be retrieved using:
```python
>>> loaded_dataset = datasets.Dataset.load_from_disk("/path/on/your/disk")
```

Also, I'd recommend posting your question directly in the issue section of the [elasticsearch repo](https://github.com/elastic/elasticsearch)
Issue: ❓ Sharing ElasticSearch indexed dataset 

Comment:
Hi @SBrandeis,

Thanks a lot for picking up my request. 

Maybe I can clarify my use-case with a bit of context. Say I have the IMDb dataset. I create an ES index on it. Now I can save and reload the dataset from disk normally. Once I reload the dataset, it is easy to retrieve the ES index on my machine. I was wondering: is there a way I can share the (now) indexed version of the IMDb dataset with my colleagues without requiring them to re-index it?

Thanks a lot in advance for your consideration.

Best,

Pietro
Issue: ❓ Sharing ElasticSearch indexed dataset 

Comment:
Thanks for the clarification.

I am not familiar with ElasticSearch, but if I understand well you're trying to migrate your data along with the ES index.
My advice would be to check out ES documentation, for instance, this might help you: https://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html

Let me know if it helps
Issue: can't load "german_legal_entity_recognition" dataset

Comment:
Please if you could tell me more about the error? 

1. Please check the directory you've been working on
2. Check for any typos
Issue: can't load "german_legal_entity_recognition" dataset

Comment:
> Please if you could tell me more about the error?
> 
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.
Issue: can't load "german_legal_entity_recognition" dataset

Comment:
Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)
Issue: imdb dataset cannot be downloaded

Comment:
Hi @rabeehk , I am unable to reproduce your problem locally.
Can you try emptying the cache (removing the content of `/idiap/temp/rkarimi/cache_home_1/datasets`) and retry ?
Issue: imdb dataset cannot be downloaded

Comment:
Hi,
thanks, I did remove the cache and still the same error here

```
>>> a = datasets.load_dataset("imdb", split="train")
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Downloading and preparing dataset imdb/plain_text (download: 80.23 MiB, generated: 127.06 MiB, post-processed: Unknown size, total: 207.28 MiB) to /idiap/temp/rkarimi/cache_home_1/datasets/imdb/plain_text/1.0.0/90099cb476936b753383ba2ae6ab2eae419b2e87f71cd5189cb9c8e5814d12a3...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):        
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=4902716, num_examples=3680, dataset_name='imdb')}]
```

datasets version
```
datasets                  1.1.2                     <pip>
tensorflow-datasets       4.1.0                     <pip>

```
Issue: 'iwslt2017-ro-nl', cannot be downloaded 

Comment:
Looks like the data has been moved from its original location to google drive

New url: https://drive.google.com/u/0/uc?id=12ycYSzLIG253AFN35Y6qoyf9wtkOjakp&export=download
Issue: [libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0):  terminate called after throwing an instance of 'google::protobuf::FatalException'   what():  CHECK failed: (index) >= (0):  Aborted

Comment:
I remember also getting the same issue for several other translation datasets like all the iwslt2017 group, this is blokcing me and I really need to fix it and I was wondering if you have an idea on this. @lhoestq  thanks,.
Issue: [libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0):  terminate called after throwing an instance of 'google::protobuf::FatalException'   what():  CHECK failed: (index) >= (0):  Aborted

Comment:
maybe there is an empty line or something inside these datasets? could you tell me why this is happening? thanks
Issue: [libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0):  terminate called after throwing an instance of 'google::protobuf::FatalException'   what():  CHECK failed: (index) >= (0):  Aborted

Comment:
I just checked and the wmt16 en-ro doesn't have empty lines
```python
from datasets import load_dataset

d = load_dataset("wmt16", "ro-en", split="train")
len(d)  # 610320
len(d.filter(lambda x: len(x["translation"]["en"].strip()) > 0))  # 610320
len(d.filter(lambda x: len(x["translation"]["ro"].strip()) > 0))  # 610320
# also tested for split="validation" and "test"
```

Can you open an issue on the `transformers` repo ? also cc @sgugger
Issue: boolq does not work 

Comment:
here is the minimal code to reproduce

`datasets>>> datasets.load_dataset("boolq", "train")

the errors

```
`cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Using custom data configuration train
Downloading and preparing dataset boolq/train (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /idiap/temp/rkarimi/cache_home_1/datasets/boolq/train/0.1.0/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11...
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 149, in download_custom
    custom_download(url, path)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 516, in copy_v2
    compat.path_to_bytes(src), compat.path_to_bytes(dst), overwrite)



```
Issue: boolq does not work 

Comment:
This has been fixed by #881 
this fix will be available in the next release soon.

If you don't want to wait for the release you can actually load the latest version of boolq by specifying `script_version="master"` in `load_dataset`
Issue: ❓ On-the-fly tokenization with datasets, tokenizers, and torch Datasets and Dataloaders

Comment:
We're working on adding on-the-fly transforms in datasets.
Currently the only on-the-fly functions that can be applied are in `set_format` in which we transform the data in either numpy/torch/tf tensors or pandas.
For example
```python
dataset.set_format("torch")
```
applies `torch.Tensor` to the dataset entries on-the-fly.

We plan to extend this to user-defined formatting transforms.
For example
```python
dataset.set_format(transform=tokenize)
```

What do you think ?
Issue: Using a feature named "_type" fails with certain operations

Comment:
Thanks for reporting !

Indeed this is a keyword in the library that is used to encode/decode features to a python dictionary that we can save/load to json.
We can probably change `_type` to something that is less likely to collide with user feature names.
In this case we would want something backward compatible though.

Feel free to try a fix and open a PR, and to ping me if I can help :)
Issue: Add support to download kaggle datasets

Comment:
Hey, I think this is great idea. Any plan to integrate kaggle private datasets loading to `datasets`?
Issue: Add support to download kaggle datasets

Comment:
The workflow for downloading a Kaggle dataset and turning it into an HF dataset is pretty simple:
```python
!kaggle datasets download -p path
ds = load_dataset(path)
```

Native support would make our download logic even more complex, and I don't think this is a good idea considering this particular feature is not requested often. 

PS: Kaggle should integrate their API with `fsspec` to allow us to use a common interface if they are interested in tighter integrations
Issue: Not support links with 302 redirect 

Comment:
> Hi !
> This kind of links is now supported by the library since #1316

I updated links in TLC datasets to be the github links in this pull request 
 https://github.com/huggingface/datasets/pull/1737

Everything works now. Thank you.
Issue: Dataset.map() turns tensors into lists?

Comment:
A solution is to have the tokenizer return a list instead of a tensor, and then use `dataset_tok.set_format(type = 'torch')` to convert that list into a tensor. Still not sure if bug.
Issue: Dataset.map() turns tensors into lists?

Comment:
It is expected behavior, you should set the format to `"torch"` as you mentioned to get pytorch tensors back.
By default datasets returns pure python objects.
Issue: how large datasets are handled under the hood 

Comment:
This library uses Apache Arrow under the hood to store datasets on disk.
The advantage of Apache Arrow is that it allows to memory map the dataset. This allows to load datasets bigger than memory and with almost no RAM usage. It also offers excellent I/O speed.

For example when you access one element or one batch
```python
from datasets import load_dataset

squad = load_dataset("squad", split="train")
first_element = squad[0]
one_batch = squad[:8]
```

then only this element/batch is loaded in memory, while the rest of the dataset is memory mapped.
Issue: how large datasets are handled under the hood 

Comment:
How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

EDIT:
My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.
Issue: how large datasets are handled under the hood 

Comment:
> How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

Loading arrow data from disk is done with memory-mapping. This allows to load huge datasets without filling your RAM.
Memory mapping is almost instantaneous and is done within one process.

Then, the speed of querying examples from the dataset is I/O bounded depending on your disk. If it's an SSD then fetching examples from the dataset will be very fast.
But since the I/O speed of an SSD is lower than the one of RAM it's expected to be slower to fetch data from disk than from memory.
Still, if you load the dataset in different processes then it can be faster but there will still be the I/O bottleneck of the disk.

> EDIT:
> My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

Ok let me know if that helps !
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
Looks like the google drive download failed.
I'm getting a `Google Drive - Quota exceeded` error while looking at the downloaded file.

We should consider finding a better host than google drive for this dataset imo
related : #873 #864
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
It is working now, thank you. 

Should I leave this issue open to address the Quota-exceeded error?
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
I've looked into it and couldn't find a solution. This looks like a Google Drive limitation..
Please try to use other hosts when possible
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
The original links are google drive links.  Would it be feasible for HF to maintain their own servers for this? Also, I think the same issue must also exist with TFDS.
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
It's possible to host data on our side but we should ask the authors. TFDS has the same issue and doesn't have a solution either afaik.
Otherwise you can use the google drive link, but it it's not that convenient because of this quota issue.
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
Okay. I imagine asking every author who shares their dataset on Google Drive will also be cumbersome.
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
Not as long as the data is stored on GG drive unfortunately.
Maybe we can ask if there's a mirror ?

Hi @JafferWilson is there a download link to get cnn dailymail from another host than GG drive ?

To give you some context, this library provides tools to download and process datasets. For CNN DailyMail the data are downloaded from the link you provide on your github repository. Unfortunately because of GG drive quotas, many users are not able to load this dataset.
Issue: NotADirectoryError while loading the CNN/Dailymail dataset

Comment:
Thanks for the link @mrazizi !

Apparently the original authors don't host the dataset themselves ("for legal reasons", source [here](https://github.com/abisee/cnn-dailymail/issues/9)).
Issue: Problem downloading amazon_reviews_multi

Comment:
Hi @hfawaz ! This is working fine for me. Is it a repeated occurence? Have you tried from the latest verion?
Issue: making sure datasets are not loaded in memory and distributed training of them

Comment:
my implementation of sharding per TPU core: https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/trainers/t5_trainer.py#L316 
my implementation of dataloader for this case https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/tasks/tasks.py#L496
Issue: making sure datasets are not loaded in memory and distributed training of them

Comment:
Hi!  You can use the `assert not bool(dataset.cache_files)` assertion to ensure your data is in memory. And I suggest using `accelerate` for distributed training.
Issue: sample multiple datasets 

Comment:
here I share my dataloader currently for multiple tasks: https://gist.github.com/rabeehkarimimahabadi/39f9444a4fb6f53dcc4fca5d73bf8195 

I need to train my model distributedly with this dataloader, "MultiTasksataloader", currently this does not work in distributed fasion,
to save on memory I tried to use iterative datasets, could you have a look in this dataloader and tell me if this is indeed the case? not sure how to make datasets being iterative to not load them in memory, then I remove the sampler for dataloader, and shard the data per core, could you tell me please how I should implement this case in datasets library? and how do you find my implementation in terms of correctness? thanks
Issue: sample multiple datasets 

Comment:
Thanks @rabeehk for sharing. 

The sampler basically returns a list of integers to sample from each task's dataset. I was wondering how to use it with two `torch.Dataset` of different tasks. Also, do I need to shard across processes while creating an Iterable Dataset?
Issue: sample multiple datasets 

Comment:
We now have `interleave_datasets` in the API that allows you to cycle/sample with probabilities (with various stopping strategies) through a list of datasets. However, more specific behavior should be implemented manually.
Issue: Local machine/cluster Beam Datasets example/tutorial

Comment:
I tried to make it run once on the SparkRunner but it seems that this runner has some issues when it is run locally.
From my experience the DirectRunner is fine though, even if it's clearly not memory efficient.

It would be awesome though to make it work locally on a SparkRunner !
Did you manage to make your processing work ?
Issue: wrong length with datasets 

Comment:
Also, I cannot first convert it to torch format, since huggingface seq2seq_trainer codes process the datasets afterwards during datacollector function to make it optimize for TPUs.
Issue: Shall we change the hashing to encoding to reduce potential replicated cache files?

Comment:
This is an interesting idea !
Do you have ideas about how to approach the decoding and the normalization ?
Issue: Shall we change the hashing to encoding to reduce potential replicated cache files?

Comment:
@lhoestq
I think we first need to save the transformation chain to a list in `self._fingerprint`. Then we can
- decode all the current saved datasets to see if there is already one that is equivalent to the transformation we need now.
- or, calculate all the possible hash value of the current chain for comparison so that we could continue to use hashing.
If we find one, we can adjust the list in `self._fingerprint` to it.

As for the transformation reordering rules, we can just start with some manual rules, like two sort on the same column should merge to one, filter and select can change orders.

And for encoding and decoding, we can just manually specify `sort` is 0, `shuffling` is 2 and create a base-n number or use some general algorithm like `base64.urlsafe_b64encode`.

Because we are not doing lazy evaluation now, we may not be able to normalize the transformation to its minimal form. If we want to support that, we can provde a `Sequential` api and let user input a list or transformation, so that user would not use the intermediate datasets. This would look like tf.data.Dataset.
Issue: Dataset viewer issues

Comment:
Thanks for reporting !
cc @srush for the empty feature list issue and the encoding issue
cc @julien-c maybe we can update the url and just have a redirection from the old url to the new one ?
Issue: Dataset viewer issues

Comment:
Ok, I redirected on our side to a new url. ⚠️ @srush: if you update the Streamlit config too to `/datasets/viewer`, let me know because I'll need to change our nginx config at the same time
Issue: Nested lists are zipped unexpectedly

Comment:
Yes following the Tensorflow Datasets convention, objects with type `Sequence of a Dict` are actually stored as a `dictionary of lists`.
See the [documentation](https://huggingface.co/docs/datasets/features.html?highlight=features) for more details
Issue: Nested lists are zipped unexpectedly

Comment:
Thanks.
This is a bit (very) confusing, but I guess if its intended, I'll just work with it as if its how my data was originally structured :)