ahmedfarazsyk/datasets-github-issues huggingface dataset:
Inconsistent argument names.
Author: Fraser-Greenlee
State: closed

Just find it a wee bit odd that in the transformers library `predictions` are those made by the model:
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_utils.py#L51-L61

While in many datasets metrics they are the ground truth labels:
https://github.com/huggingface/datasets/blob/c3f53792a744ede18d748a1133b6597fdd2d8d18/metrics/accuracy/accuracy.py#L31-L40

Do you think predictions & references should be swapped? I'd be willing to do some refactoring here if you agree.

Comments: Also for the `Accuracy` metric the `accuracy_score` method should have its args in the opposite order so `accuracy_score(predictions, references,,,)`.

Thanks for pointing this out ! 🕵🏻 
Predictions and references should indeed be swapped in the docstring.
However, the call to `accuracy_score` should not be changed, it [signature](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) being:
```
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)
```

Feel free to open a PR if you want to fix this :)
SNLI dataset contains labels with value -1
Author: aarnetalman
State: closed

```
import datasets
nli_data = datasets.load_dataset("snli")
train_data = nli_data['train']
train_labels = train_data['label']
label_set = set(train_labels)
print(label_set)
```

**Output:**
`{0, 1, 2, -1}`

Comments: I believe the `-1` label is used for missing/NULL data as per HuggingFace Dataset conventions. If I recall correctly SNLI has some entries with no (gold) labels in the dataset.

Ah, you're right. The dataset has some pairs with missing labels. Thanks for reminding me.
FileNotFound remotly, can't load a dataset
Author: sadakmed
State: closed

```py
!pip install datasets
import datasets as ds

corpus = ds.load_dataset('large_spanish_corpus')
```
gives the error

> FileNotFoundError: Couldn't find file locally at large_spanish_corpus/large_spanish_corpus.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/large_spanish_corpus/large_spanish_corpus.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/large_spanish_corpus/large_spanish_corpus.py

not just `large_spanish_corpus`,  `zest` too, but `squad` is available. 

this was using colab and localy 

Comments: This dataset will be available in version-2 of the library. If you want to use this dataset now, install datasets from `master` branch rather.

Command to install datasets from `master` branch:
`!pip install git+https://github.com/huggingface/datasets.git@master`

Closing this, thanks @VasudevGupta7
❓ Sharing ElasticSearch indexed dataset 
Author: pietrolesci
State: open
Labels: name: dataset request

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label

Comments: Hello @pietrolesci , I am not sure to understand what you are trying to do here.

If you're looking for ways to save a dataset on disk, you can you the `save_to_disk` method:
```python
>>> import datasets
>>> loaded_dataset = datasets.load("dataset_name")
>>> loaded_dataset.save_to_disk("/path/on/your/disk")
```

The saved dataset can later be retrieved using:
```python
>>> loaded_dataset = datasets.Dataset.load_from_disk("/path/on/your/disk")
```

Also, I'd recommend posting your question directly in the issue section of the [elasticsearch repo](https://github.com/elastic/elasticsearch)

Hi @SBrandeis,

Thanks a lot for picking up my request. 

Maybe I can clarify my use-case with a bit of context. Say I have the IMDb dataset. I create an ES index on it. Now I can save and reload the dataset from disk normally. Once I reload the dataset, it is easy to retrieve the ES index on my machine. I was wondering: is there a way I can share the (now) indexed version of the IMDb dataset with my colleagues without requiring them to re-index it?

Thanks a lot in advance for your consideration.

Best,

Pietro

Thanks for the clarification.

I am not familiar with ElasticSearch, but if I understand well you're trying to migrate your data along with the ES index.
My advice would be to check out ES documentation, for instance, this might help you: https://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html

Let me know if it helps
can't load "german_legal_entity_recognition" dataset
Author: nataly-obr
State: closed

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py


Comments: Please if you could tell me more about the error? 

1. Please check the directory you've been working on
2. Check for any typos

> Please if you could tell me more about the error?
> 
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.

Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)
Incorrect URL for MRQA SQuAD train subset
Author: yuxiang-wu
State: closed

https://github.com/huggingface/datasets/blob/4ef4c8f8b7a60e35c6fa21115fca9faae91c9f74/datasets/mrqa/mrqa.py#L53

The URL for `train+SQuAD` subset of MRQA points to the dev set instead of train set. It should be `https://s3.us-east-2.amazonaws.com/mrqa/release/v2/train/SQuAD.jsonl.gz`.

Comments: good catch !
Using a feature named "_type" fails with certain operations
Author: dcfidalgo
State: closed

A column named `_type` leads to a `TypeError: unhashable type: 'dict'` for certain operations:
```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({"_type": ["whatever"]}).map()
concatenate_datasets([ds])
# or simply
Dataset(ds._data)
```
Context: We are using datasets to persist data coming from elasticsearch to feed to our pipeline, and elasticsearch has a `_type` field, hence the strange name of the column.

Not sure if you wish to support this specific column name, but if you do i would be happy to try a fix and provide a PR. I already had a look into it and i think the culprit is the `datasets.features.generate_from_dict` function. It uses the hard coded `_type` string to figure out if it reached the end of the nested feature object from a serialized dict.

Best wishes and keep up the awesome work!

Comments: Thanks for reporting !

Indeed this is a keyword in the library that is used to encode/decode features to a python dictionary that we can save/load to json.
We can probably change `_type` to something that is less likely to collide with user feature names.
In this case we would want something backward compatible though.

Feel free to try a fix and open a PR, and to ping me if I can help :)
Add support to download kaggle datasets
Author: abhishekkrthakur
State: closed
Labels: name: enhancement

We can use API key

Comments: Hey, I think this is great idea. Any plan to integrate kaggle private datasets loading to `datasets`?

The workflow for downloading a Kaggle dataset and turning it into an HF dataset is pretty simple:
```python
!kaggle datasets download -p path
ds = load_dataset(path)
```

Native support would make our download logic even more complex, and I don't think this is a good idea considering this particular feature is not requested often. 

PS: Kaggle should integrate their API with `fsspec` to allow us to use a common interface if they are interested in tighter integrations
Not support links with 302 redirect 
Author: chameleonTK
State: closed
Labels: name: bug

name: enhancement

I have an issue adding this download link https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz

it might be because it is not a direct link (it returns 302 and redirects to aws that returns 403 for head requests). 

```
r.head("https://github.com/jitkapat/thailitcorpus/releases/download/v.2.0/tlc_v.2.0.tar.gz", allow_redirects=True)                                      
# <Response [403]>
```

Comments: Hi !
This kind of links is now supported by the library since #1316

> Hi !
> This kind of links is now supported by the library since #1316

I updated links in TLC datasets to be the github links in this pull request 
 https://github.com/huggingface/datasets/pull/1737

Everything works now. Thank you.
Dataset.map() turns tensors into lists?
Author: tombosc
State: closed

I apply `Dataset.map()` to a function that returns a dict of torch tensors (like a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists!

```import datasets
import torch  
from datasets import load_dataset                                                                                                                 
print("version datasets", datasets.__version__)

dataset = load_dataset("snli", split='train[0:50]')  

def tokenizer_fn(example):
    # actually uses a tokenizer which does something like:
    return {'input_ids': torch.tensor([[0, 1, 2]])}

print("First item in dataset:\n", dataset[0])
tokenized = tokenizer_fn(dataset[0])
print("Tokenized hyp:\n", tokenized)
dataset_tok = dataset.map(tokenizer_fn, batched=False,
        remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
dataset_tok = dataset.map(tokenizer_fn, batched=False,
                          remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
```

The output is:

```
version datasets 1.1.3
Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c)
First item in dataset:
 {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1}
Tokenized hyp:
 {'input_ids': tensor([[0, 1, 2]])}
Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow
Tokenized using map:
 {'input_ids': [[0, 1, 2]]}
<class 'torch.Tensor'> <class 'list'>
```

Or am I doing something wrong?


Comments: A solution is to have the tokenizer return a list instead of a tensor, and then use `dataset_tok.set_format(type = 'torch')` to convert that list into a tensor. Still not sure if bug.

It is expected behavior, you should set the format to `"torch"` as you mentioned to get pytorch tensors back.
By default datasets returns pure python objects.
Hi
Author: suemori87
State: closed

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
how large datasets are handled under the hood 
Author: rabeehkarimimahabadi
State: closed

Hi
I want to use multiple large datasets with a mapping style dataloader, where they cannot fit into memory, could you tell me how you handled the datasets under the hood? is this you bring all in memory in case of mapping style ones? or is this some sharding under the hood and you bring in memory when necessary, thanks 

Comments: This library uses Apache Arrow under the hood to store datasets on disk.
The advantage of Apache Arrow is that it allows to memory map the dataset. This allows to load datasets bigger than memory and with almost no RAM usage. It also offers excellent I/O speed.

For example when you access one element or one batch
```python
from datasets import load_dataset

squad = load_dataset("squad", split="train")
first_element = squad[0]
one_batch = squad[:8]
```

then only this element/batch is loaded in memory, while the rest of the dataset is memory mapped.

How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

EDIT:
My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

> How can we change how much data is loaded to memory with Arrow? I think that I am having some performance issue with it. When Arrow loads the data from disk it does it in multiprocess? It's almost twice slower training with arrow than in memory.

Loading arrow data from disk is done with memory-mapping. This allows to load huge datasets without filling your RAM.
Memory mapping is almost instantaneous and is done within one process.

Then, the speed of querying examples from the dataset is I/O bounded depending on your disk. If it's an SSD then fetching examples from the dataset will be very fast.
But since the I/O speed of an SSD is lower than the one of RAM it's expected to be slower to fetch data from disk than from memory.
Still, if you load the dataset in different processes then it can be faster but there will still be the I/O bottleneck of the disk.

> EDIT:
> My fault! I had not seen the `dataloader_num_workers` in `TrainingArguments` ! Now I can parallelize and go fast! Sorry, and thanks.

Ok let me know if that helps !
Problem downloading amazon_reviews_multi
Author: hfawaz
State: closed

Thanks for adding the dataset. 
After trying to load the dataset, I am getting the following error: 
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json
`
I used the following code to load the dataset: 
`load_dataset(
            dataset_name,
            "all_languages",
            cache_dir=".data"
        )`

I am using version 1.1.3 of `datasets`

Note that I can perform a successfull `wget https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json`

Comments: Hi @hfawaz ! This is working fine for me. Is it a repeated occurence? Have you tried from the latest verion?

Hi, it seems a connection problem. 
Now it says: 
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_ja_train.json`
making sure datasets are not loaded in memory and distributed training of them
Author: rabeehk
State: closed

Hi
I am dealing with large-scale datasets which I need to train distributedly, I used the shard function to divide the dataset across the cores, without any sampler, this does not work for distributed training and does not become any faster than 1 TPU core. 1) how I can make sure data is not loaded in memory 2) in case of distributed training with iterative datasets which measures needs to be taken? Is this all sharding the data only. I was wondering if there can be possibility for me to discuss this with someone with distributed training with iterative datasets using dataset library. thanks 

Comments: my implementation of sharding per TPU core: https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/trainers/t5_trainer.py#L316 
my implementation of dataloader for this case https://github.com/google-research/ruse/blob/d4dd58a2d8efe0ffb1a9e9e77e3228d6824d3c3c/seq2seq/tasks/tasks.py#L496 

Hi!  You can use the `assert not bool(dataset.cache_files)` assertion to ensure your data is in memory. And I suggest using `accelerate` for distributed training.
sample multiple datasets 
Author: rabeehk
State: closed

Hi
I am dealing with multiple datasets, I need to have a dataloader over them with a condition that in each batch data samples are coming from one of the datasets. My main question is: 
-  I need to have a way to sample the datasets first with some weights, lets say 2x dataset1 1x dataset2, could you point me how I can do it

sub-questions:
- I want to concat sampled datasets and define one dataloader on it, then I need a way to make sure batches come from 1 dataset in each iteration, could you assist me how I can do?
- I use iterative-type of datasets, but I need a method of shuffling still since it brings accuracy performance issues if not doing it, thanks for the help. 

Comments: here I share my dataloader currently for multiple tasks: https://gist.github.com/rabeehkarimimahabadi/39f9444a4fb6f53dcc4fca5d73bf8195 

I need to train my model distributedly with this dataloader, "MultiTasksataloader", currently this does not work in distributed fasion,
to save on memory I tried to use iterative datasets, could you have a look in this dataloader and tell me if this is indeed the case? not sure how to make datasets being iterative to not load them in memory, then I remove the sampler for dataloader, and shard the data per core, could you tell me please how I should implement this case in datasets library? and how do you find my implementation in terms of correctness? thanks 


Hi @rabeehkarimimahabadi any luck with updating the multi-task data loader to work with distributed training?

Hi @pushkalkatara yes I solved it back then, here please find my implementation https://github.com/rabeehk/hyperformer/blob/main/hyperformer/data/multitask_sampler.py 

Thanks @rabeehk for sharing. 

The sampler basically returns a list of integers to sample from each task's dataset. I was wondering how to use it with two `torch.Dataset` of different tasks. Also, do I need to shard across processes while creating an Iterable Dataset?


We now have `interleave_datasets` in the API that allows you to cycle/sample with probabilities (with various stopping strategies) through a list of datasets. However, more specific behavior should be implemented manually.

Hi @mariosasko, @pushkalkatara 

I have multi dataset for ASR task. I have multilingual such as: English, China, Japan, Vietnamese. Each dataset is loaded as dataset and they are imbalance dataset. Now I training, I want to equal sampler, it means each batch loader in training has equal number sampler in each language.

For example: Batch_size = 16:
English sample:4
China sample:4
Japan sample:4
Vietnamese sample:4 

How can I do it? I saw that  interleave_datasets only splits data with probability an concatenates all of them after that, it not affect in training data loader. Forgiving me if I wrong.

Thank you for your help.
D
Author: CryptoMiKKi
State: closed

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Local machine/cluster Beam Datasets example/tutorial
Author: shangw-nvidia
State: closed

Hi,

I'm wondering if https://huggingface.co/docs/datasets/beam_dataset.html has an non-GCP or non-Dataflow version example/tutorial? I tried to migrate it to run on DirectRunner and SparkRunner, however, there were way too many runtime errors that I had to fix during the process, and even so I wasn't able to get either runner correctly producing the desired output.

Thanks!
Shang

Comments: I tried to make it run once on the SparkRunner but it seems that this runner has some issues when it is run locally.
From my experience the DirectRunner is fine though, even if it's clearly not memory efficient.

It would be awesome though to make it work locally on a SparkRunner !
Did you manage to make your processing work ?

We've deprecated the Beam API in `datasets`. As part of it, the Beam datasets have also been converted to non-Beam-based to make using them straightforward.
wrong length with datasets 
Author: rabeehk
State: closed

Hi
I have a MRPC dataset which I convert it to seq2seq format, then this is of this format:

`Dataset(features: {'src_texts': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 10)
`

I feed it to a dataloader:
```
dataloader = DataLoader(
            train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
        )
```

now if I type len(dataloader) this is 1, which is wrong, and this needs to be 10. could you assist me please? thanks 


Comments: Also, I cannot first convert it to torch format, since huggingface seq2seq_trainer codes process the datasets afterwards during datacollector function to make it optimize for TPUs. 

sorry I misunderstood length of dataset with dataloader, closed. thanks