NyssaW/github-issues huggingface dataset:
Add tedlium

## Adding a Dataset
- **Name:** *tedlium*
- **Description:** *The TED-LIUM 1-3 corpus is English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech.*
- **Paper:** Homepage: http://www.openslr.org/7/, https://lium.univ-lemans.fr/en/ted-lium2/ &, https://www.openslr.org/51/
- **Data:** http://www.openslr.org/7/
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/tedlium
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Hi @patrickvonplaten 
I can have a look to this dataset later since I am trying to add the OpenSLR dataset https://github.com/huggingface/datasets/pull/2173
Hopefully I have enough space since the compressed file is 21GB. The release 3 is even bigger: 54GB :-0

Resolved via https://github.com/huggingface/datasets/pull/4309
Add VCTK

## Adding a Dataset
- **Name:** *VCTK*
- **Description:** *This CSTR VCTK Corpus includes speech data uttered by 110 English speakers with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.*
- **Paper:** Homepage: https://datashare.ed.ac.uk/handle/10283/3443
- **Data:** https://datashare.ed.ac.uk/handle/10283/3443
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/vctk

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
@patrickvonplaten I'd like to take this, if nobody has already done it. I have added datasets before through the datasets sprint, but I feel rusty on the details, so I'll look at the guide as well as similar audio PRs (#1878 in particular comes to mind). If there is any detail I should be aware of please, let me know! Otherwise, I'll try to write up a PR in the coming days.

That sounds great @jaketae - let me know if you need any help i.e. feel free to ping me on a first PR :-)
test.json has been removed from the limit dataset repo (breaks dataset)

https://github.com/huggingface/datasets/blob/16042b233dbff2a7585110134e969204c69322c3/datasets/limit/limit.py#L51

The URL is not valid anymore since test.json has been removed in master for some reason. Directly referencing the last commit works:

`https://raw.githubusercontent.com/ilmgut/limit_dataset/0707d3989cd8848f0f11527c77dcf168fefd2b23/data`

Comments:
Thanks for the heads up ! I'm opening a PR to fix that
Add CHiME4 dataset

## Adding a Dataset
- **Name:** Chime4
- **Description:** Chime4 is a dataset for automatic speech recognition. It is especially useful for evaluating models in a noisy environment and for multi-channel ASR
- **Paper:** Dataset comes from a channel: http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/ . Results paper: 
- **Data:** http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html
- **Motivation:** So far there are very little datasets for speech in `datasets`. Only `lbirispeech_asr` so far.

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
@patrickvonplaten not sure whether it is still needed, but willing to tackle this issue

Hey @patrickvonplaten, I have managed to download the zip on [here]( http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html) and successfully uploaded all the files on a hugging face dataset: 

https://huggingface.co/datasets/ksbai123/Chime4

However I am getting this error when trying to use the dataset viewer:

![Screenshot 2023-12-27 at 18 40 59](https://github.com/huggingface/datasets/assets/35923560/a5a9ed3d-8dbd-41c4-b83a-4e80728b1450)

Can you take a look and let me know if I have missed any files please

@patrickvonplaten ?

Hi @KossaiSbai,

Thanks for your contribution.

As the issue is not strictly related to the `datasets` library, but to the specific implementation of the CHiME4 dataset, I have opened an issue in the Discussion tab of the dataset: https://huggingface.co/datasets/ksbai123/Chime4/discussions/2
Let's continue the discussion there!

Hi, please help me understand this dataset. I am trying to run ESPnet chime4 recepie. I downloaded the dataset (https://www.chimechallenge.org/challenges/chime4/download) and it doesn't look like the official site (https://www.chimechallenge.org/challenges/chime4/data#Audio):

<img width="220" alt="Image" src="https://github.com/user-attachments/assets/91d2d727-bc65-4c45-ad87-b643fc26a40e" />

There is no 
├── audio
├── transcriptions
└── WSJ0
only 
├── annotations

same in your huggingface card. 
How can I access the missing data?
Add OSCAR dataset card

I added more information and completed the dataset card for OSCAR which was started by @lhoestq in his previous [PR](https://github.com/huggingface/datasets/pull/1824).

Comments:
@lhoestq Thanks for the suggestions! I agree with all of them. Should I accept them one by one or can I accept them all at once? When I try to load the whole diff GitHub is complaining and it does no render them well (probably my browser?) 😅 

I just merged the tables as suggested 😄 . However I noticed something weird, the train sizes are identical for both the original and deduplicated files ... This is not normal, in general the original files are almost twice as big as the deduplicated ones 🤔 

Good catch @pjox ! I just checked and this is because the scripts doesn't handle having several blank lines in a row.
Blank lines introduced by deduplication are currently not ignored so we end up with the same number of examples in the dataset as the original version (but with empty examples...)
I fixed that in this [commit](https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383). I'm re-running the metadata generation for deduplicated configs.

I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

> I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

great, I just wanted to report that I got error message "NonMatchingSplitsSizesError" when I tried to load one of the oscar dataset.

Hi @cahya-wirawan, which configuration of oscar do you have this issue with ?

Ok I see you're having this issue because I haven't updated the sizes yet ! I'm opening a PR

I just checked and indeed there's an issue with the `deduplicated` configurations since the commit I mentioned above.
I'm fixing this by using the new sizes I got yesterday :) 


I just updated the size in the table @pjox it should be good now :) 
I also updated the sizes in the dataset_infos.json in https://github.com/huggingface/datasets/pull/1868 (merged)

Thanks @lhoestq for fixing the issue, it works now

Thank you so much @lhoestq !
Add Tweet Eval Dataset

Closes Draft PR #1407. 

Notes:
1. I have excluded `mapping.txt` from the dataset at it only contained the name mappings, which are already present in the ClassLabels.
2. I have also exluded the textual names for the emojis mentioned in the [mapping](https://github.com/cardiffnlp/tweeteval/blob/main/datasets/emoji/mapping.txt).
3. I do not understand @abhishekkrthakur's example generator on #1407. Maybe he was trying to build up on code from some other dataset.

Requesting @lhoestq  to review.

Comments:
Regarding On-the-fly Data Loading

Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan

Comments:
Possible duplicate

#1776 https://github.com/huggingface/datasets/issues/

really looking PR for this feature

Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).

Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?

Hi @lhoestq,

I apologize for the late response. This answers my question. Thanks a lot.
Add OSCAR dataset card

I started adding the dataset card for OSCAR !

For now it's just basic info for all the different configurations in `Dataset Structure`.
In particular the Data Splits section tells how may samples there are for each config. The Data Instances section show an example for each config, and it also shows the size in MB. Since the Data Instances section is very long the user has to click to expand the info. I was able to generate it thanks to the tools made by  @madlag and @yjernite :D

Cc @pjox could you help me with the other sections ? (Dataset Description, Dataset Creation, Considerations for Using the Data, Additional Information)


Comments:
Hi @lhoestq! When are you planning to release the version with this dataset?

BTW: What a huge README file :astonished:

Next week !

Closing in favor of #1833
Provide better exception message when one of many files results in an exception

I find when I process many files, i.e.

```
train_files = glob.glob('rain*.csv')
validation_files = glob.glob(validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))
```

I sometimes encounter an error due to one of the files being misformed (i.e. no data, or a comma in a field that isn't quoted, etc).

For example, this is the tail of an exception which I suspect is due to a stray comma.

>   File "pandas/_libs/parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read
>   File "pandas/_libs/parsers.pyx", line 783, in pandas._libs.parsers.TextReader._read_low_memory
>   File "pandas/_libs/parsers.pyx", line 827, in pandas._libs.parsers.TextReader._read_rows
>   File "pandas/_libs/parsers.pyx", line 814, in pandas._libs.parsers.TextReader._tokenize_rows
>   File "pandas/_libs/parsers.pyx", line 1951, in pandas._libs.parsers.raise_parser_error
> pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 559, saw 3

It would be nice if the exception trace contained the name of the file being processed (I have 250 separate files!)

Comments:
Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```
Add metrics usage examples and tests

All metrics finally have usage examples and proper fast + slow tests :)

I added examples of usage for every metric, and I use doctest to make sure they all work as expected.

For "slow" metrics such as bert_score or bleurt which require to download + run a transformer model, the download + forward pass are only done in the slow test.
In the fast test on the other hand, the download + forward pass are monkey patched.

Metrics that need to be installed from github are not added to setup.py because it prevents uploading the `datasets` package to pypi.
An additional-test-requirements.txt file is used instead. This file also include `comet` in order to not have to resolve its *impossible* dependencies.

Also `comet` is not tested on windows because one of its dependencies (fairseq) can't be installed in the CI for some reason.

Comments:
pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500

I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!

Comments:
Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```

You saved my life.
Support future datasets

If a dataset is available at the version of the local installation of `datasets` (e.g. 1.2.0), then loading this dataset means loading the script at this version.

However when trying to load a dataset that is only available on master, currently users have to specify `script_version="master"` in `load_dataset` to make it work.

However we could automatically get the dataset from master instead in this case.

I added this feature in this PR.
I also added a warning if a dataset is not available at the version of the local installation of `datasets` but is loaded from master:
```python
>>> load_dataset("silicone", "dyda_da")
Couldn't find file locally at silicone/silicone.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.2.0/datasets/silicone/silicone.py.
The file was picked from the master branch on github instead at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/silicone/silicone.py.
Downloading and preparing dataset silicone/dyda_da (download: 8.46 MiB, generated: 9.39 MiB, post-processed: Unknown size, total: 17.86 MiB) to /Users/quentinlhoest/.cache/huggingface/datasets/silicone/dyda_da/1.0.0/d41d8c0b73c6df035b1369c45774418f0051163ea689b5502b8bda783adf6342...
...
```


Comments:
Unable to add Multi-label Datasets

I am trying to add [CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. The dataset contains two labels per image - `fine label` and `coarse label`. Using just one label in supervised keys as 
`supervised_keys=("img", "fine_label")` raises no issue. But trying `supervised_keys=("img", "fine_label","coarse_label")` leads to this error : 

```python
Traceback (most recent call last):
  File "test_script.py", line 2, in <module>
    d = load_dataset('./datasets/cifar100')
  File "~/datasets/src/datasets/load.py", line 668, in load_dataset
    **config_kwargs,
  File "~/datasets/src/datasets/builder.py", line 896, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "~/datasets/src/datasets/builder.py", line 247, in __init__
    info.update(self._info())
  File "~/.cache/huggingface/modules/datasets_modules/datasets/cifar100/61d2489b2d4a4abc34201432541b7380984ec714e290817d9a1ee318e4b74e0f/cifar100.py", line 79, in _info
    citation=_CITATION,
  File "<string>", line 19, in __init__
  File "~/datasets/src/datasets/info.py", line 136, in __post_init__
    self.supervised_keys = SupervisedKeysData(*self.supervised_keys)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
```
Is there a way I can fix this?

Also, what does adding `supervised_keys` do? Is it necessary? How would I specify `supervised_keys` for a multi-input, multi-label dataset?

Thanks,
Gunjan

Comments:
Thanks for adding this dataset! As far as I know `supervised_keys` is mostly a holdover from TFDS, but isn't really used, so feel free to drop it (@lhoestq  or @thomwolf correct me if I'm wrong). It definitely shouldn't be blocking :) 

I can confirm that it comes from TFDS and is not used at the moment.

Thanks @yjernite @lhoestq 

The template for new dataset makes it slightly confusing. I suppose the comment suggesting its update can be removed.

Closing this issue since it was answered.
Add Hateful Memes Dataset

## Add Hateful Memes Dataset
- **Name:** Hateful Memes
- **Description:** [https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set]( https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set)
- **Paper:** [https://arxiv.org/pdf/2005.04790.pdf](https://arxiv.org/pdf/2005.04790.pdf)
- **Data:** [This link](https://drivendata-competition-fb-hateful-memes-data.s3.amazonaws.com/XjiOc5ycDBRRNwbhRlgH.zip?AWSAccessKeyId=AKIARVBOBDCY4MWEDJKS&Signature=DaUuGgZWUgDHzEPPbyJ2PhSJ56Q%3D&Expires=1612816874)
- **Motivation:** Including multi-modal datasets to 🤗 datasets.

I will be adding this dataset. It requires the user to sign an agreement on DrivenData. So, it will be used with a manual download.

The issue with this dataset is that the images are of different sizes. The image datasets added so far (CIFAR-10 and MNIST) have a uniform shape throughout.
So something like 
```python
 datasets.Array2D(shape=(28, 28), dtype="uint8")
```
won't work for the images. How would I add image features then? I checked `datasets/features.py` but couldn't figure out the appropriate class for this. I'm assuming I would want to avoid re-sizing at all since we want the user to be able to access the original images.

Also, in case I want to load only a subset of the data, since the actual data is around 8.8GB, how would that be possible?

Thanks,
Gunjan

Comments:
I am not sure, but would `datasets.Sequence(datasets.Sequence(datasets.Sequence(datasets.Value("int")))` work?

Also, I found the information for loading only subsets of the data [here](https://github.com/huggingface/datasets/blob/master/docs/source/splits.rst).

Hi @lhoestq,

Request you to check this once.

Thanks,
Gunjan

Hi @gchhablani since Array2D doesn't support images of different sizes, I would suggest to store in the dataset the paths to the image file instead of the image data. This has the advantage of not decompressing the data (images are often compressed using jpeg, png etc.). Users can still apply `.map` to load the images if they want to. Though it would en up being Sequences features.

In the future we'll add support for ragged tensors for this case and update the relevant dataset with this feature.
Add FreebaseQA dataset

Adding FreebaseQA dataset suggested in PR #1435 with minor edits. Also closes that PR.
Requesting @lhoestq to review.

Comments:
Hi ! It looks like this PR contains changes about other datasets than freebase_qa such as DuoRC.

Can you remove these changes please ?

Hi @lhoestq,

I think this happened because of rebasing. I'm unable to remove the duorc commit from the branch.  GEM, Arabic sarcasm datasets are also there. I can't see any merge conflicts, however. Before commiting I always rebase (shouldn't have done that).
Can you explain what is to be done? Should I create a clean PR?

Hi @gchhablani 
I think you can simply create another branch and another PR.

If I understand correctly the github diff is messed up because you rebased instead of merge.
Rebasing is supposed to be used only before pushing the branch the first time, or github messes up the diff.
If you want to include changes from master on a branch that is already push you need to use git merge.

Thanks @lhoestq.

I understand the issue now. I missed the instructions on the template. Sorry for bothering you unnecessarily, I'm pretty new to contributing on GitHub. I'll make a fresh PR.


No problem, I'm not a big fan of this weird behavior tbh.
Thanks for making a new PR

@lhoestq Haha, well, it's not as weird as not reading the [instructions](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#open-a-pull-request-on-the-main-huggingface-repo-and-share-your-work).
Also, I'm enjoying adding new datasets so it's all cool :)
writing Datasets in a human readable format

Hi
I see there is a save_to_disk function to save data, but this is not human readable format, is there a way I could save a Dataset object in a human readable  format to a file like json? thanks @lhoestq 

Comments:
AFAIK, there is currently no built-in method on the `Dataset` object to do this.
However, a workaround is to directly use the Arrow table backing the dataset, **but it implies loading the whole dataset in memory** (correct me if I'm mistaken @lhoestq).

You can convert the Arrow table to a pandas dataframe to save the data as csv as follows:
```python
arrow_table = dataset.data
dataframe = arrow_table.to_pandas()
dataframe.to_csv("/path/to/file.csv")
```

Similarly, you can convert the dataset to a Python dict and save it as JSON:
```python
import json
arrow_table = dataset.data
py_dict = arrow_table.to_pydict()
with open("/path/to/file.json", "w+") as f:
    json.dump(py_dict, f)
```

Indeed this works as long as you have enough memory.
It would be amazing to have export options like csv, json etc. !

It should be doable to implement something that iterates through the dataset batch by batch to write to csv for example.
There is already an `export` method but currently the only export type that is supported is `tfrecords`.

Hi! `datasets` now supports `Dataset.to_csv` and `Dataset.to_json` for saving data in a human readable format.
add github of contributors

This PR will add contributors GitHub id at the end of every dataset cards.

Comments:
@lhoestq Can you confirm if this format is fine? I will update cards based on your feedback.

On HuggingFace side we also have a mapping of hf user => github user (GitHub info used to be required when signing up until not long ago – cc @gary149 @beurkinger) so we can also add a link to HF profile

All the dataset cards have been updated with GitHub ids :)
[GEM] Updated the source link of the data to update correct tokenized version.


Comments:
@mounicam we'll keep the original version in the Turk dataset proper, and use the updated file in the GEM aggregated dataset which I'll add later today

@lhoestq do not merge, I'll close when I've submitted the GEM dataset PR :) 

Closed by https://github.com/huggingface/datasets/pull/1807
Add DuoRC Dataset

Hi,

DuoRC SelfRC is one type of the [DuoRC Dataset](https://duorc.github.io/). DuoRC SelfRC is a crowdsourced Abstractive/Extractive Question-Answering dataset based on Wikipedia movie plots. It contains examples that may have answers in the movie plot, synthesized answers which are not present in the movie plot, or no answers.  I have also added ParaphraseRC - the other type of DuoRC dataset where questions are based on Wikipedia movie plots and answers are based on corresponding IMDb movie plots.

Paper :  [https://arxiv.org/abs/1804.07927](https://arxiv.org/abs/1804.07927)

I want to add this to 🤗 datasets to make it more accessible to the community. I have added  all the details that I could find. Please let me know if anything else is needed from my end.

Thanks,
Gunjan


Comments:
Thanks for approving @lhoestq!
Will apply these changes for the other datasets I've added too.
Update: SWDA - Fixed code to use all metadata features. Added comments and cleaned c…

This is a dataset I currently use my research and I realized some features are not being returned.

Previous code was not using all available metadata and was kind of messy

I fixed code to use all metadata and made some modification to be more efficient and better formatted.


Please let me know if I need to make any changes.

Comments:
@yjernite Pushed all the changes you recommended. Thank you for your help!
Connection error

Hi
I am hitting to the error, help me and thanks.

`train_data = datasets.load_dataset("xsum", split="train")`
`ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/xsum/xsum.py`

Comments:
Hi ! For future references let me add a link to our discussion here : https://github.com/huggingface/datasets/issues/759#issuecomment-770684693

Let me know if you manage to fix your proxy issue or if we can do something on our end to help you :)
Allow loading dataset in-memory

Allow loading datasets either from:
- memory-mapped file (current implementation)
- from file descriptor, copying data to physical memory

Close #708

Comments:
I am wondering how to test their difference...

> ring how to test their difference...

Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.

Otherwise we can still check the difference of RAM used when loading a big chunk of data.

> Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.
> 
> Otherwise we can still check the difference of RAM used when loading a big chunk of data.

@lhoestq I think I found a way: `pa.total_allocated_bytes()` :smirk:
How to use split dataset 

![Capture1](https://user-images.githubusercontent.com/78090287/106057436-cb6a1f00-6111-11eb-8c9c-3658065b1fdf.PNG)

Hey,
I want to split the lambada dataset into corpus, test, train and valid txt files (like penn treebank) but I am not able to achieve this. What I am doing is, executing the lambada.py file in my project but its not giving desired results. Any help will be appreciated!

Comments:
By default, all 3 splits will be loaded if you run the following:

```python
from datasets import load_dataset
dataset = load_dataset("lambada")
print(dataset["train"])
print(dataset["valid"])

```

If you wanted to do load this manually, you could do this:

```python
from datasets import load_dataset
data_files = {
   "train": "data/lambada/train.txt",
   "valid": "data/lambada/valid.txt",
   "test": "data/lambada/test.txt",
}
ds = load_dataset("text", data_files=data_files)
```

Thank you for the quick response!
Not enough disk space (Needed: Unknown size) when caching on a cluster

I'm running some experiments where I'm caching datasets on a cluster and accessing it through multiple compute nodes. However, I get an error when loading the cached dataset from the shared disk.

The exact error thrown:

```bash
>>> load_dataset(dataset, cache_dir="/path/to/cluster/shared/path")
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
```


[`utils.has_sufficient_disk_space`](https://github.com/huggingface/datasets/blob/8a03ab7d123a76ee744304f21ce868c75f411214/src/datasets/utils/py_utils.py#L332) fails on each job because of how the cluster system is designed (`disk_usage(".").free` can't compute on the cluster's shared disk).


This is exactly where the error gets thrown:
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L502

```python
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
    raise IOError(
          "Not enough disk space. Needed: {} (download: {}, generated: {}, post-processed: {})".format(
          utils.size_str(self.info.size_in_bytes or 0),
          utils.size_str(self.info.download_size or 0),
          utils.size_str(self.info.dataset_size or 0),
          utils.size_str(self.info.post_processing_size or 0),
       )
    )

```

What would be a good way to circumvent this? my current fix is to manually comment out that part, but that is not ideal. 
Would it be possible to pass a flag to skip this check on disk space?

Comments:
Hi ! 

What do you mean by "disk_usage(".").free` can't compute on the cluster's shared disk" exactly ?
Does it return 0 ?

Yes, that's right. It shows 0 free space even though there is. I suspect it might have to do with permissions on the shared disk.

```python
>>> disk_usage(".")
usage(total=999999, used=999999, free=0)
```

That's an interesting behavior...
Do you know any other way to get the free space that works in your case ?
Also if it's a permission issue could you try fix the permissions and let mus know if that helped ?

I think its an issue on the clusters end (unclear exactly why -- maybe something with docker containers?), will close the issue

Were you able to figure it out?

@philippnoah I had fixed it with a small hack where I patched `has_sufficient_disk_space` to always return `True`. you can do that with an import without having to modify the `datasets` package

@olinguyen Thanks for the suggestion, it works but I had to to edit builder.py in the installed package. Can you please explain how were you able to do this using import?

I was able to patch the builder code in my notebook before the load data call and it works. 
```
import datasets
datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
```

import datasets
    datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
JSONDecodeError on JSON with multiple lines

Hello :),

I have been trying to load data using a JSON file. Based on the [docs](https://huggingface.co/docs/datasets/loading_datasets.html#json-files), the following format is supported:

```json
{"key1":11, "key2":12, "key3":13}
{"key1":21, "key2":22, "key3":23}
```
 But, when I try loading a dataset with the same format, I get a JSONDecodeError : `JSONDecodeError: Extra data: line 2 column 1 (char 7142)`. Now, this is expected when using `json` to load a JSON file. But I was wondering if there are any special arguments to pass when using `load_dataset` as the docs suggest that this format is supported.

When I convert the JSON file to a list of dictionaries format, I get AttributeError: `AttributeError: 'list' object has no attribute 'keys'`. So, I can't convert them to list of dictionaries either.

Please let me know :)

Thanks,
Gunjan

Comments:
Hi !

The `json` dataset script does support this format. For example loading a dataset with this format works on my side:
```json
{"key1":11, "key2":12, "key3":13}
{"key1":21, "key2":22, "key3":23}
```

Can you show the full stacktrace please ? Also which version of datasets and pyarrow are you using ?



Hi Quentin!

I apologize for bothering you. There was some issue with my pyarrow version as far as I understand. I don't remember the exact version I was using as I didn't check it.

I repeated it with `datasets 1.2.1` and `pyarrow  2.0.0` and it worked.

Closing this issue. Again, sorry for the bother.

Thanks,
Gunjan
Dataset Examples Explorer

In the Older version of the Dataset, there are a useful Dataset Explorer that allow user to visualize the examples (training, test and validation) of a particular dataset, it is no longer there in current version.

Hope HuggingFace can re-enable the feature that at least allow viewing of  the first 20  examples of a particular dataset, or alternatively can extract 20 examples for each datasets and make those part of the Dataset Card Documentation.

Comments:
Hi @ChewKokWah,

We're working on it! In the meantime, you can still find the dataset explorer at the following URL: https://huggingface.co/datasets/viewer/

Glad to see that it still exist, this existing one is more than good enough for me, it is feature rich, simple to use and concise. 
Hope similar feature can be retain in the future version.
AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 

I'm using Colab. And suddenly this morning, there is this error. Have a look below!

![screenshot-colab research google com-2021 01 26-08-15-36](https://user-images.githubusercontent.com/45964869/105799890-fdaf3b80-5fae-11eb-8f06-11b65cdccc30.png)


Comments:
Hi ! I'm not able to reproduce the issue. Can you try restarting your runtime ?

The PyExtensionType is available in pyarrow starting 0.17.1 iirc. If restarting your runtime doesn't fix this, can you try updating pyarrow ?
```
pip install pyarrow --upgrade
```

We should bump up the version test of pyarrow maybe no?

https://github.com/huggingface/datasets/blob/master/src/datasets/__init__.py#L60

Yes indeed.

Also it looks like Pyarrow 3.0.0 got released on pypi 10 hours ago. This might be related to the bug, I'll investigate
EDIT: looks like the 3.0.0 release doesn't have unexpected breaking changes for us, so I don't think the issue comes from that

Maybe colab moved to pyarrow 0.16 by default (instead of 0.14 before)?

Installing datasets installs pyarrow>=0.17.1 so in theory it doesn't matter which version of pyarrow colab has by default (which is currently pyarrow 0.14.1).

Also now the colab runtime refresh the pyarrow version automatically after the update from pip (previously you needed to restart your runtime).

I guess what happened is that Colab didn't refresh pyarrow for some reason, and the AttributeError was raised *before* the pyarrow version check from `datasets` at https://github.com/huggingface/datasets/blob/master/src/datasets/__init__.py#L60

Yes colab doesn’t reload preloaded library unless you restart the instance. Maybe we should move the check on top of the init 

Yes I'll do that :)

I updated the pyarrow version check in #1782

restarting worked for me
> ```

yes it worked for me
Ignore definition line number of functions for caching

As noticed in #1718 , when a function used for processing with `map` is moved inside its python file, then the change of line number causes the caching mechanism to consider it as a different function. Therefore in this case, it recomputes everything.

This is because we were not ignoring the line number definition for such functions (even though we're doing it for lambda functions).

For example this code currently prints False:
```python
from datasets.fingerprint import Hasher

# define once
def foo(x):
    return x

h = Hasher.hash(foo)

# define a second time elsewhere
def foo(x):
    return x

print(h == Hasher.hash(foo))
```

I changed this by ignoring the line number for all functions.

Comments:
Narrative QA Manual

Submitting the manual version of Narrative QA script which requires a manual download from the original repository

Comments:
@lhoestq sorry I opened a new pull request because of some issues with the previous code base. This pull request is originally from #1364

Excellent comments. Thanks for those valuable suggestions. I changed everything as you have pointed out :) 

I've copied the same template as NarrativeQA now. Please let me know if this is fine. 

> Awesome thank you !!
> This looks all good :)
> 
> Just before we merge, I was wondering if you knew why the number of examples in the train set went from 1102 to 32747 in your last commit ? I can't see why the changes in the code would cause such a big difference

Ok the change was the way I presented the data. 
In my previous code, I presented a story with a list of questions-answers related to the story per sample. So the total 1102 was the number of stories (not questions) in the train set. 

In the case of `NarrativeQA`, the code presented each sample data with one single question. So the story gets replicated as many times based on number of questions per story. I felt this was not really memory efficient so I had coded the way I did earlier. 

But since this would be inconsistent as you pointed out, I modified my code to suit the `NarrativeQA` approach. Hope it's clear now :) 

Ok I see ! that makes sense

Thanks for your time and helping me with all this :) Really appreciate the hardwork you guys do.
GPT2 MNLI training using run_glue.py

Edit: I'm closing this because I actually meant to post this in `transformers `not `datasets`

Running this on Google Colab,

```
!python run_glue.py \
  --model_name_or_path gpt2 \
  --task_name mnli \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 10 \
  --gradient_accumulation_steps 32\
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir models/gpt2/mnli/
```

I get the following error,

```
 "Asking to pad but the tokenizer does not have a padding token. "
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

Do I need to modify the trainer to work with GPT2 ?

Comments:
[Question & Bug Report] Can we preprocess a dataset on the fly?

I know we can use `Datasets.map` to preprocess a dataset, but I'm using it with very large corpus which generates huge cache file (several TB cache from a 400 GB text file). I have no disk large enough to save it.  Can we preprocess a dataset on the fly without generating cache?

BTW, I tried raising `writer_batch_size`. Seems that argument doesn't have any effect when it's larger than `batch_size`, because you are saving all the batch instantly after it's processed. Please check the following code:

https://github.com/huggingface/datasets/blob/0281f9d881f3a55c89aeaa642f1ba23444b64083/src/datasets/arrow_dataset.py#L1532

Comments:
We are very actively working on this. How does your dataset look like in practice (number/size/type of files)?

It's a text file with many lines (about 1B) of Chinese sentences. I use it to train language model using https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm_wwm.py

Indeed I will submit a PR in a fez days to enable processing on-the-fly :)
This can be useful in language modeling for tokenization, padding etc.


any update on this issue? ...really look forward to use it 

Hi @acul3,

Please look at the discussion on a related Issue #1825. I think using `set_transform` after building from source should do.

@gchhablani thank you so much

will try look at it
Efficient ways to iterate the dataset

For a large dataset that does not fits the memory, how can I select only a subset of features from each example?

If I iterate over the dataset and then select the subset of features one by one, the resulted memory usage will be huge. Any ways to solve this?

Thanks

Comments:
It seems that selecting a subset of colums directly from the dataset, i.e., dataset["column"], is slow.

I was wrong, ```dataset["column"]``` is fast.
is it possible to make slice to be more compatible like python list and numpy?

Hi,
see below error:
```
AssertionError: Requested slice [:10000000000000000] incompatible with 20 examples.
```

Comments:
Hi ! Thanks for reporting.
I am working on changes in the way data are sliced from arrow. I can probably fix your issue with the changes I'm doing.
If you have some code to reproduce the issue it would be nice so I can make sure that this case will be supported :)
I'll make a PR in a few days 

Good if you can take care at your side.
Here is the [colab notebook](https://colab.research.google.com/drive/19c-abm87RTRYgW9G1D8ktfwRW95zDYBZ?usp=sharing)
Adding SICK dataset

Hi
It would be great to include SICK dataset.

## Adding a Dataset
- **Name:** SICK
- **Description:** a well known entailment dataset 
- **Paper:** http://marcobaroni.org/composes/sick.html
- **Data:** http://marcobaroni.org/composes/sick.html
- **Motivation:** this is an important NLI benchmark

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).



thanks

Comments:
Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/csv/csv.py

Hi,
When I load_dataset from local csv files, below error happened, looks raw.githubusercontent.com was blocked by the chinese government. But why it need to download csv.py? should it include when pip install the dataset?

```
Traceback (most recent call last):
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
    max_retries=download_config.max_retries,
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 617, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/csv/csv.py

```

Comments:
I temporary manually download csv.py as custom dataset loading script

Indeed in 1.2.1 the script to process csv file is downloaded. Starting from the next release though we include the csv processing directly in the library.
See PR #1726 
We'll do a new release soon :)

Thanks.
how can I combine 2 dataset with different/same features?

to combine 2 dataset by one-one map like ds = zip(ds1, ds2):
ds1: {'text'}, ds2: {'text'}, combine ds:{'src', 'tgt'} 
or different feature:
ds1: {'src'}, ds2: {'tgt'}, combine ds:{'src', 'tgt'}

Comments:
Hi ! Currently we don't have a way to `zip` datasets but we plan to add this soon :)
For now you'll need to use `map` to add the fields from one dataset to the other. See the comment here for more info : https://github.com/huggingface/datasets/issues/853#issuecomment-727872188

Good to hear.
Currently I did not use map , just fetch src and tgt from the 2 dataset and merge them.
It will be a release if you can deal with it at the backend.
Thanks.

Hi! You can rename the columns and concatenate the datasets along `axis=1` to get the desired result as follows:
```python
ds1 = ds1.rename_column("text", "src")
ds2 = ds2.rename_column("text", "tgt")
ds = datasets.concatenate_datasets(["ds1", "ds2"], axis=1)
```
Add Librispeech ASR

This PR adds the librispeech asr dataset: https://www.tensorflow.org/datasets/catalog/librispeech

There are 2 configs: "clean" and "other" whereas there are two "train" datasets for "clean", hence the name "train.100" and "train.360".

As suggested by @lhoestq, due to the enormous size of the dataset in `.arrow` format, the speech files are not directly prepared to a float32-array, but instead just the path to the array file is stored.

Comments:
> Awesome thank you !
> 
> The dummy data are quite big but it was expected given that the raw files are flac files.
> Given that the script doesn't even read the flac files I think we can remove them. Or maybe use empty flac files (see [here](https://hydrogenaud.io/index.php?topic=118685.0) for example). What do you think ?
> 
> We'll find a better solution to be able to have bigger dummy_data (max 1MB instead of a few KB, maybe using git LFS.

Hmm, I already made the dummy data as small as possible (a single flac filie per split only). I'd like to keep them at least to have complete dummy data and don't think 500KB for all datasets together is a problem (the long-range summarization datasets are similarly heavy). The moment we allow dummy data to be loaded directly for testing, we need the flac files IMO.

But I agree that longterm, we need a better solution for the dummy data (maybe stop hosting it on github to not make the repo too heavy)
Issues when run two programs compute the same metrics

I got the following error when running two different programs that both compute sacreblue metrics. It seems that both read/and/write to the same location (.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow) where it caches the batches:

```
File "train_matching_min.py", line 160, in <module>ch_9_label
    avg_loss = valid(epoch, args.batch, args.validation, args.with_label)
  File "train_matching_min.py", line 93, in valid
    bleu += eval.compute()
  File "/u/tlhoang/projects/seal/match/models/eval.py", line 23, in compute
    return self.metric.compute()['score']
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/metric.py", line 387, in compute
    self._finalize()
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/metric.py", line 355, in _finalize
    self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 231, in read_files
    pa_table = self._read_files(files)
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 170, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 299, in _get_dataset_from_filename
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 481, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 1819307375 metadata bytes, but only read 454396
``` 

Comments:
Hi ! To avoid collisions you can specify a `experiment_id` when instantiating your metric using `load_metric`. It will replace "default_experiment" with the experiment id that you provide in the arrow filename. 

Also when two `experiment_id` collide we're supposed to detect it using our locking mechanism. Not sure why it didn't work in your case. Could you share some code that reproduces the issue ? This would help us investigate.

Thank you for your response. I fixed the issue by set "keep_in_memory=True" when load_metric. 
I cannot share the entire source code but below is the wrapper I wrote:

```python
class Evaluation:
    def __init__(self, metric='sacrebleu'):
        # self.metric = load_metric(metric, keep_in_memory=True)
        self.metric = load_metric(metric)

    def add(self, predictions, references):
        self.metric.add_batch(predictions=predictions, references=references)

    def compute(self):
        return self.metric.compute()['score']
```

Then call the given wrapper as follows:

```python
eval = Evaluation(metric='sacrebleu')
for query, candidates,  labels in tqdm(dataset):
    predictions = net.generate(query)
    references = [[s] for s in labels]
    eval.add(predictions, references)
    if n % 100 == 0:
        bleu += eval.compute()
        eval = Evaluation(metric='sacrebleu')