konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Add MDD Dataset

- **Name:** *MDD Dataset*
- **Description:** The Movie Dialog dataset (MDD) is designed to measure how well models can perform at goal and non-goal orientated dialog centered around the topic of movies (question answering, recommendation and discussion), from various movie reviews sources such as MovieLens and OMDb.
- **Paper:** [arXiv](https://arxiv.org/pdf/1511.06931.pdf)
- **Data:**  https://research.fb.com/downloads/babi/
- **Motivation:** This is one of the popular dialog datasets, a part of Facebook Research's "bAbI project".

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


**Note**: I haven't included the following from the data files: `entities` (the file containing list of all entities in the first three subtasks), `dictionary`(the dictionary of words they use in their models), `movie_kb`(contains the knowledge base of information about the movies, actors and other entities that are mentioned in the dialogs). Please let me know if those are needed, and if yes, should I make separate configurations for them?

Hi @lhoestq,

I have added changes from review.

Thanks for approving :)
Build custom dataset to fine-tune Wav2Vec2

Thank you for your recent tutorial on how to finetune Wav2Vec2 on a custom dataset. The example you gave here (https://huggingface.co/blog/fine-tune-xlsr-wav2vec2) was on the CommonVoice dataset. However, what if I want to load my own dataset?  I have a manifest (transcript and their audio files) in a JSON file. 


@lhoestq - We could simply use the "general" json dataset for this no? 

Sure you can use the json loader
```python
data_files = {"train": "path/to/your/train_data.json", "test": "path/to/your/test_data.json"}
train_dataset = load_dataset("json", data_files=data_files, split="train")
test_dataset = load_dataset("json", data_files=data_files, split="test")
```

You just need to make sure that the data contain the paths to the audio files.
If not, feel free to use `.map()` to add them.

Many thanks! that was what I was looking for.
Multilingual dIalogAct benchMark (miam)

My collaborators (@EmileChapuis, @PierreColombo) and I within the Affective Computing team at Telecom Paris would like to anonymously publish the miam dataset. It is assocated with a publication currently under review. We will update the dataset with full citations once the review period is over.

Hello. All aforementioned changes have been made. I've also re-run black on miam.py. :-)

I will run isort again. Hopefully it resolves the current check_code_quality test failure.

Once the review period is over, feel free to open a PR to add all the missing information ;)

Hi! I will follow up right now with one more pull request as I have new anonymous citation information to include.
Preserve column ordering in Dataset.rename_column

Currently `Dataset.rename_column` doesn't necessarily preserve the order of the columns:
```python
>>> from datasets import Dataset
>>> d = Dataset.from_dict({'sentences': ["s1", "s2"], 'label': [0, 1]})
>>> d
Dataset({
    features: ['sentences', 'label'],
    num_rows: 2
})
>>> d.rename_column('sentences', 'text')
Dataset({
    features: ['label', 'text'],
    num_rows: 2
})
```
This PR fixes this.

Not sure why CI isn't triggered.

@lhoestq Can you please help me with this? 

I don't know how to trigger it manually, but an empty commit should do the job
Add CBT dataset

This PR adds the [CBT Dataset](https://arxiv.org/abs/1511.02301).

Note that I have also added the `raw` dataset as a separate configuration. I couldn't find a suitable "task" for it in YAML tags.

The dummy files have one example each, as the examples are slightly big. For `raw` dataset, I just used top few lines, because they are entire books and would take up a lot of space.

Let me know in case of any issues.

Hi @lhoestq,

I have added changes from the review.

Thanks for approving @lhoestq
ValueError: datasets' indices [1] come from memory and datasets' indices [0] come from disk

Hi there,

I am trying to concat two datasets that I've previously saved to disk via `save_to_disk()` like so (note that both are saved as `DataDict`, `PATH_DATA_CLS_*` are `Path`-objects):
```python
concatenate_datasets([load_from_disk(PATH_DATA_CLS_A)['train'], load_from_disk(PATH_DATA_CLS_B)['train']])
```
Yielding the following error:
```python
ValueError: Datasets' indices should ALL come from memory, or should ALL come from disk.
However datasets' indices [1] come from memory and datasets' indices [0] come from disk.
```
Been trying to solve this for quite some time now. Both `DataDict` have been created by reading in a `csv` via `load_dataset` and subsequently processed using the various `datasets` methods (i.e. filter, map, remove col, rename col). Can't figure out tho...

`load_from_disk(PATH_DATA_CLS_A)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 785
})
```
`load_from_disk(PATH_DATA_CLS_B)['train']` yields:
```python
Dataset({
    features: ['labels', 'text'],
    num_rows: 3341
})
```

Hi ! To help me understand the situation, can you print the values of  `load_from_disk(PATH_DATA_CLS_A)['train']._indices_data_files` and `load_from_disk(PATH_DATA_CLS_B)['train']._indices_data_files` ?
They should both have a path to an arrow file

Also note that from #2025 concatenating datasets will no longer have such restrictions.

Sure, thanks for the fast reply!

For dataset A: `[{'filename': 'drive/MyDrive/data_target_task/dataset_a/train/cache-4797266bf4db1eb7.arrow'}]`
For dataset B: `[]`

No clue why for B it returns nothing. `PATH_DATA_CLS_B` is exactly the same in `save_to_disk` and `load_from_disk`... Also I can verify that the folder physically exists under 'drive/MyDrive/data_target_task/dataset_b/'

In the next release you'll be able to concatenate any kinds of dataset (either from memory or from disk).

For now I'd suggest you to flatten the indices of the A and B datasets. This will remove the indices mapping and you will be able to concatenate them. You can flatten the indices with
```python
dataset = dataset.flatten_indices()
```

Indeed this works. Not the most elegant solution, but it does the trick. Thanks a lot!
outdated dataset_infos.json might fail verifications

The [doc2dial/dataset_infos.json](https://github.com/huggingface/datasets/blob/master/datasets/doc2dial/dataset_infos.json) is outdated. It would fail data_loader when verifying download checksum etc..

Could you please update this file or point me how to update this file?

Thank you.

Hi ! Thanks for reporting.

To update the dataset_infos.json you can run:
```
datasets-cli test ./datasets/doc2dial --all_configs --save_infos --ignore_verifications
```

Fixed by #2041, thanks again @songfeng !
Fix: Wikipedia - save memory by replacing root.clear with elem.clear

see: https://github.com/huggingface/datasets/issues/2031

What I did:
- replace root.clear with elem.clear
- remove lines to get root element
- $ make style
- $ make test
  - some tests required some pip packages, I installed them.

test results on origin/master and my branch are same. I think it's not related on my modification, isn't it?
```
==================================================================================== short test summary info ====================================================================================
FAILED tests/test_arrow_writer.py::TypedSequenceTest::test_catch_overflow - AssertionError: OverflowError not raised
============================================================= 1 failed, 2332 passed, 5138 skipped, 70 warnings in 91.75s (0:01:31) ==============================================================
make: *** [Makefile:19: test] Error 1

```

Is there anything else I should do?

The error you got is minor and appeared in the last version of pyarrow, we'll fix the CI to take this into account. You can ignore it
Cannot load wikitext

when I execute these codes
```
>>> from datasets import load_dataset
>>> test_dataset = load_dataset("wikitext")
```

I got an error,any help?

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/load.py", line 589, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/home/xxx/anaconda3/envs/transformer/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 487, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/wikitext/wikitext.py
```

Solved!
Use Arrow filtering instead of writing a new arrow file for Dataset.filter

Currently the filter method reads the dataset batch by batch to write a new, filtered, arrow file on disk. Therefore all the reading + writing can take some time.

Using a mask directly on the arrow table doesn't do any read or write operation therefore it's significantly quicker.

I think there are two cases:
- if the dataset doesn't have an indices mapping, then one can simply use the arrow filtering on the main arrow table `dataset._data.filter(...)`
- if the dataset an indices mapping, then the mask should be applied on the indices mapping table `dataset._indices.filter(...)`

The indices mapping is used to map between the idx at `dataset[idx]` in `__getitem__` and the idx in the actual arrow table.

The new filter method should therefore be faster, and allow users to pass either a filtering function (that returns a boolean given an example), or directly a mask.

Feel free to discuss this idea in this thread :)

One additional note: the refactor at #2025 would make all the pickle-related stuff work directly with the arrow filtering, so that we only need to change the Dataset.filter method without having to deal with pickle.

cc @theo-m @gchhablani 

related issues: #1796 #1949 

Actually table.filter returns a new table in memory, which can fill users RAM.

Therefore it's not a good solution if we want to keep supporting bigger than RAM datastes
wikipedia.py generator that extracts XML doesn't release memory

I tried downloading Japanese wikipedia, but it always failed because of out of memory maybe.

I found that the generator function that extracts XML data in wikipedia.py doesn't release memory in the loop.

https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L464-L502

`root.clear()` intend to clear memory, but it doesn't.
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L490
https://github.com/huggingface/datasets/blob/13a5b7db992ad5cf77895e4c0f76595314390418/datasets/wikipedia/wikipedia.py#L494
I replaced them with `elem.clear()`, then it seems to work correctly.

here is the notebook to reproduce it.
https://gist.github.com/miyamonz/dc06117302b6e85fa51cbf46dde6bb51#file-xtract_content-ipynb

Hi @miyamonz 
Thanks for investigating this issue, good job !
It would be awesome to integrate your fix in the library, could you open a pull request ?

OK! I'll send it later.
Add Romanian to XQuAD

On Jan 18, XQuAD was updated with a new Romanian validation file ([xquad commit link](https://github.com/deepmind/xquad/commit/60cac411649156efb6aab9dd4c9cde787a2c0345))


Hi ! Thanks for updating XQUAD :)

The slow test is failing though since there's no dummy data nor metadata in dataset_infos.json for the romanian configuration.

Could you please generate the dummy data with
```
datasets-cli dummy_data ./datasets/xquad --auto_generate --json_field data
```
This will update all the dummy data files, and also add the new one for the romanian configuration.


You can also update the metadata with
```
datasets-cli test ./datasets/xquad --name xquad.ro --save_infos
```
This will update the dataset_infos.json file with the metadata of the romanian config :)

Thanks in advance !

Hello Quentin, and thanks for your help.

I found that running

```python
datasets-cli test ./datasets/xquad --name xquad.ro --save_infos
```

was not enough to pass the slow tests, because it was not adding the new `xquad.ro.json` checksum to the other configs infos and becuase of that an `UnexpectedDownloadedFile` error was being thrown, so instead I used:

```python
datasets-cli test ./datasets/xquad --save_infos --all_configs --ignore_verifications
```

`--ignore_verifications` was necessary to bypass the same `UnexpectedDownloadedFile` error.

Additionally, I deleted `dummy_data_copy.zip` and the `copy.sh` script because they both seem now unnecessary.

The slow tests for both the real and dummy data now pass successfully, so I hope that I didn't mess anything up :)


You're right, you needed the `--ignore_verifications` flag !
Thanks for updating them :)

Although I just noticed that the new dummy_data.zip files are quite big (170KB each) because they contain the json files of all the languages, while only one json file per language is necessary. Could you remove the unnecessary json files to reduce the size of the dummy_data.zip files if you don't mind ?

Done. I created a script (`remove_unnecessary_langs.sh`) to automate the process.
ValueError when rename_column on splitted dataset

Hi there,
I am loading `.tsv` file via `load_dataset` and subsequently split the rows into training and test set via the `ReadInstruction` API like so:

```python
split = {
    'train': ReadInstruction('train', to=90, unit='%'),
    'test': ReadInstruction('train', from_=-10, unit='%')
}

dataset = load_dataset(
    path='csv',               # use 'text' loading script to load from local txt-files
    delimiter='\t',           # xxx
    data_files=text_files,    # list of paths to local text files
    split=split,              # xxx
)

dataset
```

Part of output:
```python
DatasetDict({
    train: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 900
    })
    test: Dataset({
        features: ['sentence', 'sentiment'],
        num_rows: 100
    })
})
```
Afterwards I'd like to rename the 'sentence' column to 'text' in order to be compatible with my modelin pipeline. If I run the following code I experience a `ValueError` however:
```python
dataset['train'].rename_column('sentence', 'text')
```
```python
/usr/local/lib/python3.7/dist-packages/datasets/splits.py in __init__(self, name)
    353         for split_name in split_names_from_instruction:
    354             if not re.match(_split_re, split_name):
--> 355                 raise ValueError(f"Split name should match '{_split_re}'' but got '{split_name}'.")
    356 
    357     def __str__(self):

ValueError: Split name should match '^\w+(\.\w+)*$'' but got 'ReadInstruction('.
```
In particular, these behavior does not arise if I use the deprecated `rename_column_` method. Any idea what causes the error? Would assume something in the way I defined the split.

Thanks in advance! :)

Hi,

This is a bug so thanks for reporting it. `Dataset.__setstate__`  is the problem, which is called when `Dataset.rename_column` tries to copy the dataset with `copy.deepcopy(self)`. This only happens if the `split` arg in `load_dataset` was defined as `ReadInstruction`.

To overcome this issue, use the named splits API (for now):
```python
train_ds, test_ds = load_dataset(
    path='csv',               
    delimiter='\t',          
    data_files=text_files,    
    split=['train[:90%]', 'train[-10%:]'],
)

train_ds = train_ds.rename_column('sentence', 'text')
```

This has been fixed in #2043 , thanks @mariosasko 
The fix is available on master and we'll do a new release soon :)

feel free to re-open if you still have issues
Interactively doing  save_to_disk and load_from_disk corrupts the datasets object?

 dataset_info.json file saved after using  save_to_disk gets corrupted as follows. 
 
 
![image](https://user-images.githubusercontent.com/16892570/110568474-ed969880-81b7-11eb-832f-2e5129656016.png)

Is there a way to disable the cache that will save to /tmp/huggiface/datastes ? 
I have a feeling there is a serious issue with cashing.

Hi,

Can you give us a minimal reproducible example? This [part](https://huggingface.co/docs/datasets/master/processing.html#controling-the-cache-behavior) of the docs explains how to control caching.
Md gender card update

I updated the descriptions of the datasets as they appear in the HF repo and the descriptions of the source datasets according to what I could find from the paper and the references. I'm still a little unclear about some of the fields of the different configs, and there was little info on the word list and name list. I'll contact the authors to see if they have any additional information or suggested changes.

Link to the card: https://github.com/mcmillanmajora/datasets/blob/md-gender-card/datasets/md_gender_bias/README.md

dataset card* @sgugger :p 

Ahah that's what I wanted to say @lhoestq, thanks for fixing. Not used to review the Datasets side ;-)
No upstream branch

Feels like the documentation on adding a new dataset is outdated?

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L49-L54

There is no upstream branch on remote. 

What's the issue exactly ?

Given an `upstream` remote repository with url `https://github.com/huggingface/datasets.git`, you can totally rebase from `upstream/master`.

It's mentioned at the beginning how to add the `upstream` remote repository

https://github.com/huggingface/datasets/blob/987df6b4e9e20fc0c92bc9df48137d170756fd7b/ADD_NEW_DATASET.md#L10-L14

~~What difference is there with the default `origin` remote that is set when you clone the repo?~~ I've just understood that this applies to **forks** of the repo 🤡
Local testing fails

I'm following the CI setup as described in 

https://github.com/huggingface/datasets/blob/8eee4fa9e133fe873a7993ba746d32ca2b687551/.circleci/config.yml#L16-L19

in a new conda environment, at commit https://github.com/huggingface/datasets/commit/4de6dbf84e93dad97e1000120d6628c88954e5d4

and getting

```
FAILED tests/test_caching.py::RecurseDumpTest::test_dump_ipython_function - TypeError: an integer is required (got type bytes)
1 failed, 2321 passed, 5109 skipped, 10 warnings in 124.32s (0:02:04)
```

Seems like a discrepancy with CI, perhaps a lib version that's not controlled? 
Tried with `pyarrow=={1.0.0,0.17.1,2.0.0}`

I'm not able to reproduce on my side.
Can you provide the full stacktrace please ?
What version of `python` and `dill` do you have ? Which OS are you using ?

```
co_filename = '<ipython-input-2-e0383a102aae>', returned_obj = [0]
                                                                                                                                                                       
    def create_ipython_func(co_filename, returned_obj):
        def func():
            return returned_obj
     
        code = func.__code__
>       code = CodeType(*[getattr(code, k) if k != "co_filename" else co_filename for k in code_args])
E       TypeError: an integer is required (got type bytes)

tests/test_caching.py:152: TypeError
```

Python 3.8.8 
dill==0.3.1.1


I managed to reproduce. This comes from the CodeType init signature that is different in python 3.8.8
I opened a PR to fix this test
Thanks !
Ambiguous documentation

https://github.com/huggingface/datasets/blob/2ac9a0d24a091989f869af55f9f6411b37ff5188/templates/new_dataset_script.py#L156-L158

Looking at the template, I find this documentation line to be confusing, the method parameters don't include the `gen_kwargs` so I'm unclear where they're coming from.

Happy to push a PR with a clearer statement when I understand the meaning.

Hi @theo-m !

A few lines above this line, you'll find that the `_split_generators` method returns a list of `SplitGenerator`s objects:

```python
datasets.SplitGenerator(
    name=datasets.Split.VALIDATION,
    # These kwargs will be passed to _generate_examples
    gen_kwargs={
        "filepath": os.path.join(data_dir, "dev.jsonl"),
        "split": "dev",
    },
),
```

Notice the `gen_kwargs` argument passed to the constructor of `SplitGenerator`: this dict will be unpacked as keyword arguments to pass to the `_generat_examples` method (in this case the `filepath` and `split` arguments).

Let me know if that helps!

Oh ok I hadn't made the connection between those two, will offer a tweak to the comment and the template then - thanks!
Fix various typos/grammer in the docs

This PR:
* fixes various typos/grammer I came across while reading the docs
* adds the "Install with conda" installation instructions

Closes #1959 

What do yo think of the documentation btw ?
What parts would you like to see improved ?

I like how concise and straightforward the docs are.

Few things that would further improve the docs IMO:
* the usage example of `Dataset.formatted_as` in https://huggingface.co/docs/datasets/master/processing.html
* the "Open in Colab" button would be nice where it makes sense (we can borrow this from the transformers project + link to HF Forum)
How to not load huggingface datasets into memory 

Hi
I am running this example from transformers library version 4.3.3:
(Here is the full documentation https://github.com/huggingface/transformers/issues/8771 but the running command should work out of the box)

 USE_TF=0  deepspeed  run_seq2seq.py --model_name_or_path google/mt5-base --dataset_name wmt16 --dataset_config_name ro-en --source_prefix "translate English to Romanian: " --task translation_en_to_ro   --output_dir /test/test_large  --do_train --do_eval --predict_with_generate  --max_train_samples 500   --max_val_samples 500  --max_source_length 128 --max_target_length 128 --sortish_sampler --per_device_train_batch_size 8   --val_max_target_length 128 --deepspeed ds_config.json --num_train_epochs 1 --eval_steps 25000 --warmup_steps 500 --overwrite_output_dir

(Here please find the script: https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py)

If you do not pass max_train_samples in above command to load the full dataset, then I get memory issue on a gpu with 24 GigBytes of memory.
 
I need to train large-scale mt5 model on large-scale datasets of wikipedia (multiple of them concatenated or other datasets in multiple languages like OPUS), could you help me how I can avoid loading the full data into memory? to make the scripts not related to data size? 

In above example, I was hoping the script could work without relying on dataset size, so I can still train the model without subsampling training set.

thank you so much @lhoestq for your great help in advance




So maybe a summary here: 
If I could fit a large model with batch_size = X into memory, is there a way I could train this model for huge datasets with keeping setting the same? thanks 

The `datastets` library doesn't load datasets into memory. Therefore you can load a dataset that is terabytes big without filling up your RAM.

The only thing that's loaded into memory during training is the batch used in the training step.
So as long as your model works with batch_size = X, then you can load an even bigger dataset and it will work as well with the same batch_size.

Note that you still have to take into account that some batches take more memory than others, depending on the texts lengths. If it works for a batch with batch_size = X and with texts of maximum length, then it will work for all batches.

In your case I guess that there are a few long sentences in the dataset. For those long sentences you get a memory error on your GPU because they're too long. By passing `max_train_samples` you may have taken a subset of the dataset that only contain short sentences. That's probably why in your case it worked only when you set `max_train_samples`.
I'd suggest you to reduce the batch size so that the batches with long sentences can be loaded on the GPU.

Let me know if that helps or if you have other questions
Messages are being printed to the `stdout`

In this code segment, we can see some messages are being printed to the `stdout`.
https://github.com/huggingface/datasets/blob/7e60bb509b595e8edc60a87f32b2bacfc065d607/src/datasets/builder.py#L545-L554
According to the comment, it is done intentionally, but I don't really understand why don't we log it with a higher level or print it directly to the `stderr`.
In my opinion, this kind of messages should never printed to the stdout. At least some configuration/flag should make it possible to provide in order to explicitly prevent the package to contaminate the stdout.


This is expected to show this message to the user via stdout.
This way the users see it directly and can cancel the downloading if they want to.
Could you elaborate why it would be better to have it in stderr instead of stdout ?

@lhoestq, sorry for the late reply

I completely understand why you decided to output a message that is always shown. The only problem is that the message is printed to the `stdout`. For example, if the user runs `python run_glue.py > log_file`, it will redirect `stdout` to the file named  `log_file`, and the message will not be shown to the user.

Instead, we should print this message to `stderr`.  Even in the case of `python run_glue.py > log_file` only `stdout` is being redirected and so the message is always shown.

We now log these messages to `stderr` (the built-in `logging` module's default behavior).
Empty evidence document ("provenance") in KILT ELI5 dataset

In the  original KILT benchmark(https://github.com/facebookresearch/KILT), 

all samples has its evidence document (i.e. wikipedia page id) for prediction.

For example, a sample in ELI5 dataset has the format including provenance (=evidence document) like this

`{"id": "1kiwfx", "input": "In Trading Places (1983, Akroyd/Murphy) how does the scheme at the end of the movie work? Why would buying a lot of OJ at a high price ruin the Duke Brothers?", "output": [{"answer": "I feel so old. People have been askinbg what happened at the end of this movie for what must be the last 15 years of my life. It never stops. Every year/month/fortnight, I see someone asking what happened, and someone explaining. Andf it will keep on happening, until I am 90yrs old, in a home, with nothing but the Internet and my bladder to keep me going. And there it will be: \"what happens at the end of Trading Places?\""}, {"provenance": [{"wikipedia_id": "242855", "title": "Futures contract", "section": "Section::::Abstract.", "start_paragraph_id": 1, "start_character": 14, "end_paragraph_id": 1, "end_character": 612, "bleu_score": 0.9232808519770748}]}], "meta": {"partial_evidence": [{"wikipedia_id": "520990", "title": "Trading Places", "section": "Section::::Plot.\n", "start_paragraph_id": 7, "end_paragraph_id": 7, "meta": {"evidence_span": ["On television, they learn that Clarence Beeks is transporting a secret USDA report on orange crop forecasts.", "On television, they learn that Clarence Beeks is transporting a secret USDA report on orange crop forecasts. Winthorpe and Valentine recall large payments made to Beeks by the Dukes and realize that the Dukes plan to obtain the report to corner the market on frozen orange juice.", "Winthorpe and Valentine recall large payments made to Beeks by the Dukes and realize that the Dukes plan to obtain the report to corner the market on frozen orange juice."]}}]}}`

However, KILT ELI5 dataset from huggingface datasets library only contain empty list of provenance.

`{'id': '1oy5tc', 'input': 'in football whats the point of wasting the first two plays with a rush - up the middle - not regular rush plays i get those', 'meta': {'left_context': '', 'mention': '', 'obj_surface': [], 'partial_evidence': [], 'right_context': '', 'sub_surface': [], 'subj_aliases': [], 'template_questions': []}, 'output': [{'answer': 'In most cases the O-Line is supposed to make a hole for the running back to go through. If you run too many plays to the outside/throws the defense will catch on.\n\nAlso, 2 5 yard plays gets you a new set of downs.', 'meta': {'score': 2}, 'provenance': []}, {'answer': "I you don't like those type of plays, watch CFL.  We only get 3 downs so you can't afford to waste one.  Lots more passing.", 'meta': {'score': 2}, 'provenance': []}]}
`

should i perform other procedure to obtain evidence documents?

Why did you close this issue? How did you end up finding the evidence documents? I'm running into a similar issue with other KILT tasks.
Error when exploring `arabic_speech_corpus`

Navigate to https://huggingface.co/datasets/viewer/?dataset=arabic_speech_corpus

Error:
```
ImportError: To be able to use this dataset, you need to install the following dependencies['soundfile'] using 'pip install soundfile' for instance'
Traceback:
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 233, in <module>
    configs = get_confs(option)
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 604, in wrapped_func
    return get_or_create_cached_value()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/streamlit/caching.py", line 588, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/home/sasha/nlp-viewer/run.py", line 145, in get_confs
    module_path = nlp.load.prepare_module(path, dataset=True
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/datasets/load.py", line 342, in prepare_module
    f"To be able to use this {module_type}, you need to install the following dependencies"
```

Thanks for reporting! We'll fix that as soon as possible

Actually soundfile is not a dependency of this dataset.
The error comes from a bug that was fixed in this commit: https://github.com/huggingface/datasets/pull/1767/commits/c304e63629f4453367de2fd42883a78768055532
Basically the library used to consider the `import soundfile` in the docstring as a dependency, while it's just here as a code example.

Updating the viewer to the latest version of `datasets` should fix this issue


Hi! The viewer at https://huggingface.co/datasets/arabic_speech_corpus works fine. Closing.
[Timit_asr] Make sure not only the first sample is used 

When playing around with timit I noticed that only the first sample is used for all indices. I corrected this typo so that the dataset is correctly loaded.

cc @lhoestq  @vrindaprabhu

Failing `run (push)` is unrelated -> merging

Thanks for fixing this, it was affecting my runs for https://github.com/huggingface/transformers/pull/10581/

I am seeing this very late! Sorry for the blunder everyone! :(
How to load a dataset with load_from disk and save it again after doing transformations without changing the original? 

I am using the latest datasets library.  In my work, I first use **load_from_disk** to load a data set that contains 3.8Gb information. Then during my training process, I update that dataset object and add new elements and save it in a different place.  

When I save the dataset with **save_to_disk**, the original dataset which is already in the disk also gets updated. I do not want to update it.  How to prevent from this?


Hi ! That looks like a bug, can you provide some code so that we can reproduce ?
It's not supposed to update the original dataset

Hi, I experimented with RAG. 

Actually, you can run the [use_own_knowldge_dataset.py](https://github.com/shamanez/transformers/blob/rag-end-to-end-retrieval/examples/research_projects/rag/use_own_knowledge_dataset.py#L80). In the 80 you can save the dataset object to the disk with save_to_disk. Then in order to compute the embeddings in this use **load_from_disk**. 

Then finally save it. You can see the original dataset object (CSV after splitting also will be changed)

One more thing- when I save the dataset object with **save_to_disk** it name the arrow file with cache.... rather than using dataset. arrow.  Can you add a variable that we can feed a name to save_to_disk function?

@lhoestq   I also found that cache in tmp directory gets updated after transformations. This is really problematic when using datasets interactively. Let's say we use the shards function to a dataset loaded with csv, atm when we do transformations to shards and combine them it updates the original csv cache. 

I plan to update the save_to_disk method in #2025 so I can make sure the new save_to_disk doesn't corrupt your cache files.
But from your last message it looks like save_to_disk isn't the root cause right ?

ok, one more thing. When we use save_to_disk there are two files other than .arrow. dataset_info.json and state.json. Sometimes most of the fields in the dataset_infor.json are null, especially when saving dataset objects. Anyways I think load_from_disk uses the arrow files mentioned in state.json right? 

> Anyways I think load_from_disk uses the arrow files mentioned in state.json right?

Yes exactly

Perfect.  For now, I am loading the dataset from CSV in my interactive process and will wait until you make the PR!
Readme.md is misleading about kinds of datasets?

Hi!

At the README.MD, you say: "efficient data pre-processing: simple, fast and reproducible data pre-processing for the above public datasets as well as your own local datasets in CSV/JSON/text. "

But here:
https://github.com/huggingface/datasets/blob/master/templates/new_dataset_script.py#L82-L117

You mention other kinds of datasets, with images and so on. I'm confused. 

Is it possible to use it to store, say, imagenet locally? 

Hi ! Yes it's possible to use image data. There are already a few of them available (MNIST, CIFAR..)
wmt datasets fail to load

~\.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
    758         # Extract manually downloaded files.
    759         manual_files = dl_manager.extract(manual_paths_dict)
--> 760         extraction_map = dict(downloaded_files, **manual_files)
    761 
    762         for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list

caching issue, seems to work again..
Optimize int precision

Optimize int precision to reduce dataset file size.

Close #1973, close #1825, close #861.

@lhoestq, are the tests OK? Some other cases I missed? Do you agree with this approach?

I just tested this and it works like a charm :) 

However tokenizing and then setting the format to "torch" to feed the tokens into a model doesn't seem to work anymore, since the pytorch tensors have the int32/int8 precisions instead of int64 that is required as model inputs.

For example:

```python
import torch
from datasets import Dataset
from transformers import BertModel, BertTokenizer

torch.set_grad_enabled(False)

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")

dataset = Dataset.from_dict({"text": ["hello there !"]})
dataset = dataset.map(tokenizer, input_columns="text", remove_columns=dataset.column_names)
dataset = dataset.with_format("torch")

print(dataset.features)
# {'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
#  'input_ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),  # this should be int32 though
#  'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None)}

model(**dataset[:1])
# RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.CharTensor instead (while checking arguments for embedding)

dataset = dataset.with_format("torch", dtype=torch.int64)

model(**dataset[:1])
# works as expected
```

Pinging @sgugger here to make sure we take the right decision here.

Do we want the "torch" format to always return int64 ? Or does it have to keep the precision defined by the `dataset.features` 
 and therefore we would need to specify "torch" with `dtype=torch.int64` ?

From a user perspective, I think it's fine if the "torch" format converts all ints types to `torch.int64` by default since it's what the model will need almost all the time. I don't see a case where you would want to keep the low precision at the top of my head, and one can always write a custom transform for an edge case.

Sounds good to me !
For consistency maybe we should make the float precision fixed as well (float32, I guess)

Yes, that would be the one used by default.

Do we have the same requirements for TensorFlow?

Yes I we should do the same for tensorflow as well since tf models would have the same issue

Thanks for adding this :)

@lhoestq I think this PR is ready... :)
The size of CoNLL-2003 is not consistant with the official release.

Thanks for the dataset sharing! But when I use conll-2003, I meet some questions.
The statistics of conll-2003 in this repo is  : 
\#train 14041  \#dev 3250 \#test 3453
While the official statistics is:
\#train 14987 \#dev 3466 \#test 3684
Wish for your reply~

Hi,

if you inspect the raw data, you can find there are 946 occurrences of `-DOCSTART- -X- -X- O` in the train split and `14041 + 946 = 14987`, which is exactly the number of sentences the authors report. `-DOCSTART-` is a special line that acts as a boundary between two different documents and is filtered out in our implementation.

@lhoestq What do you think about including these lines? ([Link](https://github.com/flairNLP/flair/issues/1097) to a similar issue in the flairNLP repo)

We should mention in the Conll2003 dataset card that these lines have been removed indeed.

If some users are interested in using these lines (maybe to recombine documents ?) then we can add a parameter to the conll2003 dataset to include them.

But IMO the default config should stay the current one (without the `-DOCSTART-` stuff), so that you can directly train NER models without additional preprocessing. Let me know what you think

@lhoestq Yes, I agree adding a small note should be sufficient.

Currently, NLTK's `ConllCorpusReader` ignores the `-DOCSTART-` lines so I think it's ok if we do the same. If there is an interest in the future to use these lines, then we can include them.

I added a mention of this in conll2003's dataset card:
https://github.com/huggingface/datasets/blob/fc9796920da88486c3b97690969aabf03d6b4088/datasets/conll2003/README.md#conll2003

Edit: just saw your PR @mariosasko (noticed it too late ^^)
Let me take a look at it :)
Fix NestedDataStructure.data for empty dict

Fix #1981

I validated that this fixed the problem, thank you, @albertvillanova!


still facing the same issue or similar:
from datasets import load_dataset
wtm14_test = load_dataset('wmt14',"de-en",cache_dir='./datasets')

~\.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
    758         # Extract manually downloaded files.
    759         manual_files = dl_manager.extract(manual_paths_dict)
--> 760         extraction_map = dict(downloaded_files, **manual_files)
    761 
    762         for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list

Hi @sabania 
We released a patch version that fixes this issue (1.4.1), can you try with the new version please ?
```
pip install --upgrade datasets
```

I re-validated with the hotfix and the problem is no more.

It's working. thanks a lot.
wmt datasets fail to load

on master:
```
python -c 'from datasets import load_dataset; load_dataset("wmt14", "de-en")'
Downloading and preparing dataset wmt14/de-en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/stas/.cache/huggingface/datasets/wmt14/de-en/1.0.0/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 578, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 634, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/stas/.cache/huggingface/modules/datasets_modules/datasets/wmt14/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e/wmt_utils.py", line 760, in _split_generators
    extraction_map = dict(downloaded_files, **manual_files)
```

it worked fine recently. same problem if I try wmt16.

git bisect points to this commit from Feb 25 as the culprit https://github.com/huggingface/datasets/commit/792f1d9bb1c5361908f73e2ef7f0181b2be409fa

@albertvillanova 

@stas00 Mea culpa... May I fix this tomorrow morning?

yes, of course, I reverted to the version before that and it works ;)

but since a new release was just made you will probably need to make a hotfix.

and add the wmt to the tests?

Sure, I will implement a regression test!

@stas00 it is fixed. @lhoestq are you releasing the hot fix or would you prefer me to do it?

I'll do a patch release for this issue early tomorrow.

And yes we absolutly need tests for the wmt datasets: The missing tests for wmt are an artifact from the early development of the lib but now we have tools to generate automatically the dummy data used for tests :)

still facing the same issue or similar:
from datasets import load_dataset
wtm14_test = load_dataset('wmt14',"de-en",cache_dir='./datasets')

~.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
758 # Extract manually downloaded files.
759 manual_files = dl_manager.extract(manual_paths_dict)
--> 760 extraction_map = dict(downloaded_files, **manual_files)
761
762 for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list
Loading all answers from drop

Hello all,

I propose this change to the DROP loading script so that all answers are loaded no matter their type. Currently, only "span" answers are loaded, which excludes a significant amount of answers from drop (i.e. "number" and "date").

I updated the script with the version I use for my work. However, I couldn't find a way to verify that all is working when integrated with the datasets repo, since the `load_dataset` method seems to always download the script from github and not local files.

Note that 9 items from the train set have no answers, as well as 1 from the validation set. The script I propose simply do not load them.

Let me know if there is anything else I can do,
Clément

Nice thanks for the change !
This looks all good to me

Before we merge can you just update the dataset_infos.json file of drop ? You can do it by running
```
datasets-cli test ./datasets/drop --all_configs --save_infos --ignore_verifications
```

Done!
Add article_id and process test set template for semeval 2020 task 11…

… dataset

- `article_id` is needed to create the submission file for the task at https://propaganda.qcri.org/semeval2020-task11/
- The `technique classification` task provides the span indices in a template for the test set that is necessary to complete the task. This PR implements processing of that template for the dataset.

Thanks !
Now to fix the CI the only thing left is to add a dummy `test-task-tc-template.out` file inside the `dummy_data.zip` at `./datasets/sem_eval_2020_task_11/dummy/1.1.0`
It must contain the labels template for each dummy article of the test set included in `dummy_data.zip`

After that we should be good to merge this one :)

@lhoestq Made the changes! The failure now seems to be unrelated to the changes. Any idea what's going on?

This is a bug on master that we're investigating. You can ignore it
Adding ro sts dataset

Adding [RO-STS](https://github.com/dumitrescustefan/RO-STS) dataset

@lhoestq thank you very much for the quick review and useful comments! 

I have tried to address them all, and a few comments that you left for ro_sts I have applied to the ro_sts_parallel as well (in read-me: fixed source_datasets, links to homepage, repository, leaderboard, thanks to me message, in ro_sts_parallel.py changed to camel case as well). In the ro_sts_parallel I have changed the order on the languages, also in the example, as you said order doesn't matter, but just to have them listed in the readme in the same order.

I have commented above on why we would like to keep them as separate datasets, hope it makes sense.

If there is anything else I should change please let me know.

Thanks again!

@lhoestq I tried to adjust the ro_sts_parallel, locally when I run the tests they are passing, but somewhere it has the old name of rosts-parallel-ro-en which I am trying to change to ro_sts_parallel. I don't think I have left anything related to rosts-parallel-ro-en, but when the dataset_infos.json is regenerated it adds it. Could you please help me out, how can I fix this? Thanks in advance!

Great, thanks for all your help!