concatenate_datasets loads all the data into memory

bug

Something isn't working

Therefore, when I try to concatenate larger datasets (5x 35GB data sets) I also get an out of memory error, since over 90GB of swap space was used at the time of the crash:

```
---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-6-9766d77530b9> in <module>
     20         print(file_name)
     21         cv_batch = load_from_disk(file_name)
---> 22         cv_sampled_train = concatenate_datasets([cv_sampled_train, cv_batch])
     23
     24 print("Saving to disk!")

C:\ProgramData\Anaconda3\lib\site-packages\datasets\arrow_dataset.py in concatenate_datasets(dsets, info, split, axis)
   2891
   2892     # Concatenate tables
-> 2893     table = concat_tables([dset._data for dset in dsets if len(dset._data) > 0], axis=axis)
   2894     table = update_metadata_with_features(table, None)
   2895

C:\ProgramData\Anaconda3\lib\site-packages\datasets\table.py in concat_tables(tables, axis)
    837     if len(tables) == 1:
    838         return tables[0]
--> 839     return ConcatenationTable.from_tables(tables, axis=axis)
    840
    841

C:\ProgramData\Anaconda3\lib\site-packages\datasets\table.py in from_tables(cls, tables, axis)
    697             return result
    698
--> 699         blocks = to_blocks(tables[0])
    700         for table in tables[1:]:
    701             table_blocks = to_blocks(table)

C:\ProgramData\Anaconda3\lib\site-packages\datasets\table.py in to_blocks(table)
    669                 return [[InMemoryTable(table)]]
    670             elif isinstance(table, ConcatenationTable):
--> 671                 return copy.deepcopy(table.blocks)
    672             else:
    673                 return [[table]]

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_list(x, memo, deepcopy)
    203     append = y.append
    204     for a in x:
--> 205         append(deepcopy(a, memo))
    206     return y
    207 d[list] = _deepcopy_list

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_list(x, memo, deepcopy)
    203     append = y.append
    204     for a in x:
--> 205         append(deepcopy(a, memo))
    206     return y
    207 d[list] = _deepcopy_list

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    151             copier = getattr(x, "__deepcopy__", None)
    152             if copier is not None:
--> 153                 y = copier(memo)
    154             else:
    155                 reductor = dispatch_table.get(cls)

C:\ProgramData\Anaconda3\lib\site-packages\datasets\table.py in __deepcopy__(self, memo)
    143         # by adding it to the memo, self.table won't be copied
    144         memo[id(self.table)] = self.table
--> 145         return _deepcopy(self, memo)
    146
    147     def __getstate__(self):

C:\ProgramData\Anaconda3\lib\site-packages\datasets\table.py in _deepcopy(x, memo)
     62     memo[id(x)] = result
     63     for k, v in x.__dict__.items():
---> 64         setattr(result, k, copy.deepcopy(v, memo))
     65     return result
     66

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_list(x, memo, deepcopy)
    203     append = y.append
    204     for a in x:
--> 205         append(deepcopy(a, memo))
    206     return y
    207 d[list] = _deepcopy_list

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    170                     y = x
    171                 else:
--> 172                     y = _reconstruct(x, memo, *rv)
    173
    174     # If is its own copy, don't memoize.

C:\ProgramData\Anaconda3\lib\copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    262     if deep and args:
    263         args = (deepcopy(arg, memo) for arg in args)
--> 264     y = func(*args)
    265     if deep:
    266         memo[id(x)] = y

C:\ProgramData\Anaconda3\lib\copy.py in <genexpr>(.0)
    261     deep = memo is not None
    262     if deep and args:
--> 263         args = (deepcopy(arg, memo) for arg in args)
    264     y = func(*args)
    265     if deep:

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_list(x, memo, deepcopy)
    203     append = y.append
    204     for a in x:
--> 205         append(deepcopy(a, memo))
    206     return y
    207 d[list] = _deepcopy_list

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    170                     y = x
    171                 else:
--> 172                     y = _reconstruct(x, memo, *rv)
    173
    174     # If is its own copy, don't memoize.

C:\ProgramData\Anaconda3\lib\copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
    262     if deep and args:
    263         args = (deepcopy(arg, memo) for arg in args)
--> 264     y = func(*args)
    265     if deep:
    266         memo[id(x)] = y

C:\ProgramData\Anaconda3\lib\copy.py in <genexpr>(.0)
    261     deep = memo is not None
    262     if deep and args:
--> 263         args = (deepcopy(arg, memo) for arg in args)
    264     y = func(*args)
    265     if deep:

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_tuple(x, memo, deepcopy)
    208
    209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 210     y = [deepcopy(a, memo) for a in x]
    211     # We're not going to put the tuple in the memo, but it's still important we
    212     # check for it, in case the tuple contains recursive mutable structures.

C:\ProgramData\Anaconda3\lib\copy.py in <listcomp>(.0)
    208
    209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 210     y = [deepcopy(a, memo) for a in x]
    211     # We're not going to put the tuple in the memo, but it's still important we
    212     # check for it, in case the tuple contains recursive mutable structures.

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_list(x, memo, deepcopy)
    203     append = y.append
    204     for a in x:
--> 205         append(deepcopy(a, memo))
    206     return y
    207 d[list] = _deepcopy_list

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_tuple(x, memo, deepcopy)
    208
    209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 210     y = [deepcopy(a, memo) for a in x]
    211     # We're not going to put the tuple in the memo, but it's still important we
    212     # check for it, in case the tuple contains recursive mutable structures.

C:\ProgramData\Anaconda3\lib\copy.py in <listcomp>(.0)
    208
    209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy):
--> 210     y = [deepcopy(a, memo) for a in x]
    211     # We're not going to put the tuple in the memo, but it's still important we
    212     # check for it, in case the tuple contains recursive mutable structures.

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    144     copier = _deepcopy_dispatch.get(cls)
    145     if copier is not None:
--> 146         y = copier(x, memo)
    147     else:
    148         if issubclass(cls, type):

C:\ProgramData\Anaconda3\lib\copy.py in _deepcopy_list(x, memo, deepcopy)
    203     append = y.append
    204     for a in x:
--> 205         append(deepcopy(a, memo))
    206     return y
    207 d[list] = _deepcopy_list

C:\ProgramData\Anaconda3\lib\copy.py in deepcopy(x, memo, _nil)
    159                     reductor = getattr(x, "__reduce_ex__", None)
    160                     if reductor is not None:
--> 161                         rv = reductor(4)
    162                     else:
    163                         reductor = getattr(x, "__reduce__", None)

C:\ProgramData\Anaconda3\lib\site-packages\pyarrow\io.pxi in pyarrow.lib.Buffer.__reduce_ex__()

C:\ProgramData\Anaconda3\lib\site-packages\pyarrow\io.pxi in pyarrow.lib.Buffer.to_pybytes()

MemoryError:

```

Hi ! this looks like an important issue. Let me try to reproduce this.
Cc @samsontmr this might be related to the memory issue you have in #2134

@lhoestq Just went to open a similar issue.

It seems like deep copying (tested on master) the dataset object writes the table's record batches (`dset._data._batches`) into RAM.

To find the bug, I modified the `_deepcopy` function in `table.py` as follows:
```python
def _deepcopy(x, memo: dict):
    """deepcopy a regular class instance"""
    import psutil # pip install this package
    import time
    cls = x.__class__
    result = cls.__new__(cls)
    memo[id(x)] = result
    for k, v in x.__dict__.items():
        print("="* 50)
        print("Current memory:", psutil.virtual_memory().percent)
        print(f"Saving object {k} with value {v}")
        setattr(result, k, copy.deepcopy(v, memo))
        time.sleep(5)
        print("Memory after copy:", psutil.virtual_memory().percent)
    return result
```
Test script:
```python
import copy
from datasets import load_dataset
bk = load_dataset("bookcorpus", split="train")
bk_copy = copy.deepcopy(bk)
```

Thanks for the insights @mariosasko ! I'm working on a fix.
Since this is a big issue I'll make a patch release as soon as this is fixed

Hi @samsontmr @TaskManager91 the fix is on the master branch, feel free to install `datasets` from source and let us know if you still have issues

We just released `datasets` 1.6.2 that includes the fix :)

## Describe the bug
When I try to concatenate 2 datasets (10GB each) , the entire data is loaded into memory instead of being written directly to disk.

Interestingly, this happens when trying to save the new dataset to disk or concatenating it again.

![image](https://user-images.githubusercontent.com/7063207/116420321-2b21b480-a83e-11eb-9006-8f6ca729fb6f.png)

## Steps to reproduce the bug
```python
from datasets import concatenate_datasets, load_from_disk

test_sampled_pro = load_from_disk("test_sampled_pro")
val_sampled_pro = load_from_disk("val_sampled_pro")

big_set = concatenate_datasets([test_sampled_pro, val_sampled_pro])

# Loaded to memory
big_set.save_to_disk("big_set")

# Loaded to memory
big_set = concatenate_datasets([big_set, val_sampled_pro])
```

## Expected results
The data should be loaded into memory in batches and then saved directly to disk.

## Actual results
The entire data set is loaded into the memory and then saved to the hard disk.

## Versions
Paste the output of the following code:
```python
- Datasets: 1.6.1
- Python: 3.8.8 (default, Apr 13 2021, 19:58:26)
[GCC 7.3.0]
- Platform: Linux-5.4.72-microsoft-standard-WSL2-x86_64-with-glibc2.10
```


SNLI dataset has labels of -1

Hi @puzzler10,
Those examples where `gold_label` field was empty, -1 label was alloted to it. In order to remove it you can filter the samples from train/val/test splits. Here's how you can drop those rows from the dataset:
`dataset = load_dataset("snli")`
`dataset_test_filter = dataset['test'].filter(lambda example: example['label'] != -1)`

I agree it should have been mentioned in the documentation. I'll raise a PR regarding the same. Thanks for pointing out!

There are a number of rows with a label of -1 in the SNLI dataset. The dataset descriptions [here](https://nlp.stanford.edu/projects/snli/) and [here](https://github.com/huggingface/datasets/tree/master/datasets/snli) don't list  -1 as a label possibility, and neither does the dataset viewer. As examples, see index 107 or 124 of the test set.

It isn't clear what these labels mean. I found a [line of code](https://github.com/huggingface/datasets/blob/80e59ef178d3bb2090d091bc32315c655eb0633d/datasets/snli/snli.py#L94) that seems to put them in but it seems still unclear why they are there. The current workaround is to just drop the rows from any model being trained.

Perhaps the documentation should be updated.


Always update metadata in arrow schema

We store a redundant copy of the features in the metadata of the schema of the arrow table. This is used to recover the features when doing `Dataset.from_file`. These metadata are updated after each transfor, that changes the feature types.

For each function that transforms the feature types of the dataset, I added a step in the tests to make sure the metadata in the arrow schema are up to date.

I also added a line to update the metadata directly in the Dataset.__init__ method.
This way even a dataset instantiated with __init__ will have a table with the right metadata.

cc @mariosasko


Added CUAD metrics

`EM`, `F1`, `AUPR`, `Precision@80%Recall`, and `Precision@90%Recall` metrics supported for CUAD


Bug in Dataset.class_encode_column

Something isn't working

This has been fixed in this commit: https://github.com/huggingface/datasets/pull/2254/commits/88676c930216cd4cc31741b99827b477d2b46cb6

It was introduced in #2246 : using map with `input_columns` doesn't return the other columns anymore

## Describe the bug

All the rest of the columns except the one passed to `Dataset.class_encode_column` are discarded.

## Expected results

All the original columns should be kept.

This needs regression tests.


Synchronize table metadata with features

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

As pointed out in this [comment](https://github.com/huggingface/datasets/pull/2145#discussion_r621326767):
> Metadata stored in the schema is just a redundant information regarding the feature types.
It is used when calling Dataset.from_file to know which feature types to use.
These metadata are stored in the schema of the pyarrow table by using `update_metadata_with_features`.
However this something that's almost never tested properly.

**Describe the solution you'd like**

We should find a way to always make sure that the metadata (in `self.data.schema.metadata`) are synced with the actual feature types (in `self.info.features`).


Fix iterable interface expected by numpy

It's been fixed in this commit: https://github.com/huggingface/datasets/commit/549110e08238b3716a5904667095fb003acda54e

Basically #2246 broke querying an index with a simple iterable.
With the fix, it's again possible to use iterables and we can keep RandIter as it is.

Closing since the fix is already on master

Numpy expects the old iterable interface with `__getitem__` instead of `__iter__`.


Fix query table with iterable

The benchmark runs are failing on master because it tries to use an iterable to query the dataset.
However there's currently an issue caused by the use of `np.array` instead of `np.fromiter` on the iterable.
This PR fixes it


Don't use pyarrow 4.0.0 since it segfaults when casting a sliced ListArray of integers

@lhoestq note that the segfault also occurs on Linux.

Created the ticket at
https://issues.apache.org/jira/browse/ARROW-12568

@lhoestq the ticket you mentioned is now in state resolved. Pyarrow supports AArch64 after version 4.0.0. Because of this restriction `datasets` is not installing in AArch64 systems.

This test `tests/test_table.py::test_concatenation_table_cast` segfaults with the latest update of pyarrow 4.0.0.
Setting `pyarrow<4.0.0` for now. I'll open an issue on JIRA once I know more about the origin of the issue


DatasetDict save load Failing test in 1.6 not in 1.5

bug

Something isn't working

Thanks for reporting ! We're looking into it

I'm not able to reproduce this, do you think you can provide a code that creates a DatasetDict that has this issue when saving and reloading ?

Hi, I just ran into a similar error. Here is the minimal code to reproduce:
```python
from datasets import load_dataset, DatasetDict
ds = load_dataset('super_glue', 'multirc')

ds.save_to_disk('tempds')

ds = DatasetDict.load_from_disk('tempds')

```

```bash
Reusing dataset super_glue (/home/idahl/.cache/huggingface/datasets/super_glue/multirc/1.0.2/2fb163bca9085c1deb906aff20f00c242227ff704a4e8c9cfdfe820be3abfc83)
Traceback (most recent call last):
  File "/home/idahl/eval-util-expl/multirc/tmp.py", line 7, in <module>
    ds = DatasetDict.load_from_disk('tempds')
  File "/home/idahl/miniconda3/envs/eval-util-expl/lib/python3.9/site-packages/datasets/dataset_dict.py", line 710, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/home/idahl/miniconda3/envs/eval-util-expl/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 687, in load_from_disk
    return Dataset(
  File "/home/idahl/miniconda3/envs/eval-util-expl/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 274, in __init__
    raise ValueError(
ValueError: External features info don't match the dataset:
Got
{'answer': Value(dtype='string', id=None), 'idx': {'answer': Value(dtype='int32', id=None), 'paragraph': Value(dtype='int32', id=None), 'question': Value(dtype='int32', id=None)}, 'label': ClassLabel(num_classes=2, names=['False', 'True'], names_file=None, id=None), 'paragraph': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None)}
with type
struct<answer: string, idx: struct<answer: int32, paragraph: int32, question: int32>, label: int64, paragraph: string, question: string>

but expected something like
{'answer': Value(dtype='string', id=None), 'idx': {'paragraph': Value(dtype='int32', id=None), 'question': Value(dtype='int32', id=None), 'answer': Value(dtype='int32', id=None)}, 'label': Value(dtype='int64', id=None), 'paragraph': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None)}
with type
struct<answer: string, idx: struct<paragraph: int32, question: int32, answer: int32>, label: int64, paragraph: string, question: string>

```

The non-matching part seems to be
`'label': ClassLabel(num_classes=2, names=['False', 'True'], names_file=None, id=None),`
vs
`'label': Value(dtype='int64', id=None),`

And the order in the `<struct...` being different, which might cause the [features.type != inferred_features.type](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L274) condition to become true and raise this ValueError.

I am using datasets version 1.6.2.

Edit: can confirm, this works without error in version 1.5.0

My current workaround is to remove the idx feature:

```

from datasets import load_dataset, DatasetDict, Value
ds = load_dataset('super_glue', 'multirc')
ds = ds.remove_columns('idx')

ds.save_to_disk('tempds')

ds = DatasetDict.load_from_disk('tempds')

```

works.

It looks like this issue comes from the order of the fields in the 'idx' struct that is different for some reason.
I'm looking into it. Note that as a workaround you can also flatten the nested features with `ds = ds.flatten()`

I just pushed a fix on `master`. We'll do a new release soon !

Thanks for reporting

## Describe the bug

We have a test that saves a DatasetDict to disk and then loads it from disk. In 1.6 there is an incompatibility in the schema.

Downgrading to `>1.6` -- fixes the problem.

## Steps to reproduce the bug
```python

### Load a dataset dict from jsonl

path = '/test/foo'

ds_dict.save_to_disk(path)

ds_from_disk = DatasetDict.load_from_disk(path).  ## <-- this is where I see the error on 1.6
```

## Expected results

Upgrading to 1.6 shouldn't break that test. We should be able to serialize to and from disk.

## Actual results
```
        # Infer features if None
        inferred_features = Features.from_arrow_schema(arrow_table.schema)
        if self.info.features is None:
            self.info.features = inferred_features

        # Infer fingerprint if None

        if self._fingerprint is None:
            self._fingerprint = generate_fingerprint(self)

        # Sanity checks

        assert self.features is not None, "Features can't be None in a Dataset object"
        assert self._fingerprint is not None, "Fingerprint can't be None in a Dataset object"
        if self.info.features.type != inferred_features.type:
>           raise ValueError(
                "External features info don't match the dataset:\nGot\n{}\nwith type\n{}\n\nbut expected something like\n{}\nwith type\n{}".format(
                    self.info.features, self.info.features.type, inferred_features, inferred_features.type
                )
            )
E           ValueError: External features info don't match the dataset:
E           Got
E           {'_input_hash': Value(dtype='int64', id=None), '_task_hash': Value(dtype='int64', id=None), '_view_id': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'encoding__ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'encoding__offsets': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None), 'encoding__overflowing': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None), 'encoding__tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'encoding__words': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'ner_ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'ner_labels': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'relations': [{'child': Value(dtype='int64', id=None), 'child_span': {'end': Value(dtype='int64', id=None), 'label': Value(dtype='string', id=None), 'start': Value(dtype='int64', id=None), 'token_end': Value(dtype='int64', id=None), 'token_start': Value(dtype='int64', id=None)}, 'color': Value(dtype='string', id=None), 'head': Value(dtype='int64', id=None), 'head_span': {'end': Value(dtype='int64', id=None), 'label': Value(dtype='string', id=None), 'start': Value(dtype='int64', id=None), 'token_end': Value(dtype='int64', id=None), 'token_start': Value(dtype='int64', id=None)}, 'label': Value(dtype='string', id=None)}], 'spans': [{'end': Value(dtype='int64', id=None), 'label': Value(dtype='string', id=None), 'start': Value(dtype='int64', id=None), 'text': Value(dtype='string', id=None), 'token_end': Value(dtype='int64', id=None), 'token_start': Value(dtype='int64', id=None), 'type': Value(dtype='string', id=None)}], 'text': Value(dtype='string', id=None), 'tokens': [{'disabled': Value(dtype='bool', id=None), 'end': Value(dtype='int64', id=None), 'id': Value(dtype='int64', id=None), 'start': Value(dtype='int64', id=None), 'text': Value(dtype='string', id=None), 'ws': Value(dtype='bool', id=None)}]}
E           with type
E           struct<_input_hash: int64, _task_hash: int64, _view_id: string, answer: string, encoding__ids: list<item: int64>, encoding__offsets: list<item: list<item: int64>>, encoding__overflowing: list<item: null>, encoding__tokens: list<item: string>, encoding__words: list<item: int64>, ner_ids: list<item: int64>, ner_labels: list<item: string>, relations: list<item: struct<child: int64, child_span: struct<end: int64, label: string, start: int64, token_end: int64, token_start: int64>, color: string, head: int64, head_span: struct<end: int64, label: string, start: int64, token_end: int64, token_start: int64>, label: string>>, spans: list<item: struct<end: int64, label: string, start: int64, text: string, token_end: int64, token_start: int64, type: string>>, text: string, tokens: list<item: struct<disabled: bool, end: int64, id: int64, start: int64, text: string, ws: bool>>>
E
E           but expected something like
E           {'_input_hash': Value(dtype='int64', id=None), '_task_hash': Value(dtype='int64', id=None), '_view_id': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'encoding__ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'encoding__offsets': Sequence(feature=Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), length=-1, id=None), 'encoding__overflowing': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None), 'encoding__tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'encoding__words': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'ner_ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'ner_labels': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'relations': [{'head': Value(dtype='int64', id=None), 'child': Value(dtype='int64', id=None), 'head_span': {'start': Value(dtype='int64', id=None), 'end': Value(dtype='int64', id=None), 'token_start': Value(dtype='int64', id=None), 'token_end': Value(dtype='int64', id=None), 'label': Value(dtype='string', id=None)}, 'child_span': {'start': Value(dtype='int64', id=None), 'end': Value(dtype='int64', id=None), 'token_start': Value(dtype='int64', id=None), 'token_end': Value(dtype='int64', id=None), 'label': Value(dtype='string', id=None)}, 'color': Value(dtype='string', id=None), 'label': Value(dtype='string', id=None)}], 'spans': [{'text': Value(dtype='string', id=None), 'start': Value(dtype='int64', id=None), 'token_start': Value(dtype='int64', id=None), 'token_end': Value(dtype='int64', id=None), 'end': Value(dtype='int64', id=None), 'type': Value(dtype='string', id=None), 'label': Value(dtype='string', id=None)}], 'text': Value(dtype='string', id=None), 'tokens': [{'text': Value(dtype='string', id=None), 'start': Value(dtype='int64', id=None), 'end': Value(dtype='int64', id=None), 'id': Value(dtype='int64', id=None), 'ws': Value(dtype='bool', id=None), 'disabled': Value(dtype='bool', id=None)}]}
E           with type
E           struct<_input_hash: int64, _task_hash: int64, _view_id: string, answer: string, encoding__ids: list<item: int64>, encoding__offsets: list<item: list<item: int64>>, encoding__overflowing: list<item: null>, encoding__tokens: list<item: string>, encoding__words: list<item: int64>, ner_ids: list<item: int64>, ner_labels: list<item: string>, relations: list<item: struct<head: int64, child: int64, head_span: struct<start: int64, end: int64, token_start: int64, token_end: int64, label: string>, child_span: struct<start: int64, end: int64, token_start: int64, token_end: int64, label: string>, color: string, label: string>>, spans: list<item: struct<text: string, start: int64, token_start: int64, token_end: int64, end: int64, type: string, label: string>>, text: string, tokens: list<item: struct<text: string, start: int64, end: int64, id: int64, ws: bool, disabled: bool>>>

../../../../../.virtualenvs/tf_ner_rel_lib/lib/python3.8/site-packages/datasets/arrow_dataset.py:274: ValueError
```
## Versions
- Datasets: 1.6.1
- Python: 3.8.5 (default, Jan 26 2021, 10:01:04)
[Clang 12.0.0 (clang-1200.0.32.2)]
- Platform: macOS-10.15.7-x86_64-i386-64bit

```


Make tests run faster

LOL, I was also working on something similar 😅. I'm gonna have a look!!!

Sorry I didn't know you were also working on it ^^'
And yes I 100% agree with you on the points you mentioned. We should definitely improve the coverage. It would be nice to have a clearer separation to know which tests in the suite are unit tests and which ones are integration tests

Never mind: we both noticed tests can be improved. More PRs to come... 😉

According to the literature, unit tests are those that test a behavior unit, isolated from the other components and must be very fast: for me, this last requirement implies that they must be performed completely _in memory_.

As opposed, integration tests are those which also test interactions with _external_ components, like web services, databases, file system, etc.

The problem I see is that our code is still too coupled and it is difficult to isolate components for testing. Therefore, I would suggest acting iteratively, by refactoring to decouple components and then implement unit tests for each component in isolation.

From 7min to 2min to run pytest.
Ideally we should keep the whole CI run time below 10min.

In this PR I removed the remote tests that were never used.
I also replaced nested parametrized tests with unit tests.
This makes me think that we could still add more high level tests to check for a few combinations of parameters (but not all of them since there are too many of them).
Let me know what you think

Finally in another PR we can also separate in two circleci jobs:
- the tests of the code code of the lib
- the tests of the all the dataset/metric scripts.


Update black

Latest black version 21.4b0 requires to reformat most dataset scripts and also the core code of the lib.
This makes the CI currently fail on master


Fix memory issue in multiprocessing: Don't pickle table index

The code quality check is going to be fixed by #2265

The memory issue didn't come from `self.__dict__.copy()` but from the fact that this dict contains `_batches` which has all the batches of the table in it.
Therefore for a MemoryMappedTable all the data in `_batches` were copied in memory when pickling and this is the issue.

I'm still investigating why we didn't catch this issue in the tests.
This test should have caught it but didn't:

https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/tests/test_table.py#L350-L353

I'll focus on the patch release and fix the test in another PR after the release

The table index is currently being pickled when doing multiprocessing, which brings all the record batches of the dataset in memory.

I fixed that by not pickling the index attributes. Therefore each process has to rebuild the index when unpickling the table.

Fix issue #2256

We'll do a patch release asap !


test data added, dataset_infos updated

Fixes #2262. Thanks for pointing out issue with dataset @jinmang2!


NewsPH NLI dataset script fails to access test data.

dataset bug

A bug in a dataset script provided in the library

Thanks @bhavitvyamalik for the fix !
The fix will be available in the next release.
It's already available on the `master` branch. For now you can either install `datasets` from source or use `script_version="master"` in `load_dataset` to use the fixed version of this dataset.

In Newsph-NLI Dataset (#1192), it fails to access test data.

According to the script below, the download manager will download the train data when trying to download the test data.

https://github.com/huggingface/datasets/blob/2a2dd6316af2cc7fdf24e4779312e8ee0c7ed98b/datasets/newsph_nli/newsph_nli.py#L71

If you download it according to the script above, you can see that train and test receive the same data as shown below.
```python
>>> from datasets import load_dataset
>>> newsph_nli = load_dataset(path="./datasets/newsph_nli.py")
>>> newsph_nli
DatasetDict({
    train: Dataset({
        features: ['premise', 'hypothesis', 'label'],
        num_rows: 420000
    })
    test: Dataset({
        features: ['premise', 'hypothesis', 'label'],
        num_rows: 420000
    })
    validation: Dataset({
        features: ['premise', 'hypothesis', 'label'],
        num_rows: 90000
    })
})
>>> newsph_nli["train"][0]
{'hypothesis': 'Ito ang dineklara ni Atty. Romulo Macalintal, abogado ni Robredo, kaugnay ng pagsisimula ng preliminary conference ngayong hapon sa Presidential Electoral Tribunal (PET).',
 'label': 1,
 'premise': '"Hindi ko ugali ang mamulitika; mas gusto kong tahimik na magtrabaho. Pero sasabihin ko ito ngayon: ang tapang, lakas, at diskarte, hindi nadadaan sa mapanirang salita. Ang kailangan ng taumbayan ay tapang sa gawa," ayon kay Robredo sa inilabas nitong statement.'}
>>> newsph_nli["test"][0]
{'hypothesis': 'Ito ang dineklara ni Atty. Romulo Macalintal, abogado ni Robredo, kaugnay ng pagsisimula ng preliminary conference ngayong hapon sa Presidential Electoral Tribunal (PET).',
 'label': 1,
 'premise': '"Hindi ko ugali ang mamulitika; mas gusto kong tahimik na magtrabaho. Pero sasabihin ko ito ngayon: ang tapang, lakas, at diskarte, hindi nadadaan sa mapanirang salita. Ang kailangan ng taumbayan ay tapang sa gawa," ayon kay Robredo sa inilabas nitong statement.'}
```

In local, I modified the code of the source as below and got the correct result.
```python
71    test_path = os.path.join(download_path, "test.csv")
```
```python
>>> from datasets import load_dataset
>>> newsph_nli = load_dataset(path="./datasets/newsph_nli.py")
>>> newsph_nli
DatasetDict({
    train: Dataset({
        features: ['premise', 'hypothesis', 'label'],
        num_rows: 420000
    })
    test: Dataset({
        features: ['premise', 'hypothesis', 'label'],
        num_rows: 9000
    })
    validation: Dataset({
        features: ['premise', 'hypothesis', 'label'],
        num_rows: 90000
    })
})
>>> newsph_nli["train"][0]
{'hypothesis': 'Ito ang dineklara ni Atty. Romulo Macalintal, abogado ni Robredo, kaugnay ng pagsisimula ng preliminary conference ngayong hapon sa Presidential Electoral Tribunal (PET).',
 'label': 1,
 'premise': '"Hindi ko ugali ang mamulitika; mas gusto kong tahimik na magtrabaho. Pero sasabihin ko ito ngayon: ang tapang, lakas, at diskarte, hindi nadadaan sa mapanirang salita. Ang kailangan ng taumbayan ay tapang sa gawa," ayon kay Robredo sa inilabas nitong statement.'}
>>> newsph_nli["test"][0]
{'hypothesis': '-- JAI (@JaiPaller) September 13, 2019',
 'label': 1,
 'premise': 'Pinag-iingat ng Konsulado ng Pilipinas sa Dubai ang publiko, partikular ang mga donor, laban sa mga scam na gumagamit ng mga charitable organization.'}
```

I don't have experience with open source pull requests, so I suggest that you reflect them in the source.

Thank you for reading :)


Improve ReadInstruction logic and update docs

Improve ReadInstruction logic and docs.


GooAQ dataset added

Thanks for adding this one !
The download manager does support downloading files on git lfs via their github url. No need for a manual download option ;)

@lhoestq here the dataset is stored with Git LFS. Should I add option for manual downloading of dataset using `git lfs pull` post repo cloning or can we accommodate this in the current `download_and_extract`?


Add support for Split.ALL

Honestly, I think we should fix some other issues in Split API before this change. E. g. currently the following will not work, even though it should:
```python
import datasets
datasets.load_dataset("sst", split=datasets.Split.TRAIN+datasets.Split.TEST)  # AssertionError
```

EDIT:
Actually, think it's OK to merge this PR because the fix will not touch this PR's code.

The title says it all.


Fix incorrect update_metadata_with_features calls in ArrowDataset

@lhoestq Maybe a test that runs the functions that call `update_metadata_with_features` and checks if metadata was updated would be nice to prevent this from happening in the future.

Fixes bugs in the `unpdate_metadata_with_features` calls (caused by changes in #2151)


added metrics for CUAD

> For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here

@bhavitvyamalik  I guess the mentioned metrics are enough but it would be better if exact match is also added since the standard SQUAD dataset also has it.

I would like to quote it from the website that I am following to learn
these things.
Exact Match:
This metric is as simple as it sounds. For each question+answer pair, if
the characters of the model's prediction exactly match the characters of
*(one
of) the True Answer(s)*, EM = 1, otherwise EM = 0. This is a strict
all-or-nothing metric; being off by a single character results in a score
of 0. When assessing against a negative example, if the model predicts any
text at all, it automatically receives a 0 for that example.

So, I guess you need to ensure at least 1 predicted answer matches for EM
to be 1.
Source:
https://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html

You can go to their homepage and read the other links. They have detailed
explanations on evaluation metrics. You can also have a look at the
squad_v2 metric file for further clarification.

Regards,
Mohammed Rakib

On Sun, 25 Apr 2021 at 15:20, Bhavitvya Malik ***@***.***>
wrote:

> I'm a little confused when it comes to 2 ground truths which can be a
> possible answer. Like here for eg.
>
> predictions = [{'prediction_text': ['The seller:', 'The buyer/End-User:
> Shenzhen LOHAS Supply Chain Management Co., Ltd.'], 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> references = [{'answers': {'answer_start': [143, 49], 'text': ['The
> seller:', 'The buyer/End-User: Shenzhen LOHAS Supply Chain Management Co.,
> Ltd.']}, 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> Should I ensure at least 1 predicted answer matches or both predicted
> answers should match (like in this case) for EM to be 1?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/2257#issuecomment-826289753>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHMYZAZSAEZNFWEMVAPK6M3TKPNHLANCNFSM43QFZVPQ>
> .
>

Updated the same @MohammedRakib! Even if a single answer matches I'm returning 1 in that case for EM (not traversing all predictions once we have one `exact_match` from prediction)


Running `datase.map` with `num_proc > 1` uses a lot of memory

bug

Something isn't working

Thanks for reporting ! We are working on this and we'll do a patch release very soon.

We did a patch release to fix this issue.
It should be fixed in the new version 1.6.1

Thanks again for reporting and for the details :)

## Describe the bug
Running `datase.map` with `num_proc > 1`  leads to a tremendous memory usage that requires swapping on disk and it becomes very slow.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dstc8_datset = load_dataset("roskoN/dstc8-reddit-corpus", keep_in_memory=False)

def _prepare_sample(batch):
    return {"input_ids": list(), "attention_mask": list()}

for split_name, dataset_split in list(dstc8_datset.items()):
    print(f"Processing {split_name}")
    encoded_dataset_split = dataset_split.map(
        function=_prepare_sample,
        batched=True,
        num_proc=4,
        remove_columns=dataset_split.column_names,
        batch_size=10,
        writer_batch_size=10,
        keep_in_memory=False,
    )
    print(encoded_dataset_split)

    path = f"./data/encoded_{split_name}"

    encoded_dataset_split.save_to_disk(path)
```

## Expected results
Memory usage should stay within reasonable boundaries.

## Actual results
This is htop-output from running the provided script.

![image](https://user-images.githubusercontent.com/8143425/115954836-66954980-a4f3-11eb-8340-0153bdc3a475.png)

## Versions
```
- Datasets: 1.6.0
- Python: 3.8.8 (default, Apr 13 2021, 19:58:26)
[GCC 7.3.0]
- Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.10
```
Running on WSL2


Task casting for text classification & question answering

Looks really nice so far, thanks !
Maybe if a dataset doesn't have a template for a specific task we could try the default template of this task ?

hey @SBrandeis @lhoestq,

i now have a better idea about what you guys are trying to achieve with the task templates and have a few follow-up questions:

1. how did you envision using `DatasetInfo` for running evaluation? my understanding is that all `dataset_infos.json` files are stored in the `datasets` repo (unlike `transformers` where each model's weights etc are stored in a dedicated repo).
this suggests the following workflow:

```
- git clone datasets
- load target dataset to evaluate
- load `dataset_infos.json` for target dataset
- run eval for each task template in `task_templates`
- store metrics as evaluation cards (similar to what is done in `autonlp`)
```
2. assuming the above workflow, i see that the current `TaskTemplate` attributes of `task`, `input_schema`, and `label_schema` still require some wrangling from `dataset_infos.json` to reproduce additional mappings like `label2id` that we'd need for e.g. text classification. an alternative would be to instantiate the task template class directly from the JSON with something like
```python
from datasets.tasks import TextClassification
from transformers import AutoModelForSequenceClassification, AutoConfig

tc = TextClassification.from_json("path/to/dataset_infos.json")
# load a model with the desired config
model_ckpt = ...
config = AutoConfig.from_pretrained(model_ckpt, label2id=tc.label2id, id2label=tc.id2label)
model = AutoModelForSequenceClassification.from_pretrained(model_ckpt, config=config)
# run eval ...
```
perhaps this is what @SBrandeis had in mind with the `TaskTemplate.from_dict` method?

3. i personally prefer using `task_templates` over `supervised_keys` because it encourages the contributor to think in terms of 1 or more tasks. my question here is do we currently use `supervised_keys` for anything important in the `datasets` library?

1. How do you envision using DatasetInfo for running evaluation?

The initial idea was to be able to do something like this:
```python
from datasets import load_dataset
dset = load_dataset("name", task="binary_classification")
# OR
dset = load_dataset("name")
dset = dset.prepare_for_task("binary_classification")
```

2. I don't think that's needed if we proceed as mentioned above

3. `supervised_keys` are mostly a legacy compatibility thing with TF datasets, not sure it's used for anything right now. I'll let @lhoestq give more details on that

[Edit 1] Typo

> The initial idea was to be able to do something like this:
>
> ```python
> from datasets import load_dataset
> dset = load_dataset("name", task="binary_classification")
> # OR
> dset = load_dataset("name")
> dset = dset.prepare_for_task("binary_classification")
> ```

ah that's very elegant! just so i've completely understood, the result would be that the relevant column names of `dset` would be mapped to e.g. `text` and `label` and thus we'd have a uniform schema for the evaluation of all `binary_classification` tasks?

That's correct! Also, the features need to be appropriately casted
For a classification task for example, we would need to cast the datasets features to something like this:
```python
datasets.Features({
    "text": datasets.Value("string"),
    "label": datasets.ClassLabel(names=[...]),
})
```

3. We can ignore `supervised_keys` (it came from TFDS and we're not using it) and use `task_templates`

great, thanks a lot for your answers! now it's much clearer what i need to do next 😃

hey @lhoestq @SBrandeis,

i've made some small tweaks to @SBrandeis's code so that `Dataset.prepare_for_task` is called in `DatasetBuilder`. using the `emotion` dataset as a test case, the following now works:

 ```python
# DatasetDict with default columns
ds = load_dataset("./datasets/emotion/")
# DatasetDict({
#     train: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 16000
#     })
#     validation: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 2000
#     })
#     test: Dataset({
#         features: ['tweet', 'emotion'],
#         num_rows: 2000
#     })
# })

# DatasetDict with remapped columns
ds = load_dataset("./datasets/emotion/", task="text_classification")
DatasetDict({
#     train: Dataset({
#         features: ['text', 'label'],
#         num_rows: 16000
#     })
#     validation: Dataset({
#         features: ['text', 'label'],
#         num_rows: 2000
#     })
#     test: Dataset({
#         features: ['text', 'label'],
#         num_rows: 2000
#     })
# })

# Dataset with default columns
ds = load_dataset("./datasets/emotion/", split="train")
# Map/cast features
ds = ds.prepare_for_task("text_classification")
# Dataset({
#     features: ['text', 'label'],
#     num_rows: 16000
# })
```

i have a few follow-up questions / remarks:

1. i'm working under the assumption that contributors / users only provide a unique set of task types. in particular, the current implementation does not support something like:
```python
task_templates=[TextClassification(labels=class_names, text_column="tweet", label_column="emotion"), TextClassification(labels=class_names, text_column="some_other_column", label_column="some_other_column")]
```
since we use `TaskTemplate.task` and the filter for compatible templates in `Dataset.prepare_for_task`. should we support these scenarios? my hunch is that this is rare in practice, but please correct me if i'm wrong.

2. when we eventually run evaluation for `transformers` models, i expect we'll be using the `Trainer` for which we can pass the standard label names to `TrainingArguments.label_names`. if that's the case, it might be prudent to heed the warning from the [docs](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#trainer) and use `labels` instead of `label` in the schema:
> your model can accept multiple label arguments (use the label_names in your TrainingArguments to indicate their name to the Trainer) but none of them should be named "label".

3. i plan to forge ahead on the rest of the pipeline taxonomy. please let me know if you'd prefer smaller, self-contained pull requests (e.g. one per task)

hey @lhoestq @SBrandeis, i think this is ready for another review 😃

in addition to a few comments / questions i've left in the pr, here's a few remarks:

1. after some experimentation, i decided against allowing the user to specify nested column names for question-answering. i couldn't find a simple solution with the current api and suspect that i'd have to touch many areas of `datasets` to "unflatten" columns in a generic fashion.
2. in the current implementation, the user can specify the outer column name for question-answering, but is expected to follow the inner schema for e.g. `answers.text` and `answers.answer_start`. we can decide later how much flexibility we want to give users
3. i added a few unit tests
4. as discussed, let's keep this pr focused on text classification / question answering and i'll add the other tasks in separate prs
5. i renamed the tasks e.g. `text_classification` -> `text-classification` for consistency with the `Trainer` model cards [here](https://github.com/huggingface/transformers/pull/11599#pullrequestreview-656371007).

> i'm not sure why the benchmarks are getting cancelled - is this expected?

Hmm I don't know. It's certainly unrelated to this PR though. Maybe github has some issues

Something is happening with actions: https://www.githubstatus.com/

hey @lhoestq and @SBrandeis, i've:

* extended the `prepare_for_task` API along the lines that @lhoestq suggested. i wasn't entirely sure what the `datasets` convention is for docstrings with mixed types, so please see if my proposal makes sense
* added a few new tests to check that we trigger the value errors on incorrect input

i think this is ready for another review :)

> Looks all good thank you :)
>
> Can you also add `prepare_for_task` in the `main_classes.rst` file of the documentation ?

Done! I also remembered that I needed to do the same for `DatasetDict`, so included this as well :)

This PR implements task preparation for a given task, in the continuation of #2143

Task taxonomy follows 🤗 Transformers's pipelines taxonomy: https://github.com/huggingface/transformers/tree/master/src/transformers/pipelines

Edit by @lewtun:

This PR implements support for the following tasks:

* `text-classification`
* `question-answering`

The intended usage is as follows:

```python
# Load a dataset with default column names / features
ds = load_dataset("dataset_name")
# Cast column names / features to schema. Casting is defined in the dataset's `DatasetInfo`
ds = ds.prepare_for_task(task="text-classification")
# Casting can also be realised during load
ds = load_dataset("dataset_name", task="text-classification")
# We can also combine shared tasks across dataset concatenation
ds1 = load_dataset("dataset_name_1", task="text-classification")
ds2 = load_dataset("dataset_name_2", task="text-classification")
# If the tasks have the same schema, so will `ds_concat`
ds_concat = concatenate_datasets([ds1, ds2])
```

Note that the current implementation assumes that `DatasetInfo.task_templates` has been pre-defined by the user / contributor when overriding the `MyDataset(GeneratorBasedBuilder)._info` function.

As pointed out by @SBrandeis, for evaluation we'll need a way to detect which datasets are already have a compatible schema so we don't have to edit hundreds of dataset scripts. One possibility is to check if the schema features are a subset of the dataset ones, e.g.

```python
squad = load_dataset("./datasets/squad", split="train")
qa = QuestionAnswering()
schema = Features({**qa.input_schema, **qa.label_schema})
assert all(item in squad.features.items() for item in schema.items())
```


Update format, fingerprint and indices after add_item

I renamed the variable, added a test for dataset._indices and fixed an issue with class_encode_column

Added fingerprint and format update wrappers + update the indices by adding the index of the newly added item in the table.


Perform minor refactoring: use config

refactoring

Restructuring existing code without changing its external behavior

@lhoestq is there a problem in the master branch? I got a segmentation fault...
```
tests/test_table.py::test_concatenation_table_cast[in_memory] Fatal Python error: Segmentation fault
```

Oh wow. Let me re-run the CI just to make sure

Hmm interesting, the segfault is still there. I'm investigating this issue on my windows machine

Feel free to merge master into this branch to fix the CI :)

Perform minor refactoring related to `config`.


Slow dataloading with big datasets issue persists

Hi ! Sorry to hear that. This may come from another issue then.

First can we check if this latency comes from the dataset itself ?
You can try to load your dataset and benchmark the speed of querying random examples inside it ?
```python
import time
import numpy as np

from datasets import load_from_disk

dataset = load_from_disk(...) # or from load_dataset...

_start = time.time()
n = 100
for i in np.random.default_rng(42).integers(0, len(dataset), size=n):
    _ = dataset[i]
print(time.time() - _start)
```

If we see a significant speed difference between your two datasets then it would mean that there's an issue somewhere

Hi @lhoestq, here is the result. I additionally measured time to `load_from_disk`:
* 60GB
```
loading took:  22.618776321411133
ramdom indexing 100 times took: 0.10214924812316895
```

* 600GB
```
loading took:  1176.1764674186707
ramdom indexing 100 times took: 2.853600025177002
```

Hmm.. I double checked that it's version 1.6.0. The difference seems quite big, could it be related to the running environment?

I'm surprised by the speed change. Can you give more details about your dataset ?
The speed depends on the number of batches in the arrow tables and the distribution of the lengths of the batches.
You can access the batches by doing `dataset.data.to_batches()` (use only for debugging) (it doesn't bring data in memory).

Also can you explain what parameters you used if you used `map` calls ?
Also if you have some code that reproduces the issue I'd be happy to investigate it.

Also if you could give us more info about your env like your OS, version of pyarrow and if you're using an HDD or a SSD

Here are some details of my 600GB dataset. This is a dataset AFTER the `map` function and once I load this dataset, I do not use `map` anymore in the training. Regarding the distribution of the lengths, it is almost uniform (90% is 512 tokens, and 10% is randomly shorter than that -- typical setting for language modeling).
```
len(batches):
492763

batches[0]:
pyarrow.RecordBatch
attention_mask: list<item: uint8>
  child 0, item: uint8
input_ids: list<item: int16>
  child 0, item: int16
special_tokens_mask: list<item: uint8>
  child 0, item: uint8
token_type_ids: list<item: uint8>
  child 0, item: uint8
```

Here the some parameters to `map` function just in case it is relevant:
```
num_proc=1    # as multi processing is slower in my case
load_from_cache_file=False
```

Regarding the environment, I am running the code on a cloud server. Here are some info:
```
Ubuntu 18.04.5 LTS   # cat /etc/issue
pyarrow                 3.0.0  # pip list | grep pyarrow
```
The data is stored in SSD and it is mounted to the machine via Network File System.

If you could point me to some of the commands to check the details of the environment, I would be happy to provide relevant information @lhoestq !

I am not sure how I could provide you with the reproducible code, since the problem only arises when the data is big. For the moment, I would share the part that I think is relevant. Feel free to ask me for more info.

```python
class MyModel(pytorch_lightning.LightningModule)
    def setup(self, stage):
        self.dataset = datasets.load_from_disk(path)
        self.dataset.set_format("torch")

    def train_dataloader(self):
        collate_fn = transformers.DataCollatorForLanguageModeling(
                tokenizer=transformers.ElectraTokenizerFast.from_pretrained(tok_path)
        )
        dataloader = torch.utils.DataLoader(
                self.dataset,
                batch_size=32,
                collate_fn=collate_fn,
                num_workers=8,
                pin_memory=True,
       )
```

Hi ! Sorry for the delay I haven't had a chance to take a look at this yet. Are you still experiencing this issue ?
I'm asking because the latest patch release 1.6.2 fixed a few memory issues that could have lead to slow downs

Hi! I just ran the same code with different datasets (one is 60 GB and another 600 GB), and the latter runs much slower. ETA differs by 10x.

@lhoestq and @hwijeen

Despite upgrading to datasets 1.6.2, still experiencing extremely slow (2h00) loading for a 300Gb local dataset shard size 1.1Gb on local HDD (40Mb/s read speed). This corresponds almost exactly to total data divided by reading speed implying that it reads the entire dataset at each load.

Stack details:
=========

> GCC version: Could not collect
> Clang version: Could not collect
> CMake version: Could not collect
>
> Python version: 3.7 (64-bit runtime)
> Is CUDA available: True
> CUDA runtime version: 10.2.89
> GPU models and configuration: GPU 0: GeForce GTX 1050
> Nvidia driver version: 457.63
> cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin\cudnn64_7.dll
> HIP runtime version: N/A
> MIOpen runtime version: N/A
>
> Versions of relevant libraries:
> [pip3] datasets==1.6.2
> [pip3] transformers==4.5.1
> [pip3] numpy==1.19.1
> [pip3] numpydoc==1.1.0
> [pip3] pytorch-metric-learning==0.9.98
> [pip3] torch==1.8.1
> [pip3] torchaudio==0.8.1
> [pip3] torchvision==0.2.2
> [conda] blas                      2.16                        mkl    conda-forge
> [conda] cudatoolkit               10.2.89              hb195166_8    conda-forge
> [conda] libblas                   3.8.0                    16_mkl    conda-forge
> [conda] libcblas                  3.8.0                    16_mkl    conda-forge
> [conda] liblapack                 3.8.0                    16_mkl    conda-forge
> [conda] liblapacke                3.8.0                    16_mkl    conda-forge
> [conda] mkl                       2020.1                      216
> [conda] numpy                     1.19.1           py37hae9e721_0    conda-forge
> [conda] numpydoc                  1.1.0                      py_1    conda-forge
> [conda] pytorch                   1.8.1           py3.7_cuda10.2_cudnn7_0    pytorch
> [conda] pytorch-metric-learning   0.9.98             pyh39e3cac_0    metric-learning
> [conda] torchaudio                0.8.1                      py37    pytorch
> [conda] torchvision               0.2.2                      py_3    pytorch

Hi @BenoitDalFerro how do your load your dataset ?

Hi @lhoestq thanks for the quick turn-around, actually the plain vanilla way, without an particular knack or fashion, I tried to look into the documentation for some alternative but couldn't find any

> dataset = load_from_disk(dataset_path=os.path.join(datasets_dir,dataset_dir))

I’m facing the same issue when loading a 900GB dataset (stored via `save_to_disk`): `load_from_disk(path_to_dir)` takes 1.5 hours and htop consistently shows high IO rates > 120 M/s.

@tsproisl same here, smells like ~~teen spirit~~ intended generator inadvertently ending up iterator

@lhoestq perhaps solution to detect bug location in code is to track its signature via HD read usage monitoring, option is to add tracking decorator on top each function and sequentially close all hatches from top to bottom, suggest PySmart https://pypi.org/project/pySMART/ a Smartmontools implementation

I wasn't able to reproduce this on a toy dataset of around 300GB:

```python
import datasets as ds

s = ds.load_dataset("squad", split="train")
s4000 = ds.concatenate_datasets([s] * 4000)
print(ds.utils.size_str(s4000.data.nbytes))  # '295.48 GiB'

s4000.save_to_disk("tmp/squad_4000")
```

```python
import psutil
import time
from datasets import load_from_disk

disk = "disk0"  # You may have to change your disk here
iocnt1 = psutil.disk_io_counters(perdisk=True)[disk]
time1 = time.time()

s4000_reloaded = load_from_disk("tmp/squad_4000")

time2 = time.time()
iocnt2 = psutil.disk_io_counters(perdisk=True)[disk]

print(f"Blocks read {iocnt2.read_count - iocnt1.read_count}")  # Blocks read 18
print(f"Elapsed time: {time2 - time1:.02f}s")  # Elapsed time: 14.60s
```

Could you run this on your side and tell me if how much time it takes ? Please run this when your machine is idle so that other processes don't interfere.

I got these results on my macbook pro on datasets 1.6.2

@lhoestq thanks, test running as we speak, bear with me

Just tried on google colab and got ~1min for a 15GB dataset (only 200 times SQuAD), while it should be instantaneous. The time is spent reading the Apache Arrow table from the memory mapped file. This might come a virtual disk management issue. I'm trying to see if I can still speed it up on colab.

@lhoestq what is Google Colab's HD read speed, is it possible to introspect incl. make like SSD or HDD ?

@lhoestq Thank you! The issue is getting more interesting. The second script is still running, but it's definitely taking much longer than 15 seconds.

Okay, here’s the ouput:
Blocks read 158396
Elapsed time: 529.10s

Also using datasets 1.6.2. Do you have any ideas, how to pinpoint the problem?

@lhoestq, @tsproisl mmmh still writing on my side about 1h to go, thinking on it are your large datasets all monoblock unsharded ? mine is 335 times 1.18Gb shards.

The 529.10s was a bit too optimistic. I cancelled the reading process once before running it completely, therefore the harddrive cache probably did its work.

Here are three consecutive runs
First run (freshly written to disk):
Blocks read 309702
Elapsed time: 1267.74s
Second run (immediately after):
Blocks read 113944
Elapsed time: 417.55s
Third run (immediately after):
Blocks read 42518
Elapsed time: 199.19s

@lhoestq
First test
> elapsed time: 11219.05s

Second test running bear with me, for Windows users slight trick to modify original "disk0" string:

First find physical unit relevant key in dictionnary
```
import psutil
psutil.disk_io_counters(perdisk=True)
```

> {'PhysicalDrive0': sdiskio(read_count=18453286, write_count=4075333, read_bytes=479546467840, write_bytes=161590275072, read_time=20659, write_time=2464),
>  'PhysicalDrive1': sdiskio(read_count=1495778, write_count=388781, read_bytes=548628622336, write_bytes=318234849280, read_time=426066, write_time=19085)}

In my case it's _PhysicalDrive1_

Then insert relevant key's string as _disk_ variable

```
psutil.disk_io_counters()
disk = 'PhysicalDrive1'  # You may have to change your disk here
iocnt1 = psutil.disk_io_counters(perdisk=True)[disk]
time1 = time.time()
s4000_reloaded = load_from_disk("your path here")
time2 = time.time()
iocnt2 = psutil.disk_io_counters(perdisk=True)[disk]
print(f"Blocks read {iocnt2.read_count - iocnt1.read_count}")  # Blocks read 18
print(f"Elapsed time: {time2 - time1:.02f}s")  # Elapsed time: 14.60s
```

@lhoestq
Second test

> Blocks read 1265609
> Elapsed time: 11216.55s

Unfortunately no. Thanks for running the benchmark though, it shows that you machine does a lot of read operations. This is not expected: in other machines it does almost no read operations which enables a very fast loading.

I did some tests on google colab and have the same issue. The first time the dataset arrow file is memory mapped takes always a lot of time (time seems linear with respect to the dataset size). Reloading the dataset is then instantaneous since the arrow file has already been memory mapped.

I also tried using the Arrow IPC file format (see #1933) instead of the current streaming format that we use but it didn't help.

Memory mapping is handled by the OS and depends on the disk you're using, so I'm not sure we can do much about it. I'll continue to investigate anyway, because I still don't know why in some cases it would go through the entire file (high `Blocks read ` as in your tests) and in other cases it would do almost no reading.

@lhoestq thanks for the effort, let's stay in touch

Just want to say that I am seeing the same issue. Dataset size if 268GB and it takes **3 hours** to load `load_from_disk`, using dataset version `1.9.0`. Filesystem underneath is `Lustre`

Hi @lhoestq, confirmed Windows issue, exact same code running on Linux OS total loading time about 3 minutes.

Hmm that's different from what I got. I was on Ubuntu when reporting the initial issue.

Hi,

I reported too slow data fetching when data is large(#2210) a couple of weeks ago, and @lhoestq referred me to the fix (#2122).
However, the problem seems to persist. Here is the profiled results:

1) Running with 60GB
```
Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  517.96         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
model_backward                     	|  0.26144        	|100            	|  26.144         	|  5.0475         	|
model_forward                      	|  0.11123        	|100            	|  11.123         	|  2.1474         	|
get_train_batch                    	|  0.097121       	|100            	|  9.7121         	|  1.8751         	|
```

3) Running with 600GB, datasets==1.6.0
```
Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  4563.2         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
get_train_batch                    	|  5.1279         	|100            	|  512.79         	|  11.237         	|
model_backward                     	|  4.8394         	|100            	|  483.94         	|  10.605         	|
model_forward                      	|  0.12162        	|100            	|  12.162         	|  0.26652        	|
```

I see that `get_train_batch` lags when data is large. Could this be related to different issues?
I would be happy to provide necessary information to investigate.


while running run_qa.py, ran into a value error

command:

python3 run_qa.py   --model_name_or_path hyunwoongko/kobart    --dataset_name squad_kor_v2   --do_train   --do_eval   --per_device_train_batch_size 8   --learning_rate 3e-5   --num_train_epochs 3   --max_seq_length 512   --doc_stride 128   --output_dir /tmp/debug_squad/

error:

ValueError: External features info don't match the dataset:
Got
{'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answer': {'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None), 'html_answer_start': Value(dtype='int32', id=None)}, 'url': Value(dtype='string', id=None), 'raw_html': Value(dtype='string', id=None)}
with type
struct<answer: struct<text: string, answer_start: int32, html_answer_start: int32>, context: string, id: string, question: string, raw_html: string, title: string, url: string>

but expected something like
{'answer': {'answer_start': Value(dtype='int32', id=None), 'html_answer_start': Value(dtype='int32', id=None), 'text': Value(dtype='string', id=None)}, 'context': Value(dtype='string', id=None), 'id': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'raw_html': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'url': Value(dtype='string', id=None)}
with type
struct<answer: struct<answer_start: int32, html_answer_start: int32, text: string>, context: string, id: string, question: string, raw_html: string, title: string, url: string>

I didn't encounter this error 4 hours ago. any solutions for this kind of issue?
looks like gained dataset format refers to 'Data Fields', while expected refers to 'Data Instances'.


some issue in loading local txt file as Dataset for run_mlm.py

Hi,

1. try
    ```python
    dataset = load_dataset("text", data_files={"train": ["a1.txt", "b1.txt"], "test": ["c1.txt"]})
    ```
    instead.

    Sadly, I can't reproduce the error on my machine. If the above code doesn't resolve the issue, try to update the library to the
    newest version (`pip install datasets --upgrade`).

2. https://github.com/huggingface/transformers/blob/3ed5e97ba04ce9b24b4a7161ea74572598a4c480/examples/pytorch/language-modeling/run_mlm.py#L258-L259
This is the original code. You'll have to modify the example source to work with multiple train files. To make it easier, let's say "|" will act as a delimiter between files:
    ```python
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file.split("|")  # + .split("|")
    ```
    Then call the script as follows (**dataset_name must be None**):
    ```bash
    python run_mlm.py [... other args] --train_file a1.txt|b1.txt
    ```

i meet the same error with datasets 1.11.0, is there any insight about this?

![image](https://user-images.githubusercontent.com/14968123/115773877-18cef300-a3c6-11eb-8e58-a9cbfd1001ec.png)

first of all, I tried to load 3 .txt files as a dataset (sure that the directory and permission is OK.), I face with the below error.

> FileNotFoundError: [Errno 2] No such file or directory: 'c'

by removing one of the training .txt files It's fixed and although if I put all file as training it's ok
![image](https://user-images.githubusercontent.com/14968123/115774207-867b1f00-a3c6-11eb-953b-905cfb112d25.png)
![image](https://user-images.githubusercontent.com/14968123/115774264-9b57b280-a3c6-11eb-9f36-7b109f0e5a31.png)

after this, my question is how could I use this defined Dataset for run_mlm.py for from scratch pretraining.
by using  --train_file path_to_train_file just can use one .txt , .csv or, .json file. I tried to set my defined Dataset as --dataset_name but the below issue occurs.

> Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 336, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py", line 291, in cached_path
    use_auth_token=download_config.use_auth_token,
  File "/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py", line 621, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/dataset/dataset.py

> During handling of the above exception, another exception occurred:

> Traceback (most recent call last):
  File "run_mlm.py", line 486, in <module>
    main()
  File "run_mlm.py", line 242, in main
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir)
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 719, in load_dataset
    use_auth_token=use_auth_token,
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 347, in prepare_module
    combined_path, github_file_path
FileNotFoundError: Couldn't find file locally at dataset/dataset.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.6.0/datasets/dataset/dataset.py.
The file is also not present on the master branch on github.


enhancement

New feature or request

Next minor release

1.12

> If you pass a dictionary like this:
>
> ```
> {"main_metadata": url_to_main_data,
> "secondary_metadata": url_to_sec_data,
> "train": url_train_data,
> "test": url_test_data}
> ```
>
> then only the train or test keys will be kept, which I feel not intuitive.
>
> For example if the users asks to load the "train" split, then the main and secondary metadata won't be downloaded.
> You can fix that by keeping all the keys except the splits to ignore

Hi @lhoestq, I have been thinking about this and I think it is worth that we discuss about it.

When I created this PR, my first idea was to create a "hack" inside the download manager that will be able to filter some split(s) without touching any dataset script. Of course, the download manager does not know about splits logic, and thus this trick would only work for some very specific datasets: only the ones containing that pass a dict to the download manager containing only the keys "train", "validation", "test" (or the one passed by the user for advanced users knowing they can do it), e.g. the `natural_questions` dataset (which was one of the targets).

The big inconvenient of this approach is that it is not applicable to many datasets (or worse, it should be constantly tweaked to cope with exceptional cases). One exceptional case is the one you pointed out. But I see others:
- the split keys can be different: train, test, dev, val, validation, eval,...
- in `hope_edi` dataset, the split keys are: TRAIN_DOWNLOAD_URL, VALIDATION_DOWNLOAD_URL
- in `few_rel` dataset, the split keys are: train_wiki, val_nyt, val_pubmed,..., pid2name
- in `curiosity_dialogs`, the split keys are: train, val, test, test_zero; this means that for every split we pass, we will also get test_zero
- in `deal_or_no_dialog`, each of the splits URL is passed separately to the download manager, so all splits would be always downloaded
- etc.

Then after discussing, another idea emerged: pass a `split` parameter to `_split_generators`, which know about the splits logic, so that it can handle which splits are passed to the download manager. This approach is more accurate and can be tweaked so that it works with all the datasets we want. The only inconvenient is that then for every target dataset, we must modify its corresponding `_split_generators` script method.

My point is that I don't think it is a good idea to implement both approaches. They could even interfere with each other!

If you agree, I would implement ONLY the second one, which is simpler, more consistent and stable and will avoid future problems.

Hi @albertvillanova !
Yup I agree with you, implementing the 2nd approach seems to be the right solution

Allow downloading/processing/caching only specific splits without downloading/processing/caching the other splits.

This PR implements two steps to handle only specific splits:
- it allows processing/caching only specific splits into Arrow files
- for some simple cases, it allows downloading only specific splits (which is more intricate as it depends on the user-defined method `_split_generators`)

This PR makes several assumptions:
- `DownloadConfig` contains the configuration settings for downloading
- the parameter `split` passed to `load_dataset` is just a parameter for loading (from cache), not for downloading


Implement Dataset to JSON

New feature or request

Next minor release

1.7

Implement `Dataset.to_json`.


Implement Dataset from Parquet

New feature or request

Next minor release

1.11

Hi @albertvillanova , I'll implement the parquet builder as an ArrowBasedBuilder if you don't mind

closing in favor of #2537 that is already merged

Implement instantiation of Dataset from Parquet file.


Faster map w/ input_columns & faster slicing w/ Iterable keys

@lhoestq Just fixed the code style issues— I think it should be good to merge now :)

@lhoestq Fixes #2193

- `map` now uses `with_format` to only load needed columns in memory when `input_columns` is set
- Slicing datasets with Iterables of indices now uses a new `Table.fast_gather` method, implemented with `np.searchsorted`, to find the appropriate batch indices all at once. `pa.concat_tables` is no longer used for this; we just call `pa.Table.from_batches` with a list of all the batch slices.

Together these changes have sped up batched `map()` calls over subsets of columns quite considerably in my initial testing.


Add `key` type and duplicates verification with hashing

@lhoestq The tests for key type and duplicate keys have been added and verified successfully.
After generating with an intentionally faulty `mnist` script, when there is an incompatible key type, it shows:

```
Downloading and preparing dataset mnist/mnist (download: 11.06 MiB, generated: 60.62 MiB, post-processed: Unknown size, total: 71.67 MiB) to C:\Users\nikhil\.cache\huggingface\datasets\mnist\mnist\1.0.0\5064c25e57a1678f700d2dc798ffe8a6d519405cca7d33670fffda477857a994...
0 examples [00:00, ? examples/s]2021-04-26 02:50:03.703836: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll

FAILURE TO GENERATE DATASET: Invalid key type detected
Found Key [0, 0] of type <class 'list'>
Keys should be either str, int or bytes type
```

In the case of duplicate keys, it now gives:

```
Downloading and preparing dataset mnist/mnist (download: 11.06 MiB, generated: 60.62 MiB, post-processed: Unknown size, total: 71.67 MiB) to C:\Users\nikhil\.cache\huggingface\datasets\mnist\mnist\1.0.0\5064c25e57a1678f700d2dc798ffe8a6d519405cca7d33670fffda477857a994...
0 examples [00:00, ? examples/s]2021-04-26 02:53:13.498579: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "f:\datasets\datasets-1\src\datasets\load.py", line 746, in load_dataset
    builder_instance.download_and_prepare(
  File "f:\datasets\datasets-1\src\datasets\builder.py", line 587, in download_and_prepare
    self._download_and_prepare(
  File "f:\datasets\datasets-1\src\datasets\builder.py", line 665, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "f:\datasets\datasets-1\src\datasets\builder.py", line 1002, in _prepare_split
    writer.write(example, key)
  File "f:\datasets\datasets-1\src\datasets\arrow_writer.py", line 321, in write
    self.check_duplicates()
  File "f:\datasets\datasets-1\src\datasets\arrow_writer.py", line 331, in check_duplicates
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 234467
Keys should be unique and deterministic in nature
```
Please let me know if this is what we wanted to implement. Thanks!

This looks pretty cool !
We can make focus on the GeneratorBasedBuilder for now yes.

Do you think we could make the ArrowWriter not look for duplicates by default ?
This way we can just enable duplicate detections when instantiating the writer in the GeneratorBasedBuilder for now.

Thank you @lhoestq

> Do you think we could make the ArrowWriter not look for duplicates by default ?

We can definitely do that by including a `check_duplicates` argument while instantiating `ArrowWriter()`.

However, since only `GeneratorBasedBuilder` uses the `write()` function (which includes the detection code) and the others like `ArrowBasedBuilder` use `write_table()` which remains as it was (without duplicate detection). I don't think it would be necessary.

Nonetheless, doing this would require just some small changes. Please let me know your thoughts on this. Thanks!

I like the idea of having the duplicate detection optional for other uses of the ArrowWriter.
This class is the main tool to write python data in arrow format so I'd expect it to be flexible.
That's why I think by default it shouldn't require users to provide keys or do any duplicates detection.

An alternative would be to subclass the writer to include duplicates detection in another class.

Both options are fine for me, let me know what you think !

> This class is the main tool to write python data in arrow format so I'd expect it to be flexible.
> That's why I think by default it shouldn't require users to provide keys or do any duplicates detection.

Well, that makes sense as the writer can indeed be used for other purposes as well.

> We can definitely do that by including a `check_duplicates` argument while instantiating `ArrowWriter()`.

I think that this would be the simplest and the more efficient option for achieving this as subclassing the writer only for this would lead to unnecessary complexity and code duplication (in case of `writer()`).

I will be adding the changes soon. Thanks for the feedback @lhoestq!

@lhoestq I have pushed the final changes just now.
Now, the keys and duplicate checking will be necessary only when the `ArrowWriter` is initialized with `check_duplicates=True` specifically (in this case, for `GeneratorBasedBuilders`)

Let me know if this is what was required. Thanks!

@lhoestq Thanks for the feedback! I will be adding the tests for the same very soon.

However, I'm not sure as to what exactly is causing the `segmentation fault` in the failing CI tests. It seems to be something from `test_concatenation_table_cast` from `test_table.py`, but I'm not sure as to what exactly. Would be great if you could help. Thanks!

You can merge master into your branch to fix this issue.
Basically pyarrow 4.0.0 has a segfault issue (which has now been resolved on the master branch of pyarrow).
So until 4.0.1 comes out we changed to using `pyarrow<4.0.0` recently.

@lhoestq Thanks for the help with the CI failures. Apologies for the multiple merge commits. My local repo got messy while merging which led to this.
Will be pushing the commit for the tests soon!

Hey @lhoestq, I've just added the required tests for checking key duplicates and invalid key data types.
I think we have caught a nice little issue as 27 datasets are currently using non-unique keys (hence, the failing tests: All these datasets are giving `DuplicateKeysError` during testing).
These datasets were not detected earlier as there was no key checking when `num_examples < writer_batch_size` due to which they passed the dummy data generation test. This bug was fixed by adding the test to `writer.finalize()` method as well for  checking any leftover examples from batches.

I'd like to make changes to the faulty datasets' scripts. However, I was wondering if I should do that in this PR itself or open a new PR as this might get messy in the same PR. Let me know your thoughts on this. Thanks!

Hi ! Once https://github.com/huggingface/datasets/pull/2333 is merged, feel free to merge master into your branch to fix the CI :)

Thanks a lot for the help @lhoestq. Besides merging the new changes, I guess this PR is completed for now :)

I just merged the PR, feel free to merge `master` into your branch. It should fix most most of the CI issues. If there are some left we can fix them in this PR :)

@lhoestq Looks like the PR is completed now. Thanks for helping me out so much in this :)

Hey @lhoestq, I've added the test and corrected the Cl errors as well. Do let me know if this requires any change. Thanks!

Merging. I'll update the comment on the master branch (for some reason I can edit files on this branch)

@lhoestq Thank you for the help and feedback. Feels great to contribute!

Closes #2230
There is currently no verification for the data type and the uniqueness of the keys yielded by the `dataset_builder`.
This PR is currently a work in progress with the following goals:

-  [x] Adding `hash_salt` to `ArrowWriter` so that the keys belonging to different splits have different hash
-  [x] Add `key` arrtibute to `ArrowWriter.write()` for hashing
-  [x] Add a hashing class which takes an input key of certain type (`str`/`int`/anything convertible to string) and produces a 128-bit hash using `hashlib.md5`
-  [x] Creating a function giving a custom error message when non-unique keys are found
   **[This will take care of type-checking for keys]**
-  [x] Checking for duplicate keys in `writer.write()` for each batch

[**NOTE**: This PR is currently concerned with `GeneratorBasedBuilder` only, for simplification. A subsequent PR will be made in future for `ArrowBasedBuilder`]

@lhoestq Thank you for the feedback. It would be great to have your guidance on this!


Set specific cache directories per test function call

Next minor release

1.12

@lhoestq, I think this reaches some memory limit on Linux instances... (?)

It looks like the `comet` metric test fails because it tries to load a model in memory.
In the tests I think we have `patch_comet` that mocks the model download + inference. Not sure why it didn't work though.
I can take a look tomorrow (this afternoon is the pytorch ecosystem day)

@lhoestq thanks for the hint: I'm going to have a look at that mock... ;)

@lhoestq finally I did not find out why the mock is not used... If you can give me some other hint tomorrow...

Implement specific cache directories (datasets, metrics and modules) per test function call.

Currently, the cache directories are set within the temporary test directory, but they are shared across all test function calls.

This PR implements specific cache directories for each test function call, so that tests are atomic and there are no side effects.


Map is slow and processes batches one after another

bug

Something isn't working

Hi @villmow, thanks for reporting.

Could you please try with the Datasets version 1.6? We released it yesterday and it fixes some issues about the processing speed. You can see the fix implemented by @lhoestq here: #2122.

Once you update Datasets, please confirm if the problem persists.

Hi @albertvillanova, thanks for the reply. I just tried the new version and the problem still persists.

Do I need to rebuild the saved dataset (which I load from disk) with the 1.6.0 version of datasets? My script loads this dataset and creates new datasets from it. I tried it without rebuilding.

See this short video of what happens. It does not create all processes at the same time:

https://user-images.githubusercontent.com/2743060/115720139-0da3a500-a37d-11eb-833a-9bbacc70868d.mp4

There can be a bit of delay between the creations of the processes but this delay should be the same for both your `map` calls. We should look into this.
Also if you hav some code that reproduces this issue on google colab that'd be really useful !

Regarding the speed differences:
This looks like a similar issue as https://github.com/huggingface/datasets/issues/1992 who is experiencing the same speed differences between processes.
This is a known bug that we are investigating. As of now I've never managed to reproduce it on my machine so it's pretty hard for me to find where this issue comes from.

Upgrade to 1.6.1 solved my problem somehow. I did not change any of my code, but now it starts all processes around the same time.

Nice ! I'm glad this works now.
Closing for now, but feel free to re-open if you experience this issue again.

## Describe the bug

I have a somewhat unclear bug to me, where I can't figure out what the problem is. The code works as expected on a small subset of my dataset (2000 samples) on my local machine, but when I execute the same code with a larger dataset (1.4 million samples) this problem occurs. Thats why I can't give exact steps to reproduce, I'm sorry.

I process a large dataset in a two step process. I first call map on a dataset I load from disk and create a new dataset from it. This works like expected and `map` uses all workers I started it with. Then I process the dataset created by the first step, again with `map`, which is really slow and starting only one or two process at a time. Number of processes is the same for both steps.

pseudo code:
```python
ds = datasets.load_from_disk("path")
new_dataset = ds.map(work, batched=True, ...)  # fast uses all processes
final_dataset = new_dataset.map(work2, batched=True, ...)  # slow starts one process after another
```

## Expected results
Second stage should be as fast as the first stage.

## Versions
Paste the output of the following code:
- Datasets: 1.5.0
- Python: 3.8.8 (default, Feb 24 2021, 21:46:12)
- Platform: Linux-5.4.0-60-generic-x86_64-with-glibc2.10

Do you guys have any idea? Thanks a lot!


Link to datasets viwer on Quick Tour page returns "502 Bad Gateway"

Something isn't working

Link to datasets viwer (https://huggingface.co/datasets/viewer/)  on Quick Tour page  (https://huggingface.co/docs/datasets/quicktour.html) returns "502 Bad Gateway"

The same error with https://huggingface.co/datasets/viewer/?dataset=glue&config=mrpc


Add SLR32 to OpenSLR

> And yet another one ! Thanks a lot :)

I just hope you don’t get fed up with openslr PR 😊 there are still few other datasets created by google in openslr that is not in hf dataset yet

I would like to add SLR32 to OpenSLR. It contains four South African languages: Afrikaans, Sesotho, Setswana and isiXhosa


Clarify how to load wikihow

Explain clearer how to load the dataset in the manual download instructions.

En relation with #2239.


Error loading wikihow dataset

Something isn't working

Hi @odellus, thanks for reporting.

The `wikihow` dataset has 2 versions:
- `all`: Consisting of the concatenation of all paragraphs as the articles and the bold lines as the reference summaries.
- `sep`: Consisting of each paragraph and its summary.

Therefore, in order to load it, you have to specify which version you would like, for example:
```python
dataset = load_dataset('wikihow', 'all')
```

Please, tell me if this solves your problem.

Good call out. I did try that and that's when it told me to download the
dataset. Don't believe I have tried it with local files. Will try first
thing in the morning and get back to you.

On Mon, Apr 19, 2021, 11:17 PM Albert Villanova del Moral <
***@***.***> wrote:

> Hi @odellus <https://github.com/odellus>, thanks for reporting.
>
> The wikihow dataset has 2 versions:
>
>    - all: Consisting of the concatenation of all paragraphs as the
>    articles and the bold lines as the reference summaries.
>    - sep: Consisting of each paragraph and its summary.
>
> Therefore, in order to load it, you have to specify which version you
> would like, for example:
>
> dataset = load_dataset('wikihow', 'all')
>
> Please, tell me if this solves your problem.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2239#issuecomment-823004146>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABDYI3HVRTBI2QT3BOG262DTJUL57ANCNFSM43GV5BZQ>
> .
>

Hi @odellus, yes you are right.

Due to the server where the `wikihow` dataset is hosted, the dataset can't be downloaded automatically by `huggingface` and you have to download it manually as you did.

Nevertheless, you have to specify which dataset version you would like to load anyway:
```python
dataset = load_dataset('wikihow', 'all', data_dir='./wikihow')
```
or
```python
dataset = load_dataset('wikihow', 'sep', data_dir='./wikihow')
```
I find that the instructions given by `huggingface` are not clear enough: I am going to fix this.
Please tell me if this eventually works for you.

## Describe the bug

When attempting to load wikihow into a dataset with
```python
from datasets import load_dataset
dataset = load_dataset('wikihow', data_dir='./wikihow')
```
I get the message:
```
AttributeError: 'BuilderConfig' object has no attribute 'filename'
```
at the end of a [full stack trace](https://gist.github.com/odellus/602c3b2de52f541d353b1022f320ffc2).

## Steps to reproduce the bug

I have followed the instructions for creating a wikihow dataset. The [wikihow dataset site](https://huggingface.co/datasets/wikihow) says to use
```python
from datasets import load_dataset
dataset = load_dataset('wikihow')
```
to load the dataset. I do so and I get the message
```
AssertionError: The dataset wikihow with config all requires manual data.
 Please follow the manual download instructions:   You need to manually download two wikihow files. An overview of which files to download can be seen at https://github.com/mahnazkoupaee/WikiHow-Dataset.
  You need to download the following two files manually:
    1) https://ucsb.app.box.com/s/ap23l8gafpezf4tq3wapr6u8241zz358 and save the file under <path/to/folder>/wikihowAll.csv
    2) https://ucsb.app.box.com/s/7yq601ijl1lzvlfu4rjdbbxforzd2oag and save the file under <path/to/folder>/wikihowSep.csv

  The <path/to/folder> can e.g. be "~/manual_wikihow_data".

  Wikihow can then be loaded using the following command `datasets.load_dataset("wikihow", data_dir="<path/to/folder>")`.
  .
 Manual data can be loaded with `datasets.load_dataset(wikihow, data_dir='<path/to/manual/data>')
```

So I create a directory `./wikihow` and download `wikihowAll.csv` and `wikihowSep.csv` into the new directory.

Then I run
```python
from datasets import load_dataset
dataset = load_dataset('wikihow', data_dir='./wikihow')
```

that's when I get the [stack trace](https://gist.github.com/odellus/602c3b2de52f541d353b1022f320ffc2)

## Expected results
I expected it to load the downloaded files into a dataset.

## Actual results
```python
Using custom data configuration default-data_dir=.%2Fwikihow
Downloading and preparing dataset wikihow/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/azureuser/.cache/huggingface/datasets/wikihow/default-data_dir=.%2Fwikihow/0.0.0/58f42f8f0e4d459811a0f69aaab35870093830ccd58006769e7e1eb3e0e686c2...                                                    ---------------------------------------------------------------------------
AttributeError
Traceback (most recent call last)
<ipython-input-9-5e4d40142f30> in <module>
----> 1 dataset = load_dataset('wikihow',data_dir='./wikihow')
~/.local/lib/python3.6/site-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, script_version, use_auth_token, **config_kwargs)
745         try_from_hf_gcs=try_from_hf_gcs,
746         base_path=base_path,-->
747         use_auth_token=use_auth_token,
748     )
749
~/.local/lib/python3.6/site-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
577                     if not downloaded_from_gcs:
578                         self._download_and_prepare(                                                             -->
579                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
580                         )
581                     # Sync info
~/.local/lib/python3.6/site-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
632         split_dict = SplitDict(dataset_name=self.name)
633         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)                      -->
634         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
635
636         # Checksums verification
~/.cache/huggingface/modules/datasets_modules/datasets/wikihow/58f42f8f0e4d459811a0f69aaab35870093830ccd58006769e7e1eb3e0e686c2/wikihow.py in _split_generators(self, dl_manager)
132
133         path_to_manual_file = os.path.join(
--> 134             os.path.abspath(os.path.expanduser(dl_manager.manual_dir)), self.config.filename
135         )
136
AttributeError: 'BuilderConfig' object has no attribute 'filename'
```
## Versions
Paste the output of the following code:
```python
import datasets
import sys
import platform

print(f"""
- Datasets: {datasets.__version__}
- Python: {sys.version}
- Platform: {platform.platform()}
""")
```
```
- Datasets: 1.5.0
- Python: 3.6.9 (default, Jan 26 2021, 15:33:00)                                      [GCC 8.4.0]
- Platform: Linux-5.4.0-1046-azure-x86_64-with-Ubuntu-18.04-bionic
```


NLU evaluation data

New intent classification dataset from https://github.com/xliuhw/NLU-Evaluation-Data


Update Dataset.dataset_size after transformed with map

enhancement

New feature or request

@albertvillanova I would like to take this up. It would be great if you could point me as to how the dataset size is calculated in HF. Thanks!

After loading a dataset, if we transform it by using `.map` its `dataset_size` attirbute is not updated.


Request to add StrategyQA dataset

dataset request

Requesting to add a new dataset

## Request to add StrategyQA dataset
- **Name:** StrategyQA
- **Description:** open-domain QA [(project page)](https://allenai.org/data/strategyqa)
- **Paper:** [url](https://arxiv.org/pdf/2101.02235.pdf)
- **Data:** [here](https://allenai.org/data/strategyqa)
- **Motivation:** uniquely-formulated dataset that also includes a question-decomposition breakdown and associated Wikipedia annotations for each step. Good for multi-hop reasoning modeling.


Update README.md

Adding relevant citations (paper accepted at AAAI 2020 & EMNLP 2020) to the benchmark


Fix bash snippet formatting in ADD_NEW_DATASET.md

This PR indents the paragraphs around the bash snippets in ADD_NEW_DATASET.md to fix formatting.


Fix `xnli` dataset tuple key

Closes #2229
The `xnli` dataset yields a tuple key in case of `ar` which is inconsistant with the acceptable key types (str/int).
The key was thus ported to `str` keeping the original information intact.


Start filling GLUE dataset card

I replaced all the "we" and applied your suggestion

Merging this for now, we can continue improving this card in other PRs :)

The dataset card was pretty much empty.

I added the descriptions (mainly from TFDS since the script is the same), and I also added the tasks tags as well as examples for a subset of the tasks.

cc @sgugger


Fix map when removing columns on a formatted dataset

This should fix issue #2226

The `remove_columns` argument was ignored on formatted datasets