added spans field for the wikiann datasets

Hi @lhoestq
Thanks a lot for taking time checking it. I update "dataset_infos.json", I added description to the function of _generate_samples  in wikiann.py but I was not sure about the format to write in README. thanks.

Thanks !

For the fields description in the dataset card, something like this does the job:
```
- `tokens`: a `list` of `string` features.
- `langs`: a `list` of `string` features that correspond to the language of each token.
- `ner_tags`: a `list` of classification labels, with possible values including `O` (0), `B-PER` (1), `I-PER` (2), `B-ORG` (3), `I-ORG` (4), `B-LOC` (5), `I-LOC` (6).
- `spans`: a `list` of `string` features, that is the list of named entities in the input text formatted as ``<TAG>: <mention>``
```

Also for information, I think the trailer of rick and morty season 5 is out now :)

Hi @lhoestq
thank you! This is updated now, please feel free to let me know if I need to modify something :)  thanks

Hi @lhoestq
I tried to add spans to the wikiann datasets.
Thanks a lot for kindly having a look.
This addresses https://github.com/huggingface/datasets/issues/2130.
Best regards
Rabeeh


add banking77 dataset

Intent classification/detection dataset from banking category with 77 unique intents.


TypeError when using save_to_disk in a dataset loaded with ReadInstruction split

Hi !
I think this has been fixed recently on `master`.
Can you try again by installing `datasets` from `master` ?
```
pip install git+https://github.com/huggingface/datasets.git
```

Hi!

Using that version of the code solves the issue. Thanks!

Hi,

Loading a dataset with `load_dataset` using a split defined via `ReadInstruction` and then saving it to disk results in the following error: `TypeError: Object of type ReadInstruction is not JSON serializable`.

Here is the minimal reproducible example:

```python
from datasets import load_dataset
from datasets import ReadInstruction

data_1 = load_dataset(
    "wikiann",
    "en",
    split="validation",
)

data_1.save_to_disk("temporary_path_1")

print("Save with regular split works.")

data_2 = load_dataset(
    "wikiann",
    "en",
    split=ReadInstruction("validation", to=50, unit="%"),
)

data_2.save_to_disk("temporary_path_2")
```

and the corresponding output:

```
Reusing dataset wikiann (/xxxxx/.cache/huggingface/datasets/wikiann/en/1.1.0/0b11a6fb31eea02f38ca17610657bfba3206100685283014daceb8da291c3be9)
Save with regular split works.
Reusing dataset wikiann (/xxxxx/.cache/huggingface/datasets/wikiann/en/1.1.0/0b11a6fb31eea02f38ca17610657bfba3206100685283014daceb8da291c3be9)
Traceback (most recent call last):
  File "bug.py", line 20, in <module>
    data_2.save_to_disk("temporary_path_2")
  File "/xxxxx/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 645, in save_to_disk
    json.dump(state, state_file, indent=2, sort_keys=True)
  File "/usr/lib/python3.7/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib/python3.7/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ReadInstruction is not JSON serializable
```

Let me know if there is some misuse from my end.

Thanks in advance.


Add CER metric

Add Character Error Rate (CER) metric that is used in evaluation in ASR. I also have written unittests (hopefully thorough enough) but I'm not sure how to integrate them into the existed codebase.

```python
from cer import CER

cer = CER()

class TestCER(unittest.TestCase):
    def test_cer_case_senstive(self):
        refs = ['White House']
        preds = ['white house']
        # S = 2, D = 0, I = 0, N = 11, CER = 2 / 11
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.1818181818) < 1e-6)

    def test_cer_whitespace(self):
        refs = ['were wolf']
        preds = ['werewolf']
        # S = 0, D = 0, I = 1, N = 9, CER = 1 / 9
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.1111111) < 1e-6)

        refs = ['werewolf']
        preds = ['weae     wolf']
        # S = 1, D = 1, I = 0, N = 8, CER = 0.25
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.25) < 1e-6)

        # consecutive whitespaces case 1
        refs = ['were wolf']
        preds = ['were               wolf']
        # S = 0, D = 0, I = 0, N = 9, CER = 0
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.0) < 1e-6)

        # consecutive whitespaces case 2
        refs = ['were   wolf']
        preds = ['were               wolf']
        # S = 0, D = 0, I = 0, N = 9, CER = 0
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.0) < 1e-6)

    def test_cer_sub(self):
        refs = ['werewolf']
        preds = ['weaewolf']
        # S = 1, D = 0, I = 0, N = 8, CER = 0.125
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.125) < 1e-6)

    def test_cer_del(self):
        refs = ['werewolf']
        preds = ['wereawolf']
        # S = 0, D = 1, I = 0, N = 8, CER = 0.125
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.125) < 1e-6)

    def test_cer_insert(self):
        refs = ['werewolf']
        preds = ['wereolf']
        # S = 0, D = 0, I = 1, N = 8, CER = 0.125
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.125) < 1e-6)

    def test_cer_equal(self):
        refs = ['werewolf']
        char_error_rate = cer.compute(predictions=refs, references=refs)
        self.assertEqual(char_error_rate, 0.0)

    def test_cer_list_of_seqs(self):
        refs = ['werewolf', 'I am your father']
        char_error_rate = cer.compute(predictions=refs, references=refs)
        self.assertEqual(char_error_rate, 0.0)

        refs = ['werewolf', 'I am your father', 'doge']
        preds = ['werxwolf', 'I       am your father', 'doge']
        # S = 1, D = 0, I = 0, N = 28, CER = 1 / 28
        char_error_rate = cer.compute(predictions=preds, references=refs)
        self.assertTrue(abs(char_error_rate - 0.03571428) < 1e-6)

    def test_cer_unicode(self):
        ref = [u'我能吞下玻璃而不伤身体']
        pred = [u' 能吞虾玻璃而 不霜身体啦']
        # S = 3, D = 2, I = 0, N = 11
        # CER = 5 / 11
        char_error_rate = cer.compute(predictions=pred, references=ref)
        self.assertTrue(abs(char_error_rate - 0.4545454545) < 1e-6)

        ref = [u'我能吞', u'下玻璃而不伤身体']
        pred = [u'我    能 吞 下 玻 璃', u'而不伤身体']
        # S = 0, D = 5, I = 0, N = 11
        # CER = 5 / 11
        char_error_rate = cer.compute(predictions=pred, references=ref)
        self.assertTrue(abs(char_error_rate - 0.454545454545) < 1e-6)

        ref = [u'我能吞下玻璃而不伤身体']
        char_error_rate = cer.compute(predictions=ref, references=ref)
        self.assertFalse(char_error_rate, 0.0)

    def test_cer_empty(self):
        ref = ''
        pred = 'Hypothesis'
        with self.assertRaises(ValueError):
            char_error_rate = cer.compute(predictions=pred, references=ref)

if __name__ == '__main__':
    unittest.main()
```


Fix missing infos from concurrent dataset loading

This should fix issue #2131

When calling `load_dataset` at the same time from 2 workers, one of the worker could have missing split infos when reloading the dataset from the cache.


fix dialogue action slot name and value

fix #2128


en language data from MLQA dataset is missing

Hi ! Indeed only the languages of the `translate-train` data are included...
I can't find a link to download the english train set on https://github.com/facebookresearch/MLQA though, do you know where we can download it ?

Hi @lhoestq
thank you very much for coming back to me, now I see, you are right, in the link you sent I see split of  {split}-context-{context_language}-question-{question_language}.json with context_language=question_language=en, TFDS most probably has extracted english ones from these files as en language files, but translate-train/test do not have en indeed. thanks a lot for the great explanations

I close the ticket, since I do not see any en existing, they have trained on "SQuAD V1.1" instead. Thanks.

Hi
I need mlqa-translate-train.en dataset, but it is missing from the MLQA dataset. could you have a look please? @lhoestq  thank you for your help to fix this issue.


Saving large in-memory datasets with save_to_disk crashes because of pickling

bug

Something isn't working

Hi !
Indeed `save_to_disk` doesn't call pickle anymore. Though the `OverflowError` can still appear for in-memory datasets bigger than 4GB. This happens when doing this for example:
```python
import pyarrow as pa
import pickle

arr = pa.array([0] * ((4 * 8 << 30) // 64))
table = pa.Table.from_arrays([a], names=["foo"])
pickle.dumps(table)  # fails with an OverflowError
pickle.dumps(table, 4)  # works !
```
We'll do the change to use `protocol=4`.

Moreover I've also seen other users complain about this error
```
struct.error: 'I' format requires 0 <= number <= 4294967295
```

It looks like something related to the 4GB limit as well but I'm not able to reproduce on my side.
Do you think you can provide a script that reproduces the issue ?
How big is your dataset ? (number of bytes, number of rows)

Hi!
So I've managed to created a minimum working (well technically crashing) example for the multiprocessing case, I create a huge list of zeros, like in your example, and then I try to .map(None, num_proc=2) over it, which then crashes, here's the code:

```python
from datasets import  Dataset

if __name__ == '__main__':
    ton_of_zeroes = [0] * ((12 * 8 << 30) // 64)
    large_dataset = Dataset.from_dict({'col': ton_of_zeroes})
    print("Start")
    large_dataset.map(function=None, num_proc=2)
    print("Done - should not print")
```

The amount of zeros could probably be reduced, I haven't tried to minimize it to find the breaking point, I just increased it from your code (which by quick glance I assumed tried to allocate over 4 GiB)

Running this results in the following traceback:

```
Parameter 'indices'=[        0         1         2 ... 805306365 805306366 805306367] of the transform datasets.arrow_dataset.Dataset.select couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
Traceback (most recent call last):
  File "./crash_multiproc_pickle.py", line 7, in <module>
    large_dataset.map(function=None, num_proc=2)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in map
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 454, in dump
    StockPickler.dump(self, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 732, in save_bytes
    self._write_large_bytes(BINBYTES + pack("<I", n), obj)
struct.error: 'I' format requires 0 <= number <= 4294967295
```

My datasets usually have hundreds of thousands to low millions of rows, with each row containing a list of 10 strings and list of vectors of different length (the strings tokenized), which in the worst case have 10\*512\*8 = 40960 bytes (but usually it is much smaller, as the vectors tend to be shorter. I need these groups of text lines to create training data for the Inverse Cloze Task.

Anyway I don't think my particular dataset is relevant, as the tiny script I created also manages to crash.
But I think the issue is the same as the save_to_disk, from the traceback it seems that in multiprocessing, it tries to use dill to return the result of the map workers, which tries to pickle the data and can't do it, probably because it's again using the older pickle protocol. That's my guess anyway.

I just merged a fix #2150 that allows to pickle tables bigger than 4GiB
Feel free to try it on the `master` branch !

awesome! I started getting this error as well when I tried to tokenize with a longer sequence length

@prokopCerny does this fix work for you? I found that with the latest master, my container with 500GB RAM starts crashing when I try to map a large dataset using `num_proc`.

@lhoestq would it be possible to implement some logic to keep the individual cache files small (say below 100mb)? I find this helps with loading large datasets, but the "hack" I was using (increasing `num_proc` to a large number) doesn't work anymore with the latest master; my container crashes even with `num_proc=200` now

Closing since the original issue was fixed in #2150
Feel free to reopen if you are still experiencing it.
For the other problems, please open separate issues

Using Datasets 1.5.0 on Python 3.7.
Recently I've been working on medium to large size datasets (pretokenized raw text sizes from few gigabytes to low tens of gigabytes), and have found out that several preprocessing steps are massively faster when done in memory, and I have the ability to requisition a lot of RAM, so I decided to do these steps completely out of the datasets library.

 So my workflow is to do several .map() on datasets object, then for the operation which is faster in memory to extract the necessary columns from the dataset and then drop it whole, do the transformation in memory, and then create a fresh Dataset object using .from_dict() or other method.

When I then try to call save_to_disk(path) on the dataset, it crashes because of pickling, which appears to be because of using old pickle protocol which doesn't support large files (over 4 GiB).
```
Traceback (most recent call last):
  File "./tokenize_and_chunkify_in_memory.py", line 80, in <module>
    main()
  File "./tokenize_and_chunkify_in_memory.py", line 75, in main
    tokenize_and_chunkify(config)
  File "./tokenize_and_chunkify_in_memory.py", line 60, in tokenize_and_chunkify
    contexts_dataset.save_to_disk(chunked_path)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 457, in save_to_disk
    self = pickle.loads(pickle.dumps(self))
OverflowError: cannot serialize a bytes object larger than 4 GiB
```
From what I've seen this issue may be possibly fixed, as the line `self = pickle.loads(pickle.dumps(self))` does not appear to be present in the current state of the repository.

To save these datasets to disk, I've resorted to calling .map() over them with `function=None` and specifying the .arrow cache file, and then creating a new dataset using the .from_file() method, which I can then safely save to disk.

Additional issue when working with these large in-memory datasets is when using multiprocessing, is again to do with pickling. I've tried to speed up the mapping with function=None by specifying num_proc to the available cpu count, and I again get issues with transferring the dataset, with the following traceback. I am not sure if I should open a separate issue for that.
```
Traceback (most recent call last):
  File "./tokenize_and_chunkify_in_memory.py", line 94, in <module>
    main()
  File "./tokenize_and_chunkify_in_memory.py", line 89, in main
    tokenize_and_chunkify(config)
  File "./tokenize_and_chunkify_in_memory.py", line 67, in tokenize_and_chunkify
    contexts_dataset.map(function=None, cache_file_name=str(output_dir_path / "tmp.arrow"), writer_batch_size=50000, num_proc=config.threads)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in map
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 454, in dump
    StockPickler.dump(self, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 732, in save_bytes
    self._write_large_bytes(BINBYTES + pack("<I", n), obj)
struct.error: 'I' format requires 0 <= number <= 4294967295Traceback (most recent call last):
  File "./tokenize_and_chunkify_in_memory.py", line 94, in <module>
    main()
  File "./tokenize_and_chunkify_in_memory.py", line 89, in main
    tokenize_and_chunkify(config)
  File "./tokenize_and_chunkify_in_memory.py", line 67, in tokenize_and_chunkify
    contexts_dataset.map(function=None, cache_file_name=str(output_dir_path / "tmp.arrow"), writer_batch_size=50000, num_proc=config.threads)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in map
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1485, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 657, in get
    raise self._value
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/connection.py", line 209, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 454, in dump
    StockPickler.dump(self, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/cernypro/dev/envs/huggingface_gpu/lib/python3.7/site-packages/dill/_dill.py", line 941, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 789, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 843, in _batch_appends
    save(x)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 638, in save_reduce
    save(args)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 774, in save_tuple
    save(element)
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/mnt/appl/software/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/pickle.py", line 732, in save_bytes
    self._write_large_bytes(BINBYTES + pack("<I", n), obj)
struct.error: 'I' format requires 0 <= number <= 4294967295
```


bug in mlqa dataset

Hi @dorost1234.

In Python 3, strings are sequences of Unicode _code points_. Unicode is a specification that maps all characters (and emoji symbols) with its unique representation in terms of code points. That is what you see: Unicode code points (represented by a \u escaped sequence of 16-bit hex values).

Characters are usually represented (on screen and papers) with a graphical element called _glyph_. That is what you would like to see: glyphs. But Python does not care about glyphs: that is the job of the GUI or the terminal; glyphs are what you get with the `print` function (if your terminal is properly configured to display those glyphs).

You have more detailed information about Unicode in the Python documentation: https://docs.python.org/3/howto/unicode.html

thank you so much for the insightful comments.

Hi
Looking into MLQA dataset for langauge "ar":

```
 "question": [
    "\u0645\u062a\u0649 \u0628\u062f\u0627\u062a \u0627\u0644\u0645\u062c\u0644\u0629 \u0627\u0644\u0645\u062f\u0631\u0633\u064a\u0629 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645 \u0628\u0627\u0644\u0646\u0634\u0631?",
    "\u0643\u0645 \u0645\u0631\u0629 \u064a\u062a\u0645 \u0646\u0634\u0631\u0647\u0627 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
    "\u0645\u0627 \u0647\u064a \u0627\u0644\u0648\u0631\u0642\u0629 \u0627\u0644\u064a\u0648\u0645\u064a\u0629 \u0644\u0644\u0637\u0644\u0627\u0628 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
    "\u0643\u0645 \u0639\u062f\u062f \u0627\u0644\u0627\u0648\u0631\u0627\u0642 \u0627\u0644\u0627\u062e\u0628\u0627\u0631\u064a\u0629 \u0644\u0644\u0637\u0644\u0627\u0628 \u0627\u0644\u062a\u064a \u0648\u062c\u062f\u062a \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?",
    "\u0641\u064a \u0627\u064a \u0633\u0646\u0629 \u0628\u062f\u0627\u062a \u0648\u0631\u0642\u0629 \u0627\u0644\u0637\u0627\u0644\u0628 \u0627\u0644\u062d\u0633 \u0627\u0644\u0633\u0644\u064a\u0645 \u0628\u0627\u0644\u0646\u0634\u0631 \u0641\u064a \u0646\u0648\u062a\u0631\u062f\u0627\u0645?"
  ]
```

the questions are in the wrong format, and not readable, could you please have a look? thanks @lhoestq


TydiQA dataset is mixed and is not split per language

You can filter the languages this way:
```python
tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
```

Otherwise maybe we can have one configuration per language ?
What do you think of this for example ?

```python
load_dataset("tydiqa", "primary_task.en")
```

Hi
thank you very much for the great response, this will be really wonderful
to have one configuration per language, as one need the dataset in majority
of case per language for cross-lingual evaluations.
This becomes also then more close to TFDS format, which is separated per
language https://www.tensorflow.org/datasets/catalog/tydi_qa which will be
really awesome to have.
thanks

On Mon, Mar 29, 2021 at 6:17 PM Quentin Lhoest ***@***.***>
wrote:

> You can filter the languages this way:
>
> tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
>
> Otherwise maybe we can have one configuration per language ?
> What do you think of this for example ?
>
> load_dataset("tydiqa", "primary_task.en")
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2132#issuecomment-809516799>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXPW2PWSQ2RHG73O7TTGCY4LANCNFSM4Z7ER7IA>
> .
>

@lhoestq  I greatly appreciate any updates on this. thanks a lot

Hi @lhoestq
Currently TydiQA is mixed and user can only access the whole training set of all languages:
https://www.tensorflow.org/datasets/catalog/tydi_qa

for using this dataset, one need to train/evaluate in each separate language, and having them mixed, makes it hard to use this dataset. This is much convenient for user to have  them split and I appreciate your help on this.

Meanwhile, till hopefully this is split per language, I greatly appreciate telling me how I can preprocess and get data per language. thanks a lot


When training with Multi-Node Multi-GPU the worker 2 has TypeError: 'NoneType' object

Something isn't working

Hi ! Thanks for reporting
I was able to reproduce this issue. This was caused by missing split infos if a worker reloads the cache of the other worker.

I just opened https://github.com/huggingface/datasets/pull/2137 to fix this issue

The PR got merged :)
Feel free to try it out on the `master` branch

Sorry for the late reply.
Now everything just works well XD

version: 1.5.0
met a very strange error, I am training large scale language model, and need train on 2 machines(workers).
And sometimes I will get this error `TypeError: 'NoneType' object is not iterable`
This is traceback
```

71 |   | Traceback (most recent call last):
-- | -- | --
72 |   | File "run_gpt.py", line 316, in <module>
73 |   | main()
74 |   | File "run_gpt.py", line 222, in main
75 |   | delimiter="\t", column_names=["input_ids", "attention_mask", "chinese_ref"])
76 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/load.py", line 747, in load_dataset
77 |   | use_auth_token=use_auth_token,
78 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/builder.py", line 513, in download_and_prepare
79 |   | self.download_post_processing_resources(dl_manager)
80 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/builder.py", line 673, in download_post_processing_resources
81 |   | for split in self.info.splits:
82 |   | TypeError: 'NoneType' object is not iterable
83 |   | WARNING:datasets.builder:Reusing dataset csv (/usr/local/app/.cache/huggingface/datasets/csv/default-1c257ebd48e225e7/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2)
84 |   | Traceback (most recent call last):
85 |   | File "/data/miniconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
86 |   | "__main__", mod_spec)
87 |   | File "/data/miniconda3/lib/python3.7/runpy.py", line 85, in _run_code
88 |   | exec(code, run_globals)
89 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in <module>
90 |   | main()
91 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
92 |   | sigkill_handler(signal.SIGTERM, None)  # not coming back
93 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
94 |   | raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)

```
On worker 1 it loads the dataset well, however on worker 2 will get this error.
And I will meet this error from time to time, sometimes it just goes well.


wikiann dataset is missing columns

good first issue

Good for newcomers

Here please find TFDS format of this dataset: https://www.tensorflow.org/datasets/catalog/wikiann
where there is a span column, this is really necessary to be able to use the data, and I appreciate your help @lhoestq

Hi !
Apparently you can get the spans from the NER tags using `tags_to_spans` defined here:

https://github.com/tensorflow/datasets/blob/c7096bd38e86ed240b8b2c11ecab9893715a7d55/tensorflow_datasets/text/wikiann/wikiann.py#L81-L126

It would be nice to include the `spans` field in this dataset as in TFDS. This could be a good first issue for new contributors !

The objective is to use `tags_to_spans` in the `_generate_examples` method [here](https://github.com/huggingface/nlp/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L292-L316) to create he `spans` for each example.

Hi @lhoestq
thank you very much for the help, it would be very nice to have it included, here is the full code, one need to also convert tags to string first:

```
import datasets
from datasets import load_dataset

def tags_to_spans(tags):
  """Convert tags to spans."""
  spans = set()
  span_start = 0
  span_end = 0
  active_conll_tag = None
  for index, string_tag in enumerate(tags):
    # Actual BIO tag.
    bio_tag = string_tag[0]
    assert bio_tag in ["B", "I", "O"], "Invalid Tag"
    conll_tag = string_tag[2:]
    if bio_tag == "O":
      # The span has ended.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = None
      # We don't care about tags we are
      # told to ignore, so we do nothing.
      continue
    elif bio_tag == "B":
      # We are entering a new span; reset indices and active tag to new span.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = conll_tag
      span_start = index
      span_end = index
    elif bio_tag == "I" and conll_tag == active_conll_tag:
      # We're inside a span.
      span_end += 1
    else:
      # This is the case the bio label is an "I", but either:
      # 1) the span hasn't started - i.e. an ill formed span.
      # 2) We have IOB1 tagging scheme.
      # We'll process the previous span if it exists, but also include this
      # span. This is important, because otherwise, a model may get a perfect
      # F1 score whilst still including false positive ill-formed spans.
      if active_conll_tag:
        spans.add((active_conll_tag, (span_start, span_end)))
      active_conll_tag = conll_tag
      span_start = index
      span_end = index
  # Last token might have been a part of a valid span.
  if active_conll_tag:
    spans.add((active_conll_tag, (span_start, span_end)))
  # Return sorted list of spans
  return sorted(list(spans), key=lambda x: x[1][0])

dataset = load_dataset('wikiann', 'en', split="train")
ner_tags = {
   0:"O",
   1:"B-PER",
   2:"I-PER",
   3:"B-ORG",
   4:"I-ORG",
   5:"B-LOC",
   6:"I-LOC"
}

def get_spans(tokens, tags):
  """Convert tags to textspans."""
  spans = tags_to_spans(tags)
  text_spans = [
      x[0] + ": " + " ".join([tokens[i]
                              for i in range(x[1][0], x[1][1] + 1)])
      for x in spans
  ]
  if not text_spans:
    text_spans = ["None"]
  return text_spans

for i, d in enumerate(dataset):
   tokens = d['tokens']
   tags = d['ner_tags']
   tags = [ner_tags[i] for i in tags]
   spans = get_spans(tokens, tags)
   print("spans ", spans)
   print(d)
   if i > 10:
     break;
```
I am not sure how to contribute to the repository and how things work, could you let me know how one can access the datasets to be able to contribute to the repository? Maybe I could do it then
thanks

Cool ! Let me give you some context:

#### Contribution guide

You can find the contribution guide here:

https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md

It explains how to set up your dev environment in a few steps.

#### Dataset loading

Each Dataset is defined by a Table that have many rows (one row = one example) and columns (one column = one feature).
To change how a dataset is constructed, you have to modify its dataset script that you can find here:

https://github.com/huggingface/datasets/blob/master/datasets/wikiann/wikiann.py

It includes everything needed to load the WikiANN dataset.
You can load locally a modified version of `wikiann.py` with `load_dataset("path/to/wikiann.py")`.

#### Define a new column

Each column has a name and a type. You can see how the features of WikiANN are defined here:

https://github.com/huggingface/datasets/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L245-L263

Ideally we would have one additional feature "spans":
```python
        "spans": datasets.Sequence(datasets.Value("string")),
```

#### Compute the content of each row

To build the WikiANN rows, the _generate_examples method from [here](https://github.com/huggingface/nlp/blob/c98e4b8f23e3770c401c6d9326e243e1ffd599ec/datasets/wikiann/wikiann.py#L292-L316) is used. This function `yield` one python dictionary for each example:
```python
yield guid_index, {"tokens": tokens, "ner_tags": ner_tags, "langs": langs}
```

The objective would be to return instead something like
```python
spans = spans = get_spans(tokens, tags)
yield guid_index, {"tokens": tokens, "ner_tags": ner_tags, "langs": langs, "spans": spans}
```

Let me know if you have questions !

The PR was merged. Issue should be closed.

CC: @lhoestq

Hi
Wikiann dataset needs to have "spans" columns, which is necessary to be able to use this dataset, but this column is missing from huggingface datasets, could you please have a look? thank you @lhoestq


How to train BERT model with next sentence prediction?

Hi !
We're not using `TextDatasetForNextSentencePrediction` in `datasets`.
Although you can probably use the `TextDatasetForNextSentencePrediction.create_examples_from_document` on a dataset to prepare it for next sentence prediction.

Thanks.

Do you mean that `TextDatasetForNextSentencePrediction.create_exapmles_from_document` can be applied to dataset object other than `TextDatasetForNextSentencePrediction`  e.g. a `Dataset` object which is  loaded by  `datasets.load_dataset`?

It would probably require a bit of tweaking, but you can apply it to a dataset, yes.
This should give you a new dataset with sentence pairs you can train a model on.

You can find the documentation about dataset processing here:
https://huggingface.co/docs/datasets/processing.html#processing-data-with-map

Thank you for detail information.

I'll try to apply `create_examples_from_document` to `Dataset` object.

Hello.

I'm trying to pretrain the BERT model with next sentence prediction. Is there any function that supports next sentence prediction
like ` TextDatasetForNextSentencePrediction` of `huggingface/transformers` ?


Dialogue action slot name and value are reversed in MultiWoZ 2.2

dataset bug

A bug in a dataset script provided in the library

Hi
Good catch ! Thanks for reporting

If you are interested in contributing, feel free to open a PR to fix this :)

Hi @yjernite, thank you for adding MultiWoZ 2.2 in the huggingface datasets platform. It is beneficial!

I spot an error that the order of Dialogue action slot names and values are reversed.

https://github.com/huggingface/datasets/blob/649b2c469779bc4221e1b6969aa2496d63eb5953/datasets/multi_woz_v22/multi_woz_v22.py#L251-L262


make documentation more clear to use different cloud storage

This PR extends the cloud storage documentation. To show you can use a different `fsspec` implementation.


Replace legacy torch.Tensor constructor with torch.tensor

The title says it all (motivated by [this issue](https://github.com/pytorch/pytorch/issues/53146) in the pytorch repo).


Is dataset timit_asr broken?

Hi,

thanks for the report, but this is a duplicate of #2052.

@mariosasko
Thank you for your quick response! Following #2052, I've fixed the problem.

Using `timit_asr` dataset, I saw all records are the same.

``` python
from datasets import load_dataset, load_metric

timit = load_dataset("timit_asr")

from datasets import ClassLabel
import random
import pandas as pd
from IPython.display import display, HTML

def show_random_elements(dataset, num_examples=10):
    assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
    picks = []
    for _ in range(num_examples):
        pick = random.randint(0, len(dataset)-1)
        while pick in picks:
            pick = random.randint(0, len(dataset)-1)
        picks.append(pick)

    df = pd.DataFrame(dataset[picks])
    display(HTML(df.to_html()))

show_random_elements(timit['train'].remove_columns(["file", "phonetic_detail", "word_detail", "dialect_region", "id",
                                                    "sentence_type", "speaker_id"]), num_examples=20)

```

`output`

I double-checked it [here](https://huggingface.co/datasets/viewer/), and met the same problem.


Adding ScaNN library to do MIPS?

I haven't played with it (yet) but it sounds really cool !

@lhoestq Hi I am thinking of adding this new google library to do the MIPS similar to **add_faiss_idex**. As the paper suggests, it is really fast when it comes to retrieving the nearest neighbors.

https://github.com/google-research/google-research/tree/master/scann

![image](https://user-images.githubusercontent.com/16892570/112738294-78ec9800-8fc6-11eb-9a5f-3d7ee5818e76.png)


Problem downloading GEM wiki_auto_asset_turk dataset

Hi,

sadly I can't replicate the problem on my Windows machine. Try to update the library to the newest version with:
```bash
pip install git+https://github.com/huggingface/datasets
```

Thanks for the answer! I updated the library but unfortunately it didn't solve the problem.

Is there an error message ?
What stacktrace do you get if you interrupt the execution of the program while downloading ?

Sorry for the long time since my last comment, I tried again and don't seem to have the problem anymore, thanks for your support!

Great ! I'm closing the issue then. Feel free to re-open if you experience this issue again

@yjernite

### Summary

I am currently working on the GEM datasets and do not manage to download the wiki_auto_asset_turk data, whereas all other datasets download well with the same code.

### Steps to reproduce
Code snippet:

from datasets import load_dataset
#dataset = load_dataset('gem', 'web_nlg_en')
dataset = load_dataset('gem', 'wiki_auto_asset_turk')

```

**Expected behavior:**

I expect the dataset to start downloading (download bar appears and progresses toward 100%)

**Actual behavior:**
Instead of seeing the download bar appearing, nothing happens; the following appears in the console as expected, but nothing more:

Downloading: 36.6kB [00:00, 37.2MB/s]
Downloading: 41.7kB [00:00, ?B/s]
Downloading and preparing dataset gem/wiki_auto_asset_turk (download: 121.37 MiB, generated: 145.69 MiB, post-processed: Unknown size, total: 267.07 MiB) to C:\Users\sfmil\.cache\huggingface\datasets\gem\wiki_auto_asset_turk\1.0.0\f252756d7f1b8f019aac71a1623b2950acfe10d25d956668ac4eae4e93c58b8d...

### Is this a regression?
No, it was the first time I was trying to download this dataset (same for the other ones).

### Debug info
- Python version: Python 3.8.2
- OS version: Windows 10 Family


Fast table queries with interpolation search

## Intro

This should fix issue #1803

Currently querying examples in a dataset is O(n) because of the underlying pyarrow ChunkedArrays implementation.
To fix this I implemented interpolation search that is pretty effective since datasets usually verifies the condition of evenly distributed chunks (the default chunk size is fixed).

## Benchmark

Here is a [benchmark](https://pastebin.com/utEXUqsR) I did on bookcorpus (74M rows):

for the current implementation
```python
>>> python speed.py
Loaded dataset 'bookcorpus', len=74004228, nbytes=4835358766

========================= Querying unshuffled bookcorpus =========================

Avg access time key=1                                                 : 0.018ms
Avg access time key=74004227                                          : 0.215ms
Avg access time key=range(74003204, 74004228)                         : 1.416ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 92.532ms

========================== Querying shuffled bookcorpus ==========================

Avg access time key=1                                                 : 0.187ms
Avg access time key=74004227                                          : 6.642ms
Avg access time key=range(74003204, 74004228)                         : 90.941ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 3448.456ms
```

for the new one using interpolation search:
```python
>>> python speed.py
Loaded dataset 'bookcorpus', len=74004228, nbytes=4835358766

========================= Querying unshuffled bookcorpus =========================

Avg access time key=1                                                 : 0.076ms
Avg access time key=74004227                                          : 0.056ms
Avg access time key=range(74003204, 74004228)                         : 1.807ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 24.028ms

========================== Querying shuffled bookcorpus ==========================

Avg access time key=1                                                 : 0.061ms
Avg access time key=74004227                                          : 0.058ms
Avg access time key=range(74003204, 74004228)                         : 22.166ms
Avg access time key=RandIter(low=0, high=74004228, size=1024, seed=42): 42.757ms
```

The RandIter class is just an iterable of 1024 random indices from 0 to 74004228.

Here is also a plot showing the speed improvement depending on the dataset size:
![image](https://user-images.githubusercontent.com/42851186/112673587-32335c80-8e65-11eb-9a0c-58ad774abaec.png)

## Implementation details:
- `datasets.table.Table` objects implement interpolation search for the `slice` method
- The interpolation search requires to store the offsets of all the chunks of a table. The offsets are stored when the `Table` is initialized.
- `datasets.table.Table.slice` returns a `datasets.table.Table` using interpolation search
- `datasets.table.Table.fast_slice` returns a `pyarrow.Table` object using interpolation search. This is useful to get a part of a dataset if we don't need the indexing structure for future computations. For example it's used when querying an example as a dictionary.
- Now a `Dataset` object is always backed by a `datasets.table.Table` object. If one passes a `pyarrow.Table` to initialize a `Dataset`, then it's converted to a `datasets.table.Table`

## Checklist:

- [x] implement interpolation search
- [x] use `datasets.table.Table` in `Dataset` objects
- [x] update current tests
- [x] add tests for interpolation search
- [x] comments and docstring
- [x] add the benchmark to the CI

Fix #1803.


Add Validation For README

Good start! Here are some proposed next steps:
- We want the Class structure to reflect the template - so the parser know what section titles to expect and when something has gone wrong
- As a result, we don't need to parse the table of contents, since it will always be the same
- For each section/subsection it would be cool to have a variable saying whether it's filled out or not (when it's either empty or has `[More Information Needed]`)
- `attributes` should probably be `text`

@yjernite @lhoestq

I have added basic validation checking in the class. It works based on a YAML string. The YAML string determines the expected structure and which text is to be checked. The `text` can be true or false showing whether the text has to be checked or not for emptiness. Similarly, each subsection is parsed recursively. I have used print statement currently so that all issues are shown.

Please let me know your thoughts.

I haven't added a variable that keeps a track of whether the text is empty or not but it can be done easliy if required.

This looks like a good start !
Maybe we can use a field named `allow_empty` instead of `text` ?
Also +1 for keeping track of empty texts

Do you think you can have a way to collect all the validation fails of a readme and then raise an error showing all the failures instead of using print ?

Then we can create a `tests/test_dataset_cards.py` test file to make sure all the readmes of the repo are valid !

Hi @lhoestq

I have added changes accordingly. I prepared a list which stores all the errors and raises them at the end. I'm not sure if there is a better way.

Hi @lhoestq @yjernite

Please find the output for the existing READMEs here: http://p.ip.fi/2vYU

Thanks,
Gunjan

Hi @lhoestq

I have added some basic tests, also have restructured `ReadMe` class slightly.

There is one print statement currently, I'm not sure how to remove it. Basically, I want to warn but not stop further validation. I can't append to a list because the `error_list` and `warning_list` are both only present in `validate` method, and this print is present in the `parse` method. This is done when someone has repeated a section multiple times.  For e.g.:

```markdown
---
---

# Dataset Card for FashionMNIST
## Dataset Description
## Dataset Description
```

In this case, I check for validation only in the latest entry.

I can also raise an error (ideal case scenario), but still, it is in the `parse`. Should I add `error_lines` and `warning_lines` as instance variables? That would probably solve the issue.

In tests, I'm using a dummy YAML string for structure, we can also make it into a file but I feel that is not a hard requirement. Let me know your thoughts.

I will add tests for `from_readme` as well.

However, I would love to be able to check the exact message in the test when an error is raised. I checked a couple of methods but couldn't get it working. Let me know if you're aware of a way to do that.

Hi @lhoestq

Thanks for merging. :)
Thanks a lot to you and @yjernite  for guiding me and helping me out.

Yes, I'll also use the next PR for combining the readme and tags validation. ^_^

Hi @lhoestq, @yjernite

This is a simple Readme parser. All classes specific to different sections can inherit `Section` class, and we can define more attributes in each.

Let me know if this is going in the right direction :)

Currently the output looks like this, for `to_dict()` on `FashionMNIST` `README.md`:

```json
{
    "name": "./datasets/fashion_mnist/README.md",
    "attributes": "",
    "subsections": [
        {
            "name": "Dataset Card for FashionMNIST",
            "attributes": "",
            "subsections": [
                {
                    "name": "Table of Contents",
                    "attributes": "- [Dataset Description](#dataset-description)\n  - [Dataset Summary](#dataset-summary)\n  - [Supported Tasks](#supported-tasks-and-leaderboards)\n  - [Languages](#languages)\n- [Dataset Structure](#dataset-structure)\n  - [Data Instances](#data-instances)\n  - [Data Fields](#data-instances)\n  - [Data Splits](#data-instances)\n- [Dataset Creation](#dataset-creation)\n  - [Curation Rationale](#curation-rationale)\n  - [Source Data](#source-data)\n  - [Annotations](#annotations)\n  - [Personal and Sensitive Information](#personal-and-sensitive-information)\n- [Considerations for Using the Data](#considerations-for-using-the-data)\n  - [Social Impact of Dataset](#social-impact-of-dataset)\n  - [Discussion of Biases](#discussion-of-biases)\n  - [Other Known Limitations](#other-known-limitations)\n- [Additional Information](#additional-information)\n  - [Dataset Curators](#dataset-curators)\n  - [Licensing Information](#licensing-information)\n  - [Citation Information](#citation-information)\n  - [Contributions](#contributions)",
                    "subsections": []
                },
                {
                    "name": "Dataset Description",
                    "attributes": "- **Homepage:** [GitHub](https://github.com/zalandoresearch/fashion-mnist)\n- **Repository:** [GitHub](https://github.com/zalandoresearch/fashion-mnist)\n- **Paper:** [arXiv](https://arxiv.org/pdf/1708.07747.pdf)\n- **Leaderboard:**\n- **Point of Contact:**",
                    "subsections": [
                        {
                            "name": "Dataset Summary",
                            "attributes": "Fashion-MNIST is a dataset of Zalando's article images\u2014consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. We intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits.",
                            "subsections": []
                        },
                        {
                            "name": "Supported Tasks and Leaderboards",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        },
                        {
                            "name": "Languages",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Dataset Structure",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Data Instances",
                            "attributes": "A data point comprises an image and its label.",
                            "subsections": []
                        },
                        {
                            "name": "Data Fields",
                            "attributes": "- `image`: a 2d array of integers representing the 28x28 image.\n- `label`: an integer between 0 and 9 representing the classes with the following mapping:\n  | Label | Description |\n  | --- | --- |\n  | 0 | T-shirt/top |\n  | 1 | Trouser |\n  | 2 | Pullover |\n  | 3 | Dress |\n  | 4 | Coat |\n  | 5 | Sandal |\n  | 6 | Shirt |\n  | 7 | Sneaker |\n  | 8 | Bag |\n  | 9 | Ankle boot |",
                            "subsections": []
                        },
                        {
                            "name": "Data Splits",
                            "attributes": "The data is split into training and test set. The training set contains 60,000 images and the test set 10,000 images.",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Dataset Creation",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Curation Rationale",
                            "attributes": "**From the arXiv paper:**\nThe original MNIST dataset contains a lot of handwritten digits. Members of the AI/ML/Data Science community love this dataset and use it as a benchmark to validate their algorithms. In fact, MNIST is often the first dataset researchers try. \"If it doesn't work on MNIST, it won't work at all\", they said. \"Well, if it does work on MNIST, it may still fail on others.\"\nHere are some good reasons:\n- MNIST is too easy. Convolutional nets can achieve 99.7% on MNIST. Classic machine learning algorithms can also achieve 97% easily. Check out our side-by-side benchmark for Fashion-MNIST vs. MNIST, and read \"Most pairs of MNIST digits can be distinguished pretty well by just one pixel.\"\n- MNIST is overused. In this April 2017 Twitter thread, Google Brain research scientist and deep learning expert Ian Goodfellow calls for people to move away from MNIST.\n- MNIST can not represent modern CV tasks, as noted in this April 2017 Twitter thread, deep learning expert/Keras author Fran\u00e7ois Chollet.",
                            "subsections": []
                        },
                        {
                            "name": "Source Data",
                            "attributes": "",
                            "subsections": [
                                {
                                    "name": "Initial Data Collection and Normalization",
                                    "attributes": "**From the arXiv paper:**\nFashion-MNIST is based on the assortment on Zalando\u2019s website. Every fashion product on Zalando has a set of pictures shot by professional photographers, demonstrating different aspects of the product, i.e. front and back looks, details, looks with model and in an outfit. The original picture has a light-gray background (hexadecimal color: #fdfdfd) and stored in 762 \u00d7 1000 JPEG format. For efficiently serving different frontend components, the original picture is resampled with multiple resolutions, e.g. large, medium, small, thumbnail and tiny.\nWe use the front look thumbnail images of 70,000 unique products to build Fashion-MNIST. Those products come from different gender groups: men, women, kids and neutral. In particular, whitecolor products are not included in the dataset as they have low contrast to the background. The thumbnails (51 \u00d7 73) are then fed into the following conversion pipeline:\n1. Converting the input to a PNG image.\n2. Trimming any edges that are close to the color of the corner pixels. The \u201ccloseness\u201d is defined by the distance within 5% of the maximum possible intensity in RGB space.\n3. Resizing the longest edge of the image to 28 by subsampling the pixels, i.e. some rows and columns are skipped over.\n4. Sharpening pixels using a Gaussian operator of the radius and standard deviation of 1.0, with increasing effect near outlines.\n5. Extending the shortest edge to 28 and put the image to the center of the canvas.\n6. Negating the intensities of the image.\n7. Converting the image to 8-bit grayscale pixels.",
                                    "subsections": []
                                },
                                {
                                    "name": "Who are the source image producers?",
                                    "attributes": "**From the arXiv paper:**\nEvery fashion product on Zalando has a set of pictures shot by professional photographers, demonstrating different aspects of the product, i.e. front and back looks, details, looks with model and in an outfit.",
                                    "subsections": []
                                }
                            ]
                        },
                        {
                            "name": "Annotations",
                            "attributes": "",
                            "subsections": [
                                {
                                    "name": "Annotation process",
                                    "attributes": "**From the arXiv paper:**\nFor the class labels, they use the silhouette code of the product. The silhouette code is manually labeled by the in-house fashion experts and reviewed by a separate team at Zalando. Each product Zalando is the Europe\u2019s largest online fashion platform. Each product contains only one silhouette code.",
                                    "subsections": []
                                },
                                {
                                    "name": "Who are the annotators?",
                                    "attributes": "**From the arXiv paper:**\nThe silhouette code is manually labeled by the in-house fashion experts and reviewed by a separate team at Zalando.",
                                    "subsections": []
                                }
                            ]
                        },
                        {
                            "name": "Personal and Sensitive Information",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Considerations for Using the Data",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Social Impact of Dataset",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        },
                        {
                            "name": "Discussion of Biases",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        },
                        {
                            "name": "Other Known Limitations",
                            "attributes": "[More Information Needed]",
                            "subsections": []
                        }
                    ]
                },
                {
                    "name": "Additional Information",
                    "attributes": "",
                    "subsections": [
                        {
                            "name": "Dataset Curators",
                            "attributes": "Han Xiao and Kashif Rasul and Roland Vollgraf",
                            "subsections": []
                        },
                        {
                            "name": "Licensing Information",
                            "attributes": "MIT Licence",
                            "subsections": []
                        },
                        {
                            "name": "Citation Information",
                            "attributes": "@article{DBLP:journals/corr/abs-1708-07747,\n  author    = {Han Xiao and\n               Kashif Rasul and\n               Roland Vollgraf},\n  title     = {Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning\n               Algorithms},\n  journal   = {CoRR},\n  volume    = {abs/1708.07747},\n  year      = {2017},\n  url       = {http://arxiv.org/abs/1708.07747},\n  archivePrefix = {arXiv},\n  eprint    = {1708.07747},\n  timestamp = {Mon, 13 Aug 2018 16:47:27 +0200},\n  biburl    = {https://dblp.org/rec/bib/journals/corr/abs-1708-07747},\n  bibsource = {dblp computer science bibliography, https://dblp.org}\n}",
                            "subsections": []
                        },
                        {
                            "name": "Contributions",
                            "attributes": "Thanks to [@gchhablani](https://github.com/gchablani) for adding this dataset.",
                            "subsections": []
                        }
                    ]
                }
            ]
        }
    ]
}
```

Thanks,
Gunjan


dataset viewer does not work anymore

Thanks for reporting :) We're looking into it

Hi
I normally use this link to see all datasets and how I can load them

https://huggingface.co/datasets/viewer/

Now I am getting

502 Bad Gateway
nginx/1.18.0 (Ubuntu)

could you bring this webpage back ? this was very helpful @lhoestq
thanks for your help


copy.deepcopy os.environ instead of copy

Fixes: https://github.com/huggingface/datasets/issues/2115

- bug fix: using envrion.copy() returns a dict.
- using deepcopy(environ) returns an `_environ` object
- Changing the datatype of the _environ object can break code, if subsequent libraries perform operations using apis exclusive to the environ object, like `environ.getenv()` for example.

Testing:

Tested the change on my terminal:

```
>>> import os
>>> x = deepcopy(os.environ)
>>> y = os.environ
>>> x is y
False
>>> isinstance(x, type(os.environ))
True
>>> z = os.environ.copy()
>>> isinstance(z, type(os.environ))
False
>>> isinstance(z, dict)
True
```


Remove os.environ.copy in Dataset.map

I thought deepcopy on `os.environ` is unsafe (see [this](https://stackoverflow.com/questions/13142972/using-copy-deepcopy-on-os-environ-in-python-appears-broken)), but I can't replicate the behavior described in the linked SO thread.

Closing this one because #2119 has a much cleaner approach.

Replace `os.environ.copy` with in-place modification
Fixes #2115


load_metric from local "glue.py" meet error 'NoneType' object is not callable

> @Frankie123421 what was the resolution to this?

use glue_metric.py instead of glue.py in load_metric

actual_task = "mnli" if task == "mnli-mm" else task
dataset = load_dataset(path='/home/glue.py', name=actual_task)
metric = load_metric(path='/home/glue.py', name=actual_task)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-7ab77a465d81> in <module>
      1 actual_task = "mnli" if task == "mnli-mm" else task
      2 dataset = load_dataset(path='/home/jcli/glue.py', name=actual_task)
----> 3 metric = load_metric(path='/home/jcli/glue.py', name=actual_task)

~/anaconda3/envs/pytorch/lib/python3.6/site-packages/datasets/load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
    508         keep_in_memory=keep_in_memory,
    509         experiment_id=experiment_id,
--> 510         **metric_init_kwargs,
    511     )
    512

TypeError: 'NoneType' object is not callable

Please help


Creating custom dataset results in error while calling the map() function

Hi,

the `_data` attribute is missing due to `MyDataset.__init__` not calling the parent `__init__`. However, I don't think it's a good idea to subclass the `datasets.Dataset` class (e.g. it's kind of dangerous to override `datasets.Dataset.__getitem__`). Instead, it's better to follow the "association over inheritance" approach with a simple wrapper class that delegates calls to a wrapped `Dataset` (map, etc.). Btw, the library offers the `datasets.Dataset.from_pandas` class method to directly create a `datasets.Dataset` from the dataframe.

calling `map()` of `datasets` library results into an error while defining a Custom dataset.
Reproducible example:
```
import datasets
class MyDataset(datasets.Dataset):

    def __init__(self, sentences):
        "Initialization"
        self.samples = sentences

    def __len__(self):
        "Denotes the total number of samples"
        return len(self.samples)

    def __getitem__(self, index):
        "Generates one sample of data"
        # Select sample
        # Load data and get label
        samples = self.samples[index]

        return samples

def preprocess_function_train(examples):
        inputs = examples
        labels = [example+tokenizer.eos_token for example in examples ]
        inputs = tokenizer(inputs, max_length=30, padding=True, truncation=True)
        labels = tokenizer(labels, max_length=30, padding=True, truncation=True)
        model_inputs = inputs
        model_inputs["labels"] = labels["input_ids"]
        print("about to return")
        return model_inputs

##train["sentence"] is dataframe column
train_dataset = MyDataset(train['sentence'].values.tolist())
train_dataset = train_dataset.map(
            preprocess_function,
            batched = True,
            batch_size=32
        )
```

Stack trace of error:
```
Traceback (most recent call last):
  File "dir/train_generate.py", line 362, in <module>
    main()
  File "dir/train_generate.py", line 245, in main
    train_dataset = train_dataset.map(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1244, in map
    return self._map_single(
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 149, in wrapper
    unformatted_columns = set(self.column_names) - set(self._format_columns or [])
  File "anaconda_dir/anaconda3/envs/env1/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 526, in column_names
    return self._data.column_names
AttributeError: 'MyDataset' object has no attribute '_data'
```


The datasets.map() implementation modifies the datatype of os.environ object

In our testing, we noticed that the datasets.map() implementation is modifying the datatype of python os.environ object from '_Environ' to 'dict'.

This causes following function calls to fail as follows:

`
     x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    TypeError: get() takes no keyword arguments
`
It looks like the following line in datasets.map implementation introduced this functionality.

https://github.com/huggingface/datasets/blob/0cb1ac06acb0df44a1cf4128d03a01865faa2504/src/datasets/arrow_dataset.py#L1421

Here is the test script to reproduce this error.

```
from datasets import load_dataset
from transformers import AutoTokenizer
import os

def test_train():
    model_checkpoint = "distilgpt2"
    datasets = load_dataset('wikitext', 'wikitext-2-raw-v1')
    tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)
    tokenizer.pad_token = tokenizer.eos_token

    def tokenize_function(examples):
        y = tokenizer(examples['text'], truncation=True, max_length=64)
        return y

    x = os.environ.get("TEST_ENV_VARIABLE_BEFORE_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_BEFORE_dataset_map {x}")
    print(f"Data type of os.environ before datasets.map = {os.environ.__class__.__name__}")
    datasets.map(tokenize_function, batched=True, num_proc=2, remove_columns=["text"])
    print(f"Data type of os.environ after datasets.map = {os.environ.__class__.__name__}")
    x = os.environ.get("TEST_ENV_VARIABLE_AFTER_dataset_map", default=None)
    print(f"Testing environment variable: TEST_ENV_VARIABLE_AFTER_dataset_map {x}")

if __name__ == "__main__":
    test_train()

```


Support for legal NLP datasets (EURLEX, ECtHR cases and EU-REG-IR)

> Awesome thank you :)
> This is really cool
>
> I left a few comments.
>
> Also it looks like the dummy data are quite big (100-200KB each). Can you try to reduce their sizes please ? For example I noticed that all the jsonl files inside the `dummy_data.zip` files have 20 lines. Can you only keep 2 lines instead ?

Hi @lhoestq, I did my best to improve the README files, while I also decreased dummy data examples. I included one more legal dataset.

@lhoestq thanks for your review.

 I shortened the examples in README files and removed `DEFAULT_CONFIG_BUILDER` from `eu_regulatory_ir.py`.

Add support for two legal NLP datasets:

- EURLEX (https://www.aclweb.org/anthology/P19-1636/)
- ECtHR cases (https://arxiv.org/abs/2103.13084)
- EU-REG-IR (https://arxiv.org/abs/2101.10726)


Implement Dataset as context manager

When used as context manager, it would be safely deleted if some exception is raised.

This will avoid
> During handling of the above exception, another exception occurred:


Support for legal NLP datasets (EURLEX and ECtHR cases)

Add support for two legal NLP datasets:
- EURLEX (https://www.aclweb.org/anthology/P19-1636/)
- ECtHR cases (https://arxiv.org/abs/2103.13084)


Compute WER metric iteratively

I discussed with Patrick and I think we could have a nice addition: have a parameter `concatenate_texts` that, if `True`, uses the old implementation.

By default `concatenate_texts` would be `False`, so that sentences are evaluated independently, and to save resources (the WER computation has a quadratic complexity).

Some users might still want to use the old implementation.

@lhoestq @patrickvonplaten are you sure of the parameter name `concatenate_texts`? I was thinking about something like `iter`...

Not sure about the name, if you can improve it feel free to do so ^^'
The old implementation computes the WER on the concatenation of all the input texts, while the new one makes WER measures computation independent for each reference/prediction pair.
That's why I thought of `concatenate_texts`

@lhoestq yes, but the end user does not necessarily know the details of the implementation of the WER computation.

From the end user perspective I think it might make more sense: how do you want to compute the metric?
- all in once, more RAM memory needed?
- iteratively, less RAM requirements?

Because of that I was thinking of something like `iter` or `iterative`...

Personally like `concatenate_texts` better since I feel like `iter` or `iterate` are quite vague

Compute WER metric iteratively to avoid MemoryError.

Fix #2078.


Fix incorrect assertion in builder.py

Hi ! The SplitInfo is not always available. By default you would get `split_info.num_examples == 0`
So unfortunately we can't use this assertion you suggested

> Hi ! The SplitInfo is not always available. By default you would get `split_info.num_examples == 0`
> So unfortunately we can't use this assertion you suggested

Then it would be better to just remove the assertion, because the existing assertion does nothing.

Fix incorrect num_examples comparison assertion in builder.py


Add more issue templates and customize issue template chooser

If you agree, I could also add a link to [Discussions](https://github.com/huggingface/datasets/discussions) in order to reinforce the use of Discussion to make Questions (instead of Issues).

I could also add some other templates: Bug, Feature Request,...

@theo-m we wrote our same comments at the same time... 😉

When opening an issue, it is not evident for the users how to choose a blank issue template. There is a link at the bottom of all the other issue templates (`Don’t see your issue here? Open a blank issue.`), but this is not very visible for users. This is the reason why many users finally chose the `add-dataset` template instead (this is more visible) for issues that indeed are not requesting the addition of a new dataset.

~~With this PR, the default blank issue template would be as visible as the other templates (as the `add-dataset` template), thus making easier for the users to choose it.~~

With this PR:
- more issue templates, besides `add-dataset`, are added: `bug-report` and `feature-request`
- the issue template chooser is customized, so that it now includes a link to `Discussions` for questions


Is there a way to use a GPU only when training an Index  in the process of add_faisis_index?

question

Further information is requested

Motivation - Some FAISS indexes like IVF consist of the training step that clusters the dataset into a given number of indexes. It would be nice if we can use a GPU to do the training step and covert the index back to CPU as mention in [this faiss example](https://gist.github.com/mdouze/46d6bbbaabca0b9778fca37ed2bcccf6).


Metadata validation

> Also I was wondering this is really needed to have `utils.metadata` as a submodule of `datasets` ? This is only used by the CI so I'm not sure we should have this in the actual `datasets` package.

I'm unclear on the suggestion, would you rather have a root-level `./metadata.py` file? I think it's well where it is, if anything we could move it out of utils and into `datasets` as it could be used by e.g. `DatasetDict` so that users can pull the metadata easily rather than have to reparse the readme.

Ok that makes sense if we want to have functions that parse the metadata for users

Hi @theo-m @lhoestq

This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)

Sorry for the delay in responding.

Thanks,
Gunjan

> Hi @theo-m @lhoestq
>
> This seems very interesting. Should I add the descriptions to the PR on `datasets-tagging`? Alternatively, I can also create a google-sheet/markdown table :)
>
> Sorry for the delay in responding.
>
> Thanks,
> Gunjan

Hi @gchhablani, yes I think at the moment the best solution is for you to write in `datasets-tagging`, as the PR will allow us to discuss and review, even though the work will be ported to this repo in the end.
Or we wait for this to be merged and you reopen the PR here, your call :)

- `pydantic` metadata schema with dedicated validators against our taxonomy
- ci script to validate new changes against this schema and start a vertuous loop
- soft validation on tasks ids since we expect the taxonomy to undergo some changes in the near future

for reference with the current validation we have ~365~ 378 datasets with invalid metadata! full error report [_here_.](https://gist.github.com/theo-m/61b3c0c47fc6121d08d3174bd4c2a26b)


WMT19 Dataset for Kazakh-English is not formatted correctly

Hi ! Thanks for reporting

By looking at the raw `news-commentary-v14.en-kk.tsv` file, it looks like there are at least 17 lines with this issue.
Moreover these issues are not always the same:
- L97 is only `kk` text and must be appended at the end of the `kk` text of the **next** line
- L2897 is only `kk` text and must be appended at the end of the `kk` text of the **previous** line
- L1247 and L1248 are only `kk` texts and must be inserted at the **beginning** of the `kk` text of the next line
- (and there are many others)

It would be nice to have a corrected version of this file ! The file is available in the `wmt/news-commentary` repository on the Datasets Hub here:
https://huggingface.co/datasets/wmt/news-commentary/tree/main/v14/training

Then maybe we can notify the WMT authors and host the corrected version somewhere

In addition to the bug of languages being switched from Issue @415, there are incorrect translations in the dataset because the English-Kazakh translations have a one off formatting error.

The News Commentary v14 parallel data set for kk-en from http://www.statmt.org/wmt19/translation-task.html has a bug here:

> Line 94. The Swiss National Bank, for its part, has been battling with the deflationary effects of the franc’s dramatic appreciation over the past few years.	Швейцарияның Ұлттық банкі өз тарапынан, соңғы бірнеше жыл ішінде франк құнының қатты өсуінің дефляциялық әсерімен күресіп келеді.
>
> Line 95. Дефляциялық күштер 2008 жылы терең және ұзаққа созылған жаһандық дағдарысқа байланысты орын алған ірі экономикалық және қаржылық орын алмасулардың арқасында босатылды.  Жеке қарыз қаражаты үлесінің қысқаруы орталық банктің рефляцияға жұмсалған күш-жігеріне тұрақты соққан қарсы желдей болды.
>
> Line 96. The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.	2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.

As you can see, line 95 has only the Kazakh translation which should be part of line 96. This causes all of the following English-Kazakh translation pairs to be one off rendering ALL of those translations incorrect. This issue was not fixed when the dataset was imported to Huggingface. By running this code

```
import datasets
from datasets import load_dataset
dataset = load_dataset('wmt19', 'kk-en')
for key in dataset['train']['translation']:
    if 'The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008.' in key['kk']:
        print(key['en'])
        print(key['kk'])
        break
```
we get:
> 2009 жылы, алдыңғы қатарлы экономикалардың шамамен үштен бірі бағаның төмендеуін көрсетті, бұл соғыстан кейінгі жоғары деңгей болды.
> The deflationary forces were unleashed by the major economic and financial dislocations associated with the deep and protracted global crisis that erupted in 2008. Private deleveraging became a steady headwind to central bank efforts to reflate.

which shows that the issue still persists in the Huggingface dataset. The Kazakh sentence matches up to the next English sentence in the dataset instead of the current one.

Please let me know if there's you have any ideas to fix this one-off error from the dataset or if this can be fixed by Huggingface.


Request to remove S2ORC dataset

Hello @kyleclo! Currently, we are getting the data from your bucket, so if you remove it the HF script won't work anymore :)

Until you solve things on your end, @lhoestq suggested we just return a warning message when people try to load that dataset from HF. What would you like it to say?

Hi @kyleclo, as of today, you have not removed your bucket data yet, and therefore HuggingFace can download it from there.

Is it OK? Are you planning to eventually delete it? Thank you.

Hi!  Sorry I missed @yjernite 's previous message, thanks for responding!

Is there an option where we can keep our data in our bucket, but the HF script no longer pulls data from it?

Hi!  I was wondering if it's possible to remove [S2ORC](https://huggingface.co/datasets/s2orc) from hosting on Huggingface's platform?  Unfortunately, there are some legal considerations about how we make this data available.  Happy to add back to Huggingface's platform once we work out those hurdles!  Thanks!


Trouble loading wiki_movies

Hi ! `wiki_movies` was added in `datasets==1.2.0`. However it looks like you have `datasets==1.1.2`.

To use `wiki_movies`, please update `datasets` with
```
pip install --upgrade datasets
```

Thanks a lot! That solved it and I was able to upload a model trained on it as well :)

Hello,
I am trying to load_dataset("wiki_movies") and it gives me this error -

`FileNotFoundError: Couldn't find file locally at wiki_movies/wiki_movies.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/wiki_movies/wiki_movies.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/wiki_movies/wiki_movies.py`

Trying to do `python run_mlm.py \
    --model_name_or_path roberta-base \
    --dataset_name wiki_movies \` also gives the same error.

Is this something on my end? From what I can tell, this dataset was re-added by @lhoestq a few months ago.
Thank you!


citation, homepage, and license fields of `dataset_info.json` are duplicated many times

enhancement

New feature or request

good first issue

Good for newcomers

Thanks for reporting :)
Maybe we can concatenate fields only if they are different.

Currently this is done here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/src/datasets/info.py#L180-L196

This can be a good first contribution to the library.
Please comment if you'd like to improve this and open a PR :)

This happens after a `map` operation when `num_proc` is set to `>1`. I tested this by cleaning up the json before running the `map` op on the dataset so it's unlikely it's coming from an earlier concatenation.

Example result:
```
"citation": "@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n    url    = {https://dumps.wikimedia.org}\n}\n\n@ONLINE {wikidump,\n    author = {Wikimedia Foundation},\n    title  = {Wikimedia Downloads},\n
```

@lhoestq and I believe this is happening due to the fields being concatenated `num_proc` times.


Move Dataset.to_csv to csv module

refactoring

Restructuring existing code without changing its external behavior

Move the implementation of `Dataset.to_csv` to module `datasets.io.csv`.