kardwalker/my_dataset huggingface dataset:
GitHub issue #3189: conll2003 incorrect label explanation
State: closed | Created: 2021-11-01T11:03:30Z

In the [conll2003](https://huggingface.co/datasets/conll2003#data-fields) README, the labels are described as follows

> - `id`: a `string` feature.
> - `tokens`: a `list` of `string` features.
> - `pos_tags`: a `list` of classification labels, with possible values including `"` (0), `''` (1), `#` (2), `$` (3), `(` (4).
> - `chunk_tags`: a `list` of classification labels, with possible values including `O` (0), `B-ADJP` (1), `I-ADJP` (2), `B-ADVP` (3), `I-ADVP` (4).
> - `ner_tags`: a `list` of classification labels, with possible values including `O` (0), `B-PER` (1), `I-PER` (2), `B-ORG` (3), `I-ORG` (4) `B-LOC` (5), `I-LOC` (6) `B-MISC` (7), `I-MISC` (8).

First of all, it would be great if we can get a list of ALL possible pos_tags.

Second, the chunk tags labels cannot be correct. The description says the values go from 0 to 4 whereas the data shows values from at least 11 to 21 and 0.

EDIT: not really a bug, sorry for mistagging.
GitHub issue #3188: conll2002 issues
State: closed | Created: 2021-11-01T09:49:24Z

**Link:** https://huggingface.co/datasets/conll2002

The dataset viewer throws a server error when trying to preview the dataset. 

```
Message:       Extraction protocol 'train' for file at 'https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/esp.train' is not implemented yet
```

In addition, the "point of contact" has encoding issues and does not work when clicked.

Am I the one who added this dataset ? No, @lhoestq  did
GitHub issue #3186: Dataset viewer for nli_tr
State: closed | Created: 2021-10-31T03:56:33Z

## Dataset viewer issue for '*nli_tr*'

**Link:** https://huggingface.co/datasets/nli_tr

Hello,

Thank you for the new dataset preview feature that will help the users to view the datasets online.

We just noticed that the dataset viewer widget in the `nli_tr` dataset shows the error below. The error must be due to a temporary problem that may have blocked access to the dataset through the dataset viewer. But the dataset is currently accessible through the link in the error message.   May we kindly ask if it would be possible to rerun the job so that it can access the dataset for the dataset viewer function?

Thank you.
Emrah


------------------------------------------
Server Error
Status code:   404
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: 'zip://snli_tr_1.0_train.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip
------------------------------------------

Am I the one who added this dataset ? Yes
GitHub issue #3185: 7z dataset preview not implemented?
State: closed | Created: 2021-10-30T20:18:27Z

## Dataset viewer issue for dataset 'samsum'

**Link:** https://huggingface.co/datasets/samsum

Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol '7z' for file at 'https://arxiv.org/src/1911.12237v2/anc/corpus.7z' is not implemented yet
GitHub issue #3184: RONEC v2
State: closed | Created: 2021-10-30T10:50:03Z

Hi, as we've recently finished with the new RONEC (Romanian Named Entity Corpus), we'd like to update the dataset here as well. It's actually essential as links to V1 are no longer valid. 

In reality we'd like to replace completely v1, as v2 is a full re-annotation of v1 with additional data (up to 2x size vs v1).

I've run the make style and all the dummy and real data test, and they passed.

I hope it's okay to merge the new RONEC v2 in the datasets. 

Thanks!
GitHub issue #3182: Don't memoize strings when hashing since two identical strings may have different python ids
State: closed | Created: 2021-10-29T16:26:17Z

When hashing an object that has several times the same string, the hashing could return a different hash if the identical strings share the same python `id()` or not.

Here is an example code that shows how the issue can affect the caching:
```python
import json
import pyarrow as pa
from datasets.features import Features
from datasets.fingerprint import Hasher

schema = pa.schema([pa.field("some_string", pa.string()), pa.field("another_string", pa.string())])
features_from_schema = Features.from_arrow_schema(schema)
Hasher.hash(features_from_schema) # dffa9dca9a73fd8c

features_dict = json.loads('{"some_string": {"dtype": "string", "id": null, "_type": "Value"}, "another_string": {"dtype": "string", "id": null, "_type": "Value"}}')
features_from_json = Features.from_dict(features_dict)
Hasher.hash(features_from_json) # 3812e76b15e6420e

features_from_schema == features_from_json # True
```

This is because in `features_dict`, some strings like "dtype" are repeated but don't share the same id, contrary to the ones in `features_from_schema`.

I fixed that by disabling memoization for strings.

This could be optimized in the future by implementing a smarter memoization with a special handling for strings.
GitHub issue #3181: `None` converted to `"None"` when loading a dataset
State: closed | Created: 2021-10-29T15:23:53Z

## Describe the bug
When loading a dataset `None` values of the type `NoneType` are converted to `'None'` of the type `str`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
qasper = load_dataset("qasper", split="train", download_mode="reuse_cache_if_exists")
print(qasper[60]["full_text"]["section_name"])
```

When installing version 1.1.40, the output is
`[None, 'Introduction', 'Benchmark Datasets', ...]`

When installing from the master branch, the output is
`['None', 'Introduction', 'Benchmark Datasets', ...]`

Notice how the first element was changed from `NoneType` to `str`.

## Expected results
`None` should stay as is.

## Actual results
`None` is converted to a string.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1
GitHub issue #3180: fix label mapping
State: closed | Created: 2021-10-29T14:42:24Z

Fixing label mapping for hlgd.
0 correponds to same event and 1 corresponds to different event
<img width="642" alt="Capture d’écran 2021-10-29 à 10 39 58 AM" src="https://user-images.githubusercontent.com/16107619/139454810-1f225e3d-ad48-44a8-b8b1-9205c9533839.png">
<img width="638" alt="Capture d’écran 2021-10-29 à 10 40 09 AM" src="https://user-images.githubusercontent.com/16107619/139454813-93066a3c-7d33-4f56-b133-2f1a7661e438.png">
nt
GitHub issue #3177: More control over TQDM when using map/filter with multiple processes
State: closed | Created: 2021-10-29T11:56:16Z

It would help with the clutter in my terminal if tqdm is only shown for rank 0 when using `num_proces>0` in the map and filter methods of datasets.

```python
dataset.map(lambda examples: tokenize(examples["text"]), batched=True, num_proc=6)
```

The above snippet leads to a lot of TQDM bars and depending on your terminal, these will not overwrite but keep pushing each other down.

```
 #0:   0%|          | 0/13 [00:00<?, ?ba/s]
 #1:   0%|          | 0/13 [00:00<?, ?ba/s]
 #2:   0%|          | 0/13 [00:00<?, ?ba/s]
 #3:   0%|          | 0/13 [00:00<?, ?ba/s]
 #4:   0%|          | 0/13 [00:00<?, ?ba/s]
 #5:   0%|          | 0/13 [00:00<?, ?ba/s]
 #0:   8%|          | 1/13 [00:00<?, ?ba/s]
 #1:   8%|          | 1/13 [00:00<?, ?ba/s]
...
```

Instead, it would be welcome if we had the option to only show the progress of rank 0.
GitHub issue #3176: OpenSLR dataset: update generate_examples to properly extract data for SLR83
State: closed | Created: 2021-10-29T00:59:27Z

Fixed #3168.

The SLR38 indices are CSV files and there wasn't any code in openslr.py to process these files properly. The end result was an empty table.

I've added code to properly process these CSV files.
GitHub issue #3175: Add docs for `to_tf_dataset`
State: closed | Created: 2021-10-28T20:55:22Z

This PR adds some documentation for new features released in v1.13.0, with the main addition being `to_tf_dataset`:

- Show how to use `to_tf_dataset` in the tutorial, and move `set_format(type='tensorflow'...)` to the Process section (let me know if I'm missing anything @Rocketknight1 😅).
- Add an example for loading dataset from multiple zipped CSV files to the Load section.
- Add an example for removing columns for an `IterableDataset`.
- Add graphic for visualizing streaming.
GitHub issue #3172: `SystemError 15` thrown in `Dataset.__del__` when using `Dataset.map()` with `num_proc>1`
State: closed | Created: 2021-10-28T10:29:00Z

## Describe the bug
I use `datasets.map` to preprocess some data in my application. The error `SystemError 15` is thrown at the end of the execution of `Dataset.map()` (only with `num_proc>1`. Traceback included bellow. 
 
The exception is raised only when the code runs within a specific context. Despite ~10h spent investigating this issue, I have failed to isolate the bug, so let me describe my setup. 

In my project, `Dataset` is wrapped into a `LightningDataModule` and the data is preprocessed when calling `LightningDataModule.setup()`. Calling `.setup()` in an isolated script works fine (even when wrapped with `hydra.main()`). However, when calling `.setup()` within the experiment script (depends on `pytorch_lightning`), the script crashes and `SystemError 15`.

I could avoid throwing this error by modifying ` Dataset.__del__()` (see bellow), but I believe this only moves the problem somewhere else. I am completely stuck with this issue, any hint would be welcome. 

```python
class Dataset()
    ...
    def __del__(self):
        if hasattr(self, "_data"):
            _ = self._data # <- ugly trick that allows avoiding the issue.
            del self._data
        if hasattr(self, "_indices"):
            del self._indices
```

## Steps to reproduce the bug
```python
# Unfortunately I couldn't isolate the bug.
```

## Expected results
Calling `Dataset.map()` without throwing an exception. Or at least raising a more detailed exception/traceback.

## Actual results
```
Exception ignored in: <function Dataset.__del__ at 0x7f7cec179160>███████████████████████████████████████████████████| 5/5 [00:05<00:00,  1.17ba/s]
Traceback (most recent call last):
  File ".../python3.8/site-packages/datasets/arrow_dataset.py", line 906, in __del__
    del self._data
  File ".../python3.8/site-packages/ray/worker.py", line 1033, in sigterm_handler
    sys.exit(signum)
SystemExit: 15

```

## Environment info

Tested on 2 environments:

**Environment 1.**
- `datasets` version: 1.14.0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.0

**Environment 2.**
- `datasets` version: 1.14.0
- Platform: Linux-4.18.0-305.19.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- PyArrow version: 6.0.0
GitHub issue #3171: Raise exceptions instead of using assertions for control flow
State: closed | Created: 2021-10-27T18:26:52Z

Motivated by https://github.com/huggingface/transformers/issues/12789 in Transformers, one welcoming change would be replacing assertions with proper exceptions. The only type of assertions we should keep are those used as sanity checks.

Currently, there is a total of 87 files with the `assert` statements (located under `datasets` and `src/datasets`), so when working on this, to manage the PR size, only modify 4-5 files at most before submitting a PR.
GitHub issue #3168: OpenSLR/83 is empty
State: closed | Created: 2021-10-26T19:42:21Z

## Describe the bug
As the summary says, openslr / SLR83 / train is empty.

The dataset returned after loading indicates there are **zero** rows. The correct number should be **17877**.
## Steps to reproduce the bug
```python
import datasets


datasets.load_dataset('openslr', 'SLR83')
```

## Expected results
```
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'sentence'],
        num_rows: 17877
    })
})
```
## Actual results
```
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'sentence'],
        num_rows: 0
    })
})
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.1.dev0 (master HEAD)
- Platform: Ubuntu 20.04
- Python version: 3.7.10
- PyArrow version: 3.0.0
GitHub issue #3167: bookcorpusopen no longer works
State: closed | Created: 2021-10-26T16:06:15Z

## Describe the bug

When using the latest version of datasets (1.14.0), I cannot use the `bookcorpusopen` dataset. The process blocks always around `9924 examples [00:06, 1439.61 examples/s]` when preparing the dataset. I also noticed that after half an hour the process is automatically killed because of the RAM usage (the machine has 1TB of RAM...).

This did not happen with 1.4.1.
I tried also `rm -rf ~/.cache/huggingface` but did not help.
Changing python version between 3.7, 3.8 and 3.9 did not help too.

## Steps to reproduce the bug
```python
import datasets
d = datasets.load_dataset('bookcorpusopen')
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.4.0-1054-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1
GitHub issue #3164: Add raw data files to the Hub with GitHub LFS for canonical dataset
State: closed | Created: 2021-10-25T23:28:21Z

I'm interested in sharing the CaseHOLD dataset (https://arxiv.org/abs/2104.08671) as a canonical dataset on the HuggingFace Hub and would like to add the raw data files to the Hub with GitHub LFS, since it seems like a more sustainable long term storage solution, compared to other storage solutions available to my team. From what I can tell, this option is not immediately supported if one follows the sharing steps detailed here: [https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset](https://huggingface.co/docs/datasets/share_dataset.html#sharing-a-canonical-dataset), since GitHub LFS is not supported for public forks. Is there a way to request this? Thanks!
GitHub issue #3163: Add Image feature
State: closed | Created: 2021-10-25T19:07:48Z

Adds the Image feature. This feature is heavily inspired by the recently added Audio feature (#2324). Currently, this PR is pretty simple.

Some considerations that need further discussion:
* I've decided to use `Pillow`/`PIL` as the image decoding library. Another candidate I considered is `torchvision`, mostly because of its `accimage` backend, which should be faster for loading `jpeg` images than `Pillow`. However, `torchvision`'s io module only supports png and jpeg images, has `torch` as a hard dependency, and requires magic to work with image bytes ( `torch.ByteTensor(torch.ByteStorage.from_buffer(image_bytes)))`).
* Currently, I'm converting `PIL`'s `Image` type to `np.ndarray`. The vision models in Transformers such as ViT prefer the raw `Image` type and not the decoded tensors, so there is a small overhead due to [this conversion](https://github.com/huggingface/transformers/blob/3e8761ab8077e3bb243fe2f78b2a682bd2257cf1/src/transformers/image_utils.py#L62-L73). IMO this is justified to keep this part aligned with the Audio feature, which also returns `np.ndarray`. What do you think?
* Still have to work on the channel decoding logic:
    * PyTorch prefers the channel-first ordering (C, H, W); TF and Flax the channel-last ordering (H, W, C). One cool feature would be adjusting the channel order based on the selected formatter (`torch`, `tf`, `jax`). 
    * By default, `Image.open` returns images of shape (H, W, C). However, ViT's feature extractor expects the format (C, H, W) if the image is passed as an array (explained [here](https://huggingface.co/transformers/model_doc/vit.html#transformers.ViTFeatureExtractor.__call__)), so I'm more inclined to the format (C, H, W). Which one do you prefer, (C, H, W) or (H, W, C)?
* Are there any options you'd like to see? (the user could change those via `cast_column`, such as `sampling_rate` in the Audio feature)


TODOs:
* [x] tests
* in subsequent PRs:
  * docs - a section in the docs, which gives some additional info on the Image and Audio feature and compares them to 
  `ArrayND`  
  * streaming (waiting for #3129 and #3133 to get merged first)
  * update the image tasks and the datasets to use the new feature
  * Image/Audio formatting

[Colab Notebook](https://colab.research.google.com/drive/1mIrTnqTVkWLJWoBzT1ABSe-LFelIep1c?usp=sharing) where you can play with this feature.

I'm also adding a link to the [Image](https://github.com/tensorflow/datasets/blob/7ac7d506488d46038a5854961d068926b3f93c7f/tensorflow_datasets/core/features/image_feature.py#L155) feature in TFDS because one of our goals is to parse TFDS scripts eventually, so our Image feature has to (at least) support all the formats theirs does.
Feel free to cc anyone who might be interested.

P.S. Please ignore the changes in the `datasets/**/*.py` files 😄.
GitHub issue #3162: `datasets-cli test` should work with datasets without scripts
State: open | Created: 2021-10-25T18:52:30Z

It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached  to them (whether the datasets are private or not).

I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!
GitHub issue #3160: Better error msg if `len(predictions)` doesn't match `len(references)` in metrics
State: closed | Created: 2021-10-25T15:25:05Z

Improve the error message in `Metric.add_batch` if `len(predictions)` doesn't match `len(references)`.

cc: @BramVanroy (feel free to test this code on your examples and review this PR)
GitHub issue #3158: Fix string encoding for Value type
State: closed | Created: 2021-10-25T13:44:13Z

Some metrics have `string` features but currently it fails if users pass integers instead. Indeed feature encoding that handles the conversion of the user's objects to the right python type is missing a case for `string`, while it already works as expected for integers, floats and booleans

Here is an example code that didn't work previously, but that works with this fix:
```python
import datasets

# Note that 'id' is an integer while the SQuAD metric uses strings
predictions = [{'prediction_text': '1976', 'id': 5}]
references = [{'answers': {'answer_start': [97], 'text': ['1976']}, 'id': 5}] 

squad_metric = datasets.load_metric("squad") 
squad_metric.add_batch(predictions=predictions, references=references) 
results = squad_metric.compute()
# {'exact_match': 100.0, 'f1': 100.0}
```

cc @sgugger @philschmid
GitHub issue #3155: Illegal instruction (core dumped) at datasets import
State: closed | Created: 2021-10-24T17:21:36Z

## Describe the bug
I install datasets using conda and when I import datasets I get: "Illegal instruction (core dumped)"

## Steps to reproduce the bug

```
conda create --prefix path/to/env
conda activate path/to/env
conda install -c huggingface -c conda-forge datasets
# exits with output "Illegal instruction (core dumped)"
python -m datasets
```

## Environment info

When I run "datasets-cli env", I also get "Illegal instruction (core dumped)"

If I run the following commands:

```
conda create --prefix path/to/another/new/env
conda activate path/to/another/new/env
conda install -c huggingface transformers
transformers-cli env
```

Then I get:

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-67-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Let me know what additional information you need in order to debug this issue. Thanks in advance!
GitHub issue #3154: Sacrebleu unexpected behaviour/requirement for data format
State: closed | Created: 2021-10-24T08:55:33Z

## Describe the bug
When comparing with the original `sacrebleu` implementation, the `datasets` implementation does some strange things that I do not quite understand. This issue was triggered when I was trying to implement TER and found the datasets implementation of BLEU [here](https://github.com/huggingface/datasets/pull/3153).

In the below snippet, the original sacrebleu snippet works just fine whereas the datasets implementation throws an error.

## Steps to reproduce the bug
```python
import sacrebleu
import datasets

refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'],
]

hyps = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

expected_bleu = 48.530827

ds_bleu = datasets.load_metric("sacrebleu")

bleu_score_sb = sacrebleu.corpus_bleu(hyps, refs).score
print(bleu_score_sb, expected_bleu)
# works: 48.5308...
bleu_score_ds = ds_bleu.compute(predictions=hyps, references=refs)["score"]
print(bleu_score_ds, expected_bleu)
# ValueError: Predictions and/or references don't match the expected format.
```
This seems to be related to how datasets forces the features format here:

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L94-L99

and then manipulates the references during the compute stage here

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/metrics/sacrebleu/sacrebleu.py#L119-L122

I do not quite understand why that is required since sacrebleu handles argument parsing quite well [by itself](https://github.com/mjpost/sacrebleu/blob/2787185dd0f8d224c72ee5a831d163c2ac711a47/sacrebleu/metrics/base.py#L229). 
## Actual results
Traceback (most recent call last):
  File "C:\Users\bramv\AppData\Roaming\JetBrains\PyCharm2020.3\scratches\scratch_23.py", line 23, in <module>
    bleu_score_ds = ds_bleu.compute(predictions=hyps, references=refs)["score"]
  File "C:\dev\python\datasets\src\datasets\metric.py", line 392, in compute
    self.add_batch(predictions=predictions, references=references)
  File "C:\dev\python\datasets\src\datasets\metric.py", line 439, in add_batch
    raise ValueError(
ValueError: Predictions and/or references don't match the expected format.
Expected format: {'predictions': Value(dtype='string', id='sequence'), 'references': Sequence(feature=Value(dtype='string', id='sequence'), length=-1, id='references')},
Input predictions: ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.'],
Input references: [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'], ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2
- PyArrow version: 4.0.1
GitHub issue #3153: Add TER (as implemented in sacrebleu)
State: closed | Created: 2021-10-23T14:26:45Z

Implements TER (Translation Edit Rate) as per its implementation in sacrebleu. Sacrebleu for BLEU scores is already implemented in `datasets` so I thought this would be a nice addition.

I started from the sacrebleu implementation, as the two metrics have a lot in common.

Verified with sacrebleu's [testing suite](https://github.com/mjpost/sacrebleu/blob/078c440168c6adc89ba75fe6d63f0d922d42bcfe/test/test_ter.py) that this indeed works as intended.

```python
import datasets


test_cases = [
    (['aaaa bbbb cccc dddd'], ['aaaa bbbb cccc dddd'], 0),  # perfect match
    (['dddd eeee ffff'], ['aaaa bbbb cccc'], 1),  # no overlap
    ([''], ['a'], 1),  # corner case, empty hypothesis
    (['d e f g h a b c'], ['a b c d e f g h'], 1 / 8),  # a single shift fixes MT
    (
        [
            'wählen Sie " Bild neu berechnen , " um beim Ändern der Bildgröße Pixel hinzuzufügen oder zu entfernen , damit das Bild ungefähr dieselbe Größe aufweist wie die andere Größe .',
            'wenn Sie alle Aufgaben im aktuellen Dokument aktualisieren möchten , wählen Sie im Menü des Aufgabenbedienfelds die Option " Alle Aufgaben aktualisieren . "',
            'klicken Sie auf der Registerkarte " Optionen " auf die Schaltfläche " Benutzerdefiniert " und geben Sie Werte für " Fehlerkorrektur-Level " und " Y / X-Verhältnis " ein .',
            'Sie können beispielsweise ein Dokument erstellen , das ein Auto über die Bühne enthält .',
            'wählen Sie im Dialogfeld " Neu aus Vorlage " eine Vorlage aus und klicken Sie auf " Neu . "',
        ],
        [
            'wählen Sie " Bild neu berechnen , " um beim Ändern der Bildgröße Pixel hinzuzufügen oder zu entfernen , damit die Darstellung des Bildes in einer anderen Größe beibehalten wird .',
            'wenn Sie alle Aufgaben im aktuellen Dokument aktualisieren möchten , wählen Sie im Menü des Aufgabenbedienfelds die Option " Alle Aufgaben aktualisieren . "',
            'klicken Sie auf der Registerkarte " Optionen " auf die Schaltfläche " Benutzerdefiniert " und geben Sie für " Fehlerkorrektur-Level " und " Y / X-Verhältnis " niedrigere Werte ein .',
            'Sie können beispielsweise ein Dokument erstellen , das ein Auto enthalt , das sich über die Bühne bewegt .',
            'wählen Sie im Dialogfeld " Neu aus Vorlage " eine Vorlage aus und klicken Sie auf " Neu . "',
        ],
        0.136  # realistic example from WMT dev data (2019)
    ),
]

ter = datasets.load_metric(r"path\to\datasets\metrics\ter")

predictions = ["hello there general kenobi", "foo bar foobar"]
references = [["hello there general kenobi", "hello there !"], ["foo bar foobar", "foo bar foobar"]]
print(ter.compute(predictions=predictions, references=references))

for hyp, ref, score in test_cases:
    # Note the reference transformation which is different from scarebleu's input format
    results = ter.compute(predictions=hyp, references=[[r] for r in ref])
    assert 100*score == results["score"], f"expected {100*score}, got {results['score']}"
```
GitHub issue #3151: Re-add faiss to windows testing suite
State: closed | Created: 2021-10-22T19:34:29Z

In recent versions, `faiss-cpu` seems to be available for Windows as well. See the [PyPi page](https://pypi.org/project/faiss-cpu/#files) to confirm. We can therefore included it for Windows in the setup file.

At first tests didn't pass due to problems with permissions as caused by `NamedTemporaryFile` on Windows. This built-in library is notoriously poor in playing nice on Windows. The required change isn't pretty, but it works. First set `delete=False` to not automatically try to delete the file on `exit`. Then, manually delete the file with `unlink`. It's weird, I know, but it works.

```python
with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
    # do stuff
os.unlink(tmp_file.name)
```

closes #3150
GitHub issue #3150: Faiss _is_ available on Windows
State: closed | Created: 2021-10-22T18:07:16Z

In the setup file, I find the following:

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/setup.py#L171

However, FAISS does install perfectly fine on Windows on my system. You can also confirm this on the [PyPi page](https://pypi.org/project/faiss-cpu/#files), where Windows wheels are available. Maybe this was true for older versions? For current versions, this can be removed I think.

(This isn't really a bug but didn't know how else to tag.)

If you agree I can do a quick PR and remove that line.
GitHub issue #3149: Add CMU Hinglish DoG Dataset for MT
State: closed | Created: 2021-10-22T16:17:25Z

Address part of #2841 

Added the CMU Hinglish DoG Dataset as in GLUECoS. Added it as a seperate dataset as unlike other tasks of GLUE CoS this can't be evaluated for a BERT like model. 
Consists of parallel dataset between Hinglish (Hindi-English) and English, can be used for Machine Translation between the two. 

The data processing part is inspired from the GLUECoS repo [here](https://github.com/microsoft/GLUECoS/blob/7fdc51653e37a32aee17505c47b7d1da364fa77e/Data/Preprocess_Scripts/preprocess_mt_en_hi.py)
The dummy data part is not working properly, it shows 
``` UnboundLocalError: local variable 'generator_splits' referenced before assignment ``` 
when I run without ``--auto_generate``.

Please let me know how I can fix that.
Thanks
GitHub issue #3148: Streaming with num_workers != 0
State: closed | Created: 2021-10-22T15:07:17Z

## Describe the bug
When using dataset streaming with pytorch DataLoader, the setting num_workers to anything other than 0 causes the code to freeze forever before yielding the first batch.

The code owner is likely @lhoestq 

## Steps to reproduce the bug

For your convenience, we've prepped a colab notebook that reproduces the bug
https://colab.research.google.com/drive/1Mgl0oTZSNIE3UeGl_oX9wPCOIxRg19h1?usp=sharing
```python
!pip install datasets==1.14.0

should_freeze_forever = True
# ^-- set this to True in order to freeze forever, set to False in order to work normally

import torch
from datasets import load_dataset

data = load_dataset("oscar", "unshuffled_deduplicated_bn", split="train", streaming=True)
data = data.map(lambda x: {"text": x["text"], "orig": f"oscar[{x['id']}]"}, batched=True)
data = data.shuffle(100, seed=1337)

data = data.with_format("torch")
loader = torch.utils.data.DataLoader(data, batch_size=2, num_workers=2 if should_freeze_forever else 0)

# v-- the code should freeze forever at this line
for i, row in enumerate(loader):
  print(row)
  if i > 10: break
print("DONE!")
```

## Expected results
The code should not freeze forever with num_workers=2

## Actual results
The code freezes forever with num_workers=2

## Environment info
- `datasets` version: 1.14.0 (also found in previous versions)
- Platform: google colab (also locally)
- Python version: 3.7, (also 3.8)
- PyArrow version: 3.0.0
GitHub issue #3146: CLI test command throws NonMatchingSplitsSizesError when saving infos
State: closed | Created: 2021-10-22T13:50:53Z

When trying to generate a datset JSON metadata, a `NonMatchingSplitsSizesError` is thrown:
```
$ datasets-cli test datasets/arabic_billion_words --save_infos --all_configs
Testing builder 'Alittihad' (1/10)
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: Unknown size, post-processed: Unknown size, total: 332.13 MiB) to .cache\arabic_billion_words\Alittihad\1.1.0\8175ff1c9714c6d5d15b1141b6042e5edf048276bb81a9c14e35e149a7a62ae4...
Traceback (most recent call last):
  File "path\huggingface\datasets\.venv\Scripts\datasets-cli-script.py", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "path\huggingface\datasets\src\datasets\commands\datasets_cli.py", line 33, in main
    service.run()
  File "path\huggingface\datasets\src\datasets\commands\test.py", line 144, in run
    builder.download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "path\huggingface\datasets\src\datasets\utils\info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='arabic_billion_words'), 'recorded': SplitInfo(name='train', num_bytes=1601790302, num_examples=349342, dataset_name='arabic_billion_words')}]
```

This is due because a previous run generated a wrong `dataset_info.json`.

This error can be avoided by passing `--ignore_verifications`, but I think this should be assumed when passing `--save_infos`.
GitHub issue #3145: [when Image type will exist] provide a way to get the data as binary + filename
State: closed | Created: 2021-10-22T13:23:49Z

**Is your feature request related to a problem? Please describe.**

When a dataset cell contains a value of type Image (be it from a remote URL, an Array2D/3D, or any other way to represent images), I want to be able to write the image to the disk, with the correct filename, and optionally to know its mimetype, in order to serve it on the web.

Note: this issue would apply exactly the same for the `Audio` type.

**Describe the solution you'd like**

If a "cell" has the type `Image`, provide a way to get the binary content of the file, and the filename, eg as:

```python
  filename: str
  data: bytes
```

**Describe alternatives you've considered**

A way to write the cell to the disk (passing a local directory), and then return the pathname, filename, and mimetype.
GitHub issue #3144: Infer the features if missing
State: closed | Created: 2021-10-22T13:17:33Z

**Is your feature request related to a problem? Please describe.**

Some datasets, in particular community datasets, have no info file, thus no features.

**Describe the solution you'd like**

If a dataset has no features, the first loaded data (5-10 rows) could be used to infer the type.

Related: `datasets` would provide a way to load the data, and get the rows AND the features as the result.

**Describe alternatives you've considered**

The HF hub could also provide some UI to help the dataset maintainers to explicit the types of their rows, or automatically infer them as an initial proposal.
GitHub issue #3143: Provide a way to check if the features (in info) match with the data of a split
State: open | Created: 2021-10-22T13:13:36Z

**Is your feature request related to a problem? Please describe.**

I understand that currently the data loaded has not always the type described in the info features

**Describe the solution you'd like**

Provide a way to check if the rows have the type described by info features

**Describe alternatives you've considered**

Always check it, and raise an error when loading the data if their type doesn't match the features.
GitHub issue #3142: Provide a way to write a streamed dataset to the disk
State: open | Created: 2021-10-22T13:09:53Z

**Is your feature request related to a problem? Please describe.**

The streaming mode allows to get the 100 first rows of a dataset very quickly. But it does not cache the answer, so a posterior call to get the same 100 rows will send a request to the server again and again.

**Describe the solution you'd like**

Provide a way to write the streamed rows of a dataset on the disk, and to load from it later.

**Describe alternatives you've considered**

Provide a third mode: `lazy`, which would use the local cache for the data that have already been fetched previously, and use streaming to get the rest of the requested data.
GitHub issue #3139: Fix file/directory deletion on Windows  
State: open | Created: 2021-10-22T12:22:08Z

Currently, on Windows, some attempts to delete a dataset file/directory will fail with the `PerimissionError`. 

Examples:
- download a dataset, then force redownload it in the same session while keeping a reference to the downloaded dataset
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset = load_dataset("sst", split="train", download_mode="force_redownload")
```
- try to clean up the cache files while keeping a reference to those files (via the mapped dataset):
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset_mapped = dset.map(lambda _: {"dummy_col": 1})
dset.cleanup_cache_files()
```
We should fix those.
GitHub issue #3138: More fine-grained taxonomy of error types
State: open | Created: 2021-10-22T09:35:29Z

**Is your feature request related to a problem? Please describe.**

Exceptions like `FileNotFoundError` can be raised by different parts of the code, and it's hard to detect which one did

**Describe the solution you'd like**

Give a specific exception type for every group of similar errors

**Describe alternatives you've considered**

Rely on the error message, using regex
GitHub issue #3137: Fix numpy deprecation warning for ragged tensors
State: closed | Created: 2021-10-22T09:17:46Z

Numpy shows a deprecation warning when we call `np.array` on a list of ragged tensors without specifying the `dtype`. If their shapes match, the tensors can be collated together, otherwise the resulting array should have `dtype=np.object`.

Fix #3084 
cc @Rocketknight1
GitHub issue #3136: Fix script of Arabic Billion Words dataset to return all data
State: closed | Created: 2021-10-22T09:14:24Z

The script has a bug and only parses and generates a portion of the entire dataset.

This PR fixes the loading script so that is properly parses the entire dataset. 

Current implementation generates the same number of examples as reported in the [original paper](https://arxiv.org/abs/1611.04033) for all configurations except for one:
- For "Youm7" we generate more examples (1172136) than the ones reported by the paper (1025027)

|                | Number of examples | Number of examples according to the source |
|:---------------|-------------------:|-----:|
| Alittihad      |             349342 |349342 |
| Almasryalyoum  |             291723 |291723 |
| Almustaqbal    |             446873 |446873 |
| Alqabas        |             817274 |817274 |
| Echoroukonline |             139732 |139732 |
| Ryiadh         |             858188 | 858188 |
| Sabanews       |              92149 |92149 |
| SaudiYoum      |             888068 |888068 |
| Techreen       |             314597 |314597 |
| Youm7          |            1172136 |1025027 |

Fix #3126.
GitHub issue #3135: Make inspect.get_dataset_config_names always return a non-empty list of configs
State: closed | Created: 2021-10-22T08:02:50Z

**Is your feature request related to a problem? Please describe.**

Currently, some datasets have a configuration, while others don't. It would be simpler for the user to always have configuration names to refer to

**Describe the solution you'd like**

In that sense inspect.get_dataset_config_names should always return at least one configuration name, be it `default` or `Check___region_1` (for community datasets like `Check/region_1`).

https://github.com/huggingface/datasets/blob/c5747a5e1dde2670b7f2ca6e79e2ffd99dff85af/src/datasets/inspect.py#L161
GitHub issue #3134: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py
State: closed | Created: 2021-10-22T07:07:52Z

datasets version: 1.12.1

`metric = datasets.load_metric('rouge')`

The error:

> ConnectionError                           Traceback (most recent call last)
> <ipython-input-3-dd10a0c5212f> in <module>
> ----> 1 metric = datasets.load_metric('rouge')
> 
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in load_metric(path, config_name, process_id, num_process, cache_dir, experiment_id, keep_in_memory, download_config, download_mode, script_version, **metric_init_kwargs)
>     613         download_config=download_config,
>     614         download_mode=download_mode,
> --> 615         dataset=False,
>     616     )
>     617     metric_cls = import_main_class(module_path, dataset=False)
> 
> /usr/local/lib/python3.6/dist-packages/datasets/load.py in prepare_module(path, script_version, download_config, download_mode, dataset, force_local_path, dynamic_modules_path, return_resolved_file_path, **download_kwargs)
>     328                 file_path = hf_github_url(path=path, name=name, dataset=dataset, version=script_version)
>     329                 try:
> --> 330                     local_path = cached_path(file_path, download_config=download_config)
>     331                 except FileNotFoundError:
>     332                     if script_version is not None:
> 
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in cached_path(url_or_filename, download_config, **download_kwargs)
>     296             use_etag=download_config.use_etag,
>     297             max_retries=download_config.max_retries,
> --> 298             use_auth_token=download_config.use_auth_token,
>     299         )
>     300     elif os.path.exists(url_or_filename):
> 
> /usr/local/lib/python3.6/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
>     603             raise FileNotFoundError("Couldn't find file at {}".format(url))
>     604         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
> --> 605         raise ConnectionError("Couldn't reach {}".format(url))
>     606
>     607     # Try a second time
> 
> ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.11.0/metrics/rouge/rouge.py


Is there any remedy to solve the connection issue ?
GitHub issue #3131: Add ADE20k
State: closed | Created: 2021-10-21T10:13:09Z

## Adding a Dataset
- **Name:** ADE20k (actually it's called the MIT Scene Parsing Benchmark, it's actually a subset of ADE20k but a lot of authors still call it ADE20k)
- **Description:** A semantic segmentation dataset, consisting of 150 classes.
- **Paper:** http://people.csail.mit.edu/bzhou/publication/scene-parse-camera-ready.pdf
- **Data:** http://sceneparsing.csail.mit.edu/
- **Motivation:** I am currently adding Transformer-based semantic segmentation models that achieve SOTA on this dataset. It would be great to directly access this dataset using HuggingFace Datasets, in order to make example scripts in HuggingFace Transformers.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
GitHub issue #3127: datasets-cli: convertion of a tfds dataset to a huggingface one.
State: open | Created: 2021-10-21T06:14:27Z

### Discussed in https://github.com/huggingface/datasets/discussions/3079

<div type='discussions-op-text'>

<sup>Originally posted by **vitalyshalumov** October 14, 2021</sup>
I'm trying to convert a tfds dataset to a huggingface one.

I've tried:

1.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/mnist/3.0.1/

2.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/


and other permutations.
The script appears to be running and finishing without an error but when looking in the  huggingface/datasets/ folder nothing is created.


</div>
GitHub issue #3126: "arabic_billion_words" dataset does not create the full dataset
State: closed | Created: 2021-10-21T06:02:38Z

## Describe the bug
When running: 
raw_dataset = load_dataset('arabic_billion_words','Alittihad')
the correct dataset file is pulled from the url.
But, the generated dataset includes just a small portion of the data included in the file.
This is true for all other portions of the "arabic_billion_words" dataset ('Almasryalyoum',.....)

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
raw_dataset = load_dataset('arabic_billion_words','Alittihad')

#The screen message
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: 20.62 MiB, post-processed: Unknown size, total: 352.74 MiB) 

## Expected results
over 100K sentences

## Actual results
only 11K sentences

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.8.0-63-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 4.0.1
GitHub issue #3124: More efficient nested features encoding
State: closed | Created: 2021-10-21T01:55:31Z

Nested encoding of features wastes a lot of time on operations which are effectively doing nothing when lists are used.
For example, if in the input we have a list of integers, `encoded_nested_example` will iterate over it and apply `encoded_nested_example` on every element even though it just return the int as is.

A similar issue is handled at an earlier stage when casting pytorch/tensorflow/pandas objects to python lists/numpy arrays:
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L149-L156
https://github.com/huggingface/datasets/blob/c98c23c4260edadab00f997d1a5d66b7f2e93ce9/src/datasets/features/features.py#L212-L228

In this pull request I suggest to use the same approach in `encoded_nested_example`.
In my setup there was a major speedup with this change: loading the data was at least x4 faster.
GitHub issue #3123: Segmentation fault when loading datasets from file
State: closed | Created: 2021-10-20T20:16:11Z

## Describe the bug
Custom dataset loading sometimes segfaults and kills the process if chunks contain a variety of features/

## Steps to reproduce the bug

Download an example file:
```
wget https://gist.githubusercontent.com/TevenLeScao/11e2184394b3fa47d693de2550942c6b/raw/4232704d08fbfcaf93e5b51def9e5051507651ad/tiny_kelm.jsonl
```
Then in Python:
```
import datasets
tiny_kelm = datasets.load_dataset("json", data_files="tiny_kelm.jsonl", chunksize=100000)
```

## Expected results
a `tiny_kelm` functional dataset

## Actual results
☠️ `Segmentation fault (core dumped)` ☠️

## Environment info
- `datasets` version: 1.14.0
- Platform: Linux-5.11.0-38-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 5.0.0
GitHub issue #3122: OSError with a custom dataset loading script
State: closed | Created: 2021-10-20T20:08:39Z

## Describe the bug
I am getting an OS error when trying to load the newly uploaded dataset classla/janes_tag. What puzzles me is that I have already uploaded a very similar dataset - classla/reldi_hr - with no issues. The loading scripts for the two datasets are almost identical and they have the same directory structure, yet I am only getting an error with janes_tag.

## Steps to reproduce the bug
```python
dataset = datasets.load_dataset('classla/janes_tag', split='validation')
```

## Expected results
Dataset correctly loaded.

## Actual results

Traceback (most recent call last):
  File "C:/mypath/test.py", line 91, in <module>
    load_and_print('janes_tag')
  File "C:/mypath/test.py", line 32, in load_and_print
    dataset = datasets.load_dataset('classla/{}'.format(ds_name), split='validation')
  File "C:\mypath\venv\lib\site-packages\datasets\load.py", line 1632, in load_dataset
    use_auth_token=use_auth_token,
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 608, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "C:\mypath\venv\lib\site-packages\datasets\builder.py", line 704, in _download_and_prepare
    ) from None
OSError: Cannot find data file. 
Original error:
[Errno 2] No such file or directory: 'C:\\mypath\\.cache\\huggingface\\datasets\\downloads\\2c9996e44bdc5af9c89bffb9e6d7a3e42fdb2f56bacab45de13b20f3032ea7ca\\data\\train_all.conllup'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.5
- PyArrow version: 3.0.0
GitHub issue #3121: Use huggingface_hub.HfApi to list datasets/metrics
State: closed | Created: 2021-10-20T17:48:29Z

Delete `datasets.inspect.HfApi` and use `huggingface_hub.HfApi` instead.

WIP until https://github.com/huggingface/huggingface_hub/pull/429 is merged, then wait for the new release of  `huggingface_hub`, update the `huggingface_hub` version in `setup.py` and merge this PR.

cc: @lhoestq