NyssaW/github-issues huggingface dataset:
FileNotFoundError for TupleIE dataste

Hi,
`dataset = datasets.load_dataset('tuple_ie', 'all')`

returns a FileNotFound error. Is the data not available? 

Many thanks.


Comments:
@mariosasko @lhoestq Could you give me an update on how to load the dataset after the fix?
Thanks.

Hi @arda-vianai,

first, you can try:
```python
import datasets
dataset = datasets.load_dataset('tuple_ie', 'all', revision="master")
```
If this doesn't work, your version of `datasets` is missing some features that are required to run the dataset script, so install the master version with the following command:
```
pip install git+https://github.com/huggingface/datasets.git
```
and then:
```python
import datasets
dataset = datasets.load_dataset('tuple_ie', 'all')
```
should work (even without `revision`).

@mariosasko 
Thanks, it is working now. I actually did that before but I didn't restart the kernel. I restarted it and it works now. My bad!!!
Many thanks and great job!
-arda
Updated: DaNE - updated URL for download

It seems that DaNLP has updated their download URLs and it therefore also needs to be updated in here...

Comments:
Actually it looks like the old URL is still working, and it's also the one that is mentioned in https://github.com/alexandrainst/danlp/blob/master/docs/docs/datasets.md

What makes you think we should use the new URL ?

@lhoestq Sorry! I might have jumped to conclusions a bit too fast here... 

I was working in Google Colab and got an error that it was unable to use the URL. I then forked the project, updated the URL, ran it locally and it worked. I therefore assumed that my URL update fixed the issue, however, I see now that it might rather be a Google Colab issue... 

Still - this seems to be the official URL for downloading the dataset, and I think that it will be most beneficial to use. :-) 

It looks like they're using these new urls for their new datasets. Maybe let's change to the new URL in case the old one stops working at one point. Thanks
Add mIoU metric

**Is your feature request related to a problem? Please describe.**
Recently, some semantic segmentation models were added to HuggingFace Transformers, including [SegFormer](https://huggingface.co/transformers/model_doc/segformer.html) and [BEiT](https://huggingface.co/transformers/model_doc/beit.html).

Semantic segmentation (which is the task of labeling every pixel of an image with a corresponding class) is typically evaluated using the Mean Intersection and Union (mIoU). Together with the upcoming Image Feature, adding this metric could be very handy when creating example scripts to fine-tune any Transformer-based model on a semantic segmentation dataset.

An implementation can be found [here](https://github.com/open-mmlab/mmsegmentation/blob/504965184c3e6bc9ec43af54237129ef21981a5f/mmseg/core/evaluation/metrics.py#L132) for instance.


Comments:
Resolved via https://github.com/huggingface/datasets/pull/3745.
Add GSM8K dataset

## Adding a Dataset
- **Name:** GSM8K (short for Grade School Math 8k)
- **Description:** GSM8K is a dataset of 8.5K high quality linguistically diverse grade school math word problems created by human problem writers.
- **Paper:** https://openai.com/blog/grade-school-math/
- **Data:** https://github.com/openai/grade-school-math
- **Motivation:** The dataset is useful to investigate the reasoning abilities of large Transformer models, such as GPT-3.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Closed via https://github.com/huggingface/datasets/pull/4103
Multiprocessing filter/map (tests) not working on Windows

While running the tests, I found that the multiprocessing examples fail on Windows, or rather they do not complete: they cause a deadlock. I haven't dug deep into it, but they do not seem to work as-is. I currently have no time to tests this in detail but at least the tests seem not to run correctly (deadlocking).

## Steps to reproduce the bug
```shell
pytest tests/test_arrow_dataset.py -k "test_filter_multiprocessing"
pytest tests/test_arrow_dataset.py -k "test_map_multiprocessing"
```

## Expected results
The functionality to work on all platforms.

## Actual results
Deadlock.

## Environment info
- `datasets` version: 1.14.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2, also tested with 3.7.9
- PyArrow version: 4.0.1



Comments:
Dataset viewer issue for '*compguesswhat*'

## Dataset viewer issue for '*compguesswhat*'

**Link:** https://huggingface.co/datasets/compguesswhat

File not found

Am I the one who added this dataset ? No


Comments:
```python
>>> import datasets
>>> dataset = datasets.load_dataset('compguesswhat', name='compguesswhat-original',split='train', streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/compguesswhat/4d08b9e0a8d1cf036c9626c93be4a759fdd9fcce050ea503ea14b075e830c799/compguesswhat.py", line 251, in _generate_examples
    with gzip.open(filepath) as in_file:
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/gzip.py", line 58, in open
    binary_file = GzipFile(filename, gz_mode, compresslevel)
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/gzip.py", line 173, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'zip://compguesswhat-original/0.2.0/compguesswhat.train.jsonl.gz::https://www.dropbox.com/s/l0nc13udml6vs0w/compguesswhat-original.zip?dl=1'
```

It's an issue with the streaming mode. Note that normal mode is used by the dataset viewer when streaming is failing, but only for the smallest datasets. This dataset is above the limit, hence the error.

Same case as https://github.com/huggingface/datasets/issues/3186#issuecomment-1096549774.

cc @huggingface/datasets 

There is an issue with the URLs of their data files: https://www.dropbox.com/s/l0nc13udml6vs0w/compguesswhat-original.zip?dl=1
> Dropbox Error: That didn't work for some reason

Error reported to their repo:
- https://github.com/CompGuessWhat/compguesswhat.github.io/issues/1

Closed by:
- #4968
combination of shuffle and filter results in a bug

## Describe the bug
Hi,
I would like to shuffle a dataset, then filter it based on each existing label. however, the combination of `filter`, `shuffle` seems to results in a bug. In the minimal example below, as you see in the filtered results, the filtered labels are not unique, meaning filter has not worked. Any suggestions as a temporary fix is appreciated @lhoestq.

Thanks.
 Best regards
Rabeeh 

## Steps to reproduce the bug
```python
import numpy as np
import datasets 

datasets = datasets.load_dataset('super_glue', 'rte', script_version="master")
shuffled_data = datasets["train"].shuffle(seed=42)
for label in range(2):
    print("label ", label)
    data = shuffled_data.filter(lambda example: int(example['label']) == label)
    print("length ", len(data), np.unique(data['label']))
```

## Expected results
Filtering per label, should only return the data with that specific label.

## Actual results
As you can see, filtered data per label, has still two labels of [0, 1]
```
label  0
length  1249 [0 1]
label  1
length  1241 [0 1]
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1 
- Platform: linux 
- Python version: 3.7.11 
- PyArrow version: 5.0.0 


Comments:
I cannot reproduce this on master and pyarrow==4.0.1.


Hi ! There was a regression in `datasets` 1.12 that introduced this bug. It has been fixed in #3019 in 1.13

Can you try to update `datasets` and try again ?

Thanks a lot, fixes with 1.13
conll2003 incorrect label explanation

In the [conll2003](https://huggingface.co/datasets/conll2003#data-fields) README, the labels are described as follows

> - `id`: a `string` feature.
> - `tokens`: a `list` of `string` features.
> - `pos_tags`: a `list` of classification labels, with possible values including `"` (0), `''` (1), `#` (2), `$` (3), `(` (4).
> - `chunk_tags`: a `list` of classification labels, with possible values including `O` (0), `B-ADJP` (1), `I-ADJP` (2), `B-ADVP` (3), `I-ADVP` (4).
> - `ner_tags`: a `list` of classification labels, with possible values including `O` (0), `B-PER` (1), `I-PER` (2), `B-ORG` (3), `I-ORG` (4) `B-LOC` (5), `I-LOC` (6) `B-MISC` (7), `I-MISC` (8).

First of all, it would be great if we can get a list of ALL possible pos_tags.

Second, the chunk tags labels cannot be correct. The description says the values go from 0 to 4 whereas the data shows values from at least 11 to 21 and 0.

EDIT: not really a bug, sorry for mistagging.

Comments:
Hi @BramVanroy,

since these fields are of type `ClassLabel` (you can check this with `dset.features`), you can inspect the possible values with:
```python
dset.features[field_name].feature.names  # .feature because it's a sequence of labels
```

and to find the mapping between names and integers, use: 
```python
dset.features[field_name].feature.int2str(value_or_values_list)  # map integer value to string value
# or
dset.features[field_name].feature.str2int(value_or_values_list)  # map string value to integer value
```
conll2002 issues

**Link:** https://huggingface.co/datasets/conll2002

The dataset viewer throws a server error when trying to preview the dataset. 

```
Message:       Extraction protocol 'train' for file at 'https://raw.githubusercontent.com/teropa/nlp/master/resources/corpora/conll2002/esp.train' is not implemented yet
```

In addition, the "point of contact" has encoding issues and does not work when clicked.

Am I the one who added this dataset ? No, @lhoestq  did

Comments:
Hi ! Thanks for reporting :)

This is related to https://github.com/huggingface/datasets/issues/2742, I'm working on it. It should fix the viewer for around 80 datasets.


Ah, hadn't seen that sorry.

The scrambled "point of contact" is a separate issue though, I think.

@lhoestq The "point of contact" is still an issue.

It will be fixed in https://github.com/huggingface/datasets/pull/3274, thanks
Dataset viewer for nli_tr

## Dataset viewer issue for '*nli_tr*'

**Link:** https://huggingface.co/datasets/nli_tr

Hello,

Thank you for the new dataset preview feature that will help the users to view the datasets online.

We just noticed that the dataset viewer widget in the `nli_tr` dataset shows the error below. The error must be due to a temporary problem that may have blocked access to the dataset through the dataset viewer. But the dataset is currently accessible through the link in the error message.   May we kindly ask if it would be possible to rerun the job so that it can access the dataset for the dataset viewer function?

Thank you.
Emrah


------------------------------------------
Server Error
Status code:   404
Exception:     FileNotFoundError
Message:       [Errno 2] No such file or directory: 'zip://snli_tr_1.0_train.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip
------------------------------------------

Am I the one who added this dataset ? Yes


Comments:
It's an issue with the streaming mode:

```python
>>> import datasets
>>> dataset = datasets.load_dataset('nli_tr', name='snli_tr',split='test', streaming=True)
>>> next(iter(dataset))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 497, in __iter__
    for key, example in self._iter():
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 494, in _iter
    yield from ex_iterable
  File "/home/slesage/hf/datasets-preview-backend/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 87, in __iter__
    yield from self.generate_examples_fn(**self.kwargs)
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/nli_tr/c2ddd0c0a70caddac6a81c2dae5ca7939f00060d517d08f1983927818dba6521/nli_tr.py", line 155, in _generate_examples
    with codecs.open(filepath, encoding="utf-8") as f:
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/codecs.py", line 905, in open
    file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: 'zip://snli_tr_1.0_test.jsonl::https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip'
```

Note that normal mode is used by the dataset viewer when streaming is failing, but only for the smallest datasets. `nli_tr` is above the limit, hence the error.

cc @huggingface/datasets 

Apparently there is an issue with the data source URLs: Server Not Found
- https://tabilab.cmpe.boun.edu.tr/datasets/nli_datasets/snli_tr_1.0.zip

We are contacting the authors to ask them: 
@e-budur you are one of the authors: are you aware of the issue with the URLs of your data ?

Reported to their repo:
- https://github.com/boun-tabi/NLI-TR/issues/9

The server issue was temporary and is now resolved.

Once we have implemented support for streaming, the viewer works: https://huggingface.co/datasets/nli_tr
7z dataset preview not implemented?

## Dataset viewer issue for dataset 'samsum'

**Link:** https://huggingface.co/datasets/samsum

Server Error
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol '7z' for file at 'https://arxiv.org/src/1911.12237v2/anc/corpus.7z' is not implemented yet


Comments:
It's a bug in the dataset viewer: the dataset cannot be downloaded in streaming mode, but since the dataset is relatively small, the dataset viewer should have fallback to normal mode. Working on a fix.

Fixed. https://huggingface.co/datasets/samsum/viewer/samsum/train

<img width="1563" alt="Capture d’écran 2022-04-12 à 13 47 45" src="https://user-images.githubusercontent.com/1676121/162953339-cd8922d7-9037-408b-b896-eac1af0bb54f.png">

Thanks for reporting!
RONEC v2

Hi, as we've recently finished with the new RONEC (Romanian Named Entity Corpus), we'd like to update the dataset here as well. It's actually essential as links to V1 are no longer valid. 

In reality we'd like to replace completely v1, as v2 is a full re-annotation of v1 with additional data (up to 2x size vs v1).

I've run the make style and all the dummy and real data test, and they passed.

I hope it's okay to merge the new RONEC v2 in the datasets. 

Thanks!

Comments:
@lhoestq Thanks for the review. I totally understand what you are saying. Normally, I would definitely agree with you, but in this particular case, the quality of v1 is poor, and the dataset itself is small (at the time we created v1 it was the only RO NER dataset, and its size was limited by the available resources). 

This is why we worked to build a larger one, with much better inter-annotator agreement. Fact is, models trained on v1 will be of very low quality and I would not recommend to anybody to use/do that. That's why I'd strongly suggest we replace v1 with v2, and kindof make v1 vanish :)  

What do you think? If you insist on having v1 accessible, I'll add the required code. Thanks!



Ok I see ! I think it's fine then, no need to re-add V1
Don't memoize strings when hashing since two identical strings may have different python ids

When hashing an object that has several times the same string, the hashing could return a different hash if the identical strings share the same python `id()` or not.

Here is an example code that shows how the issue can affect the caching:
```python
import json
import pyarrow as pa
from datasets.features import Features
from datasets.fingerprint import Hasher

schema = pa.schema([pa.field("some_string", pa.string()), pa.field("another_string", pa.string())])
features_from_schema = Features.from_arrow_schema(schema)
Hasher.hash(features_from_schema) # dffa9dca9a73fd8c

features_dict = json.loads('{"some_string": {"dtype": "string", "id": null, "_type": "Value"}, "another_string": {"dtype": "string", "id": null, "_type": "Value"}}')
features_from_json = Features.from_dict(features_dict)
Hasher.hash(features_from_json) # 3812e76b15e6420e

features_from_schema == features_from_json # True
```

This is because in `features_dict`, some strings like "dtype" are repeated but don't share the same id, contrary to the ones in `features_from_schema`.

I fixed that by disabling memoization for strings.

This could be optimized in the future by implementing a smarter memoization with a special handling for strings.

Comments:
This change slows down the hash computation a little bit but from my tests it doesn't look too impactful. So I think it's fine to merge this.
`None` converted to `"None"` when loading a dataset

## Describe the bug
When loading a dataset `None` values of the type `NoneType` are converted to `'None'` of the type `str`.

## Steps to reproduce the bug
```python
from datasets import load_dataset
qasper = load_dataset("qasper", split="train", download_mode="reuse_cache_if_exists")
print(qasper[60]["full_text"]["section_name"])
```

When installing version 1.1.40, the output is
`[None, 'Introduction', 'Benchmark Datasets', ...]`

When installing from the master branch, the output is
`['None', 'Introduction', 'Benchmark Datasets', ...]`

Notice how the first element was changed from `NoneType` to `str`.

## Expected results
`None` should stay as is.

## Actual results
`None` is converted to a string.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1


Comments:
Hi @eladsegal, thanks for reporting.

@mariosasko I saw you are already working on this, but maybe my comment will be useful to you.

All values are casted to their corresponding feature type (including `None` values). For example if the feature type is `Value("bool")`, `None` is casted to `False`.

It is true that strings were an exception, but this was recently fixed by @lhoestq (see #3158).

Thanks for reporting.

This is actually a breaking change that I think can cause issues when users preprocess their data. String columns used to be nullable. Maybe we can correct https://github.com/huggingface/datasets/pull/3158 to keep the None values and avoid this breaking change ?

EDIT: the other types (bool, int, etc) can also become nullable IMO

So what would be the best way to handle a feature that can have a null value in some of the instances? So far I used `None`.
Using the empty string won't be a good option, as it can be an actual value in the data and is not the same as not having a value at all.

Hi @eladsegal,

Use `None`. As @albertvillanova correctly pointed out, this change in conversion was introduced (by mistake) in #3158. To avoid it, install the earlier revision with:
```
pip install git+https://github.com/huggingface/datasets.git@8107844ec0e7add005db0585c772ee20adc01a5e
```

I'm making all the feature types nullable as we speak, and the fix will be merged probably early next week.

Hi @mariosasko, is there an estimation as to when this issue will be fixed?

https://github.com/huggingface/datasets/pull/3195 fixed it, we'll do a new release soon :)

For now feel free to install `datasets` from the master branch

Thanks, but unfortunately looks like it isn't fixed yet 😢 
[notebook for 1.14.0](https://colab.research.google.com/drive/1SV3sFXPJMWSQgbm4pr9Y1Q8OJ4JYKcDo?usp=sharing)
[notebook for master](https://colab.research.google.com/drive/145wDpuO74MmsuI0SVLcI1IswG6aHpyhi?usp=sharing)

Oh, sorry. I deleted the fix by accident when I was resolving a merge conflict. Let me fix this real quick.

Thank you, it works! 🎊
fix label mapping

Fixing label mapping for hlgd.
0 correponds to same event and 1 corresponds to different event
<img width="642" alt="Capture d’écran 2021-10-29 à 10 39 58 AM" src="https://user-images.githubusercontent.com/16107619/139454810-1f225e3d-ad48-44a8-b8b1-9205c9533839.png">
<img width="638" alt="Capture d’écran 2021-10-29 à 10 40 09 AM" src="https://user-images.githubusercontent.com/16107619/139454813-93066a3c-7d33-4f56-b133-2f1a7661e438.png">
nt

Comments:
heck, test failings. moving to draft. will come back to this later today hopefully

Thanks for fixing this :)
I just updated the dataset_infos.json and added the missing `pretty_name` tag to the dataset card

thank you @lhoestq! running around as always it felt through as a lower priority...
More control over TQDM when using map/filter with multiple processes

It would help with the clutter in my terminal if tqdm is only shown for rank 0 when using `num_proces>0` in the map and filter methods of datasets.

```python
dataset.map(lambda examples: tokenize(examples["text"]), batched=True, num_proc=6)
```

The above snippet leads to a lot of TQDM bars and depending on your terminal, these will not overwrite but keep pushing each other down.

```
 #0:   0%|          | 0/13 [00:00<?, ?ba/s]
 #1:   0%|          | 0/13 [00:00<?, ?ba/s]
 #2:   0%|          | 0/13 [00:00<?, ?ba/s]
 #3:   0%|          | 0/13 [00:00<?, ?ba/s]
 #4:   0%|          | 0/13 [00:00<?, ?ba/s]
 #5:   0%|          | 0/13 [00:00<?, ?ba/s]
 #0:   8%|          | 1/13 [00:00<?, ?ba/s]
 #1:   8%|          | 1/13 [00:00<?, ?ba/s]
...
```

Instead, it would be welcome if we had the option to only show the progress of rank 0.

Comments:
Hi,

It's hard to provide an API that would cover all use-cases with tqdm in this project.

However, you can make it work by defining a custom decorator (a bit hacky tho) as follows:
```python
import datasets

def progress_only_on_rank_0(func):
    def wrapper(*args, **kwargs):
        rank = kwargs.get("rank")
        disable_tqdm = kwargs.get("disable_tqdm", False)
        disable_tqdm = True if rank is not None and rank > 0 else disable_tqdm
        kwargs["disable_tqdm"] = disable_tqdm
        return func(*args, **kwargs)
    return wrapper
    
datasets.Dataset._map_single = progress_only_on_rank_0(datasets.Dataset._map_single)
``` 

EDIT: Ups, closed by accident.

Thanks for the provided links. `Trainer` requires this for training in multi-node distributed setting. However, `Dataset.map` doesn't support that yet.

Do you have an API for this in mind? `Dataset.map` is already bloated with the arguments, so IMO it's not a good idea to add a new arg there.



Inspiration may be found at `transformers`.

https://github.com/huggingface/transformers/blob/4a394cf53f05e73ab9bbb4b179a40236a5ffe45a/src/transformers/trainer.py#L1231-L1233

To get unique IDs for each worker, see https://stackoverflow.com/a/10192611/1150683
Add docs for `to_tf_dataset`

This PR adds some documentation for new features released in v1.13.0, with the main addition being `to_tf_dataset`:

- Show how to use `to_tf_dataset` in the tutorial, and move `set_format(type='tensorflow'...)` to the Process section (let me know if I'm missing anything @Rocketknight1 😅).
- Add an example for loading dataset from multiple zipped CSV files to the Load section.
- Add an example for removing columns for an `IterableDataset`.
- Add graphic for visualizing streaming.

Comments:
This looks great, thank you!

Thanks !

For some reason the new GIF is 6MB, which is a bit heavy for an image on a website. The previous one was around 200KB though which is perfect. For a good experience we usually expect images to be less than 500KB - otherwise for users with poor connection it takes too long to load. Could you try to reduce its size ? Than I think we can merge :)
Asserts replaced by exceptions (huggingface#3171)

I've replaced two asserts with their proper exceptions following the guidelines described in issue #3171 by following the contributing guidelines.

PS: This is one of my first PRs, hoping I don't break anything!

Comments:
Your first PR went smoothly, well done!
You are welcome to continue contributing to this project.
Gràcies, @joseporiolayats! 😉
Raise exceptions instead of using assertions for control flow

Motivated by https://github.com/huggingface/transformers/issues/12789 in Transformers, one welcoming change would be replacing assertions with proper exceptions. The only type of assertions we should keep are those used as sanity checks.

Currently, there is a total of 87 files with the `assert` statements (located under `datasets` and `src/datasets`), so when working on this, to manage the PR size, only modify 4-5 files at most before submitting a PR.

Comments:
Adding the remaining tasks for this issue to help new code contributors.  
$ cd src/datasets && ack assert -lc 
- [x] commands/convert.py:1
- [x] arrow_reader.py:3
- [x] load.py:7
- [x] utils/py_utils.py:2
- [x] features/features.py:9
- [x] arrow_writer.py:7
- [x] search.py:6
- [x] table.py:1
- [x] metric.py:3
- [x] tasks/image_classification.py:1
- [x] arrow_dataset.py:17
- [x] fingerprint.py:6
- [x] io/json.py:1
- [x] io/csv.py:1

Hi all,
I am interested in taking up `fingerprint.py`, `search.py`, `arrow_writer.py` and `metric.py`. Will raise a PR soon!

Let me look into `arrow_dataset.py`, `table.py`, `data_files.py` & `features.py` 

All the tasks are completed for this issue. This can be closed.
OpenSLR/83 is empty

## Describe the bug
As the summary says, openslr / SLR83 / train is empty.

The dataset returned after loading indicates there are **zero** rows. The correct number should be **17877**.
## Steps to reproduce the bug
```python
import datasets


datasets.load_dataset('openslr', 'SLR83')
```

## Expected results
```
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'sentence'],
        num_rows: 17877
    })
})
```
## Actual results
```
DatasetDict({
    train: Dataset({
        features: ['path', 'audio', 'sentence'],
        num_rows: 0
    })
})
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.1.dev0 (master HEAD)
- Platform: Ubuntu 20.04
- Python version: 3.7.10
- PyArrow version: 3.0.0


Comments:
Hi @tyrius02, thanks for reporting. I see you self-assigned this issue: are you working on this?

@albertvillanova Yes. Figured I introduced the broken config, I should fix it too.

I've got it working, but I'm struggling with one of the tests. I've started a PR so I/we can work through it.

Looks like the tests all passed on the PR.
bookcorpusopen no longer works

## Describe the bug

When using the latest version of datasets (1.14.0), I cannot use the `bookcorpusopen` dataset. The process blocks always around `9924 examples [00:06, 1439.61 examples/s]` when preparing the dataset. I also noticed that after half an hour the process is automatically killed because of the RAM usage (the machine has 1TB of RAM...).

This did not happen with 1.4.1.
I tried also `rm -rf ~/.cache/huggingface` but did not help.
Changing python version between 3.7, 3.8 and 3.9 did not help too.

## Steps to reproduce the bug
```python
import datasets
d = datasets.load_dataset('bookcorpusopen')
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.14.0
- Platform: Linux-5.4.0-1054-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1


Comments:
Hi ! Thanks for reporting :) I think #3280 should fix this

I tried with the latest changes from #3280 on google colab and it worked fine :)
We'll do a new release soon, in the meantime you can use the updated version with:
```python
load_dataset("bookcorpusopen", revision="master")
```

Fixed by #3280.
`datasets-cli test` should work with datasets without scripts

It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached  to them (whether the datasets are private or not).

I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!


Comments:
> It would be really useful to be able to run `datasets-cli test`for datasets that don't have scripts attached to them (whether the datasets are private or not).
> 
> I wasn't able to run the script for a private test dataset that I had created on the hub (https://huggingface.co/datasets/huggingface/DataMeasurementsTest/tree/main) -- although @lhoestq came to save the day!

why don't you try to share that info with people, so you can also save some days.

Hi ! You can run the command if you download the repository
```
git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
```
and run the command
```
datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
```

(though on my side it doesn't manage to download the data since the dataset is private ^^)

> Hi ! You can run the command if you download the repository
> 
> ```
> git clone https://huggingface.co/datasets/huggingface/DataMeasurementsTest
> ```
> 
> and run the command
> 
> ```
> datasets-cli test DataMeasurementsTest/DataMeasurementsTest.py
> ```
> 
> (though on my side it doesn't manage to download the data since the dataset is private ^^)

Hi! Thanks for the info. 
git cannot find the repository. Do you know if they have depreciated these tests and created a new one?

I think it's become private, but feel free to try with any other dataset like `lhoestq/test` for example at `https://huggingface.co/datasets/lhoestq/test`

> I think it's become private, but feel free to try with any other dataset like `lhoestq/test` for example at `https://huggingface.co/datasets/lhoestq/test`

your example repo and this page `https://huggingface.co/docs/datasets/add_dataset.html` helped me to solve.. thanks a lot
Better error msg if `len(predictions)` doesn't match `len(references)` in metrics

Improve the error message in `Metric.add_batch` if `len(predictions)` doesn't match `len(references)`.

cc: @BramVanroy (feel free to test this code on your examples and review this PR)

Comments:
Can't test this now but it may be a good improvement indeed.

I added a function, but it only works with the `list` type. For arrays/tensors, we delegate formatting to the frameworks.
Make inspect.get_dataset_config_names always return a non-empty list

Make all named configs cases, so that no special unnamed config case needs to be handled differently.

Fix #3135.

Comments:
This PR is already working (although not very beautiful; see below): the idea was to have the `DatasetModule.builder_kwargs` accessible from the `builder_cls`, so that this can generate the default builder config (at the class level, without requiring the builder to be instantiated).

I have a plan for a follow-up refactoring (same functionality, better implementation, much nicer), but I think we could already merge this, so that @severo can test it in the datasets previewer and report any potential issues.

Yes @lhoestq you are completely right. Indeed I was exclusively using `builder_cls.kwargs` to get the community dataset `name` (nothing else): "lhoestq___demo1"

See et: https://github.com/huggingface/datasets/pull/3159/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R413-R415

In your example, the `name` I was getting from `builder_cls.kwargs` was:
```python
{"name": "lhoestq___demo1",...}
```

I'm going to refactor all the approach... as I only need the name for this specific case ;)

I think this makes more sense now, @lhoestq @severo 😅 

It works well, thanks!
Fix string encoding for Value type

Some metrics have `string` features but currently it fails if users pass integers instead. Indeed feature encoding that handles the conversion of the user's objects to the right python type is missing a case for `string`, while it already works as expected for integers, floats and booleans

Here is an example code that didn't work previously, but that works with this fix:
```python
import datasets

# Note that 'id' is an integer while the SQuAD metric uses strings
predictions = [{'prediction_text': '1976', 'id': 5}]
references = [{'answers': {'answer_start': [97], 'text': ['1976']}, 'id': 5}] 

squad_metric = datasets.load_metric("squad") 
squad_metric.add_batch(predictions=predictions, references=references) 
results = squad_metric.compute()
# {'exact_match': 100.0, 'f1': 100.0}
```

cc @sgugger @philschmid 

Comments:
That was fast!
Illegal instruction (core dumped) at datasets import

## Describe the bug
I install datasets using conda and when I import datasets I get: "Illegal instruction (core dumped)"

## Steps to reproduce the bug

```
conda create --prefix path/to/env
conda activate path/to/env
conda install -c huggingface -c conda-forge datasets
# exits with output "Illegal instruction (core dumped)"
python -m datasets
```

## Environment info

When I run "datasets-cli env", I also get "Illegal instruction (core dumped)"

If I run the following commands:

```
conda create --prefix path/to/another/new/env
conda activate path/to/another/new/env
conda install -c huggingface transformers
transformers-cli env
```

Then I get:

- `transformers` version: 4.11.3
- Platform: Linux-5.4.0-67-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Let me know what additional information you need in order to debug this issue. Thanks in advance!

Comments:
It seems to be an issue with how conda-forge is building the binaries. It works on some machines, but not a machine with AMD Opteron 8384 processors.
Re-add faiss to windows testing suite

In recent versions, `faiss-cpu` seems to be available for Windows as well. See the [PyPi page](https://pypi.org/project/faiss-cpu/#files) to confirm. We can therefore included it for Windows in the setup file.

At first tests didn't pass due to problems with permissions as caused by `NamedTemporaryFile` on Windows. This built-in library is notoriously poor in playing nice on Windows. The required change isn't pretty, but it works. First set `delete=False` to not automatically try to delete the file on `exit`. Then, manually delete the file with `unlink`. It's weird, I know, but it works.

```python
with tempfile.NamedTemporaryFile(delete=False) as tmp_file:
    # do stuff
os.unlink(tmp_file.name)
```

closes #3150 

Comments:
Faiss _is_ available on Windows

In the setup file, I find the following:

https://github.com/huggingface/datasets/blob/87c71b9c29a40958973004910f97e4892559dfed/setup.py#L171

However, FAISS does install perfectly fine on Windows on my system. You can also confirm this on the [PyPi page](https://pypi.org/project/faiss-cpu/#files), where Windows wheels are available. Maybe this was true for older versions? For current versions, this can be removed I think.

(This isn't really a bug but didn't know how else to tag.)

If you agree I can do a quick PR and remove that line.

Comments:
Sure, feel free to open a PR.
Add CMU Hinglish DoG Dataset for MT

Address part of #2841 

Added the CMU Hinglish DoG Dataset as in GLUECoS. Added it as a seperate dataset as unlike other tasks of GLUE CoS this can't be evaluated for a BERT like model. 
Consists of parallel dataset between Hinglish (Hindi-English) and English, can be used for Machine Translation between the two. 

The data processing part is inspired from the GLUECoS repo [here](https://github.com/microsoft/GLUECoS/blob/7fdc51653e37a32aee17505c47b7d1da364fa77e/Data/Preprocess_Scripts/preprocess_mt_en_hi.py)
The dummy data part is not working properly, it shows 
``` UnboundLocalError: local variable 'generator_splits' referenced before assignment ``` 
when I run without ``--auto_generate``.

Please let me know how I can fix that.
Thanks

Comments:
Hi @lhoestq, thanks a lot for the help. I have moved the part as suggested. 
Although still while running the dummy data script, I face this issue

```
Traceback (most recent call last):
  File "/home/ishan/anaconda3/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/commands/dummy_data.py", line 318, in run
    self._autogenerate_dummy_data(
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/commands/dummy_data.py", line 363, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/builder.py", line 1103, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/features/features.py", line 981, in encode_example
    return encode_nested_example(self, example)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/features/features.py", line 775, in encode_nested_example
    return {
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/features/features.py", line 775, in <dictcomp>
    return {
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 99, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/home/ishan/anaconda3/lib/python3.8/site-packages/datasets/utils/py_utils.py", line 99, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
KeyError: 'status'
```
This KeyError is at times different from 'status' also.
when I run 
```
datasets-cli dummy_data datasets/cmu_hinglish_dog --auto_generate --json_field='history'
```
I have tried removing unnecessary feature type definition, but that didn't help. Please let me know if I am missing something, thanks!

The CI fail is unrelated to this PR and fixed on master. Merging !
Streaming with num_workers != 0

## Describe the bug
When using dataset streaming with pytorch DataLoader, the setting num_workers to anything other than 0 causes the code to freeze forever before yielding the first batch.

The code owner is likely @lhoestq 

## Steps to reproduce the bug

For your convenience, we've prepped a colab notebook that reproduces the bug
https://colab.research.google.com/drive/1Mgl0oTZSNIE3UeGl_oX9wPCOIxRg19h1?usp=sharing
```python
!pip install datasets==1.14.0

should_freeze_forever = True
# ^-- set this to True in order to freeze forever, set to False in order to work normally

import torch
from datasets import load_dataset

data = load_dataset("oscar", "unshuffled_deduplicated_bn", split="train", streaming=True)
data = data.map(lambda x: {"text": x["text"], "orig": f"oscar[{x['id']}]"}, batched=True)
data = data.shuffle(100, seed=1337)

data = data.with_format("torch")
loader = torch.utils.data.DataLoader(data, batch_size=2, num_workers=2 if should_freeze_forever else 0)

# v-- the code should freeze forever at this line
for i, row in enumerate(loader):
  print(row)
  if i > 10: break
print("DONE!")
```

## Expected results
The code should not freeze forever with num_workers=2

## Actual results
The code freezes forever with num_workers=2

## Environment info
- `datasets` version: 1.14.0 (also found in previous versions)
- Platform: google colab (also locally)
- Python version: 3.7, (also 3.8)
- PyArrow version: 3.0.0



Comments:
I can confirm that I was able to reproduce the bug. This seems odd given that #3423 reports duplicate data retrieval when `num_workers` and `streaming` are used together, which is obviously different from what is reported here. 

Any update? A possible solution is to have multiple arrow files as shards, and handle them like what webdatasets does.
![image](https://user-images.githubusercontent.com/11533479/148176637-72746b2c-c122-47aa-bbfe-224b13ee9a71.png)

Pytorch's new dataset RFC is supporting sharding now, which may helps avoid duplicate data under streaming mode. (https://github.com/pytorch/pytorch/blob/master/torch/utils/data/datapipes/iter/grouping.py#L13)


Hi ! Thanks for the insights :) Note that in streaming mode there're usually no arrow files. The data are streamed from TAR, ZIP, text, etc. files directly from the web. Though for sharded datasets we can definitely adopt a similar strategy !

fixed by #4375
CLI test command throws NonMatchingSplitsSizesError when saving infos

When trying to generate a datset JSON metadata, a `NonMatchingSplitsSizesError` is thrown:
```
$ datasets-cli test datasets/arabic_billion_words --save_infos --all_configs
Testing builder 'Alittihad' (1/10)
Downloading and preparing dataset arabic_billion_words/Alittihad (download: 332.13 MiB, generated: Unknown size, post-processed: Unknown size, total: 332.13 MiB) to .cache\arabic_billion_words\Alittihad\1.1.0\8175ff1c9714c6d5d15b1141b6042e5edf048276bb81a9c14e35e149a7a62ae4...
Traceback (most recent call last):
  File "path\huggingface\datasets\.venv\Scripts\datasets-cli-script.py", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "path\huggingface\datasets\src\datasets\commands\datasets_cli.py", line 33, in main
    service.run()
  File "path\huggingface\datasets\src\datasets\commands\test.py", line 144, in run
    builder.download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
  File "path\huggingface\datasets\src\datasets\builder.py", line 709, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "path\huggingface\datasets\src\datasets\utils\info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='arabic_billion_words'), 'recorded': SplitInfo(name='train', num_bytes=1601790302, num_examples=349342, dataset_name='arabic_billion_words')}]
```

This is due because a previous run generated a wrong `dataset_info.json`.

This error can be avoided by passing `--ignore_verifications`, but I think this should be assumed when passing `--save_infos`.

Comments:
[when Image type will exist] provide a way to get the data as binary + filename

**Is your feature request related to a problem? Please describe.**

When a dataset cell contains a value of type Image (be it from a remote URL, an Array2D/3D, or any other way to represent images), I want to be able to write the image to the disk, with the correct filename, and optionally to know its mimetype, in order to serve it on the web.

Note: this issue would apply exactly the same for the `Audio` type.

**Describe the solution you'd like**

If a "cell" has the type `Image`, provide a way to get the binary content of the file, and the filename, eg as:

```python
  filename: str
  data: bytes
```

**Describe alternatives you've considered**

A way to write the cell to the disk (passing a local directory), and then return the pathname, filename, and mimetype.


Comments:
@severo, maybe somehow related to this PR ?
- #3129

@severo I'll keep that in mind.

You can track progress on the Image feature in #3163 (still in the early stage).  

Hi ! As discussed with @severo offline it looks like the dataset viewer already supports reading PIL images, so maybe the dataset viewer doesn't need to disable decoding after all

Fixed with https://github.com/huggingface/datasets/pull/3163
Infer the features if missing

**Is your feature request related to a problem? Please describe.**

Some datasets, in particular community datasets, have no info file, thus no features.

**Describe the solution you'd like**

If a dataset has no features, the first loaded data (5-10 rows) could be used to infer the type.

Related: `datasets` would provide a way to load the data, and get the rows AND the features as the result.

**Describe alternatives you've considered**

The HF hub could also provide some UI to help the dataset maintainers to explicit the types of their rows, or automatically infer them as an initial proposal.

Comments:
Done by @lhoestq here: https://github.com/huggingface/datasets/pull/4500 (https://github.com/huggingface/datasets/pull/4500/files#diff-02930e1d966f4b41f9ddf15d961f16f5466d9bee583138657018c7329f71aa43R1255 in particular)
Provide a way to check if the features (in info) match with the data of a split

**Is your feature request related to a problem? Please describe.**

I understand that currently the data loaded has not always the type described in the info features

**Describe the solution you'd like**

Provide a way to check if the rows have the type described by info features

**Describe alternatives you've considered**

Always check it, and raise an error when loading the data if their type doesn't match the features.


Comments:
Related: #3144
Provide a way to write a streamed dataset to the disk

**Is your feature request related to a problem? Please describe.**

The streaming mode allows to get the 100 first rows of a dataset very quickly. But it does not cache the answer, so a posterior call to get the same 100 rows will send a request to the server again and again.

**Describe the solution you'd like**

Provide a way to write the streamed rows of a dataset on the disk, and to load from it later.

**Describe alternatives you've considered**

Provide a third mode: `lazy`, which would use the local cache for the data that have already been fetched previously, and use streaming to get the rest of the requested data. 


Comments:
Yes, I agree this feature is much needed. We could do something similar to what TF does (https://www.tensorflow.org/api_docs/python/tf/data/Dataset#cache). 

Ideally, if the entire streamed dataset is consumed/cached, the generated cache should be reusable for the Arrow dataset.

@mariosasko Hi big brother,any update on this? It's 2024 with large streamed dataset loading consume too much time(exp.  2day..), really need this feature for what TF does
Fix file/directory deletion on Windows  

Currently, on Windows, some attempts to delete a dataset file/directory will fail with the `PerimissionError`. 

Examples:
- download a dataset, then force redownload it in the same session while keeping a reference to the downloaded dataset
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset = load_dataset("sst", split="train", download_mode="force_redownload")
```
- try to clean up the cache files while keeping a reference to those files (via the mapped dataset):
```python
from datasets import load_dataset
dset = load_dataset("sst", split="train")
dset_mapped = dset.map(lambda _: {"dummy_col": 1})
dset.cleanup_cache_files()
```
We should fix those.


Comments:
More fine-grained taxonomy of error types

**Is your feature request related to a problem? Please describe.**

Exceptions like `FileNotFoundError` can be raised by different parts of the code, and it's hard to detect which one did

**Describe the solution you'd like**

Give a specific exception type for every group of similar errors

**Describe alternatives you've considered**

Rely on the error message, using regex


Comments:
related: #4995