jamesLeeeeeee/datasets-github-issues huggingface dataset:
Add RVL-CDIP dataset
Author: dnaveenr | Pull request: yes | State: closed

Resolves #2762 

Dataset Request : Add RVL-CDIP dataset [#2762](https://github.com/huggingface/datasets/issues/2762)
This PR adds the RVL-CDIP dataset.

The dataset contains Google Drive link for download and wasn't getting downloaded automatically, so I have provided manual_download_instructions.
- I have added the dummy_data.zip as well.

Needed inputs on how I can run the real data and the dummy data tests for datasets with manual download ?

Inputs and suggestions for improvement are welcome. Thank you.
Split size error on `amazon_us_reviews` / `PC_v1_00` dataset
Author: trentonstrong | Pull request: no | State: closed

## Describe the bug
When downloading this subset as of 3-28-2022 you will encounter a split size error after the dataset is extracted. The extracted dataset has roughly ~6m rows while the split expects <1m. 

Upon digging a little deeper, I downloaded the raw files from `https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_PC_v1_00.tsv.gz` and extracted them. A line count via `wc -l` confirms the ~6m number that we see and the data looks valid at a glance (I did not check for duplicate rows). My guess is this file has either been updated in place or there is a bug in the dataset metadata.

Happy to submit a PR and fix this up if turns out to be a metadata issue but wanted to get some other :eyes: on it first.

## Steps to reproduce the bug
```python
load_dataset('amazon_us_reviews', 'PC_v1_00')
```

## Expected results
Dataset is downloaded and extracted successfully.

## Actual results
An split size exception is thrown.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1
Dataset.unique(column: str) -> ArrowNotImplementedError
Author: orkenstein | Pull request: no | State: closed

## Describe the bug
I'm trying to use `unique()` function, but it fails
## Steps to reproduce the bug
1. Get dataset
2. Call `unique`
3. Error

# Sample code to reproduce the bug
```python
!pip show datasets

from datasets import load_dataset

dataset = load_dataset('wikiann', 'en')

dataset['train'].column_names
dataset['train'].unique(dataset['train'].column_names[0])
```

## Expected results
It would be nice to actually see unique items

## Actual results
Error:
```python
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
[<ipython-input-10-5e0de07ed42c>](https://s0qyv2vjaji-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220324-060046-RC00_436956229#) in <module>()
      6 
      7 dataset['train'].column_names
----> 8 dataset['train'].unique(dataset['train'].column_names[0])

5 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Function unique has no kernel matching input types (array[list<item: string>])
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Google Collab
- Python version: 3.7.13
- PyArrow version: 6.0.1
CLI dummy data generation is broken
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
We get a TypeError when running CLI dummy data generation:
```shell
datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate
```
gives:
```
  File ".../huggingface/datasets/src/datasets/commands/dummy_data.py", line 361, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
TypeError: _prepare_split() missing 1 required positional argument: 'check_duplicate_keys'
```
Error while building documentation
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```
Fix building of documentation
Author: albertvillanova | Pull request: yes | State: closed

Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```

Fix #4037.
can't download cats_vs_dogs dataset
Author: RRaphaell | Pull request: no | State: closed

## Describe the bug
can't download cats_vs_dogs dataset. error: Checksums didn't match for dataset source files

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("cats_vs_dogs")
```

## Expected results
loaded successfully.

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip']

## Environment info
fresh google colab notebook
Cannot load the dataset conll2012_ontonotesv5
Author: cathyxl | Pull request: no | State: closed

## Describe the bug
Cannot load the dataset conll2012_ontonotesv5

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset
dataset = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test")
print(dataset)

```

## Expected results
The datasets should be downloaded successfully

## Actual results
raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/zmycy7t9h9-1.zip']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 7.0.0
Use a constant for the articles regex in SQuAD v2
Author: bryant1410 | Pull request: yes | State: closed

The main reason for doing this is to be able to change the articles list if using another language, for example. It's not the most elegant solution but at least it makes the metric more extensible with no drawbacks.

BTW, what could be the best way to make this more generic (i.e., SQuAD in other languages)? Maybe receive a regex as an optional param, with the current value as the default? Similarly for SQuAD v1 (can't they re-use code?).
Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold
Author: MoritzLaurer | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
I would like to retrieve all texts from a dataset, which are semantically similar to a specific input text (query), above a certain (cosine) similarity threshold. My dataset is very large (Wikipedia), so I need to use Datasets and FAISS for this. I would like to be able to repeat many different queries on the dataset quickly. 

**Describe the solution you'd like**
dataset objects currently have the .get_nearest_examples() method for text retrieval via FAISS. But this only allows retrieving a specific number of K texts instead of everything above a specified similarity threshold. 
It would be great if HF Datasets would also support the FAISS method .range_search() for retrieving texts above a certain similarity threshold. 
see details here: https://github.com/facebookresearch/faiss/issues/1273

**Describe alternatives you've considered**
I've considered using native FAISS, but doing this via HF datasets would be better. My assumption is that Dataset features like dataset streaming make it easier to work with large datasets


**Additional context**
The concrete use-case is: I have a large dataset (wikipedia) and I would like to retrieve all paragraphs which are similar to a query. I will use sentence-transformers for encoding the texts.
Fix docs on audio feature installation
Author: albertvillanova | Pull request: yes | State: closed

This PR:
- Removes the explicit installation of `librosa` (this is installed with `pip install datasets[audio]`
- Adds the warning for Linux users to install manually the non-Python package `libsndfile`
- Explains that the installation of `torchaudio` is only necessary to support loading audio datasets containing MP3 audio files

Related to #4000.
ElasticSearch Indexing example: TypeError: __init__() missing 1 required positional argument: 'scheme'
Author: MoritzLaurer | Pull request: no | State: closed

## Describe the bug
I am following the example in the documentation for elastic search step by step (on google colab): https://huggingface.co/docs/datasets/faiss_es#elasticsearch

```
from datasets import load_dataset
squad = load_dataset('crime_and_punish', split='train[:1000]')
```

When I run the line: 
`squad.add_elasticsearch_index("context", host="localhost", port="9200")`

I get the error: 
`TypeError: __init__() missing 1 required positional argument: 'scheme'`


## Expected results
No error message

## Actual results

```
TypeError                                 Traceback (most recent call last)

[<ipython-input-23-9205593edef3>](https://localhost:8080/#) in <module>()
      1 import elasticsearch
----> 2 squad.add_elasticsearch_index("text", host="localhost", port="9200")


6 frames

[/usr/local/lib/python3.7/dist-packages/elasticsearch/_sync/client/utils.py](https://localhost:8080/#) in host_mapping_to_node_config(host)
    209         options["path_prefix"] = options.pop("url_prefix")
    210 
--> 211     return NodeConfig(**options)  # type: ignore
    212 
    213 

TypeError: __init__() missing 1 required positional argument: 'scheme'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.0
- Platform: Linux, Google Colab
- Python version: Google Colab (probably 3.7)
- PyArrow version: ?
Missing argument in precision/recall
Author: Dref360 | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
[`sklearn.metrics.precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html) accepts an argument `zero_division`, but it is not available in [precision Metric](https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py#L117)

Same issue is present for Recall.

**Describe the solution you'd like**
Support for **kwargs or adding a new field for `zero_division`.

**Describe alternatives you've considered**
I could filter the warnings myself, but that is not ideal.

**Additional context**
I can make the requested changes if this is approved.
Fix `map` remove_columns on empty dataset
Author: lhoestq | Pull request: yes | State: closed

On an empty dataset, the `remove_columns` parameter of `map` currently doesn't actually remove the columns:
```python
>>> ds = datasets.load_dataset("glue", "rte")
>>> ds_filtered = ds.filter(lambda x: x["label"] != -1)
>>> ds_mapped = ds_filtered.map(lambda x: x, remove_columns=["label"])
>>> print(repr(ds_mapped.column_names))
{
    'train': ['sentence1', 'sentence2', 'idx'],
    'validation': ['sentence1', 'sentence2', 'idx'],
    'test': ['sentence1', 'sentence2', 'label', 'idx']
}
```

I fixed this error and updated the tests
Can not correctly parse the classes with imagefolder
Author: YiSyuanChen | Pull request: no | State: closed

## Describe the bug
I try to load my own image dataset with imagefolder, but the parsing of classes is incorrect.

## Steps to reproduce the bug
I organized my dataset (ImageNet) in the following structure:
```
  - imagenet/
      - train/
          - n01440764/
              - ILSVRC2012_val_00000293.jpg
              - ...... 
          - n01695060/
          - ......
      - val/ 
          - n01440764/
          - n01695060/
          - ......
```

At first, I followed the instructions from the Huggingface [example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification#using-your-own-data) to load my data as:
```
from datasets import load_dataset
data_files = {'train': 'imagenet/train', 'val': 'imagenet/val'}
ds = load_dataset("nateraw/image-folder", data_files=data_files, task="image-classification")
```
but it resulted following error (I mask my personal path as <PERSONAL_PATH>):
```
FileNotFoundError: Unable to find 'https://huggingface.co/datasets/nateraw/image-folder/resolve/main/imagenet/train' at <PERSONAL_PATH>/ImageNet/https:/huggingface.co/datasets/nateraw/image-folder/resolve/main
```

Next, I followed a recent issue #3960 to load data as:
```
from datasets import load_dataset
data_files = {'train': ['imagenet/train/**'], 'val': ['imagenet/val/**']}
ds = load_dataset("imagefolder", data_files=data_files, task="image-classification")
```
and the data can be loaded without error as: (I copy val folder to train folder for illustration)
```
>>> ds
DatasetDict({
    train: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
    val: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
})
```
However, the parsed classes is wrong (should be 1000 classes):
```
>>> ds["train"].features
{'image': Image(decode=True, id=None), 'labels': ClassLabel(num_classes=1, names=['val'], id=None)}
```

## Expected results
I expect that the "labels" in ds["train"].features should contain 1000 classes.

## Actual results
The "labels" in ds["train"].features contains only 1 wrong class.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Ubuntu 18.04
- Python version: Python 3.7.12
- PyArrow version: 7.0.0
Cannot preview "hazal/Turkish-Biomedical-corpus-trM"
Author: hazalturkmen | Pull request: no | State: closed

## Dataset viewer issue for '*hazal/Turkish-Biomedical-corpus-trM'

**Link:** *https://huggingface.co/datasets/hazal/Turkish-Biomedical-corpus-trM*

*I cannot see the dataset preview.*

```
Server Error

Status code:   400
Exception:     HTTPError
Message:       403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/hazal/Turkish-Biomedical-corpus-trM?full=true
```


Am I the one who added this dataset ? Yes
Fix None issue with Sequence of dict
Author: lhoestq | Pull request: yes | State: closed

`Features.encode_example` currently fails if it contains a sequence if dict like `Sequence({"subcolumn": Value("int32")})` and if `None` is passed instead of the dict.

```python
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 1310, in encode_example
    return encode_nested_example(self, example)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in encode_nested_example
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in <dictcomp>
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 998, in encode_nested_example
    for k, (sub_schema, sub_objs) in zip_dict(schema.feature, obj):
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
TypeError: 'NoneType' object is not subscriptable
```

I fixed this issue and updated the tests (this case was missing in the tests)
AMI load_dataset error: sndfile library not found
Author: i-am-neo | Pull request: no | State: closed

## Describe the bug
Getting error message when loading AMI dataset.

## Steps to reproduce the bug
`python3 -c "from datasets import load_dataset; print(load_dataset('ami', 'headset-single', split='validation')[0])"

`

## Expected results
A clear and concise description of the expected results.

## Actual results
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/load.py", line 1707, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 595, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 690, in _download_and_prepare
    ) from None
OSError: Cannot find data file. 
Original error:
sndfile library not found

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.3
- PyArrow version: 7.0.0
set_format does not work with multi dimension tensor
Author: phihung | Pull request: no | State: closed

## Describe the bug
set_format only transforms the last dimension of a multi-dimension list to tensor

## Steps to reproduce the bug
```python
import torch
from datasets import Dataset

ds = Dataset.from_dict({"A": [torch.rand((2, 2))]})
# ds = Dataset.from_dict({"A": [np.random.rand(2, 2)]})  # => same result
ds = ds.with_format("torch")
print(ds[0])
```

## Expected results
```
{'A': [tensor([[0.6689, 0.1516], [0.1403, 0.5567]])]}
```

## Actual results
```
{'A': [tensor([0.6689, 0.1516]), tensor([0.1403, 0.5567])]}
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- datasets version: 2.0.0
- Platform: Mac OSX
- Python version: 3.8.12
- PyArrow version: 7.0.0
Use audio feature in ASR task template
Author: lhoestq | Pull request: yes | State: closed

The AutomaticSpeechRecognition task template is outdated: it still uses the file path column as input instead of the audio column.

I changed that and updated all the datasets as well as the tests.

The only community dataset that will need to be updated is `facebook/multilingual_librispeech`. It has almost zero usage unfortunately (probably because users load the duplicate `multilingual_librispeech` directly instead), but it means we can update it.

(this makes me think that we should deprecate `multilingual_librispeech` it and redirect users to `facebook/multilingual_librispeech`).

This PR is also useful for the AudioFolder in https://github.com/huggingface/datasets/pull/3963
Yelp not working
Author: patrickvonplaten | Pull request: no | State: closed

## Dataset viewer issue for '*name of the dataset*'

**Link:** https://huggingface.co/datasets/yelp_review_full/viewer/yelp_review_full/train

Doesn't work:
```
Server error
Status code:   400
Exception:     Error
Message:       line contains NULL
```

Am I the one who added this dataset ? No

A seamingly copy of the dataset: https://huggingface.co/datasets/SetFit/yelp_review_full works . The original one: https://huggingface.co/datasets/yelp_review_full has > 20K downloads.
ASSIN2 dataset checksum bug
Author: ruanchaves | Pull request: no | State: closed

## Describe the bug

Checksum error after trying to load the [ASSIN 2 dataset](https://huggingface.co/datasets/assin2).

`NonMatchingChecksumError` triggered by calling `load_dataset("assin2")`.

Similar to #3952 , #3942 , #3941 , etc.

```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
[<ipython-input-13-c664a92ad5e7>](https://localhost:8080/#) in <module>()
----> 1 load_dataset('assin2')

4 frames
[/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py](https://localhost:8080/#) in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?id=1Q9j1a83CuKzsHCGaNulSkNxBm7Dkn7Ln&export=download']
```

## Steps to reproduce the bug
```python
from datasets import load_dataset
load_dataset("assin2")
```

## Expected results
Load the dataset.

## Actual results
The dataset won't load. 

## Environment info
- `datasets` version: 2.0.1.dev0
- Platform: Google Colab
- Python version:  3.7.12
- PyArrow version: 6.0.1
How to use generate this multitask dataset for SQUAD? I am getting a value error. 
Author: gsk1692 | Pull request: no | State: closed

## Dataset viewer issue for 'squad_multitask*'

**Link:** https://huggingface.co/datasets/vershasaxena91/squad_multitask

*short description of the issue*

I am trying to generate the multitask dataset for squad dataset. However, gives the error in dataset explorer as well as my local machine. 
I tried the command: dataset = load_dataset("vershasaxena91/squad_multitask", 'highlight_qg_format')
Error:
Status code:   400
Exception:     TypeError
Message:       argument of type 'Value' is not iterable

Kindly advice.