konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Unexpected behavior of _TempDirWithCustomCleanup

## Describe the bug
This is not 100% a bug in `datasets`, but behavior that surprised me and I think this could be made more robust on the `datasets`side. 

When using `datasets.disable_caching()`, cache files are written to a temporary directory. This directory should be based on the environment variable TMPDIR. I want to set TMPDIR at runtime using os.ENVIRON["TMPDIR"] = something, but depending on other imported modules this can fail to take effect. 

## Steps to reproduce the bug
`_TempDirWithCustomCleanup` relies on `tempfile` to generate a path to a temporary directory. However, `tempfile` generates the path only once. This can be a problem when trying to set TMPDIR at runtime whenever other code imports `tempfile` first and does something unexpected. 

For example (after too much trial and error) I found out that a different part of the code base I work with defines a class `PatchedDataCollatorForLanguageModeling(transformers.DataCollatorForLanguageModeling)` based on a `transformers` class.  This import is enough to trigger `tempfile` to generate `tempfile` to generate a temporary path and leading to the wrong path being cached in `tempfile.tempdir`.

## Suggestion:
I could file this also as bug with `transformers`, but I think fixing this on the datasets would be much more robust:
Datasets could recompute the temporary path once (technically possible via `tempfile._get_default_tempdir` or resetting 
 the global variable `tempfile.tmpdir` to None) before setting its own global  `_TEMP_DIR_FOR_TEMP_CACHE_FILES`.




Hi ! Would setting TMPDIR at the beginning of your python script/session work ? I mean, even before importing transformers, datasets, etc. and using them ? I think this would be the most robust solution given any library that uses `tempfile`. I don't think we aim to support environment variables to be changed at run time

Hi, yeah setting the environment variable before the imports / as environment variable outside is another way to fix this. I am just arguing that `datasets` already uses its own global variable to track temporary files: `_TEMP_DIR_FOR_TEMP_CACHE_FILES`, and the creation of this global variable should respect TMPDIR instead of relying on tempfile to do so.
Modify datatype from `int32` to `float` for pearsonr, spearmanr.

**Is your feature request related to a problem? Please describe.**
- Now [Pearsonr](https://github.com/huggingface/datasets/blob/master/metrics/pearsonr/pearsonr.py) and [Spearmanr](https://github.com/huggingface/datasets/blob/master/metrics/spearmanr/spearmanr.py) both get input data as 'int32'. 


**Describe the solution you'd like**
- Considering that those metrics are widely used for the STS task(labels are in 'float' data type),
it would be better to modify datatype from 'int32' to 'float' for getting exact values of similarity.

@Woodywarhol9 good catch, thanks for reporting.

We are fixing this.
metric = metric_cls( TypeError: 'NoneType' object is not callable

Hi, friend. I meet a problem.
When I run the code:
`metric = load_metric('glue', 'rte')`

There is a problem raising:
`metric = metric_cls(
TypeError: 'NoneType' object is not callable `

I don't know why. Thanks for your help!



Hi @klyuhang9,

I'm sorry but I can't reproduce your problem:
```python
In [2]: metric = load_metric('glue', 'rte')
Downloading builder script: 5.76kB [00:00, 2.40MB/s]
```

Could you please, retry to load the metric? Sometimes there are temporary connectivity issues.

Feel free to re-open this issue of the problem persists.
Split size error on `amazon_us_reviews` / `PC_v1_00` dataset

## Describe the bug
When downloading this subset as of 3-28-2022 you will encounter a split size error after the dataset is extracted. The extracted dataset has roughly ~6m rows while the split expects <1m. 

Upon digging a little deeper, I downloaded the raw files from `https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_us_PC_v1_00.tsv.gz` and extracted them. A line count via `wc -l` confirms the ~6m number that we see and the data looks valid at a glance (I did not check for duplicate rows). My guess is this file has either been updated in place or there is a bug in the dataset metadata.

Happy to submit a PR and fix this up if turns out to be a metadata issue but wanted to get some other :eyes: on it first.

## Steps to reproduce the bug
```python
load_dataset('amazon_us_reviews', 'PC_v1_00')
```

## Expected results
Dataset is downloaded and extracted successfully.

## Actual results
An split size exception is thrown.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


Follow-up: I have confirmed there are no duplicate lines via `sort amazon_reviews_us_PC_v1_00.tsv | uniq -cd` after extracting the raw file.

Hi @trentonstrong, thanks for reporting!

I confirm that loading this dataset configuration throws a `NonMatchingSplitsSizesError`:
```
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=350242049, num_examples=785730, dataset_name='amazon_us_reviews'), 'recorded': SplitInfo(name='train', num_bytes=3982712078, num_examples=6908554, dataset_name='amazon_us_reviews')}]
```

Also thank you for your offer to fix this. You can find information about how to update the metadata JSON file here: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#automatically-add-code-metadata
```shell
datasets-cli test datasets/amazon_us_reviews --save_infos --all_configs
```
Please, feel free to open a PR with this fix. And do not hesitate to ping me if you need any help.

No sweat. Will get it patched up ASAP.
Dataset.unique(column: str) -> ArrowNotImplementedError

## Describe the bug
I'm trying to use `unique()` function, but it fails
## Steps to reproduce the bug
1. Get dataset
2. Call `unique`
3. Error

# Sample code to reproduce the bug
```python
!pip show datasets

from datasets import load_dataset

dataset = load_dataset('wikiann', 'en')

dataset['train'].column_names
dataset['train'].unique(dataset['train'].column_names[0])
```

## Expected results
It would be nice to actually see unique items

## Actual results
Error:
```python
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
[<ipython-input-10-5e0de07ed42c>](https://s0qyv2vjaji-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220324-060046-RC00_436956229#) in <module>()
      6 
      7 dataset['train'].column_names
----> 8 dataset['train'].unique(dataset['train'].column_names[0])

5 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Function unique has no kernel matching input types (array[list<item: string>])
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Google Collab
- Python version: 3.7.13
- PyArrow version: 6.0.1


Hi @orkenstein, thanks for reporting.

Please note that for this case, our `datasets` library uses under the hood the Apache Arrow `unique` function: https://arrow.apache.org/docs/python/generated/pyarrow.compute.unique.html#pyarrow.compute.unique

And currently the Apache Arrow `unique` function is only implemented for these input types (see info in their [docs](https://arrow.apache.org/docs/cpp/compute.html#array-wise-vector-functions)): Boolean, Null, Numeric, Temporal, Binary- and String-like.

However, the data types of the `wikiann` dataset are all `list<item: string>` (see its [dataset card](https://huggingface.co/datasets/wikiann#data-fields)), and thus, not yet supported by the Apache Arrow `unique` function.

As a workaround solution you can use pandas:
```python
from datasets import load_dataset

dataset = load_dataset('wikiann', 'en', split='train')
df = dataset.to_pandas()
unique_df = df[~df.tokens.apply(tuple).duplicated()]  # from https://stackoverflow.com/a/46958336/17517845
```

Note that pandas loads the dataset in memory (this one is small so it's fine).

@lhoestq thank you! I will fall back to this method for now
CLI dummy data generation is broken

## Describe the bug
We get a TypeError when running CLI dummy data generation:
```shell
datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate
```
gives:
```
  File ".../huggingface/datasets/src/datasets/commands/dummy_data.py", line 361, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
TypeError: _prepare_split() missing 1 required positional argument: 'check_duplicate_keys'
```
[DO NOT MERGE] Test doc-builder with skipped installation feature

This PR is just for testing that we can build PR docs with changes made on the [`skip-install-for-real`](https://github.com/huggingface/doc-builder/tree/skip-install-for-real) branch of `doc-builder`.

_The documentation is not available anymore as the PR was closed or merged._

Fix in https://github.com/huggingface/doc-builder/pull/162 works as expected (docs build), closing this
Error while building documentation

## Describe the bug
Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```



After some investigation, maybe the bug is in `doc-builder`.

I've opened an issue there:
- huggingface/doc-builder#160

Fixed by @lewtun (thank you):
- huggingface/doc-builder@31fe6c8bc7225810e281c2f6c6cd32f38828c504
Fix building of documentation

Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```

Fix #4037.

_The documentation is not available anymore as the PR was closed or merged._

Superseded by huggingface/doc-builder@31fe6c8bc7225810e281c2f6c6cd32f38828c504
can't download cats_vs_dogs dataset

## Describe the bug
can't download cats_vs_dogs dataset. error: Checksums didn't match for dataset source files

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("cats_vs_dogs")
```

## Expected results
loaded successfully.

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip']

## Environment info
fresh google colab notebook


Thnaks for reporting @RRaphaell.

We are fixing it.
Cannot load the dataset conll2012_ontonotesv5

## Describe the bug
Cannot load the dataset conll2012_ontonotesv5

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset
dataset = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test")
print(dataset)

```

## Expected results
The datasets should be downloaded successfully

## Actual results
raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/zmycy7t9h9-1.zip']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 7.0.0



Hi @cathyxl, thanks for reporting.

Indeed, we have recently updated the loading script of that dataset (and fixed that bug as well):
- #4002

That fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by:
- installing `datasets` from our GitHub repo:
```bash
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
- forcing the data files to be redownloaded
```python
ds = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test", download_mode="force_redownload")
```

Feel free to re-open this issue if the problem persists.
Use a constant for the articles regex in SQuAD v2

The main reason for doing this is to be able to change the articles list if using another language, for example. It's not the most elegant solution but at least it makes the metric more extensible with no drawbacks.

BTW, what could be the best way to make this more generic (i.e., SQuAD in other languages)? Maybe receive a regex as an optional param, with the current value as the default? Similarly for SQuAD v1 (can't they re-use code?).

_The documentation is not available anymore as the PR was closed or merged._
Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold

**Is your feature request related to a problem? Please describe.**
I would like to retrieve all texts from a dataset, which are semantically similar to a specific input text (query), above a certain (cosine) similarity threshold. My dataset is very large (Wikipedia), so I need to use Datasets and FAISS for this. I would like to be able to repeat many different queries on the dataset quickly. 

**Describe the solution you'd like**
dataset objects currently have the .get_nearest_examples() method for text retrieval via FAISS. But this only allows retrieving a specific number of K texts instead of everything above a specified similarity threshold. 
It would be great if HF Datasets would also support the FAISS method .range_search() for retrieving texts above a certain similarity threshold. 
see details here: https://github.com/facebookresearch/faiss/issues/1273

**Describe alternatives you've considered**
I've considered using native FAISS, but doing this via HF datasets would be better. My assumption is that Dataset features like dataset streaming make it easier to work with large datasets


**Additional context**
The concrete use-case is: I have a large dataset (wikipedia) and I would like to retrieve all paragraphs which are similar to a query. I will use sentence-transformers for encoding the texts. 


Hi ! You can access the faiss index with
```python
faiss_index = my_dataset.get_index("my_index_name").faiss_index
```
and then do whatever you want with it, e.g. query it using range_search:
```python
threshold = 0.95
limits, distances, indices = faiss_index.range_search(x=xq, thresh=threshold)

texts = dataset[indices]
```

wow, that's great, thank you for the explanation. (if that's not already in the documentation, could be worth adding it)

which type of faiss index is Datasets using? I looked into faiss recently and I understand that there are several different types of indexes and the choice is important, e.g. regarding which distance metric you use (euclidian vs. cosine/dot product), the size of my dataset etc. can I chose the type of index somehow as well?

`Dataset.add_faiss_index` has a `string_factory` parameter, used to set the type of index (see the faiss documentation about [index factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)). Alternatively, you can pass an index you've defined yourself using faiss with the `custom_index` parameter of `Dataset.add_faiss_index` 

Here is the full documentation of `Dataset.add_faiss_index`: https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index

great thanks, I will try it out
Fix docs on audio feature installation

This PR:
- Removes the explicit installation of `librosa` (this is installed with `pip install datasets[audio]`
- Adds the warning for Linux users to install manually the non-Python package `libsndfile`
- Explains that the installation of `torchaudio` is only necessary to support loading audio datasets containing MP3 audio files

Related to #4000.

_The documentation is not available anymore as the PR was closed or merged._
ElasticSearch Indexing example: TypeError: __init__() missing 1 required positional argument: 'scheme'

## Describe the bug
I am following the example in the documentation for elastic search step by step (on google colab): https://huggingface.co/docs/datasets/faiss_es#elasticsearch

```
from datasets import load_dataset
squad = load_dataset('crime_and_punish', split='train[:1000]')
```

When I run the line: 
`squad.add_elasticsearch_index("context", host="localhost", port="9200")`

I get the error: 
`TypeError: __init__() missing 1 required positional argument: 'scheme'`


## Expected results
No error message

## Actual results

```
TypeError                                 Traceback (most recent call last)

[<ipython-input-23-9205593edef3>](https://localhost:8080/#) in <module>()
      1 import elasticsearch
----> 2 squad.add_elasticsearch_index("text", host="localhost", port="9200")


6 frames

[/usr/local/lib/python3.7/dist-packages/elasticsearch/_sync/client/utils.py](https://localhost:8080/#) in host_mapping_to_node_config(host)
    209         options["path_prefix"] = options.pop("url_prefix")
    210 
--> 211     return NodeConfig(**options)  # type: ignore
    212 
    213 

TypeError: __init__() missing 1 required positional argument: 'scheme'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.0
- Platform: Linux, Google Colab
- Python version: Google Colab (probably 3.7)
- PyArrow version: ?


Hi, @MoritzLaurer, thanks for reporting.

Normally this is due to a mismatch between the versions of your Elasticsearch client and server:
- your ES client is passing only keyword arguments to your ES server
- whereas your ES server expects a positional argument called 'scheme'

In order to fix this, you should align the major versions of both Elasticsearch client and server.

You can have more info:
- on this other issue page: https://github.com/huggingface/datasets/issues/3956#issuecomment-1072115173
- Elasticsearch client docs: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/overview.html#_compatibility

Feel free to re-open this issue if the problem persists.

Duplicate of:
- #3956

1. Check elasticsearch version
```
import elasticsearch
print(elasticsearch.__version__)
```
Ex: 7.9.1
2. Uninstall current elasticsearch package
`pip uninstall elasticsearch`
3. Install elasticsearch 7.9.1 package
`pip install elasticsearch==7.9.1`
Missing argument in precision/recall

**Is your feature request related to a problem? Please describe.**
[`sklearn.metrics.precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html) accepts an argument `zero_division`, but it is not available in [precision Metric](https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py#L117)

Same issue is present for Recall.

**Describe the solution you'd like**
Support for **kwargs or adding a new field for `zero_division`.

**Describe alternatives you've considered**
I could filter the warnings myself, but that is not ideal.

**Additional context**
I can make the requested changes if this is approved.

Thanks for the suggestion, @Dref360.

We are adding that argument.
Doc: image_process small tip

I've added a small tip in the `image_process` doc

_The documentation is not available anymore as the PR was closed or merged._

This tip is unnecessary, i.e., Pillow will already be installed since the `Image` feature requires it for encoding and decoding. Thanks anyway.

cc @stevhliu I've noticed we are missing the installation section in the doc (`pip install datasets[vision]`). I can add it myself.
Fix `map` remove_columns on empty dataset

On an empty dataset, the `remove_columns` parameter of `map` currently doesn't actually remove the columns:
```python
>>> ds = datasets.load_dataset("glue", "rte")
>>> ds_filtered = ds.filter(lambda x: x["label"] != -1)
>>> ds_mapped = ds_filtered.map(lambda x: x, remove_columns=["label"])
>>> print(repr(ds_mapped.column_names))
{
    'train': ['sentence1', 'sentence2', 'idx'],
    'validation': ['sentence1', 'sentence2', 'idx'],
    'test': ['sentence1', 'sentence2', 'label', 'idx']
}
```

I fixed this error and updated the tests

_The documentation is not available anymore as the PR was closed or merged._
Can not correctly parse the classes with imagefolder

## Describe the bug
I try to load my own image dataset with imagefolder, but the parsing of classes is incorrect.

## Steps to reproduce the bug
I organized my dataset (ImageNet) in the following structure:
```
  - imagenet/
      - train/
          - n01440764/
              - ILSVRC2012_val_00000293.jpg
              - ...... 
          - n01695060/
          - ......
      - val/ 
          - n01440764/
          - n01695060/
          - ......
```

At first, I followed the instructions from the Huggingface [example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification#using-your-own-data) to load my data as:
```
from datasets import load_dataset
data_files = {'train': 'imagenet/train', 'val': 'imagenet/val'}
ds = load_dataset("nateraw/image-folder", data_files=data_files, task="image-classification")
```
but it resulted following error (I mask my personal path as <PERSONAL_PATH>):
```
FileNotFoundError: Unable to find 'https://huggingface.co/datasets/nateraw/image-folder/resolve/main/imagenet/train' at <PERSONAL_PATH>/ImageNet/https:/huggingface.co/datasets/nateraw/image-folder/resolve/main
```

Next, I followed a recent issue #3960 to load data as:
```
from datasets import load_dataset
data_files = {'train': ['imagenet/train/**'], 'val': ['imagenet/val/**']}
ds = load_dataset("imagefolder", data_files=data_files, task="image-classification")
```
and the data can be loaded without error as: (I copy val folder to train folder for illustration)
```
>>> ds
DatasetDict({
    train: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
    val: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
})
```
However, the parsed classes is wrong (should be 1000 classes):
```
>>> ds["train"].features
{'image': Image(decode=True, id=None), 'labels': ClassLabel(num_classes=1, names=['val'], id=None)}
```

## Expected results
I expect that the "labels" in ds["train"].features should contain 1000 classes.

## Actual results
The "labels" in ds["train"].features contains only 1 wrong class.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Ubuntu 18.04
- Python version: Python 3.7.12
- PyArrow version: 7.0.0


I found that the problem arises because the image files in my folder are actually symbolic links (for my own reasons).  After modifications, the classes can now be correctly parsed. Therefore, I close this issue.

HI, I have a question. How much time did you load the ImageNet data files?
Cannot preview "hazal/Turkish-Biomedical-corpus-trM"

## Dataset viewer issue for '*hazal/Turkish-Biomedical-corpus-trM'

**Link:** *https://huggingface.co/datasets/hazal/Turkish-Biomedical-corpus-trM*

*I cannot see the dataset preview.*

```
Server Error

Status code:   400
Exception:     HTTPError
Message:       403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/hazal/Turkish-Biomedical-corpus-trM?full=true
```


Am I the one who added this dataset ? Yes


Hi @hazalturkmen, thanks for reporting.

Note that your dataset repository does not contain any loading script; it only contains a data file named `tr_article_2`.

When there is no loading script but only data files, the `datasets` library tries to infer how to load the data by looking at the data file extensions. However, your data file does not have any extension.

Note that current supported data file extensions are: 'csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip'.

You have more info on our docs: [How to share a dataset](https://huggingface.co/docs/datasets/share).

thanks for reply :)
Fix None issue with Sequence of dict

`Features.encode_example` currently fails if it contains a sequence if dict like `Sequence({"subcolumn": Value("int32")})` and if `None` is passed instead of the dict.

```python
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 1310, in encode_example
    return encode_nested_example(self, example)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in encode_nested_example
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in <dictcomp>
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 998, in encode_nested_example
    for k, (sub_schema, sub_objs) in zip_dict(schema.feature, obj):
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
TypeError: 'NoneType' object is not subscriptable
```

I fixed this issue and updated the tests (this case was missing in the tests)

_The documentation is not available anymore as the PR was closed or merged._

Merging since I'd like do do a patch release soon for this one
AMI load_dataset error: sndfile library not found

## Describe the bug
Getting error message when loading AMI dataset.

## Steps to reproduce the bug
`python3 -c "from datasets import load_dataset; print(load_dataset('ami', 'headset-single', split='validation')[0])"

`

## Expected results
A clear and concise description of the expected results.

## Actual results
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/load.py", line 1707, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 595, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 690, in _download_and_prepare
    ) from None
OSError: Cannot find data file. 
Original error:
sndfile library not found

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.3
- PyArrow version: 7.0.0


Issue unresolved, see [4000](https://github.com/huggingface/datasets/issues/4009#issue-1179658611)
set_format does not work with multi dimension tensor

## Describe the bug
set_format only transforms the last dimension of a multi-dimension list to tensor

## Steps to reproduce the bug
```python
import torch
from datasets import Dataset

ds = Dataset.from_dict({"A": [torch.rand((2, 2))]})
# ds = Dataset.from_dict({"A": [np.random.rand(2, 2)]})  # => same result
ds = ds.with_format("torch")
print(ds[0])
```

## Expected results
```
{'A': [tensor([[0.6689, 0.1516], [0.1403, 0.5567]])]}
```

## Actual results
```
{'A': [tensor([0.6689, 0.1516]), tensor([0.1403, 0.5567])]}
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- datasets version: 2.0.0
- Platform: Mac OSX
- Python version: 3.8.12
- PyArrow version: 7.0.0

Hi! Use the `ArrayXD` feature type (where X is the number of dimensions) to get correctly formated tensors. So in your case, define the dataset as follows :
```python
ds = Dataset.from_dict({"A": [torch.rand((2, 2))]}, features=Features({"A": Array2D(shape=(2, 2), dtype="float32")}))
```


Hi @mariosasko I'm facing the same issue and the only work around I've found so far is to convert my `DatasetDict` to a dictionary and then create new objects with `Dataset.from_dict`.
```
dataset = load_dataset("my_dataset.py")
dataset = dataset.map(lambda example: blabla(example))
dict_dataset_test = dataset["test"].to_dict()
...
dataset_test = Dataset.from_dict(dict_dataset_test, features=Features(features))
```
However, converting a `Dataset` object to a dict takes quite a lot of time and memory... Is there a way to directly create an `Array2D` without having to transform the original `Dataset` to a dict?

Hi! Yes, you can directly pass the `Features` dictionary as `features` in `map` to cast the column to `Array2D`:
```python
dataset = dataset.map(lambda example: blabla(example), features=Features(features))
```
Or you can use `cast` after `map` to do that:
```python
dataset = dataset.map(lambda example: blabla(example))
dataset = dataset.cast(Features(features))
```

Fantastic thank you @mariosasko
The first option you suggested is indeed way faster 😃
Use audio feature in ASR task template

The AutomaticSpeechRecognition task template is outdated: it still uses the file path column as input instead of the audio column.

I changed that and updated all the datasets as well as the tests.

The only community dataset that will need to be updated is `facebook/multilingual_librispeech`. It has almost zero usage unfortunately (probably because users load the duplicate `multilingual_librispeech` directly instead), but it means we can update it.

(this makes me think that we should deprecate `multilingual_librispeech` it and redirect users to `facebook/multilingual_librispeech`).

This PR is also useful for the AudioFolder in https://github.com/huggingface/datasets/pull/3963

_The documentation is not available anymore as the PR was closed or merged._
How to use generate this multitask dataset for SQUAD? I am getting a value error. 

## Dataset viewer issue for 'squad_multitask*'

**Link:** https://huggingface.co/datasets/vershasaxena91/squad_multitask

*short description of the issue*

I am trying to generate the multitask dataset for squad dataset. However, gives the error in dataset explorer as well as my local machine. 
I tried the command: dataset = load_dataset("vershasaxena91/squad_multitask", 'highlight_qg_format')
Error:
Status code:   400
Exception:     TypeError
Message:       argument of type 'Value' is not iterable

Kindly advice. 


Hi! Replacing `nlp.<obj>` with `datasets.<obj>` in the script should fix the problem. `nlp` has been renamed to `datasets` more than a year ago, so please use `datasets` instead to avoid weird issues.

Thank You! Was able to solve with the help of this.

But I request you to please fix the same in the dataset hub explorer as well...

May I ask how to get this dataset?
Fix Audio.encode_example() when writing an array

Closes  #3996

_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova do you think [this line](https://github.com/huggingface/datasets/pull/3998/files#diff-58e348f6e4deaa5f3119e420a5d48ebb82875a78c28628831748fb54f59b2c78R67) is enough? that's why we missed this bug, we didn't check this case
Sync Features dictionaries

This PR adds a wrapper to the `Features` class to keep the secondary dict, `_column_requires_decoding`, aligned with the main dict (as discussed in https://github.com/huggingface/datasets/pull/3723#discussion_r806912731).

A more elegant approach would be to subclass `UserDict` and override `__setitem__` and `__delitem__`, but this PR doesn't implement it for the following reasons:
* it requires replacing all occurrences of `isinstance(obj, dict)` with `isinstance(obj, Mapping)`, which is five times slower than `isinstance(obj, dict)` on my machine, in `features.py`
* is a breaking change, i.e., `isinstance(Features(...), dict)` would return `False` after it
* IMO, it makes sense to be consistent in the user-facing API and subclass either `dict` or `UserDict`. The problem with the latter is that it can't be used for `DatasetDict` because `DatasetDict` exposes the `data` property, which is also used by `UserDict`, so this would result in a collision.



_The documentation is not available anymore as the PR was closed or merged._
Close `PIL.Image` file handler in `Image.decode_example`

Closes the file handler of the PIL image object in `Image.decode_example` to avoid the `Too many open files` error.

To pass [the image equality checks](https://app.circleci.com/pipelines/github/huggingface/datasets/10774/workflows/d56670e6-16bb-4c64-b601-a152c5acf5ed/jobs/65825) in CI, `Image.decode_example` calls `image.load()` regardless of how the image object is created (not only for the `PIL.Image.open(local_path)` case). This is needed because `load()` sets the `readonly` attribute of a `PIL.Image` object to 0 (it's 1 after `PIL.Image.open(file_like)`), and in the older PIL versions (only fixed on main), that attribute is considered in `PIL.Image.__eq__`. More info can be found here: https://github.com/python-pillow/Pillow/issues/5926.

Fix #3985 





_The documentation is not available anymore as the PR was closed or merged._
Streaming dataset + interleave + DataLoader hangs with multiple workers

## Describe the bug
Interleaving multiple iterable datasets that use `load_dataset` on streaming mode hangs when passed to `torch.utils.data.DataLoader` with multiple workers.

## Steps to reproduce the bug
```python
from datasets import interleave_datasets, load_dataset
from torch.utils.data import DataLoader

en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)
fr_dataset = load_dataset('oscar', "unshuffled_deduplicated_fr", split='train', streaming=True)
it_dataset = load_dataset('oscar', "unshuffled_deduplicated_it", split='train', streaming=True)
de_dataset = load_dataset('oscar', "unshuffled_deduplicated_de", split='train', streaming=True)
multilingual_dataset = interleave_datasets([en_dataset, fr_dataset, de_dataset, it_dataset])
multilingual_dataset = multilingual_dataset.with_format('torch')

next(iter(multilingual_dataset))  # works fairly fast

dataloader = DataLoader(multilingual_dataset, batch_size=8, num_workers=4)
for batch in dataloader:
    print(len(batch))  # prints nothing after 30 min of waiting

dataloader = DataLoader(multilingual_dataset, batch_size=8, num_workers=0)
for batch in dataloader:
    print(len(batch))  # prints right away

```

## Expected results
It should be able to iterate the dataset with multiple workers.

## Actual results
Prints with results with `next(iter(multilingual_dataset)) ` and `num_workers=0`  but it prints nothing with `num_workers=4` or any number above 0.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.1.dev0
- `pytorch` version: 1.10.0+cu113
- Python version: 3.7
- PyArrow version: 6.0.1


Same thing occurs when streaming files loaded from disk.

Hi ! Thanks for reporting, could this be related to https://github.com/huggingface/datasets/issues/3950 ?

Currently streaming datasets only works in single process, but we're working on having in work in distributed setups as well :) (EDIT: done)

Hi, thanks for your reply. It seems related :)

+1

Please update `datasets` if you're having this issue. What version are you using ?
Image column is not decoded in map when using with with_transform

## Describe the bug
Image column is not _decoded_ in **map** when using with `with_transform`

## Steps to reproduce the bug
```python
from datasets import Image, Dataset

def add_C(batch):
    batch["C"] = batch["A"]
    return batch

ds = Dataset.from_dict({"A": ["image.png"]}).cast_column("A",  Image())
ds = ds.with_transform(lambda x: x)  # <= This line causes the problem
ds = ds.map(add_C, batched=True)
print(ds[0])
```

## Expected results
```
{'C': <PIL.PngImagePlugin.PngImageFile>,  ...}
```

## Actual results
```
{'C': {'bytes': None, 'path': 'image.png'}, ...}
```

If we remove the `with_transform` line, we get the expected result.

## Environment info
- `datasets` version: 2.0.0
- Platform: Mac OSX
- Python version: 3.8.12
- PyArrow version: 7.0.0


Hi! This behavior stems from this line: https://github.com/huggingface/datasets/blob/799b817d97590ddc97cbd38d07469403e030de8c/src/datasets/arrow_dataset.py#L1919
Basically, the `Image`/`Audio` columns are decoded only if the `format_type` attribute is `None` (`set_format`/`with_format` and `set_transform`/`with_transform` assign a non-`None` value to it) and the  `input_columns` param is not specified (see https://github.com/huggingface/datasets/issues/3756). We will remove these limitations soon.
Add Lung Image Database Consortium image collection (LIDC-IDRI) dataset

## Adding a Dataset

- **Name:** *Lung Image Database Consortium image collection (LIDC-IDRI)*
- **Description:** *Consists of diagnostic and lung cancer screening thoracic computed tomography (CT) scans with marked-up annotated lesions. It is a web-accessible international resource for development, training, and evaluation of computer-assisted diagnostic (CAD) methods for lung cancer detection and diagnosis. Initiated by the National Cancer Institute (NCI), further advanced by the Foundation for the National Institutes of Health (FNIH), and accompanied by the Food and Drug Administration (FDA) through active participation, this public-private partnership demonstrates the success of a consortium founded on a consensus-based process.*
- **Data:** *[link to the Github repository or current dataset location](https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI)*
- **Motivation:** *Key dataset in the healthcare community*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

FYI @osanseviero @abidlabs
Improve AutomaticSpeechRecognition task template

**Is your feature request related to a problem? Please describe.**
[AutomaticSpeechRecognition task template](https://github.com/huggingface/datasets/blob/master/src/datasets/tasks/automatic_speech_recognition.py) is outdated as it uses path to audiofile as an audio column instead of a Audio feature itself (I guess it's because Audio feature didn't exist at the time this template was created).

**Describe the solution you'd like**
Change audio columns from string path to Audio feature.


There is an open PR to do that: #3364. I just haven't had time to finish it... 

> There is an open PR to do that: #3364. I just haven't had time to finish it...

😬 thanks...
Remove old wikipedia leftovers

After updating Wikipedia dataset, remove old wikipedia leftovers from doc.


_The documentation is not available anymore as the PR was closed or merged._

> This makes me think we shouldn't advise the use of load_dataset in dataset scripts, since it doesn't guarantee that the cache will work as expected (the cache directory is not set correctly, and the required disk space for downloaded files is not recorded)

@lhoestq, do you think it could be a good idea to add a comment in this script WARNING that using load_dataset in a script is not good practice and that people should avoid using that script as a template to create other scripts? 

good idea ! :)
[image feature] Too many files open error when image feature is returned as a path

## Describe the bug

PR in context: #3967. If I load the dataset in this PR (TextVQA), and do a simple list comprehension on the dataset, I get `Too many open files error`. This is happening due to the way we are loading the image feature when a str path is returned from the `_generate_examples`. Specifically at https://github.com/huggingface/datasets/blob/508eb4ab5d52f590baa677b4f64b1cc069139f7b/src/datasets/features/image.py#L110, we are open the file handle to the image but never closing it. This in my understanding is causing the issue.

## Steps to reproduce the bug
Pull the PR locally and run the following code
```python
from datasets import load_dataset

dataset = load_dataset("./datasets/textvqa")["train"]
data = [item for item in dataset]

# Error happens
```

## Expected results

List comprehension should work smoothly

## Actual results

`Too many open files error`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.1.dev0
- Platform: macOS-12.2-arm64-arm-64bit
- Python version: 3.10.0
- PyArrow version: 7.0.0
- Pandas version: 1.4.1
Exclude Google Drive tests of the CI

These tests make the CI spam the Google Drive API, the CI now gets banned by Google Drive very often.

I think we can just skip these tests from the CI for now.

In the future we could have a CI job that runs only once a day or once a week for such cases

cc @albertvillanova @mariosasko @severo 

Close #3415

![image](https://user-images.githubusercontent.com/42851186/159283608-fdeca1ac-b57f-4fa3-bf09-6fa5361c494f.png)


_The documentation is not available anymore as the PR was closed or merged._

I was thinking exactly the same: running unit tests that request continuously a third-party API is not a good idea.
Add tip on how to speed up loading with ImageFolder

This PR does two things:
* adds a tip on how to speed up loading of a large number of files with ImageFolder (motivated by [this issue](https://github.com/huggingface/datasets/issues/3960))
* replaces the current references to the `Dataset` methods in the Image Processing doc with their fully qualified counterparts (to align it with the Audio Processing doc)

cc @stevhliu 

_The documentation is not available anymore as the PR was closed or merged._

Thanks for adding that tip! 👍 

For the docs syntax, it might be better if we hide the package name/full path to the class or function and only show the name of it. I think it's easier for users to read the function name (eg,`cast_column`) instead of the full path which can be a bit lengthy for some functions like `datasets.IterableDataset.remove_columns` (and if we like this idea, we can align the rest of the docs on it). 

> For the docs syntax, it might be better if we hide the package name/full path to the class or function and only show the name of it. I think it's easier for users to read the function name (eg,cast_column) instead of the full path which can be a bit lengthy for some functions like datasets.IterableDataset.remove_columns (and if we like this idea, we can align the rest of the docs on it).

That's also OK, as long as we are consistent.

@lhoestq @albertvillanova @polinaeterna Which one of these two styles do you prefer?

Agree on hiding `datasets` name. Not sure about hiding class name as it's anyway not visible for users if they use `Dataset.cast_column` or  `IterableDataset.cast_column` when working with their datasets. But I agree that the most important thing is to be consistent :)

Good points! :)

I think it'll be good to show the class name since some functions have different parameters. For example, if users click on `IterableDataset.map` and then `Dataset.map`, they'll see different parameters and have to figure out why (which isn't too difficult I guess lol). But showing the class name avoids any confusion upfront.
Fix google drive streaming for small files

Google drive did another change recently, following #3787 #3843 .

In particular Google Drive now returns 403 for GET requests with `confirm=t` when a files doesn't have a virus warning message. I fixed this by passing `confirm=t` if and only if when there is one (i.e. when status code is 200 for HEAD)

_The documentation is not available anymore as the PR was closed or merged._

Actually the CI fails because of this
![image](https://user-images.githubusercontent.com/42851186/159281771-78e611b1-6b04-4a87-8324-b6ba2d8c6a6a.png)

It looks like we can't have a proper way to test google drive in the CI right now. Though it seems to work locally if you're not banned. I think I'll just disable those tests for now

this fix will not be included?

No we can't do anything except stop using google drive when possible
Fix main classes reference in docs

Currently the section index (on the page's right side) of the [main classes reference](https://huggingface.co/docs/datasets/master/en/package_reference/main_classes) incorrectly displays `Tensor returned:`, this PR fixes this issue by wrapping code examples in this page with markdown code block.

There are other examples in datasets library having this issue.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3976). All of your documentation changes will be reflected on that endpoint.

Not sure why some section titles end with `[[datasets.xxx]]`, like this: https://moon-ci-docs.huggingface.co/docs/datasets/pr_3976/en/package_reference/main_classes#datasetdict[[datasets.datasetdict]]

Thanks ! I think this has been fixed already in https://github.com/huggingface/datasets/pull/3925 though

I'm closing this one then if it's fine for you
Add XFUN dataset

This PR adds XFUN dataset.

Home page and repository: https://github.com/doc-analysis/XFUND

Source code: https://github.com/microsoft/unilm/blob/master/layoutlmft/layoutlmft/data/datasets/xfun.py

_The documentation is not available anymore as the PR was closed or merged._

Not sure how to generate dummy data.

The downloaded file structure is 

- document file paths
    - (a json file containing all documents info, document images folder)
    - (a json file containing all documents info, document images folder)
    - ...

Hey @mariosasko, thanks for the review. I'm not sure how to suggest these changes to the owner @ranpox, and I did spend some time to write the model card and hope to get it on the official repo. Is that possible?

Since the author is not responding, maybe we can go ahead with this PR ?

Go for it!

On Tue, Apr 12, 2022 at 10:24 AM Quentin Lhoest ***@***.***>
wrote:

> Since the author is not responding, maybe we can go ahead with this PR ?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/3974#issuecomment-1096797650>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ATFNL66EVUFWS3P2FOAS7SLVEWBP3ANCNFSM5RFH3MXA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


@qqaatw Do you plan to finish this PR? I can give you some pointers and help you with the code if needed.

@mariosasko Yes, I'll apply all of the suggestions when I have some time.

Thanks for your contribution, @qqaatw.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you propose this changes there to the original repo. Please, feel free to tell us if you need some help.