josteece/github-issues huggingface dataset:
Dataset.unique(column: str) -> ArrowNotImplementedError
Opened by orkenstein (closed)

## Describe the bug
I'm trying to use `unique()` function, but it fails
## Steps to reproduce the bug
1. Get dataset
2. Call `unique`
3. Error

# Sample code to reproduce the bug
```python
!pip show datasets

from datasets import load_dataset

dataset = load_dataset('wikiann', 'en')

dataset['train'].column_names
dataset['train'].unique(dataset['train'].column_names[0])
```

## Expected results
It would be nice to actually see unique items

## Actual results
Error:
```python
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
[<ipython-input-10-5e0de07ed42c>](https://s0qyv2vjaji-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20220324-060046-RC00_436956229#) in <module>()
      6 
      7 dataset['train'].column_names
----> 8 dataset['train'].unique(dataset['train'].column_names[0])

5 frames
/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Function unique has no kernel matching input types (array[list<item: string>])
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Google Collab
- Python version: 3.7.13
- PyArrow version: 6.0.1


Comments:
Hi @orkenstein, thanks for reporting.

Please note that for this case, our `datasets` library uses under the hood the Apache Arrow `unique` function: https://arrow.apache.org/docs/python/generated/pyarrow.compute.unique.html#pyarrow.compute.unique

And currently the Apache Arrow `unique` function is only implemented for these input types (see info in their [docs](https://arrow.apache.org/docs/cpp/compute.html#array-wise-vector-functions)): Boolean, Null, Numeric, Temporal, Binary- and String-like.

However, the data types of the `wikiann` dataset are all `list<item: string>` (see its [dataset card](https://huggingface.co/datasets/wikiann#data-fields)), and thus, not yet supported by the Apache Arrow `unique` function.

As a workaround solution you can use pandas:
```python
from datasets import load_dataset

dataset = load_dataset('wikiann', 'en', split='train')
df = dataset.to_pandas()
unique_df = df[~df.tokens.apply(tuple).duplicated()]  # from https://stackoverflow.com/a/46958336/17517845
```

Note that pandas loads the dataset in memory (this one is small so it's fine).

@lhoestq thank you! I will fall back to this method for now
CLI dummy data generation is broken
Opened by albertvillanova (closed)

## Describe the bug
We get a TypeError when running CLI dummy data generation:
```shell
datasets-cli dummy_data datasets/<your-dataset-folder> --auto_generate
```
gives:
```
  File ".../huggingface/datasets/src/datasets/commands/dummy_data.py", line 361, in _autogenerate_dummy_data
    dataset_builder._prepare_split(split_generator)
TypeError: _prepare_split() missing 1 required positional argument: 'check_duplicate_keys'
```



Comments:
[DO NOT MERGE] Test doc-builder with skipped installation feature
Opened by lewtun (closed)

This PR is just for testing that we can build PR docs with changes made on the [`skip-install-for-real`](https://github.com/huggingface/doc-builder/tree/skip-install-for-real) branch of `doc-builder`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Fix in https://github.com/huggingface/doc-builder/pull/162 works as expected (docs build), closing this
Error while building documentation
Opened by albertvillanova (closed)

## Describe the bug
Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```



Comments:
After some investigation, maybe the bug is in `doc-builder`.

I've opened an issue there:
- huggingface/doc-builder#160

Fixed by @lewtun (thank you):
- huggingface/doc-builder@31fe6c8bc7225810e281c2f6c6cd32f38828c504
Fix building of documentation
Opened by albertvillanova (closed)

Documentation building is failing:
- https://github.com/huggingface/datasets/runs/5716300989?check_suite_focus=true

```
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Unable to find datasets.filesystems.S3FileSystem in datasets. Make sure the path to that object is correct.
```

Fix #4037.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Superseded by huggingface/doc-builder@31fe6c8bc7225810e281c2f6c6cd32f38828c504
can't download cats_vs_dogs dataset
Opened by RRaphaell (closed)

## Describe the bug
can't download cats_vs_dogs dataset. error: Checksums didn't match for dataset source files

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("cats_vs_dogs")
```

## Expected results
loaded successfully.

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_3367a.zip']

## Environment info
fresh google colab notebook


Comments:
Thnaks for reporting @RRaphaell.

We are fixing it.
Cannot load the dataset conll2012_ontonotesv5
Opened by cathyxl (closed)

## Describe the bug
Cannot load the dataset conll2012_ontonotesv5

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import load_dataset
dataset = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test")
print(dataset)

```

## Expected results
The datasets should be downloaded successfully

## Actual results
raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/zmycy7t9h9-1.zip']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Linux-5.4.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 7.0.0



Comments:
Hi @cathyxl, thanks for reporting.

Indeed, we have recently updated the loading script of that dataset (and fixed that bug as well):
- #4002

That fix will be available in our next `datasets` library release. In the meantime, you can incorporate that fix by:
- installing `datasets` from our GitHub repo:
```bash
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
- forcing the data files to be redownloaded
```python
ds = load_dataset('conll2012_ontonotesv5', 'english_v4', split="test", download_mode="force_redownload")
```

Feel free to re-open this issue if the problem persists.
Use a constant for the articles regex in SQuAD v2
Opened by bryant1410 (closed)

The main reason for doing this is to be able to change the articles list if using another language, for example. It's not the most elegant solution but at least it makes the metric more extensible with no drawbacks.

BTW, what could be the best way to make this more generic (i.e., SQuAD in other languages)? Maybe receive a regex as an optional param, with the current value as the default? Similarly for SQuAD v1 (can't they re-use code?).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold
Opened by MoritzLaurer (closed)

**Is your feature request related to a problem? Please describe.**
I would like to retrieve all texts from a dataset, which are semantically similar to a specific input text (query), above a certain (cosine) similarity threshold. My dataset is very large (Wikipedia), so I need to use Datasets and FAISS for this. I would like to be able to repeat many different queries on the dataset quickly. 

**Describe the solution you'd like**
dataset objects currently have the .get_nearest_examples() method for text retrieval via FAISS. But this only allows retrieving a specific number of K texts instead of everything above a specified similarity threshold. 
It would be great if HF Datasets would also support the FAISS method .range_search() for retrieving texts above a certain similarity threshold. 
see details here: https://github.com/facebookresearch/faiss/issues/1273

**Describe alternatives you've considered**
I've considered using native FAISS, but doing this via HF datasets would be better. My assumption is that Dataset features like dataset streaming make it easier to work with large datasets


**Additional context**
The concrete use-case is: I have a large dataset (wikipedia) and I would like to retrieve all paragraphs which are similar to a query. I will use sentence-transformers for encoding the texts. 


Comments:
Hi ! You can access the faiss index with
```python
faiss_index = my_dataset.get_index("my_index_name").faiss_index
```
and then do whatever you want with it, e.g. query it using range_search:
```python
threshold = 0.95
limits, distances, indices = faiss_index.range_search(x=xq, thresh=threshold)

texts = dataset[indices]
```

wow, that's great, thank you for the explanation. (if that's not already in the documentation, could be worth adding it)

which type of faiss index is Datasets using? I looked into faiss recently and I understand that there are several different types of indexes and the choice is important, e.g. regarding which distance metric you use (euclidian vs. cosine/dot product), the size of my dataset etc. can I chose the type of index somehow as well?

`Dataset.add_faiss_index` has a `string_factory` parameter, used to set the type of index (see the faiss documentation about [index factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)). Alternatively, you can pass an index you've defined yourself using faiss with the `custom_index` parameter of `Dataset.add_faiss_index` 

Here is the full documentation of `Dataset.add_faiss_index`: https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index

great thanks, I will try it out
Fix docs on audio feature installation
Opened by albertvillanova (closed)

This PR:
- Removes the explicit installation of `librosa` (this is installed with `pip install datasets[audio]`
- Adds the warning for Linux users to install manually the non-Python package `libsndfile`
- Explains that the installation of `torchaudio` is only necessary to support loading audio datasets containing MP3 audio files

Related to #4000.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
ElasticSearch Indexing example: TypeError: __init__() missing 1 required positional argument: 'scheme'
Opened by MoritzLaurer (closed)

## Describe the bug
I am following the example in the documentation for elastic search step by step (on google colab): https://huggingface.co/docs/datasets/faiss_es#elasticsearch

```
from datasets import load_dataset
squad = load_dataset('crime_and_punish', split='train[:1000]')
```

When I run the line: 
`squad.add_elasticsearch_index("context", host="localhost", port="9200")`

I get the error: 
`TypeError: __init__() missing 1 required positional argument: 'scheme'`


## Expected results
No error message

## Actual results

```
TypeError                                 Traceback (most recent call last)

[<ipython-input-23-9205593edef3>](https://localhost:8080/#) in <module>()
      1 import elasticsearch
----> 2 squad.add_elasticsearch_index("text", host="localhost", port="9200")


6 frames

[/usr/local/lib/python3.7/dist-packages/elasticsearch/_sync/client/utils.py](https://localhost:8080/#) in host_mapping_to_node_config(host)
    209         options["path_prefix"] = options.pop("url_prefix")
    210 
--> 211     return NodeConfig(**options)  # type: ignore
    212 
    213 

TypeError: __init__() missing 1 required positional argument: 'scheme'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.2.0
- Platform: Linux, Google Colab
- Python version: Google Colab (probably 3.7)
- PyArrow version: ?


Comments:
Hi, @MoritzLaurer, thanks for reporting.

Normally this is due to a mismatch between the versions of your Elasticsearch client and server:
- your ES client is passing only keyword arguments to your ES server
- whereas your ES server expects a positional argument called 'scheme'

In order to fix this, you should align the major versions of both Elasticsearch client and server.

You can have more info:
- on this other issue page: https://github.com/huggingface/datasets/issues/3956#issuecomment-1072115173
- Elasticsearch client docs: https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/overview.html#_compatibility

Feel free to re-open this issue if the problem persists.

Duplicate of:
- #3956

1. Check elasticsearch version
```
import elasticsearch
print(elasticsearch.__version__)
```
Ex: 7.9.1
2. Uninstall current elasticsearch package
`pip uninstall elasticsearch`
3. Install elasticsearch 7.9.1 package
`pip install elasticsearch==7.9.1`
Missing argument in precision/recall
Opened by Dref360 (closed)

**Is your feature request related to a problem? Please describe.**
[`sklearn.metrics.precision_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html) accepts an argument `zero_division`, but it is not available in [precision Metric](https://github.com/huggingface/datasets/blob/master/metrics/precision/precision.py#L117)

Same issue is present for Recall.

**Describe the solution you'd like**
Support for **kwargs or adding a new field for `zero_division`.

**Describe alternatives you've considered**
I could filter the warnings myself, but that is not ideal.

**Additional context**
I can make the requested changes if this is approved.

Comments:
Thanks for the suggestion, @Dref360.

We are adding that argument.
Doc: image_process small tip
Opened by FrancescoSaverioZuppichini (closed)

I've added a small tip in the `image_process` doc

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This tip is unnecessary, i.e., Pillow will already be installed since the `Image` feature requires it for encoding and decoding. Thanks anyway.

cc @stevhliu I've noticed we are missing the installation section in the doc (`pip install datasets[vision]`). I can add it myself.
Fix `map` remove_columns on empty dataset
Opened by lhoestq (closed)

On an empty dataset, the `remove_columns` parameter of `map` currently doesn't actually remove the columns:
```python
>>> ds = datasets.load_dataset("glue", "rte")
>>> ds_filtered = ds.filter(lambda x: x["label"] != -1)
>>> ds_mapped = ds_filtered.map(lambda x: x, remove_columns=["label"])
>>> print(repr(ds_mapped.column_names))
{
    'train': ['sentence1', 'sentence2', 'idx'],
    'validation': ['sentence1', 'sentence2', 'idx'],
    'test': ['sentence1', 'sentence2', 'label', 'idx']
}
```

I fixed this error and updated the tests

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Make yelp_polarity streamable
Opened by lhoestq (closed)

It was using `dl_manager.download_and_extract` on a TAR archive, which is not supported in streaming mode. I replaced this by `dl_manager.iter_archive`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The CI is failing because of the incomplete dataset card - this is unrelated to the goal of this PR so we can ignore it
Can not correctly parse the classes with imagefolder
Opened by YiSyuanChen (closed)

## Describe the bug
I try to load my own image dataset with imagefolder, but the parsing of classes is incorrect.

## Steps to reproduce the bug
I organized my dataset (ImageNet) in the following structure:
```
  - imagenet/
      - train/
          - n01440764/
              - ILSVRC2012_val_00000293.jpg
              - ...... 
          - n01695060/
          - ......
      - val/ 
          - n01440764/
          - n01695060/
          - ......
```

At first, I followed the instructions from the Huggingface [example](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-classification#using-your-own-data) to load my data as:
```
from datasets import load_dataset
data_files = {'train': 'imagenet/train', 'val': 'imagenet/val'}
ds = load_dataset("nateraw/image-folder", data_files=data_files, task="image-classification")
```
but it resulted following error (I mask my personal path as <PERSONAL_PATH>):
```
FileNotFoundError: Unable to find 'https://huggingface.co/datasets/nateraw/image-folder/resolve/main/imagenet/train' at <PERSONAL_PATH>/ImageNet/https:/huggingface.co/datasets/nateraw/image-folder/resolve/main
```

Next, I followed a recent issue #3960 to load data as:
```
from datasets import load_dataset
data_files = {'train': ['imagenet/train/**'], 'val': ['imagenet/val/**']}
ds = load_dataset("imagefolder", data_files=data_files, task="image-classification")
```
and the data can be loaded without error as: (I copy val folder to train folder for illustration)
```
>>> ds
DatasetDict({
    train: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
    val: Dataset({
        features: ['image', 'labels'],
        num_rows: 50000
    })
})
```
However, the parsed classes is wrong (should be 1000 classes):
```
>>> ds["train"].features
{'image': Image(decode=True, id=None), 'labels': ClassLabel(num_classes=1, names=['val'], id=None)}
```

## Expected results
I expect that the "labels" in ds["train"].features should contain 1000 classes.

## Actual results
The "labels" in ds["train"].features contains only 1 wrong class.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.0.0
- Platform: Ubuntu 18.04
- Python version: Python 3.7.12
- PyArrow version: 7.0.0


Comments:
I found that the problem arises because the image files in my folder are actually symbolic links (for my own reasons).  After modifications, the classes can now be correctly parsed. Therefore, I close this issue.

HI, I have a question. How much time did you load the ImageNet data files?
Cannot preview "hazal/Turkish-Biomedical-corpus-trM"
Opened by hazalturkmen (closed)

## Dataset viewer issue for '*hazal/Turkish-Biomedical-corpus-trM'

**Link:** *https://huggingface.co/datasets/hazal/Turkish-Biomedical-corpus-trM*

*I cannot see the dataset preview.*

```
Server Error

Status code:   400
Exception:     HTTPError
Message:       403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/hazal/Turkish-Biomedical-corpus-trM?full=true
```


Am I the one who added this dataset ? Yes


Comments:
Hi @hazalturkmen, thanks for reporting.

Note that your dataset repository does not contain any loading script; it only contains a data file named `tr_article_2`.

When there is no loading script but only data files, the `datasets` library tries to infer how to load the data by looking at the data file extensions. However, your data file does not have any extension.

Note that current supported data file extensions are: 'csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip'.

You have more info on our docs: [How to share a dataset](https://huggingface.co/docs/datasets/share).

thanks for reply :)
Fix None issue with Sequence of dict
Opened by lhoestq (closed)

`Features.encode_example` currently fails if it contains a sequence if dict like `Sequence({"subcolumn": Value("int32")})` and if `None` is passed instead of the dict.

```python
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 1310, in encode_example
    return encode_nested_example(self, example)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in encode_nested_example
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 973, in <dictcomp>
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/features/features.py", line 998, in encode_nested_example
    for k, (sub_schema, sub_objs) in zip_dict(schema.feature, obj):
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in zip_dict
    yield key, tuple(d[key] for d in dicts)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/utils/py_utils.py", line 207, in <genexpr>
    yield key, tuple(d[key] for d in dicts)
TypeError: 'NoneType' object is not subscriptable
```

I fixed this issue and updated the tests (this case was missing in the tests)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging since I'd like do do a patch release soon for this one
AMI load_dataset error: sndfile library not found
Opened by i-am-neo (closed)

## Describe the bug
Getting error message when loading AMI dataset.

## Steps to reproduce the bug
`python3 -c "from datasets import load_dataset; print(load_dataset('ami', 'headset-single', split='validation')[0])"

`

## Expected results
A clear and concise description of the expected results.

## Actual results
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/load.py", line 1707, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 595, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/neo/.virtualenvs/hubert/lib/python3.7/site-packages/datasets/builder.py", line 690, in _download_and_prepare
    ) from None
OSError: Cannot find data file. 
Original error:
sndfile library not found

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-4.19.0-18-cloud-amd64-x86_64-with-debian-10.11
- Python version: 3.7.3
- PyArrow version: 7.0.0


Comments:
Issue unresolved, see [4000](https://github.com/huggingface/datasets/issues/4009#issue-1179658611)