noamaanMulla-03/datasets-issues huggingface dataset:
Add FAISS .range_search() method for retrieving all texts from dataset above similarity threshold

**Is your feature request related to a problem? Please describe.**
I would like to retrieve all texts from a dataset, which are semantically similar to a specific input text (query), above a certain (cosine) similarity threshold. My dataset is very large (Wikipedia), so I need to use Datasets and FAISS for this. I would like to be able to repeat many different queries on the dataset quickly. 

**Describe the solution you'd like**
dataset objects currently have the .get_nearest_examples() method for text retrieval via FAISS. But this only allows retrieving a specific number of K texts instead of everything above a specified similarity threshold. 
It would be great if HF Datasets would also support the FAISS method .range_search() for retrieving texts above a certain similarity threshold. 
see details here: https://github.com/facebookresearch/faiss/issues/1273

**Describe alternatives you've considered**
I've considered using native FAISS, but doing this via HF datasets would be better. My assumption is that Dataset features like dataset streaming make it easier to work with large datasets


**Additional context**
The concrete use-case is: I have a large dataset (wikipedia) and I would like to retrieve all paragraphs which are similar to a query. I will use sentence-transformers for encoding the texts. 


Labels: name: enhancement

Hi ! You can access the faiss index with
```python
faiss_index = my_dataset.get_index("my_index_name").faiss_index
```
and then do whatever you want with it, e.g. query it using range_search:
```python
threshold = 0.95
limits, distances, indices = faiss_index.range_search(x=xq, thresh=threshold)

texts = dataset[indices]
```

wow, that's great, thank you for the explanation. (if that's not already in the documentation, could be worth adding it)

which type of faiss index is Datasets using? I looked into faiss recently and I understand that there are several different types of indexes and the choice is important, e.g. regarding which distance metric you use (euclidian vs. cosine/dot product), the size of my dataset etc. can I chose the type of index somehow as well?

`Dataset.add_faiss_index` has a `string_factory` parameter, used to set the type of index (see the faiss documentation about [index factory](https://github.com/facebookresearch/faiss/wiki/The-index-factory)). Alternatively, you can pass an index you've defined yourself using faiss with the `custom_index` parameter of `Dataset.add_faiss_index` 

Here is the full documentation of `Dataset.add_faiss_index`: https://huggingface.co/docs/datasets/v2.0.0/en/package_reference/main_classes#datasets.Dataset.add_faiss_index

great thanks, I will try it out
Replace data URL in SAMSum dataset and support streaming

This PR replaces data URL in SAMSum dataset:
- original host (arxiv.org) does not allow HTTP Range requests
- we have hosted the data on the Hub (license: CC BY-NC-ND 4.0)

Moreover, it implements support for streaming.

Fix #4146.
Related to: #4236.

CC: @severo 


_The documentation is not available anymore as the PR was closed or merged._
Support to load dataset with TSV files by passing only dataset name

This PR implements support to load a dataset (w/o script) containing TSV files by passing only the dataset name (no need to pass `sep='\t'`):
```python
ds = load_dataset("dataset/name")
```

The refactoring allows for future builder kwargs customizations based on file extension.

Related to #4238.


_The documentation is not available anymore as the PR was closed or merged._
Update GH template for dataset viewer issues

Update template to use new issue forms instead.

With this PR we can check if this new feature is useful for us.

Once validated, we can update the other templates.

CC: @severo 


_The documentation is not available anymore as the PR was closed or merged._

You can see rendering at: https://github.com/huggingface/datasets/blob/6b48fedbdafe12a42c7b6edcecc32820af1a4822/.github/ISSUE_TEMPLATE/dataset-viewer.yml
Don't do unnecessary list type casting to avoid replacing None values by empty lists

In certain cases, `None` values are replaced by empty lists when casting feature types.

It happens every time you cast an array of nested lists like [None, [0, 1, 2, 3]] to a different type (to change the integer precision for example). In this case you'd get [[], [0, 1, 2, 3]] for example. This issue comes from PyArrow, see the discussion in https://github.com/huggingface/datasets/issues/3676

This issue also happens when no type casting is needed, because casting is supposed to be a no-op in this case. But as https://github.com/huggingface/datasets/issues/3676 shown, it's not the case and `None` are replaced by empty lists even if we cast to the exact same type.

In this PR I just workaround this bug in the case where no type casting is needed. In particular, I only call `pa.ListArray.from_arrays` only when necessary.

I also added a warning when some `None` are effectively replaced by empty lists. I wanted to raise an error in this case, but maybe we should wait a major update to do so

This PR fixes this particular case, that is occurring in `run_qa.py` in `transformers`:
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
# before:
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
#
# now:
#              b
# 0  [None, [0]]
# 1  [None, [0]]
# 2  [None, [0]]
# 3  [None, [0]]
```

cc @sgugger 


_The documentation is not available anymore as the PR was closed or merged._

Quick question about the message in the warning. You say "will be fixed in a future major version" but don't you mean "will raise an error in a future major version"?

Right ! Good catch, thanks, I updated the message to say "will raise an error in a future major version"
Autoeval config

Added autoeval config to imdb as pilot


_The documentation is not available anymore as the PR was closed or merged._

Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424

The tests are failing due to the changed metadata:

```
got an unexpected keyword argument 'train-eval-index'
```

I think you can fix this by updating the `DatasetMetadata` class and implementing an appropriate `validate_train_eval_index()` function

@lhoestq we are working with an arbitrary set of tags for `autoeval config`. See https://github.com/huggingface/autonlp-backend/issues/414
I need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this?

On the question of validating the `train-eval-index` metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).

For me, the required fields include:

* `config`
* `task`
* `task_id`
* `splits` (train / validation / eval)
* `col_mapping`
* `metrics` (checking that each one has `type`, `name`) 

Here I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.

WDYT @lhoestq ?

Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it

I ended up improving the metadata validation in this PR x)

In particular:
- I added support YAML keys with dashes instead of underscores for `train-eval-index`
- I added `train-eval-index` validation  with `validate_train_eval_index`. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want

Let me know if it sounds good to you ! I think we can improve `validate_train_eval_index` in another PR

Come on windows... I didn't do anything advanced...

Anyway, will try to fix this when I get back home x)

> Come on windows... I didn't do anything advanced...
> 
> Anyway, will try to fix this when I get back home x)

Hehe, thanks!

Thanks, @lhoestq this is great! 

Did I just fix it for windows and now it fails on linux ? xD

> Did I just fix it for windows and now it fails on linux ? xD

Looks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅 

The worst is that the tests pass locally both on my windows and my linux x)

Ok fixed it, the issue came from python 3.6 that doesn't return the right `__origin__` for Dict and List types

> Alright thanks for adding the first Autoeval config ! :D

Woohoo! Thank you so much 🤗 

This is cool!
Enable label alignment for token classification datasets

This PR extends the `Dataset.align_labels_with_mapping()` method to support alignment of label mappings between datasets and models for token classification (e.g. NER).

Example of usage:

```python
from datasets import load_dataset

ner_ds = load_dataset("conll2003", split="train")
# returns [3, 0, 7, 0, 0, 0, 7, 0, 0]
ner_ds[0]["ner_tags"]
# hypothetical model mapping with O <--> B-LOC
label2id = {
    "B-LOC": "0",
    "B-MISC": "7",
    "B-ORG": "3",
    "B-PER": "1",
    "I-LOC": "6",
    "I-MISC": "8",
    "I-ORG": "4",
    "I-PER": "2",
    "O": "5"
  }
ner_aligned_ds = ner_ds.align_labels_with_mapping(label2id, "ner_tags")
# returns [3, 5, 7, 5, 5, 5, 7, 5, 5]
ner_aligned_ds[0]["ner_tags"]
```

Context: we need this in AutoTrain to automatically align datasets / models during evaluation. cc @abhishekkrthakur 


_The documentation is not available anymore as the PR was closed or merged._

Hmm, not sure why the Windows tests are failing with:

```
Did not find path entry C:\tools\miniconda3\bin
C:\tools\miniconda3\envs\py37\python.exe: No module named pytest
```

Edit: running the CI again fixed the problem 🙃 

> One last nit and we can merge then

Thanks, done!
wmt19 is broken (zh-en)

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wmt19", 'zh-en')
```

## Expected results
The dataset should download.

## Actual results
`ConnectionError: Couldn't reach ftp://cwmt-wmt:cwmt-wmt@datasets.nju.edu.cn/parallel/casia2015.zip`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux
- Python version: 3.8 


Labels: name: bug

name: dataset bug

This issue is not reproducible.
Adjusting chunk size for streaming datasets

**Is your feature request related to a problem? Please describe.**
I want to use mc4 which I cannot save locally, so I stream it. However, I want to process the entire dataset and filter some documents from it. With the current chunk size of around 1000 documents (right?) I hit a performance bottleneck because of the frequent decompressing. 

**Describe the solution you'd like**
I would appreciate a parameter in the load_dataset function, that allows me to set the chunksize myself (to a value like 100'000 in my case). Like that, I hope to improve the processing time.



Labels: name: enhancement

Hi ! Data streaming uses `fsspec` to read the data files progressively. IIRC the block size for buffering is 5MiB by default. So every time you finish iterating over a block, it downloads the next one. You can still try to increase the `fsspec` block size for buffering if it can help. To do so you just need to increase `fsspec.spec.AbstractBufferedFile.DEFAULT_BLOCK_SIZE `

Currently this is unfortunately done in a single thread, so it blocks the processing to download and uncompress the next block. At one point it would be nice to be able to do that in parallel !

Hi! Thanks for the help, I will try it :)
Dataset Viewer issue for strombergnlp/ipm_nel : preview is empty, no error message

### Link

https://huggingface.co/datasets/strombergnlp/ipm_nel/viewer/ipm_nel/train

### Description

The viewer is blank. I tried my best to emulate a dataset with a working viewer, but this one just doesn't seem to want to come up. What did I miss?

### Owner

Yes

Labels: name: dataset-viewer

Hi @leondz, thanks for reporting.

Indeed, the dataset viewer relies on the dataset being streamable (passing `streaming=True` to `load_dataset`). Whereas most of the datastes are streamable out of the box (thanks to our implementation of streaming), there are still some exceptions.

In particular, in your case, that is due to the data file being TAR. This format is not streamable out of the box (it does not allow random access to the archived files), but we use a trick to allow streaming: using `dl_manager.iter_archive`.

Let me know if you need some help: I could push a commit to your repo with the fix.

Ah, right! The preview is working now, but this explanation is good to know, thank you. I'll prefer formats with random file access supported in datasets.utils.extract in future, and try out this fix for the tarfiles :)
Datasets YAML tagging space is down

## Describe the bug
The neat hf spaces app for generating YAML tags for dataset `README.md`s is down

## Steps to reproduce the bug
1. Visit https://huggingface.co/spaces/huggingface/datasets-tagging

## Expected results
There'll be a HF spaces web app for generating dataset metadata YAML

## Actual results
There's an error message; here's the step where it breaks:

```
Step 18/29 : RUN pip install -r requirements.txt
 ---> Running in e88bfe7e7e0c
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/huggingface/datasets.git@update-task-list (from -r requirements.txt (line 4))
  Cloning https://github.com/huggingface/datasets.git (to revision update-task-list) to /tmp/pip-req-build-bm8t0r0k
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/datasets.git /tmp/pip-req-build-bm8t0r0k
  WARNING: Did not find branch or tag 'update-task-list', assuming revision or ref.
  Running command git checkout -q update-task-list
  error: pathspec 'update-task-list' did not match any file(s) known to git
  error: subprocess-exited-with-error
  
  × git checkout -q update-task-list did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git checkout -q update-task-list did not run successfully.
│ exit code: 1
╰─> See above for output.
```

## Environment info

- Platform: Linux / Brave


Labels: name: bug

@lhoestq @albertvillanova `update-task-list` branch does not exist anymore, should point to `main` now i guess

Thanks for reporting, fixing it now

It's up again :)
Altered faiss installation comment



_The documentation is not available anymore as the PR was closed or merged._

Hi ! Can you explain why this change is needed ?

Facebook recommends installing FAISS using conda (https://github.com/facebookresearch/faiss/blob/main/INSTALL.md). pip does not seem to have the latest version of FAISS. The latest version of faiss is 1.7.2 (https://anaconda.org/conda-forge/faiss), but the latest one available through pip is 1.5.3 (https://pypi.org/project/faiss/).
Language code search does direct matches

## Describe the bug

Hi. Searching for bcp47 tags that are just the language prefix (e.g. `sq` or `da`) excludes datasets that have added extra information in their language metadata (e.g. `sq-AL` or `da-bornholm`). The example codes given in the [tagging app](https://huggingface.co/spaces/huggingface/datasets-tagging) encourages addition of the additional codes ("_expected format is BCP47 tags separated for ';' e.g. 'en-US;fr-FR'_") but this would lead to those datasets being hidden in datasets search.

## Steps to reproduce the bug
1. Add a dataset using a variant tag (e.g. [`sq-AL`](https://huggingface.co/datasets?languages=languages:sq-AL))
2. Look for datasets using the full code 
3. Note that they're missing when just the language is searched for (e.g. [`sq`](https://huggingface.co/datasets?languages=languages:sq))

Some datasets are already affected by this - e.g. `AmazonScience/massive` is listed under `sq-AL` but not `sq`.

One workaround is for dataset creators to add an additional root language tag to dataset YAML metadata, but it's unclear how to communicate this. It might be possible to index the search on `languagecode.split('-')[0]` but I wanted to float this issue before trying to write any code :)

## Expected results
Datasets using longer bcp47 tags also appear under searches for just the language code; e.g. Quebecois datasets (`fr-CA`) would come up when looking for French datasets with no region specification (`fr`), or US English (`en-US`) datasets would come up when searching for English datasets (`en`).

## Actual results
The language codes seem to be directly string matched, excluding datasets with specific language tags from non-specific searches.

## Environment info
(web app)

Labels: name: bug

Thanks for reporting ! I forwarded the issue to the front-end team :)

Will keep you posted !

I also changed the tagging app to suggest two letters code for now.
[Imagefolder] Docs + Don't infer labels from file names when there are metadata + Error messages when metadata and images aren't linked correctly

I updated the `docs/source/image_process.mdx` documentation and added an example for image captioning and object detection using `ImageFolder`.

While doing so I also improved a few aspects:
- we don't need to infer labels from file names when there are metadata - they can just be in the metadata if necessary
- raise informative error messages when metadata and images aren't linked correctly:
  - when an image is missing a metadata file
  - when a metadata file is missing an image

I added some tests for these changes as well

cc @mariosasko 


_The documentation is not available anymore as the PR was closed or merged._

Merging this one since mario is off, I took care of adding some tests to make sure everything is fine. Will do the release after it
Fix cnn_dailymail (dm stories were ignored)

https://github.com/huggingface/datasets/pull/4188 introduced a bug in `datasets` 2.2.0: DailyMail stories are ignored when generating the dataset.

I fixed that, and removed the google drive link (it has annoying quota limitations issues)

We can do a patch release after this is merged


_The documentation is not available anymore as the PR was closed or merged._
Task search function on hub not working correctly

When I want to look at all datasets of the category: `speech-processing`  *i.e.* https://huggingface.co/datasets?task_categories=task_categories:speech-processing&sort=downloads , then the following dataset doesn't show up for some reason:

- https://huggingface.co/datasets/speech_commands

even thought it's task tags seem correct: 
https://raw.githubusercontent.com/huggingface/datasets/master/datasets/speech_commands/README.md

Labels: name: bug

known issue due to https://github.com/huggingface/datasets/pull/2362 (and [internal](https://github.com/huggingface/moon-landing/issues/946)) , will be solved soon

hmm actually i have no recollection of why I said that

Because it has dots in some YAML keys, it can't be parsed and indexed by the back-end
Failing CI on Windows for sari and wiki_split metrics

## Describe the bug
Our CI is failing from yesterday on Windows for metrics: sari and wiki_split
```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_sari - ...
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_wiki_split
```

See: https://app.circleci.com/pipelines/github/huggingface/datasets/11928/workflows/79daa5e7-65c9-4e85-829b-00d2bfbd076a/jobs/71594

Labels: name: bug
Dataset Viewer issue for strombergnlp/offenseval_2020, strombergnlp/polstance

### Link

https://huggingface.co/datasets/strombergnlp/offenseval_2020/viewer/ar/train

### Description

The viewer isn't running for these two datasets. I left it overnight because a wait sometimes helps things get loaded, and the error messages have all gone, but the datasets are still turning up blank in viewer. Maybe it needs a bit more time.

* https://huggingface.co/datasets/strombergnlp/polstance/viewer/PolStance/train

* https://huggingface.co/datasets/strombergnlp/offenseval_2020/viewer/ar/train

While offenseval_2020 is gated w. prompt, the other gated previews I have run fine in Viewer, e.g. https://huggingface.co/datasets/strombergnlp/shaj , so I'm a bit stumped!

### Owner

Yes

Labels: name: dataset-viewer

Not sure if it's related... I was going to raise an issue for https://huggingface.co/datasets/domenicrosati/TruthfulQA which also has the same issue... https://huggingface.co/datasets/domenicrosati/TruthfulQA/viewer/domenicrosati--TruthfulQA/train 



Yes, it's related. The backend behind the dataset viewer is currently under too much load, and these datasets are still in the jobs queue. We're actively working on this issue, and we expect to fix the issue permanently soon. Thanks for your patience 🙏  

Thanks @severo and no worries! - a suggestion for a UI usability thing maybe is to indicate that the dataset processing is in the job queue (rather than no data?)

Thanks, these are working great now (including @domenicrosati 's, afaics!)
Support >1 PWC dataset per dataset card

**Is your feature request related to a problem? Please describe.**
Some datasets cover more than one dataset on PapersWithCode. For example, the OffensEval 2020 challenge involved five languages, and there's one dataset to cover all five datasets, [`strombergnlp/offenseval_2020`](https://huggingface.co/datasets/strombergnlp/offenseval_2020). However, the yaml `paperswithcode_id:` dataset card entry only supports one value; when multiple are added, the PWC link disappears from the dataset page.

Because the link from a PapersWithCode dataset to a Hugging Face Hub entry can't be entered manually and seems to be scraped, this means end users don't have a way of getting a dataset reader link to appear on all the PWC datasets supported by one HF Hub Dataset reader.

It's not super unusual to have papers introduce multiple parallel variants of a dataset and would be handy to reflect this, so e.g. dataset maintainers can DRY, and so dataset users can keep what they're doing simple.

**Describe the solution you'd like**
I'd like `paperswithcode_id:` to support lists and be able to connect with multiple PWC datasets.

**Describe alternatives you've considered**
De-normalising the datasets on HF Hub to create multiple readers for each variation on a task, i.e. instead of a single `offenseval_2020`, having `offenseval_2020_ar`, `offenseval_2020_da`, `offenseval_2020_gr`, ...

**Additional context**
Hope that's enough

**Priority**
Low

Labels: name: enhancement

Hi @leondz, I agree it would be nice. We'll see what we can do ;)
Fix never ending GH Action to build documentation

There was an unclosed code block introduced by:
- #4313 

https://github.com/huggingface/datasets/pull/4313/files#diff-f933ce41f71c6c0d1ce658e27de62cbe0b45d777e9e68056dd012ac3eb9324f7R538 

This causes the "Make documentation" step in the "Build documentation" workflow to never finish.
- I think this issue should also be addressed in the `doc-builder` lib.


Fix #4346.


_The documentation is not available anymore as the PR was closed or merged._