noamaanMulla-03/datasets-issues huggingface dataset:
`FaissIndex.save` throws error on GPU

## Describe the bug

After training an index with a factory string `OPQ16_128,IVF512,PQ32` on GPU, `.save_faiss_index` throws this error.

```
  File "index_wikipedia.py", line 119, in <module>
    data["train"].save_faiss_index("text_emb", index_save_path)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 470, in save_faiss_index
    index.save(file)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 334, in save
    faiss.write_index(index, str(file))
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/faiss/swigfaiss_avx2.py", line 5654, in write_index
    return _swigfaiss.write_index(*args)
RuntimeError: Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /root/miniconda3/conda-bld/faiss-pkg_1613235005464/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index
```

## Steps to reproduce the bug

Any dataset will do, I just selected a familiar one.

```python
import numpy as np
import datasets
INDEX_STR = "OPQ16_128,IVF512,PQ32"
INDEX_SAVE_PATH = "will_not_save.faiss"

data = datasets.load_dataset("Fraser/news-category-dataset", split=f"train[:10000]")

def encode(item):
    return {"text_emb": np.random.randn(768).astype(np.float32)}

data = data.map(encode)

data.add_faiss_index(column="text_emb", string_factory=INDEX_STR, train_size=10_000, device=0)
data.save_faiss_index("text_emb", INDEX_SAVE_PATH)
```

## Expected results
Saving the index

## Actual results
Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) ... don't know how to serialize this type of index

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


I will be proposing a fix in a couple of minutes

Labels: name: bug

Just in case, this is a workaround that I use in my code and it seems to do the job.

```python
if use_gpu_index:
    data["train"]._indexes["text_emb"].faiss_index = faiss.index_gpu_to_cpu(data["train"]._indexes["text_emb"].faiss_index)
```
Fix web_nlg metadata

Our metadata storage system does not support `.` inside keys. cc @Pierrci 




Hi ! `release_v2.1` and the others are dataset configuration names.

The configuration names are used to show the right code snippet in the UI to load the dataset.
For example if the parsing of the web_nlg tags worked correctly we would have:
![image](https://user-images.githubusercontent.com/42851186/118475444-8d1e5d00-b70c-11eb-98e9-844d4daf6139.png)

Therefore I don't think it's a good idea to rename the configurations from `release_v2.1` to `release_v2_1` as the code snippet would be wrong in this case.

Moreover we can't really disallow dots in configuration names and rename the configurations since it would be a big breaking change. It's commonly used, especially with multilingual datasets. For example `load_dataset("indic_glue", "sna.bn")`.

Is this something that can be fixed on the moonlanding side instead ?

> Is this something that can be fixed on the moonlanding side instead ?

Not really unless we change database:)

We'll maybe try to find another workaround, but super low-prio given that it's the only dataset that has those dotted keys in the YAML metadata

Ok, should we close this PR then ?
README updated for SNLI, MNLI

Closes #2275. Mentioned about -1 labels in MNLI, SNLI and how they should be removed before training. @lhoestq `check_code_quality` test might fail for MNLI as the license name `other-Open Portion of the American National Corpus`  is not a registered tag for 'licenses'


Regarding the license issue, I think we should allow it since it starts with `other-`. Cc @gchhablani what do you think ?

@lhoestq I agree, I'll look into it.
Add Qasper Dataset

[Question Answering on Scientific Research Papers](https://allenai.org/project/qasper/home)

Doing NLP on NLP papers to do NLP ♻️ I had to add it~

- [x] Add README (just gotta fill out some more )
- [x] Dataloader code
- [x] Make dummy dataset
- [x] generate dataset infos
- [x] Tests



I saw that the README [template](https://github.com/huggingface/datasets/blob/master/templates/README.md) changed while I was working on this 😅 Some TOC titles may be different but I filled it to the best of my knowledge & readme quality check passes now.
ready for review @lhoestq
Loading dataset from local path

I'm trying to load a local dataset with the code below

```
ds = datasets.load_dataset('my_script.py', 
                           data_files='corpus.txt', 
                           data_dir='/data/dir', 
                           cache_dir='.')
```
But internally a BuilderConfig is created, which tries to use getmtime on the data_files string, without using data_dir. Is this a bug or am I not using the load_dataset correctly?

https://github.com/huggingface/datasets/blob/bc61954083f74e6460688202e9f77dde2475319c/src/datasets/builder.py#L153


Version below works, checked again in the docs, and data_files should be a path.
```
ds = datasets.load_dataset('my_script.py', 
                           data_files='/data/dir/corpus.txt', 
                           cache_dir='.')
```
Add missing dataset_infos.json files

Some of the datasets in `datasets` are missing a `dataset_infos.json` file, e.g.

```
[PosixPath('datasets/chr_en/chr_en.py'), PosixPath('datasets/chr_en/README.md')]
[PosixPath('datasets/telugu_books/README.md'), PosixPath('datasets/telugu_books/telugu_books.py')]
[PosixPath('datasets/reclor/README.md'), PosixPath('datasets/reclor/reclor.py')]
[PosixPath('datasets/json/README.md')]
[PosixPath('datasets/csv/README.md')]
[PosixPath('datasets/wikihow/wikihow.py'), PosixPath('datasets/wikihow/README.md')]
[PosixPath('datasets/c4/c4.py'), PosixPath('datasets/c4/README.md')]
[PosixPath('datasets/text/README.md')]
[PosixPath('datasets/lm1b/README.md'), PosixPath('datasets/lm1b/lm1b.py')]
[PosixPath('datasets/pandas/README.md')]
```

For `json`, `text`, csv`, and `pandas` this is expected, but not for the others which should be fixed


Labels: name: enhancement
Roman Urdu Stopwords List

A list of most frequently used Roman Urdu words with different spellings and usages.
This is a very basic effort to collect some basic stopwords for Roman Urdu to help efforts of analyzing text data in roman Urdu which makes up a huge part of daily internet interaction of Roman-Urdu users.


Hi ! Thanks for sharing :)
I think the best place to share this is probably the `Languages at Hugging Face` section of the forum:
https://discuss.huggingface.co/c/languages-at-hugging-face/15

Since this is not a dataset, I'm closing this PR if you don't mind

Thank you I will look into the link that you have shared with me.




On Mon, May 17, 2021 at 7:05 PM Quentin Lhoest ***@***.***>
wrote:

> Closed #2358 <https://github.com/huggingface/datasets/pull/2358>.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/2358#event-4754836267>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AN7SJYJVY4C5XQRDNET743DTOEPC7ANCNFSM443AZ3MA>
> .
>
maintain YAML structure reading from README

How YAML used be loaded earlier in the string (structure of YAML was affected because of this and YAML for datasets with multiple configs was not being loaded correctly):
```
annotations_creators:
labeled_final:
- expert-generated
labeled_swap:
- expert-generated
unlabeled_final:
- machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
labeled_final:
- 10K<n<100K
labeled_swap:
- 10K<n<100K
unlabeled_final:
- 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```
How YAML is loaded in string now:
```
annotations_creators:
  labeled_final:
  - expert-generated
  labeled_swap:
  - expert-generated
  unlabeled_final:
  - machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
  labeled_final:
  - 10K<n<100K
  labeled_swap:
  - 10K<n<100K
  unlabeled_final:
  - 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```
A syntax error in example

![image](https://user-images.githubusercontent.com/6883957/117315905-b47a5c00-aeba-11eb-91eb-b2a4a0212a56.png)

Sorry to report with an image, I can't find the template source code of this snippet.

Labels: name: bug

cc @beurkinger but I think this has been fixed internally and will soon be updated right ?

This issue has been fixed.
[Question & Bug Report] Can we preprocess a dataset on the fly?

I know we can use `Datasets.map` to preprocess a dataset, but I'm using it with very large corpus which generates huge cache file (several TB cache from a 400 GB text file). I have no disk large enough to save it.  Can we preprocess a dataset on the fly without generating cache?

BTW, I tried raising `writer_batch_size`. Seems that argument doesn't have any effect when it's larger than `batch_size`, because you are saving all the batch instantly after it's processed. Please check the following code:

https://github.com/huggingface/datasets/blob/0281f9d881f3a55c89aeaa642f1ba23444b64083/src/datasets/arrow_dataset.py#L1532


We are very actively working on this. How does your dataset look like in practice (number/size/type of files)?

It's a text file with many lines (about 1B) of Chinese sentences. I use it to train language model using https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm_wwm.py

Indeed I will submit a PR in a fez days to enable processing on-the-fly :)
This can be useful in language modeling for tokenization, padding etc.


any update on this issue? ...really look forward to use it 

Hi @acul3,

Please look at the discussion on a related Issue #1825. I think using `set_transform` after building from source should do.

@gchhablani thank you so much

will try look at it
Set to_json default to JSON lines

With this PR, the method `Dataset.to_json`:
- is added to the docs
- defaults to JSON lines


This is perfect, @albertvillanova - thank you! Tested it to work.

Might it be a good idea to document the args to `to_json`?

and also even a very basic progress bar? took 10min for 8M large records for `openwebtext` so perhaps some indication of it's being alive every min or so?

@lhoestq I added tests for both `lines` and `orient`.
Add tests for dataset cards

Adding tests for dataset cards

This PR will potentially remove the scripts being used for dataset tags and readme validation.

Additionally, this will allow testing dataset readmes by providing the name as follows:

```bash
pytest tests/test_dataset_cards.py::test_dataset_tags[fashion_mnist]
```
and

```bash
pytest tests/test_dataset_cards.py::test_readme_content[fashion_mnist]
```
or a combined test as:

```bash
pytest tests/test_dataset_cards.py::test_dataset_card[fashion_mnist]
```
@lhoestq 


@lhoestq

Should I remove the scripts? or atleast remove running them from the CircleCI config?

Also, I hope it is okay that the combined method (metadata+content) is only a slow test, and for the Circle CI, I assume only non-slow tests are run? If yes, this would mean separate tests for content and metadata.

Also feel free to remove the scripts from the CI and also remove the scripts files :)
Accessing Arrow dataset cache_files

## Describe the bug
In datasets 1.5.0 the following code snippet would have printed the cache_files:

```
train_data = load_dataset('conll2003', split='train', cache_dir='data')
print(train_data.cache_files[0]['filename'])

```

However, in the newest release (1.6.1), it prints an empty list.

I also tried loading the dataset with `keep_in_memory=True` argument but still `cache_files` is empty.

Was wondering if this is a bug or I need to pass additional arguments so I can access the cache_files.


Labels: name: bug

Thanks @bhavitvyamalik for referencing the workaround. Setting `keep_in_memory=False` is working.
Add args description to DatasetInfo

Closes #2354 

I am not sure what `post_processed` and `post_processing_size` correspond to, so have left them empty for now. I also took a guess at some of the other fields like `dataset_size` vs `size_in_bytes`, so might have misunderstood their meaning.


Thanks for the suggestions! I've included them and made a few minor tweaks along the way

Please merge master into this branch to fix the CI, I just fixed metadata validation tests.
News_commentary Dataset Translation Pairs are of Incorrect Language Specified Pairs

I used load_dataset to load the news_commentary dataset for "ar-en" translation pairs but found translations from Arabic to Hindi.  

```
train_ds = load_dataset("news_commentary", "ar-en", split='train[:98%]')
val_ds = load_dataset("news_commentary", "ar-en", split='train[98%:]')

# filtering out examples that are not ar-en translations but ar-hi
val_ds = val_ds.filter(lambda example, indice: indice not in chain(range(1312,1327) ,range(1384,1399), range(1030,1042)), with_indices=True)
```

* I'm fairly new to using datasets so I might be doing something wrong


Hi @anassalamah,

Could you please try with this:
```python
train_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[:98%]')
val_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[98%:]')
```

Hello @albertvillanova, 

Thanks for the suggestion. I didn't know you could do that. however, it didn't resolve the issue

![image](https://user-images.githubusercontent.com/8571003/114169966-ec819400-993a-11eb-8a67-930f9a9b2290.png)
Concatenate several datasets with removed columns is not working.

## Describe the bug

You can't concatenate datasets when you removed columns before.

## Steps to reproduce the bug
```python
from datasets import load_dataset, concatenate_datasets

wikiann= load_dataset("wikiann","en")

wikiann["train"] = wikiann["train"].remove_columns(["langs","spans"])
wikiann["test"] = wikiann["test"].remove_columns(["langs","spans"])

assert wikiann["train"].features.type == wikiann["test"].features.type

concate = concatenate_datasets([wikiann["train"],wikiann["test"]])
```

## Expected results
Merged dataset 


## Actual results
```python
ValueError: External features info don't match the dataset:
Got
{'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'ner_tags': Sequence(feature=ClassLabel(num_classes=7, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'], names_file=None, id=None), length=-1, id=None), 'langs': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'spans': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
with type
struct<langs: list<item: string>, ner_tags: list<item: int64>, spans: list<item: string>, tokens: list<item: string>>

but expected something like
{'ner_tags': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
with type
struct<ner_tags: list<item: int64>, tokens: list<item: string>>
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: ~1.6.2~ 1.5.0
- Platform: macos
- Python version: 3.8.5
- PyArrow version: 3.0.0


Labels: name: bug

Hi,

did you fill out the env info section manually or by copy-pasting the output of the `datasets-cli env` command?

This code should work without issues on 1.6.2 version (I'm working on master (1.6.2.dev0 version) and can't reproduce this error).

@mariosasko you are right I was still on `1.5.0`.
Fix number of classes in indic_glue sna.bn dataset

As read in the [paper](https://www.aclweb.org/anthology/2020.findings-emnlp.445.pdf), Table 11.


@lhoestq there are many things missing in the README.md file, but this correction is right despite not passing the validation tests...

Yes indeed. We run the validation in all modified readme because we think that it is the time when contributors are the most likely to fix a dataset card - or it will never be done
ConvQuestions benchmark added

Hello,
I would like to integrate our dataset on conversational QA. The answers are grounded in the KG.
The work was published in CIKM 2019 (https://dl.acm.org/doi/10.1145/3357384.3358016).
We hope for further research on how to deal with the challenges of factoid conversational QA.
Thanks! :)


Thanks for your helpful comments and suggestions! :)
I integrated the additional fields, and extended some of the README/dataset card.
And I actually realized that we had the cc-by-4.0 for the dataset, so this was also changed.

I added the answers to the test set actually :)

Oh great ! Let me revert my change then
Paperswithcode dataset mapping

This is a continuation of https://github.com/huggingface/huggingface_hub/pull/43, encoded directly inside dataset cards.

As discussed:
- `paperswithcode_id: null` when the dataset doesn't exist on paperswithcode's side.
- I've added this new key at the end of the yaml instead of ordering all keys alphabetically as pyyaml's default. No strong opinion on that one though



messed up my branch, repushing

live mapping can be found at https://huggingface.co/api/pwc/datasets-mapping and will be kept up to date going forward
Allow passing `desc` to `tqdm` in `Dataset.map()`

It's normal to have many `map()` calls, and some of them can take a few minutes,
it would be nice to have a description on the progress bar.

Alternative solution:
Print the description before/after the `map()` call.

Labels: name: enhancement

name: good first issue

Hi @lhoestq,
Should we change `desc` in [pbar](https://github.com/huggingface/datasets/blob/81fcf88172ed5e3026ef68aed4c0ec6980372333/src/datasets/arrow_dataset.py#L1860) to something meaningful?

I think the user could pass the `desc` parameter to `map` so that it can be displayed in the tqdm progress bar, as suggested by @cccntu.

When there's no multiprocessing, the `desc` of the progress bar could be the `desc` passed by the user.
In multiprocessing, we were already using a `desc` equal to `"#" + str(rank)`.
We can change it to be `(desc or "") + "#" + str(rank)` instead.

In the end, since both `desc` and `rank` could be None, we can have:
```python
pbar_desc = (desc or "") + "#" + str(rank) if rank is not None else desc
```

Finally let's remember that if we add `desc` as a new parameter to `map`, we should add it to the `ignore_kwargs` list of the `@fingerprint_transform` decorator of `Dataset._map_single` since we don't want this parameter to affect the fingerprint of the resulting dataset.
PermissionError on Windows when using temp dir for caching

Currently, the following code raises a PermissionError on master if working on Windows:

```python
# run as a script or call exit() in REPL to initiate the temp dir cleanup
from datasets import *
d = load_dataset("sst", split="train", keep_in_memory=False)
set_caching_enabled(False)
d.map(lambda ex: ex)
```

Error stack trace:
```
Traceback (most recent call last):
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\weakref.py", line 624, in _exitfunc
    f()
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\weakref.py", line 548, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\tempfile.py", line 799, in _cleanup
    _shutil.rmtree(name)
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 500, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 395, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 393, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Mario\\AppData\\Local\\Temp\\tmp20epyhmq\\cache-87a87ffb5a956e68.arrow'
```

Labels: name: bug
Add env variable for MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES

Add env variable for `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES`.

This will allow to turn off default behavior: loading in memory (and not caching) small datasets.

Fix #2387.


Thank you for clarifying the precedence, @albertvillanova 

Isn't it typically the case where env vars have the highest precedence? 

In my understanding the point of env vars is to be able to override software w/o needing to touch the code. 

Please correct me if this is not so in the general case.

Hi @stas00, 

Well, I'm not an expert on this topic, but the precedence hierarchy I have normally found is from higher to lower:
- command line parameters
- env vars
- config files
So yes, normally env vars have precedence over configuration files.

Anyway, for Datasets, there are no configuration files. The _in-memory_ config is set from default values or env vars (which have precedence over default values). But this is done at import.

However, once the library is imported, the user can modify the in-memory config, and this will have precedence over the rest of mechanisms (which take place only at import).

In my limited experience env vars are typically above cmd line args, so that one can override scripts with cmd lines using env vars, but usually one then uses env vars inside cmd line args, so it's loud and clear.

For example specifying a specific gpu number on a command line will depend on `CUDA_VISIBLE_DEVICES` so gpu0 will be different if someone sets `CUDA_VISIBLE_DEVICES=2,3` vs  `CUDA_VISIBLE_DEVICES=0,1`.

> However, once the library is imported, the user can modify the in-memory config, and this will have precedence over the rest of mechanisms (which take place only at import).

And this is exactly the problem we are trying to solve here. For a good reason HF examples don't want to use `keep_in_memory=False`, and they may choose to now set `datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES` and which means we again can't override it via env var.



oops, sorry, didn't think earlier - do we need to prefix this with `HF_DATASETS` or `HF_` like all the other env vars? or is it long enough already to be unique - it's just not telling the user in the config file what projet this variable is for...

You're right, I just opened https://github.com/huggingface/datasets/pull/2409
Perform minor refactoring: use config

Perform minor refactoring related to `config`.

Labels: name: refactoring

@lhoestq is there a problem in the master branch? I got a segmentation fault...
```
tests/test_table.py::test_concatenation_table_cast[in_memory] Fatal Python error: Segmentation fault
```

Oh wow. Let me re-run the CI just to make sure

Hmm interesting, the segfault is still there. I'm investigating this issue on my windows machine

Feel free to merge master into this branch to fix the CI :)