helmo/github-issues huggingface dataset:
load_from_disk and save_to_disk are not compatible with each other
State: closed | Author: roholazandie (NONE)

## Describe the bug
load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks like the load_from_disk only works on one split

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("art")
dataset.save_to_disk("mydir")
d = Dataset.load_from_disk("mydir")
```

## Expected results
It is expected that these two functions be the reverse of each other without more manipulation

## Actual results
FileNotFoundError: [Errno 2] No such file or directory: 'mydir/art/state.json'

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>



Comments: Hi,

`load_dataset` returns an instance of `DatasetDict` if `split` is not specified, so instead of `Dataset.load_from_disk`, use `DatasetDict.load_from_disk` to load the dataset from disk.

Thanks it worked!

Though I see a stream of issues open by people lost between datasets and datasets dicts so maybe there is here something that could be better in terms of UX. Could be better error handling or something else smarter to even avoid said errors but maybe we should think about this. Reopening to use this issue as a discussion place but feel free to open a new open if you prefer @lhoestq @albertvillanova 

We should probably improve the error message indeed.

Also note that there exists a function `load_from_disk` that can load a Dataset or a DatasetDict. Under the hood it calls either `Dataset.load_from_disk` or `DatasetDict.load_from_disk`:


```python
from datasets import load_from_disk

dataset_dict = load_from_disk("path/to/dataset/dict")
single_dataset = load_from_disk("path/to/single/dataset")
```

I just opened #2437 to improve the error message

Superseded by #2462
add `desc` in `map` for `DatasetDict` object
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

`desc` in `map` currently only works with `Dataset` objects. This PR adds support for `DatasetDict` objects as well

Comments: The CI error is unrelated to the PR, merging

@lhoestq, can we release this feature if you guys are planning for any patch release for Datasets. It'll slow down [#11927](https://github.com/huggingface/transformers/pull/11927) otherwise :/ 

Sure definitely, having a discrepancy between Dataset.map and DatasetDict.map is an issue that we should fix and include in a patch release. Will do it in the coming days
adds license information for DailyDialog.
State: closed | Author: aditya2211 (CONTRIBUTOR)


Comments: Thanks! Can you also add it as metadata in the YAML block at the top of the file?

Should be in the form:

```
licenses:
- cc-by-sa-4.0
```

seems like we need to add all the other tags ? 
```       
if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           __init__() missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'languages', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
```

I'll let @lhoestq or @yjernite chime in (and maybe complete/merge). Thanks!

Looks like CircleCI has an incident. Let's wait for it to be working again and make sure the CI is green

The remaining error is unrelated to this PR, merging
add utf-8 while reading README
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

It was causing tests to fail in Windows (see #2416). In Windows, the default encoding is CP1252 which is unable to decode the character byte 0x9d 

Comments: Can you please add encoding to this line as well to fix the issue (and maybe replace `path.open(...)` with `open(path, ...)`)?
https://github.com/huggingface/datasets/blob/7bee4be44706a59b084b9b69c4cd00f73ee72f76/src/datasets/utils/metadata.py#L58

Sure, in fact even I was thinking of adding this in order to maintain the consistency!
Make datasets PEP-561 compliant
State: closed | Author: SBrandeis (CONTRIBUTOR)

Allows to type-check datasets with `mypy` when imported as a third-party library

PEP-561: https://www.python.org/dev/peps/pep-0561
MyPy doc on the subject: https://mypy.readthedocs.io/en/stable/installed_packages.html


Comments: This is super cool, I love that ❤️
Add KLUE dataset
State: closed | Author: jungwhank (CONTRIBUTOR)

Add `KLUE (Korean Language Understanding Evaluation)` dataset released recently from [paper](https://arxiv.org/abs/2105.09680), [github](https://github.com/KLUE-benchmark/KLUE) and [webpage](https://klue-benchmark.com/tasks).
Please let me know if there's anything missing in the code or README.
Thanks!




Comments: I'm not sure why I got error like below when I auto-generate dummy data "mrc" 
```
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

> I'm not sure why I got error like below when I auto-generate dummy data "mrc"
> 
> ```
> datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
> Found duplicate Key: 0
> Keys should be unique and deterministic in nature
> ```

Please check out the suggestion below. I think it might be a cause.

> > I'm not sure why I got error like below when I auto-generate dummy data "mrc"
> > ```
> > datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
> > Found duplicate Key: 0
> > Keys should be unique and deterministic in nature
> > ```
> 
> Please check out the suggestion below. I think it might be a cause.

The problem was `id_` in mrc when yield was not unique. (I used index in `enumerate(paragraphs)` by mistake)
I fixed it and update all the things

To fix the CI you can just merge master into your branch and it should be all green hopefully :)

@lhoestq
Thanks for reviewing!

It's harder than I thought to add dataset card. 😅 
I checked and updated your suggestion (script, readme details, dummy data). 

dummy data is little bit larger than expected because `ner` dataset is about 80 lines and `dp` dataset is about 25 lines to avoid 0 examples.

I'm not sure why some CI keep fails, can u check for this?

Thanks ! That makes sense for ner and dp

For mrc on the other hand there are still too many examples, maybe you can generate the dummy data for 5 examples for all tasks except ner and dp ?

> Thanks ! That makes sense for ner and dp
> 
> For mrc on the other hand there are still too many examples, maybe you can generate the dummy data for 5 examples for all tasks except ner and dp ?

Yes, I generate default lines in dataset-cli for other dataset except "dp" and "ner"
I fixed mrc dataset, hope it's fine now :)

the reason CI failed was I forgot to merge master into my branch 😅
AttributeError: 'DatasetInfo' object has no attribute 'task_templates'
State: closed | Author: jungwhank (CONTRIBUTOR)
Labels: name: bug

## Describe the bug
Hello, 
I'm trying to add dataset and contribute, but test keep fail with below cli.
` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<my_dataset>`

## Steps to reproduce the bug
It seems like a bug when I see an error with the existing dataset, not the dataset I'm trying to add.

` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<any_dataset>`


## Expected results
All test passed

## Actual results
```
                # check that dataset is not empty
                self.parent.assertListEqual(sorted(dataset_builder.info.splits.keys()), sorted(dataset))
                for split in dataset_builder.info.splits.keys():
                    # check that loaded datset is not empty
                    self.parent.assertTrue(len(dataset[split]) > 0)
    
                # check that we can cast features for each task template
>               task_templates = dataset_builder.info.task_templates
E               AttributeError: 'DatasetInfo' object has no attribute 'task_templates'

tests/test_dataset_common.py:175: AttributeError
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Darwin-20.4.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments: Hi ! Can you try using a more up-to-date version ? We added the task_templates in `datasets` 1.7.0.

Ideally when you're working on new datasets, you should install and use the local version of your fork of `datasets`. Here I think you tried to run the 1.7.0 tests with the 1.6.2 code
Docstring mistake: dataset vs. metric
State: closed | Author: PhilipMay (CONTRIBUTOR)

This:

https://github.com/huggingface/datasets/blob/d95b95f8cf3cb0cff5f77a675139b584dcfcf719/src/datasets/load.py#L582

Should better be something like:

`a metric identifier on HuggingFace AWS bucket (list all available metrics and ids with ``datasets.list_metrics()``)`

I can provide a PR l8er...

Comments: > I can provide a PR l8er...

See #2425
fix #2391 add original answers in kilt-TriviaQA
State: closed | Author: PaulLerner (CONTRIBUTOR)

cc @yjernite is it ok like this?

Comments: LGTM, but I'm not sure what's going on with the Unix tests @lhoestq 

The CI error is unrelated to this PR, it's been fixed now on master.

Thanks @PaulLerner !

> #- [ ] - Hey![image](https://user-images.githubusercontent.com/71971234/121969638-00030e00-cd75-11eb-9512-25d32ac08051.jpeg)@fr[fr_fr**fr~~fr `fr```
FR
````~~**_]()

Oh that was unexpected. I didn't know pokemons were into NLP
.map() function got an unexpected keyword argument 'cache_file_name'
State: closed | Author: cindyxinyiwang (NONE)
Labels: name: bug

## Describe the bug

I'm trying to save the result of datasets.map() to a specific file, so that I can easily share it among multiple computers without reprocessing the dataset. However, when I try to pass an argument 'cache_file_name' to the .map() function, it throws an error that ".map() function got an unexpected keyword argument 'cache_file_name'". 

I believe I'm using the latest dataset 1.6.2. Also seems like the document and the actual code indicates there is an argument 'cache_file_name' for the .map() function.

Here is the code I use
## Steps to reproduce the bug
```datasets = load_from_disk(dataset_path=my_path)

[...]

def tokenize_function(examples):
    return tokenizer(examples[text_column_name])

logger.info("Mapping dataset to tokenized dataset.")
tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=preprocessing_num_workers,
    remove_columns=column_names,
    load_from_cache_file=True,
   cache_file_name="my_tokenized_file"
)
```

## Actual results
    tokenized_datasets = datasets.map(
TypeError: map() got an unexpected keyword argument 'cache_file_name'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.6.2
- Platform:Linux-4.18.0-193.28.1.el8_2.x86_64-x86_64-with-glibc2.10
- Python version:3.8.5
- PyArrow version:3.0.0


Comments: Hi @cindyxinyiwang,
Did you try adding `.arrow` after `cache_file_name` argument? Here I think they're expecting something like that only for a cache file:
https://github.com/huggingface/datasets/blob/e08362256fb157c0b3038437fc0d7a0bbb50de5c/src/datasets/arrow_dataset.py#L1556-L1558

Hi ! `cache_file_name` is an argument of the `Dataset.map` method. Can you check that your `dataset` is indeed a `Dataset` object ?

If you loaded several splits, then it would actually be a `DatasetDict` (one dataset per split, in a dictionary).
In this case, since there are several datasets in the dict, the `DatasetDict.map` method requires a `cache_file_names` argument (with an 's'), so that you can provide one file name per split.

I think you are right. I used cache_file_names={data1: name1, data2: name2} and it works. Thank you!
Paperswithcode dataset mapping
State: closed | Author: julien-c (MEMBER)

This is a continuation of https://github.com/huggingface/huggingface_hub/pull/43, encoded directly inside dataset cards.

As discussed:
- `paperswithcode_id: null` when the dataset doesn't exist on paperswithcode's side.
- I've added this new key at the end of the yaml instead of ordering all keys alphabetically as pyyaml's default. No strong opinion on that one though


Comments: messed up my branch, repushing

live mapping can be found at https://huggingface.co/api/pwc/datasets-mapping and will be kept up to date going forward
PermissionError on Windows when using temp dir for caching
State: closed | Author: mariosasko (COLLABORATOR)
Labels: name: bug

Currently, the following code raises a PermissionError on master if working on Windows:

```python
# run as a script or call exit() in REPL to initiate the temp dir cleanup
from datasets import *
d = load_dataset("sst", split="train", keep_in_memory=False)
set_caching_enabled(False)
d.map(lambda ex: ex)
```

Error stack trace:
```
Traceback (most recent call last):
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\weakref.py", line 624, in _exitfunc
    f()
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\weakref.py", line 548, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\tempfile.py", line 799, in _cleanup
    _shutil.rmtree(name)
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 500, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 395, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\shutil.py", line 393, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Mario\\AppData\\Local\\Temp\\tmp20epyhmq\\cache-87a87ffb5a956e68.arrow'
```
Concatenate several datasets with removed columns is not working.
State: closed | Author: philschmid (CONTRIBUTOR)
Labels: name: bug

## Describe the bug

You can't concatenate datasets when you removed columns before.

## Steps to reproduce the bug
```python
from datasets import load_dataset, concatenate_datasets

wikiann= load_dataset("wikiann","en")

wikiann["train"] = wikiann["train"].remove_columns(["langs","spans"])
wikiann["test"] = wikiann["test"].remove_columns(["langs","spans"])

assert wikiann["train"].features.type == wikiann["test"].features.type

concate = concatenate_datasets([wikiann["train"],wikiann["test"]])
```

## Expected results
Merged dataset 


## Actual results
```python
ValueError: External features info don't match the dataset:
Got
{'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'ner_tags': Sequence(feature=ClassLabel(num_classes=7, names=['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC'], names_file=None, id=None), length=-1, id=None), 'langs': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'spans': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
with type
struct<langs: list<item: string>, ner_tags: list<item: int64>, spans: list<item: string>, tokens: list<item: string>>

but expected something like
{'ner_tags': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None), 'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
with type
struct<ner_tags: list<item: int64>, tokens: list<item: string>>
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: ~1.6.2~ 1.5.0
- Platform: macos
- Python version: 3.8.5
- PyArrow version: 3.0.0


Comments: Hi,

did you fill out the env info section manually or by copy-pasting the output of the `datasets-cli env` command?

This code should work without issues on 1.6.2 version (I'm working on master (1.6.2.dev0 version) and can't reproduce this error).

@mariosasko you are right I was still on `1.5.0`.
Add env variable for MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES
State: closed | Author: albertvillanova (MEMBER)

Add env variable for `MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES`.

This will allow to turn off default behavior: loading in memory (and not caching) small datasets.

Fix #2387.

Comments: Thank you for clarifying the precedence, @albertvillanova 

Isn't it typically the case where env vars have the highest precedence? 

In my understanding the point of env vars is to be able to override software w/o needing to touch the code. 

Please correct me if this is not so in the general case.

Hi @stas00, 

Well, I'm not an expert on this topic, but the precedence hierarchy I have normally found is from higher to lower:
- command line parameters
- env vars
- config files
So yes, normally env vars have precedence over configuration files.

Anyway, for Datasets, there are no configuration files. The _in-memory_ config is set from default values or env vars (which have precedence over default values). But this is done at import.

However, once the library is imported, the user can modify the in-memory config, and this will have precedence over the rest of mechanisms (which take place only at import).

In my limited experience env vars are typically above cmd line args, so that one can override scripts with cmd lines using env vars, but usually one then uses env vars inside cmd line args, so it's loud and clear.

For example specifying a specific gpu number on a command line will depend on `CUDA_VISIBLE_DEVICES` so gpu0 will be different if someone sets `CUDA_VISIBLE_DEVICES=2,3` vs  `CUDA_VISIBLE_DEVICES=0,1`.

> However, once the library is imported, the user can modify the in-memory config, and this will have precedence over the rest of mechanisms (which take place only at import).

And this is exactly the problem we are trying to solve here. For a good reason HF examples don't want to use `keep_in_memory=False`, and they may choose to now set `datasets.config.MAX_IN_MEMORY_DATASET_SIZE_IN_BYTES` and which means we again can't override it via env var.



oops, sorry, didn't think earlier - do we need to prefix this with `HF_DATASETS` or `HF_` like all the other env vars? or is it long enough already to be unique - it's just not telling the user in the config file what projet this variable is for...

You're right, I just opened https://github.com/huggingface/datasets/pull/2409
News_commentary Dataset Translation Pairs are of Incorrect Language Specified Pairs
State: closed | Author: anassalamah (NONE)
Labels: name: bug

I used load_dataset to load the news_commentary dataset for "ar-en" translation pairs but found translations from Arabic to Hindi.  

```
train_ds = load_dataset("news_commentary", "ar-en", split='train[:98%]')
val_ds = load_dataset("news_commentary", "ar-en", split='train[98%:]')

# filtering out examples that are not ar-en translations but ar-hi
val_ds = val_ds.filter(lambda example, indice: indice not in chain(range(1312,1327) ,range(1384,1399), range(1030,1042)), with_indices=True)
```

* I'm fairly new to using datasets so I might be doing something wrong

Comments: These ranges seem to be valid English. Closing.
Fix number of classes in indic_glue sna.bn dataset
State: closed | Author: albertvillanova (MEMBER)

As read in the [paper](https://www.aclweb.org/anthology/2020.findings-emnlp.445.pdf), Table 11.

Comments: @lhoestq there are many things missing in the README.md file, but this correction is right despite not passing the validation tests...

Yes indeed. We run the validation in all modified readme because we think that it is the time when contributors are the most likely to fix a dataset card - or it will never be done
strange datasets from OSCAR corpus
State: open | Author: cosmeowpawlitan (CONTRIBUTOR)
Labels: name: bug

![image](https://user-images.githubusercontent.com/50871412/119260850-4f876b80-bc07-11eb-8894-124302600643.png)
![image](https://user-images.githubusercontent.com/50871412/119260875-675eef80-bc07-11eb-9da4-ee27567054ac.png)
From the [official site ](https://oscar-corpus.com/), the Yue Chinese dataset should have 2.2KB data.
7 training instances is obviously not a right number.
As I can read Yue Chinese, I call tell the last instance is definitely not something that would appear on Common Crawl.
And even if you don't read Yue Chinese, you can tell the first six instance are problematic.
(It is embarrassing, as the 7 training instances look exactly like something from a pornographic novel or flitting messages in a chat of a dating app)
It might not be the problem of the huggingface/datasets implementation, because when I tried to download the dataset from the official site, I found out that the zip file is corrupted.
I will try to inform the host of OSCAR corpus later.
Awy a remake about this dataset in huggingface/datasets is needed, perhaps after the host of the dataset fixes the issue.

> Hi @jerryIsHere , sorry for the late response! Sadly this is normal, the problem comes form fasttext's classifier which we used to create the original corpus. In general the classifier is not really capable of properly recognizing Yue Chineese so the file ends un being just noise from Common Crawl. Some of these problems with OSCAR were already discussed [here](https://arxiv.org/pdf/2103.12028.pdf) but we are working on explicitly documenting the problems by language on our website. In fact, could please you open an issue on [our repo](https://github.com/oscar-corpus/oscar-website/issues) as well so that we can track it?

Thanks a lot, the new post is here:
https://github.com/oscar-corpus/oscar-website/issues/11

Comments: Hi ! Thanks for reporting
cc @pjox is this an issue from the data ?

Anyway we should at least mention that OSCAR could contain such contents in the dataset card, you're totally right @jerryIsHere 

Hi @jerryIsHere , sorry for the late response! Sadly this is normal, the problem comes form fasttext's classifier which we used to create the original corpus. In general the classifier is not really capable of properly recognizing Yue Chineese so the file ends un being just noise from Common Crawl. Some of these problems with OSCAR were already discussed [here](https://arxiv.org/pdf/2103.12028.pdf) but we are working on explicitly documenting the problems by language on our website. In fact, could please you open an issue on [our repo](https://github.com/oscar-corpus/oscar-website/issues) as well so that we can track it?