SurAyush/github_issues huggingface dataset:
Issue: load_from_disk and save_to_disk are not compatible with each other
State: closed
Author: roholazandie

## Describe the bug
load_from_disk and save_to_disk are not compatible. When I use save_to_disk to save a dataset to disk it works perfectly but given the same directory load_from_disk throws an error that it can't find state.json. looks like the load_from_disk only works on one split

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("art")
dataset.save_to_disk("mydir")
d = Dataset.load_from_disk("mydir")
```

## Expected results
It is expected that these two functions be the reverse of each other without more manipulation

## Actual results
FileNotFoundError: [Errno 2] No such file or directory: 'mydir/art/state.json'

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>



Comments:
Hi,

`load_dataset` returns an instance of `DatasetDict` if `split` is not specified, so instead of `Dataset.load_from_disk`, use `DatasetDict.load_from_disk` to load the dataset from disk.

Thanks it worked!

Though I see a stream of issues open by people lost between datasets and datasets dicts so maybe there is here something that could be better in terms of UX. Could be better error handling or something else smarter to even avoid said errors but maybe we should think about this. Reopening to use this issue as a discussion place but feel free to open a new open if you prefer @lhoestq @albertvillanova 

We should probably improve the error message indeed.

Also note that there exists a function `load_from_disk` that can load a Dataset or a DatasetDict. Under the hood it calls either `Dataset.load_from_disk` or `DatasetDict.load_from_disk`:


```python
from datasets import load_from_disk

dataset_dict = load_from_disk("path/to/dataset/dict")
single_dataset = load_from_disk("path/to/single/dataset")
```

I just opened #2437 to improve the error message

Superseded by #2462
Issue: add `desc` in `map` for `DatasetDict` object
State: closed
Author: bhavitvyamalik

`desc` in `map` currently only works with `Dataset` objects. This PR adds support for `DatasetDict` objects as well

Comments:
The CI error is unrelated to the PR, merging

@lhoestq, can we release this feature if you guys are planning for any patch release for Datasets. It'll slow down [#11927](https://github.com/huggingface/transformers/pull/11927) otherwise :/ 

Sure definitely, having a discrepancy between Dataset.map and DatasetDict.map is an issue that we should fix and include in a patch release. Will do it in the coming days
Issue: adds license information for DailyDialog.
State: closed
Author: aditya2211


Comments:
Thanks! Can you also add it as metadata in the YAML block at the top of the file?

Should be in the form:

```
licenses:
- cc-by-sa-4.0
```

seems like we need to add all the other tags ? 
```       
if error_messages:
>           raise ValueError("\n".join(error_messages))
E           ValueError: The following issues have been found in the dataset cards:
E           YAML tags:
E           __init__() missing 8 required positional arguments: 'annotations_creators', 'language_creators', 'languages', 'multilinguality', 'size_categories', 'source_datasets', 'task_categories', and 'task_ids'
```

I'll let @lhoestq or @yjernite chime in (and maybe complete/merge). Thanks!

Looks like CircleCI has an incident. Let's wait for it to be working again and make sure the CI is green

The remaining error is unrelated to this PR, merging
Issue: add utf-8 while reading README
State: closed
Author: bhavitvyamalik

It was causing tests to fail in Windows (see #2416). In Windows, the default encoding is CP1252 which is unable to decode the character byte 0x9d 

Comments:
Can you please add encoding to this line as well to fix the issue (and maybe replace `path.open(...)` with `open(path, ...)`)?
https://github.com/huggingface/datasets/blob/7bee4be44706a59b084b9b69c4cd00f73ee72f76/src/datasets/utils/metadata.py#L58

Sure, in fact even I was thinking of adding this in order to maintain the consistency!
Issue: Make datasets PEP-561 compliant
State: closed
Author: SBrandeis

Allows to type-check datasets with `mypy` when imported as a third-party library

PEP-561: https://www.python.org/dev/peps/pep-0561
MyPy doc on the subject: https://mypy.readthedocs.io/en/stable/installed_packages.html


Comments:
This is super cool, I love that ❤️
Issue: Add KLUE dataset
State: closed
Author: jungwhank

Add `KLUE (Korean Language Understanding Evaluation)` dataset released recently from [paper](https://arxiv.org/abs/2105.09680), [github](https://github.com/KLUE-benchmark/KLUE) and [webpage](https://klue-benchmark.com/tasks).
Please let me know if there's anything missing in the code or README.
Thanks!




Comments:
I'm not sure why I got error like below when I auto-generate dummy data "mrc" 
```
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 0
Keys should be unique and deterministic in nature
```

> I'm not sure why I got error like below when I auto-generate dummy data "mrc"
> 
> ```
> datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
> Found duplicate Key: 0
> Keys should be unique and deterministic in nature
> ```

Please check out the suggestion below. I think it might be a cause.

> > I'm not sure why I got error like below when I auto-generate dummy data "mrc"
> > ```
> > datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
> > Found duplicate Key: 0
> > Keys should be unique and deterministic in nature
> > ```
> 
> Please check out the suggestion below. I think it might be a cause.

The problem was `id_` in mrc when yield was not unique. (I used index in `enumerate(paragraphs)` by mistake)
I fixed it and update all the things

To fix the CI you can just merge master into your branch and it should be all green hopefully :)

@lhoestq
Thanks for reviewing!

It's harder than I thought to add dataset card. 😅 
I checked and updated your suggestion (script, readme details, dummy data). 

dummy data is little bit larger than expected because `ner` dataset is about 80 lines and `dp` dataset is about 25 lines to avoid 0 examples.

I'm not sure why some CI keep fails, can u check for this?

Thanks ! That makes sense for ner and dp

For mrc on the other hand there are still too many examples, maybe you can generate the dummy data for 5 examples for all tasks except ner and dp ?

> Thanks ! That makes sense for ner and dp
> 
> For mrc on the other hand there are still too many examples, maybe you can generate the dummy data for 5 examples for all tasks except ner and dp ?

Yes, I generate default lines in dataset-cli for other dataset except "dp" and "ner"
I fixed mrc dataset, hope it's fine now :)

the reason CI failed was I forgot to merge master into my branch 😅
Issue: AttributeError: 'DatasetInfo' object has no attribute 'task_templates'
State: closed
Author: jungwhank

## Describe the bug
Hello, 
I'm trying to add dataset and contribute, but test keep fail with below cli.
` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<my_dataset>`

## Steps to reproduce the bug
It seems like a bug when I see an error with the existing dataset, not the dataset I'm trying to add.

` RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<any_dataset>`


## Expected results
All test passed

## Actual results
```
                # check that dataset is not empty
                self.parent.assertListEqual(sorted(dataset_builder.info.splits.keys()), sorted(dataset))
                for split in dataset_builder.info.splits.keys():
                    # check that loaded datset is not empty
                    self.parent.assertTrue(len(dataset[split]) > 0)
    
                # check that we can cast features for each task template
>               task_templates = dataset_builder.info.task_templates
E               AttributeError: 'DatasetInfo' object has no attribute 'task_templates'

tests/test_dataset_common.py:175: AttributeError
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Darwin-20.4.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Hi ! Can you try using a more up-to-date version ? We added the task_templates in `datasets` 1.7.0.

Ideally when you're working on new datasets, you should install and use the local version of your fork of `datasets`. Here I think you tried to run the 1.7.0 tests with the 1.6.2 code
Issue: Docstring mistake: dataset vs. metric
State: closed
Author: PhilipMay

This:

https://github.com/huggingface/datasets/blob/d95b95f8cf3cb0cff5f77a675139b584dcfcf719/src/datasets/load.py#L582

Should better be something like:

`a metric identifier on HuggingFace AWS bucket (list all available metrics and ids with ``datasets.list_metrics()``)`

I can provide a PR l8er...

Comments:
> I can provide a PR l8er...

See #2425
Issue: fix #2391 add original answers in kilt-TriviaQA
State: closed
Author: PaulLerner

cc @yjernite is it ok like this?

Comments:
LGTM, but I'm not sure what's going on with the Unix tests @lhoestq 

The CI error is unrelated to this PR, it's been fixed now on master.

Thanks @PaulLerner !

> #- [ ] - Hey![image](https://user-images.githubusercontent.com/71971234/121969638-00030e00-cd75-11eb-9512-25d32ac08051.jpeg)@fr[fr_fr**fr~~fr `fr```
FR
````~~**_]()

Oh that was unexpected. I didn't know pokemons were into NLP
Issue: .map() function got an unexpected keyword argument 'cache_file_name'
State: closed
Author: cindyxinyiwang

## Describe the bug

I'm trying to save the result of datasets.map() to a specific file, so that I can easily share it among multiple computers without reprocessing the dataset. However, when I try to pass an argument 'cache_file_name' to the .map() function, it throws an error that ".map() function got an unexpected keyword argument 'cache_file_name'". 

I believe I'm using the latest dataset 1.6.2. Also seems like the document and the actual code indicates there is an argument 'cache_file_name' for the .map() function.

Here is the code I use
## Steps to reproduce the bug
```datasets = load_from_disk(dataset_path=my_path)

[...]

def tokenize_function(examples):
    return tokenizer(examples[text_column_name])

logger.info("Mapping dataset to tokenized dataset.")
tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=preprocessing_num_workers,
    remove_columns=column_names,
    load_from_cache_file=True,
   cache_file_name="my_tokenized_file"
)
```

## Actual results
    tokenized_datasets = datasets.map(
TypeError: map() got an unexpected keyword argument 'cache_file_name'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:1.6.2
- Platform:Linux-4.18.0-193.28.1.el8_2.x86_64-x86_64-with-glibc2.10
- Python version:3.8.5
- PyArrow version:3.0.0


Comments:
Hi @cindyxinyiwang,
Did you try adding `.arrow` after `cache_file_name` argument? Here I think they're expecting something like that only for a cache file:
https://github.com/huggingface/datasets/blob/e08362256fb157c0b3038437fc0d7a0bbb50de5c/src/datasets/arrow_dataset.py#L1556-L1558

Hi ! `cache_file_name` is an argument of the `Dataset.map` method. Can you check that your `dataset` is indeed a `Dataset` object ?

If you loaded several splits, then it would actually be a `DatasetDict` (one dataset per split, in a dictionary).
In this case, since there are several datasets in the dict, the `DatasetDict.map` method requires a `cache_file_names` argument (with an 's'), so that you can provide one file name per split.

I think you are right. I used cache_file_names={data1: name1, data2: name2} and it works. Thank you!