helmo/github-issues huggingface dataset:
Missing original answers in kilt-TriviaQA
State: closed | Author: PaulLerner (CONTRIBUTOR)
Labels: name: bug

I previously opened an issue at https://github.com/facebookresearch/KILT/issues/42 but from the answer of @fabiopetroni it seems that the problem comes from HF-datasets

## Describe the bug
The `answer` field in kilt-TriviaQA, e.g. `kilt_tasks['train_triviaqa'][0]['output']['answer']` contains a list of alternative answer which are accepted for the question.  
However it'd be nice to know the original answer to the question (the only fields in `output` are `'answer', 'meta', 'provenance'`)

## How to fix
It can be fixed by retrieving the original answer from the original TriviaQA (e.g. `trivia_qa['train'][0]['answer']['value']`), perhaps at the same place as here where one retrieves the questions https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md#loading-the-kilt-knowledge-source-and-task-data

cc @yjernite who previously answered to an issue about KILT and TriviaQA :)


Comments: That could be useful indeed! Feel free to open a PR on the dataset card if you already have some code that runs, otherwise we'll take care of it soon :) 

I can open a PR but there is 2 details to fix:
- the name for the corresponding key (e.g. `original_answer`)
- how to implement it: I’m not sure what happens when you map `lambda x: {'input': ...}` as it keeps the other keys (e.g. `output`) intact but here since we want to set a nested value (e.g. `x['output']['original_answer']`) I implemented it with a regular function (not lambda), see below

```py
def add_original_answer(x, trivia_qa, triviaqa_map):
    i = triviaqa_map[x['id']]
    x['output']['original_answer'] = trivia_qa['validation'][i]['answer']['value']
    return x
```
Accessing Arrow dataset cache_files
State: closed | Author: Mehrad0711 (NONE)
Labels: name: bug

## Describe the bug
In datasets 1.5.0 the following code snippet would have printed the cache_files:

```
train_data = load_dataset('conll2003', split='train', cache_dir='data')
print(train_data.cache_files[0]['filename'])

```

However, in the newest release (1.6.1), it prints an empty list.

I also tried loading the dataset with `keep_in_memory=True` argument but still `cache_files` is empty.

Was wondering if this is a bug or I need to pass additional arguments so I can access the cache_files.


Comments: Thanks @bhavitvyamalik for referencing the workaround. Setting `keep_in_memory=False` is working.
Add args description to DatasetInfo
State: closed | Author: lewtun (MEMBER)

Closes #2354 

I am not sure what `post_processed` and `post_processing_size` correspond to, so have left them empty for now. I also took a guess at some of the other fields like `dataset_size` vs `size_in_bytes`, so might have misunderstood their meaning.

Comments: Thanks for the suggestions! I've included them and made a few minor tweaks along the way

Please merge master into this branch to fix the CI, I just fixed metadata validation tests.
DuplicatedKeysError: FAILURE TO GENERATE DATASET ! load_dataset('head_qa', 'en')
State: closed | Author: helloworld123-lab (NONE)

Hello everyone,

I try to use head_qa dataset in [https://huggingface.co/datasets/viewer/?dataset=head_qa&config=en](url)

```
!pip install datasets
from datasets import load_dataset
dataset = load_dataset(
   'head_qa', 'en')
```
When I write above load_dataset(.), it throws the following:

```
DuplicatedKeysError                       Traceback (most recent call last)

<ipython-input-6-ea87002d32f0> in <module>()
      2 from datasets import load_dataset
      3 dataset = load_dataset(
----> 4    'head_qa', 'en')

5 frames

/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py in check_duplicate_keys(self)
    347         for hash, key in self.hkey_record:
    348             if hash in tmp_record:
--> 349                 raise DuplicatedKeysError(key)
    350             else:
    351                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 1
Keys should be unique and deterministic in nature
```
How can I fix the error? Thanks
maintain YAML structure reading from README
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

How YAML used be loaded earlier in the string (structure of YAML was affected because of this and YAML for datasets with multiple configs was not being loaded correctly):
```
annotations_creators:
labeled_final:
- expert-generated
labeled_swap:
- expert-generated
unlabeled_final:
- machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
labeled_final:
- 10K<n<100K
labeled_swap:
- 10K<n<100K
unlabeled_final:
- 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```
How YAML is loaded in string now:
```
annotations_creators:
  labeled_final:
  - expert-generated
  labeled_swap:
  - expert-generated
  unlabeled_final:
  - machine-generated
language_creators:
- machine-generated
languages:
- en
licenses:
- other
multilinguality:
- monolingual
size_categories:
  labeled_final:
  - 10K<n<100K
  labeled_swap:
  - 10K<n<100K
  unlabeled_final:
  - 100K<n<1M
source_datasets:
- original
task_categories:
- text-classification
- text-scoring
task_ids:
- semantic-similarity-classification
- semantic-similarity-scoring
- text-scoring-other-paraphrase-identification
```
Add missing dataset_infos.json files
State: open | Author: lewtun (MEMBER)
Labels: name: enhancement

Some of the datasets in `datasets` are missing a `dataset_infos.json` file, e.g.

```
[PosixPath('datasets/chr_en/chr_en.py'), PosixPath('datasets/chr_en/README.md')]
[PosixPath('datasets/telugu_books/README.md'), PosixPath('datasets/telugu_books/telugu_books.py')]
[PosixPath('datasets/reclor/README.md'), PosixPath('datasets/reclor/reclor.py')]
[PosixPath('datasets/json/README.md')]
[PosixPath('datasets/csv/README.md')]
[PosixPath('datasets/wikihow/wikihow.py'), PosixPath('datasets/wikihow/README.md')]
[PosixPath('datasets/c4/c4.py'), PosixPath('datasets/c4/README.md')]
[PosixPath('datasets/text/README.md')]
[PosixPath('datasets/lm1b/README.md'), PosixPath('datasets/lm1b/lm1b.py')]
[PosixPath('datasets/pandas/README.md')]
```

For `json`, `text`, csv`, and `pandas` this is expected, but not for the others which should be fixed
ArrowDataset.save_to_disk produces files that cannot be read using pyarrow.feather
State: open | Author: Ark-kun (NONE)
Labels: name: bug

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from pyarrow import feather

dataset = load_dataset('imdb', split='train')
dataset.save_to_disk('dataset_dir')
table = feather.read_table('dataset_dir/dataset.arrow')
```

## Expected results
I expect that the saved dataset can be read by the official Apache Arrow methods.

## Actual results
```
  File "/usr/local/lib/python3.7/site-packages/pyarrow/feather.py", line 236, in read_table
    reader.open(source, use_memory_map=memory_map)
  File "pyarrow/feather.pxi", line 67, in pyarrow.lib.FeatherReader.open
  File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not a Feather V1 or Arrow IPC file
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets-1.6.2
- Platform: Linux
- Python version: 3.7
- PyArrow version: 0.17.1, also 2.0.0


Comments: Hi ! This is because we are actually using the arrow streaming format. We plan to switch to the arrow IPC format.
More info at #1933 

Not sure if this was resolved, but I am getting a similar error when trying to load a dataset.arrow file directly: `ArrowInvalid: Not an Arrow file`

Since we're using the streaming format, you need to use `open_stream`:

```python
import pyarrow as pa

def in_memory_arrow_table_from_file(filename: str) -> pa.Table:
    in_memory_stream = pa.input_stream(filename)
    opened_stream = pa.ipc.open_stream(in_memory_stream)
    pa_table = opened_stream.read_all()
    return pa_table

def memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
    memory_mapped_stream = pa.memory_map(filename)
    opened_stream = pa.ipc.open_stream(memory_mapped_stream)
    pa_table = opened_stream.read_all()
    return pa_table
```

> 由于我们使用流格式，因此您需要使用`open_stream`：
> 
> ```python
> import pyarrow as pa
> 
> def in_memory_arrow_table_from_file(filename: str) -> pa.Table:
>     in_memory_stream = pa.input_stream(filename)
>     opened_stream = pa.ipc.open_stream(in_memory_stream)
>     pa_table = opened_stream.read_all()
>     return pa_table
> 
> def memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
>     memory_mapped_stream = pa.memory_map(filename)
>     opened_stream = pa.ipc.open_stream(memory_mapped_stream)
>     pa_table = opened_stream.read_all()
>     return pa_table
> ```
Thank you very much for providing the code that can read arrow file to pa_table and finally to dict, but how to implement the reverse process, how to restore a dict to arrow file?
Improve task api code quality
State: closed | Author: mariosasko (COLLABORATOR)

Improves the code quality of the `TaskTemplate` dataclasses.

Changes:
* replaces `return NotImplemented` with raise `NotImplementedError` 
* replaces `sorted` with `len` in the uniqueness check 
* defines `label2id` and `id2label` in the `TextClassification` template as properties
* replaces the `object.__setattr__(self, attr, value)` syntax with (IMO nicer) `self.__dict__[attr] = value`

Comments: Looks good thanks, what do you think @lewtun ?

thanks for including the lazy `ClassLabel` class @mariosasko ! from my side this LGTM!
add `desc` to `tqdm` in `Dataset.map()`
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

Fixes #2330. Please let me know if anything is also required in this 

Comments: Once this is merged, let's update `transformers` examples to use this new code. As currently all those tqdm bars are who knows what they are....

https://github.com/huggingface/transformers/issues/11797

Sure @stas00! Once this is merged let's discuss what all changes can be done on `transformers` side

@bhavitvyamalik, as it has been merged would you like to tackle https://github.com/huggingface/transformers/issues/11797?


Definitely @stas00. From what I could gather, you guys want more meaningful `.map` calls for all examples [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch)?

That's exactly right, @bhavitvyamalik 

Perhaps the best approach is to do one example, see that other maintainers agree on it. and then replicate to other.
Loading dataset from local path
State: closed | Author: kolakows (NONE)

I'm trying to load a local dataset with the code below

```
ds = datasets.load_dataset('my_script.py', 
                           data_files='corpus.txt', 
                           data_dir='/data/dir', 
                           cache_dir='.')
```
But internally a BuilderConfig is created, which tries to use getmtime on the data_files string, without using data_dir. Is this a bug or am I not using the load_dataset correctly?

https://github.com/huggingface/datasets/blob/bc61954083f74e6460688202e9f77dde2475319c/src/datasets/builder.py#L153

Comments: Version below works, checked again in the docs, and data_files should be a path.
```
ds = datasets.load_dataset('my_script.py', 
                           data_files='/data/dir/corpus.txt', 
                           cache_dir='.')
```
ConvQuestions benchmark added
State: closed | Author: PhilippChr (CONTRIBUTOR)

Hello,
I would like to integrate our dataset on conversational QA. The answers are grounded in the KG.
The work was published in CIKM 2019 (https://dl.acm.org/doi/10.1145/3357384.3358016).
We hope for further research on how to deal with the challenges of factoid conversational QA.
Thanks! :)

Comments: Thanks for your helpful comments and suggestions! :)
I integrated the additional fields, and extended some of the README/dataset card.
And I actually realized that we had the cc-by-4.0 for the dataset, so this was also changed.

I added the answers to the test set actually :)

Oh great ! Let me revert my change then
Align question answering tasks with sub-domains
State: closed | Author: lewtun (MEMBER)
Labels: name: enhancement

As pointed out by @thomwolf in #2255 we should consider breaking with the pipeline taxonomy of `transformers` to account for the various types of question-answering domains:

> `question-answering` exists in two forms: abstractive and extractive question answering.
> 
> we can keep a generic `question-answering` but then it will probably mean diferrent schema of input/output for both (abstractive will have text for both while extractive can use spans indication as well as text).
> 
> Or we can also propose to use `abstractive-question-answering` and `extractive-question-answering` for instance.
> Maybe we could have `question-answering-abstractive` and `question-answering-extractive` if somehow we can use a for a completion or search in the future (detail).
> Actually I see that people are more organizing in terms of general and sub-tasks, for instance on paperwithcode: https://paperswithcode.com/area/natural-language-processing and on nlpprogress: https://github.com/sebastianruder/NLP-progress/blob/master/english/question_answering.md#squad
> 
> Probably the best is to align with one of these in terms of denomination, PaperWithCode is probably the most active and maintained and we work with them as well.
> Maybe you want to check with a few QA datasets that this schema make sense. Typically NaturalQuestions, TriviaQA and can be good second datasets to compare to and be sure of the generality of the schema.
> 
> A good recent list of QA datasets to compare the schemas among, is for instance in the UnitedQA paper: https://arxiv.org/abs/2101.00178

Investigate which grouping of QA is best suited for `datasets` and adapt / extend the QA task template accordingly.

Comments: Closing this issue as the `task_templates` API has been deprecated.
Adding HendrycksTest dataset
State: closed | Author: andyzoujm (CONTRIBUTOR)

Adding Hendrycks test from https://arxiv.org/abs/2009.03300.
I'm having a bit of trouble with dummy data creation because some lines in the csv files aren't being loaded properly (only the first entry loaded in a row of length 6). The dataset is loading just fine. Hope you can kindly help!
Thank you!

Comments: @lhoestq Thank you for the review. I've made the suggested changes. There still might be some problems with dummy data though due to some csv loading issues (which I haven't found the cause to).

I took a look at the dummy data and some csv lines were cropped. I fixed them :)

@andyzoujm Any reason why this dataset scrip was called "hendrycks_test" instead of "mmlu"?

We are thinking of renaming it...

That's because we didn't call it MMLU in the paper (the shorthand didn't
emerge until over a year later), and people at OpenAI were calling it that.

Andy

On Wed, Apr 26, 2023 at 8:44 AM Albert Villanova del Moral <
***@***.***> wrote:

> @andyzoujm <https://github.com/andyzoujm> Any reason why this dataset
> scrip was called "hendrycks_test" instead of "mmlu"?
>
> We are thinking of renaming it...
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/2370#issuecomment-1523358110>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AKLQJMZZFZBGJTBOIFWJ5KDXDEKB5ANCNFSM45BAOSIQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Thanks for your reply. Just for the records: we have renamed it to "cais/mmlu": https://huggingface.co/datasets/cais/mmlu
correct labels of conll2003
State: closed | Author: philschmid (CONTRIBUTOR)

# What does this PR

It fixes/extends the `ner_tags` for conll2003 to include all. 
Paper reference https://arxiv.org/pdf/cs/0306050v1.pdf
Model reference https://huggingface.co/elastic/distilbert-base-cased-finetuned-conll03-english/blob/main/config.json
Json loader fails if user-specified features don't match the json data fields order
State: closed | Author: lhoestq (MEMBER)
Labels: name: bug

If you do
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then depending on the order of the features in the json data field it fails:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ValueError: Target schema's field names are not matching the table's field names: ['tokens', 'ner_tags'], ['ner_tags', 'tokens']
```

This is because one must first re-order the columns of the table to match the `self.config.schema` before calling cast.

One way to fix the `cast` would be to replace it with:
```python
# reorder the arrays if necessary + cast to schema
# we can't simply use .cast here because we may need to change the order of the columns
pa_table = pa.Table.from_arrays([pa_table[name] for name in schema.names], schema=schema)
```
Missing ClassLabel encoding in Json loader
State: closed | Author: lhoestq (MEMBER)
Labels: name: bug

Currently if you want to load a json dataset this way
```python
dataset = load_dataset("json", data_files=data_files, features=features)
```
Then if your features has ClassLabel types and if your json data needs class label encoding (i.e. if the labels in the json files are strings and not integers), then it would fail:
```python
[...]
~/Desktop/hf/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files)
     94             if self.config.schema:
     95                 # Cast allows str <-> int/float, while parse_option explicit_schema does NOT
---> 96                 pa_table = pa_table.cast(self.config.schema)
     97             yield i, pa_table
[...]
ArrowInvalid: Failed to parse string: 'O' as a scalar of type int64
```

This is because it just tries to cast the string data to integers, without applying the mapping str->int first

The current workaround is to do instead
```python
dataset = load_dataset("json", data_files=data_files)
dataset = dataset.map(features.encode_example, features=features)
```
README updated for SNLI, MNLI
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

Closes #2275. Mentioned about -1 labels in MNLI, SNLI and how they should be removed before training. @lhoestq `check_code_quality` test might fail for MNLI as the license name `other-Open Portion of the American National Corpus`  is not a registered tag for 'licenses'

Comments: Regarding the license issue, I think we should allow it since it starts with `other-`. Cc @gchhablani what do you think ?

@lhoestq I agree, I'll look into it.
Fix web_nlg metadata
State: closed | Author: julien-c (MEMBER)

Our metadata storage system does not support `.` inside keys. cc @Pierrci 



Comments: Hi ! `release_v2.1` and the others are dataset configuration names.

The configuration names are used to show the right code snippet in the UI to load the dataset.
For example if the parsing of the web_nlg tags worked correctly we would have:
![image](https://user-images.githubusercontent.com/42851186/118475444-8d1e5d00-b70c-11eb-98e9-844d4daf6139.png)

Therefore I don't think it's a good idea to rename the configurations from `release_v2.1` to `release_v2_1` as the code snippet would be wrong in this case.

Moreover we can't really disallow dots in configuration names and rename the configurations since it would be a big breaking change. It's commonly used, especially with multilingual datasets. For example `load_dataset("indic_glue", "sna.bn")`.

Is this something that can be fixed on the moonlanding side instead ?

> Is this something that can be fixed on the moonlanding side instead ?

Not really unless we change database:)

We'll maybe try to find another workaround, but super low-prio given that it's the only dataset that has those dotted keys in the YAML metadata

Ok, should we close this PR then ?
Preserve dtype for numpy/torch/tf/jax arrays
State: closed | Author: bhavitvyamalik (CONTRIBUTOR)

Fixes #625. This lets the user preserve the dtype of numpy array to pyarrow array which was getting lost due to conversion of numpy array -> list -> pyarrow array. 

Comments: Hi @lhoestq, 
It turns out that pyarrow `ListArray` are not recognized as list-like when we get output from `numpy_to_pyarrow_listarray`. This might cause tests to fail. If possible can we convert that `ListArray` output to list inorder for tests to pass? Under the hood it'll maintain the dtype as that of numpy array passed during input only

Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In `test_map_torch` https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039 and `test_map_tf`https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056 
they're expecting `float64`. Shouldn't that be `float32` now?

It's normal: pytorch and tensorflow use `float32` by default, unlike numpy which uses `float64`.

I think that we should always keep the precision of the original tensor (torch/tf/numpy).
It means that as it is in this PR it's fine (the precision is conserved when doing the torch/tf -> numpy conversion).

This is a breaking change but in my opinion the fact that we had Value("float64") for torch.float32 tensors was an issue already.

Let me know what you think. Cc @albertvillanova if you have an opinion on this

If we agree on doing this breaking change, we can just change the test. 

Hi @lhoestq, 
Merged master into this branch. Only changing the test is left for now (mentioned below) after which all tests should pass.

> Brought down the failing tests from 7 to 4. Let me know if that part looks good. Failing tests are looking quite similar. In `test_map_torch`
> 
> https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1039
> 
> and `test_map_tf`
> https://github.com/huggingface/datasets/blob/3d46bc384f811435e59e3916faa3aa20a1cf87bc/tests/test_arrow_dataset.py#L1056
> 
> 
> they're expecting `float64`. Shouldn't that be `float32` now?



> they're expecting float64. Shouldn't that be float32 now?

Yes feel free to update those tests :)

It would be nice to have the same test for JAX as well

Added same test for for JAX too. Also, I saw that I missed changing `test_cast_to_python_objects_jax` like I did for TF and PyTorch. Finished that as well