jamesLeeeeeee/datasets-github-issues huggingface dataset:
Don't do unnecessary list type casting to avoid replacing None values by empty lists
Author: lhoestq | Pull request: yes | State: closed

In certain cases, `None` values are replaced by empty lists when casting feature types.

It happens every time you cast an array of nested lists like [None, [0, 1, 2, 3]] to a different type (to change the integer precision for example). In this case you'd get [[], [0, 1, 2, 3]] for example. This issue comes from PyArrow, see the discussion in https://github.com/huggingface/datasets/issues/3676

This issue also happens when no type casting is needed, because casting is supposed to be a no-op in this case. But as https://github.com/huggingface/datasets/issues/3676 shown, it's not the case and `None` are replaced by empty lists even if we cast to the exact same type.

In this PR I just workaround this bug in the case where no type casting is needed. In particular, I only call `pa.ListArray.from_arrays` only when necessary.

I also added a warning when some `None` are effectively replaced by empty lists. I wanted to raise an error in this case, but maybe we should wait a major update to do so

This PR fixes this particular case, that is occurring in `run_qa.py` in `transformers`:
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
# before:
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
#
# now:
#              b
# 0  [None, [0]]
# 1  [None, [0]]
# 2  [None, [0]]
# 3  [None, [0]]
```

cc @sgugger
Enable label alignment for token classification datasets
Author: lewtun | Pull request: yes | State: closed

This PR extends the `Dataset.align_labels_with_mapping()` method to support alignment of label mappings between datasets and models for token classification (e.g. NER).

Example of usage:

```python
from datasets import load_dataset

ner_ds = load_dataset("conll2003", split="train")
# returns [3, 0, 7, 0, 0, 0, 7, 0, 0]
ner_ds[0]["ner_tags"]
# hypothetical model mapping with O <--> B-LOC
label2id = {
    "B-LOC": "0",
    "B-MISC": "7",
    "B-ORG": "3",
    "B-PER": "1",
    "I-LOC": "6",
    "I-MISC": "8",
    "I-ORG": "4",
    "I-PER": "2",
    "O": "5"
  }
ner_aligned_ds = ner_ds.align_labels_with_mapping(label2id, "ner_tags")
# returns [3, 5, 7, 5, 5, 5, 7, 5, 5]
ner_aligned_ds[0]["ner_tags"]
```

Context: we need this in AutoTrain to automatically align datasets / models during evaluation. cc @abhishekkrthakur
OpenBookQA has missing and inconsistent field names
Author: vblagoje | Pull request: no | State: closed

## Describe the bug
OpenBookQA implementation is inconsistent with the original dataset.

We need to:

1. The dataset field [question][stem] is flattened into question_stem. Unflatten it to match the original format.
2. Add missing additional fields:
    - 'fact1': row['fact1'],
    - 'humanScore': row['humanScore'],
    - 'clarity': row['clarity'],
    - 'turkIdAnonymized': row['turkIdAnonymized']
3. Ensure the structure and every data item in the original OpenBookQA matches our OpenBookQA version.

## Expected results
The structure and every data item in the original OpenBookQA matches our OpenBookQA version.

## Actual results
TBD

## Environment info
- `datasets` version: 2.1.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
CommonSenseQA has missing and inconsistent field names
Author: vblagoje | Pull request: no | State: open

## Describe the bug
In short, CommonSenseQA implementation is inconsistent with the original dataset.

More precisely, we need to:

1. Add the dataset matching "id" field. The current dataset, instead, regenerates monotonically increasing id.  
2. The [“question”][“stem”] field is flattened into "question". We should match the original dataset and unflatten it
3. Add the missing "question_concept" field in the question tree node
4. Anything else? Go over the data structure of the newly repaired CommonSenseQA and make sure it matches the original

## Expected results
Every data item of the CommonSenseQA should structurally and data-wise match the original CommonSenseQA dataset.

## Actual results
TBD

## Environment info
- `datasets` version: 2.1.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
error downloading bigscience-catalogue-lm-data/lm_en_wiktionary_filtered
Author: i-am-neo | Pull request: no | State: closed

## Describe the bug
Error generated when attempting to download dataset

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("bigscience-catalogue-lm-data/lm_en_wiktionary_filtered")
```

## Expected results
A clear and concise description of the expected results.

## Actual results
```
ExpectedMoreDownloadedFiles               Traceback (most recent call last)

[<ipython-input-62-4ac5cf959477>](https://localhost:8080/#) in <module>()
      1 from datasets import load_dataset
      2 
----> 3 dataset = load_dataset("bigscience-catalogue-lm-data/lm_en_wiktionary_filtered")

3 frames

[/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py](https://localhost:8080/#) in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     31         return
     32     if len(set(expected_checksums) - set(recorded_checksums)) > 0:
---> 33         raise ExpectedMoreDownloadedFiles(str(set(expected_checksums) - set(recorded_checksums)))
     34     if len(set(recorded_checksums) - set(expected_checksums)) > 0:
     35         raise UnexpectedDownloadedFile(str(set(recorded_checksums) - set(expected_checksums)))

ExpectedMoreDownloadedFiles: {'/home/leandro/catalogue_data/datasets/lm_en_wiktionary_filtered/data/file-01.jsonl.gz', '/home/leandro/catalogue_data/datasets/lm_en_wiktionary_filtered/data/file-01.jsonl.gz.lock'}
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
Add HF Speech Bench to Librispeech Dataset Card
Author: sanchit-gandhi | Pull request: yes | State: closed

Adds the HF Speech Bench to Librispeech Dataset Card in place of the Papers With Code Leaderboard. Should improve usage and visibility of this leaderboard! Wondering whether this can also be done for [Common Voice 7](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) and [8](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0) through someone with permissions? 

cc @patrickvonplaten: more leaderboard promotion!
Rename imagenet2012 -> imagenet-1k
Author: lhoestq | Pull request: yes | State: closed

On the Hugging Face Hub, users refer to imagenet2012 (from #4178 ) as imagenet-1k in their model tags.

To correctly link models to imagenet, we should rename this dataset `imagenet-1k`.

Later we can add `imagenet-21k` as a new dataset if we want.

Once this one is merged we can delete the `imagenet2012` dataset repository on the Hub.

EDIT: to complete the rationale on why we should name it `imagenet-1k`:
If users specifically added the tag `imagenet-1k` , then it could be for two reasons (not sure which one is predominant), either they
- wanted to make it explicit that it’s not 21k -> the distinction is important for the community
- or they have been following this convention from other models -> the convention implicitly exists already
data leakage in `webis/conclugen` dataset
Author: xflashxx | Pull request: no | State: closed

## Describe the bug
Some samples (argument-conclusion pairs) in the *training* split of the `webis/conclugen` dataset are present in both the *validation* and *test* splits, creating data leakage and distorting model results.
Furthermore, all splits contain duplicate samples.

## Steps to reproduce the bug
```python
from datasets import load_dataset

training = load_dataset("webis/conclugen", "base", split="train")
validation = load_dataset("webis/conclugen", "base", split="validation")
testing = load_dataset("webis/conclugen", "base", split="test")

# collect which sample id's are present in the training split
ids_validation = list()
ids_testing = list()

for train_sample in training:
    train_argument = train_sample["argument"]
    train_conclusion = train_sample["conclusion"]
    train_id = train_sample["id"]
    
    # test if current sample is in validation split
    if train_argument in validation["argument"]:
        for validation_sample in validation:
            validation_argument = validation_sample["argument"]
            validation_conclusion = validation_sample["conclusion"]
            validation_id = validation_sample["id"]
            if train_argument == validation_argument and train_conclusion == validation_conclusion:
                ids_validation.append(validation_id)
    
    # test if current sample is in test split
    if train_argument in testing["argument"]:
        for testing_sample in testing:
            testing_argument = testing_sample["argument"]
            testing_conclusion = testing_sample["conclusion"]
            testing_id = testing_sample["id"]
            if train_argument == testing_argument and train_conclusion == testing_conclusion:
                ids_testing.append(testing_id)
```

## Expected results
Length of both lists `ids_validation` and `ids_testing` should be zero.

## Actual results
Length of `ids_validation` = `2556`
Length of `ids_testing` = `287`

Furthermore, there seems to be duplicate samples in (at least) the *training* split, since:
`print(len(set(ids_validation)))` = `950`
`print(len(set(ids_testing)))` = `101`

All in all, around 7% of the samples of each the *validation* and *test* split seems to be present in the *training* split.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: macOS-12.3.1-arm64-arm-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0
Add mr_polarity movie review sentiment classification
Author: mo6zes | Pull request: yes | State: closed

Add the MR (Movie Review) dataset. The original dataset contains sentences from Rotten Tomatoes labeled as either "positive" or  "negative". 

Homepage: [https://www.cs.cornell.edu/people/pabo/movie-review-data/](https://www.cs.cornell.edu/people/pabo/movie-review-data/)
paperswithcode: [https://paperswithcode.com/dataset/mr](https://paperswithcode.com/dataset/mr)

- [ ] I was not able to generate dummy data, the original dataset files have ".pos" and ".neg" as file extensions so the auto-generator does not work. Is it fine like this or should dummy data be added?
Replace data URL in SAMSum dataset and support streaming
Author: albertvillanova | Pull request: yes | State: closed

This PR replaces data URL in SAMSum dataset:
- original host (arxiv.org) does not allow HTTP Range requests
- we have hosted the data on the Hub (license: CC BY-NC-ND 4.0)

Moreover, it implements support for streaming.

Fix #4146.
Related to: #4236.

CC: @severo
conll2003 dataset loads original data.
Author: sue991 | Pull request: no | State: closed

## Describe the bug
I load `conll2003` dataset to use refined data like [this](https://huggingface.co/datasets/conll2003/viewer/conll2003/train)  preview, but it is original data that contains `'-DOCSTART- -X- -X- O'` text.

Is this a bug or should I use another dataset_name like `lhoestq/conll2003` ?

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset
dataset = load_dataset("conll2003")
```

## Expected results
{
    "chunk_tags": [11, 12, 12, 21, 13, 11, 11, 21, 13, 11, 12, 13, 11, 21, 22, 11, 12, 17, 11, 21, 17, 11, 12, 12, 21, 22, 22, 13, 11, 0],
    "id": "0",
    "ner_tags": [0, 3, 4, 0, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
    "pos_tags": [12, 22, 22, 38, 15, 22, 28, 38, 15, 16, 21, 35, 24, 35, 37, 16, 21, 15, 24, 41, 15, 16, 21, 21, 20, 37, 40, 35, 21, 7],
    "tokens": ["The", "European", "Commission", "said", "on", "Thursday", "it", "disagreed", "with", "German", "advice", "to", "consumers", "to", "shun", "British", "lamb", "until", "scientists", "determine", "whether", "mad", "cow", "disease", "can", "be", "transmitted", "to", "sheep", "."]
}

## Actual results
```python
print(dataset)

DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 219554
    })
    test: Dataset({
        features: ['text'],
        num_rows: 50350
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 55044
    })
})
```

```python
for i in range(20):
    print(dataset['train'][i])

{'text': '-DOCSTART- -X- -X- O'}
{'text': ''}
{'text': 'EU NNP B-NP B-ORG'}
{'text': 'rejects VBZ B-VP O'}
{'text': 'German JJ B-NP B-MISC'}
{'text': 'call NN I-NP O'}
{'text': 'to TO B-VP O'}
{'text': 'boycott VB I-VP O'}
{'text': 'British JJ B-NP B-MISC'}
{'text': 'lamb NN I-NP O'}
{'text': '. . O O'}
{'text': ''}
{'text': 'Peter NNP B-NP B-PER'}
{'text': 'Blackburn NNP I-NP I-PER'}
{'text': ''}
{'text': 'BRUSSELS NNP B-NP B-LOC'}
{'text': '1996-08-22 CD I-NP O'}
{'text': ''}
{'text': 'The DT B-NP O'}
{'text': 'European NNP I-NP B-ORG'}
```
Support to load dataset with TSV files by passing only dataset name
Author: albertvillanova | Pull request: yes | State: closed

This PR implements support to load a dataset (w/o script) containing TSV files by passing only the dataset name (no need to pass `sep='\t'`):
```python
ds = load_dataset("dataset/name")
```

The refactoring allows for future builder kwargs customizations based on file extension.

Related to #4238.
Fix yield for crd3
Author: shanyas10 | Pull request: yes | State: closed

Modified the `_generate_examples` function to consider all the turns for a chunk id as a single example
Modified the features accordingly

```
"turns": [
                        {
                            "names": datasets.features.Sequence(datasets.Value("string")),
                            "utterances": datasets.features.Sequence(datasets.Value("string")),
                            "number": datasets.Value("int32"),
                        }
                    ],
                }
```

I wasn't able to run `datasets-cli dummy_data datasets` command. Is there a workaround for this?
How to load VERY LARGE dataset?
Author: CaoYiqingT | Pull request: no | State: closed

### System Info

```shell
I am using transformer trainer while meeting the issue.
The trainer requests torch.utils.data.Dataset as input, which loads the whole dataset into the memory at once. Therefore, when the dataset is too large to load, there's nothing I can do except using IterDataset, which loads samples of data seperately, and results in low efficiency. 
I wonder if there are any tricks like Sharding in huggingface trainer.
Looking forward to your reply.
```


### Who can help?

Trainer: @sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

```shell
I wonder if there are any tricks like fairseq Sharding very large datasets https://fairseq.readthedocs.io/en/latest/getting_started.html.
Thanks a lot!
```
Why the `conll2003` dataset on huggingface only contains the `en` subset? Where is the  German data?
Author: beyondguo | Pull request: no | State: closed

![image](https://user-images.githubusercontent.com/37113676/165416606-96b5db18-b16c-4b6b-928c-de8620fd943e.png)

But on huggingface datasets:
![image](https://user-images.githubusercontent.com/37113676/165416649-8fd77980-ca0d-43f0-935e-f398ba8323a4.png)

Where is the  German data?
Avoid recursion error in map if example is returned as dict value
Author: mariosasko | Pull request: yes | State: closed

I noticed this bug while answering [this question](https://discuss.huggingface.co/t/correct-way-to-create-a-dataset-from-a-csv-file/15686/11?u=mariosasko). 

This code replicates the bug:
```python
from datasets import Dataset
dset = Dataset.from_dict({"en": ["aa", "bb"], "fr": ["cc", "dd"]})
dset.map(lambda ex: {"translation": ex})
```
and this is the fix for it (before this PR):
```python
from datasets import Dataset
dset = Dataset.from_dict({"en": ["aa", "bb"], "fr": ["cc", "dd"]})
dset.map(lambda ex: {"translation": dict(ex)})
```

Internally, this can be fixed by merging two dicts via dict unpacking (instead of `dict.update) `in `Dataset.map`, which avoids creating recursive dictionaries.

P.S. `{**a, **b}` is slightly more performant than `a.update(b)` in my bencmarks.
DatasetDict containing Datasets with different features when pushed to hub gets remapped features
Author: pietrolesci | Pull request: no | State: closed

Hi there,

I am trying to load a dataset to the Hub. This dataset is a `DatasetDict` composed of various splits. Some splits have a different `Feature` mapping. Locally, the DatasetDict preserves the individual features but if I `push_to_hub` and then `load_dataset`, the features are all the same.

Dataset and code to reproduce available [here](https://huggingface.co/datasets/pietrolesci/robust_nli).

In short:

I have 3 feature mapping
```python
Tri_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"]),
    }
)

Ent_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=2, names=["non-entailment", "entailment"]),
    }
)

Con_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=2, names=["non-contradiction", "contradiction"]),
    }
)
```

Then I create different datasets

```python
dataset_splits = {}

for split in df["split"].unique():
    print(split)
    df_split = df.loc[df["split"] == split].copy()
    
    if split in Tri_dataset:
        df_split["label"] = df_split["label"].map({"entailment": 0, "neutral": 1, "contradiction": 2})
        ds = Dataset.from_pandas(df_split, features=Tri_features)
    
    elif split in Ent_bin_dataset:
        df_split["label"] = df_split["label"].map({"non-entailment": 0, "entailment": 1})
        ds = Dataset.from_pandas(df_split, features=Ent_features)
    
    elif split in Con_bin_dataset:
        df_split["label"] = df_split["label"].map({"non-contradiction": 0, "contradiction": 1})
        ds = Dataset.from_pandas(df_split, features=Con_features)

    else:
        print("ERROR:", split)
    dataset_splits[split] = ds
datasets = DatasetDict(dataset_splits)
```

I then push to hub

```python
datasets.push_to_hub("pietrolesci/robust_nli", token="<token>")
```

Finally, I load it from the hub

```python
datasets_loaded_from_hub = load_dataset("pietrolesci/robust_nli")
```

And I get that

```python
datasets["LI_TS"].features != datasets_loaded_from_hub["LI_TS"].features
```

since 

```python
"label": ClassLabel(num_classes=2, names=["non-contradiction", "contradiction"])
```

gets remapped to 

```python
 "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"])
```
Add CMU MoCap Dataset
Author: dnaveenr | Pull request: yes | State: closed

Resolves #3457 

Dataset Request : Add CMU Graphics Lab Motion Capture dataset [#3457](https://github.com/huggingface/datasets/issues/3457)
This PR adds the CMU MoCap Dataset.

The authors didn't respond even after multiple follow ups, so I ended up crawling the website to get categories, subcategories and description information. Some of the subjects do not have category/subcategory/description as well. I am using a subject to categories, subcategories and description map (metadata file).

Currently the loading of the dataset works for "asf/amc" and "avi" formats since they have a single download link. But "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Any suggestions/inputs on this would be helpful. Thank you.
Add Recall Metric Card
Author: emibaylor | Pull request: yes | State: closed

What this PR mainly does:
- add metric card for recall metric
- update docs in recall python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)
Add Precision Metric Card
Author: emibaylor | Pull request: yes | State: closed

What this PR mainly does:
- add metric card for precision metric
- update docs in precision python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)
Add remove_columns=True
Author: thomasw21 | Pull request: yes | State: closed

This should fix all the issue we have with in place operations in mapping functions. This is crucial as where we do some weird things like:
```
def apply(batch):
    batch_size = len(batch["id"])
    batch["text"] = ["potato" for _ range(batch_size)]
    return {}

# Columns are: {"id": int}
dset.map(apply, batched=True, remove_columns="text") # crashes because `text` is not in the original columns
dset.map(apply, batched=True) # mapped datasets has `text` column
```

In this PR we suggest to have `remove_columns=True` so that we ignore the input completely, and just use the output to generate mapped dataset. This means that inplace operations won't have any effects anymore.
Embed image and audio files in `save_to_disk`
Author: lhoestq | Pull request: no | State: closed

Following https://github.com/huggingface/datasets/pull/4184, currently a dataset saved using `save_to_disk` doesn't actually contain the bytes of the image or audio files. Instead it stores the path to your local files. 

Adding `embed_external_files` and set it to True by default to save_to_disk would be kind of a breaking change since some users will get bigger Arrow files when updating the lib, but the advantages are nice:

- the resulting dataset is self contained, in case you want to delete your cache for example or share it with someone else
- users also upload these Arrow files to cloud storage via the fs parameter, and in this case they would expect to upload a self-contained dataset
- consistency with push_to_hub

This can be implemented at the same time as sharding for `save_to_disk` for efficiency, and reuse the helpers from `push_to_hub` to embed the external files.

cc @mariosasko