jamesLeeeeeee/datasets-github-issues huggingface dataset:
Bug in `openbookqa` dataset
Author: lucadiliello | Pull request: no | State: closed

## Describe the bug
Dataset entries contains a typo.

## Steps to reproduce the bug

```python
>>> from datasets import load_dataset
>>> obqa = load_dataset('openbookqa', 'main')
>>> obqa['train'][0]
```

## Expected results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['A', 'B', 'C', 'D']}, 'answerKey': 'D'}
```

## Actual results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting']}, 'answerKey': 'D'}
```

The bug is present in all configs and all splits.

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.4.0-1057-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1
Specify the feature types of a dataset on the Hub without needing a dataset script
Author: lhoestq | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
Currently if I upload a CSV with paths to audio files, the column type is string instead of Audio.

**Describe the solution you'd like**
I'd like to be able to specify the types of the column, so that when loading the dataset I directly get the features types I want.

The feature types could read from the `dataset_infos.json` for example.

**Describe alternatives you've considered**
Create a dataset script to specify the features, but that seems complicated for a simple thing.

cc @abidlabs
Datasets created with `push_to_hub` can't be accessed in offline mode
Author: TevenLeScao | Pull request: no | State: closed

## Describe the bug
In offline mode, one can still access previously-cached datasets. This fails with datasets created with `push_to_hub`.

## Steps to reproduce the bug
in Python:
```
import datasets
mpwiki = datasets.load_dataset("teven/matched_passages_wikidata")
```
in bash:
```
export HF_DATASETS_OFFLINE=1
```
in Python:
```
import datasets
mpwiki = datasets.load_dataset("teven/matched_passages_wikidata")
```

## Expected results
`datasets` should find the previously-cached dataset.

## Actual results
ConnectionError: Couln't reach the Hugging Face Hub for dataset 'teven/matched_passages_wikidata': Offline mode is enabled

## Environment info
- `datasets` version: 1.16.2.dev0
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 3.0.0
Ability to split a dataset in multiple files.
Author: Dref360 | Pull request: no | State: open

Hello,

**Is your feature request related to a problem? Please describe.**

My use case is that I have one writer that adds columns and multiple workers reading the same `Dataset`. Each worker should have access to columns added by the writer when they reload the dataset.

I understand that we shouldn't overwrite an arrow file as this could cause Segfault and so on. Before 1.16, I was able to overwrite the dataset and that would work most of the time with some retries.

**Describe the solution you'd like**

I was thinking that if we could append `Dataset._data_files`, when the workers reload the Dataset, they would get the new columns.


**Describe alternatives you've considered**
I currently need to
1. Save multiple "versions" of the dataset and load the latest.
2. Try working with cache files to get the latest columns.

**Additional context**

I think this would be a great addition to HFDataset as Parquet supports multi-files input out of the box!

I can make a PR myself with some pointers as needed :)
Allow loading community metrics from the hub, just like datasets
Author: eladsegal | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
Currently, I can load a metric implemented by me by providing the local path to the file in `load_metric`.
However, there is no option to do it with the metric uploaded to the hub.
This means that if I want to allow other users to use it, they must download it first which makes the usage less smooth.

**Describe the solution you'd like**
Load metrics from the hub just like datasets are loaded.
In order to not break stuff, the convention can be to put the metric file in a "metrics" folder in the hub.
How to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?
Author: CindyTing | Pull request: no | State: open

Hi,
I use torch.utils.data.Dataset to define my own data, but I need to use the 'map' function of datasets.arrow_dataset.Dataset later, so I hope to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset.

Here is an example.
```
from torch.utils.data import Dataset
from datasets.arrow_dataset import Dataset as HFDataset

class ADataset(Dataset):
    def __init__(self, data): 
        super().__init__()
        self.data = data
    def __getitem__(self, index):
        return self.data[index]
    def __len__(self):
        return self.len
class MDataset():
    def __init__(self, tokenizer: AutoTokenizer, data_args, training_args):
        self.train_dataset = ADataset(data_args)
        self.tokenizer = tokenizer
        self.data_args = data_args

        self.train_dataset = self.train_dataset.map(
            self.process_function,
            batched=True,
            remove_columns=column_names,
            load_from_cache_file=True,
            desc="Running tokenizer on train dataset",
             )
    def process_function(self, examples):
        sentences = [" ".join(sample[0][3]) for sample in examples]

        tokenized = self.tokenizer(
            sentences,
            max_length=self.max_seq_len,
            padding=self.padding, 
            truncation=True)
```
But it would raise an ERROR, AttributeError: 'ADataset' object has no attribute 'map'.

so how to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Thanks in advance!
Task search function on hub not working correctly
Author: patrickvonplaten | Pull request: no | State: open

When I want to look at all datasets of the category: `speech-processing`  *i.e.* https://huggingface.co/datasets?task_categories=task_categories:speech-processing&sort=downloads , then the following dataset doesn't show up for some reason:

- https://huggingface.co/datasets/speech_commands

even thought it's task tags seem correct: 
https://raw.githubusercontent.com/huggingface/datasets/master/datasets/speech_commands/README.md
Give clearer instructions to add the YAML tags
Author: albertvillanova | Pull request: no | State: closed

## Describe the bug
As reported by @julien-c, many community datasets contain the line `YAML tags:` at the top of the YAML section in the header of the README file.  See e.g.: https://huggingface.co/datasets/bigscience/P3/commit/a03bea08cf4d58f268b469593069af6aeb15de32

Maybe we should give clearer instruction/hints in the README template.
wmt19 is broken (zh-en)
Author: AjayP13 | Pull request: no | State: closed

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wmt19", 'zh-en')
```

## Expected results
The dataset should download.

## Actual results
`ConnectionError: Couldn't reach ftp://cwmt-wmt:cwmt-wmt@datasets.nju.edu.cn/parallel/casia2015.zip`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux
- Python version: 3.8
Add PubMed Central Open Access dataset
Author: albertvillanova | Pull request: no | State: closed

## Adding a Dataset
- **Name:** PubMed Central Open Access
- **Description:** The PMC Open Access Subset includes more than 3.4 million journal articles and preprints that are made available under license terms that allow reuse.
- **Paper:** *link to the dataset paper if available*
- **Data:** https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
`wiki_dpr` details for Open Domain Question Answering tasks
Author: pk1130 | Pull request: no | State: closed

Hey guys!

Thanks for creating the `wiki_dpr` dataset!

I am currently trying to use the dataset for context retrieval using DPR on NQ questions and need details about what each of the files and data instances mean, which version of the Wikipedia dump it uses, etc. Please respond at your earliest convenience regarding the same! Thanks a ton!

P.S.: (If one of @thomwolf @lewtun @lhoestq could respond, that would be even better since they have the first-hand details of the dataset. If anyone else has those, please reach out! Thanks!)
Discuss whether support canonical datasets w/o dataset_infos.json and/or dummy data
Author: albertvillanova | Pull request: no | State: closed

I open this PR to have a public discussion about this topic and make a decision.

As previously discussed, once we have the metadata in the dataset card (README file, containing both Markdown info and YAML tags), what is the point of having also the JSON metadata (dataset_infos.json file)?

On the other hand, the dummy data is necessary for testing (in our CI suite) that the canonical dataset loads correctly. However:
- the dataset preview feature is already an indirect test that the dataset loads correctly (it also tests it is streamable though)
- we are migrating canonical datasets to the Hub
Do we really need to continue testing them in out CI?

Also note that for generating both (dataset_infos.json file and dummy data), the entire dataset needs being downloaded. This can be an issue for huge datasets (like WIT, with 400 GB of data).

Feel free to ping other people for the discussion.

CC: @lhoestq @mariosasko @thomwolf @julien-c @patrickvonplaten @anton-l @LysandreJik @yjernite @nateraw
Batched in filter throws error
Author: gpucce | Pull request: no | State: closed

I hope this is really a bug, I could not find it among the open issues

## Describe the bug
using `batched=False` in DataSet.filter throws error
```python
TypeError: filter() got an unexpected keyword argument 'batched'
```
but in the docs it is lister as an argument.

## Steps to reproduce the bug
```python
task = "mnli"
max_length = 128
tokenizer = AutoTokenizer.from_pretrained("./pretrained_models/pretrained_models_drozd/sl250.m.gsic.titech.ac.jp:8000/21.11.17_06.30.32_roberta-base_a0057/checkpoints/smpl_400M/hf/")

dataset = load_dataset("glue", task)

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

##### tokenization_parameters
sentence1_key, sentence2_key = task_to_keys[task]
def preprocess_function(examples, max_length):
    if sentence2_key is None:
        return tokenizer(
            examples[sentence1_key], truncation=True, max_length=max_length
        )
    return tokenizer(
        examples[sentence1_key],
        examples[sentence2_key],
        truncation=False,
        padding="max_length",
        max_length=max_length,
    )

encoded_dataset = dataset.map(
    lambda x: preprocess_function(x, max_length=max_length), batched=False
)

encoded_dataset.filter(lambda x: len(x['input_ids']) <= max_length, batched=False)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1, 1.17.0
- Platform: ubuntu
- Python version: 3.8.12
Adjusting chunk size for streaming datasets
Author: JoelNiklaus | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
I want to use mc4 which I cannot save locally, so I stream it. However, I want to process the entire dataset and filter some documents from it. With the current chunk size of around 1000 documents (right?) I hit a performance bottleneck because of the frequent decompressing. 

**Describe the solution you'd like**
I would appreciate a parameter in the load_dataset function, that allows me to set the chunksize myself (to a value like 100'000 in my case). Like that, I hope to improve the processing time.
update `pretty_name` for first 200 datasets
Author: bhavitvyamalik | Pull request: yes | State: closed

I made a script some time back to fetch `pretty_names` from `papers_with_code` dataset along with some other rules incase that dataset wasn't available on `papers_with_code`. Updating them in the `README` of `datasets`. Took only the first 200 datasets into consideration and after some eyeballing, most of them were looking good to me!
Add VoxLingua107
Author: jaketae | Pull request: no | State: open

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models. 
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** 107 languages, totaling 6628 hours for the train split.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Avoid unnecessary list creations
Author: bryant1410 | Pull request: yes | State: open

Like in `join([... for s in ...])`. Also changed other things that I saw:
* Use a `with` statement for many `open` that missed them, so the files don't remain open.
* Remove unused variables.
* Many HTTP links converted into HTTPS (verified).
* Remove unnecessary "r" mode arg in `open` (double-checked it was actually the default in each case).
* Remove Python 2 style of using `super`.
* Run `pyupgrade $(find . -name "*.py" -type f) --py36-plus` (which already does some of the previous points).
* Run `dos2unix $(find . -name "*.py" -type f)` (CRLF to LF line endings).
* Fix typos.
URL query parameters are set as path in the compression hop for fsspec
Author: albertvillanova | Pull request: no | State: open

## Describe the bug
There is an ssue with `StreamingDownloadManager._extract`.

I don't know how the test `test_streaming_gg_drive_gzipped` passes:

For 
```python
TEST_GG_DRIVE_GZIPPED_URL = "https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz"
urlpath = StreamingDownloadManager().download_and_extract(TEST_GG_DRIVE_GZIPPED_URL)
```
gives `urlpath`:
```python 
'gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz::https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz'
```

The gzip path makes no sense: `gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz`


## Steps to reproduce the bug
```python
from datasets.utils.streaming_download_manager import StreamingDownloadManager

dl_manager = StreamingDownloadManager()
urlpath = dl_manager.extract("https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz")
print(urlpath)
```

## Expected results
The query parameters should not be set as part of the path.
skip columns which cannot set to specific format when set_format
Author: tshu-w | Pull request: no | State: closed

**Is your feature request related to a problem? Please describe.**
When using `dataset.set_format("torch")`, I must make sure every columns in datasets can convert to `torch`, however, sometimes I want to keep some string columns.

**Describe the solution you'd like**
skip columns which cannot set to specific format when set_format instead of raise an error.
make shape verification to use ArrayXD instead of nested lists for map
Author: tshu-w | Pull request: no | State: open

As describe in https://github.com/huggingface/datasets/issues/2005#issuecomment-793716753 and mentioned by @mariosasko in [image feature example](https://colab.research.google.com/drive/1mIrTnqTVkWLJWoBzT1ABSe-LFelIep1c#scrollTo=ow3XHDvf2I0B&line=1&uniqifier=1), IMO make shape verifcaiton to use ArrayXD instead of nested lists for map can help user reduce unnecessary cast. I notice datasets have done something special for `input_ids` and `attention_mask` which is also unnecessary after this feature added.
Fix duplicate keys in NewsQA
Author: bryant1410 | Pull request: yes | State: closed

* Fix duplicate keys in NewsQA when loading from CSV files.
* Fix s/narqa/newsqa/ in the download manually error message.
* Make the download manually error message show nicely when printed. Otherwise, is hard to read due to spacing issues.
* Fix the format of the license text.
* Reformat the code to make it simpler.
Fix overriding of filesystem info
Author: albertvillanova | Pull request: yes | State: closed

Previously, `BaseCompressedFileFileSystem.info` was overridden and transformed from function to dict.

This generated a bug for filesystem methods that use `self.info()`, like e.g. `fs.isfile()`.

This PR:
- Adds tests for `fs.isfile` (that use `fs.info`).
- Fixes custom `BaseCompressedFileFileSystem.info` by removing its overriding.
the compression format requested when saving a dataset in json format is not respected
Author: SaulLu | Pull request: no | State: closed

## Describe the bug
In the documentation of the `to_json` method, it is stated in the parameters that 
> **to_json_kwargs – Parameters to pass to pandas’s pandas.DataFrame.to_json.

however when we pass for example `compression="gzip"`, the saved file is not compressed.

Would you also have expected compression to be applied? :relaxed: 

## Steps to reproduce the bug
```python
my_dict = {"a": [1, 2, 3], "b": [1, 2, 3]}
```
### Result with datasets
```python
from datasets import Dataset

dataset = Dataset.from_dict(my_dict)
dataset.to_json("dic_with_datasets.jsonl.gz", compression="gzip")
!cat dic_with_datasets.jsonl.gz
```
output
```
{"a":1,"b":1}
{"a":2,"b":2}
{"a":3,"b":3}
```
Note: I would expected to see binary data here

### Result with pandas
```python
import pandas as pd

df = pd.DataFrame(my_dict)
df.to_json("dic_with_pandas.jsonl.gz", lines=True, orient="records", compression="gzip")
!cat dic_with_pandas.jsonl.gz
```
output
```
4��a�dic_with_pandas.jsonl��VJT�2�QJ��\� ��g��yƵ���������)���
```
Note: It looks like binary data

## Expected results

I would have expected that the saved result with datasets would also be a binary file

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyArrow version: 5.0.0
Dataset preview is not available (I think for all Hugging Face datasets)
Author: Abirate | Pull request: no | State: closed

## Dataset viewer issue for '*french_book_reviews*'

**Link:** https://huggingface.co/datasets/Abirate/french_book_reviews

**short description of the issue**
For my dataset, the dataset preview is no longer functional (it used to work: The dataset had been added the day before and it was fine...)  
And, after looking over the datasets, I discovered that this issue affects all Hugging Face datasets (as of yesterday, December 23, 2021, around 10 p.m. (CET)).  
**Am I the one who added this dataset** :  Yes   

**Note**: here a screenshot showing the issue
![Dataset preview is not available for my dataset](https://user-images.githubusercontent.com/66887439/147333078-60734578-420d-4e91-8691-a90afeaa8948.jpg)  

**And here for glue dataset :**  

![Dataset preview is not available for other Hugging Face datasets(glue)](https://user-images.githubusercontent.com/66887439/147333492-26fa530c-befd-4992-8361-70c51397a25a.jpg)
Extend support for streaming datasets that use os.walk
Author: albertvillanova | Pull request: yes | State: closed

This PR extends the support in streaming mode for datasets that use `os.walk`, by patching that function.

This PR adds support for streaming mode to datasets:
1. autshumato
1. code_x_glue_cd_code_to_text
1. code_x_glue_tc_nl_code_search_adv
1. nchlt

CC: @severo
Use `iter_files` instead of `str(Path(...)` in image dataset
Author: mariosasko | Pull request: yes | State: closed

Use `iter_files` in the `beans` and the `cats_vs_dogs` dataset scripts as suggested by @albertvillanova.

Additional changes:
* Fix `iter_files` in `MockDownloadManager` (see this [CI error](https://app.circleci.com/pipelines/github/huggingface/datasets/9247/workflows/2657ff8a-b531-4fd9-a9fc-6541a72e8d83/jobs/57028))
* Add support for `os.path.isdir` and `os.path.isfile` in streaming (`os.path.isfile` is needed in `StreamingDownloadManager`'s `iter_files` to make `cats_vs_dogs` streamable)

TODO:
- [ ] add tests for `xisdir` and `xisfile`
Extend support for streaming datasets that use ET.parse
Author: albertvillanova | Pull request: yes | State: closed

This PR extends the support in streaming mode for datasets that use `ET.parse`, by patching the function.

This PR adds support for streaming mode to datasets:
1. ami
1. assin
1. assin2
1. counter
1. enriched_web_nlg
1. europarl_bilingual
1. hyperpartisan_news_detection
1. polsum
1. qa4mre
1. quail
1. ted_talks_iwslt
1. udhr
1. web_nlg
1. winograd_wsc

CC: @severo
The rotten_tomatoes dataset of movie reviews contains some reviews in Spanish 
Author: puzzler10 | Pull request: no | State: open

## Describe the bug
See title. I don't think this is intentional and they probably should be removed. If they stay the dataset description should be at least updated to make it clear to the user. 

## Steps to reproduce the bug
Go to the [dataset viewer](https://huggingface.co/datasets/viewer/?dataset=rotten_tomatoes) for the dataset, set the offset to 4160 for the train dataset, and scroll through the results. I found ones at index 4166 and 4173. There's others too (e.g. index 2888) but those two are easy to find like that. 

## Expected results
English movie reviews only. 

## Actual results
Example of a Spanish movie review (4173): 
> "É uma pena que , mais tarde , o próprio filme abandone o tom de paródia e passe a utilizar os mesmos clichês que havia satirizado "
Iterating over a vision dataset doesn't decode the images
Author: lhoestq | Pull request: no | State: closed

## Describe the bug

If I load `mnist` and I iterate over the dataset, the images are not decoded, and the dictionary with the bytes is returned.

## Steps to reproduce the bug
```python
from datasets import load_dataset
import PIL

mnist = load_dataset("mnist", split="train")

first_image = mnist[0]["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # passes
first_image = next(iter(mnist))["image"]
assert isinstance(first_image, PIL.PngImagePlugin.PngImageFile)  # fails
```

## Expected results

The image should be decoded, as a PIL Image

## Actual results

We get a dictionary
```
{'bytes': b'\x89PNG\r\n\x1a\n\x00..., 'path': None}
```

## Environment info

- `datasets` version: 1.17.1.dev0
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyArrow version: 6.0.0

The bug also exists in 1.17.0

## Investigation

I think the issue is that decoding is disabled in `__iter__`:

https://github.com/huggingface/datasets/blob/dfe5b73387c5e27de6a16b0caeb39d3b9ded66d6/src/datasets/arrow_dataset.py#L1651-L1661

Do you remember why it was disabled in the first place @albertvillanova ?
Also cc @mariosasko @NielsRogge