noamaanMulla-03/datasets-issues huggingface dataset:
Batched in filter throws error

I hope this is really a bug, I could not find it among the open issues

## Describe the bug
using `batched=False` in DataSet.filter throws error
```python
TypeError: filter() got an unexpected keyword argument 'batched'
```
but in the docs it is lister as an argument.

## Steps to reproduce the bug
```python
task = "mnli"
max_length = 128
tokenizer = AutoTokenizer.from_pretrained("./pretrained_models/pretrained_models_drozd/sl250.m.gsic.titech.ac.jp:8000/21.11.17_06.30.32_roberta-base_a0057/checkpoints/smpl_400M/hf/")

dataset = load_dataset("glue", task)

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

##### tokenization_parameters
sentence1_key, sentence2_key = task_to_keys[task]
def preprocess_function(examples, max_length):
    if sentence2_key is None:
        return tokenizer(
            examples[sentence1_key], truncation=True, max_length=max_length
        )
    return tokenizer(
        examples[sentence1_key],
        examples[sentence2_key],
        truncation=False,
        padding="max_length",
        max_length=max_length,
    )

encoded_dataset = dataset.map(
    lambda x: preprocess_function(x, max_length=max_length), batched=False
)

encoded_dataset.filter(lambda x: len(x['input_ids']) <= max_length, batched=False)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1, 1.17.0
- Platform: ubuntu
- Python version: 3.8.12


Labels: name: bug
Audio datacard update - first pass

Filling out data card "Personal and Sensitive Information" for speech datasets to note PII concerns


I'm not sure that we want to change the tags at the top of the cards by hand. Those are used to create the tags in the hub. Although looking at all the tags now, we might want to normalize the current tags again (hyphens or no, ".0" or no). Maybe we could add a binary tag for public domain or not?

> 

That's a good point, I didn't realize these were auto-populated.
At the same time, some of them are wrong -- how/where are they auto-populated?  Seems like we should fix it at that source for the future.
In the mean time, I see that "cc0-1.0" is the desired tag for public domain, so I will change that for now.
the compression format requested when saving a dataset in json format is not respected

## Describe the bug
In the documentation of the `to_json` method, it is stated in the parameters that 
> **to_json_kwargs – Parameters to pass to pandas’s pandas.DataFrame.to_json.

however when we pass for example `compression="gzip"`, the saved file is not compressed.

Would you also have expected compression to be applied? :relaxed: 

## Steps to reproduce the bug
```python
my_dict = {"a": [1, 2, 3], "b": [1, 2, 3]}
```
### Result with datasets
```python
from datasets import Dataset

dataset = Dataset.from_dict(my_dict)
dataset.to_json("dic_with_datasets.jsonl.gz", compression="gzip")
!cat dic_with_datasets.jsonl.gz
```
output
```
{"a":1,"b":1}
{"a":2,"b":2}
{"a":3,"b":3}
```
Note: I would expected to see binary data here

### Result with pandas
```python
import pandas as pd

df = pd.DataFrame(my_dict)
df.to_json("dic_with_pandas.jsonl.gz", lines=True, orient="records", compression="gzip")
!cat dic_with_pandas.jsonl.gz
```
output
```
4��a�dic_with_pandas.jsonl��VJT�2�QJ��\� ��g��yƵ���������)���
```
Note: It looks like binary data

## Expected results

I would have expected that the saved result with datasets would also be a binary file

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.11
- PyArrow version: 5.0.0


Labels: name: bug

Thanks for reporting @SaulLu.

At first sight I think the problem is caused because `pandas` only takes into account the `compression` parameter if called with a non-null file path or buffer. And in our implementation, we call pandas `to_json` with `None` `path_or_buf`.

We should fix this:
- either handling directly the `compression` parameter ourselves
- or refactoring to pass non-null path or buffer to pandas

CC: @lhoestq

I was thinking if we can handle the `compression` parameter by ourselves? Compression types will be similar to what `pandas` offer. Initially, we can try this with 2-3 compression types and see how good/bad it is? Let me know if it sounds good, I can raise a PR for this next week

Hi ! Thanks for your help @bhavitvyamalik :)
Maybe let's start with `gzip` ? I think it's the most common use case, then if we're fine with it we can add other compression methods
make shape verification to use ArrayXD instead of nested lists for map

As describe in https://github.com/huggingface/datasets/issues/2005#issuecomment-793716753 and mentioned by @mariosasko in [image feature example](https://colab.research.google.com/drive/1mIrTnqTVkWLJWoBzT1ABSe-LFelIep1c#scrollTo=ow3XHDvf2I0B&line=1&uniqifier=1), IMO make shape verifcaiton to use ArrayXD instead of nested lists for map can help user reduce unnecessary cast. I notice datasets have done something special for `input_ids` and `attention_mask` which is also unnecessary after this feature added.

Labels: name: enhancement

Hi! 

Yes, this makes sense for numeric values, but first I have to finish https://github.com/huggingface/datasets/pull/3336 because currently ArrayXD only allows the first dimension to be dynamic.
URL query parameters are set as path in the compression hop for fsspec

## Describe the bug
There is an ssue with `StreamingDownloadManager._extract`.

I don't know how the test `test_streaming_gg_drive_gzipped` passes:

For 
```python
TEST_GG_DRIVE_GZIPPED_URL = "https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz"
urlpath = StreamingDownloadManager().download_and_extract(TEST_GG_DRIVE_GZIPPED_URL)
```
gives `urlpath`:
```python 
'gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz::https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz'
```

The gzip path makes no sense: `gzip://uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz`


## Steps to reproduce the bug
```python
from datasets.utils.streaming_download_manager import StreamingDownloadManager

dl_manager = StreamingDownloadManager()
urlpath = dl_manager.extract("https://drive.google.com/uc?export=download&id=1Bt4Garpf0QLiwkJhHJzXaVa0I0H5Qhwz")
print(urlpath)
```

## Expected results
The query parameters should not be set as part of the path.



Labels: name: bug

I think the test passes because it simply ignore what's after `gzip://`.

The returned urlpath is expected to look like `gzip://filename::url`, and the filename is currently considered to be what's after the final `/`, hence the result.

We can decide to change this and simply have `gzip://::url`, this way we don't need to guess the filename, what do you think ?
Add desc parameter to Dataset filter method

**Is your feature request related to a problem? Please describe.**
As I was filtering very large datasets I noticed the filter method doesn't have the desc parameter which is available in the map method. Why don't we add a desc parameter to the filter method both for consistency and it's nice to give some feedback to users during long operations on Datasets?

**Describe the solution you'd like**
Add desc parameter to Dataset filter method

**Describe alternatives you've considered**
N/A

**Additional context**
N/A


Labels: name: enhancement

Hi,

`Dataset.map` allows more generic transforms compared to `Dataset.filter`, which purpose is very specific (to filter examples based on a condition). That's why I don't think we need the `desc` parameter there for consistency. #3196 has added descriptions to the  `Dataset` methods that call `.map` internally, but not for the `filter` method, so we should do that.

Do you have a description in mind? Maybe `"Filtering the dataset"` or `"Filtering the indices"`? If yes, feel free to open a PR.

I'm personally ok with adding the `desc` parameter actually. Let's say you have different filters, it can be nice to differentiate between the different filters when they're running no ?

@mariosasko the use case is filtering of a dataset prior to tokenization and subsequent training. As the dataset is huge it's just a matter of giving a user (model trainer) some feedback on what's going on. Otherwise, feedback is given for all steps in training preparation and not for filtering and the filtering in my use case lasts about 4-5 minutes. And yes, if there are more filtering stages, as @lhoestq pointed out, it would be nice to give some feedback. I thought desc is there already and got confused when I got the script error. 

I don't have a strong opinion on that, so having `desc` as a parameter is also OK.
Fix to_tf_dataset references in docs

Fix the `to_tf_dataset` references in the docs. The currently failing example of usage will be fixed by #3338.  


The code snippet in [this section](https://huggingface.co/docs/datasets/master/use_dataset.html?highlight=to_tf_dataset#tensorflow) is missing an import (`DataCollatorWithPadding`) and doesn't initialize the TF model before the `model.fit` call.
How to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Hi,
I use torch.utils.data.Dataset to define my own data, but I need to use the 'map' function of datasets.arrow_dataset.Dataset later, so I hope to convert torch.utils.data.Dataset to datasets.arrow_dataset.Dataset.

Here is an example.
```
from torch.utils.data import Dataset
from datasets.arrow_dataset import Dataset as HFDataset

class ADataset(Dataset):
    def __init__(self, data): 
        super().__init__()
        self.data = data
    def __getitem__(self, index):
        return self.data[index]
    def __len__(self):
        return self.len
class MDataset():
    def __init__(self, tokenizer: AutoTokenizer, data_args, training_args):
        self.train_dataset = ADataset(data_args)
        self.tokenizer = tokenizer
        self.data_args = data_args

        self.train_dataset = self.train_dataset.map(
            self.process_function,
            batched=True,
            remove_columns=column_names,
            load_from_cache_file=True,
            desc="Running tokenizer on train dataset",
             )
    def process_function(self, examples):
        sentences = [" ".join(sample[0][3]) for sample in examples]

        tokenized = self.tokenizer(
            sentences,
            max_length=self.max_seq_len,
            padding=self.padding, 
            truncation=True)
```
But it would raise an ERROR, AttributeError: 'ADataset' object has no attribute 'map'.

so how to convert  torch.utils.data.Dataset to datasets.arrow_dataset.Dataset?

Thanks in advance!






Labels: name: enhancement
Ability to split a dataset in multiple files.

Hello,

**Is your feature request related to a problem? Please describe.**

My use case is that I have one writer that adds columns and multiple workers reading the same `Dataset`. Each worker should have access to columns added by the writer when they reload the dataset.

I understand that we shouldn't overwrite an arrow file as this could cause Segfault and so on. Before 1.16, I was able to overwrite the dataset and that would work most of the time with some retries.

**Describe the solution you'd like**

I was thinking that if we could append `Dataset._data_files`, when the workers reload the Dataset, they would get the new columns.


**Describe alternatives you've considered**
I currently need to
1. Save multiple "versions" of the dataset and load the latest.
2. Try working with cache files to get the latest columns.

**Additional context**

I think this would be a great addition to HFDataset as Parquet supports multi-files input out of the box!

I can make a PR myself with some pointers as needed :)

Labels: name: enhancement
Error about load_metric

## Describe the bug
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1371, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable

## Steps to reproduce the bug
```python
metric = load_metric("glue", "sst2")
```


## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 6.0.1


Labels: name: bug

Hi ! I wasn't able to reproduce your error.

Can you try to clear your cache at `~/.cache/huggingface/modules` and try again ?
fix: 🐛 pass token when retrieving the split names



Currently, it does not work with https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0/blob/main/common_voice_7_0.py#L146 (which was the goal), because `dl_manager.download_config.use_auth_token` is ignored, and the authentication is required to be use `huggingface-cli login`.
In my use case (dataset viewer), I'd prefer to use a specific "User Token Access", with only the "read" role (https://huggingface.co/settings/token).

See https://github.com/huggingface/datasets-preview-backend/issues/74#issuecomment-1007316853 for the context

> Simply passing download_config is ok :)

hmm, I prefer only passing use_auth_token. But the question is more: is it correct, in the (convoluted) case if `download_config.use_auth_token` exists and is different from `use_auth_token`? Which one should be used?

If both are passed, `use_auth_token` should have the priority (more specific parameters have the higher priority)
[Quick poll] Give your opinion on the future of the Hugging Face Open Source ecosystem!

Thanks to all of you, `datasets` will pass 11.5k stars :star2: this week!

If you have a couple of minutes and want to participate in shaping the future of the ecosystem, please share your thoughts: 

[**hf.co/oss-survey**](https://hf.co/oss-survey)

(please reply in the above feedback form rather than to this thread)

Thank you all on behalf of the HuggingFace team! 🤗
Add the KMWP & DKTC dataset.

Add the KMWP & DKTC dataset.
  
Additional notes:
- Both datasets will be released on January 10 through the GitHub link below.
  - https://github.com/tunib-ai/DKTC
  - https://github.com/tunib-ai/KMWP
- So it doesn't work as a link at the moment, but the code will work soon (after it is released on January 10).
Fix bug in `ImageClassifcation` task template

Fixes a bug in the `ImageClassification` task template which requires specifying class labels twice in dataset scripts. Additionally, this PR refactors the API around the classification task templates for cleaner `labels` handling.

CC: @lewtun @nateraw


The CI failures are unrelated to the changes in this PR.

> The CI failures are unrelated to the changes in this PR.

It seems that some of the failures are due to the tests on the dataset cards (e.g. CIFAR, MNIST, FASHION_MNIST). Perhaps it's worth addressing those in this PR to avoid confusing downstream developers who branch off `master` and suddenly have a failing CI?

@lewtun We only run these tests against the modified datasets on the PR branch, so this will not lead to errors after merging.
Add align_labels_with_mapping function

This PR adds a helper function to align the `label2id` mapping between a `datasets.Dataset` and a classifier (e.g. a transformer with a `PretrainedConfig.label2id` dict), with the alignment performed on the dataset itself.

This will help us with the Hub evaluation, where we won't know in advance whether a model that is fine-tuned on say MNLI has the same mappings as the MNLI dataset we load from `datasets`.

An example where this is needed is if we naively try to evaluate `microsoft/deberta-base-mnli` on `mnli` because the model config has the following mappings:

```python
  "id2label": {
    "0": "CONTRADICTION",
    "1": "NEUTRAL",
    "2": "ENTAILMENT"
  },
  "label2id": {
    "CONTRADICTION": 0,
    "ENTAILMENT": 2,
    "NEUTRAL": 1
  }
```

while the `mnli` dataset has the `contradiction` and `neutral` labels swapped:

```python
id2label = {0: 'entailment', 1: 'neutral', 2: 'contradiction'}
label2id = {'contradiction': 2, 'entailment': 0, 'neutral': 1}
```

As a result, we get a much lower accuracy during evaluation:

```python
from datasets import load_dataset
from transformers.trainer_utils import EvalPrediction
from transformers import AutoModelForSequenceClassification, Trainer

# load dataset for evaluation
mnli = load_dataset("glue", "mnli", split="test")
# load model
model_ckpt = "microsoft/deberta-base-mnli"
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
# preprocess, create trainer ...
mnli_enc = ...
trainer = Trainer(model, args=args, tokenizer=tokenizer)
# generate preds
preds = trainer.predict(mnli_enc)
# preds.label_ids misalinged with model.config => returns wrong accuracy (too low)!
compute_metrics(EvalPrediction(preds.predictions, preds.label_ids))
```

The fix is to use the helper function before running the evaluation to make sure the label IDs are aligned:

```python
mnli_enc_aligned = mnli_enc.align_labels_with_mapping(label2id=config.label2id, label_column="label")
# preds now aligned and everyone is happy :)
preds = trainer.predict(mnli_enc_aligned)
```

cc @thomwolf @lhoestq 


@lhoestq i think this is ready for another review 🙂 

@lhoestq thanks for the feedback - it's now integrated :) 

i also added a comment about sorting the input label IDs

Created the PR here: https://github.com/huggingface/datasets/pull/2510

> Thanks ! Looks all good now :)
> 
> We will also need to have the `DatasetDict.align_labels_with_mapping` method. Let me quickly add it

thanks a lot! i always forget about `DatasetDict` - will be happy when it's just one "dataset" object :)

So, there seems to be a problem with the function align_labels_with_mapping for models like this: https://huggingface.co/huggingface/distilbert-base-uncased-finetuned-mnli]. At least with this model, but perhaps also with others, the model.config.label2id values are of type str not int, which crashes said function. After manually converting the model.config.label2id values to int, the script runs smoothly.
Add RedCaps dataset

## Adding a Dataset
- **Name:** RedCaps
- **Description:** Web-curated image-text data created by the people, for the people
- **Paper:** https://arxiv.org/abs/2111.11431
- **Data:** https://redcaps.xyz/
- **Motivation:** Multimodal image-text dataset:  12M+ Image-text pairs 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Proposed by @patil-suraj 

Labels: name: dataset request

name: vision
Add RedCaps dataset

Add the RedCaps dataset. I'm not adding the generated `dataset_infos.json` file for now due to its size (11 MB). 

TODOs:
- [x] dummy data
- [x] dataset card

Close #3316 


Cool ! If you want you can include `dataset_infos.json` but only for the main configurations. That's what we do for example for translation datasets when there are too many configs

@lhoestq I've added an example that uses `map` to download the images.
DuplicatedKeysError when loading tweet_qa dataset

When loading the tweet_qa dataset with `load_dataset('tweet_qa')`, the following error occurs: 

`DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 2a167f9e016ba338e1813fed275a6a1e
Keys should be unique and deterministic in nature
`
Might be related to issues #2433 and #2333

- `datasets` version: 1.17.0
- Python version: 3.8.5


Labels: name: bug

Hi, we've just merged the PR with the fix. The fixed version of the dataset can be downloaded as follows:
```python
import datasets
dset = datasets.load_dataset("tweet_qa", revision="master")
```
Add parameter `preserve_index` to `from_pandas`

Added optional parameter, so that user can get rid of useless index preserving. [Issue](https://github.com/huggingface/datasets/issues/3563)


> 

I did `make style` and it affected over 500 files

```
All done! ✨ 🍰 ✨
575 files reformatted, 372 files left unchanged.
isort tests src benchmarks datasets/**/*.py metri
```

(result)
![image](https://user-images.githubusercontent.com/20703486/149166681-2f9d1bc4-116a-4f53-ad42-e54e3b8bd605.png)


Nvm I was using wrong black version
Dataset.from_pandas preserves useless index

## Describe the bug
Let's say that you want to create a Dataset object from pandas dataframe. Most likely you will write something like this:

```
import pandas as pd
from datasets import Dataset


df = pd.read_csv('some_dataset.csv')
# Some DataFrame preprocessing code...
dataset = Dataset.from_pandas(df)
```
If your preprocessing code contain indexing operations like this:
```
df = df[df.col1 == some_value]
```
then your df.index can be changed from (default) ```RangeIndex(start=0, stop=16590, step=1)``` to something like this ```Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,
                9,
            ...
            83979, 83980, 83981, 83982, 83983, 83984, 83985, 83986, 83987,
            83988],
           dtype='int64', length=16590)```

In this case, PyArrow (by default) will preserve this non-standard index. In the result, your dataset object will have the extra field that you likely don't want to have: '__index_level_0__'.

You can easily fix this by just adding extra argument ```preserve_index=False``` to call of ```InMemoryTable.from_pandas``` in ```arrow_dataset.py```.

If you approve that this isn't desirable behavior, I can make a PR fixing that.

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-44-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1



Labels: name: bug

Hi! That makes sense. Sure, feel free to open a PR! Just a small suggestion: let's make `preserve_index` a parameter of `Dataset.from_pandas` (which we then pass to `InMemoryTable.from_pandas`) with `None` as a default value to not have this as a breaking change.
Add Species-800



thanks @lhoestq ! I probably need to do the same change in the `SplitGenerator`s (lines 107, 110 and 113). I'll open a new PR for that

Yes indeed ! Good catch 👍 
Feel free to open a PR and ping me

Hi , theres a issue pulling species_800 dataset , throws  google drive  error 

error: 

```
raise ConnectionError(f"Couldn't reach {url} ({repr(head_error)})")
ConnectionError: Couldn't reach https://drive.google.com/u/0/uc?id=1OletxmPYNkz2ltOr9pyT0b0iBtUWxslh&export=download/ (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='drive.google.com', port=443): Read timed out. (read timeout=10)")))
```
code: 
```
from datasets import load_dataset

dataset = load_dataset("species_800")
```

Hi @obonyojimmy! I am running the same commands and they work for me. Did you check your internet connection?