ahmedfarazsyk/datasets-github-issues huggingface dataset:
Can't import `datasets` in python 3.10
Author: lhoestq
State: closed

When importing `datasets` I'm getting this error in python 3.10:
```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_dataset.py", line 47, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_reader.py", line 33, in <module>
    from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 334, in <module>
    class InMemoryTable(TableBlock):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 361, in InMemoryTable
    def from_pandas(cls, *args, **kwargs):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 24, in wrapper
    out = wraps(arrow_table_method)(method)
  File "/Users/quentinlhoest/.pyenv/versions/3.10.0/lib/python3.10/functools.py", line 61, in update_wrapper
    wrapper.__wrapped__ = wrapped
AttributeError: readonly attribute
```

This makes the conda build fail.
I'm opening a PR to fix this and do a patch release 1.16.1
Can't get tatoeba.rus dataset
Author: mmg10
State: closed
Labels: name: bug

## Describe the bug
It gives an error.

> FileNotFoundError: Couldn't find file at https://github.com/facebookresearch/LASER/raw/master/data/tatoeba/v1/tatoeba.rus-eng.rus

## Steps to reproduce the bug
```python
data=load_dataset("xtreme","tatoeba.rus", split="validation")
```

## Solution
The library tries to access the **master** branch. In the github repo of facebookresearch, it is in the **main** branch.
Add desc parameter to Dataset filter method
Author: vblagoje
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
As I was filtering very large datasets I noticed the filter method doesn't have the desc parameter which is available in the map method. Why don't we add a desc parameter to the filter method both for consistency and it's nice to give some feedback to users during long operations on Datasets?

**Describe the solution you'd like**
Add desc parameter to Dataset filter method

**Describe alternatives you've considered**
N/A

**Additional context**
N/A


Comments: Hi,

`Dataset.map` allows more generic transforms compared to `Dataset.filter`, which purpose is very specific (to filter examples based on a condition). That's why I don't think we need the `desc` parameter there for consistency. #3196 has added descriptions to the  `Dataset` methods that call `.map` internally, but not for the `filter` method, so we should do that.

Do you have a description in mind? Maybe `"Filtering the dataset"` or `"Filtering the indices"`? If yes, feel free to open a PR.

I'm personally ok with adding the `desc` parameter actually. Let's say you have different filters, it can be nice to differentiate between the different filters when they're running no ?

@mariosasko the use case is filtering of a dataset prior to tokenization and subsequent training. As the dataset is huge it's just a matter of giving a user (model trainer) some feedback on what's going on. Otherwise, feedback is given for all steps in training preparation and not for filtering and the filtering in my use case lasts about 4-5 minutes. And yes, if there are more filtering stages, as @lhoestq pointed out, it would be nice to give some feedback. I thought desc is there already and got confused when I got the script error. 

I don't have a strong opinion on that, so having `desc` as a parameter is also OK.
Add RedCaps dataset
Author: albertvillanova
State: closed
Labels: name: dataset request

name: vision

## Adding a Dataset
- **Name:** RedCaps
- **Description:** Web-curated image-text data created by the people, for the people
- **Paper:** https://arxiv.org/abs/2111.11431
- **Data:** https://redcaps.xyz/
- **Motivation:** Multimodal image-text dataset:  12M+ Image-text pairs 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Proposed by @patil-suraj
TriviaQA License Mismatch
Author: akhilkedia
State: closed
Labels: name: bug

## Describe the bug

TriviaQA Webpage at http://nlp.cs.washington.edu/triviaqa/ says they do not own the copyright to the data. However, Huggingface datasets at https://huggingface.co/datasets/trivia_qa mentions that the dataset is released under Apache License

Is the License Information on HuggingFace correct?

Comments: Hi ! You're completely right, this must be mentioned in the dataset card.
If you're interesting in contributing, feel free to open a pull request to mention this in the `trivia_qa` dataset card in the "Licensing Information" section at https://github.com/huggingface/datasets/blob/master/datasets/trivia_qa/README.md
Add WebSRC
Author: NielsRogge
State: open
Labels: name: dataset request

## Adding a Dataset
- **Name:** WebSRC
- **Description:** WebSRC is a novel Web-based Structural Reading Comprehension dataset. It consists of 0.44M question-answer pairs, which are collected from 6.5K web pages with corresponding HTML source code, screenshots and metadata. 
- **Paper:** https://arxiv.org/abs/2101.09465
- **Data:** https://x-lance.github.io/WebSRC/dashboard.html#
- **Motivation:** Currently adding MarkupLM to HuggingFace Transformers, which achieves SOTA on this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
"dataset_infos.json" missing for chr_en and mc4
Author: amitness
State: open
Labels: name: bug

name: dataset bug

## Describe the bug
In the repository, every dataset has its metadata in a file called`dataset_infos.json`. But, this file is missing for two datasets: `chr_en` and `mc4`.

## Steps to reproduce the bug
Check [chr_en](https://github.com/huggingface/datasets/tree/master/datasets/chr_en) and [mc4](https://github.com/huggingface/datasets/tree/master/datasets/mc4)

Comments: Hi ! Thanks for reporting :) 
We can easily add the metadata for `chr_en` IMO, but for mC4 it will take more time, since it requires to count the number of examples in each language

No problem. I am trying to do some analysis on the metadata of all available datasets. Is reading `metadata_infos.json` for each dataset the correct way to go? 

I noticed that the same information is also available as special variables inside .py file of each dataset. So, I was wondering if `metadata_infos.json` has been deprecated?

![image](https://user-images.githubusercontent.com/8587189/142914413-a95a1abf-6f3e-4fbe-96e5-16d3ca39c831.png)


The `dataset_infos.json` files have more information and are made to be used to analyze the datasets without having to run/parse the python scripts. Moreover some datasets on the Hugging face don't even have a python script, and for those ones we'll make tools to generate the JSON file automatically :)
nested sequence feature won't encode example if the first item of the outside sequence is an empty list
Author: function2-llx
State: closed
Labels: name: bug

## Describe the bug
As the title, nested sequence feature won't encode example if the first item of the outside sequence is an empty list.

## Steps to reproduce the bug
```python
from datasets import Features, Sequence, ClassLabel
features = Features({
    'x': Sequence(Sequence(ClassLabel(names=['a', 'b']))),
})
print(features.encode_batch({
    'x': [
        [['a'], ['b']],
        [[], ['b']],
    ]
}))
```

## Expected results
print `{'x': [[[0], [1]], [[], ['1']]]}`

## Actual results
print `{'x': [[[0], [1]], [[], ['b']]]}`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux-5.13.0-21-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.0

## Additional information
I think the issue stems from [here](https://github.com/huggingface/datasets/blob/8555197a3fe826e98bd0206c2d031c4488c53c5c/src/datasets/features/features.py#L847-L848).



Comments: knock knock

Hi, thanks for reporting! I've linked a PR that should fix the issue.

I've checked the PR and it looks great, thanks a lot!
Dataset object has no attribute `to_tf_dataset`
Author: RajkumarGalaxy
State: closed
Labels: name: bug

I am following HuggingFace Course. I am at Fine-tuning a model. 
Link: https://huggingface.co/course/chapter3/2?fw=tf

I use tokenize_function and `map` as mentioned in the course to process data.

`# define a tokenize function`
`def Tokenize_function(example):`
`    return tokenizer(example['sentence'], truncation=True)`

`# tokenize entire data`
`tokenized_data = raw_data.map(Tokenize_function, batched=True)`

I get Dataset object at this point. When I try converting this to a TF dataset object as mentioned in the course, it throws the following error.

`# convert to TF dataset`
`train_data = tokenized_data["train"].to_tf_dataset( `
`   columns = ['attention_mask', 'input_ids', 'token_type_ids'], `
`    label_cols = ['label'], `
`    shuffle = True, `
`    collate_fn = data_collator, `
`    batch_size = 8 `
`)`

Output:

`---------------------------------------------------------------------------`
`AttributeError                            Traceback (most recent call last)`
`/tmp/ipykernel_42/103099799.py in <module>`
`      1 # convert to TF dataset`
`----> 2 train_data = tokenized_data["train"].to_tf_dataset( \`
`      3     columns = ['attention_mask', 'input_ids', 'token_type_ids'], \`
`      4     label_cols = ['label'], \`
`      5     shuffle = True, \`
`AttributeError: 'Dataset' object has no attribute 'to_tf_dataset'`

When I look for `dir(tokenized_data["train"])`, there is no method or attribute in the name of `to_tf_dataset`.

Why do I get this error? And how to clear this?

Please help me.

Comments: The issue is due to the older version of transformers and datasets. It has been resolved by upgrading their versions.

```
# upgrade transformers and datasets to latest versions
!pip install --upgrade transformers
!pip install --upgrade datasets
```

Regards!
DataCollatorWithPadding: TypeError
Author: RajkumarGalaxy
State: closed
Labels: name: bug

Hi,
I am following the HuggingFace course. I am now at Fine-tuning [https://huggingface.co/course/chapter3/3?fw=tf](https://huggingface.co/course/chapter3/3?fw=tf).  When I set up `DataCollatorWithPadding` as following I got an error while trying to reproduce the course code in Kaggle. This error occurs with either a CPU-only-device or a GPU-device.

Input:
```checkpoint = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```

Output:
```---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_42/1563280798.py in <module>
      1 checkpoint = 'bert-base-uncased'
      2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")
TypeError: __init__() got an unexpected keyword argument 'return_tensors'
```

When I call `help` method, it too confirms that there is no argument `return_tensors`.
Input:
```
help(DataCollatorWithPadding.__init__)
```
Output:
```
Help on function __init__ in module transformers.data.data_collator:

__init__(self, tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, padding: Union[bool, str, transformers.file_utils.PaddingStrategy] = True, max_length: Union[int, NoneType] = None, pad_to_multiple_of: Union[int, NoneType] = None) -> None
```

But, the source file *[Data Collator - docs](https://huggingface.co/transformers/main_classes/data_collator.html#datacollatorwithpadding)* says that there is such an argument. By default, it returns Pytorch tensors while I need TF tensors.

Where do I miss?
Please help me.

Comments: 
> 
> Input:
> 
> ```
> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
> ```
> 
> Output:
> 
> ```
> TypeError                                 Traceback (most recent call last)
> /tmp/ipykernel_42/1563280798.py in <module>
>       1 checkpoint = 'bert-base-uncased'
>       2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
> ----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")
> TypeError: __init__() got an unexpected keyword argument 'return_tensors'
> ```
> 

The issue is due to the older version of transformers and datasets. It has been resolved by upgrading their versions.

`# upgrade transformers and datasets to latest versions`
`!pip install --upgrade transformers`
`!pip install --upgrade datasets`

Cheers!
Add option to find unique elements in nested sequences when calling `Dataset.unique`
Author: mariosasko
State: open
Labels: name: enhancement

It would be nice to have an option to flatten nested sequences to find unique elements stored in them when calling `Dataset.unique`. ~~Currently, `Dataset.unique` only supports finding unique sequences and not unique elements in that situation.~~

Comments: Hi @mariosasko!

Has this been patched into any of the releases?

Hi! Not yet, would you be interested in contributing a PR? I can give you some pointers if needed. 

@mariosasko did this ever get implemented? Willing to help if you are still up for it.

@dcruiz01 No, but here is an example of how to do this with the existing API:


```python
ds = Dataset.from_dict({"tokens": [["a", "b"], ["c", "a"], ["c", "e"]]})

def flatten_tokens(pa_table):
    return pa.table([pc.list_flatten(pa_table["tokens"])], ["flat_tokens"])

ds = ds.with_format("arrow")
ds = ds.map(flatten_tokens, batched=True)
ds = ds.with_format(None)

unique_tokens = ds.unique("flat_tokens")
```

When I think about it, `.unique` on `Sequence(Value(...))` should return unique sequences/arrays, not unique elements of these sequences...
Agnews dataset viewer is not working
Author: pietrolesci
State: closed
Labels: name: dataset-viewer

## Dataset viewer issue for '*name of the dataset*'

**Link:** https://huggingface.co/datasets/ag_news

Hi there, the `ag_news` dataset viewer is not working.

Am I the one who added this dataset? No


Comments: Hi ! Thanks for reporting
We've already fixed the code that generates the preview for this dataset, we'll release the fix soon :)

Hi @lhoestq, thanks for your feedback!

Fixed in the viewer.

https://huggingface.co/datasets/ag_news
Temporary dataset_path for remote fs URIs not built properly in arrow_dataset.py::load_from_disk
Author: francisco-perez-sorrosal
State: closed
Labels: name: bug

## Describe the bug
When trying to build a temporary dataset path from a remote URI in this block of code:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

the result is not the expected when passing an absolute path in an URI like `hdfs:///absolute/path`.

## Steps to reproduce the bug
```python
dataset_path = "hdfs:///absolute/path"
src_dataset_path = extract_path_from_uri(dataset_path)
tmp_dir = get_temporary_cache_files_directory()
dataset_path = Path(tmp_dir, src_dataset_path)
print(dataset_path)
```

## Expected results
With the code above, we would expect a value in `dataset_path` similar to:
`/tmp/tmpnwxyvao5/absolute/path`

## Actual results
However, we get a `dataset_path` value like:
`/absolute/path`

This is because this line here: https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1041
returns the last absolute path when two absolute paths (the one in `tmp_dir` and the one extracted from the URI in `src_dataset_path`) are passed as arguments.

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0


Comments: Hi ! Good catch and thanks for opening a PR :)

I just responded in your PR
Add Natural Adversarial Objects dataset
Author: osanseviero
State: open
Labels: name: dataset request

name: vision

## Adding a Dataset
- **Name:** Natural Adversarial Objects (NAO)
- **Description:** Natural Adversarial Objects (NAO) is a new dataset to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence.
- **Paper:** https://arxiv.org/abs/2111.04204v1
- **Data:** https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8
- **Motivation:** interesting object detection dataset useful for miscclassifications

cc @NielsRogge 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add IEMOCAP dataset
Author: osanseviero
State: open
Labels: name: dataset request

name: speech

name: vision

## Adding a Dataset
- **Name:** IEMOCAP
- **Description:** acted, multimodal and multispeaker database
- **Paper:** https://sail.usc.edu/iemocap/Busso_2008_iemocap.pdf
- **Data:** https://sail.usc.edu/iemocap/index.html
- **Motivation:** Useful multimodal dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: The IEMOCAP dataset is private and available only on request.
```
To obtain the IEMOCAP data you just need to fill out an electronic release form below.
```

- [Request form](https://sail.usc.edu/iemocap/release_form.php)
- [License ](https://sail.usc.edu/iemocap/Data_Release_Form_IEMOCAP.pdf)


> We do not share the dataset for commercial purposes due to privacy concerns surrounding the participants of the research. The login details will only be emailed to the given academic email address.

I think it won't be possible to add this dataset to 🤗 datasets.

Hi @dnaveenr ! We can contact the authors to see if they are interested in hosting the dataset on the Hub. In the meantime, feel free to work on a script with manual download.

Hi @mariosasko . Thanks for your response. Sure, I will mail them and find out if they're open to this.

Work on a script with manual download ? This is new to me, any guidelines would be helpful here.


> Thanks for your response. Sure, I will mail them and find out if they're open to this.

It's best to leave this part to us because we have to explain how login would work and (potentially) set up a custom verification for the dataset.

> Work on a script with manual download ? This is new to me, any guidelines would be helpful here.

For instance, this is one of the scripts with manual download: https://huggingface.co/datasets/arxiv_dataset. Compared to the standard dataset, it has the `manual_download_instructions` attribute and uses `dl_manager.manual_dir` (derived from `load_dataset(..., data_dir="path/to/data")`) to access the dataset's data files.

> It's best to leave this part to us because we have to explain how login would work and (potentially) set up a custom verification for the dataset.

Yes. That would be perfect. Thanks.

----
Okay. Thanks for giving a reference. This is helpful. I will go through it.



@mariosasko  has this been solved? I would like to use login and custom verification for training on my private dataset.

@flckv I think the [gating mechanism](https://huggingface.co/docs/hub/datasets-gated) is what you are looking for. 

@mariosasko Thanks, but no. I would like to keep my HuggingFace Dataset private and train a model on it. Is this possible?
Add VoxLingua107 dataset
Author: osanseviero
State: open
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models. The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** Nice audio classification dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: #self-assign
Add Speech Commands dataset
Author: osanseviero
State: closed
Labels: name: dataset request

name: speech

## Adding a Dataset
- **Name:** Speech commands
- **Description:** A Dataset for Limited-Vocabulary Speech Recognition
- **Paper:** https://arxiv.org/abs/1804.03209
- **Data:** https://www.tensorflow.org/datasets/catalog/speech_commands,  Available:
http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
- **Motivation:** Nice dataset for audio classification training

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: #self-assign