NyssaW/github-issues huggingface dataset:
iter_archive for zip files 

* In this PR, I added the option to iterate through zipfiles for `download_manager.py` only. 
* Next PR will be the same applied to `streaming_download_manager.py`.
* Related issue #3272.
## Comments : 
* There is no `.isreg()` equivalent in zipfile library to check if file is Regular so I used `.is_dir()` instead to skip directories.
* For now I got `streaming_download_manager.py`  working for local zip files, but not for urls. I get the following error when I test it on an archive in google drive, so still working on it.  `BlockSizeError: Got more bytes so far (>2112) than requested (22)`


## Tasks : 
- [x] download_manager.py
- [ ] streaming_download_manager.py

Comments:
And also don't always try streaming with Google Drive - it can have issues because of how Google Drive works (with quotas, restrictions, etc.) and it can indeed cause `BlockSizeError`.

Feel free to host your test data elsewhere, such as in a dataset repository on https://huggingface.co (see [here](https://huggingface.co/docs/datasets/upload_dataset.html#upload-your-files) for a tutorial on how to upload files)
Better error message when download fails

From our discussions in https://github.com/huggingface/datasets/issues/3269 and https://github.com/huggingface/datasets/issues/3282 it would be nice to have better messages if a download fails.

In particular the error now shows:
- the error from the HEAD request if there's one
- otherwise the response code of the HEAD request

I also added an error to tell users to pass `use_auth_token` when the Hugging Face Hub returns 401 (Unauthorized).

While paying around with this I also fixed a minor issue with the `force_download` parameter that was not always taken into account

Comments:
Fix ASSET dataset data URLs

Change the branch name "master" to "main" in the data URLs, since facebookresearch has changed that.

Comments:
> Hi @tianjianjiang, thanks for the fix.
> The links should also be updated in the `dataset_infos.json` file.
> The failing tests are due to the missing tag in the header of the `README.md` file:

Hi @albertvillanova, thank you for the info! My apologies for the messy PR.
Mirror the canonical datasets to the Hugging Face Hub

- [ ] create a repo on https://hf.co/datasets for every canonical dataset
- [ ] on every commit related to a dataset, update the hf.co repo

See https://github.com/huggingface/moon-landing/pull/1562

@SBrandeis: I let you edit this description if needed to precise the intent.

Comments:
I created a GitHub project to keep track of what needs to be done:
https://github.com/huggingface/datasets/projects/3

I also store my code in a (private for now) repository at https://github.com/huggingface/mirror_canonical_datasets_on_hub

I understand that the datasets are mirrored on the Hub now, right? Might I close @lhoestq @SBrandeis?
Fix JSON ClassLabel casting for integers

Loading a JSON dataset with ClassLabel feature types currently fails if the JSON data already has integers. Indeed currently it tries to convert the strings to integers without even checking if the data are not integers already.

For example this currently fails:
```python
from datasets import load_dataset, Features, ClassLabel

path = "data.json"
f = Features({"a": ClassLabel(names=["neg", "pos"])})
d = load_dataset("json", data_files=path, features=f)
```
data.json
```json
{"a": 0}
{"a": 1}
```

I fixed that by adding a line that checks the type of the JSON data before trying to convert them

cc @albertvillanova let me know if it sounds good to you

Comments:
Typing of Dataset.__getitem__ could be improved.

## Describe the bug

The newly added typing for Dataset.__getitem__ is Union[Dict, List]. This makes tools like mypy a bit awkward to use as we need to check the type manually. We could use type overloading to make this easier. [Documentation](https://docs.python.org/3/library/typing.html#typing.overload)

## Steps to reproduce the bug
Let's have a file `test.py`

```python
from typing import List, Dict, Any

from datasets import Dataset

ds = Dataset.from_dict({
    'a': [1,2,3],
    'b': ["1", "2", "3"]
})

one_colum: List[str] = ds['a']
some_index: Dict[Any, Any] = ds[1]
```

## Expected results

Running `mypy test.py` should not give any error.


## Actual results

```
test.py:10: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "List[str]")
test.py:11: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "Dict[Any, Any]")
Found 2 errors in 1 file (checked 1 source file)
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.1


Comments:
Hi ! Thanks for the suggestion, I didn't know about this decorator.

If you are interesting in contributing, feel free to open a pull request to add the overload methods for each typing combination :) To assign you to this issue, you can comment `#self-assign` in this thread.

`Dataset.__getitem__` is defined right here: https://github.com/huggingface/datasets/blob/e6f1352fe19679de897f3d962e616936a17094f5/src/datasets/arrow_dataset.py#L1840

#self-assign
Add support for multiple dynamic dimensions and to_pandas conversion for dynamic arrays

Add support for multiple dynamic dimensions (e.g. `(None, None, 3)` for arbitrary sized images) and `to_pandas()` conversion for dynamic arrays.

TODOs:
* [ ] Cleaner code
* [ ] Formatting issues (if NumPy doesn't allow broadcasting even though dtype is np.object)
* [ ] Fix some issues with zero-dim tensors 
* [ ] Tests


Comments:
Integrate Polars library

Check potential integration of the Polars library: https://github.com/pola-rs/polars
- Benchmark: https://h2oai.github.io/db-benchmark/

CC: @thomwolf @lewtun 


Comments:
If possible, a neat API could be something like `Dataset.to_polars()`, as well as `Dataset.set_format("polars")`

Note they use a "custom" implementation of Arrow: [Arrow2](https://github.com/jorgecarleitao/arrow2).

Polars has grown rapidly in popularity over the last year - could you consider integrating the Polars functionality again?

I don't think the "custom" implementation should be a barrier, it still conforms to the Arrow specification 

Is there some direction regarding this from the HF team @lewtun ? Can conversion from polars to HF dataset be implemented with limited/zero copy? So, something like ``Dataset.from_polars()`` and ``Dataset.to_polars()`` like you mentioned. Happy to contribute if I can get some pointers on how this may be implemented.

Hi, is there any updates? Thanks!

> Hi, is there any updates? Thanks!

The feature has been there for a bit 😊 You can call `dataset.to_polars()` (on a `Dataset`, not a `DatasetDict`). The issue can be closed, I guess! @lhoestq 

Looks great and thanks!

Thank you.
Fix error message and add extension fallback

Fix the error message raised if `infered_module_name` is `None` in `CommunityDatasetModuleFactoryWithoutScript.get_module` and make `infer_module_for_data_files` more robust. 

In the linked issue, `infer_module_for_data_files` returns `None` because `json` is the second most common extension due to the suffix ordering. Now,  we go from the most common to the least common extension and try to map it or return `None`.

Fix #3331

Comments:
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'

## Describe the bug
I add a new question answering dataset to huggingface datasets manually. Here is the link: [luozhouyang/question-answering-datasets](https://huggingface.co/datasets/luozhouyang/question-answering-datasets)

But when I load the dataset, an error raised: 

```bash
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
```

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("luozhouyang/question-answering-datasets", data_files=["dureader_robust.train.json"])
```

## Expected results
Load dataset successfully without any error.

## Actual results
```bash
Traceback (most recent call last):
  File "/mnt/home/zhouyang.lzy/github/naivenlp/naivenlp/tests/question_answering_tests/dataset_test.py", line 89, in test_load_dataset_with_hf
    data_files=["dureader_robust.train.json"],
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1616, in load_dataset
    **config_kwargs,
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1443, in load_dataset_builder
    path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1157, in dataset_module_factory
    raise e1 from None
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1144, in dataset_module_factory
    download_mode=download_mode,
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 798, in get_module
    raise FileNotFoundError(f"No data files or dataset script found in {self.path}")
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: linux
- Python version: 3.6.13
- PyArrow version: 6.0.1


Comments:
Hi,

the fix was merged and will be available in the next release of `datasets`.
In the meantime, you can use it by installing `datasets` directly from master as follows:
```
pip install git+https://github.com/huggingface/datasets.git
```
Map function: Type error on iter #999

## Describe the bug
Using the map function, it throws a type error on iter #999

Here is the code I am calling:
```
dataset = datasets.load_dataset('squad')
dataset['validation'].map(text_numbers_to_int, input_columns=['context'], fn_kwargs={'column': 'context'})
``` 
text_numbers_to_int returns the input text with numbers replaced in the format {'context': text}

It happens at 
`
File "C:\Users\lonek\anaconda3\envs\ai\Lib\site-packages\datasets\arrow_writer.py", line 289, in <listcomp>
    [row[0][col] for row in self.current_examples], type=col_type, try_type=col_try_type, col=col
`

The issue is that the list comprehension expects self.current_examples to be type tuple(dict, str), but for some reason 26 out of 1000 of the sefl.current_examples are type tuple(str, str)

Here is an example of what self.current_examples should be
({'context': 'Super Bowl 50 was an...merals 50.'}, '')

Here is an example of what self.current_examples are when it throws the error:
('The Panthers used th... Marriott.', '')


Comments:
Hi, thanks for reporting.

It would be really helpful if you could provide the actual code of the `text_numbers_to_int` function so we can reproduce the error.

```
def text_numbers_to_int(text, column=""):
    """
    Convert text numbers to int.

    :param text: text numbers
    :return: int
    """
    try:
        numbers = find_numbers(text)
        if not numbers:
            return text
        result = ""
        i, j = 0, 0
        while i < len(text):
            if j < len(numbers) and i == numbers[j][1]:
                n = int(numbers[j][0]) if numbers[j][0] % 1 == 0 else float(numbers[j][0])
                result += str(n)
                i = numbers[j][2] #end
                j += 1
            else:
                result += text[i]
                i += 1
        if column:
            return{column: result}
        else:
            return {column: result}
    except Exception as e:
        print(e)
        return {column: result}
```

Maybe this is because of the `return text` line ? I think it should return a dictionary rather than a string

Yes that was it, good catch! Thanks
Quick fix error formatting

While working on a dataset, I got the error
```
TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types {[type(x) for x in processed_inputs.values()]}. When using `batched=True`, make sure provided `function` returns a `dict` of types like `{allowed_batch_return_types}`.
```

This PR should fix the formatting of this error

Comments:
"Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"

## Describe the bug
Passing a correctly shaped Numpy-Array to get_nearest_examples leads to the Exception

"Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"

Probably the reason for this is a wrongly converted assertion.

1.15.1:

`assert len(query.shape) == 1 or (len(query.shape) == 2 and query.shape[0] == 1)`

1.16.1:

```
 if len(query.shape) != 1 or (len(query.shape) == 2 and query.shape[0] != 1):
            raise ValueError("Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)")
```

## Steps to reproduce the bug

follow the steps described here: https://huggingface.co/course/chapter5/6?fw=tf

```python
 question_embedding.shape # (1, 768)

 scores, samples = embeddings_dataset.get_nearest_examples(
    "embeddings", question_embedding, k=5 # Error
)

# "Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"
```

## Expected results
Should work without exception

## Actual results
Throws exception

## Environment info
- `datasets` version: 1.15.1
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.12
- PyArrow version: 6.0.


Comments:
#3323
Can't import `datasets` in python 3.10

When importing `datasets` I'm getting this error in python 3.10:
```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_dataset.py", line 47, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_reader.py", line 33, in <module>
    from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 334, in <module>
    class InMemoryTable(TableBlock):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 361, in InMemoryTable
    def from_pandas(cls, *args, **kwargs):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 24, in wrapper
    out = wraps(arrow_table_method)(method)
  File "/Users/quentinlhoest/.pyenv/versions/3.10.0/lib/python3.10/functools.py", line 61, in update_wrapper
    wrapper.__wrapped__ = wrapped
AttributeError: readonly attribute
```

This makes the conda build fail.
I'm opening a PR to fix this and do a patch release 1.16.1

Comments:
Update URL of tatoeba subset of xtreme

Updates the URL of the tatoeba subset of xtreme. Additionally, replaces `os.path.join` with `xjoin` to correctly join the URL segments on Windows.

Fix #3320 

Comments:

To be more precise: `os.path.join` is replaced on-the-fly by `xjoin` anyway with patching, to extend it to remote files


Oh actually just ignore what I said: they were used to concatenate URLs, which is not recommended. Let me fix that again by appending using `+`
Can't get tatoeba.rus dataset

## Describe the bug
It gives an error.

> FileNotFoundError: Couldn't find file at https://github.com/facebookresearch/LASER/raw/master/data/tatoeba/v1/tatoeba.rus-eng.rus

## Steps to reproduce the bug
```python
data=load_dataset("xtreme","tatoeba.rus", split="validation")
```

## Solution
The library tries to access the **master** branch. In the github repo of facebookresearch, it is in the **main** branch.

Comments:
Add push_to_hub docs

Since #3098 it's now possible to upload a dataset on the Hub directly from python using the `push_to_hub` method.
I just added a section in the "Upload a dataset to the Hub" tutorial.

I kept the section quite simple but let me know if it sounds good to you @LysandreJik @stevhliu :)

Comments:
Looks good to me! :)

Maybe we can mention that users can also set the `private` argument if they want to keep their dataset private? It would lead nicely into the next section on Privacy.

Thanks for your comments, I fixed the capitalization for consistency and added an passage to mention the `private` parameter and to have a nice transition to the Privacy section :)

I also added the login instruction that was missing before the user can actually upload a dataset.
Add desc parameter to Dataset filter method

**Is your feature request related to a problem? Please describe.**
As I was filtering very large datasets I noticed the filter method doesn't have the desc parameter which is available in the map method. Why don't we add a desc parameter to the filter method both for consistency and it's nice to give some feedback to users during long operations on Datasets?

**Describe the solution you'd like**
Add desc parameter to Dataset filter method

**Describe alternatives you've considered**
N/A

**Additional context**
N/A


Comments:
Hi,

`Dataset.map` allows more generic transforms compared to `Dataset.filter`, which purpose is very specific (to filter examples based on a condition). That's why I don't think we need the `desc` parameter there for consistency. #3196 has added descriptions to the  `Dataset` methods that call `.map` internally, but not for the `filter` method, so we should do that.

Do you have a description in mind? Maybe `"Filtering the dataset"` or `"Filtering the indices"`? If yes, feel free to open a PR.

I'm personally ok with adding the `desc` parameter actually. Let's say you have different filters, it can be nice to differentiate between the different filters when they're running no ?

@mariosasko the use case is filtering of a dataset prior to tokenization and subsequent training. As the dataset is huge it's just a matter of giving a user (model trainer) some feedback on what's going on. Otherwise, feedback is given for all steps in training preparation and not for filtering and the filtering in my use case lasts about 4-5 minutes. And yes, if there are more filtering stages, as @lhoestq pointed out, it would be nice to give some feedback. I thought desc is there already and got confused when I got the script error. 

I don't have a strong opinion on that, so having `desc` as a parameter is also OK.
Add RedCaps dataset

## Adding a Dataset
- **Name:** RedCaps
- **Description:** Web-curated image-text data created by the people, for the people
- **Paper:** https://arxiv.org/abs/2111.11431
- **Data:** https://redcaps.xyz/
- **Motivation:** Multimodal image-text dataset:  12M+ Image-text pairs 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Proposed by @patil-suraj 

Comments:
Adding arg to pass process rank to `map`

This PR adds a `with_rank` argument to `map` that gives the user the possibility to pass the rank of each process to their function. This is mostly designed for multi-GPU map (each process can be sent to a different device thanks to the rank). I've also added tests. I'm putting the PR up so you can check the code, I'll add a multi-GPU example to the doc (+ write a bit in the doc for the new arg)

Comments:
Some commits seem to be there twice (made the mistake of rebasing because I wasn't sure whether the doc had changed), is this an issue @lhoestq ?
TriviaQA License Mismatch

## Describe the bug

TriviaQA Webpage at http://nlp.cs.washington.edu/triviaqa/ says they do not own the copyright to the data. However, Huggingface datasets at https://huggingface.co/datasets/trivia_qa mentions that the dataset is released under Apache License

Is the License Information on HuggingFace correct?

Comments:
Hi ! You're completely right, this must be mentioned in the dataset card.
If you're interesting in contributing, feel free to open a pull request to mention this in the `trivia_qa` dataset card in the "Licensing Information" section at https://github.com/huggingface/datasets/blob/master/datasets/trivia_qa/README.md
add bl books genre dataset

First of all thanks for the fantastic library/collection of datasets 🤗

This pull request adds a dataset of metadata from digitised (mostly 19th Century) books from the British Library The [data](https://bl.iro.bl.uk/concern/datasets/1e1ccb46-65b4-4481-b6f8-b8129d5da053) contains various metadata about the books. In addition, a subset of the data includes 'genre' information which can be used for supervised text classification tasks. I hope that this offers easier access to a dataset for doing text classification on GLAM (galleries, libraries, archives and museums) data. 

I have tried to create three configurations that provide both an 'easy' version of the dataset if you want to use it for training a genre classification model and a more 'raw' version of the data for other potential use cases for the data. I am open to suggestions if this doesn't make sense. 

Similarly, for some of the arrow datatypes, I have had to fall back to strings since there are missing values for some fields/rows but I may have missed a more elegant way of dealing with it. 

Comments:
To fix the CI, feel free to run the `make style` command to format the code.

Then it also looks like the dummy_data.zip archives are all empty, which makes the tests fail. Can you try regenerating them ? They should have one file inside which is a dummy version of the file at https://bl.iro.bl.uk/downloads/36c7cd20-c8a7-4495-acbe-469b9132c6b1?locale=en

@lhoestq, thanks for that feedback. 

I should have made most of these changes now. The `--auto_generate` flag wasn't working because the file wasn't downloaded with a `.csv` extension. I used `--match_text_files "*"` to get around this. Because there is a lot of data that isn't annotated using the default line number for the dummy data causes the `annotated_raw` and the `title_genre_classifiction` configs to fail because they don't generate any examples — bumping the line numbers to `250` fixes this. This does make the dummy data a bit bigger, though. 

The total directory size for the dataset is now `150kb`. Is this okay, or do you want me to generate the dummy data manually instead? 

Hi ! yes 150kB is fine :)
Feel free to push your new dummy_data.zip files (I think the current one are still the empty ones)

@lhoestq I've pushed those dummy files now and added your other suggestions.

The CI failure is unrelated to this PR, merging :)

@lhoestq, thanks for all your help with this pull request 😀
Add WebSRC

## Adding a Dataset
- **Name:** WebSRC
- **Description:** WebSRC is a novel Web-based Structural Reading Comprehension dataset. It consists of 0.44M question-answer pairs, which are collected from 6.5K web pages with corresponding HTML source code, screenshots and metadata. 
- **Paper:** https://arxiv.org/abs/2101.09465
- **Data:** https://x-lance.github.io/WebSRC/dashboard.html#
- **Motivation:** Currently adding MarkupLM to HuggingFace Transformers, which achieves SOTA on this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
"dataset_infos.json" missing for chr_en and mc4

## Describe the bug
In the repository, every dataset has its metadata in a file called`dataset_infos.json`. But, this file is missing for two datasets: `chr_en` and `mc4`.

## Steps to reproduce the bug
Check [chr_en](https://github.com/huggingface/datasets/tree/master/datasets/chr_en) and [mc4](https://github.com/huggingface/datasets/tree/master/datasets/mc4)

Comments:
Hi ! Thanks for reporting :) 
We can easily add the metadata for `chr_en` IMO, but for mC4 it will take more time, since it requires to count the number of examples in each language

No problem. I am trying to do some analysis on the metadata of all available datasets. Is reading `metadata_infos.json` for each dataset the correct way to go? 

I noticed that the same information is also available as special variables inside .py file of each dataset. So, I was wondering if `metadata_infos.json` has been deprecated?

![image](https://user-images.githubusercontent.com/8587189/142914413-a95a1abf-6f3e-4fbe-96e5-16d3ca39c831.png)


The `dataset_infos.json` files have more information and are made to be used to analyze the datasets without having to run/parse the python scripts. Moreover some datasets on the Hugging face don't even have a python script, and for those ones we'll make tools to generate the JSON file automatically :)
nested sequence feature won't encode example if the first item of the outside sequence is an empty list

## Describe the bug
As the title, nested sequence feature won't encode example if the first item of the outside sequence is an empty list.

## Steps to reproduce the bug
```python
from datasets import Features, Sequence, ClassLabel
features = Features({
    'x': Sequence(Sequence(ClassLabel(names=['a', 'b']))),
})
print(features.encode_batch({
    'x': [
        [['a'], ['b']],
        [[], ['b']],
    ]
}))
```

## Expected results
print `{'x': [[[0], [1]], [[], ['1']]]}`

## Actual results
print `{'x': [[[0], [1]], [[], ['b']]]}`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux-5.13.0-21-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.0

## Additional information
I think the issue stems from [here](https://github.com/huggingface/datasets/blob/8555197a3fe826e98bd0206c2d031c4488c53c5c/src/datasets/features/features.py#L847-L848).



Comments:
knock knock

Hi, thanks for reporting! I've linked a PR that should fix the issue.

I've checked the PR and it looks great, thanks a lot!
Dataset object has no attribute `to_tf_dataset`

I am following HuggingFace Course. I am at Fine-tuning a model. 
Link: https://huggingface.co/course/chapter3/2?fw=tf

I use tokenize_function and `map` as mentioned in the course to process data.

`# define a tokenize function`
`def Tokenize_function(example):`
`    return tokenizer(example['sentence'], truncation=True)`

`# tokenize entire data`
`tokenized_data = raw_data.map(Tokenize_function, batched=True)`

I get Dataset object at this point. When I try converting this to a TF dataset object as mentioned in the course, it throws the following error.

`# convert to TF dataset`
`train_data = tokenized_data["train"].to_tf_dataset( `
`   columns = ['attention_mask', 'input_ids', 'token_type_ids'], `
`    label_cols = ['label'], `
`    shuffle = True, `
`    collate_fn = data_collator, `
`    batch_size = 8 `
`)`

Output:

`---------------------------------------------------------------------------`
`AttributeError                            Traceback (most recent call last)`
`/tmp/ipykernel_42/103099799.py in <module>`
`      1 # convert to TF dataset`
`----> 2 train_data = tokenized_data["train"].to_tf_dataset( \`
`      3     columns = ['attention_mask', 'input_ids', 'token_type_ids'], \`
`      4     label_cols = ['label'], \`
`      5     shuffle = True, \`
`AttributeError: 'Dataset' object has no attribute 'to_tf_dataset'`

When I look for `dir(tokenized_data["train"])`, there is no method or attribute in the name of `to_tf_dataset`.

Why do I get this error? And how to clear this?

Please help me.

Comments:
The issue is due to the older version of transformers and datasets. It has been resolved by upgrading their versions.

```
# upgrade transformers and datasets to latest versions
!pip install --upgrade transformers
!pip install --upgrade datasets
```

Regards!
DataCollatorWithPadding: TypeError

Hi,
I am following the HuggingFace course. I am now at Fine-tuning [https://huggingface.co/course/chapter3/3?fw=tf](https://huggingface.co/course/chapter3/3?fw=tf).  When I set up `DataCollatorWithPadding` as following I got an error while trying to reproduce the course code in Kaggle. This error occurs with either a CPU-only-device or a GPU-device.

Input:
```checkpoint = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```

Output:
```---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_42/1563280798.py in <module>
      1 checkpoint = 'bert-base-uncased'
      2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")
TypeError: __init__() got an unexpected keyword argument 'return_tensors'
```

When I call `help` method, it too confirms that there is no argument `return_tensors`.
Input:
```
help(DataCollatorWithPadding.__init__)
```
Output:
```
Help on function __init__ in module transformers.data.data_collator:

__init__(self, tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, padding: Union[bool, str, transformers.file_utils.PaddingStrategy] = True, max_length: Union[int, NoneType] = None, pad_to_multiple_of: Union[int, NoneType] = None) -> None
```

But, the source file *[Data Collator - docs](https://huggingface.co/transformers/main_classes/data_collator.html#datacollatorwithpadding)* says that there is such an argument. By default, it returns Pytorch tensors while I need TF tensors.

Where do I miss?
Please help me.

Comments:

> 
> Input:
> 
> ```
> tokenizer = AutoTokenizer.from_pretrained(checkpoint)
> data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
> ```
> 
> Output:
> 
> ```
> TypeError                                 Traceback (most recent call last)
> /tmp/ipykernel_42/1563280798.py in <module>
>       1 checkpoint = 'bert-base-uncased'
>       2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
> ----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")
> TypeError: __init__() got an unexpected keyword argument 'return_tensors'
> ```
> 

The issue is due to the older version of transformers and datasets. It has been resolved by upgrading their versions.

`# upgrade transformers and datasets to latest versions`
`!pip install --upgrade transformers`
`!pip install --upgrade datasets`

Cheers!
Add option to find unique elements in nested sequences when calling `Dataset.unique`

It would be nice to have an option to flatten nested sequences to find unique elements stored in them when calling `Dataset.unique`. ~~Currently, `Dataset.unique` only supports finding unique sequences and not unique elements in that situation.~~

Comments:
Hi @mariosasko!

Has this been patched into any of the releases?

Hi! Not yet, would you be interested in contributing a PR? I can give you some pointers if needed. 

@mariosasko did this ever get implemented? Willing to help if you are still up for it.

@dcruiz01 No, but here is an example of how to do this with the existing API:


```python
ds = Dataset.from_dict({"tokens": [["a", "b"], ["c", "a"], ["c", "e"]]})

def flatten_tokens(pa_table):
    return pa.table([pc.list_flatten(pa_table["tokens"])], ["flat_tokens"])

ds = ds.with_format("arrow")
ds = ds.map(flatten_tokens, batched=True)
ds = ds.with_format(None)

unique_tokens = ds.unique("flat_tokens")
```

When I think about it, `.unique` on `Sequence(Value(...))` should return unique sequences/arrays, not unique elements of these sequences...
Agnews dataset viewer is not working

## Dataset viewer issue for '*name of the dataset*'

**Link:** https://huggingface.co/datasets/ag_news

Hi there, the `ag_news` dataset viewer is not working.

Am I the one who added this dataset? No


Comments:
Hi ! Thanks for reporting
We've already fixed the code that generates the preview for this dataset, we'll release the fix soon :)

Hi @lhoestq, thanks for your feedback!

Fixed in the viewer.

https://huggingface.co/datasets/ag_news
Fix temporary dataset_path creation for URIs related to remote fs

This aims to close #3295

Comments:
Hi ! Thanks for the fix :) 

I think this should be `extract_path_from_uri` 's responsibility to strip the extra `/` from a badly formatted path like `hdfs:///absolute/path` (or raise an error). Do you think you can simply do the changes in `extract_path_from_uri` ? This way this fix will be available for all the other parts of the lib that need to extract the inner path from an URI of a remote filesystem

Then we can also keep your test cases but simply apply them to `extract_path_from_uri` instead

Hi @lhoestq! No problem! Thanks for your interest! :)

I think stripping the 3rd `/` in `hdfs:///absolute/path` inside `extract_path_from_uri` is not the solution. When I provide `hdfs:///absolute/path` to `extract_path_from_uri` we want `/absolute/path` to be returned, as it does now (at least in the case of URIs with `hdfs` schemas, for `s3` is different as it should start with a bucket name).

The problem comes in line 1041 in the original code below:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

Lets assume the following parameters for line 1041 after `extract_path_from_uri` has removed the `hdfs` schema part and the `://` from `hdfs:///absolute/path`, and `get_temporary_cache_files_directory()` returns `/tmp/a1b2b3c4`, as it is shown below: 

```python
src_dataset_path = '/absolute/path'
tmp_dir = '/tmp/a1b2b3c4'
dataset_path = Path(tmp_dir, src_dataset_path)
```

After passing those paths to the `Path` object, `dataset_path` contains only `/absolute/path`; that is, it has lost the temporary directory path. This is because, when two (or more) absolute paths are passed to the `Path` function, only the last one is taken. However, if the contents of those variables are:

```python
src_dataset_path = 'relative/path'
tmp_dir = '/tmp/a1b2b3c4'
dataset_path = Path(tmp_dir, src_dataset_path)
```

then `dataset_path` contains `/tmp/a1b2b3c4/relative/path` as expected.

Absolute paths are allowed in hdfs URIs, so that's why I added the extra function `build_local_temp_path` in the PR; so in case the second argument is an absolute path, it still will create the correct absolute path by concatenating the temp dir and the path passed by converting it to a relative path (and it also works for windows paths too.) It also allows to add the tests, checking that the main combinations are ok.

I've checked all the places where the result of `extract_path_from_uri` is used, and as far as I've seen this is the only place where it is concatenated with another possible absolute path, so no need to add `build_local_temp_path` anywhere else.
Temporary dataset_path for remote fs URIs not built properly in arrow_dataset.py::load_from_disk

## Describe the bug
When trying to build a temporary dataset path from a remote URI in this block of code:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

the result is not the expected when passing an absolute path in an URI like `hdfs:///absolute/path`.

## Steps to reproduce the bug
```python
dataset_path = "hdfs:///absolute/path"
src_dataset_path = extract_path_from_uri(dataset_path)
tmp_dir = get_temporary_cache_files_directory()
dataset_path = Path(tmp_dir, src_dataset_path)
print(dataset_path)
```

## Expected results
With the code above, we would expect a value in `dataset_path` similar to:
`/tmp/tmpnwxyvao5/absolute/path`

## Actual results
However, we get a `dataset_path` value like:
`/absolute/path`

This is because this line here: https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1041
returns the last absolute path when two absolute paths (the one in `tmp_dir` and the one extracted from the URI in `src_dataset_path`) are passed as arguments.

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0


Comments:
Hi ! Good catch and thanks for opening a PR :)

I just responded in your PR
Add Natural Adversarial Objects dataset

## Adding a Dataset
- **Name:** Natural Adversarial Objects (NAO)
- **Description:** Natural Adversarial Objects (NAO) is a new dataset to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence.
- **Paper:** https://arxiv.org/abs/2111.04204v1
- **Data:** https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8
- **Motivation:** interesting object detection dataset useful for miscclassifications

cc @NielsRogge 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
Make several audio datasets streamable


Needs https://github.com/huggingface/datasets/pull/3129 to be merged first


Make those audio datasets streamable:
- [x] common_voice
- [x] openslr
- [x] vivos
- [x] librispeech_asr 
(still has some issues to read FLAC)
 *actually it's ok*
- [ ] 
multilingual_librispeech (yet to be converted)</S> *TODO in a separate PR*

Comments:
Reading FLAC (for `librispeech_asr`) works OK for me (`soundfile` version: `0.10.3`):
```python
In [2]: ds = load_dataset("datasets/librispeech_asr/librispeech_asr.py", "clean", streaming=True, split="train.100")

In [3]: item = next(iter(ds))

In [4]: item.keys()
Out[4]: dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])

In [5]: item["file"]
Out[5]: '374-180298-0000.flac'

In [6]: item["audio"].keys()
Out[6]: dict_keys(['path', 'array', 'sampling_rate'])

In [7]: item["audio"]["sampling_rate"]
Out[7]: 16000

In [8]: item["audio"]["path"]
Out[8]: '374-180298-0000.flac'

In [9]: item["audio"]["array"].shape
Out[9]: (232480,)
```

Oh cool ! I think this might have come from an issue with my local `soundfile` installation then

I'll do `multilingual_librispeech` in a separate PR since it requires the data to be in another format (in particular separate the train/dev/test splits in different files)

@lhoestq @albertvillanova - think it would have been nice to have added a big message at the top stating that this is a breaking change and ping `transformers` people a bit more here.
Allow datasets with indices table when concatenating along axis=1

Calls `flatten_indices` on the datasets with indices table in `concatenate_datasets` to fix issues when concatenating along `axis=1`.


cc @lhoestq: I decided to flatten all the datasets instead of flattening all the datasets except the largest one in the end. The latter approach fails on the following example:
```python
a = Dataset.from_dict({"a": [10, 20, 30, 40]})
b = Dataset.from_dict({"b": [10, 20, 30, 40, 50, 60]})  # largest dataset
a = a.select([1, 2, 3])
b = b.select([1, 2, 3])
concatenate_datasets([a, b], axis=1) # fails at line concat_tables(...) because the real length of b's data is 6 and a's length is 3 after flattening (was 4 before flattening)
```

Also, it requires additional re-ordering of indices to prepare them for working with the indices table of the largest dataset. IMO not worth when we save only one `flatten_indices` call. (feel free to check the code of that approach at https://github.com/huggingface/datasets/commit/6acd10481c70950dcfdbfd2bab0bf0c74ad80bcb if you are interested)

Fixes #3273


Comments:
Add IEMOCAP dataset

## Adding a Dataset
- **Name:** IEMOCAP
- **Description:** acted, multimodal and multispeaker database
- **Paper:** https://sail.usc.edu/iemocap/Busso_2008_iemocap.pdf
- **Data:** https://sail.usc.edu/iemocap/index.html
- **Motivation:** Useful multimodal dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
The IEMOCAP dataset is private and available only on request.
```
To obtain the IEMOCAP data you just need to fill out an electronic release form below.
```

- [Request form](https://sail.usc.edu/iemocap/release_form.php)
- [License ](https://sail.usc.edu/iemocap/Data_Release_Form_IEMOCAP.pdf)


> We do not share the dataset for commercial purposes due to privacy concerns surrounding the participants of the research. The login details will only be emailed to the given academic email address.

I think it won't be possible to add this dataset to 🤗 datasets.

Hi @dnaveenr ! We can contact the authors to see if they are interested in hosting the dataset on the Hub. In the meantime, feel free to work on a script with manual download.

Hi @mariosasko . Thanks for your response. Sure, I will mail them and find out if they're open to this.

Work on a script with manual download ? This is new to me, any guidelines would be helpful here.


> Thanks for your response. Sure, I will mail them and find out if they're open to this.

It's best to leave this part to us because we have to explain how login would work and (potentially) set up a custom verification for the dataset.

> Work on a script with manual download ? This is new to me, any guidelines would be helpful here.

For instance, this is one of the scripts with manual download: https://huggingface.co/datasets/arxiv_dataset. Compared to the standard dataset, it has the `manual_download_instructions` attribute and uses `dl_manager.manual_dir` (derived from `load_dataset(..., data_dir="path/to/data")`) to access the dataset's data files.

> It's best to leave this part to us because we have to explain how login would work and (potentially) set up a custom verification for the dataset.

Yes. That would be perfect. Thanks.

----
Okay. Thanks for giving a reference. This is helpful. I will go through it.



@mariosasko  has this been solved? I would like to use login and custom verification for training on my private dataset.

@flckv I think the [gating mechanism](https://huggingface.co/docs/hub/datasets-gated) is what you are looking for. 

@mariosasko Thanks, but no. I would like to keep my HuggingFace Dataset private and train a model on it. Is this possible?
Add VoxLingua107 dataset

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models. The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** Nice audio classification dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
#self-assign
Add Speech Commands dataset

## Adding a Dataset
- **Name:** Speech commands
- **Description:** A Dataset for Limited-Vocabulary Speech Recognition
- **Paper:** https://arxiv.org/abs/1804.03209
- **Data:** https://www.tensorflow.org/datasets/catalog/speech_commands,  Available:
http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
- **Motivation:** Nice dataset for audio classification training

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
#self-assign
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py

## Dataset viewer issue for '*oscar-corpus/OSCAR-2109*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/oscar-corpus/OSCAR-2109)*

*The dataset library cannot download any language from the oscar-corpus/OSCAR-2109 dataset. By entering the URL in your browser I can access the file.*

```
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

Am I the one who added this dataset ? No

Using the older version of [OSCAR](https://huggingface.co/datasets/oscar) I don't have any issues downloading languages with the dataset library.

Comments:
Hi ! Thanks for reporting :)
I think this is because the dataset is behind an access page. We can fix the dataset viewer

If you also have this error when you use the `datasets` library in python, you should probably pass `use_auth_token=True` to the `load_dataset()` function to use your account to access the dataset.

Ah ok, I didn't realise about the login page. I'll try `use_auth_token=True` and see if that solves it.

Regards!

Hi, 

Using `use_auth_token=True` and downloading the credentials with `huggingface-cli login` (stored in .huggingface/token) solved the issue.

Should I leave the issue open until you fix the Dataset viewer issue?

Cool ! Yes let's keep this issue open until the viewer is fixed - I'll close it when this is fixed. Thanks

The error I get when trying to load OSCAR 21.09 is this
```
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

The URL I get in the browser is this
```
https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
```

Maybe URL is the issue? (resolve vs blob)

> The error I get when trying to load OSCAR 21.09 is this
> 
> ```
> ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
> ```
> 
> The URL I get in the browser is this
> 
> ```
> https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/blob/main/OSCAR-2109.py
> ```
> 
> Maybe URL is the issue? (resolve vs blob)

You need to download your login credentials. See `huggingface-cli login` documentation and when loading the dataset use `use_auth_token=True`:
`
load_dataset(corpus, language, split=None, use_auth_token=True, cache_dir=cache_folder)`

Fixed.

<img width="1542" alt="Capture d’écran 2022-04-12 à 13 57 24" src="https://user-images.githubusercontent.com/1676121/162957585-af96d19c-f86c-47fe-80c4-2b071083cee4.png">