KRadim/github-issues-minimal huggingface dataset:
add new field, and get errors
State: closed

after adding new field **tokenized_examples["example_id"]**, and get errors below,
I think it is due to changing data to tensor, and **tokenized_examples["example_id"]** is string list 
**all fields**
```
***************** train_dataset 1: Dataset({
    features: ['attention_mask', 'end_positions', 'example_id', 'input_ids', 'start_positions', 'token_type_ids'],
    num_rows: 87714
})
```

**Errors**
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 705, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: too many dimensions 'str'
```
to_tf_dataset() refactor
State: closed

This is the promised cleanup to `to_tf_dataset()` now that the course is out of the way! The main changes are:

- A collator is always required (there was way too much hackiness making things like labels work without it)
- Lots of cleanup and a lot of code moved to `_get_output_signature`
- Should now handle it gracefully when the data collator adds unexpected columns
iter_archive for zip files 
State: closed

* In this PR, I added the option to iterate through zipfiles for `download_manager.py` only. 
* Next PR will be the same applied to `streaming_download_manager.py`.
* Related issue #3272.
## Comments : 
* There is no `.isreg()` equivalent in zipfile library to check if file is Regular so I used `.is_dir()` instead to skip directories.
* For now I got `streaming_download_manager.py`  working for local zip files, but not for urls. I get the following error when I test it on an archive in google drive, so still working on it.  `BlockSizeError: Got more bytes so far (>2112) than requested (22)`


## Tasks : 
- [x] download_manager.py
- [ ] streaming_download_manager.py
Failed to convert `string` with pyarrow for QED since 1.15.0
State: closed
Labels: name: bug

## Describe the bug
Loading QED was fine until 1.15.0.
related: bigscience-workshop/promptsource#659, bigscience-workshop/promptsource#670

Not sure where the root cause is, but here are some candidates:
- #3158
- #3120
- #3196
- #2891

## Steps to reproduce the bug
```python
load_dataset("qed")
```

## Expected results
Loading completed.

## Actual results
```shell
ArrowInvalid: Could not convert in with type str: tried to convert to boolean
Traceback:
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/script_runner.py", line 354, in _run_script
    exec(code, module.__dict__)
File "/Users/s0s0cr3/Documents/GitHub/promptsource/promptsource/app.py", line 260, in <module>
    dataset = get_dataset(dataset_key, str(conf_option.name) if conf_option else None)
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/caching.py", line 543, in wrapped_func
    return get_or_create_cached_value()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/streamlit/caching.py", line 527, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
File "/Users/s0s0cr3/Documents/GitHub/promptsource/promptsource/utils.py", line 49, in get_dataset
    builder_instance.download_and_prepare()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 607, in download_and_prepare
    self._download_and_prepare(
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 697, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/builder.py", line 1106, in _prepare_split
    num_examples, num_bytes = writer.finalize()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 456, in finalize
    self.write_examples_on_file()
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 325, in write_examples_on_file
    pa_array = pa.array(typed_sequence)
File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
File "/Users/s0s0cr3/Library/Python/3.9/lib/python/site-packages/datasets/arrow_writer.py", line 121, in __arrow_array__
    out = pa.array(cast_to_python_objects(self.data, only_1d_for_numpy=True), type=type)
File "pyarrow/array.pxi", line 305, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.0, 1.16.1
- Platform: macOS 1.15.7 or above
- Python version: 3.7.12 and 3.9
- PyArrow version: 3.0.0, 5.0.0, 6.0.1
Better error message when download fails
State: closed

From our discussions in https://github.com/huggingface/datasets/issues/3269 and https://github.com/huggingface/datasets/issues/3282 it would be nice to have better messages if a download fails.

In particular the error now shows:
- the error from the HEAD request if there's one
- otherwise the response code of the HEAD request

I also added an error to tell users to pass `use_auth_token` when the Hugging Face Hub returns 401 (Unauthorized).

While paying around with this I also fixed a minor issue with the `force_download` parameter that was not always taken into account
Mirror the canonical datasets to the Hugging Face Hub
State: closed
Labels: name: enhancement

- [ ] create a repo on https://hf.co/datasets for every canonical dataset
- [ ] on every commit related to a dataset, update the hf.co repo

See https://github.com/huggingface/moon-landing/pull/1562

@SBrandeis: I let you edit this description if needed to precise the intent.
Fix JSON ClassLabel casting for integers
State: closed

Loading a JSON dataset with ClassLabel feature types currently fails if the JSON data already has integers. Indeed currently it tries to convert the strings to integers without even checking if the data are not integers already.

For example this currently fails:
```python
from datasets import load_dataset, Features, ClassLabel

path = "data.json"
f = Features({"a": ClassLabel(names=["neg", "pos"])})
d = load_dataset("json", data_files=path, features=f)
```
data.json
```json
{"a": 0}
{"a": 1}
```

I fixed that by adding a line that checks the type of the JSON data before trying to convert them

cc @albertvillanova let me know if it sounds good to you
to_tf_dataset fails on TPU
State: open
Labels: name: bug

Using `to_tf_dataset` to create a dataset and then putting it in `model.fit` results in an internal error on TPUs. I've only tried on Colab and Kaggle TPUs, not GCP TPUs.


## Steps to reproduce the bug
I made a colab to show the error. https://colab.research.google.com/drive/12x_PFKzGouFxqD4OuWfnycW_1TaT276z?usp=sharing

## Expected results
dataset from `to_tf_dataset` works in `model.fit`  
Right below the first error in the colab I use `tf.data.Dataset.from_tensor_slices` and `model.fit` works just fine. This is the desired outcome.

## Actual results
```
InternalError: 5 root error(s) found.
  (0) INTERNAL: {{function_node __inference_train_function_30558}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1638231897.932218653","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3151,"referenced_errors":[{"created":"@1638231897.932216754","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":161,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[tpu_compile_succeeded_assert/_14023832043698465348/_7/_439]]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0
- Tensorflow 2.7.0
- `transformers` 4.12.5
[WIP] Add doctests for tutorials
State: closed

Opening a PR as discussed with @LysandreJik for some help with doctest issues. The goal is to add doctests for each of the tutorials in the documentation to make sure the code samples work as shown.

### Issues

A doctest has been added in the docstring of the `load_dataset_builder` function in `load.py` to handle variable outputs with the `ELLIPSIS` directive. When I run doctest on the `load_hub.rst` file, doctest should recognize the expected output from the docstring, and the corresponding code sample in `load_hub.rst` should pass. I am having the same issue with handling tracebacks in the `load_dataset` function.

From the docstring:
```
>>> dataset_builder.cache_dir #doctest: +ELLIPSIS
/Users/.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
```
Test result:
```
Failed example:
    dataset_builder.cache_dir
Expected:
    /Users/.../.cache/huggingface/datasets/imdb/plain_text/1.0.0/...
Got:
    /Users/steven/.cache/huggingface/datasets/imdb/plain_text/1.0.0/2fdd8b9bcadd6e7055e742a706876ba43f19faee861df134affd7a3f60fc38a1
```

I am able to get the doctest to pass by adding the doctest directives (`ELLIPSIS` and `NORMALIZE_WHITESPACE`) to the code samples in the `rst` file directly. But my understanding is that these directives should also work in the docstrings of the functions. I am running the test from the root of the directory:

```
python -m doctest -v docs/source/load_hub.rst
```
Typing of Dataset.__getitem__ could be improved.
State: closed
Labels: name: bug

## Describe the bug

The newly added typing for Dataset.__getitem__ is Union[Dict, List]. This makes tools like mypy a bit awkward to use as we need to check the type manually. We could use type overloading to make this easier. [Documentation](https://docs.python.org/3/library/typing.html#typing.overload)

## Steps to reproduce the bug
Let's have a file `test.py`

```python
from typing import List, Dict, Any

from datasets import Dataset

ds = Dataset.from_dict({
    'a': [1,2,3],
    'b': ["1", "2", "3"]
})

one_colum: List[str] = ds['a']
some_index: Dict[Any, Any] = ds[1]
```

## Expected results

Running `mypy test.py` should not give any error.


## Actual results

```
test.py:10: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "List[str]")
test.py:11: error: Incompatible types in assignment (expression has type "Union[Dict[Any, Any], List[Any]]", variable has type "Dict[Any, Any]")
Found 2 errors in 1 file (checked 1 source file)
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.13.3
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyArrow version: 6.0.1
Add support for multiple dynamic dimensions and to_pandas conversion for dynamic arrays
State: closed

Add support for multiple dynamic dimensions (e.g. `(None, None, 3)` for arbitrary sized images) and `to_pandas()` conversion for dynamic arrays.

TODOs:
* [ ] Cleaner code
* [ ] Formatting issues (if NumPy doesn't allow broadcasting even though dtype is np.object)
* [ ] Fix some issues with zero-dim tensors 
* [ ] Tests
load JSON files, get the errors
State: closed

Hi, does this bug be fixed? when I load JSON files, I get the same errors by the command 
`!python3 run.py --do_train --task qa --dataset squad-retrain-data/train-v2.0.json --output_dir ./re_trained_model/`

change the dateset to load json by refering to https://huggingface.co/docs/datasets/loading.html
`dataset = datasets.load_dataset('json', data_files=args.dataset)`

Errors:
`Downloading and preparing dataset json/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /root/.cache/huggingface/datasets/json/default-c1e124ad488911b8/0.0.0/45636811569ec4a6630521c18235dfbbab83b7ab572e3393c5ba68ccabe98264...
`

_Originally posted by @yanllearnn in https://github.com/huggingface/datasets/issues/730#issuecomment-981095050_
Fix error message and add extension fallback
State: closed

Fix the error message raised if `infered_module_name` is `None` in `CommunityDatasetModuleFactoryWithoutScript.get_module` and make `infer_module_for_data_files` more robust. 

In the linked issue, `infer_module_for_data_files` returns `None` because `json` is the second most common extension due to the suffix ordering. Now,  we go from the most common to the least common extension and try to map it or return `None`.

Fix #3331
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
State: closed
Labels: name: bug

## Describe the bug
I add a new question answering dataset to huggingface datasets manually. Here is the link: [luozhouyang/question-answering-datasets](https://huggingface.co/datasets/luozhouyang/question-answering-datasets)

But when I load the dataset, an error raised: 

```bash
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
```

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("luozhouyang/question-answering-datasets", data_files=["dureader_robust.train.json"])
```

## Expected results
Load dataset successfully without any error.

## Actual results
```bash
Traceback (most recent call last):
  File "/mnt/home/zhouyang.lzy/github/naivenlp/naivenlp/tests/question_answering_tests/dataset_test.py", line 89, in test_load_dataset_with_hf
    data_files=["dureader_robust.train.json"],
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1616, in load_dataset
    **config_kwargs,
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1443, in load_dataset_builder
    path, revision=revision, download_config=download_config, download_mode=download_mode, data_files=data_files
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1157, in dataset_module_factory
    raise e1 from None
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 1144, in dataset_module_factory
    download_mode=download_mode,
  File "/mnt/home/zhouyang.lzy/.conda/envs/naivenlp/lib/python3.6/site-packages/datasets/load.py", line 798, in get_module
    raise FileNotFoundError(f"No data files or dataset script found in {self.path}")
AttributeError: 'CommunityDatasetModuleFactoryWithoutScript' object has no attribute 'path'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: linux
- Python version: 3.6.13
- PyArrow version: 6.0.1
Map function: Type error on iter #999
State: closed
Labels: name: bug

## Describe the bug
Using the map function, it throws a type error on iter #999

Here is the code I am calling:
```
dataset = datasets.load_dataset('squad')
dataset['validation'].map(text_numbers_to_int, input_columns=['context'], fn_kwargs={'column': 'context'})
``` 
text_numbers_to_int returns the input text with numbers replaced in the format {'context': text}

It happens at 
`
File "C:\Users\lonek\anaconda3\envs\ai\Lib\site-packages\datasets\arrow_writer.py", line 289, in <listcomp>
    [row[0][col] for row in self.current_examples], type=col_type, try_type=col_try_type, col=col
`

The issue is that the list comprehension expects self.current_examples to be type tuple(dict, str), but for some reason 26 out of 1000 of the sefl.current_examples are type tuple(str, str)

Here is an example of what self.current_examples should be
({'context': 'Super Bowl 50 was an...merals 50.'}, '')

Here is an example of what self.current_examples are when it throws the error:
('The Panthers used th... Marriott.', '')
Quick fix error formatting
State: closed

While working on a dataset, I got the error
```
TypeError: Provided `function` which is applied to all elements of table returns a `dict` of types {[type(x) for x in processed_inputs.values()]}. When using `batched=True`, make sure provided `function` returns a `dict` of types like `{allowed_batch_return_types}`.
```

This PR should fix the formatting of this error
"Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"
State: closed
Labels: name: bug

## Describe the bug
Passing a correctly shaped Numpy-Array to get_nearest_examples leads to the Exception

"Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"

Probably the reason for this is a wrongly converted assertion.

1.15.1:

`assert len(query.shape) == 1 or (len(query.shape) == 2 and query.shape[0] == 1)`

1.16.1:

```
 if len(query.shape) != 1 or (len(query.shape) == 2 and query.shape[0] != 1):
            raise ValueError("Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)")
```

## Steps to reproduce the bug

follow the steps described here: https://huggingface.co/course/chapter5/6?fw=tf

```python
 question_embedding.shape # (1, 768)

 scores, samples = embeddings_dataset.get_nearest_examples(
    "embeddings", question_embedding, k=5 # Error
)

# "Shape of query is incorrect, it has to be either a 1D array or 2D (1, N)"
```

## Expected results
Should work without exception

## Actual results
Throws exception

## Environment info
- `datasets` version: 1.15.1
- Platform: Darwin-20.6.0-x86_64-i386-64bit
- Python version: 3.7.12
- PyArrow version: 6.0.
Can't import `datasets` in python 3.10
State: closed

When importing `datasets` I'm getting this error in python 3.10:
```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_dataset.py", line 47, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_reader.py", line 33, in <module>
    from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 334, in <module>
    class InMemoryTable(TableBlock):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 361, in InMemoryTable
    def from_pandas(cls, *args, **kwargs):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 24, in wrapper
    out = wraps(arrow_table_method)(method)
  File "/Users/quentinlhoest/.pyenv/versions/3.10.0/lib/python3.10/functools.py", line 61, in update_wrapper
    wrapper.__wrapped__ = wrapped
AttributeError: readonly attribute
```

This makes the conda build fail.
I'm opening a PR to fix this and do a patch release 1.16.1
Can't get tatoeba.rus dataset
State: closed
Labels: name: bug

## Describe the bug
It gives an error.

> FileNotFoundError: Couldn't find file at https://github.com/facebookresearch/LASER/raw/master/data/tatoeba/v1/tatoeba.rus-eng.rus

## Steps to reproduce the bug
```python
data=load_dataset("xtreme","tatoeba.rus", split="validation")
```

## Solution
The library tries to access the **master** branch. In the github repo of facebookresearch, it is in the **main** branch.
Add desc parameter to Dataset filter method
State: closed
Labels: name: enhancement

**Is your feature request related to a problem? Please describe.**
As I was filtering very large datasets I noticed the filter method doesn't have the desc parameter which is available in the map method. Why don't we add a desc parameter to the filter method both for consistency and it's nice to give some feedback to users during long operations on Datasets?

**Describe the solution you'd like**
Add desc parameter to Dataset filter method

**Describe alternatives you've considered**
N/A

**Additional context**
N/A
Add RedCaps dataset
State: closed
Labels: name: dataset request, name: vision

## Adding a Dataset
- **Name:** RedCaps
- **Description:** Web-curated image-text data created by the people, for the people
- **Paper:** https://arxiv.org/abs/2111.11431
- **Data:** https://redcaps.xyz/
- **Motivation:** Multimodal image-text dataset:  12M+ Image-text pairs 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Proposed by @patil-suraj
Removing query params for dynamic URL caching
State: closed

The main use case for this is to make dynamically generated private URLs (like the ones returned by CommonVoice API) compatible with the datasets' caching logic.

Usage example:
```python
import datasets

class CommonVoice(datasets.GeneratorBasedBuilder):
    def _info(self):
        return datasets.DatasetInfo()

    def _split_generators(self, dl_manager):
        dl_manager.download_config.ignore_url_params = True
        HUGE_URL = "https://mozilla-common-voice-datasets.s3.dualstack.us-west-2.amazonaws.com/cv-corpus-7.0-2021-07-21/cv-corpus-7.0-2021-07-21-ab.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQ3GQRTO3IU5JYB5K%2F20211125%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20211125T131423Z&X-Amz-Expires=43200&X-Amz-Security-Token=FwoGZXIvYXdzEL7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaDLsZw7Nj0d9h4rgheyKSBJJ6bxo1JdWLXAUhLMrUB8AXfhP8Ge4F8dtjwXmvGJgkIvdMT7P4YOEE1pS3mW8AyKsz7Z7IRVCIGQrOH1AbxGVVcDoCMMswXEOqL3nJFihKLf99%2F6l8iJVZdzftRUNgMhX5Hz0xSIL%2BzRDpH5nYa7C6YpEdOdW81CFVXybx7WUrX13wc8X4ZlUj7zrWcWf5p2VEIU5Utb7YHVi0Y5TQQiZSDoedQl0j4VmMuFkDzoobIO%2BvilgGeE2kIX0E62X423mEGNu4uQV5JsOuLAtv3GVlemsqEH3ZYrXDuxLmnvGj5HfMtySwI4vKv%2BlnnirD29o7hxvtidXiA8JMWhp93aP%2Fw7sod%2BPPbb5EqP%2B4Qb2GJ1myClOKcLEY0cqoy7XWm8NeVljLJojnFJVS5mNFBAzCCTJ%2FidxNsj8fflzkRoAzYaaPBuOTL1dgtZCdslK3FAuEvw0cik7P9A7IYiULV33otSHKMPcVfNHFsWQljs03gDztsIUWxaXvu6ck5vCcGULsHbfe6xoMPm2bR9jtKLONsslPcnzWIf7%2Fch2w%2F%2BjtTCd9IxaH4kytyJ6mIjpV%2FA%2F2h9qeDnDFsCphnMjAzPQn6tqCgTtPcyJ2b8c94ncgUnE4mepx%2FDa%2FanAEsrg9RPdmbdoPswzHn1IClh91IfSN74u95DZUxlPeZrHG5HxVCN3dKO6j%2Ft1xd20L0hEtazDdKOr8%2FYwGMirp8rp%2BII0pYOwQOrYHqH%2FREX2dRJctJtwE86Qj1eU8BAdXuFIkLC4NWXw%3D&X-Amz-Signature=1b8108d29b0e9c2bf6c7246e58ca8d5749a83de0704757ad8e8a44d78194691f&X-Amz-SignedHeaders=host"
        dl_path = dl_manager.download_and_extract(HUGE_URL)
        print(dl_path)
        
        HUGE_URL += "&some_new_or_changed_param=12345"
        dl_path = dl_manager.download_and_extract(HUGE_URL)
        print(dl_path)

dl_manager = datasets.DownloadManager(dataset_name="common_voice")
CommonVoice()._split_generators(dl_manager)
```

Output:
```
/home/user/.cache/huggingface/datasets/downloads/6ef2a377398ff3309554be040caa78414e6562d623dbd0ce8fc262459a7f8ec6
/home/user/.cache/huggingface/datasets/downloads/6ef2a377398ff3309554be040caa78414e6562d623dbd0ce8fc262459a7f8ec6
```
Adding arg to pass process rank to `map`
State: closed

This PR adds a `with_rank` argument to `map` that gives the user the possibility to pass the rank of each process to their function. This is mostly designed for multi-GPU map (each process can be sent to a different device thanks to the rank). I've also added tests. I'm putting the PR up so you can check the code, I'll add a multi-GPU example to the doc (+ write a bit in the doc for the new arg)
TriviaQA License Mismatch
State: closed
Labels: name: bug

## Describe the bug

TriviaQA Webpage at http://nlp.cs.washington.edu/triviaqa/ says they do not own the copyright to the data. However, Huggingface datasets at https://huggingface.co/datasets/trivia_qa mentions that the dataset is released under Apache License

Is the License Information on HuggingFace correct?
add bl books genre dataset
State: closed

First of all thanks for the fantastic library/collection of datasets 🤗

This pull request adds a dataset of metadata from digitised (mostly 19th Century) books from the British Library The [data](https://bl.iro.bl.uk/concern/datasets/1e1ccb46-65b4-4481-b6f8-b8129d5da053) contains various metadata about the books. In addition, a subset of the data includes 'genre' information which can be used for supervised text classification tasks. I hope that this offers easier access to a dataset for doing text classification on GLAM (galleries, libraries, archives and museums) data. 

I have tried to create three configurations that provide both an 'easy' version of the dataset if you want to use it for training a genre classification model and a more 'raw' version of the data for other potential use cases for the data. I am open to suggestions if this doesn't make sense. 

Similarly, for some of the arrow datatypes, I have had to fall back to strings since there are missing values for some fields/rows but I may have missed a more elegant way of dealing with it.
Add WebSRC
State: open
Labels: name: dataset request

## Adding a Dataset
- **Name:** WebSRC
- **Description:** WebSRC is a novel Web-based Structural Reading Comprehension dataset. It consists of 0.44M question-answer pairs, which are collected from 6.5K web pages with corresponding HTML source code, screenshots and metadata. 
- **Paper:** https://arxiv.org/abs/2101.09465
- **Data:** https://x-lance.github.io/WebSRC/dashboard.html#
- **Motivation:** Currently adding MarkupLM to HuggingFace Transformers, which achieves SOTA on this dataset.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Fatal error condition occurred in aws-c-io
State: closed
Labels: name: bug

## Describe the bug
Fatal error when using the library

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('wikiann', 'en')
```

## Expected results
No fatal errors

## Actual results
```
Fatal error condition occurred in D:\bld\aws-c-io_1633633258269\work\source\event_loop.c:74: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
```

## Environment info
- `datasets` version: 1.15.2.dev0
- Platform: Windows-10-10.0.22504-SP0
- Python version: 3.8.12
- PyArrow version: 6.0.0
fix: files counted twice in inferred structure
State: closed

Files were counted twice in a structure like:
```
my_dataset_local_path/
├── README.md
└── data/
    ├── train/
    │   ├── shard_0.csv
    │   ├── shard_1.csv
    │   ├── shard_2.csv
    │   └── shard_3.csv
    └── valid/
        ├── shard_0.csv
        └── shard_1.csv
```

The reason is that they were matching both `*train*/*` and `*train*/**/*`.

This PR fixes it. @lhoestq
"dataset_infos.json" missing for chr_en and mc4
State: open
Labels: name: bug, name: dataset bug

## Describe the bug
In the repository, every dataset has its metadata in a file called`dataset_infos.json`. But, this file is missing for two datasets: `chr_en` and `mc4`.

## Steps to reproduce the bug
Check [chr_en](https://github.com/huggingface/datasets/tree/master/datasets/chr_en) and [mc4](https://github.com/huggingface/datasets/tree/master/datasets/mc4)
nested sequence feature won't encode example if the first item of the outside sequence is an empty list
State: closed
Labels: name: bug

## Describe the bug
As the title, nested sequence feature won't encode example if the first item of the outside sequence is an empty list.

## Steps to reproduce the bug
```python
from datasets import Features, Sequence, ClassLabel
features = Features({
    'x': Sequence(Sequence(ClassLabel(names=['a', 'b']))),
})
print(features.encode_batch({
    'x': [
        [['a'], ['b']],
        [[], ['b']],
    ]
}))
```

## Expected results
print `{'x': [[[0], [1]], [[], ['1']]]}`

## Actual results
print `{'x': [[[0], [1]], [[], ['b']]]}`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux-5.13.0-21-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyArrow version: 6.0.0

## Additional information
I think the issue stems from [here](https://github.com/huggingface/datasets/blob/8555197a3fe826e98bd0206c2d031c4488c53c5c/src/datasets/features/features.py#L847-L848).
Dataset object has no attribute `to_tf_dataset`
State: closed
Labels: name: bug

I am following HuggingFace Course. I am at Fine-tuning a model. 
Link: https://huggingface.co/course/chapter3/2?fw=tf

I use tokenize_function and `map` as mentioned in the course to process data.

`# define a tokenize function`
`def Tokenize_function(example):`
`    return tokenizer(example['sentence'], truncation=True)`

`# tokenize entire data`
`tokenized_data = raw_data.map(Tokenize_function, batched=True)`

I get Dataset object at this point. When I try converting this to a TF dataset object as mentioned in the course, it throws the following error.

`# convert to TF dataset`
`train_data = tokenized_data["train"].to_tf_dataset( `
`   columns = ['attention_mask', 'input_ids', 'token_type_ids'], `
`    label_cols = ['label'], `
`    shuffle = True, `
`    collate_fn = data_collator, `
`    batch_size = 8 `
`)`

Output:

`---------------------------------------------------------------------------`
`AttributeError                            Traceback (most recent call last)`
`/tmp/ipykernel_42/103099799.py in <module>`
`      1 # convert to TF dataset`
`----> 2 train_data = tokenized_data["train"].to_tf_dataset( \`
`      3     columns = ['attention_mask', 'input_ids', 'token_type_ids'], \`
`      4     label_cols = ['label'], \`
`      5     shuffle = True, \`
`AttributeError: 'Dataset' object has no attribute 'to_tf_dataset'`

When I look for `dir(tokenized_data["train"])`, there is no method or attribute in the name of `to_tf_dataset`.

Why do I get this error? And how to clear this?

Please help me.
DataCollatorWithPadding: TypeError
State: closed
Labels: name: bug

Hi,
I am following the HuggingFace course. I am now at Fine-tuning [https://huggingface.co/course/chapter3/3?fw=tf](https://huggingface.co/course/chapter3/3?fw=tf).  When I set up `DataCollatorWithPadding` as following I got an error while trying to reproduce the course code in Kaggle. This error occurs with either a CPU-only-device or a GPU-device.

Input:
```checkpoint = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="tf")
```

Output:
```---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_42/1563280798.py in <module>
      1 checkpoint = 'bert-base-uncased'
      2 tokenizer = AutoTokenizer.from_pretrained(checkpoint)
----> 3 data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors="pt")
TypeError: __init__() got an unexpected keyword argument 'return_tensors'
```

When I call `help` method, it too confirms that there is no argument `return_tensors`.
Input:
```
help(DataCollatorWithPadding.__init__)
```
Output:
```
Help on function __init__ in module transformers.data.data_collator:

__init__(self, tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, padding: Union[bool, str, transformers.file_utils.PaddingStrategy] = True, max_length: Union[int, NoneType] = None, pad_to_multiple_of: Union[int, NoneType] = None) -> None
```

But, the source file *[Data Collator - docs](https://huggingface.co/transformers/main_classes/data_collator.html#datacollatorwithpadding)* says that there is such an argument. By default, it returns Pytorch tensors while I need TF tensors.

Where do I miss?
Please help me.
❓ Dataset loading script from Hugging Face Hub
State: closed
Labels: name: dataset request, name: dataset-viewer

Hi there,

I am trying to add my custom `ag_news` with its own loading script on the Hugging Face datasets hub. In particular, I would like to test the addition of a second configuration to the existing `ag_news` dataset. Once it works in my hub, I plan to make a PR to the original dataset. However, in trying to do so I have encountered certain problems as detailed below.

Issues I have encountered:
- Without a loading script, the train and test files are loaded together into a unique `dataset.Dataset` -> so I wrote a loading script. Also, I need a loading script otherwise I cannot specify multiple configurations
- Once my loading script is working locally, I do not manage to make it work on the hub. In particular, I would like to be able to load the dataset like this
```python
load_dataset("pietrolesci/ag_news", name="my_configuration")
```

Apparently, the `load_dataset` is able to pick up the loading script from the hub and run it. However, it errors because it is unable to find the files. The structure of my hub repo is the following
```
ag_news.py
train.csv
test.csv
```
and the loading script I specify  `data_dir=Path(__file__).parent` and `data_files=DataFilesDict({"train": "train.csv", "test": "test.csv"})`. In the documentation I could not find info regarding loading a dataset from the hub using a loading script present on the hub.

Any suggestion is very much appreciated.

Best,
Pietro

Link to the hub repo: https://huggingface.co/datasets/pietrolesci/ag_news

BONUS: how can I make the data viewer work in this specific case? :)
.map() cache is wrongfully reused - only happens when the mapping function is imported
State: open
Labels: name: bug

## Describe the bug
When `.map` is used with a mapping function that is imported, the cache is reused even if the mapping function has been modified.
The reason for this is that `dill` that is used for creating the fingerprint [pickles imported functions by reference](https://stackoverflow.com/a/67851411).

I guess it is not a widespread case, but it can still lead to unwanted results unnoticeably. 

## Steps to reproduce the bug
Create files `a.py` and `b.py`:
```python
# a.py
from datasets import load_dataset

def main():
    squad = load_dataset("squad")
    squad.map(mapping_func, batched=True)

def mapping_func(examples):
    ID_LENGTH = 4
    examples["id"] = [id_[:ID_LENGTH] for id_ in examples["id"]]
    return examples

if __name__ == "__main__":
    main()
```
```python
# b.py
from datasets import load_dataset
from a import mapping_func

def main():
    squad = load_dataset("squad")
    squad.map(mapping_func, batched=True)

if __name__ == "__main__":
    main()
```
Run `python b.py` twice: In the first run you will see tqdm bars showing that the data is processed, and in the second run you will see "Loading cached processed dataset at...".
Now change `ID_LENGTH` to another number in order to change the mapping function, and run `python b.py` again. You'll see that `.map` loads from the cache the result of the previous mapping function.

## Expected results
Run `python a.py` twice: In the first run you will see tqdm bars showing that the data is processed, and in the second run you will see "Loading cached processed dataset at...".
Now change `ID_LENGTH` to another number in order to change the mapping function, and run `python a.py` again. You'll see that the dataset is being processed and that there's no reuse of the previous mapping function result.

## Workaround
Put the mapping function inside a dummy class as a static method:
```python
# a.py
class MappingFuncClass:
    @staticmethod
    def mapping_func(examples):
        ID_LENGTH = 4
        examples["id"] = [id_[:ID_LENGTH] for id_ in examples["id"]]
        return examples
```
```python
# b.py
from datasets import load_dataset
from a import MappingFuncClass

def main():
    squad = load_dataset("squad")
    squad.map(MappingFuncClass.mapping_func, batched=True)

if __name__ == "__main__":
    main()
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.15.1
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1
Temporary dataset_path for remote fs URIs not built properly in arrow_dataset.py::load_from_disk
State: closed
Labels: name: bug

## Describe the bug
When trying to build a temporary dataset path from a remote URI in this block of code:

https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1038-L1042

the result is not the expected when passing an absolute path in an URI like `hdfs:///absolute/path`.

## Steps to reproduce the bug
```python
dataset_path = "hdfs:///absolute/path"
src_dataset_path = extract_path_from_uri(dataset_path)
tmp_dir = get_temporary_cache_files_directory()
dataset_path = Path(tmp_dir, src_dataset_path)
print(dataset_path)
```

## Expected results
With the code above, we would expect a value in `dataset_path` similar to:
`/tmp/tmpnwxyvao5/absolute/path`

## Actual results
However, we get a `dataset_path` value like:
`/absolute/path`

This is because this line here: https://github.com/huggingface/datasets/blob/42f6b1d18a4a1b6009b6e62d115491be16dfca22/src/datasets/arrow_dataset.py#L1041
returns the last absolute path when two absolute paths (the one in `tmp_dir` and the one extracted from the URI in `src_dataset_path`) are passed as arguments.

## Environment info
- `datasets` version: 1.13.3
- Platform: Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.33
- Python version: 3.9.7
- PyArrow version: 5.0.0
Add Natural Adversarial Objects dataset
State: open
Labels: name: dataset request, name: vision

## Adding a Dataset
- **Name:** Natural Adversarial Objects (NAO)
- **Description:** Natural Adversarial Objects (NAO) is a new dataset to evaluate the robustness of object detection models. NAO contains 7,934 images and 9,943 objects that are unmodified and representative of real-world scenarios, but cause state-of-the-art detection models to misclassify with high confidence.
- **Paper:** https://arxiv.org/abs/2111.04204v1
- **Data:** https://drive.google.com/drive/folders/15P8sOWoJku6SSEiHLEts86ORfytGezi8
- **Motivation:** interesting object detection dataset useful for miscclassifications

cc @NielsRogge 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Make several audio datasets streamable
State: closed


Needs https://github.com/huggingface/datasets/pull/3129 to be merged first


Make those audio datasets streamable:
- [x] common_voice
- [x] openslr
- [x] vivos
- [x] librispeech_asr 
(still has some issues to read FLAC)
 *actually it's ok*
- [ ] 
multilingual_librispeech (yet to be converted)</S> *TODO in a separate PR*
Allow datasets with indices table when concatenating along axis=1
State: closed

Calls `flatten_indices` on the datasets with indices table in `concatenate_datasets` to fix issues when concatenating along `axis=1`.


cc @lhoestq: I decided to flatten all the datasets instead of flattening all the datasets except the largest one in the end. The latter approach fails on the following example:
```python
a = Dataset.from_dict({"a": [10, 20, 30, 40]})
b = Dataset.from_dict({"b": [10, 20, 30, 40, 50, 60]})  # largest dataset
a = a.select([1, 2, 3])
b = b.select([1, 2, 3])
concatenate_datasets([a, b], axis=1) # fails at line concat_tables(...) because the real length of b's data is 6 and a's length is 3 after flattening (was 4 before flattening)
```

Also, it requires additional re-ordering of indices to prepare them for working with the indices table of the largest dataset. IMO not worth when we save only one `flatten_indices` call. (feel free to check the code of that approach at https://github.com/huggingface/datasets/commit/6acd10481c70950dcfdbfd2bab0bf0c74ad80bcb if you are interested)

Fixes #3273
Add IEMOCAP dataset
State: open
Labels: name: dataset request, name: speech, name: vision

## Adding a Dataset
- **Name:** IEMOCAP
- **Description:** acted, multimodal and multispeaker database
- **Paper:** https://sail.usc.edu/iemocap/Busso_2008_iemocap.pdf
- **Data:** https://sail.usc.edu/iemocap/index.html
- **Motivation:** Useful multimodal dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add VoxLingua107 dataset
State: open
Labels: name: dataset request, name: speech

## Adding a Dataset
- **Name:** VoxLingua107
- **Description:** VoxLingua107 is a speech dataset for training spoken language identification models. The dataset consists of short speech segments automatically extracted from YouTube videos and labeled according the language of the video title and description, with some post-processing steps to filter out false positives.
- **Paper:** https://arxiv.org/abs/2011.12998
- **Data:** http://bark.phon.ioc.ee/voxlingua107/
- **Motivation:** Nice audio classification dataset

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
Add Speech Commands dataset
State: closed
Labels: name: dataset request, name: speech

## Adding a Dataset
- **Name:** Speech commands
- **Description:** A Dataset for Limited-Vocabulary Speech Recognition
- **Paper:** https://arxiv.org/abs/1804.03209
- **Data:** https://www.tensorflow.org/datasets/catalog/speech_commands,  Available:
http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz
- **Motivation:** Nice dataset for audio classification training

cc @anton-l 

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
State: closed
Labels: name: dataset-viewer

## Dataset viewer issue for '*oscar-corpus/OSCAR-2109*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/oscar-corpus/OSCAR-2109)*

*The dataset library cannot download any language from the oscar-corpus/OSCAR-2109 dataset. By entering the URL in your browser I can access the file.*

```
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://huggingface.co/datasets/oscar-corpus/OSCAR-2109/resolve/main/OSCAR-2109.py
```

Am I the one who added this dataset ? No

Using the older version of [OSCAR](https://huggingface.co/datasets/oscar) I don't have any issues downloading languages with the dataset library.
[Datasets] Improve Covost 2
State: closed

It's currently quite confusing to understand the manual data download instruction of Covost and not very user-friendly.

Currenty the user has to:

1. Go on Common Voice website
2. Find the correct dataset which is **not** mentioned in the error message
3. Download it
4. Untar it
5. Create a language id folder (why? this folder does not exist in the `.tar` downloaded file)
6. pass the folder containing the created language id folder

This PR improves this to:

1. Go on Common Voice website
2. Find the correct dataset which **is** mentioned in the error message
3. Download it
4. Untar it
5. pass the untared folder

**Note**: This PR is not at all time-critical
Proposed update to the documentation for WER
State: closed

I wanted to submit a minor update to the description of WER for your consideration. 

Because of the possibility of insertions, the numerator in the WER formula can be larger than N, so the value of WER can be greater than 1.0:

```
>>> from datasets import load_metric
>>> metric = load_metric("wer")
>>> metric.compute(predictions=["hello how are you"], references=["hello"])
3.0
```

and similarly from the underlying jiwer module's `wer` function:

```
>>> from jiwer import wer
>>> wer("hello", "hello how are you")
3.0
```
f-string formatting
State: closed

**Fix #3257**

Replaced _.format()_ and _%_ by f-strings in the following modules : 
- [x] **tests**
- [x] **metrics**
- [x] **benchmarks**
- [x] **utils**
- [x] **templates**
- [x] **src/Datasets/\*.py**

Modules in **_src/Datasets/_**: 
- [x] **commands**
- [x] **features**
- [x] **formatting**
- [x] **io**
- [x] **tasks**
- [x] **utils**


Module **datasets** will not be edited as asked by @mariosasko 

-A correction of the first PR (#3267)-
Force data files extraction if download_mode='force_redownload'
State: closed

Avoids weird issues when redownloading a dataset due to cached data not being fully updated.

With this change, issues #3122 and https://github.com/huggingface/datasets/issues/2956 (not a fix, but a workaround) can be fixed as follows:
```python
dset = load_dataset(..., download_mode="force_redownload")
```
Respect row ordering when concatenating datasets along axis=1
State: closed
Labels: name: bug

Currently, there is a bug when concatenating datasets along `axis=1` if more than one dataset has the `_indices` attribute defined. In that scenario, all indices mappings except the first one get ignored.

A minimal reproducible example:
```python
>>> from datasets import Dataset, concatenate_datasets
>>> a = Dataset.from_dict({"a": [30, 20, 10]})
>>> b = Dataset.from_dict({"b": [2, 1, 3]})
>>> d = concatenate_datasets([a.sort("a"), b.sort("b")], axis=1)
>>> print(d[:3])              # expected: {'a': [10, 20, 30], 'b': [1, 2, 3]}
{'a': [10, 20, 30], 'b': [3, 1, 2]}
```

I've noticed the bug while working on #3195.