Unable to load 'cnn_dailymail' dataset

bug

Something isn't working

duplicate

This issue or pull request already exists

Hi @talha1503, thanks for reporting.

It seems there is an issue with one of the data files hosted at Google Drive:
```
Google Drive - Quota exceeded

Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
```

As you probably know, Hugging Face does not host the data, and in this case the data owner decided to host their data at Google Drive, which has quota limits.

Is there anything we could do, @lhoestq @mariosasko?

This looks related to https://github.com/huggingface/datasets/issues/996

It seems that [this](https://huggingface.co/datasets/ccdv/cnn_dailymail) copy of the dataset has fixed the problem

## Describe the bug
I wanted to load cnn_dailymail dataset from huggingface datasets on Google Colab, but I am getting an error while loading it.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset('cnn_dailymail', '3.0.0', ignore_verifications = True)
```

## Expected results
Expecting to load 'cnn_dailymail' dataset.

## Actual results
`NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 3.0.0


struct.error: 'i' format requires -2147483648 <= number <= 2147483647

bug

Something isn't working

Hi ! Can you try setting `datasets.config.MAX_TABLE_NBYTES_FOR_PICKLING` to a smaller value than `4 << 30` (4GiB), for example `500 << 20` (500MiB) ? It should reduce the maximum size of the arrow table being pickled during multiprocessing.

If it fixes the issue, we can consider lowering the default value for everyone.

## Describe the bug
A clear and concise description of what the bug is.

using latest datasets=datasets-1.16.1-py3-none-any.whl
process my own multilingual dataset by following codes, and the number of rows in all dataset is 306000, the max_length of each sentence is 256:
![image](https://user-images.githubusercontent.com/30341159/146865779-3d25d011-1f42-4026-9e1b-76f6e1d172e9.png)

then I get this error:
![image](https://user-images.githubusercontent.com/30341159/146865844-e60a404c-5f3a-403c-b2f1-acd943b5cdb8.png)

I have seen the issue in #2134 and #2150, so I don't understand why latest repo still can't deal with big dataset.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux docker
- Python version: 3.6


Update swahili_news dataset

Update dataset with latest verion data files.

Fix #3462.
Close bigscience-workshop/data_tooling#107


Update swahili_news dataset

dataset request

Requesting to add a new dataset

Please note also: the HuggingFace version at https://huggingface.co/datasets/swahili_news is outdated. An updated version, with deduplicated text and official splits, can be found at https://zenodo.org/record/5514203.

## Adding a Dataset
- **Name:** swahili_news

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Related to:
- bigscience-workshop/data_tooling#107


Fix links in metrics description

Remove Markdown syntax for links in metrics description, as it is not properly rendered.

Related to #3437.


Don't encode lists as strings when using `Value("string")`

Following https://github.com/huggingface/datasets/pull/3456#event-5792250497 it looks like `datasets` can silently convert lists to strings using `str()`, instead of raising an error.

This PR fixes this and should fix the issue with WER showing low values if the input format is not right.


dataset.filter overwriting previously set dataset._indices values, resulting in the wrong elements being selected.

bug

Something isn't working

I think this is a duplicate of [#3190](https://github.com/huggingface/datasets/issues/3190)?

Upgrading the datasets version as per #3190 fixes this bug.
I'm Marking as closed.

## Describe the bug
When using dataset.select to select a subset of a dataset, dataset._indices are set to indicate which elements are now considered in the dataset.
The same thing happens when you shuffle the dataset; dataset._indices are set to indicate what the new order of the data is.
However, if you then use a dataset.filter, that filter interacts with those dataset._indices values in a non-intuitive manner.
https://huggingface.co/docs/datasets/_modules/datasets/arrow_dataset.html#Dataset.filter

Effectively, it looks like the original set of _indices were discared and overwritten by the set created during the filter operation.

I think this is actually an issue with how the map function handles dataset._indices. Ideally it should use the _indices it gets passed, and then return an updated _indices which reflect the map transformation applied to the starting _indices.

## Steps to reproduce the bug
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print("initial 10 elements")
print(dataset['label'])   # -> [1, 1, 0, 1, 0, 0, 0, 1, 0, 0]

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)
print("filtered 10 elements looking for label 0")
print(dataset['label'])  # -> [1, 1, 1, 1, 1, 1]
```

## Actual results
```
$ python indices_bug.py
initial 10 elements
[1, 1, 0, 1, 0, 0, 0, 1, 0, 0]
filtered 10 elements looking for label 0
[1, 1, 1, 1, 1, 1]
```

This code block first shuffles the dataset (to get a mix of label 0 and label 1).
Then it selects just the first 10 elements (the number of elements does not matter, 10 is just easy to visualize). The important part is that you select some subset of the dataset.
Finally, a filter is applied to pull out just the elements with `label == 0`.

The bug is that you cannot combine any dataset operation which sets the dataset._indices with filter.
In this case I have 2, shuffle and subset.

If you just use a single dataset._indices operation (in this case shuffle) the bug still shows up.

The shuffle sets the dataset._indices and then filter uses those indices in the map, then overwrites dataset._indices with the filter results.
```python
dataset = load_dataset('imdb', split='train', keep_in_memory=True)

dataset = dataset.shuffle(keep_in_memory=True)

dataset = dataset.filter(lambda x: x['label'] == 0, keep_in_memory=True)

dataset = dataset.select(range(0, 10), keep_in_memory=True)
print(dataset['label']) # -> [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
```

## Expected results
In an ideal world, the dataset filter would respect any dataset._indices values which had previously been set.

If you use dataset.filter with the base dataset (where dataset._indices has not been set) then the filter command works as expected.

## Environment info
Here are the commands required to rebuild the conda environment from scratch.
```
# create a virtual environment
conda create -n dataset_indices python=3.8 -y

# activate the virtual environment
conda activate dataset_indices

# install huggingface datasets
conda install datasets
```

<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Linux-5.11.0-41-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 3.0.0

### Full Conda Environment
```
$ conda env export
name: dasaset_indices
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - abseil-cpp=20210324.2=h2531618_0
  - aiohttp=3.8.1=py38h7f8727e_0
  - aiosignal=1.2.0=pyhd3eb1b0_0
  - arrow-cpp=3.0.0=py38h6b21186_4
  - attrs=21.2.0=pyhd3eb1b0_0
  - aws-c-common=0.4.57=he6710b0_1
  - aws-c-event-stream=0.1.6=h2531618_5
  - aws-checksums=0.1.9=he6710b0_0
  - aws-sdk-cpp=1.8.185=hce553d0_0
  - bcj-cffi=0.5.1=py38h295c915_0
  - blas=1.0=mkl
  - boost-cpp=1.73.0=h27cfd23_11
  - bottleneck=1.3.2=py38heb32a55_1
  - brotli=1.0.9=he6710b0_2
  - brotli-python=1.0.9=py38heb0550a_2
  - brotlicffi=1.0.9.2=py38h295c915_0
  - brotlipy=0.7.0=py38h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2021.10.26=h06a4308_2
  - certifi=2021.10.8=py38h06a4308_0
  - cffi=1.14.6=py38h400218f_0
  - conllu=4.4.1=pyhd3eb1b0_0
  - cryptography=36.0.0=py38h9ce1e76_0
  - dataclasses=0.8=pyh6d0b6a4_7
  - dill=0.3.4=pyhd3eb1b0_0
  - double-conversion=3.1.5=he6710b0_1
  - et_xmlfile=1.1.0=py38h06a4308_0
  - filelock=3.4.0=pyhd3eb1b0_0
  - frozenlist=1.2.0=py38h7f8727e_0
  - gflags=2.2.2=he6710b0_0
  - glog=0.5.0=h2531618_0
  - gmp=6.2.1=h2531618_2
  - grpc-cpp=1.39.0=hae934f6_5
  - huggingface_hub=0.0.17=pyhd3eb1b0_0
  - icu=58.2=he6710b0_3
  - idna=3.3=pyhd3eb1b0_0
  - importlib-metadata=4.8.2=py38h06a4308_0
  - importlib_metadata=4.8.2=hd3eb1b0_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - krb5=1.19.2=hac12032_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libboost=1.73.0=h3ff78a5_11
  - libcurl=7.80.0=h0b77cf5_0
  - libedit=3.1.20210910=h7f8727e_0
  - libev=4.33=h7f8727e_1
  - libevent=2.1.8=h1ba5d50_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libnghttp2=1.46.0=hce63b2e_0
  - libprotobuf=3.17.2=h4ff587b_1
  - libssh2=1.9.0=h1ba5d50_1
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - libthrift=0.14.2=hcc01f38_0
  - libxml2=2.9.12=h03d6c58_0
  - libxslt=1.1.34=hc22bd24_0
  - lxml=4.6.3=py38h9120a33_0
  - lz4-c=1.9.3=h295c915_1
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py38h7f8727e_0
  - mkl_fft=1.3.1=py38hd3c417c_0
  - mkl_random=1.2.2=py38h51133e4_0
  - multiprocess=0.70.12.2=py38h7f8727e_0
  - multivolumefile=0.2.3=pyhd3eb1b0_0
  - ncurses=6.3=h7f8727e_2
  - numexpr=2.7.3=py38h22e1b3c_1
  - numpy=1.21.2=py38h20f2e39_0
  - numpy-base=1.21.2=py38h79a1101_0
  - openpyxl=3.0.9=pyhd3eb1b0_0
  - openssl=1.1.1l=h7f8727e_0
  - orc=1.6.9=ha97a36c_3
  - packaging=21.3=pyhd3eb1b0_0
  - pip=21.2.4=py38h06a4308_0
  - py7zr=0.16.1=pyhd3eb1b0_1
  - pycparser=2.21=pyhd3eb1b0_0
  - pycryptodomex=3.10.1=py38h27cfd23_1
  - pyopenssl=21.0.0=pyhd3eb1b0_1
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pyppmd=0.16.1=py38h295c915_0
  - pysocks=1.7.1=py38h06a4308_0
  - python=3.8.12=h12debd9_0
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - python-xxhash=2.0.2=py38h7f8727e_0
  - pyzstd=0.14.4=py38h7f8727e_3
  - re2=2020.11.01=h2531618_1
  - readline=8.1=h27cfd23_0
  - requests=2.26.0=pyhd3eb1b0_0
  - setuptools=58.0.4=py38h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - snappy=1.1.8=he6710b0_0
  - sqlite=3.36.0=hc218d9a_0
  - texttable=1.6.4=pyhd3eb1b0_0
  - tk=8.6.11=h1ccaba5_0
  - typing_extensions=3.10.0.2=pyh06a4308_0
  - uriparser=0.9.3=he6710b0_1
  - utf8proc=2.6.1=h27cfd23_0
  - wheel=0.37.0=pyhd3eb1b0_1
  - xxhash=0.8.0=h7f8727e_3
  - xz=5.2.5=h7b6447c_0
  - zipp=3.6.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7f8727e_4
  - zstd=1.4.9=haebb681_0
  - pip:
    - async-timeout==4.0.2
    - charset-normalizer==2.0.9
    - datasets==1.16.1
    - fsspec==2021.11.1
    - huggingface-hub==0.2.1
    - multidict==5.2.0
    - pandas==1.3.5
    - pyarrow==6.0.1
    - pytz==2021.3
    - pyyaml==6.0
    - tqdm==4.62.3
    - typing-extensions==4.0.1
    - urllib3==1.26.7
    - yarl==1.7.2

```


Fix duplicated tag in wikicorpus dataset card

CI is failing just because of empty sections - merging


Add CMU Graphics Lab Motion Capture dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

This dataset has files in ASF/AMC format. [ The skeleton file is the ASF file (Acclaim Skeleton File). The motion file is the AMC file (Acclaim Motion Capture data). ]

Some questions :
1. How do we go about representing these features using datasets.Features and generate examples ?
2. The dataset download link for ASF/AMC files does not have metadata information, for eg : category and subcategory information. We will need to crawl the website for this information. The authors mention "Please don't crawl this database for all motions." Can we mail the authors for this information ?
The dataset structure is as follows :
```
subjects
	- 01
		- 01_01.amc
		- 01_02.amc
		.
		.
		.
		- 01.asf
	- 02
		- 02_01.amc
		- 02_02.amc
		.
		.
		.
		- 02.asf
```
There is no metadata regarding the category, sub-category and motion description.

Need your inputs. @mariosasko / @lhoestq
Thank you.

Hi @dnaveenr! Thanks for working on this!

1. We can use the `Sequence(Value("string"))` feature type for the subject's AMC files and `Value("string")` for the subject's ASF file (`Value("string")` represents the file paths) + the types for categories/subcategories and descriptions.
2. We can use this URL to download the motion descriptions: http://mocap.cs.cmu.edu/search.php?subjectnumber=<subject_number>&motion=%%%&maincat=%&subcat=%&subtext=yes where `subject_number` is the number between 1 and 144. And to get categories/subcategories, feel free to contact the authors (they state in the FAQ they are happy to help) and ask them if they can provide the mapping from categories/subcategories to the AMC files to avoid crawling. You can also mention that your goal is to make their dataset more accessible by adding its loading script to the Hub.

The AMC files are also available in the tvd, c3d, mpg and avi formats (the links are in the [FAQ](http://mocap.cs.cmu.edu/faqs.php) section), so it would be nice to have one config for each of these additional formats.

And additionally, we can add a `Data Preprocessing` section to the card where we explain how to load/process the files. I can help with that.

Hi @mariosasko ,

1. Thanks for this, so we can add the file paths.
2. Yes, I had already mailed the authors a couple of days back actually, asking for the metadata details[ i.e category, sub-category and motion description] . They are yet to respond though, I will wait for a couple of days and try to follow up with them again. :) Else we can use the workaround solution.

Yes. Supporting all the formats would be helpful.

> And additionally, we can add a Data Preprocessing section to the card where we explain how to load/process the files. I can help with that.

Okay. Got it.

## Adding a Dataset
- **Name:** CMU Graphics Lab Motion Capture database
- **Description:** The database contains free motions which you can download and use.
- **Data:** http://mocap.cs.cmu.edu/
- **Motivation:** Nice motion capture dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


[WER] Better error message for wer

Hi ! I don't think this would solve this issue.
Currently it looks like there's a bug that converts the list `["hello it's nice"]` to a string `'["hello it's nice"]'` since this is what the metric expects as input. The conversion is done before the data are passed to `_compute()`.

This is `Value("string").encode_example` that is called to do the conversion. Since `str()` encoding is too permissive we should consider raising an error if the example is not a string (even though it can be converted to string).

I just opened https://github.com/huggingface/datasets/pull/3460 to fix that. It now raises an error instead of computing the wrong WER

Currently we have the following problem when using the WER. When the input format to the WER metric is wrong, instead of throwing an error message a word-error-rate is computed which is incorrect. E.g. when doing the following:

```python
from datasets import load_metric

wer = load_metric("wer")

target_str = ["hello this is nice", "hello the weather is bloomy"]
pred_str = [["hello it's nice"], ["hello it's the weather"]]

print("Wrong:", wer.compute(predictions=pred_str, references=target_str))
print("Correct", wer.compute(predictions=[x[0] for x in pred_str], references=target_str))
```
We get:
```
Wrong: 1.0
Correct 0.5555555555555556
```

meaning that we get a word-error rate for incorrectly passed input formats. We should raise an error here instead so that people don't spent hours fixing a model while it's their incorrect evaluation metric is the problem for a low WER.


Easier information editing

enhancement

New feature or request

generic discussion

Generic discussion on the library

Hi ! I guess you are talking about the dataset cards that are in this repository on github ?

I think github allows to submit a PR even for 1 line though the `Edit file` button on the page of the dataset card.

Maybe let's mention this in `CONTRIBUTING.md` ?

**Is your feature request related to a problem? Please describe.**
It requires a lot of effort to improve a datasheet.

**Describe the solution you'd like**
UI or at least a link to the place where the code that needs to be edited is (and an easy way to edit this code directly from the site, without cloning, branching, makefile etc.)

**Describe alternatives you've considered**
The current Ux is to have the 8 steps for contribution while One just wishes to change a line a type etc.


Fix iter_archive generator

This PR:
- Adds tests to DownloadManager and StreamingDownloadManager `iter_archive` for both path and file inputs
- Fixes bugs in `iter_archive` introduced in:
  - #3443

Fix #3453.


ValueError while iter_archive

bug

Something isn't working

## Describe the bug
After the merge of:
- #3443

the method `iter_archive` throws a ValueError:
```
ValueError: read of closed file
```

## Steps to reproduce the bug
```python
for path, file in dl_manager.iter_archive(archive_path):
    pass
```


enhancement

New feature or request

good second issue

Issues a bit more difficult than "Good First" issues

Hi ! It's simply not added yet :)

If someone wants to contribute to add the `stratify` parameter I'd be happy to give some pointers.

In the meantime, I guess you can use `sklearn` or other tools to do a stratified train/test split over the **indices** of your dataset and then do
```
train_dataset = dataset.select(train_indices)
test_dataset = dataset.select(test_indices)
```

Hi @lhoestq I would like to add `stratify` parameter, can you give me some pointers for adding the same ?

Hi ! Sure :)

The `train_test_split` method is defined here:

https://github.com/huggingface/datasets/blob/dc62232fa1b3bcfe2fbddcb721f2d141f8908943/src/datasets/arrow_dataset.py#L3253-L3253

and inside `train_test_split ` we need to create the right `train_indices` and `test_indices` that are passed here to `.select()`:

https://github.com/huggingface/datasets/blob/dc62232fa1b3bcfe2fbddcb721f2d141f8908943/src/datasets/arrow_dataset.py#L3450-L3464

For example if your dataset is like
|    |   label |
|---:|--------:|
|  0 |       1 |
|  1 |       1 |
|  2 |       0 |
|  3 |       0 |

and the user passes `stratify=dataset["label"]`, then you should get indices that look like this
```
train_indices = [0, 2]
test_indices = [1, 3]
```

these indices will be passed to `.select` to return the stratified train and test splits :)

Feel free to îng me if you have any question !

@lhoestq
I just added the implementation for `stratify` option here #4322

why the stratify option is omitted from test_train_split function?

is there any other way implement the stratify option while splitting the dataset? as it is important point to be considered while splitting the dataset.


[Staging] Update dataset repos automatically on the Hub

do keep us updated on how it's going in staging! cc @SBrandeis

Sure ! For now it works smoothly. We'll also do a new release today.

I can send you some repos to explore on staging, in case you want to see how they look like after being updated.
For example [swahili_news](https://moon-staging.huggingface.co/datasets/swahili_news/tree/main)

Let's have a script that updates the dataset repositories on staging for now. This way we can make sure it works fine before going in prod.

Related to https://github.com/huggingface/datasets/issues/3341

The script runs on each commit on `master`. It checks the datasets that were changed, and it pushes the changes to the corresponding repositories on the Hub.

If there's a new dataset, then a new repository is created.

If the commit is a new release of `datasets`, it also pushes the tag to all the repositories.


Unexpected behavior doing Split + Filter

bug

Something isn't working

Hi ! This is an issue with `datasets` 1.12. Sorry for the inconvenience. Can you update to `>=1.13` ?
see https://github.com/huggingface/datasets/issues/3190

Maybe we should also backport the bug fix to `1.12` (in a new version `1.12.2`)

## Describe the bug
I observed unexpected behavior when applying 'train_test_split' followed by 'filter' on dataset. Elements of the training dataset eventually end up in the test dataset (after applying the 'filter')

## Steps to reproduce the bug
```
from datasets import Dataset
import pandas as pd
dic = {'x': [1,2,3,4,5,6,7,8,9], 'y':['q','w','e','r','t','y','u','i','o']}
df = pd.DataFrame.from_dict(dic)
dataset = Dataset.from_pandas(df)
split_dataset = dataset.train_test_split(test_size=0.5, shuffle=False, seed=42)
train_dataset = split_dataset["train"]
eval_dataset = split_dataset["test"]
eval_dataset_2 = eval_dataset.filter(lambda example: example['x'] % 2 == 0)
print( eval_dataset['x'])
print(eval_dataset_2['x'])
```
One observes that elements in eval_dataset2 are actually coming from the training dataset...

## Expected results
The expected results would be that the filtered eval dataset would only contain elements from the original eval dataset.

## Actual results
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.12.1
- Platform: Windows 10
- Python version: 3.7
- PyArrow version: 5.0.0


Add `__add__()`, `__iadd__()` and similar to `Dataset` class

enhancement

New feature or request

generic discussion

Generic discussion on the library

I was going through the codebase, and I believe the implementation of __add__() and __iadd__() will be similar to concatenate_datasets() after the elimination of code for arguments other than the list of datasets (info, split, axis).
(Assuming elimination of axis means concatenating over axis 1.)

**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
I would like to be able to concatenate datasets as follows:
```python
>>> dataset["train"] += dataset["validation"]
```

... instead of using `concatenate_datasets()`:
```python
>>> raw_datasets["train"] = concatenate_datasets([raw_datasets["train"], raw_datasets["validation"]])
>>> del raw_datasets["validation"]
```

**Describe alternatives you've considered**
Well, I have considered `concatenate_datasets()` 😀

**Additional context**
N.a.


JSONDecodeError with HuggingFace dataset viewer

dataset-viewer

Related to the dataset viewer on huggingface.co

Hi ! I think the issue comes from the dataset_infos.json file: it has the "flat" field twice.

Can you try deleting this file and regenerating it please ?

Thanks! That fixed that, but now I am getting:
Server Error
Status code:   400
Exception:     KeyError
Message:       'feature'

I checked the dataset_infos.json and pubmed_neg.py script, I don't use 'feature' anywhere as a key. Is the dataset viewer expecting that I do?

It seems that the `feature` key is missing from some feature type definition in your dataset_infos.json:
```json
			"tokens": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
			},
			"tags": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
			}
```
They should be
```json
			"tokens": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence"
                                 "feature": {"dtype": "string", "id": null, "_type": "Value"}
			},
			"tags": {
				"dtype": "list",
				"id": null,
				"_type": "Sequence",
                                 "feature": {"num_classes": 5, "names": ["-", "S", "H", "N", "C"], "names_file": null, "id": null, "_type": "ClassLabel"}
			}
```

Note that you can generate the dataset_infos.json automatically to avoid mistakes:
```bash
datasets-cli test ./path/to/dataset --save_infos
```

## Dataset viewer issue for 'pubmed_neg'

**Link:** https://huggingface.co/datasets/IGESML/pubmed_neg

I am getting the error:
Status code:   400
Exception:     JSONDecodeError
Message:       Expecting property name enclosed in double quotes: line 61 column 2 (char 1202)

I have checked all files - I am not using single quotes anywhere. Not sure what is causing this issue.

Am I the one who added this dataset ? Yes


HF_DATASETS_OFFLINE=1 didn't stop datasets.builder from downloading

bug

Something isn't working

Hi ! Indeed it says "downloading and preparing" but in your case it didn't need to download anything since you used local files (it would have thrown an error otherwise). I think we can improve the logging to make it clearer in this case

@lhoestq Thank you for explaining. I am sorry but I was not clear about my intention. I didn't want to kill internet traffic; I wanted to kill all write activity. In other words, you can imagine that my storage has only read access but crashes on write.

When run_clm.py is invoked with the same parameters, the hash in the cache directory "datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/..." doesn't change, and my job can load cached data properly. This is great.

Unfortunately, when params change (which happens sometimes), the hash changes and the old cache is invalid. datasets builder would create a new cache directory with the new hash and create JSON builder there, even though every JSON builder is the same. I didn't find a way to avoid such behavior.

This problem can be resolved when using datasets.map() for tokenizing and grouping text. This function allows me to specify output filenames with --cache_file_names, so that the cached files are always valid.

This is the code that I used to freeze cache filenames for tokenization. I wish I could do the same to datasets.load_dataset()
```
        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on dataset",
            cache_file_names={k: os.path.join(model_args.cache_dir, f'{k}-tokenized') for k in raw_datasets},
        )
```

Hi ! `load_dataset` may re-generate your dataset if some parameters changed indeed. If you want to freeze a dataset loaded with `load_dataset`, I think the best solution is just to save it somewhere on your disk with `.save_to_disk(my_dataset_dir)` and reload it with `load_from_disk(my_dataset_dir)`. This way you will be able to reload the dataset without having to run `load_dataset`

## Describe the bug
According to https://huggingface.co/docs/datasets/loading_datasets.html#loading-a-dataset-builder, setting HF_DATASETS_OFFLINE to 1 should make datasets to "run in full offline mode". It didn't work for me. At the very beginning, datasets still tried to download "custom data configuration" for JSON, despite I have run the program once and cached all data into the same --cache_dir.

"Downloading" is not an issue when running with local disk, but crashes often with cloud storage because (1) multiply GPU processes try to access the same file, AND (2) FileLocker fails to synchronize all processes, due to storage throttling. 99% of times, when the main process releases FileLocker, the file is not actually ready for access in cloud storage and thus triggers "FileNotFound" errors for all other processes. Well, another way to resolve the problem is to investigate super reliable cloud storage, but that's out of scope here.

## Steps to reproduce the bug
```
export HF_DATASETS_OFFLINE=1
python run_clm.py --model_name_or_path=models/gpt-j-6B --train_file=trainpy.v2.train.json --validation_file=trainpy.v2.eval.json --cache_dir=datacache/trainpy.v2
```

## Expected results
datasets should stop all "downloading" behavior but reuse the cached JSON configuration. I think the problem here is part of the cache directory path, "default-471372bed4b51b53", is randomly generated, and it could change if some parameters changed. And I didn't find a way to use a fixed path to ensure datasets to reuse cached data every time.

## Actual results
The logging shows datasets are still downloading into "datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426".
```
12/16/2021 10:25:59 - WARNING - datasets.builder - Using custom data configuration default-471372bed4b51b53
12/16/2021 10:25:59 - INFO - datasets.builder - Generating dataset json (datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426)
Downloading and preparing dataset json/default to  datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426...
100%|██████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 17623.13it/s]
12/16/2021 10:25:59 - INFO - datasets.utils.download_manager - Downloading took 0.0 min
12/16/2021 10:26:00 - INFO - datasets.utils.download_manager - Checksum Computation took 0.0 min
100%|███████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 1206.99it/s]
12/16/2021 10:26:00 - INFO - datasets.utils.info_utils - Unable to verify checksums.
12/16/2021 10:26:00 - INFO - datasets.builder - Generating split train
12/16/2021 10:26:01 - INFO - datasets.builder - Generating split validation
12/16/2021 10:26:02 - INFO - datasets.utils.info_utils - Unable to verify splits sizes.
Dataset json downloaded and prepared to  datacache/trainpy.v2/json/default-471372bed4b51b53/0.0.0/c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 53.54it/s]
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.1
- Platform: Linux
- Python version:  3.8.10
- PyArrow version: 6.0.1


Remove redundant local path information in audio/image datasets

Cool, I'm in favor of this PR. Our official examples in speech already make use of `"audio"` so no need to change anything there. It would be great if we could prominently feature how one can get the audio path without decoding in the docs.

@patrickvonplaten Yes, I agree.

cc @stevhliu we should add an example where decoding is disabled (to read paths) to  [this section](https://github.com/huggingface/datasets/blob/master/docs/source/audio_process.rst#audio-datasets) in the docs and remove the mentions of `path`/`file` (if we merge this PR).

Remove the redundant path information in the audio/image dataset as discussed in https://github.com/huggingface/datasets/pull/3430#issuecomment-994734828

TODOs:
* [ ] merge https://github.com/huggingface/datasets/pull/3430
* [ ] merge https://github.com/huggingface/datasets/pull/3364
* [ ] re-generate the info files of the updated audio datasets

cc:  @patrickvonplaten @anton-l @nateraw (I expect this to break the audio/vision examples in Transformers; after this change you'll be able to access underlying paths as follows `dset = dset.cast_column("audio", Audio(..., decode=False)); path = dset[0]["audio"]`)


question

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Align the Dataset and IterableDataset processing API

enhancement

New feature or request

generic discussion

Generic discussion on the library

Yes I agree, these should be as aligned as possible. Maybe we can also check the feedback in the survey at http://hf.co/oss-survey and see if people mentioned related things on the API (in particular if we go the breaking change way, it would be good to be sure we are taking the right direction for the community).

I like this proposal.

> There is also an important difference in terms of behavior:
Dataset.map adds new columns (with dict.update)
BUT
IterableDataset discards previous columns (it overwrites the dict)
IMO the two methods should have the same behavior. This would be an important breaking change though.

> The main breaking change would be the change of behavior of IterableDataset.map, because currently it discards all the previous columns instead of keeping them.

Yes, this behavior of `IterableDataset.map` was surprising to me the first time I used it because I was expecting the same behavior as `Dataset.map`, so I'm OK with the breaking change here.

> IterableDataset only supports "torch" (it misses tf, jax, pandas, arrow) and is missing the parameters: columns, output_all_columns and format_kwargs

\+  it's also missing the actual formatting code (we return unformatted tensors)
> We could have a completely aligned map method if both methods were lazy by default, but this is a very big breaking change so I'm not sure we can consider doing that.

> For information, TFDS does lazy map by default, and has an additional .cache() method.

If I understand this part correctly, the idea would be for `Dataset.map` to behave similarly to `Dataset.with_transform` (lazy processing) and to have an option to cache processed data (with `.cache()`). This idea is really nice because it can also be applied to `IterableDataset` to fix https://github.com/huggingface/datasets/issues/3142 (again we get the aligned APIs). However, this change would break a lot of things, so I'm still not sure if this is a step in the right direction (maybe it's OK for Datasets 2.0?)
> If the two APIs are more aligned it would be awesome for the examples in transformers, and it would create a satisfactory experience for users that want to switch from one mode to the other.

Yes, it would be amazing to have an option to easily switch between these two modes.

I agree with the rest.

> If I understand this part correctly, the idea would be for Dataset.map to behave similarly to Dataset.with_transform (lazy processing) and to have an option to cache processed data (with .cache()). This idea is really nice because it can also be applied to IterableDataset to fix #3142 (again we get the aligned APIs). However, this change would break a lot of things, so I'm still not sure if this is a step in the right direction (maybe it's OK for Datasets 2.0?)

Yea this is too big of a change in my opinion. Anyway it's fine as it is right now with streaming=lazy and regular=eager.

Hi, IterableDataset is also missing set_format.

Yes indeed, thanks. I added it to the list of methods to align in the first post

I just encountered the problem of the missing `fn_kwargs` parameter in the `map` method. I am commenting to give a workaround in case someone has the same problem and does not find a solution.
You can wrap your function call inside a class that contains the other parameters needed by the function called by map, like this:

```python
def my_func(x, y, z):
    # Do things

class MyFuncWrapper:
    def __init__(self, y, z):
        self.y = y
        self.z = z

    def __call__(self, x):
        return my_func(x, self.y, self.z)
```

Then, give an instance of the `MyFuncWrapper` to the map function.

## Intro

items marked like <s>this</s> are done already :)

Currently the two classes have two distinct API for processing:

### The `.map()` method

Both have those parameters in common: function, batched, batch_size

- IterableDataset is missing those parameters:
<s>with_indices</s>, with_rank, <s>input_columns</s>, <s>drop_last_batch</s>, <s>remove_columns</s>, features, disable_nullable, fn_kwargs, num_proc

- Dataset also has additional parameters that are exclusive, due to caching:
keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, suffix_template, new_fingerprint

- <s>There is also an important difference in terms of behavior:
**Dataset.map adds new columns** (with dict.update)
BUT
**IterableDataset discards previous columns** (it overwrites the dict)
IMO the two methods should have the same behavior. This would be an important breaking change though.</s>

- Dataset.map is eager while IterableDataset.map is lazy

### The `.shuffle()` method

- <s>Both have an optional seed parameter, but IterableDataset requires a mandatory parameter buffer_size to control the size of the local buffer used for approximate shuffling.</s>

- <s>IterableDataset is missing the parameter generator</s>

- Also Dataset has exclusive parameters due to caching: keep_in_memory, load_from_cache_file, indices_cache_file_name, writer_batch_size, new_fingerprint

### The `.with_format()` method

- IterableDataset only supports "torch" (it misses tf, jax, pandas, arrow) and is missing the parameters: columns, output_all_columns and format_kwargs
- other methods like `set_format`, `reset_format` or `formatted_as` are also missing

### Other methods

- Both have the same `remove_columns` method
- IterableDataset is missing: <s>cast</s>, <s>cast_column</s>, <s>filter</s>, <s>rename_column</s>, <s>rename_columns</s>, class_encode_column, flatten, prepare_for_task, train_test_split, shard
- Some other methods are missing but we can discuss them: set_transform, formatted_as, with_transform
- And others don't really make sense for an iterable dataset: select, sort, add_column, add_item
- Dataset is missing skip and take, that IterableDataset implements.

## Questions

I think it would be nice to be able to switch between streaming and regular dataset easily, without changing the processing code significantly.

1. What should be aligned and what shouldn't between those two APIs ?

IMO the minimum is to align the main processing methods.

It would mean aligning breaking the current `Iterable.map` to have the same behavior as `Dataset.map` (add columns with dict.update), and add multiprocessing as well as the missing parameters. DONE ✅

It would also mean implementing the missing methods: cast, cast_column, filter, rename_column, rename_columns, class_encode_column, flatten, prepare_for_task, train_test_split, shard. WIP 🟠

2. What are the breaking changes for IterableDataset ?

The main breaking change would be the change of behavior of `IterableDataset.map`, because currently it discards all the previous columns instead of keeping them. DONE ✅

3. Shall we also do some changes for regular datasets ?

I agree the simplest would be to have the exact same methods for both Dataset and IterableDataset. However this is probably not a good idea because it would prevent users from using the best benefits of them. That's why we can keep some aspects of regular datasets as they are:
- keep the eager Dataset.map with caching
- keep the with_transform method for lazy processing
- keep Dataset.select (it could also be added to IterableDataset even though it's not recommended)

We could have a completely aligned `map` method if both methods were lazy by default, but this is a very big breaking change so I'm not sure we can consider doing that.

For information, TFDS does lazy map by default, and has an additional `.cache()` method.

## Opinions ?

I'd love to gather some opinions about this here. If the two APIs are more aligned it would be awesome for the examples in `transformers`, and it would create a satisfactory experience for users that want to switch from one mode to the other.

cc @mariosasko @albertvillanova @thomwolf @patrickvonplaten @sgugger


Extend iter_archive to support file object input

This PR adds support to passing a file object to `[Streaming]DownloadManager.iter_archive`.

With this feature, we can iterate over a tar file inside another tar file.


Extend text to support yielding lines, paragraphs or documents

> The parameter can also be named `split_by` with values "line", "paragraph" or "document" (no 's' at the end)

@lhoestq @mariosasko I would avoid the term `split` in this context and keep it only for "train", "validation" and "test" splits.
- https://huggingface.co/docs/datasets/process.html#split
  > datasets.Dataset.train_test_split() creates train and test splits, if your dataset doesn’t already have them.
- https://huggingface.co/docs/datasets/process.html#process-multiple-splits
  > Many datasets have splits that you can process simultaneously with datasets.DatasetDict.map().

Please note that in the documentation, one of the terms more frequently used in this context is **"row"**:
- https://huggingface.co/docs/datasets/access.html#features-and-columns
  > A dataset is a table of rows and typed columns.

  > Return the number of rows and columns with the following standard attributes:
  > dataset.num_columns
  > 4
  > dataset.num_rows
  > 3668

- https://huggingface.co/docs/datasets/access.html#rows-slices-batches-and-columns
  > Get several rows of your dataset at a time with slice notation or a list of indices:
- https://huggingface.co/docs/datasets/process.html#map
  > This function can even create new rows and columns.

Other of the terms more frequently used in the docs (in the code as well) is **"example"**:
- https://huggingface.co/docs/datasets/process.html#map
  > It allows you to apply a processing function to each example in a dataset, independently or in batches.
- https://huggingface.co/docs/datasets/process.html#batch-processing
  > datasets.Dataset.map() also supports working with batches of examples.
- https://huggingface.co/docs/datasets/process.html#split-long-examples
  > When your examples are too long, you may want to split them
- https://huggingface.co/docs/datasets/process.html#data-augmentation
  > With batch processing, you can even augment your dataset with additional examples.

Less frequently used: **"item"**:
- https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.add_item
  > Add item to Dataset.

Other term used in the docs (although less frequently) is **"sample"**. The advantage of this word is that it is also a verb, so we can use the parameter: "sample_by" (if you insist on using a verb instead of a noun).

In summary, these proposals:
- config.row
- config.example
- config.item
- config.sample
- config.sample_by

I like `sample_by`. Another idea I had was `separate_by`.

It could also be `sampling`, `sampling_method`, `separation_method`.

Not a big fan of the proposed nouns alone since they are very generic, that's why I tried to have something more specific.

I also agree that we actually should avoid `split` to avoid any confusion

Thanks for the analysis of the used terms. I also like `sample_by` (`separate_by` is good too).

Add `config.row` option to `text` module to allow yielding lines (default, current case), paragraphs or documents.

Feel free to comment on the name of the config parameter `row`:
- Currently, the docs state datasets are made of rows and columns
- Other names I considered: `example`, `item`


Add QuALITY dataset

dataset request

Requesting to add a new dataset

I'll take this one if no one hasn't yet!

## Adding a Dataset
- **Name:** QuALITY
- **Description:** A challenging question answering with very long contexts (Twitter [thread](https://twitter.com/sleepinyourhat/status/1471225421794529281?s=20))
- **Paper:** No ArXiv link yet, but draft is [here](https://github.com/nyu-mll/quality/blob/main/quality_preprint.pdf)
- **Data:** GitHub repo [here](https://github.com/nyu-mll/quality)
- **Motivation:** This dataset would serve as a nice way to benchmark long-range Transformer models like BigBird, Longformer and their descendants. In particular, it would be very interesting to see how the S4 model fares on this given it's impressive performance on the Long Range Arena

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


datasets keeps reading from cached files, although I disabled it

bug

Something isn't working

Hi ! What version of `datasets` are you using ? Can you also provide the logs you get before it raises the error ?

## Describe the bug
Hi,
I am trying to avoid dataset library using cached files, I get the following bug when this tried to read the cached files. I tried to do the followings:
```
from datasets import set_caching_enabled
set_caching_enabled(False)
```
also force redownlaod:
```
 download_mode='force_redownload'
```
but none worked so far, this is on a cluster and on some of the machines this reads from the cached files, I really appreciate any idea on how to fully remove caching @lhoestq
many thanks

```
File "run_clm.py", line 496, in <module>
    main()
  File "run_clm.py", line 419, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 943, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/transformers/trainer.py", line 1445, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 172, in evaluate
    output = self.eval_loop(
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 241, in eval_loop
    metrics = self.compute_pet_metrics(eval_datasets, model, self.extra_info[metric_key_prefix], task=task)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 268, in compute_pet_metrics
    centroids = self._compute_per_token_train_centroids(model, task=task)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 353, in _compute_per_token_train_centroids
    data = get_label_samples(self.get_per_task_train_dataset(task), label)
  File "/users/dara/codes/fewshot/debug/fewshot/third_party/trainers/trainer.py", line 350, in get_label_samples
    return dataset.filter(lambda example: int(example['labels']) == label)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 470, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2519, in filter
    indices = self.map(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2036, in map
    return self._map_single(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 503, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 470, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2248, in _map_single
    return Dataset.from_file(cache_file_name, info=info, split=self.split)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 654, in from_file
    return cls(
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 593, in __init__
    self.info.features = self.info.features.reorder_fields_as(inferred_features)
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/features/features.py", line 1092, in reorder_fields_as
    return Features(recursive_reorder(self, other))
  File "/users/dara/conda/envs/multisuccess/lib/python3.8/site-packages/datasets/features/features.py", line 1081, in recursive_reorder
    raise ValueError(f"Keys mismatch: between {source} and {target}" + stack_position)
ValueError: Keys mismatch: between {'indices': Value(dtype='uint64', id=None)} and {'candidates_ids': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None), 'labels': Value(dtype='int64', id=None), 'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None), 'input_ids': Sequence(feature=Value(dtype='int32', id=None), length=-1, id=None), 'extra_fields': {}, 'task': Value(dtype='string', id=None)}

```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: linux
- Python version: 3.8.12
- PyArrow version: 6.0.1


Add `cast_column` to `IterableDataset`

Closes #3369.

cc: @patrickvonplaten


Update supported versions of Python in setup.py

Update the list of supported versions of Python in `setup.py` to keep the PyPI project description updated.


Update BLEURT hyperlink

seems like a very very low-prio improvement :)

@albertvillanova thanks for the feedback! I removed the formatting altogether since I think this is a bit simpler tor read than non-rendered Markdown

The description of BLEURT on the hf.co website has a strange use of URL hyperlinking. This PR attempts to fix this, although I am not 100% sure Markdown syntax is allowed on the frontend or not.

![Screen Shot 2021-12-15 at 17 31 27](https://user-images.githubusercontent.com/26859204/146226432-c83cbdaf-f57d-4999-b53c-85da718ff7fb.png)


Add the OneStopQa dataset

Adding OneStopQA,  a multiple choice reading comprehension dataset annotated according to the STARC (Structured Annotations for Reading Comprehension) scheme.


Improve Wikipedia Loading Script

I wanted to flag a change from since we discussed this: I initially wrote a function for using the Wikimedia APIs to collect namespace aliases, but decided that adding in more http requests to the script wasn't a great idea so instead used that code to build a static list that I just added directly to the code.

Also, an FYI that python library dependencies weren't working on my local end so I wasn't able to directly test the code. I tested a copy with the problematic elements stripped (beam etc.) that worked fine, but someone with a working local copy may want to test just to make sure I didn't accidentally break anything.

Also, while I would argue more strongly for some of the changes in this code, they are five distinct changes so not so hard to remove one or two if other folks think they aren't worth the overhead etc.

I also add a comment by @geohci in the Issue page:
> See https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikipedia%20Processing.ipynb for more implementation details / some data around the overhead induced by adding the extra preprocessing steps (stripping link prefixes and magic words)

Hi ! Thanks a lot, this is very cool ! Note that unfortunately if we change the processing right now, users won't be able to load the "big" languages like english anymore, because it requires an Apache Beam runtime to process them. Some Wikipedia dumps have been processed by Hugging Face so that users don't need to run Apache Beam stuff.

Therefore, we can merge this change after we have processed dumps using this new processing, and host them on the Hugging Face google storage.

I think we can take care of this and let you know once this is ready ? What do you think @albertvillanova ?

This is also an opportunity to have the latest dumps ready, the current ones are from 2020

Related PR on updating to the latest dates: https://github.com/huggingface/datasets/pull/3612

@lhoestq if the additional processing steps are validated, we could go on generating the processed datasets for the big languages.

The only thing before doing that is that we should also validate other change (so that we include it also in the processed datasets):
- #3398

> @lhoestq if the additional processing steps are validated, we could go on generating the processed datasets for the big languages.

Cool ! Looking forward to it :)

> The only thing before doing that is that we should also validate other change (so that we include it also in the processed datasets):
>
> https://github.com/huggingface/datasets/issues/3398

Sounds good ! We can definitely add the URL as asked by the Wikipedia to provide credits to the authors.

@geohci I do not have push rights to this PR. See: [Enabling repository maintainer permissions on existing pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests).

I would like to merge the master branch so that all tests pass. Once done, I will be able approve this PR.

> @geohci I do not have push rights to this PR. See: [Enabling repository maintainer permissions on existing pull requests](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests).
>
> I would like to merge the master branch so that all tests pass. Once done, I will be able approve this PR.

@albertvillanova the `Allow edits by maintainers` box was already checked (what your instructions indicated) and indicates `If checked, users with write access to huggingface/datasets can add new commits to your wikipedia-updates branch. You can always change this setting later.` so you should have permissions already. If there's something else I'm missing or can do, please let me know. If it's not easy to resolve, I am plenty comfortable with you creating a new PR with these changes under your account too.

* More structured approach to detecting redirects
* Remove redundant template filter code (covered by strip_code)
* Add language-specific lists of additional media namespace aliases for filtering
* Add language-specific lists of category namespace aliases for new link text cleaning step
* Remove magic words (parser directions like __TOC__ that occasionally occur in text)

Fix #3400

With support from @albertvillanova
CC @yjernite


Add The People's Speech

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** The People's Speech
- **Description:** a massive English-language dataset of audio transcriptions of full sentences.
- **Paper:** https://openreview.net/pdf?id=R8CwidgJ0yT
- **Data:** https://mlcommons.org/en/peoples-speech/
- **Motivation:** With over 30,000 hours of speech, this dataset is the largest and most diverse freely available English speech recognition corpus today.

[The article](https://thegradient.pub/new-datasets-to-democratize-speech-recognition-technology-2/) which may be useful when working on the dataset.

cc: @anton-l

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add Multilingual Spoken Words dataset

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Multilingual Spoken Words
- **Description:** Multilingual Spoken Words Corpus is a large and growing audio dataset of spoken words in 50 languages for academic research and commercial applications in keyword spotting and spoken term search, licensed under CC-BY 4.0. The dataset contains more than 340,000 keywords, totaling 23.4 million 1-second spoken examples (over 6,000 hours).

  Read more: https://mlcommons.org/en/news/spoken-words-blog/
- **Paper:** https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/fe131d7f5a6b38b23cc967316c13dae2-Paper-round2.pdf
- **Data:** https://mlcommons.org/en/multilingual-spoken-words/
- **Motivation:**

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Correctly indent builder config in dataset script docs


Unable to resolve any data file after loading once

Hi ! `load_dataset` accepts as input either a local dataset directory or a dataset name from the Hugging Face Hub.

So here you are getting this error the second time because it tries to load the local `wiki_dpr` directory, instead of `wiki_dpr` from the Hub. It doesn't work since it's a **cache** directory, not a **dataset** directory in itself.

To fix that you can use another cache directory like `cache_dir="/data2/whr/lzy/open_domain_data/retrieval/cache"`

when I rerun my program, it occurs this error
" Unable to resolve any data file that matches '['**train*']' at /data2/whr/lzy/open_domain_data/retrieval/wiki_dpr with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'zip']", so how could i deal with this problem?
thx.
And below is my code .
![image](https://user-images.githubusercontent.com/84694183/146023446-d75fdec8-65c1-484f-80d8-6c20ff5e994b.png)


Make decoding of Audio and Image feature optional

Closing this PR for now due to https://github.com/huggingface/datasets/issues/3145#issuecomment-993664104.

Okay, after some more thinking, I'm re-opening this PR for three reasons:
* This feature will allow us to remove the `image_file_path`/`audio_file_path` columns in our vision/audio datasets. Currently, it makes sense to keep those columns because it's not obvious how to access the underlying path information of the Image/Audio feature. However, if the user is not aware and does `dataset[0]["image_file_path"]` on our vision/audio datasets, this will be costly because the image/audio file data has to be decoded first (stored in `dataset[0]["image"]`/`dataset[0]["audio"]`)
* In CV, we often work with the so-called "half life" datasets (RedCaps, WIT, ...) that only provide image URLs and not actual image data, and some of these image URLs even go down. Our solution to this problem is to give a note on how to efficiently download the image data using `map` in the datasets cards of such datasets. This feature will remove the need for a separate `image_url` column of type `Value("string")` in such datasets. Instead, we will be able to use the `image` column of type `Image()`  (the image feature knows how to decode image URLs using `xopen`), disable decoding and use `requests.get` for download, which I expect to be faster than `xopen`.
* This feature should help us in implementing `push_to_hub` for the Image/Audio where we transfer actual image/audio data and not paths

> This feature will allow us to remove the image_file_path/audio_file_path columns in our vision/audio datasets. Currently, it makes sense to keep those columns because it's not obvious how to access the underlying path information of the Image/Audio feature. However, if the user is not aware and does dataset[0]["image_file_path"] on our vision/audio datasets, this will be costly because the image/audio file data has to be decoded first (stored in dataset[0]["image"]/dataset[0]["audio"])

That makes sense !

> Instead, we will be able to use the image column of type Image() (the image feature knows how to decode image URLs using xopen), disable decoding and use requests.get for download, which I expect to be faster than xopen.

I feel like it's a bit convoluted compared to having the `image_url` column as string, and say to users to `map` using `requests.get` with `image_url`.

Moreover I'm not 100% sure that we should have `Image` features with both local paths and URLs, since this behavior is a bit hidden the users and they don't give the same performance at all.

> This feature should help us in implementing push_to_hub for the Image/Audio where we transfer actual image/audio data and not paths

Cool !

Thanks, @lhoestq.

> I feel like it's a bit convoluted compared to having the image_url column as string, and say to users to map using requests.get with image_url.

Yes, that makes sense.

>Moreover I'm not 100% sure that we should have Image features with both local paths and URLs, since this behavior is a bit hidden the users and they don't give the same performance at all.

Do you mean we should remove support for URLs in the Image feature? Because this is what we get for free by adding support for streaming (by using `xopen` instead of `open`) and this is also what the Audio feature does.

> Do you mean we should remove support for URLs in the Image feature? Because this is what we get for free by adding support for streaming (by using xopen instead of open) and this is also what the Audio feature does.

I think it might not be ideal to have URLs in an `Image` type column for a dataset in **non-streaming** mode, since you'd expect to have everything locally. But for a streaming dataset it must use `xopen` indeed

Yes, I agree. Let's have the `image_url` columns as `Value("string")` + a note with the map function to download images for local datasets and a note with `cast_column` (which is requested in https://github.com/huggingface/datasets/issues/3369) for streamed datasets (`ds.cast_column("image_url", Image())`).

I fixed the merge conflicts and small bugs in nested decoding introduced by #3575. Additionally, I addressed https://github.com/huggingface/datasets/issues/3473 by adding the `_iter` method to `Dataset` (inspired by the `_getitem` method). For `iter(dset)` I set `_iter(dset, decoded=True)` to enable decoding and for `map` `_iter(dset, decoded=False)` to make it lazy.

Add the `decode` argument (`True` by default) to the `Audio` and the `Image` feature to make it possible to toggle on/off decoding of these features.

Even though we've discussed that on Slack, I'm not removing the `_storage_dtype` argument of the Audio feature in this PR to avoid breaking the Audio feature tests.


Make cast cacheable (again) on Windows

`cast` currently emits the following warning when called on Windows:
```
Parameter 'function'=<function Dataset.cast.<locals>.<lambda> at 0x000001C930571EA0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting
and caching to work. If you reuse this transform, the caching mechanism will consider it to be different
from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
```

It seems like the issue stems from the `config.PYARROW_VERSION` object not being serializable on Windows (tested with `dumps(lambda: config.PYARROW_VERSION)`), so I'm fixing this by capturing `config.PYARROW_VERSION.major` before the lambda definition.


Clean squad dummy data

Some unused files were remaining, this PR removes them. We just need to keep the dummy_data.zip file


Add The Pile Enron Emails subset

Add:
- Enron Emails subset of The Pile: "enron_emails" config

Close bigscience-workshop/data_tooling#310.

CC: @StellaAthena


Update disaster_response_messages download urls (+ add validation split)

Fixes #3240, fixes #3416


Getting configs names takes too long

bug

Something isn't working

maybe related to https://github.com/huggingface/datasets/issues/2859

It looks like it's currently calling `HfFileSystem.ls()` ~8 times at the root and for each subdirectory:
- ""
- "en.noblocklist"
- "en.noclean"
- "en"
- "multilingual"
- "realnewslike"

Currently `ls` is slow because it iterates on all the files inside the repository.

An easy optimization would be to cache the result of each call to `ls`.
We can also optimize `ls` by using a tree structure per directory instead of a list of all the files.

## Steps to reproduce the bug
```python
from datasets import get_dataset_config_names
get_dataset_config_names("allenai/c4")
```

## Expected results

I would expect to get the answer quickly, at least in less than 10s

## Actual results

It takes about 45s on my environment

## Environment info

- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-1022-aws-x86_64-with-glibc2.31
- Python version: 3.9.6
- PyArrow version: 4.0.1


Add RedCaps dataset

Cool ! If you want you can include `dataset_infos.json` but only for the main configurations. That's what we do for example for translation datasets when there are too many configs

@lhoestq I've added an example that uses `map` to download the images.

Add the RedCaps dataset. I'm not adding the generated `dataset_infos.json` file for now due to its size (11 MB).

TODOs:
- [x] dummy data
- [x] dataset card

Close #3316


data duplicate when setting num_works > 1 with streaming data

Something isn't working

Hi ! Thanks for reporting :)

When using a PyTorch's data loader with `num_workers>1` and an iterable dataset, each worker streams the exact same data by default, resulting in duplicate data when iterating using the data loader.

We can probably fix this in `datasets` by checking `torch.utils.data.get_worker_info()` which gives the worker id if it happens.

> Hi ! Thanks for reporting :)
>
> When using a PyTorch's data loader with `num_workers>1` and an iterable dataset, each worker streams the exact same data by default, resulting in duplicate data when iterating using the data loader.
>
> We can probably fix this in `datasets` by checking `torch.utils.data.get_worker_info()` which gives the worker id if it happens.
Hi ! Thanks for reply

Do u have some plans to fix the problem?

Isn’t that somehow a bug on PyTorch side? (Just asking because this behavior seems quite general and maybe not what would be intended)

From PyTorch's documentation [here](https://pytorch.org/docs/stable/data.html#dataset-types):

> When using an IterableDataset with multi-process data loading. The same dataset object is replicated on each worker process, and thus the replicas must be configured differently to avoid duplicated data. See [IterableDataset](https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset) documentations for how to achieve this.

It looks like an intended behavior from PyTorch

As suggested in the [docstring of the IterableDataset class](https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset), we could pass a `worker_init_fn` to the DataLoader to fix this. It could be called `streaming_worker_init_fn` for example.

However, while this solution works, I'm worried that many users simply don't know about this parameter and just start their training with duplicate data without knowing it. That's why I'm more in favor of integrating the check on the worker id directly in `datasets` in our implementation of `IterableDataset.__iter__`.

## Describe the bug
The data is repeated num_works times when we load_dataset with streaming and set num_works > 1 when construct dataloader

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import pandas as pd
import numpy as np
import os

from datasets import load_dataset
from torch.utils.data import DataLoader
from tqdm import tqdm
import shutil

NUM_OF_USER = 1000000
NUM_OF_ACTION = 50000
NUM_OF_SEQUENCE = 10000
NUM_OF_FILES = 32
NUM_OF_WORKERS = 16

if __name__ == "__main__":
    shutil.rmtree("./dataset")
    for i in range(NUM_OF_FILES):
        sequence_data = pd.DataFrame(
            {
                "imei": np.random.randint(1, NUM_OF_USER, size=NUM_OF_SEQUENCE),
                "sequence": np.random.randint(1, NUM_OF_ACTION, size=NUM_OF_SEQUENCE)
            }
        )

        if not os.path.exists("./dataset"):
            os.makedirs("./dataset")

        sequence_data.to_csv(f"./dataset/sequence_data_{i}.csv",

 index=False)

    dataset = load_dataset("csv",
                           data_files=[os.path.join("./dataset",file) for file in os.listdir("./dataset") if file.endswith(".csv")],
                           split="train",
                           streaming=True).with_format("torch")
    data_loader = DataLoader(dataset,
                             batch_size=1024,
                             num_workers=NUM_OF_WORKERS)

    result = pd.DataFrame()
    for i, batch in tqdm(enumerate(data_loader)):
        result = pd.concat([result,
                           pd.DataFrame(batch)],
                           axis=0)
    result.to_csv(f"num_work_{NUM_OF_WORKERS}.csv", index=False)

```

## Expected results
data do not duplicate

## Actual results
data duplicate NUM_OF_WORKERS = 16
![image](https://user-images.githubusercontent.com/16486492/145748707-9d2df25b-2f4f-4d7b-a83e-242be4fc8934.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:datasets==1.14.0
- Platform:transformers==4.11.3
- Python version:3.8
- PyArrow version:


Error about load_metric

Something isn't working

Hi ! I wasn't able to reproduce your error.

Can you try to clear your cache at `~/.cache/huggingface/modules` and try again ?

## Describe the bug
  File "/opt/conda/lib/python3.8/site-packages/datasets/load.py", line 1371, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable

## Steps to reproduce the bug
```python
metric = load_metric("glue", "sst2")
```

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyArrow version: 6.0.1


Adding mMARCO dataset

Hi @albertvillanova we've made a major overhaul of the loading script including all configurations we're making available. Could you please review it again?

Thanks @lhbonifacio for adding this dataset.
Hi there, i got an error about mmarco:
ConnectionError: Couldn't reach 'unicamp-dl/mmarco' on the Hub (ConnectionError)
code:
`from datasets import list_datasets, load_dataset
dataset = load_dataset('unicamp-dl/mmarco', language='portuguese')`

Any help will be appreciated!

Hi @catqaq, we updated the loading script. Now you can load the datasets with:

```python
dataset = load_dataset('unicamp-dl/mmarco', 'portuguese')
```

You can check the list of supported languages and usage examples in [this link](https://huggingface.co/datasets/unicamp-dl/mmarco). Feel free to contact us if you have any issues.

>

> Hi @catqaq, we updated the loading script. Now you can load the datasets with:
>
> ```python
> dataset = load_dataset('unicamp-dl/mmarco', 'portuguese')
> ```
>
> You can check the list of supported languages and usage examples in [this link](https://huggingface.co/datasets/unicamp-dl/mmarco). Feel free to contact us if you have any issues.

Thanks for your quick updates. So, how can i get the fixed version, install from the source? It seems that the merging is blocked.

@catqaq you can load mMARCO using the namespace `unicamp-dl/mmarco` while this PR remains under review.

Adding mMARCO (v1.1) to HF datasets.


Add eli5_category dataset

> Thanks a lot for adding this dataset ! Good job with the dataset card and the dataset scripts - they're really good :)
>
> I just added minor changes

Thanks for fixing typos!

This pull request adds a categorized Long-form question answering dataset `ELI5_Category`. It's a new variant of the [ELI5](https://huggingface.co/datasets/eli5) dataset that uses the Reddit tags to alleviate the training/validation overlapping in the origin ELI5 dataset.

A [report](https://celeritasml.netlify.app/posts/2021-12-01-eli5c/)(Section 2) on this dataset.


`.to_json` is extremely slow after `.select`

Something isn't working

Hi ! It's slower indeed because a datasets on which `select`/`shard`/`train_test_split`/`shuffle` has been called has to do additional steps to retrieve the data of the dataset table in the right order.

Indeed, if you call `dataset.select([0, 5, 10])`, the underlying table of the dataset is not altered to keep the examples at index 0, 5, and 10. Instead, an indices mapping is added on top of the table, that says that the first example is at index 0, the second at index 5 and the last one at index 10.

Therefore accessing the examples of the dataset is slower because of the additional step that uses the indices mapping.

The step that takes the most time is to query the dataset table from a list of indices here:

https://github.com/huggingface/datasets/blob/047dc756ed20fbf06e6bcaf910464aba0e20610a/src/datasets/formatting/formatting.py#L61-L63

In your case it can be made significantly faster by checking if the indices are contiguous. If they're contiguous, we could pass a python `slice` or `range` instead of a list of integers to `_query_table`. This way `_query_table` will do only one lookup to get the queried batch instead of `batch_size` lookups.

Given that calling `select` with contiguous indices is a common use case I'm in favor of implementing such an optimization :)
Let me know what you think

Hi, thanks for the response!
I still don't understand why it is so much slower than iterating and saving:
```python
from datasets import load_dataset

original = load_dataset("squad", split="train")
original.to_json("from_original.json")  # Takes 0 seconds

selected_subset1 = original.select([i for i in range(len(original))])
selected_subset1.to_json("from_select1.json")  # Takes 99 seconds

selected_subset2 = original.select([i for i in range(int(len(original) / 2))])
selected_subset2.to_json("from_select2.json")  # Takes 47 seconds

selected_subset3 = original.select([i for i in range(len(original)) if i % 2 == 0])
selected_subset3.to_json("from_select3.json")  # Takes 49 seconds

import json
import time
def fast_to_json(dataset, path):
    start = time.time()
    with open(path, mode="w") as f:
        for example in dataset:
            f.write(json.dumps(example, separators=(',', ':')) + "\n")
    end = time.time()
    print(f"Saved {len(dataset)} examples to {path} in {end - start} seconds.")

fast_to_json(original, "from_original_fast.json")
fast_to_json(selected_subset1, "from_select1_fast.json")
fast_to_json(selected_subset2, "from_select2_fast.json")
fast_to_json(selected_subset3, "from_select3_fast.json")
```
```
Saved 87599 examples to from_original_fast.json in 8 seconds.
Saved 87599 examples to from_select1_fast.json in 10 seconds.
Saved 43799 examples to from_select2_fast.json in 6 seconds.
Saved 43800 examples to from_select3_fast.json in 5 seconds.
```

There are slight differences between what you're doing and what `to_json` is actually doing.
In particular `to_json` currently converts batches of rows (as an arrow table) to a pandas dataframe, and then to JSON Lines. From your benchmark it looks like it's faster if we don't use pandas.

Thanks for investigating, I think we can optimize `to_json` significantly thanks to your test.

Thanks for your observations, @eladsegal! I spent some time with this and tried different approaches. Turns out that https://github.com/huggingface/datasets/blob/bb13373637b1acc55f8a468a8927a56cf4732230/src/datasets/io/json.py#L100 is giving the problem when we use `to_json` after `select`. This is when `indices` parameter in `query_table` is not `None` (if it is `None` then `to_json` should work as expected)

In order to circumvent this problem, I found out instead of doing Arrow Table -> Pandas-> JSON we can directly go to JSON by using `to_pydict()` which is a little slower than the current approach but at least `select` works properly now. Lmk what you guys think of it @lhoestq, @eladsegal?

Sounds good to me ! Feel free to also share your benchmarks for reference @bhavitvyamalik

Posting it in @eladsegal's format:

For `squad`:
Saving examples using current `to_json` in 3.63 secs
Saving examples to `from_select1_fast.json` in 5.00 secs
Saving examples to `from_select2_fast.json` in 2.45 secs
Saving examples to `from_select3_fast.json` in 2.50 secs

For `squad_v2`:
Saving examples using current `to_json` in 5.26 secs
Saving examples to `from_select1_fast.json` in 7.54 secs
Saving examples to `from_select2_fast.json` in 3.80 secs
Saving examples to `from_select3_fast.json` in 3.67 secs

## Describe the bug
Saving a dataset to JSON with `to_json` is extremely slow after using `.select` on the original dataset.

## Steps to reproduce the bug
```python
from datasets import load_dataset

original = load_dataset("squad", split="train")
original.to_json("from_original.json")  # Takes 0 seconds

selected_subset1 = original.select([i for i in range(len(original))])
selected_subset1.to_json("from_select1.json")  # Takes 212 seconds

selected_subset2 = original.select([i for i in range(int(len(original) / 2))])
selected_subset2.to_json("from_select2.json")  # Takes 90 seconds
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: master (https://github.com/huggingface/datasets/commit/6090f3cfb5c819f441dd4a4bb635e037c875b044)
- Platform: Linux-4.4.0-19041-Microsoft-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 6.0.0


Add Wikisource dataset

Add loading script for Wikisource dataset.

Fix #3399.

CC: @geohci, @yjernite


Fix type of bridge field in QED

Use `Value("string")` instead of `Value("bool")` for the feature type of the `"bridge"` field in the QED dataset. If the value is `False`, set to `None`.

The following paragraph in the QED repo explains the purpose of this field:
>Each annotation in referential_equalities is a pair of spans, the question_reference and the sentence_reference, corresponding to an entity mention in the question and the selected_sentence respectively. As described in the paper, sentence_references can be "bridged in", in which case they do not correspond with any actual span in the selected_sentence. Hence, sentence_reference spans contain an additional field, bridge, which is a prepositional phrase when a reference is bridged, and is False otherwise. Prepositional phrases serve to link bridged references to an anchoring phrase in the selected_sentence. In the case a sentence_reference is bridged, the start and end, as well as the span string, map to such an anchoring phrase in the selected_sentence.

Fix #3346

cc @VictorSanh


disaster_response_messages unavailable

Hi, thanks for reporting! This is a duplicate of https://github.com/huggingface/datasets/issues/3240. We are working on a fix.

## Dataset viewer issue for '* disaster_response_messages*'

**Link:** https://huggingface.co/datasets/disaster_response_messages

Dataset unavailable. Link dead: https://datasets.appen.com/appen_datasets/disaster_response_data/disaster_response_messages_training.csv

Am I the one who added this dataset ?No


Non-deterministic tests: CI tests randomly fail

Something isn't working

I think it might come from two different issues:
1. Google Drive is an unreliable host, mainly because of quota limitations
2. the staging environment can sometimes raise some errors

For Google Drive tests we could set up some retries with backup URLs if necessary I guess.
For staging on the other hand, I guess we can investigate what causes this and discuss with the back-end team

## Describe the bug
Some CI tests fail randomly.

1. In https://github.com/huggingface/datasets/pull/3375/commits/c10275fe36085601cb7bdb9daee9a8f1fc734f48, there were 3 failing tests, only on Linux:
   ```
   =========================== short test summary info ============================
   FAILED tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol[https://drive.google.com/uc?export=download&id=1k92sUfpHxKq8PXWRr7Y5aNHXwOCNUmqh-zip]
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive - Fi...
   FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
   = 3 failed, 3553 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 192.79s (0:03:12) =
   ```

2. After re-running the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/57bfe1f342cd3c59d2510b992d5f06a0761eb147, there was only 1 failing test (one on Linux and a different one on Windows):
   - On Linux:
      ```
      =========================== short test summary info ============================
      FAILED tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped
      = 1 failed, 3555 passed, 2950 skipped, 2 xfailed, 1 xpassed, 125 warnings in 199.76s (0:03:19) =
      ```
   - On Windows:
      ```
      =========================== short test summary info ===========================
      FAILED tests/test_load.py::test_load_dataset_builder_for_community_dataset_without_script
      = 1 failed, 3551 passed, 2954 skipped, 2 xfailed, 1 xpassed, 121 warnings in 478.58s (0:07:58) =
      ```

   The test `tests/test_streaming_download_manager.py::test_streaming_gg_drive_zipped` passes locally.

3. After re-running again the CI (without any change in the code) in https://github.com/huggingface/datasets/pull/3375/commits/39f32f2119cf91b86867216bb5c356c586503c6a, ALL the tests passed.


Skip None encoding (line deleted by accident in #3195)

Return the line deleted by accident in #3195 while [resolving merge conflicts](https://github.com/huggingface/datasets/pull/3195/commits/8b0ed15be08559056b817836a07d47acda0c4510).

Fix #3181 (finally :))


Add WIDER FACE dataset

Adds the WIDER FACE face detection benchmark.

TODOs:
* [x] dataset card
* [x] dummy data


Fix flaky test again for s3 serialization

Following https://github.com/huggingface/datasets/pull/3388 that wasn't enough (see CI error [here](https://app.circleci.com/pipelines/github/huggingface/datasets/9080/workflows/b971fb27-ff20-4220-9416-c19acdfdf6f4/jobs/55985))


[chinese wwm] load_datasets behavior not as expected when using run_mlm_wwm.py script

Something isn't working

@LysandreJik not so sure who to @
Could you help?

Hi @hyusterr, I believe it is @wlhgtc from https://github.com/huggingface/transformers/pull/9887

## Describe the bug

Model I am using (Bert, XLNet ...): bert-base-chinese

The problem arises when using:
* [https://github.com/huggingface/transformers/blob/master/examples/research_projects/mlm_wwm/run_mlm_wwm.py] the official example scripts: `rum_mlm_wwm.py`

The tasks I am working on is: pretraining whole word masking with my own dataset and ref.json file
I tried follow the run_mlm_wwm.py procedure to do whole word masking on pretraining task. my file is in .txt form, where one line represents one sample, with `9,264,784` chinese lines in total. the ref.json file is also contains 9,264,784 lines of whole word masking reference data for my chinese corpus. but when I try to adapt the run_mlm_wwm.py script, it shows that somehow after
`datasets["train"] = load_dataset(...`
`len(datasets["train"])` returns `9,265,365`
then, after `tokenized_datasets = datasets.map(...`
`len(tokenized_datasets["train"])` returns `9,265,279`
I'm really confused and tried to trace code by myself but can't know what happened after a week trial.

I want to know what happened in the `load_dataset()` function and `datasets.map` here and how did I get more lines of data than I input. so I'm here to ask.
## To reproduce
Sorry that I can't provide my data here since it did not belong to me. but I'm sure I remove the blank lines.

## Expected behavior
I expect the code run as it should. but the AssertionError in line 167 keeps raise as the line of reference json and datasets['train'] differs.

Thanks for your patient reading!

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 3.0.0


Fix dependencies conflicts in Windows CI after conda update to 4.11

For some reason the CI wasn't using python 3.6 but python 3.7 after the update to conda 4.11


Pass new_fingerprint in multiprocessing

@lhoestq Hi~, does this support that `datasets.map(func, batched=True, batch_size, num_proc>1, new_fingerprint="func_v1")` even if `func` can't pickle. I also notice that you said "Unfortunately you need picklable mapping functions to make multiprocessing work :confused: Also feel free to open an issue or send me a dm if you are in a situation where the caching fails. I can help you with that :slight_smile:" in [here](https://discuss.huggingface.co/t/how-to-deal-with-unpickable-objects-in-map/1547/8). So, I want to ask that is there a way for users to use multiprocessing in `datasets.map` when the `func` can't pickle?
Thanks in advance!

Yea you need your function to be picklable for multiprocessing, otherwise the main process is not able to pass your function to the child processes.

Following https://github.com/huggingface/datasets/pull/3045

Currently one can pass `new_fingerprint` to `.map()` to use a custom fingerprint instead of the one computed by hashing the map transform. However it's ignored if `num_proc>1`.

In this PR I fixed that by passing `new_fingerprint` to `._map_single()` when `num_proc>1`.
More specifically, `new_fingerprint` with a suffix based on the process `rank` is passed, so that each process has a different `new_fingerprint`

cc @TevenLeScao @vlievin


Typo in Dataset viewer error message

Fixed, thanks
<img width="661" alt="Capture d’écran 2021-12-22 à 12 02 30" src="https://user-images.githubusercontent.com/1676121/147082881-cf700e8d-0511-4431-b214-d6cf8137db10.png">

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

When creating an empty dataset repo, the Dataset Preview provides a helpful message that no files were found. There is a tiny typo in that message: "ressource" should be "resource"

![Screen Shot 2021-12-09 at 15 31 31](https://user-images.githubusercontent.com/26859204/145415725-9cd728f0-c2c8-4b4e-a8e1-4f4d7841c94a.png)

Am I the one who added this dataset ?

N/A


Use max number of data files to infer module

Cool thanks :) Feel free to merge if it's all good for you

When inferring the module for datasets without script, set a maximum number of iterations over data files.

This PR fixes the issue of taking too long when hundred of data files present.

Please, feel free to agree on both numbers:
```
# Datasets without script
DATA_FILES_MAX_NUMBER = 10
ARCHIVED_DATA_FILES_MAX_NUMBER = 5
```

Fix #3404.


Fix module inference for archive with a directory

Fix module inference for an archive file that contains files within a directory.

Fix #3405.


ZIP format inference does not work when files located in a dir inside the archive

Something isn't working

## Describe the bug
When a zipped file contains archived files within a directory, the function `infer_module_for_data_files_in_archives` does not work.

It only works for files located in the root directory of the ZIP file.

## Steps to reproduce the bug
```python
infer_module_for_data_files_in_archives(["path/to/zip/file.zip"], False)
```


Optimize ZIP format inference

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
When hundreds of ZIP files are present in a dataset, format inference takes too long.

See: https://github.com/bigscience-workshop/data_tooling/issues/232#issuecomment-986685497

**Describe the solution you'd like**
Iterate over a maximum number of files.

CC: @lhoestq


Cannot import name 'maybe_sync'

bug

Something isn't working

Hi ! Can you try updating `fsspec` ? The minimum version is `2021.05.0`

hey @lhoestq. I'm using `fsspec-2021.11.1` but still getting that error.

Maybe this discussion can help:

https://github.com/fsspec/filesystem_spec/issues/597#issuecomment-958646964

Thanks @lhoestq. Downgrading `fsspec and s3fs` to `2021.10` fixed this issue!

## Describe the bug
Cannot seem to import datasets when running run_summarizer.py script on a VM set up on ovhcloud

## Steps to reproduce the bug
```python
from datasets import load_dataset
```

## Expected results
No error

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 48, in <module>
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence
  File "/opt/conda/lib/python3.7/site-packages/datasets/arrow_writer.py", line 27, in <module>
    from .features import (
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/__init__.py", line 2, in <module>
    from .audio import Audio
  File "/opt/conda/lib/python3.7/site-packages/datasets/features/audio.py", line 8, in <module>
    from ..utils.streaming_download_manager import xopen
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/streaming_download_manager.py", line 16, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/__init__.py", line 13, in <module>
    from .s3filesystem import S3FileSystem  # noqa: F401
  File "/opt/conda/lib/python3.7/site-packages/datasets/filesystems/s3filesystem.py", line 1, in <module>
    import s3fs
  File "/opt/conda/lib/python3.7/site-packages/s3fs/__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "/opt/conda/lib/python3.7/site-packages/s3fs/core.py", line 11, in <module>
    from fsspec.asyn import AsyncFileSystem, sync, sync_wrapper, maybe_sync
ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn' (/opt/conda/lib/python3.7/site-packages/fsspec/asyn.py)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.16.0
- Platform: OVH Cloud Tesla V100 Machine
- Python version: 3.7.9
- PyArrow version: 6.0.1


More robust first elem check in encode/cast example

Fix #3306


Add Wikimedia pre-processed datasets

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Add pre-processed data to:
  - *wikimedia/wikipedia*: https://huggingface.co/datasets/wikimedia/wikipedia
  - *wikimedia/wikisource*: https://huggingface.co/datasets/wikimedia/wikisource
- **Description:** Add pre-processed data to the Hub for all languages
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** This will be very useful for the NLP community, as the pre-processing has a high cost for lot of researchers (both in computation and in knowledge)

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite


Improve Wikipedia loading script

Requesting to add a new dataset

Thanks! See https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikipedia%20Processing.ipynb for more implementation details / some data around the overhead induced by adding the extra preprocessing steps (stripping link prefixes and magic words)

As reported by @geohci, the "wikipedia" processing/loading script could be improved by some additional small suggested processing functions:
- _extract_content(filepath):
  - Replace .startswith("#redirect") with more structured approach: if elem.find(f"./{namespace}redirect") is None: continue
- _parse_and_clean_wikicode(raw_content, parser):
  - Remove rm_template from cleaning -- this is redundant with .strip_code() from mwparserformhell
  - Build a language-specific list of namespace prefixes to filter out per below get_namespace_prefixes
  - Optional: strip prefixes like categories -- e.g., Category:Towns in Tianjin becomes Towns in Tianjin
  - Optional: strip magic words


Add Wikisource dataset

Requesting to add a new dataset

See notebook by @geohci: https://public.paws.wmcloud.org/User:Isaac_(WMF)/HuggingFace%20Wikisource%20Processing.ipynb

## Adding a Dataset
- **Name:** *wikisource*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** Additional high quality textual data, besides Wikipedia.

Add loading script as "canonical" dataset (as it is the case for ""wikipedia").

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

CC: @geohci, @yjernite


Add URL field to Wikimedia dataset instances: wikipedia,...

Requesting to add a new dataset

@geohci, I think the field "url" does not appear in the Wikimedia dumps. Therefore I guess we should generate it, using the "title" field and making some transformation of it (replacing spaces with underscores) and prepending the domain (created using the language)?

Indeed:

> To re-distribute text on Wikipedia in any form, provide credit to the authors either by including a) a [hyperlink](https://en.wikipedia.org/wiki/Hyperlink) (where possible) or [URL](https://en.wikipedia.org/wiki/URL) to the page or pages you are re-using, b) a hyperlink (where possible) or URL to an alternative, stable online copy which is freely accessible, which conforms with the license, and which provides credit to the authors in a manner equivalent to the credit given on this website, or c) a list of all authors. (Any list of authors may be filtered to exclude very small or irrelevant contributions.) This applies to text developed by the Wikipedia community. Text from external sources may attach additional attribution requirements to the work, which should be indicated on an article's face or on its talk page. For example, a page may have a banner or other notation indicating that some or all of its content was originally published somewhere else. Where such notations are visible in the page itself, they should generally be preserved by re-users.

source: https://en.wikipedia.org/wiki/Wikipedia:Copyrights

I guess it's fine to add the URL field - it can be constructed easily from the title page IIRC.

yep, sorry forgot that that wasn't already in the dumps. specifically `f"https://{language}.wikipedia.org/wiki/{title.replace(' ', '_')}` should do it

Thanks @geohci.

I had already been looking for information about the conversion from title to URL and I found that apart from replacing blanks with underscores, some other special character must also be percent-encoded (e.g. `"` to `%22`): https://meta.wikimedia.org/wiki/Help:URL

Therefore, I have finally used `urllib.parse.quote` function. This additionally percent-encodes non-ASCII characters, but Wikimedia docs say these are equivalent:
> For the other characters either the code or the character can be used in internal and external links, they are equivalent. The system does a conversion when needed.
> [[%C3%80_propos_de_M%C3%A9ta]]
> is rendered as [À_propos_de_Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), almost like [À propos de Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), which leads to this page on Meta with in the address bar the URL
> [http://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta)
> while [http://meta.wikipedia.org/wiki/À_propos_de_Méta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta) leads to the same.

As reported by @geohci, in order to host pre-processed data in the Hub, we should add the full URL to data instances (new field "url"), so that we conform to proper attribution from license requirement. See, e.g.: https://fair-trec.github.io/docs/Fair_Ranking_2021_Participant_Instructions.pdf#subsection.3.2

This should be done for all pre-processed datasets under "wikimedia" org in the Hub: https://huggingface.co/wikimedia