konrud/llm_course_chapter5_own_dataset_github_issues huggingface dataset:
Add IterableDataset.filter

_Needs https://github.com/huggingface/datasets/pull/3801 to be merged first_

I added `IterableDataset.filter` with an API that is a subset of `Dataset.filter`:
```python
def filter(self, function, batched=False, batch_size=1000, with_indices=false, input_columns=None):
```

TODO:
- [x] tests
- [x] docs

related to https://github.com/huggingface/datasets/issues/3444 and https://github.com/huggingface/datasets/issues/3753

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3826). All of your documentation changes will be reflected on that endpoint.

Indeed ! If `batch_size` is `None` or `<=0` then the full dataset should be passed. It's been mentioned in the docs for a while but never actually implemented. We can fix that later
Add Biwi Kinect Head Pose Database

## Adding a Dataset
- **Name:** Biwi Kinect Head Pose Database
- **Description:** Over 15K images of 20 people recorded with a Kinect while turning their heads around freely. For each frame, depth and rgb images are provided, together with ground in the form of the 3D location of the head and its rotation angles.
- **Data:** [*link to the Github repository or current dataset location*](https://icu.ee.ethz.ch/research/datsets.html)
- **Motivation:** Useful pose estimation dataset

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Official dataset location : https://icu.ee.ethz.ch/research/datsets.html
In the "Biwi Kinect Head Pose Database" section, I do not find any information regarding "Downloading the dataset." . Do we mail the authors regarding this ?

I found the dataset on Kaggle : [Link](https://www.kaggle.com/kmader/biwi-kinect-head-pose-database) , but since 🤗 does not host any of the datasets, this would require the user to provide their Kaggle username and API key to download. 

Any inputs on how we could proceed ? Thank you.
[ Need your inputs here, @lhoestq or @mariosasko ]

Hi @dnaveenr! Thanks for tackling this issue. This link should work: https://data.vision.ee.ethz.ch/cvl/gfanelli/kinect_head_pose_db.tgz

#self-assign

Added in https://github.com/huggingface/datasets/pull/3903, thanks @dnaveenr !

@mariosasko Hi,
this link [https://data.vision.ee.ethz.ch/cvl/gfanelli/kinect_head_pose_db.tgz]
seems not working now. Do you have alternatives?

It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose

> It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose

Hi, you mean I can download it by running python script in your link?

This dataset repository only contains a script that points to the URL of the file that doesn't exist anymore, so it's not usable anymore either :/

> This dataset repository only contains a script that points to the URL of the file that doesn't exist anymore, so it's not usable anymore either :/

Then what did you mean 
"It would be cool to find a copy of the data and host it directly in https://huggingface.co/datasets/ETHZurich/biwi_kinect_head_pose"?
I don't understand the meaning of 'host'

If we find a copy of the data we can upload the file in the dataset repository on HF, since the original data is not available anymore
Update Wikipedia dataset

This PR combines all updates to Wikipedia dataset.

Once approved, this will be used to generate the pre-processed Wikipedia datasets.

Finally, this PR will be able to be merged into master:
- NOT using squash
- BUT a regular MERGE (or REBASE+MERGE), so that all commits are preserved

TODO:
- [x] #3435
- [x] #3789
- [x] #3825
- [x] Run to get the pre-processed data for big languages (backward compatibility)
- [x] #3958

CC: @geohci

_The documentation is not available anymore as the PR was closed or merged._

I'm starting to generate the pre-processed data for some of the languages (for backward compatibility).

Once this merged, we will create the pre-processed data on the Hub under the Wikimedia namespace.

All steps have been properly done.

I'm merging all these commits into master.
`pubmed_qa` checksum mismatch

## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0


Hi @jon-tow, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

We are planning to make a patch release today.

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Simplify Common Voice code

In #3736 we introduced one method to generate examples when streaming, that is different from the one when not streaming.

In this PR I propose a new implementation which is simpler: it only has one function, based on `iter_archive`. And you still have access to local audio files when loading the dataset in non-streaming mode.

cc @patrickvonplaten @polinaeterna @anton-l @albertvillanova since this will become the template for many audio datasets to come.

This change can also trivially be applied to the other audio datasets that already exist. Using this line, you can get access to local files in non-streaming mode:
```python
local_extracted_archive = dl_manager.extract(archive_path) if not dl_manager.is_streaming else None
```

I think the script looks pretty clean and readable now! cool!
Handle Nones in PyArrow struct

This PR fixes an issue introduced by #3575 where `None` values stored in PyArrow arrays/structs would get ignored by `cast_storage` or by the `pa.array(cast_to_python_objects(..))` pattern. To fix the former, it also bumps the minimal PyArrow version to v5.0.0 to use the `mask` param in `pa.SturctArray`. 



Looks like I added my comments while you were editing - sorry about that
Add MetaShift dataset

## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


I would like to take this up and give it a shot. Any image specific - dataset guidelines to keep in mind ? Thank you.

#self-assign

I've started working on adding this dataset. I require some inputs on the following : 

Ref for the initial draft [here](https://github.com/dnaveenr/datasets/blob/add_metashift_dataset/datasets/metashift/metashift.py)
1. The dataset does not have a typical - train/test/val split. What do we do for the _split_generators() function ? How do we go about this ?
2. This dataset builds on the Visual Genome dataset, using a metadata file. The dataset is generated using generate_full_MetaShift.py script. By default, the authors choose to generate the dataset only for a SELECTED_CLASSES. The following script is used : 
Code : https://github.com/Weixin-Liang/MetaShift/blob/main/dataset/generate_full_MetaShift.py 
Info : https://metashift.readthedocs.io/en/latest/sub_pages/download_MetaShift.html#generate-the-full-metashift-dataset
Can I just copy over the required functions into the metashift.py to generate the dataset ?
3. How do we complete the _generate_examples for this dataset ?

The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Inputs, suggestions would be helpful. Thank you.

I think @mariosasko and @lhoestq should be able to help here 😄 

Hi ! Thanks for adding this dataset :) Let me answer your questions:

1. in this case you can put everything in the "train" split
2. Yes you can copy the script (provided you also include the MIT license of the code in the file header for example). Though we ideally try to not create new directories nor files when generating dataset, so if possible this script should be adapted to not create the file structure they mentioned, but instead yield the images one by one in `_generate_examples`. Let me know if you think this is feasible
3. see point 2 haha

> The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Yup ! We can also define a `selected_classes` parameter such that users can do
```python
load_dataset("metashift", selected_classes=["cat", "dog", ...])
```

Great. This is helpful. Thanks @lhoestq .
Regarding Point 2, I'll try using yield instead of creating the directories and see if its feasible. selected_classes config sounds good.

Closed via #3900
benchmark streaming speed with tar vs zip archives

# do not merge
## Hypothesis 
packing data into a single zip archive could allow us not to care about splitting data into several tar archives for efficient streaming which is annoying (since data creators usually host the data in a single tar)

## Data

I host it [here](https://huggingface.co/datasets/polinaeterna/benchmark_dataset/)

## I checked three configurations:
1. All data in one zip archive, streaming only those files that exist in split metadata file (we can access them directrly with no need to iterate over full archive), see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR196)
2.  All data in three splits, the standart way to make streaming efficient, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR174)
3. All data in single tar, iterate over the full archive and take only files existing in split metadata file, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR150) 

## Results
1. one zip
![image](https://user-images.githubusercontent.com/16348744/156567611-e3652087-7147-4cf0-9047-9cbc00ec71f5.png)
2. three tars
![image](https://user-images.githubusercontent.com/16348744/156567688-2a462107-f83e-4722-8ea3-71a13b56c998.png)
3. one tar
![image](https://user-images.githubusercontent.com/16348744/156567772-1bceb5f7-e7d9-4fa3-b31b-17fec5f9a5a7.png)

didn't check on the full data as it's time consuming but anyway it's pretty obvious that one-zip-way is not a good idea. here it's even worse than full iteration over tar containing all three splits (but that would depend on the case).


I'm closing the PR since we're not going to merge it
Checksums didn't match for datasets on Google Drive

## Describe the bug

Datasets hosted on Google Drive do not seem to work right now.
Loading them fails with a checksum error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
for dataset in ["head_qa", "yelp_review_full"]:
  try:
    load_dataset(dataset)
  except Exception as exception:
    print("Error", dataset, exception)
```

Here is a [colab](https://colab.research.google.com/drive/1wOtHBmL8I65NmUYakzPV5zhVCtHhi7uQ#scrollTo=cDzdCLlk-Bo4).

## Expected results

The datasets should be loaded.

## Actual results

```
Downloading and preparing dataset head_qa/es (download: 75.69 MiB, generated: 2.86 MiB, post-processed: Unknown size, total: 78.55 MiB) to /root/.cache/huggingface/datasets/head_qa/es/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Error head_qa Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
Downloading and preparing dataset yelp_review_full/yelp_review_full (download: 187.06 MiB, generated: 496.94 MiB, post-processed: Unknown size, total: 684.00 MiB) to /root/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/13c31a618ba62568ec8572a222a283dfc29a6517776a3ac5945fb508877dde43...
Error yelp_review_full Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZlU4dXhHTFhZQU0']
```

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Hi @muelletm, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

Until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Pre-Processing Cache Fails when using a Factory pattern

## Describe the bug
If you utilize a pre-processing function which is created using a factory pattern, the function hash changes on each run (even if the function is identical) and therefore the data will be reproduced each time.

## Steps to reproduce the bug
```python
    def preprocess_function_factory(augmentation=None):
        def preprocess_function(examples):
            # Tokenize the texts
            if augmentation:
                conversions1 = [
                    augmentation(example)
                    for example in examples[sentence1_key]
                ]
                if sentence2_key is None:
                    args = (conversions1,)
                else:
                    conversions2 = [
                        augmentation(example)
                        for example in examples[sentence2_key]
                    ]
                    args = (conversions1, conversions2)
            else:
                args = (
                    (examples[sentence1_key],)
                    if sentence2_key is None
                    else (examples[sentence1_key], examples[sentence2_key])
                )
            result = tokenizer(
                *args, padding=padding, max_length=max_seq_length, truncation=True
            )

            # Map labels to IDs (not necessary for GLUE tasks)
            if label_to_id is not None and "label" in examples:
                result["label"] = [
                    (label_to_id[l] if l != -1 else -1) for l in examples["label"]
                ]
            return result

        return preprocess_function

capitalize = lambda x: x.capitalize()
preprocess_function = preprocess_function_factory(augmentation=capitalize)
print(hash(preprocess_function)) # This will change on each run
raw_datasets = raw_datasets.map(
        preprocess_function,
        batched=True,
        load_from_cache_file=True,
        desc="Running transformation and tokenizer on dataset",
)
```

## Expected results
Running the code twice will cause the cache to be re-used.

## Actual results
Running the code twice causes the whole dataset to be re-processed


Ok - this is still an issue but I believe the root cause is different than I originally thought. I'm now able to get caching to work consistently with the above example as long as I fix the python hash seed `export PYTHONHASHSEED=1234`

Hi! 

Yes, our hasher should work with decorators. For instance, this dummy example:
```python
def f(arg):
    def f1(ex):
        return {"a": ex["col1"] + arg}
    return f1
```
gives the same hash across different Python sessions (`datasets.fingerprint.Hasher.hash(f("string1")` returns `"408c9059f89dbd6c"` on my machine).

Could you please make the example self-contained? This way, we can reproduce the bug. Additionally, you can try to find the problematic object yourself by testing their hash with `datasets.fingerprint.Hasher.hash(obj)`

This could be related to https://github.com/huggingface/datasets/issues/3638.

#3638 was indeed my issue. Thanks!
Text builder with custom separator line boundaries

**Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted. 

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None: 
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.

Gently pinging @lhoestq

Hi ! Interresting :)

Could you give more details on what kind of separators you would like to use instead ?

In my case, I just want to use `\n` but not `U+2028`.

Ok I see, maybe there can be a `sep` parameter to allow users to specify what line/paragraph separator they'd like to use

Related to:
- #3729 
- #3910

Thanks for requesting this enhancement. We have recently found a somehow related issue with another dataset:
- #3704

Let me make a PR proposal.
[Breaking] Align `map` when streaming: update instead of overwrite + add missing parameters

Currently the datasets in streaming mode and in non-streaming mode have two distinct API for `map` processing.

In this PR I'm aligning the two by changing `map` in streamign mode. This includes a **major breaking change** and will require a major release of the library: **Datasets 2.0**

In particular, `Dataset.map` adds new columns (with dict.update) BUT `IterableDataset.map` used to discard previous columns (it overwrites the dict). In this PR I'm chaning the `IterableDataset.map` to behave the same way as `Dataset.map`: it will update the examples instead of overwriting them.

I'm also adding those missing parameters to streaming `map`: with_indices, input_columns, remove_columns

### TODO
- [x] tests
- [x] docs

Related to https://github.com/huggingface/datasets/issues/3444

Right ! Will add it in another PR :)
Xtreme-S Metrics

**Added datasets (TODO)**:

- [x] MLS
- [x] Covost2
- [x] Minds-14
- [x] Voxpopuli
- [x] FLoRes (need data)

**Metrics**: Done

@lhoestq - if you could take a final review here this would be great (if you have 5min :-) ) 

Don't think the failures are related but not 100% sure

Yes the CI fail is unrelated - you can ignore it
can not flatten natural_questions dataset

## Describe the bug
after downloading the natural_questions dataset, can not flatten the dataset considering there are `long answer` and `short answer` in `annotations`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset('natural_questions',cache_dir = 'data/dataset_cache_dir')
dataset['train'].flatten()
```

## Expected results
a dataset with `long_answer` as features

## Actual results
Traceback (most recent call last):
  File "temp.py", line 5, in <module>
    dataset['train'].flatten()
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/fingerprint.py", line 413, in wrapper
    out = func(self, *args, **kwargs)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1296, in flatten
    dataset._data = update_metadata_with_features(dataset._data, dataset.features)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in update_metadata_with_features
    features = Features({col_name: features[col_name] for col_name in table.column_names})
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in <dictcomp>
    features = Features({col_name: features[col_name] for col_name in table.column_names})
KeyError: 'annotations.long_answer'
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.13
- Platform: MBP
- Python version: 3.8
- PyArrow version: 6.0.1


same issue. downgrade it to a lower version.

Thanks for reporting, I'll take a look tomorrow :)
Docs new UI actions no self hosted

Removes the need to have a self-hosted runner for the dev documentation

It seems like the doc can't be compiled right now because of the following:

```
Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 33, in <module>
    sys.exit(load_entry_point('doc-builder', 'console_scripts', 'doc-builder')())
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/commands/doc_builder_cli.py", line 39, in main
    args.func(args)
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/commands/build.py", line 95, in build_command
    build_doc(
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/build_doc.py", line 361, in build_doc
    anchors_mapping = build_mdx_files(package, doc_folder, output_dir, page_info)
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/build_doc.py", line 200, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
TypeError: There was an error when converting datasets/docs/source/package_reference/table_classes.mdx to the MDX format.
expected string or bytes-like object
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3793). All of your documentation changes will be reflected on that endpoint.

This is due to the injection of docstrings from PyArrow. I think I can fix that by moving all the docstrings and fix them manually.

> It seems like the doc can't be compiled right now because of the following:

it is expected since there is something I need to change on doc-builder side.

> This is due to the injection of docstrings from PyArrow. I think I can fix that by moving all the docstrings and fix them manually.

@lhoestq I will let you know if we need to change it manually.

@LysandreJik thanks a lot for this PR! I only had one question [here](https://github.com/huggingface/datasets/pull/3793#discussion_r816100194)

> @lhoestq I will let you know if we need to change it manually.

It would be simpler to change it manually anyway - I don't want our documentation to break if PyArrow has documentation issues

For some reason it fails when `Installing node dependencies` when running `npm ci` from the `kit` directory, any idea why @mishig25 ?

Checking it rn

It's very likely linked to an OOM error: https://github.com/huggingface/transformers/pull/15710#issuecomment-1051737337
Add doc builder scripts

I added the three scripts:
- build_dev_documentation.yml
- build_documentation.yml
- delete_dev_documentation.yml

I got them from `transformers` and did a few changes:
- I removed the `transformers`-specific dependencies
- I changed all the paths to be "datasets" instead of "transformers"
- I passed the `--library_name datasets` arg to the `doc-builder build` command (according to https://github.com/huggingface/doc-builder/pull/94/files#diff-bcc33cf7c223511e498776684a9a433810b527a0a38f483b1487e8a42b6575d3R26)

cc @LysandreJik @mishig25 

I think we're only missing the hosted runner to be configured for this repository and we should be good

Regarding the self-hosted runner, I actually encourage using the approach defined here: https://github.com/huggingface/transformers/pull/15710, which doesn't leverage a self-hosted runner. This prevents queuing jobs, which is important when we expect several concurrent jobs.

Opened a PR for that on your branch here: https://github.com/huggingface/datasets/pull/3793
Add URL and ID fields to Wikipedia dataset

This PR adds the URL field, so that we conform to proper attribution, required by their license: provide credit to the authors by including a hyperlink (where possible) or URL to the page or pages you are re-using.

About the conversion from title to URL, I found that apart from replacing blanks with underscores, some other special character must also be percent-encoded (e.g. `"` to `%22`): https://meta.wikimedia.org/wiki/Help:URL

Therefore, I have finally used `urllib.parse.quote` function. This additionally percent-encodes non-ASCII characters, but Wikimedia docs say these are equivalent:
> For the other characters either the code or the character can be used in internal and external links, they are equivalent. The system does a conversion when needed.
> [[%C3%80_propos_de_M%C3%A9ta]]
> is rendered as [À_propos_de_Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), almost like [À propos de Méta](https://meta.wikimedia.org/wiki/%C3%80_propos_de_M%C3%A9ta), which leads to this page on Meta with in the address bar the URL
> [http://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta)
> while [http://meta.wikipedia.org/wiki/À_propos_de_Méta](https://meta.wikipedia.org/wiki/%C3%80_propos_de_M%C3%A9ta) leads to the same. 

Fix #3398.
CC: @geohci

Do you think we have a dedicated branch for all the changes we want to do to wikipedia ? Then once everything looks good + we have preprocessed the main languages, we can merge it on the `master` branch

Yes, @lhoestq, I agree with you.

I have just created the dedicated branch [`update-wikipedia`](https://github.com/huggingface/datasets/tree/update-wikipedia). We can merge every PR (once validated) to that branch; once all changes are merged to that branch, we could create the preprocessed datasets and then merge the branch to master. 

@lhoestq I guess you approve this PR?
Only-data dataset loaded unexpectedly as validation split

## Describe the bug
As reported by @thomasw21 and @lhoestq, a dataset containing only a data file whose name matches the pattern `*dev*` will be returned as VALIDATION split, even if this is not the desired behavior, e.g. a file named `datosdevision.jsonl.gz`.

I see two options:
1. drop the "dev" keyword since it can be considered too generic
2. improve the pattern to something more reasonable, e.g. asking for a separator before and after "dev"
```python
["*[ ._-]dev[ ._-]*", "dev[ ._-]*"]
```

I think 2. is nice. If we agree on this one we can even decide to require the separation for the other split keywords "train", "test" etc.

Yes, I had something like that on mind: "dev" not being part of a word.
```
"[^a-zA-Z]dev[^a-zA-Z]"

Is there a reason why we want that regex? It feels like something that'll still be an issue for some weird case. "my_dataset_dev" doesn't match your regex, "my_dataset_validation" doesn't either ... Why not always "train" unless specified?

The regex is needed as part of our effort to make datasets configurable without code. In particular we define some generic dataset repository structures that users can follow

> ```
> "[^a-zA-Z]*dev[^a-zA-Z]*"
> ```

unfortunately our glob doesn't support "^": 

https://github.com/fsspec/filesystem_spec/blob/3e739db7e53f5b408319dcc9d11e92bc1f938902/fsspec/spec.py#L465-L479

> "my_dataset_dev" doesn't match your regex, "my_dataset_validation" doesn't either ... Why not always "train" unless specified?

And `my_dataset_dev.foo` would match the pattern, and we also have the same pattern but for the "validation" keyword so `my_dataset_validation.foo` would work too

> The regex is needed as part of our effort to make datasets configurable without code

This feels like coding with the filename ^^'

This is still much easier than having to write a full dataset script right ? :p
Fix Google Drive URL to avoid Virus scan warning

This PR fixes, in the datasets library instead of in every specific dataset, the issue of downloading the Virus scan warning page instead of the actual data file for Google Drive URLs.

Fix #3786, fix #3784.

Thanks for this @albertvillanova!

Once this PR merged into master and until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```

Thanks, that solved a bunch of problems we had downstream!
cf. https://github.com/ElementAI/picard/issues/61
Bug downloading Virus scan warning page from Google Drive URLs

## Describe the bug
Recently, some issues were reported with URLs from Google Drive, where we were downloading the Virus scan warning page instead of the data file itself.

See:
- #3758 
- #3773
- #3784


Once the PR merged into master and until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Fix: Bypass Virus Checks in Google Drive Links (CNN-DM dataset)

This commit fixes the issue described in #3784. By adding an extra parameter to the end of Google Drive links, we are able to bypass the virus check and download the datasets.

So, if the original link looked like https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfTHk4NFg2SndKcjQ

The new link now looks like https://drive.google.com/uc?export=download&id=0BwmD_VLjROrfTHk4NFg2SndKcjQ&confirm=t

Fixes #3784

Thank you, @albertvillanova!

Got it. Thanks for explaining this, @albertvillanova!

> On the other hand, the tests are not passing because the dummy data should also be fixed. Once done, this PR will be able to be merged into master.

Will do this 👍

Hi ! I think we need to fix the issue for every dataset. This can be done simply by fixing how we handle Google Drive links, see my comment https://github.com/huggingface/datasets/pull/3775#issuecomment-1050970157

Hi @lhoestq! I think @albertvillanova has already fixed this in #3787

Cool ! I missed this one :) thanks

No problem!

Hi, @AngadSethi, I think that once:
- #3787 

was merged, issue:
- #3784 

was also fixed.

Therefore, I think this PR is no longer necessary. I'm closing it. Let me know if you agree.

Yes, absolutely @albertvillanova! I agree :)
Unable to Download CNN-Dailymail Dataset 

## Describe the bug
I am unable to download the CNN-Dailymail dataset. Upon closer investigation, I realised why this was happening:
- The dataset sits in Google Drive, and both the CNN and DM datasets are large.
- Google is unable to scan the folder for viruses, **so the link which would originally download the dataset, now downloads the source code of this web page:**
![image](https://user-images.githubusercontent.com/58678541/155658435-c2f497d7-7601-4332-94b1-18a62dd96422.png)
- **This leads to the following error**:
```python
NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```


## Steps to reproduce the bug
```python
import datasets
dataset = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")
```

## Expected results
That the dataset is downloaded and processed just like other datasets.

## Actual results
Hit with this error:
```python
NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


#self-assign

@AngadSethi thanks for reporting and thanks for your PR!

Glad to help @albertvillanova! Just fine-tuning the PR, will comment once I am able to get it up and running 😀

Fixed by:
- #3787
Error of writing with different schema, due to nonpreservation of nullability

## 1. Case
```
dataset.map(
  batched=True,
  disable_nullable=True,
)
```
will get the following error at here https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L516

`pyarrow.lib.ArrowInvalid: Tried to write record batch with different schema`

## 2. Debugging
### 2.1 tracing
During `_map_single`, the following are called
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_dataset.py#L2523
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L511
### 2.2. Observation
The problem is, even after `table_cast`, `pa_table.schema != self._schema`
`pa_table.schema` (before/after `table_cast`)
```
input_ids: list<item: int32>
  child 0, item: int32
```
`self._schema`
```
input_ids: list<item: int32> not null
  child 0, item: int32
```
### 2.3. Reason
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1121
Here we lose nullability stored in `schema` because it seems that `Features` is always nullable and don't store nullability.
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1103
So, casting to schema from such `Features` loses nullability, and eventually causes error of writing with different schema

## 3. Solution
1. Let `Features` stores nullability.
2. Directly cast table with original schema but not schema from converted `Features`. (this PR)
3. Don't `cast_table` when `write_table`

Hi ! Thanks for reporting, indeed `disable_nullable` doesn't seem to be supported in this case. Maybe at one point we can have `disable_nullable` as a parameter of certain feature types
Reddit dataset card additions

The changes proposed are based on the "TL;DR: Mining Reddit to Learn Automatic Summarization" paper & https://zenodo.org/record/1043504#.YhaKHpbQC38
It is a Reddit dataset indeed, but the name given to the dataset by the authors is Webis-TLDR-17 (corpus), so perhaps it should be modified as well. 
The task at which the dataset is aimed is abstractive summarization.


Hello! I added the tags and created a PR. Just to note, regarding the paperswithcode_id tag, that currently has the value "reddit"; the dataset described as reddit in paperswithcode is https://paperswithcode.com/dataset/reddit and it isn't the Webis-tldr-17. I could not find Webis-tldr-17 in paperswithcode neither in the Summarization category nor using the keywords reddit, webis, & tldr. I didn't change this tag.
Not be able to download dataset - "Newsroom"

Hello,

I tried to download the **newsroom** dataset but it didn't work out for me. it said me to **download it manually**!

For manually, Link is also didn't work! It is sawing some ad or something!

If anybody has solved this issue please help me out or if somebody has this dataset please share your google drive link, it would be a great help!

Thanks
Darshan Tank

Hi @Darshan2104, thanks for reporting.

Please note that at Hugging Face we do not host the data of this dataset, but just a loading script pointing to the host of the data owners.

Apparently the data owners changed their data host server. After googling it, I found their new website at: https://lil.nlp.cornell.edu/newsroom/index.html
- Download page: https://lil.nlp.cornell.edu/newsroom/download/index.html

I'm fixing the link in our Datasets library.

@albertvillanova Thanks for the solution and link you made my day!
Start removing canonical datasets logic

I updated the source code and the documentation to start removing the "canonical datasets" logic.

Indeed this makes the documentation confusing and we don't want this distinction anymore in the future. Ideally users should share their datasets on the Hub directly.

### Changes

- the documentation about dataset loading mentions the datasets on the Hub (no difference between canonical and community, since they all have their own repository now)
- the documentation about adding a dataset doesn't explain the technical differences between canonical and community anymore, and only presents how to add a community dataset. There is still a small section at the bottom that mentions the datasets that are still on GitHub and redirects to the `ADD_NEW_DATASET.md` guide on GitHub about how to contribute a dataset to the `datasets` library
- the code source doesn't mention "canonical" anymore anywhere. There is still a `GitHubDatasetModuleFactory` class that is left, but I updated the docstring to say that it will be eventually removed in favor of the `HubDatasetModuleFactory` classes that already exist

Would love to have your feedbacks on this !

cc @julien-c @thomwolf @SBrandeis 

I'm not sure if the documentation explains why the dataset identifiers might have a namespace or not (the user/org): 'glue' vs 'severo/glue'. Do you think we should explain it, and relate it to the GitHub/Hub distinction?

> I'm not sure if the documentation explains why the dataset identifiers might have a namespace or not (the user/org): 'glue' vs 'severo/glue'. Do you think we should explain it, and relate it to the GitHub/Hub distinction?

I added an explanation, let me know if it sounds good to you:

```
Datasets used to be hosted on our GitHub repository, but all datasets have now been migrated to the Hugging Face Hub.
The legacy GitHub datasets were added originally on our GitHub repository and therefore don't have a namespace: "squad", "glue", etc. unlike the other datasets that are named "username/dataset_name" or "org/dataset_name".
```


Thanks for the feedbacks ! Merging this now - if you have some comments I can take care of them in a subsequent PR

I'll also take care of resolving the conflicts with https://github.com/huggingface/datasets/pull/3690
Allow download only some files from the Wikipedia dataset

**Is your feature request related to a problem? Please describe.**
The Wikipedia dataset can be really big. This is a problem if you want to use it locally in a laptop with the Apache Beam `DirectRunner`. Even if your laptop have a considerable amount of memory (e.g. 32gb).


**Describe the solution you'd like**
I would like to use the `data_files` argument in the `load_dataset` function to define which file in the wikipedia dataset I would like to download. Thus, I can work with the dataset in a smaller machine using the Apache Beam `DirectRunner`.

**Describe alternatives you've considered**
I've tried to use the `simple` Wikipedia dataset. But it's in English and I would like to use Portuguese texts in my model. 


Hi @jvanz, thank you for your proposal.

In fact, we are aware that it is very common the problem you mention. Because of that, we are currently working in implementing a new version of wikipedia on the Hub, with all data preprocessed (no need to use Apache Beam), from where you will be able to use `data_files` to load only a specific subset of the data files.

See:
- #3401
Update gigaword card and info

Reported on the forum: https://discuss.huggingface.co/t/error-loading-dataset/14999

I think it actually comes from an issue here:

https://github.com/huggingface/datasets/blob/810b12f763f5cf02f2e43565b8890d278b7398cd/src/datasets/utils/file_utils.py#L575-L579

and 

https://github.com/huggingface/datasets/blob/810b12f763f5cf02f2e43565b8890d278b7398cd/src/datasets/utils/streaming_download_manager.py#L386-L389

This code doesn't seem to work anymore. This can probably be fixed with

```python
if url.startswith("https://drive.google.com/"): 
    url += "&confirm=t"
    cookies = response.cookies 
```

because Google Drive doesn't return the `download_warning` cookie anymore.

Actually it seems that is has been fixed already in https://github.com/huggingface/datasets/pull/3787 :)

I think it should have fixed the gigaword dataset loading

@lhoestq The linked PR indeed fixes the issue. This PR is still worth merging IMO to update `gigaword`'s card.
DuplicatedKeysError on msr_sqa dataset

### Describe the bug
Failure to generate dataset msr_sqa because of duplicate keys.

### Steps to reproduce the bug
```
from datasets import load_dataset
load_dataset("msr_sqa")
```
### Expected results
The examples keys should be unique.

**Actual results**
```
>>> load_dataset("msr_sqa")
Downloading:
6.72k/? [00:00<00:00, 148kB/s]
Downloading:
2.93k/? [00:00<00:00, 53.8kB/s]
Using custom data configuration default
Downloading and preparing dataset msr_sqa/default (download: 4.57 MiB, generated: 26.25 MiB, post-processed: Unknown size, total: 30.83 MiB) to /root/.cache/huggingface/datasets/msr_sqa/default/0.0.0/70b2a497bd3cc8fc960a3557d2bad1eac5edde824505e15c9c8ebe4c260fd4d1...
Downloading: 100%
4.80M/4.80M [00:00<00:00, 7.49MB/s]
---------------------------------------------------------------------------
DuplicatedKeysError                       Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split(self, split_generator)
   1080                     example = self.info.features.encode_example(record)
-> 1081                     writer.write(example, key)
   1082             finally:

8 frames
DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: nt-639
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

DuplicatedKeysError                       Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in check_duplicate_keys(self)
    449         for hash, key in self.hkey_record:
    450             if hash in tmp_record:
--> 451                 raise DuplicatedKeysError(key)
    452             else:
    453                 tmp_record.add(hash)

DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: nt-639
Keys should be unique and deterministic in nature
```

### Environment info
datasets version: 1.18.3
Platform: Google colab notebook
Python version: 3.7
PyArrow version: 6.0.1


Thanks for reporting, @kolk.

We are fixing it.
`dataset = dataset.map()` causes faiss index lost

## Describe the bug
assigning the resulted dataset to original dataset causes lost of the faiss index

## Steps to reproduce the bug
`my_dataset` is a regular loaded dataset. It's a part of a customed dataset structure
```python
self.dataset.add_faiss_index('embeddings')
self.dataset.list_indexes()
# ['embeddings']

dataset2 = my_dataset.map(
    lambda x: self._get_nearest_examples_batch(x['text']), batch=True
)

# the unexpected result:
dataset2.list_indexes()
# []

self.dataset.list_indexes()
# ['embeddings']
```

in case something wrong with my `_get_nearest_examples_batch()`, it's like this
```python
def _get_nearest_examples_batch(self, examples, k=5):
    queries = embed(examples)
    scores_batch, retrievals_batch = self.dataset.get_nearest_examples_batch(self.faiss_column, queries, k)
    return {
        'neighbors': [batch['text'] for batch in retrievals_batch],
        'scores': scores_batch
    }
```

## Expected results
`map` shouldn't drop the indexes, in another word, indexes should be carried to the generated dataset

## Actual results
map drops the indexes

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: Ubuntu 20.04.3 LTS
- Python version: 3.8.12
- PyArrow version: 7.0.0


Hi ! Indeed `map` is dropping the index right now, because one can create a dataset with more or fewer rows using `map` (and therefore the index might not be relevant anymore)

I guess we could check the resulting dataset length, and if the user hasn't changed the dataset size we could keep the index, what do you think ?

doing `.add_column("x",x_data)` also removes the index. the new column might be irrelevant to the index so I don't think it should drop. 

Minimal example

```python
from datasets import load_dataset
import numpy as np

data=load_dataset("ceyda/cats_vs_dogs_sample") #just a test dataset
data=data["train"]
embd_data=data.map(lambda x: {"emb":np.random.uniform(-1,0,50).astype(np.float32)})
embd_data.add_faiss_index(column="emb")
print(embd_data.list_indexes())
embd_data=embd_data.add_column("x",[0]*data.num_rows)
print(embd_data.list_indexes())
```

I agree `add_column` shouldn't drop the index indeed ! Is it something you'd like to contribute ? I think it's just a matter of copying the `self._indexes` dictionary to the output dataset