SurAyush/github_issues huggingface dataset:
Issue: Update Wikipedia dataset
State: closed
Author: albertvillanova

This PR combines all updates to Wikipedia dataset.

Once approved, this will be used to generate the pre-processed Wikipedia datasets.

Finally, this PR will be able to be merged into master:
- NOT using squash
- BUT a regular MERGE (or REBASE+MERGE), so that all commits are preserved

TODO:
- [x] #3435
- [x] #3789
- [x] #3825
- [x] Run to get the pre-processed data for big languages (backward compatibility)
- [x] #3958

CC: @geohci

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm starting to generate the pre-processed data for some of the languages (for backward compatibility).

Once this merged, we will create the pre-processed data on the Hub under the Wikimedia namespace.

All steps have been properly done.

I'm merging all these commits into master.
Issue: `pubmed_qa` checksum mismatch
State: closed
Author: jon-tow

## Describe the bug
Loading [`pubmed_qa`](https://huggingface.co/datasets/pubmed_qa) results in a mismatched checksum error.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
import datasets
try:
    datasets.load_dataset("pubmed_qa", "pqa_labeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_unlabeled")
except Exception as e:
    print(e)

try:
    datasets.load_dataset("pubmed_qa", "pqa_artificial")
except Exception as e:
    print(e)
```

## Expected results
Successful download.

## Actual results
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/datasets/load.py", line 1702, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 594, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.9/site-packages/datasets/builder.py", line 665, in _download_and_prepare
    verify_checksums(
  File "/usr/local/lib/python3.9/site-packages/datasets/utils/info_utils.py", line 40, in verify_checksums
    raise NonMatchingChecksumError(error_msg + str(bad_urls))
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=1RsGLINVce-0GsDkCLDuLZmoLuzfmoCuQ', 'https://drive.google.com/uc?export=download&id=15v1x6aQDlZymaHGP7cZJZZYFfeJt2NdS']

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.3
- Platform: macOS
- Python version: 3.8.1
- PyArrow version: 3.0.0


Comments:
Hi @jon-tow, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

We are planning to make a patch release today.

In the meantime, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Issue: Simplify Common Voice code
State: closed
Author: lhoestq

In #3736 we introduced one method to generate examples when streaming, that is different from the one when not streaming.

In this PR I propose a new implementation which is simpler: it only has one function, based on `iter_archive`. And you still have access to local audio files when loading the dataset in non-streaming mode.

cc @patrickvonplaten @polinaeterna @anton-l @albertvillanova since this will become the template for many audio datasets to come.

This change can also trivially be applied to the other audio datasets that already exist. Using this line, you can get access to local files in non-streaming mode:
```python
local_extracted_archive = dl_manager.extract(archive_path) if not dl_manager.is_streaming else None
```

Comments:
I think the script looks pretty clean and readable now! cool!
Issue: Fix iter_archive getting reset
State: closed
Author: lhoestq

The `DownloadManager.iter_archive` method currently returns an iterator - which is  **empty** once you iter over it once. This means you can't pass the same archive iterator to several splits.

To fix that, I changed the ouput of `DownloadManager.iter_archive` to be an iterable that you can iterate over several times, instead of a one-time-use iterator.

The `StreamingDownloadManager.iter_archive` already returns an appropriate iterable, and the code added in this PR is inspired from the one in `streaming_download_manager.py`

Comments:
Issue: Handle Nones in PyArrow struct
State: closed
Author: mariosasko

This PR fixes an issue introduced by #3575 where `None` values stored in PyArrow arrays/structs would get ignored by `cast_storage` or by the `pa.array(cast_to_python_objects(..))` pattern. To fix the former, it also bumps the minimal PyArrow version to v5.0.0 to use the `mask` param in `pa.SturctArray`. 



Comments:
Looks like I added my comments while you were editing - sorry about that
Issue: Add MetaShift dataset
State: closed
Author: osanseviero

## Adding a Dataset
- **Name:** MetaShift
- **Description:**  collection of 12,868 sets of natural images across 410 classes-
- **Paper:** https://arxiv.org/abs/2202.06523v1
- **Data:** https://github.com/weixin-liang/metashift

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments:
I would like to take this up and give it a shot. Any image specific - dataset guidelines to keep in mind ? Thank you.

#self-assign

I've started working on adding this dataset. I require some inputs on the following : 

Ref for the initial draft [here](https://github.com/dnaveenr/datasets/blob/add_metashift_dataset/datasets/metashift/metashift.py)
1. The dataset does not have a typical - train/test/val split. What do we do for the _split_generators() function ? How do we go about this ?
2. This dataset builds on the Visual Genome dataset, using a metadata file. The dataset is generated using generate_full_MetaShift.py script. By default, the authors choose to generate the dataset only for a SELECTED_CLASSES. The following script is used : 
Code : https://github.com/Weixin-Liang/MetaShift/blob/main/dataset/generate_full_MetaShift.py 
Info : https://metashift.readthedocs.io/en/latest/sub_pages/download_MetaShift.html#generate-the-full-metashift-dataset
Can I just copy over the required functions into the metashift.py to generate the dataset ?
3. How do we complete the _generate_examples for this dataset ?

The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Inputs, suggestions would be helpful. Thank you.

I think @mariosasko and @lhoestq should be able to help here 😄 

Hi ! Thanks for adding this dataset :) Let me answer your questions:

1. in this case you can put everything in the "train" split
2. Yes you can copy the script (provided you also include the MIT license of the code in the file header for example). Though we ideally try to not create new directories nor files when generating dataset, so if possible this script should be adapted to not create the file structure they mentioned, but instead yield the images one by one in `_generate_examples`. Let me know if you think this is feasible
3. see point 2 haha

> The user has the ability to use default selected classes, get the complete dataset or add more specific additional classes. I think config would be a good option here.

Yup ! We can also define a `selected_classes` parameter such that users can do
```python
load_dataset("metashift", selected_classes=["cat", "dog", ...])
```

Great. This is helpful. Thanks @lhoestq .
Regarding Point 2, I'll try using yield instead of creating the directories and see if its feasible. selected_classes config sounds good.

Closed via #3900
Issue: benchmark streaming speed with tar vs zip archives
State: closed
Author: polinaeterna

# do not merge
## Hypothesis 
packing data into a single zip archive could allow us not to care about splitting data into several tar archives for efficient streaming which is annoying (since data creators usually host the data in a single tar)

## Data

I host it [here](https://huggingface.co/datasets/polinaeterna/benchmark_dataset/)

## I checked three configurations:
1. All data in one zip archive, streaming only those files that exist in split metadata file (we can access them directrly with no need to iterate over full archive), see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR196)
2.  All data in three splits, the standart way to make streaming efficient, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR174)
3. All data in single tar, iterate over the full archive and take only files existing in split metadata file, see [this func](https://github.com/huggingface/datasets/compare/master...polinaeterna:benchmark-tar-zip?expand=1#diff-4f5200d4586aec5b2a89fcf34441c5f92156f9e9d408acc7e50666f9a1921ddcR150) 

## Results
1. one zip
![image](https://user-images.githubusercontent.com/16348744/156567611-e3652087-7147-4cf0-9047-9cbc00ec71f5.png)
2. three tars
![image](https://user-images.githubusercontent.com/16348744/156567688-2a462107-f83e-4722-8ea3-71a13b56c998.png)
3. one tar
![image](https://user-images.githubusercontent.com/16348744/156567772-1bceb5f7-e7d9-4fa3-b31b-17fec5f9a5a7.png)

didn't check on the full data as it's time consuming but anyway it's pretty obvious that one-zip-way is not a good idea. here it's even worse than full iteration over tar containing all three splits (but that would depend on the case).


Comments:
I'm closing the PR since we're not going to merge it
Issue: Checksums didn't match for datasets on Google Drive
State: closed
Author: muelletm

## Describe the bug

Datasets hosted on Google Drive do not seem to work right now.
Loading them fails with a checksum error.

## Steps to reproduce the bug
```python
from datasets import load_dataset
for dataset in ["head_qa", "yelp_review_full"]:
  try:
    load_dataset(dataset)
  except Exception as exception:
    print("Error", dataset, exception)
```

Here is a [colab](https://colab.research.google.com/drive/1wOtHBmL8I65NmUYakzPV5zhVCtHhi7uQ#scrollTo=cDzdCLlk-Bo4).

## Expected results

The datasets should be loaded.

## Actual results

```
Downloading and preparing dataset head_qa/es (download: 75.69 MiB, generated: 2.86 MiB, post-processed: Unknown size, total: 78.55 MiB) to /root/.cache/huggingface/datasets/head_qa/es/1.1.0/583ab408e8baf54aab378c93715fadc4d8aa51b393e27c3484a877e2ac0278e9...
Error head_qa Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?export=download&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t']
Downloading and preparing dataset yelp_review_full/yelp_review_full (download: 187.06 MiB, generated: 496.94 MiB, post-processed: Unknown size, total: 684.00 MiB) to /root/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/13c31a618ba62568ec8572a222a283dfc29a6517776a3ac5945fb508877dde43...
Error yelp_review_full Checksums didn't match for dataset source files:
['https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZlU4dXhHTFhZQU0']
```

## Environment info
- `datasets` version: 1.18.3
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyArrow version: 6.0.1


Comments:
Hi @muelletm, thanks for reporting.

This issue was already reported and its root cause is a change in the Google Drive service. See:
- #3786 

We have already fixed it. See:
- #3787 

Until our next `datasets` library release, you can get this fix by installing our library from the GitHub master branch:
```shell
pip install git+https://github.com/huggingface/datasets#egg=datasets
```
Then, if you had previously tried to load the data and got the checksum error, you should force the redownload of the data (before the fix, you just downloaded and cached the virus scan warning page, instead of the data file):
```shell
load_dataset("...", download_mode="force_redownload")
```
Issue: Text builder with custom separator line boundaries
State: open
Author: cronoik

**Is your feature request related to a problem? Please describe.**
The current [Text](https://github.com/huggingface/datasets/blob/207be676bffe9d164740a41a883af6125edef135/src/datasets/packaged_modules/text/text.py#L23) builder implementation splits texts with `splitlines()` which splits the text on several line boundaries. Not all of them are always wanted. 

**Describe the solution you'd like**
```python
                if self.config.sample_by == "line":
                    batch_idx = 0
                    while True:
                        batch = f.read(self.config.chunksize)
                        if not batch:
                            break
                        batch += f.readline()  # finish current line
                        if self.config.custom_newline is None: 
                            batch = batch.splitlines(keepends=self.config.keep_linebreaks)
                        else:
                            batch = batch.split(self.config.custom_newline)[:-1]
                        pa_table = pa.Table.from_arrays([pa.array(batch)], schema=schema)
                        # Uncomment for debugging (will print the Arrow table size and elements)
                        # logger.warning(f"pa_table: {pa_table} num rows: {pa_table.num_rows}")
                        # logger.warning('\n'.join(str(pa_table.slice(i, 1).to_pydict()) for i in range(pa_table.num_rows)))
                        yield (file_idx, batch_idx), pa_table
                        batch_idx += 1
```

**A clear and concise description of what you want to happen.**
Creating the dataset rows with a subset of the `splitlines()` line boundaries.

Comments:
Gently pinging @lhoestq

Hi ! Interresting :)

Could you give more details on what kind of separators you would like to use instead ?

In my case, I just want to use `\n` but not `U+2028`.

Ok I see, maybe there can be a `sep` parameter to allow users to specify what line/paragraph separator they'd like to use

Related to:
- #3729 
- #3910

Thanks for requesting this enhancement. We have recently found a somehow related issue with another dataset:
- #3704

Let me make a PR proposal.
Issue: Remove deprecated methods/params (preparation for v2.0)
State: closed
Author: mariosasko

This PR removes the following deprecated methos/params:
* `Dataset.cast_`/`DatasetDict.cast_`
* `Dataset.dictionary_encode_column_`/`DatasetDict.dictionary_encode_column_`
* `Dataset.remove_columns_`/`DatasetDict.remove_columns_`
* `Dataset.rename_columns_`/`DatasetDict.rename_columns_`
* `prepare_module`
* param `script_version` in `load_dataset`/`load_metric`
* param `version` in `hf_github_url`


Comments:
Issue: [Breaking] Align `map` when streaming: update instead of overwrite + add missing parameters
State: closed
Author: lhoestq

Currently the datasets in streaming mode and in non-streaming mode have two distinct API for `map` processing.

In this PR I'm aligning the two by changing `map` in streamign mode. This includes a **major breaking change** and will require a major release of the library: **Datasets 2.0**

In particular, `Dataset.map` adds new columns (with dict.update) BUT `IterableDataset.map` used to discard previous columns (it overwrites the dict). In this PR I'm chaning the `IterableDataset.map` to behave the same way as `Dataset.map`: it will update the examples instead of overwriting them.

I'm also adding those missing parameters to streaming `map`: with_indices, input_columns, remove_columns

### TODO
- [x] tests
- [x] docs

Related to https://github.com/huggingface/datasets/issues/3444

Comments:
Right ! Will add it in another PR :)
Issue: Xtreme-S Metrics
State: closed
Author: patrickvonplaten

**Added datasets (TODO)**:

- [x] MLS
- [x] Covost2
- [x] Minds-14
- [x] Voxpopuli
- [x] FLoRes (need data)

**Metrics**: Done

Comments:
@lhoestq - if you could take a final review here this would be great (if you have 5min :-) ) 

Don't think the failures are related but not 100% sure

Yes the CI fail is unrelated - you can ignore it
Issue: Fix error message in CSV loader for newer Pandas versions
State: closed
Author: mariosasko

Fix the error message in the CSV loader for `Pandas >= 1.4`. To fix this, I directly print the current file name in the for-loop. An alternative would be to use a check similar to this:
```python
csv_file_reader.handle.handle if datasets.config.PANDAS_VERSION >= version.parse("1.4") else csv_file_reader.f
```
CC: @SBrandeis 

Comments:
Issue: can not flatten natural_questions dataset
State: closed
Author: Hannibal046

## Describe the bug
after downloading the natural_questions dataset, can not flatten the dataset considering there are `long answer` and `short answer` in `annotations`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset('natural_questions',cache_dir = 'data/dataset_cache_dir')
dataset['train'].flatten()
```

## Expected results
a dataset with `long_answer` as features

## Actual results
Traceback (most recent call last):
  File "temp.py", line 5, in <module>
    dataset['train'].flatten()
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/fingerprint.py", line 413, in wrapper
    out = func(self, *args, **kwargs)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1296, in flatten
    dataset._data = update_metadata_with_features(dataset._data, dataset.features)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in update_metadata_with_features
    features = Features({col_name: features[col_name] for col_name in table.column_names})
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in <dictcomp>
    features = Features({col_name: features[col_name] for col_name in table.column_names})
KeyError: 'annotations.long_answer'
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.13
- Platform: MBP
- Python version: 3.8
- PyArrow version: 6.0.1


Comments:
same issue. downgrade it to a lower version.

Thanks for reporting, I'll take a look tomorrow :)
Issue: Add Mahalanobis distance metric
State: closed
Author: JoaoLages

Mahalanobis distance is a very useful metric to measure the distance from one datapoint X to a distribution P. 
In this PR I implement the metric in a simple way with the help of numpy only.

Similar to the [MAUVE implementation](https://github.com/huggingface/datasets/blob/master/metrics/mauve/mauve.py), we can make this metric accept texts as input and encode them with a featurize model, if that is desirable.

Comments:
Issue: Docs new UI actions no self hosted
State: closed
Author: LysandreJik

Removes the need to have a self-hosted runner for the dev documentation

Comments:
It seems like the doc can't be compiled right now because of the following:

```
Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 33, in <module>
    sys.exit(load_entry_point('doc-builder', 'console_scripts', 'doc-builder')())
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/commands/doc_builder_cli.py", line 39, in main
    args.func(args)
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/commands/build.py", line 95, in build_command
    build_doc(
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/build_doc.py", line 361, in build_doc
    anchors_mapping = build_mdx_files(package, doc_folder, output_dir, page_info)
  File "/__w/datasets/datasets/doc-builder/src/doc_builder/build_doc.py", line 200, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
TypeError: There was an error when converting datasets/docs/source/package_reference/table_classes.mdx to the MDX format.
expected string or bytes-like object
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3793). All of your documentation changes will be reflected on that endpoint.

This is due to the injection of docstrings from PyArrow. I think I can fix that by moving all the docstrings and fix them manually.

> It seems like the doc can't be compiled right now because of the following:

it is expected since there is something I need to change on doc-builder side.

> This is due to the injection of docstrings from PyArrow. I think I can fix that by moving all the docstrings and fix them manually.

@lhoestq I will let you know if we need to change it manually.

@LysandreJik thanks a lot for this PR! I only had one question [here](https://github.com/huggingface/datasets/pull/3793#discussion_r816100194)

> @lhoestq I will let you know if we need to change it manually.

It would be simpler to change it manually anyway - I don't want our documentation to break if PyArrow has documentation issues

For some reason it fails when `Installing node dependencies` when running `npm ci` from the `kit` directory, any idea why @mishig25 ?

Checking it rn

It's very likely linked to an OOM error: https://github.com/huggingface/transformers/pull/15710#issuecomment-1051737337
Issue: Add `data_dir` to `data_files` resolution and misc improvements to HfFileSystem
State: closed
Author: mariosasko

As discussed in https://github.com/huggingface/datasets/pull/2830#issuecomment-1048989764, this PR adds a QOL improvement to easily reference the files inside a directory in `load_dataset` using the `data_dir` param (very handy for ImageFolder because it avoids globbing, but also useful for the other loaders). Additionally, it fixes the issue with `HfFileSystem.isdir`, which would previously always return `False`, and aligns the path-handling logic in `HfFileSystem` with `fsspec.GitHubFileSystem`.

Comments:
Issue: Add doc builder scripts
State: closed
Author: lhoestq

I added the three scripts:
- build_dev_documentation.yml
- build_documentation.yml
- delete_dev_documentation.yml

I got them from `transformers` and did a few changes:
- I removed the `transformers`-specific dependencies
- I changed all the paths to be "datasets" instead of "transformers"
- I passed the `--library_name datasets` arg to the `doc-builder build` command (according to https://github.com/huggingface/doc-builder/pull/94/files#diff-bcc33cf7c223511e498776684a9a433810b527a0a38f483b1487e8a42b6575d3R26)

cc @LysandreJik @mishig25 

Comments:
I think we're only missing the hosted runner to be configured for this repository and we should be good

Regarding the self-hosted runner, I actually encourage using the approach defined here: https://github.com/huggingface/transformers/pull/15710, which doesn't leverage a self-hosted runner. This prevents queuing jobs, which is important when we expect several concurrent jobs.

Opened a PR for that on your branch here: https://github.com/huggingface/datasets/pull/3793