jamesLeeeeeee/datasets-github-issues huggingface dataset:
Add KLUE-MRC metrics
Author: ingyuseong | Pull request: yes | State: closed

## Metrics for KLUE-MRC (Korean Language Understanding Evaluation — Machine Reading Comprehension)

Adding metrics for [KLUE-MRC](https://huggingface.co/datasets/klue).
KLUE-MRC is very similar to SQuAD 2.0 but has a slightly different format which is why I added metrics for KLUE-MRC.

Specifically, in the case of [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness), it leverages the scoring script of SQuAD to evaluate SQuAD 2.0 and KorQuAD. But the script isn't suitable for KLUE-MRC because KLUE-MRC is a bit different from SQuAD 2.0. And this is why I added the scoring script for KLUE-MRC.

- [x] All tests passed
- [x] Added a metric card (referred the metric card of SQuAD 2.0)
- [x] Compatibility test with [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness) passed

### References
- [KLUE: Korean Language Understanding Evaluation](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/98dce83da57b0395e163467c9dae521b-Paper-round2.pdf)
- [KLUE on Hugging Face Datasets](https://huggingface.co/datasets/klue)
- #2416
Getting a 409 error while loading xglue dataset
Author: Praful932 | Pull request: no | State: closed

### Describe the bug

Unable to load xglue dataset

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.load_dataset("xglue", "ntg")
```

> ConnectionError: Couldn't reach https://xglue.blob.core.windows.net/xglue/xglue_full_dataset.tar.gz (error 409)

### Expected behavior

Expected the dataset to load

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
The current implementation has a potential bug in the sort method
Author: wangyuxinwhy | Pull request: no | State: closed

### Describe the bug

In the sort method，here's a piece of code

```python
# column_names: Union[str, Sequence_[str]]

# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    column_names = [column_names]
```

I get an error when I pass in a tuple based on the column_names type annotation, it will raise an errror.As in the example below, while the type annotation implies that a tuple can be passed.

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

Of course, after I modified the tuple into a list, everything worked fine

Change the code to the following so there will be no problem

```python
# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    if isinstance(column_names, str):
        column_names = [column_names]
    else:
        column_names = list(column_names)
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

### Expected behavior

Passing tuple into column_names should be equivalent to passing list

### Environment info

- `datasets` version: 2.13.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.2
extend the map function so it can wrap around long text that does not fit in the context window
Author: siddhsql | Pull request: no | State: open

### Feature request

I understand `dataset` provides a [`map`](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L2849) function. This function in turn takes in a callable that is used to tokenize the text on which a model is trained. Frequently this text will not fit within a models's context window. In this case it would be useful to wrap around the text into multiple rows with each row fitting the model's context window. I tried to do it using this code as example which in turn I have borrowed from [here](https://stackoverflow.com/a/76343993/147530):

```
data = data.map(lambda samples: tokenizer(samples["text"], max_length=tokenizer.model_max_length, truncation=True, stride=4, return_overflowing_tokens=True), batched=True)
```

but running the code gives me this error:

```
File "/llm/fine-tune.py", line 117, in <module>
    data = data.map(lambda samples: tokenizer(samples["text"], max_length=tokenizer.model_max_length, truncation=True, stride=4, return_overflowing_tokens=True), batched=True)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 580, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 545, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3087, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3480, in _map_single
    writer.write_batch(batch)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_writer.py", line 556, in write_batch
    pa_table = pa.Table.from_arrays(arrays, schema=schema)
  File "pyarrow/table.pxi", line 3798, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 2962, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 394 but got length 447
```

The lambda function I have provided is correctly chopping up long text so it wraps around (and because of this 394 samples become 447 after wrap around) but the dataset `map` function does not like it.

### Motivation

please see above

### Your contribution

I'm afraid I don't have much knowledge to help
ValueError: Table schema does not match schema used to create file
Author: exs-avianello | Pull request: no | State: closed

### Describe the bug

Saving a dataset as parquet fails with a `ValueError: Table schema does not match schema used to create file`  if the dataset was obtained out of a `.select_columns()` call with columns selected out of order.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_parquet("demo.parquet")
```

```shell
>>>
ValueError: Table schema does not match schema used to create file: 
table:
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53 vs. 
file:
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53
```

--- 

I think this is because after the `.select_columns()` call with out of order columns, the output dataset features' schema ends up being out of sync with the schema of the arrow table backing it. 

```python
ds.features.arrow_schema
>>>
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53

ds.data.schema
>>>
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53
```


So when we call `.to_parquet()`, the call behind the scenes to `datasets.io.parquet.ParquetDatasetWriter(...).write()` which initialises the backend `pyarrow.parquet.ParquetWriter` with `schema = self.dataset.features.arrow_schema` triggers `pyarrow` on write when [it checks](https://github.com/apache/arrow/blob/11b140a734a516e436adaddaeb35d23f30dcce44/python/pyarrow/parquet/core.py#L1086-L1090) that the `ParquetWriter` schema matches the schema of the table being written 🙌 

https://github.com/huggingface/datasets/blob/6ed837325cb539a5deb99129e5ad181d0269e050/src/datasets/io/parquet.py#L139-L141


### Expected behavior

The dataset gets successfully saved as parquet. 

*In the same way as it does if saving it as csv:

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_csv("demo.csv")
```

### Environment info

`python==3.11`
`datasets==2.13.1`
`map` with any joblib backend
Author: lhoestq | Pull request: no | State: open

We recently enabled the (experimental) parallel backend switch for data download and extraction but not for `map` yet.

Right now we're using our `iflatmap_unordered` implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have `iflatmap_unordered` for joblib and therefore a `map` with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).
ConnectionError: Couldn't reach dataset_infos.json 
Author: yulingao | Pull request: no | State: closed

### Describe the bug

I'm trying to load codeparrot/codeparrot-clean-train, but get the following error:

ConnectionError: Couldn't reach https://huggingface.co/datasets/codeparrot/codeparrot-clean-train/resolve/main/dataset_infos.json (ConnectionError(ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))))



### Steps to reproduce the bug

train_data = load_dataset('codeparrot/codeparrot-clean-train', split='train')


### Expected behavior

download the dataset

### Environment info

centos7
Why max_shard_size is not supported in load_dataset and passed to download_and_prepare
Author: npuichigo | Pull request: no | State: closed

### Describe the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

What I can to is break the `load_dataset` and use `load_datset_builder` + `download_and_prepare` instead.

### Steps to reproduce the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

### Expected behavior

Users can define the max shard size.

### Environment info

datasets==2.13.1
AutoSharding IterableDataset's when num_workers > 1
Author: mathephysicist | Pull request: no | State: open

### Feature request


Minimal Example

```
import torch
from datasets import IterableDataset

d = IterableDataset.from_file(<file_name>)
dl = torch.utils.data.dataloader.DataLoader(d,num_workers=3)

for sample in dl:
    print(sample)

```

Warning:
Too many dataloader workers: 2 (max is dataset.n_shards=1). Stopping 1 dataloader workers.
To parallelize data loading, we give each process some shards (or data sources) to process. Therefore it's unnecessary to have a number of workers greater than dataset.n_shards=1. To enable more parallelism, please split the dataset in more files than 1.

Expected Behavior:
Dataset is sharded each cpu uses subset (contiguously - so you can do checkpoint loading/saving) 

### Motivation

I have a lot of unused cpu's and would like to be able to shard iterable datasets with pytorch's dataloader when num_workers > 1. This is for a very large single file. I am aware that we can use the `split_dataset_by_node` to ensure that each node (for distributed) gets different shards, but we should extend it so that this also continues for multiple workers. 

### Your contribution

If someone points me to what needs to change, I can create a PR.
404 on Datasets Documentation Page
Author: kmulka-bloomberg | Pull request: no | State: closed

### Describe the bug

Getting a 404 from the Hugging Face Datasets docs page:
https://huggingface.co/docs/datasets/index


### Steps to reproduce the bug

1. Go to URL https://huggingface.co/docs/datasets/index
2. Notice 404 not found

### Expected behavior

URL should either show docs or redirect to new location

### Environment info

hugginface.co
Only two cores are getting used in sagemaker with pytorch 3.10 kernel
Author: mmr-crexi | Pull request: no | State: closed

### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel
Viewing dataset card returns “502 Bad Gateway”
Author: tbenthompson | Pull request: no | State: closed

The url is: https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams

I am able to successfully view the “Files and versions” tab: [Confirm-Labs/pile_ngrams_trigrams at main](https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams/tree/main)

Any help would be appreciated! Thanks! I hope this is the right place to report an issue like this.
Avoid stuck map operation when subprocesses crashes
Author: pappacena | Pull request: yes | State: closed

I've been using Dataset.map() with `num_proc=os.cpu_count()` to leverage multicore processing for my datasets, but from time to time I get stuck processes waiting forever. Apparently, when one of the subprocesses is abruptly killed (OOM killer, segfault, SIGKILL, etc), the main process keeps waiting for the async task sent to that child process to finish.

It seems to be easy to reproduce the issue with the following script:

```
import os
from datasets import Dataset, Features, Value


def do_stuck(item):
    os.kill(os.getpid(), 9)

data = {
    "col1": list(range(5)),
    "col2": list(range(5)),
}

ds = Dataset.from_dict(
    data,
    features=Features({
        "col1": Value("int64"),
        "col2": Value("int64"),
    }),
)

print(ds.map(do_stuck, num_proc=4))
```

This is an old behavior in Python, which apparently was fixed a few years ago in `concurrent.futures.ProcessPoolExecutor` ([ref](https://bugs.python.org/issue9205)), but not in `multiprocessing.pool.Pool` / `multiprocess.pool.Pool`, which is used by `Dataset.map` ([ref](https://bugs.python.org/issue22393)).

This PR is an idea to try to detect when a child process gets killed, and raises a `RuntimeError` warning the dataset.map() caller.

EDIT: Related proposal for future improvement: https://github.com/huggingface/datasets/discussions/5977
Streaming Dataset behind Proxy - FileNotFoundError
Author: Veluchs | Pull request: no | State: closed

### Describe the bug

When trying to stream a dataset i get the following error after a few minutes of waiting.

```
FileNotFoundError: https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json
If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

I have already set the proxy environment variables. Downloading a Dataset without streaming works as expected.
Still i suspect that this is connected to being behind a proxy.

Is there a way to set the proxy for streaming datasets? Possibly a keyword argument that gets passed to ffspec?

### Steps to reproduce the bug

This is the code i use.

```
import os
os.environ['http_proxy'] = "http://example.com:xxxx" 
os.environ['https_proxy'] = "http://example.com:xxxx" 


from datasets import load_dataset

ds = load_dataset("facebook/voxpopuli", name="de", streaming=True)
```

### Expected behavior

I would expect the streaming functionality to use the set proxy settings.

### Environment info


- `datasets` version: 2.13.0
- Platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.2
Filter unsupported extensions
Author: lhoestq | Pull request: yes | State: closed

I used a regex to filter the data files based on their extension for packaged builders.

I tried and a regex is 10x faster that using `in` to check if the extension is in the list of supported extensions.

Supersedes https://github.com/huggingface/datasets/pull/5850

Close https://github.com/huggingface/datasets/issues/5849

I also did a small change to favor the parquet module in case of a draw in the extension counter.
Common Voice datasets still need `use_auth_token=True`
Author: patrickvonplaten | Pull request: no | State: closed

### Describe the bug

We don't need to pass `use_auth_token=True` anymore to download gated datasets or models, so the following should work if correctly logged in.

```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

However it throws an error - probably because something weird is hardcoded into the dataset loading script.

### Steps to reproduce the bug

1.) 
```
huggingface-cli login
```

2.) Make sure that you have accepted the license here:
https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1

3.) Run:
```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

4.)  You'll get:

```
File ~/hf/lib/python3.10/site-packages/datasets/builder.py:963, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
    961 split_dict = SplitDict(dataset_name=self.name)
    962 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 963 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    965 # Checksums verification
    966 if verification_mode == VerificationMode.ALL_CHECKS and dl_manager.record_checksums:

File ~/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_1/f4d7854c466f5bd4908988dbd39044ec4fc634d89e0515ab0c51715c0127ffe3/common_voice_6_1.py:150, in CommonVoice._split_generators(self, dl_manager)
    148 hf_auth_token = dl_manager.download_config.use_auth_token
    149 if hf_auth_token is None:
--> 150     raise ConnectionError(
    151         "Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset"
    152     )
    154 bundle_url_template = STATS["bundleURLTemplate"]
    155 bundle_version = bundle_url_template.split("/")[0]

ConnectionError: Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset
```

### Expected behavior

One should not have to pass `use_auth_token=True`. Also see discussion here: https://github.com/huggingface/blog/pull/1243#discussion_r1235131150

### Environment info

```
- `datasets` version: 2.13.0
- Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.0.dev0
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
```
Config name / split name lost after map with multiproc
Author: sanchit-gandhi | Pull request: no | State: open

### Describe the bug

Performing a `.map` method on a dataset loses it's config name / split name only if run with multiproc

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor
import numpy as np

# load dummy dataset
libri = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean")

# make train / test splits
libri = libri["validation"].train_test_split(seed=42, shuffle=True, test_size=0.1)

# example feature extractor
model_id = "ntu-spml/distilhubert"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True, return_attention_mask=True)

sampling_rate = feature_extractor.sampling_rate

libri = libri.cast_column("audio", Audio(sampling_rate=sampling_rate))

max_duration = 30.0

def preprocess_function(examples):
    audio_arrays = [x["array"] for x in examples["audio"]]
    inputs = feature_extractor(
        audio_arrays,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=int(feature_extractor.sampling_rate * max_duration),
        truncation=True,
        return_attention_mask=True,
    )
    return inputs

# single proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=1
)

print(10 * "=" ,"Single processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)

# multi proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=2
)

print(10 * "=" ,"Multi processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)
```

**Print Output:**
```
========== Single processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  clean  Split name after:  validation
========== Multi processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  None  Split name after:  None
```

=> we can see that the config/split names are lost in the multiprocessing setting




### Expected behavior

Should retain both config / split names in the multiproc setting

### Environment info

- `datasets` version: 2.13.1.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2