AniketTec/github-issues huggingface dataset:
Title: Add KLUE-MRC metrics
State: closed

## Metrics for KLUE-MRC (Korean Language Understanding Evaluation — Machine Reading Comprehension)

Adding metrics for [KLUE-MRC](https://huggingface.co/datasets/klue).
KLUE-MRC is very similar to SQuAD 2.0 but has a slightly different format which is why I added metrics for KLUE-MRC.

Specifically, in the case of [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness), it leverages the scoring script of SQuAD to evaluate SQuAD 2.0 and KorQuAD. But the script isn't suitable for KLUE-MRC because KLUE-MRC is a bit different from SQuAD 2.0. And this is why I added the scoring script for KLUE-MRC.

- [x] All tests passed
- [x] Added a metric card (referred the metric card of SQuAD 2.0)
- [x] Compatibility test with [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness) passed

### References
- [KLUE: Korean Language Understanding Evaluation](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/98dce83da57b0395e163467c9dae521b-Paper-round2.pdf)
- [KLUE on Hugging Face Datasets](https://huggingface.co/datasets/klue)
- #2416

Comments:
The metrics API in `datasets` is deprecated as of version 2.0, and `evaulate` is our new library for metrics. You can add a new metric to it by following [these steps](https://huggingface.co/docs/evaluate/creating_and_sharing).
Title: Getting a 409 error while loading xglue dataset
State: closed

### Describe the bug

Unable to load xglue dataset

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.load_dataset("xglue", "ntg")
```

> ConnectionError: Couldn't reach https://xglue.blob.core.windows.net/xglue/xglue_full_dataset.tar.gz (error 409)

### Expected behavior

Expected the dataset to load

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Thanks for reporting, @Praful932.

Let's continue the conversation on the Hub: https://huggingface.co/datasets/xglue/discussions/5
Title: The current implementation has a potential bug in the sort method
State: closed

### Describe the bug

In the sort method，here's a piece of code

```python
# column_names: Union[str, Sequence_[str]]

# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    column_names = [column_names]
```

I get an error when I pass in a tuple based on the column_names type annotation, it will raise an errror.As in the example below, while the type annotation implies that a tuple can be passed.

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

Of course, after I modified the tuple into a list, everything worked fine

Change the code to the following so there will be no problem

```python
# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    if isinstance(column_names, str):
        column_names = [column_names]
    else:
        column_names = list(column_names)
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

### Expected behavior

Passing tuple into column_names should be equivalent to passing list

### Environment info

- `datasets` version: 2.13.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.2

Comments:
Thanks for reporting, @wangyuxinwhy.
Title: ValueError: Table schema does not match schema used to create file
State: closed

### Describe the bug

Saving a dataset as parquet fails with a `ValueError: Table schema does not match schema used to create file`  if the dataset was obtained out of a `.select_columns()` call with columns selected out of order.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_parquet("demo.parquet")
```

```shell
>>>
ValueError: Table schema does not match schema used to create file: 
table:
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53 vs. 
file:
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53
```

--- 

I think this is because after the `.select_columns()` call with out of order columns, the output dataset features' schema ends up being out of sync with the schema of the arrow table backing it. 

```python
ds.features.arrow_schema
>>>
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53

ds.data.schema
>>>
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53
```


So when we call `.to_parquet()`, the call behind the scenes to `datasets.io.parquet.ParquetDatasetWriter(...).write()` which initialises the backend `pyarrow.parquet.ParquetWriter` with `schema = self.dataset.features.arrow_schema` triggers `pyarrow` on write when [it checks](https://github.com/apache/arrow/blob/11b140a734a516e436adaddaeb35d23f30dcce44/python/pyarrow/parquet/core.py#L1086-L1090) that the `ParquetWriter` schema matches the schema of the table being written 🙌 

https://github.com/huggingface/datasets/blob/6ed837325cb539a5deb99129e5ad181d0269e050/src/datasets/io/parquet.py#L139-L141


### Expected behavior

The dataset gets successfully saved as parquet. 

*In the same way as it does if saving it as csv:

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_csv("demo.csv")
```

### Environment info

`python==3.11`
`datasets==2.13.1`


Comments:
We'll do a new release of `datasets` soon to make the fix available :)

In the meantime you can use `datasets` from source (main)

Thank you very much @lhoestq ! 🚀
Title: `map` with any joblib backend
State: open
Labels: description: New feature or request
name: enhancement

We recently enabled the (experimental) parallel backend switch for data download and extraction but not for `map` yet.

Right now we're using our `iflatmap_unordered` implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have `iflatmap_unordered` for joblib and therefore a `map` with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).

Comments:
#self-assign

Hi @lhoestq 👋🏼

I’d like to work on this!

Planning to support progress tracking with `map()` using any joblib backend (like "loky") by replacing the Queue-based approach in `iflatmap_unordered` with a file-based progress tracking mechanism (e.g. shared temp file with periodic updates).

This would allow the progress bar to work even in backends where inter-process Queues aren't supported. Let me know if this sounds good — I’ll get started!


I think ideally it should have a general solution, since some joblib backends don't have a shared filesystem
Title: Set a rule on the config and split names
State: open

> should we actually allow characters like spaces? maybe it's better to add validation for whitespace symbols and directly in datasets and raise

https://github.com/huggingface/datasets-server/issues/853


Comments:
in this case we need to decide what to do with the existing datasets with white space characters (there shouldn't be a lot of them I think)

I imagine that we should stop supporting them, and help the user fix them?

See a report where the datasets server fails: https://huggingface.co/datasets/poloclub/diffusiondb/discussions/2#6374ff55b93cbdf65675f564

The config name is `random_10k [2m]`!
Title: ConnectionError: Couldn't reach dataset_infos.json 
State: closed

### Describe the bug

I'm trying to load codeparrot/codeparrot-clean-train, but get the following error:

ConnectionError: Couldn't reach https://huggingface.co/datasets/codeparrot/codeparrot-clean-train/resolve/main/dataset_infos.json (ConnectionError(ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))))



### Steps to reproduce the bug

train_data = load_dataset('codeparrot/codeparrot-clean-train', split='train')


### Expected behavior

download the dataset

### Environment info

centos7

Comments:
Unfortunately, I can't reproduce the error. What does the following code return for you?
```python
import requests
from huggingface_hub import hf_hub_url
r = requests.get(hf_hub_url("codeparrot/codeparrot-clean-train", "dataset_infos.json", repo_type="dataset"))
```

Also, can you provide more info about your network (region, proxies, etc.)?
Title: Why max_shard_size is not supported in load_dataset and passed to download_and_prepare
State: closed

### Describe the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

What I can to is break the `load_dataset` and use `load_datset_builder` + `download_and_prepare` instead.

### Steps to reproduce the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

### Expected behavior

Users can define the max shard size.

### Environment info

datasets==2.13.1

Comments:
Can you explain your use case for `max_shard_size`? 

On some systems, there is a limit to the size of a memory-mapped file, so we could consider exposing this parameter in `load_dataset`.

In my use case, users may choose a proper size to balance the cost and benefit of using large shard size. (On azure blob or hdfs which may automatically download the shard from background)

But `load_dataset` doesn't support caching (and reading) Arrow datasets from remote storage. 

`load_datset_builder` + `download_and_prepare` is not equal to `load_dataset`. The latter has one more step, `builder.as_dataset`, that memory-maps Arrow files, which only works for local files.

Thanks. So if I want to use `IterableDataset` and control the size of single arrow file, how should I organize the data loader? Maybe `load_dataset_build` + `download_and_prepare` + `builder.as_dataset` + `dataset.to_iterable_dataset`?

Yes, this should work.

I think we can expose `max_shard_size` in `load_dataset`, so feel free to open a PR.
Title: 404 on Datasets Documentation Page
State: closed

### Describe the bug

Getting a 404 from the Hugging Face Datasets docs page:
https://huggingface.co/docs/datasets/index


### Steps to reproduce the bug

1. Go to URL https://huggingface.co/docs/datasets/index
2. Notice 404 not found

### Expected behavior

URL should either show docs or redirect to new location

### Environment info

hugginface.co

Comments:
This wasn’t working for me a bit earlier, but it looks to be back up now

We had a minor issue updating the docs after the latest release. It should work now :).
Title: Only two cores are getting used in sagemaker with pytorch 3.10 kernel
State: closed

### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel

Comments:
I think it's more likely that this issue is related to PyTorch than Datasets, as PyTorch (on import) registers functions to execute when forking a process. Maybe this is the culprit: https://github.com/pytorch/pytorch/issues/99625

From reading that ticket, it may be down in mkl?  Is it worth hotfixing in the meantime, with the express intention of turning it off?  I know that's a horribly crufty solution, but it's also deeply frustrating to be limited to 2 cores for operations as simple as filtration.

This is too specific and unrelated to `datasets`, so this shouldn't be fixed here.

@mariosasko @mmr-crexi  I had the exact same problem on my kubernetes cluster. the datasets subprocess only user 1 and 17 core
Title: Viewing dataset card returns “502 Bad Gateway”
State: closed

The url is: https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams

I am able to successfully view the “Files and versions” tab: [Confirm-Labs/pile_ngrams_trigrams at main](https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams/tree/main)

Any help would be appreciated! Thanks! I hope this is the right place to report an issue like this.


Comments:
Can you try again? Maybe there was a minor outage.

Yes, it seems to be working now. In case it's helpful, the outage lasted several days. It was failing as late as yesterday morning. 

we fixed something on the server side, glad it's fixed now
Title: Docs: make "repository structure" easier to find
State: open
Labels: description: Improvements or additions to documentation
name: documentation

The page https://huggingface.co/docs/datasets/repository_structure explains how to create a simple repository structure without a dataset script.
It's the simplest way to create a dataset and should be easier to find, particularly on the docs' first pages.

Comments:
Loading a local dataset also works the same way when `data_files` are not specified, so I agree we should make this info  easier to discover 

cc @stevhliu 

Is this issue open? If so, I will self assign. 

@benjaminbrown038 Yes, it is. Maybe @stevhliu can give some pointers on improving this doc page's discoverability.

I think we can add a version of the [Main use-case](https://huggingface.co/docs/datasets/repository_structure#main-usecase) section to the [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset) tutorial. 

Currently, it doesn't tell you *how* to structure the repository; it only tells you how to create it. So adding the "main use-case" will help bridge the gap and make it easier to find. We should also add a link to the [Structure your repository](https://huggingface.co/docs/datasets/repository_structure) guide for users who want to learn about the other options.

#self-assign
Title: Issue with train_test_split maintaining the same underlying PyArrow Table
State: open

### Describe the bug

I've been using the train_test_split method in the datasets module to split my HuggingFace Dataset into separate training, validation, and testing subsets. However, I've noticed an issue where the split datasets appear to maintain the same underlying PyArrow Table.

### Steps to reproduce the bug

1. Load any dataset  ```dataset = load_dataset("lhoestq/demo1")``` 
2. Try the next code:
```python
from datasets import Dataset, DatasetDict

train_size = 0.6

split_train = dataset["train"].train_test_split(
    train_size=train_size,
)

separate_dataset_dict = DatasetDict({
    "train": split_train["train"],
    "test": split_train["test"],
})
```
3.  The next code ```print(separate_dataset_dict)``` when printing the dataset it gives the indication that they have 3 and 2 rows respectively.
4. But the next code: 
 ```python
print(len(separate_dataset_dict["train"].data['id']))
print(len(separate_dataset_dict["test"].data['id'])) 
```

 Indicates that both tables still have 5 rows.

### Expected behavior

However, I've noticed that train_test_split["train"].data, test_val_split["train"].data, and test_val_split["test"].data are identical, suggesting that they all point to the same underlying PyArrow Table. This means that the split datasets are not independent, as I expected.

I believe this is a bug in the train_test_split implementation, as I would expect this function to return datasets with separate underlying PyArrow Tables. Could you please help me understand if this is expected behavior, or if there's a workaround to create truly independent split datasets?

I would appreciate any assistance with this issue. Thank you.

### Environment info

I tried in Colab:

- `datasets` version: 2.13.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

and my PC:

- `datasets` version: 2.13.0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3

Comments:
Title: read metric glue.py from local file 
State: closed

### Describe the bug

Currently, The server is off-line.  I am using the glue metric from the local file downloaded from the hub. 
I download / cached datasets using `load_dataset('glue','sst2', cache_dir='/xxx')` to cache them and then in the off-line mode, I use `load_dataset('xxx/glue.py','sst2', cache_dir='/xxx')`. I can successfully reuse cached datasets.

My problem is about the load_metric. 
When I run `load_dataset('xxx/glue_metric.py','sst2',cache_dir='/xxx')` , it returns 

`  File "xx/lib64/python3.9/site-packages/datasets/utils/deprecation_utils.py", line 46, in wrapper
    return deprecated_function(*args, **kwargs)
  File "xx//lib64/python3.9/site-packages/datasets/load.py", line 1392, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable`

Thanks in advance for help! 
### Steps to reproduce the bug

N/A

### Expected behavior

N/A

### Environment info

`datasets == 2.12.0`

Comments:
Sorry, I solve this by call `evaluate.load('glue_metric.py','sst-2')`
Title: What is the Right way to use discofuse dataset??
State: closed

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)
**Below is the following way, as per my understanding , Is it correct :question: :question:**

The **columns/features from `DiscoFuse dataset`** that will be the **input to the `encoder` and `decoder`** are:

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

1. **coherent_first_sentence**

2. **coherent_second_sentence**

3. **incoherent_first_sentence**

4. **incoherent_second_sentence**

[Click here for Dataset link](https://huggingface.co/datasets/discofuse/viewer/discofuse-wikipedia/train?row=6)

The **`encoder` will take these four columns as input and encode them into a sequence of hidden states. The `decoder` will then take these hidden states as input and decode them into a new sentence that fuses the two original sentences together.**

The **discourse type, connective_string, has_coref_type_pronoun, and has_coref_type_nominal columns will not be used as input to the encoder or decoder.** These columns are used to provide additional information about the dataset, but they are not necessary for the task of sentence fusion.

Please correct me if I am wrong; otherwise, if this understanding is right, how shall I implement this task practically?

Comments:
Thanks for opening https://huggingface.co/datasets/discofuse/discussions/3, let's continue the discussion over there if you don't mind

I have posted there also sir, please check
@lhoestq
Title: Return the audio filename when decoding fails due to corrupt files
State: open
Labels: description: New feature or request
name: enhancement

### Feature request

Return the audio filename when the audio decoding fails. Although currently there are some checks for mp3 and opus formats with the library version there are still cases when the audio decoding could fail, eg. Corrupt file. 

### Motivation

When you try to load an object file dataset and the decoding fails you can't know which file is corrupt
```

raise LibsndfileError(err, prefix="Error opening {0!r}: ".format(self.name))
soundfile.LibsndfileError: Error opening <_io.BytesIO object at 0x7f5ab7e38290>: Format not recognised.
```

### Your contribution

Make a PR to Add exceptions for LIbsndfileError to return the audio filename or path when soundfile decoding fails.

Comments:
Hi ! The audio data don't always exist as files on disk - the blobs are often stored in the Arrow files. For now I'd suggest disabling decoding with `.cast_column("audio", Audio(decode=False))` and apply your own decoding that handles corrupted files (maybe to filter them out ?)

cc @sanchit-gandhi since it's related to our discussion about allowing users to make decoding return `None` and show a warning when there are corrupted files

Thanks @lhoestq, I wasn't aware of the decode flag. It makes more sense as you say to show a warning when there are corrupted files together with some metadata of the file that allows to filter them from the dataset.

My workaround was to catch the LibsndfileError and generate a dummy audio with an unsual sample rate to filter it later. However returning `None` seems better. 

`try:
    array, sampling_rate = sf.read(file)
except sf.LibsndfileError:
    print("bad file")
    array = np.array([0.0])
    sampling_rate = 99.000`