noamaanMulla-03/datasets-issues huggingface dataset:
Protect the main branch from inadvertent direct pushes

We have decided to implement a protection mechanism in this repository, so that nobody (not even administrators) can inadvertently push accidentally directly to the main branch.

See context here:
- d7c942228b8dcf4de64b00a3053dce59b335f618

To do:
- [x] Protect main branch
  - Settings > Branches > Branch protection rules > main > Edit
    - [x] Check: Do not allow bypassing the above settings 
      - The above settings will apply to administrators and custom roles with the "bypass branch protections" permission.
    - [x] Additionally, uncheck: Require approvals [under "Require a pull request before merging", which was already checked]
      - Before, we could exceptionally merge a non-approved PR, using Administrator bypass
      - Now that Administrator bypass is no longer possible, we would always need an approval to be able to merge; and pull request authors cannot approve their own pull requests. This could be an inconvenient in some exceptional circumstances when an urgent fix is needed
      - Nevertheless, although it is no longer enforced, it is strongly recommended to merge PRs only if they have at least one approval
- [x] #5250
  - So that direct pushes to main branch are no longer necessary

Labels: name: maintenance

It seems all the tasks have been addressed, meaning this issue can be closed, no?
The order of the splits is not preserved

We had a regression and the order of the splits is not preserved. They are alphabetically sorted, instead of preserving original "train", "validation", "test" order.

Check: In branch "main"
```python
In [9]: dataset = load_dataset("adversarial_qa", '"adversarialQA")

In [10]: dataset
Out[10]: 
DatasetDict({
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

Before (2.15.0) it was:
```python
DatasetDict({
    train: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 30000
    })
    validation: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
    test: Dataset({
        features: ['id', 'title', 'context', 'question', 'answers', 'metadata'],
        num_rows: 3000
    })
})
```

See issues: 
- https://huggingface.co/datasets/adversarial_qa/discussions/3
- https://huggingface.co/datasets/beans/discussions/4

This is a regression because it was previously fixed. See:
- #6196
- #5728

Labels: name: bug

After investigation, I think the issue was introduced by the use of the Parquet export:
- #6448

I am proposing a fix.

CC: @lhoestq
Impossible to save a mapped dataset to disk

### Describe the bug

I want to play around with different hyperparameters when training but don't want to re-map my dataset with 3 million samples each time for tens of hours when I [fully fine-tune SDXL](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py).

After I do the mapping like this:
```
train_dataset = train_dataset.map(compute_embeddings_fn, batched=True)
train_dataset = train_dataset.map(
    compute_vae_encodings_fn,
    batched=True,
    batch_size=16,
)
```
and try to save it like this:
`train_dataset.save_to_disk("test")`
i get this error ([full traceback](https://pastebin.com/kq3vt739)):
```
TypeError: Object of type function is not JSON serializable
The format kwargs must be JSON serializable, but key 'transform' isn't.
```

But what is interesting is that pushing to hub works like that:
`train_dataset.push_to_hub("kopyl/mapped-833-icons-sdxl-1024-dataset", token=True)`
Here is the link of the pushed dataset: https://huggingface.co/datasets/kopyl/mapped-833-icons-sdxl-1024-dataset

### Steps to reproduce the bug

Here is the self-contained notebook:

https://colab.research.google.com/drive/1RtCsEMVcwWcMwlWURk_cj_9xUBHz065M?usp=sharing

### Expected behavior

It should be easily saved to disk

### Environment info

NVIDIA A100, Linux (NC24ads A100 v4 from Azure), CUDA 12.2.

[pip freeze](https://pastebin.com/QTNb6iru)


I solved it with `train_dataset.with_format(None)`
But then faced some more issues (which i later solved too).

Huggingface does not seem to care, so I do. Here is an updated training script which saves a pre-processed (mapped) dataset to your local directory if you specify `--save_precomputed_data_dir=DIR_NAME`. Then use `--train_precomputed_data_dir` with the same dir to load it instead of `--dataset_name`.

[Proper SDXL trainer code](https://github.com/kopyl/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py)
[Notebook for pre-computing a dataset and saving locally](https://colab.research.google.com/drive/17Yo08hePx-NlHs99RecdeiNc8CQg4O7l?usp=sharing)

Example:

1st run (nothing is pre-computed yet):
```
accelerate launch train_text_to_image_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --dataset_name=lambdalabs/pokemon-blip-captions \
    --save_precomputed_data_dir="test-5"
```

2nd run (the pre-computed dataset is saved to `test-5` directory):
```
accelerate launch train_text_to_image_sdxl.py \
    --pretrained_model_name_or_path=stabilityai/stable-diffusion-xl-base-1.0 \
    --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix \
    --train_precomputed_data_dir test-5
```

This way when you're gonna be using your pre-computed dataset you don't need to worry about re-mapping your dataset when you change an argument for your trainer script
Adding support for netCDF (*.nc) files

### Feature request

netCDF (*.nc) is a file format for storing multidimensional scientific data, which is used by packages like `xarray` (labelled multi-dimensional arrays in Python). It would be nice to have native support for netCDF in `datasets`.

### Motivation

When uploading *.nc files onto Huggingface Hub through the `datasets` API, I would like to be able to preview the dataset without converting it to another format.

### Your contribution

I can submit a PR, provided I have the time.

Labels: name: enhancement

Related to #3113 

Conceptually, we can use xarray to load the netCDF file, then xarray -> pandas -> pyarrow.

I'd still need to verify that such a conversion would be lossless, especially for multi-dimensional data.
'Repo card metadata block was not found' when loading a pragmeval dataset

### Describe the bug

I can't load dataset subsets of 'pragmeval'.

The funny thing is I ran the dataset author's [colab notebook](https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing) and it works just fine. I tried to install exactly the same packages that are installed on colab using poetry, so my environment info only differs from the one from colab in linux version - I still get the same bug outside colab.

### Steps to reproduce the bug

Install dependencies with poetry

pyproject.toml
```
[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
datasets = "2.16.0"
pandas = "1.5.3"
pyarrow = "10.0.1"
huggingface-hub = "0.19.4"
fsspec = "2023.6.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

`poetry run python -c "import datasets; print(datasets.get_dataset_config_names('pragmeval'))`

prints ['default']

### Expected behavior

The command should print

```
['emergent',
 'emobank-arousal',
 'emobank-dominance',
 'emobank-valence',
 'gum',
 'mrda',
 'pdtb',
 'persuasiveness-claimtype',
 'persuasiveness-eloquence',
 'persuasiveness-premisetype',
 'persuasiveness-relevance',
 'persuasiveness-specificity',
 'persuasiveness-strength',
 'sarcasm',
 'squinky-formality',
 'squinky-implicature',
 'squinky-informativeness',
 'stac',
 'switchboard',
 'verifiability']
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Extreme inefficiency for `save_to_disk` when merging datasets

### Describe the bug

Hi, I tried to merge in total 22M sequences of data, where each sequence is of maximum length 2000. I found that merging these datasets and then `save_to_disk` is extremely slow because of flattening the indices. Wondering if you have any suggestions or guidance on this. Thank you very much! 

### Steps to reproduce the bug

The source data is too big to demonstrate

### Expected behavior

The source data is too big to demonstrate

### Environment info

python                     3.9.0
datasets                  2.7.0
pytorch                   2.0.0 
tokenizers                0.13.1
transformers              4.31.0


Concatenating datasets doesn't create any indices mapping - so flattening indices is not needed (unless you shuffle the dataset).
Can you share the snippet of code you are using to merge your datasets and save them to disk ?
ted_talks_iwslt | Error: Config name is missing

### Describe the bug

Running load_dataset using the newest `datasets` library like below on the ted_talks_iwslt using year pair data will throw an error "Config name is missing"

see also:
https://huggingface.co/datasets/ted_talks_iwslt/discussions/3

likely caused by #6493, where the `and not config_kwargs` part in the if logic was removed 
 https://github.com/huggingface/datasets/blob/ef3b5dd3633995c95d77f35fb17f89ff44990bc4/src/datasets/builder.py#L512

### Steps to reproduce the bug

run: 
```python
load_dataset("ted_talks_iwslt", language_pair=("ja", "en"), year="2015")
```

### Expected behavior

Load the data without error

### Environment info

datasets 2.16.0



Hi ! Thanks for reporting. I opened https://github.com/huggingface/datasets/pull/6544 to fix this

We just released 2.16.1 with a fix:

```
pip install -U datasets
```
Skip if a dataset has issues

### Describe the bug

Hello everyone,
I'm using **load_datasets** from **huggingface** to download the datasets and I'm facing an issue, the download starts but it reaches some state and then  fails with the following error:
Couldn't reach https://huggingface.co/datasets/wikimedia/wikipedia/resolve/4cb9b0d719291f1a10f96f67d609c5d442980dc9/20231101.ext/train-00000-of-00001.parquet

Failed to resolve \'huggingface.co\' ([Errno -3] Temporary failure in name resolution)"))')))


![image](https://github.com/huggingface/datasets/assets/143214684/8847d9cb-529e-4eda-9c76-282713dfa3af)

so I was wondering is there a parameter to be passed to load_dataset() to skip files that can't be downloaded??

### Steps to reproduce the bug

Parameter to be passed to load_dataset() of huggingface to skip files that can't be downloaded??

### Expected behavior

load_dataset() finishes without error

### Environment info

None


It looks like a transient DNS issue. It should work fine now if you try again.

There is no parameter in load_dataset to skip failed downloads. In your case it would have skipped every single subsequent download until the DNS issue was resolved anyway.
Datasets : wikipedia 20220301.en error 

### Describe the bug

When I used load_dataset to download this data set, the following error occurred. The main problem was that the target data did not exist.

### Steps to reproduce the bug

1.I tried downloading directly.
```python
wiki_dataset = load_dataset("wikipedia", "20220301.en")
```
An exception occurred
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`
```
2.I modified the code as prompted.
```python
wiki_dataset = load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')
```
An exception occurred:
```
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
```


### Expected behavior

I searched in the parent directory of the corresponding URL, but there was no corresponding "20220301" directory.
I really need this data set and hope to provide a download method.

### Environment info

python 3.8
datasets 2.16.0
apache-beam 2.52.0
dill 0.3.7



Hi ! We now recommend using the `wikimedia/wikipedia` dataset, can you try loading this one instead ?

```python
wiki_dataset  = load_dataset("wikimedia/wikipedia", "20231101.en")
```

This bug has been fixed in `2.16.1` thanks to https://github.com/huggingface/datasets/pull/6544, feel free to update `datasets` and re-run your code :)

```
pip install -U datasets
```
Loading from hf hub with clearer error message

### Feature request

Shouldn't this kinda work ?
```
Dataset.from_json("hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json")
```
I got an error
```
File ~/miniconda3/envs/datatrove/lib/python3.10/site-packages/datasets/data_files.py:380, in resolve_pattern(pattern, base_path, allowed_extensions, download_config)
    378     if allowed_extensions is not None:
    379         error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 380     raise FileNotFoundError(error_msg)
    381 return out

FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
(I'm logged in)
```

Fix: the correct path is
```
hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

Proposal: raise a clearer error

### Motivation

Clearer error message

### Your contribution

Can open a PR

Labels: name: enhancement

Maybe we can add a helper message like `Maybe try again using "hf://path/without/resolve"` if the path contains `/resolve/` ?

e.g.

```
FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
It looks like you used parts of the URL of the file from the Hugging Face website, but you should remove the "/resolve/<revision>" part to have a valid `hf://` path.
Please try again using this path instead:
    hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

and suggest `f"hf://datasets/HuggingFaceTB/eval_data@{revision}/eval_data_context_and_answers.json"` if revision != "main"

EDIT: I think this message should also be raised from the  `huggingface_hub`'s `HfFileSystem` implementation
datasets.DownloadMode.FORCE_REDOWNLOAD use cache to download dataset features with load_dataset function

### Describe the bug

I have updated my dataset by adding a new feature, and push it to the hub. When I want to download it on my machine which contain the old version by using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` I get an error (paste bellow).

Seems that the load_dataset function still use the old features schema instead of downloading everything new from the HUB.

I find a way to go around this issue by manually deleting the old dataset cache. But from my understanding of `datasets.DownloadMode.FORCE_REDOWNLOAD` option, the dataset cache should be ignored.

### Steps to reproduce the bug

1. Download your dataset in your machine using `datasets.load_dataset`
2. Create a new feature in your dataset and push it to the hub
3. On the same machine redownload your dataset using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)`

### Expected behavior

`
ValueError: Couldn't cast
id: string
level: string
context: list<element: string>
  child 0, element: string
type: string
answer: string
question: string
supporting_facts: list<element: string>
  child 0, element: string
fra_answer: string
fra_question: string
-- schema metadata --
huggingface: '{"info": {"features": {"id": {"dtype": "string", "_type": "' + 490
to
{'id': Value(dtype='string', id=None), 'level': Value(dtype='string', id=None), 'context': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'type': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'supporting_facts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

DatasetGenerationError
...
DatasetGenerationError: An error occurred while generating the dataset`

### Environment info

datasets-2.16.1 huggingface-hub-0.20.2
Metadata for each column

### Feature request

Being able to put some metadata for each column as a string or any other type.


### Motivation

I will bring the motivation by an example, lets say we are experimenting with embedding produced by some image encoder network, and we want to iterate through a couple of preprocessing and see which one works better in our downstream task, here as workaround right now what I do is the compute the hash of the preprocessing that the images went through as part of the new columns name, it would be nice to attach some kinda meta data in these scenarios to the each columns. metadata

### Your contribution

Maybe we could map another relational like database as the metadata? 

Labels: name: enhancement

Hi! Indeed it would be useful to support this. PyArrow natively supports schema-level and column-level metadata, so implementing this should be straightforward. The API I have in mind would work as follows:
```python
col_feature = Value("string", metadata="Some column-level metadata")

features = Features({"col": col_feature}, metadata="Some schema-level metadata")
```

WDYT?

Sorry for the late reply, 
Yes, I think this is the most straight-forward approach with the things that we already have.



@mariosasko Let me know how I can help.

Hi, is this feature to be implemented in the near future? It would be really nice if that would be the case! 

Hi, I also need this feature for tell my customer if any of the feature is encrypted with a certain key.
medical-dialog error

## Describe the bug
A clear and concise description of what the bug is.
When I attempt to download the huggingface datatset medical_dialog it errors out midway through
## Steps to reproduce the bug
```python
raw_datasets = load_dataset("medical_dialog", "en", split="train", download_mode="force_redownload", data_dir="./Medical-Dialogue-Dataset-English")
```

## Expected results
A clear and concise description of the expected results.
No error
## Actual results

```
3 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py in verify_splits(expected_splits, recorded_splits)
     72     ]
     73     if len(bad_splits) > 0:
---> 74         raise NonMatchingSplitsSizesError(str(bad_splits))
     75     logger.info("All the splits matched successfully.")
     76 

NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='medical_dialog'), 'recorded': SplitInfo(name='train', num_bytes=295097913, num_examples=229674, dataset_name='medical_dialog')}]
```
Specify the actual results or traceback.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.21.1
- Platform: colab
- Python version: colab 3.7
- PyArrow version: N/A


Labels: name: bug

Hi @smeyerhot, thanks for reporting.

You are right: there is an issue with the dataset metadata. I'm fixing it.

In the meantime, you can circumvent the issue by passing `ignore_verifications=True`:
```python
raw_datasets = load_dataset("medical_dialog", "en", split="train", download_mode="force_redownload", data_dir="./Medical-Dialogue-Dataset-English", ignore_verifications=True)
```

Hi @albertvillanova -- I see you made changes to the wikitext dataset a few hours ago.
I'm getting an error similar to the one here : https://github.com/huggingface/datasets/issues/215
After your changes. 

I deleted the original dataset cache after encountering that error and then rerun with the following command :

`traindata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train', ignore_verifications=True, download_mode='force_redownload')
    testdata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='test', ignore_verifications=True)`

But this is currently hanging.  Any ideas ? Thanks

@ldery, you can find the answer in the corresponding Discussion on the Hub: https://huggingface.co/datasets/wikitext/discussions/9

You need to update the `datasets` library:
```shell
pip install -U datasets
```
Loading a dataset from Google Colab hangs at "Resolving data files".

### Describe the bug

Hello,

I'm trying to load a dataset from Google Colab but the process hangs at `Resolving data files`:

![image](https://github.com/huggingface/datasets/assets/99779/7175ad85-e571-46ed-9f87-92653985777d)

It is happening when the `_get_origin_metadata` definition is invoked:

```python
def _get_origin_metadata(
    data_files: List[str],
    max_workers=64,
    download_config: Optional[DownloadConfig] = None,
) -> Tuple[str]:
    return thread_map(
        partial(_get_single_origin_metadata, download_config=download_config),
        data_files,
        max_workers=max_workers,
        tqdm_class=hf_tqdm,
        desc="Resolving data files",
        disable=len(data_files) <= 16,
```

The thread is then stuck at `waiter.acquire()` in the builtin `threading.py` file.

I can load the dataset just fine on my machine.

Cheers,

Thomas


### Steps to reproduce the bug

In Google Colab:

```python
!pip install datasets
from datasets import load_dataset

dataset = load_dataset("colour-science/color-checker-detection-dataset")
```

### Expected behavior

The dataset should be loaded.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.1
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0


This bug comes from the `huggingface_hub` library, see: https://github.com/huggingface/huggingface_hub/issues/1952

A fix is provided at https://github.com/huggingface/huggingface_hub/pull/1953. Feel free to install `huggingface_hub` from this PR, or wait for it to be merged and the new version of `huggingface_hub` to be released

Thanks!