SurAyush/github_issues huggingface dataset:
Issue: keep_in_memory=True does not seem to work
State: open
Author: kopyl

UPD: [Fixed](https://github.com/huggingface/datasets/issues/6568#issuecomment-1880817794) . But a new issue came up :(

Comments:
Seems like I just used the old code which did not have `keep_in_memory=True` argument, sorry.

Although i encountered a different problem – at 97% my python process just hung for around 11 minutes with no logs (when running dataset.map without `keep_in_memory=True` over around 3 million of dataset samples)...

Can you open a new issue and provide a bit more details ? What kind of map operations did you run ?

Hey. I will try to find some free time to describe it.

(can't do it now, cause i need to reproduce it myself to be sure about everything, which requires spinning a new Azuree VM, copying a huge dataset to drive from network disk for a long time etc...)

@lhoestq loading dataset like this does not spawn 50 python processes:

```
datasets.load_dataset("/preprocessed_2256k/train", num_proc=50)
```

I have 64 vCPU so i hoped it could speed up the dataset loading...

My dataset onlly has images and metadata.csv with text column alongside image file path column

now noticed
```
'Setting num_proc from 50 back to 1 for the train split to disable multiprocessing as it only contains one shard
```

Any way to work around this?

@lhoestq thanks, [this helped](https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/arrow_dataset.py#L1053)
Issue: AttributeError: 'str' object has no attribute 'to'
State: closed
Author: andysingal

### Describe the bug

```
--------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-6-80c6086794e8>](https://localhost:8080/#) in <cell line: 10>()
      8 		  report_to="wandb")
      9 
---> 10 trainer = Trainer(
     11     model=model,
     12     args=training_args,

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _move_model_to_device(self, model, device)
    688 
    689     def _move_model_to_device(self, model, device):
--> 690         model = model.to(device)
    691         # Moving a model to an XLA device disconnects the tied weights, so we have to retie them.
    692         if self.args.parallel_mode == ParallelMode.TPU and hasattr(model, "tie_weights"):

AttributeError: 'str' object has no attribute 'to'
```

### Steps to reproduce the bug

here is the notebook:
```
https://colab.research.google.com/drive/10JDBNsLlYrQdnI2FWfDK3F5M8wvVUDXG?usp=sharing

```


### Expected behavior

run the Training 

### Environment info

Colab Notebook , T4 

Comments:
I think you are reporting an issue with the `transformers` library. Note this is the repository of the `datasets` library. I recommend that you open an issue in their repository: https://github.com/huggingface/transformers/issues

EDIT: I have not the rights to transfer the issue
~~I am transferring your issue to their repository.~~

Thanks, I hope someone from transformers library addresses this issue.

On Mon, Jan 8, 2024 at 15:29 Albert Villanova del Moral <
***@***.***> wrote:

> I think you are reporting an issue with the transformers library. Note
> this is the repository of the datasets library. I am transferring your
> issue to their repository.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/6567#issuecomment-1880688586>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE4LJNOYMD6WJMXFKPMH6DLYNO7PJAVCNFSM6AAAAABBQ63HWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBQGY4DQNJYGY>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


@andysingal, I recommend that you open an issue in their repository: https://github.com/huggingface/transformers/issues
I don't have the rights to transfer this issue to their repo.
Issue:  `drop_last_batch=True` for IterableDataset map function is ignored with multiprocessing DataLoader 
State: closed
Author: naba89

### Describe the bug

Scenario:
- Interleaving two iterable datasets of unequal lengths (`all_exhausted`), followed by a batch mapping with batch size 2 to effectively merge the two datasets and get a sample from each dataset in a single batch, with `drop_last_batch=True` to skip the last batch in case it doesn't have two samples.

What works:
- Using DataLoader with `num_workers=0`

What does not work:
- Using DataLoader with `num_workers=1`, errors in the last batch.

Basically, `drop_last_batch=True` is ignored when using multiple dataloading workers.

Please take a look at the minimal repro script below.

### Steps to reproduce the bug

```python
from datasets import Dataset, interleave_datasets
from torch.utils.data import DataLoader


def merge_samples(batch):
    assert len(batch['a']) == 2, "Batch size must be 2"
    batch['c'] = [batch['a'][0]]
    batch['d'] = [batch['a'][1]]
    return batch


def gen1():
    for ii in range(1, 8385):
        yield {"a": ii}


def gen2():
    for ii in range(1, 5302):
        yield {"a": ii}


if __name__ == '__main__':

    dataset1 = Dataset.from_generator(gen1).to_iterable_dataset(num_shards=1024)
    dataset2 = Dataset.from_generator(gen2).to_iterable_dataset(num_shards=1024)

    interleaved = interleave_datasets([dataset1, dataset2], stopping_strategy="all_exhausted")
    mapped = interleaved.map(merge_samples, batched=True, batch_size=2, remove_columns=interleaved.column_names,
                             drop_last_batch=True)

    # Works
    loader = DataLoader(mapped, batch_size=32, num_workers=0)
    i = 0
    for b in loader:
        print(i, b['c'].shape, b['d'].shape)
        i += 1

    print("DataLoader with num_workers=0 works")

    # Doesn't work
    loader = DataLoader(mapped, batch_size=32, num_workers=1)
    i = 0
    for b in loader:
        print(i, b['c'].shape, b['d'].shape)
        i += 1


```

### Expected behavior

 `drop_last_batch=True` should have same behaviour for `num_workers=0` and `num_workers>=1`

### Environment info

- `datasets` version: 2.16.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0

I have also tested on Linux and got the same behavior.

Comments:
My current workaround this issue is to return `None` in the second element and then filter out samples which have `None` in  them.

```python
def merge_samples(batch):
    if len(batch['a']) == 1:
        batch['c'] = [batch['a'][0]]
        batch['d'] = [None]
    else:
        batch['c'] = [batch['a'][0]]
        batch['d'] = [batch['a'][1]]
    return batch
    
def filter_fn(x):
    return x['d'] is not None

# other code...
mapped = mapped.filter(filter_fn)
```
Issue: `Dataset.filter` missing `with_rank` parameter
State: closed
Author: kopyl

### Describe the bug

The issue shall be open: https://github.com/huggingface/datasets/issues/6435

When i try to pass `with_rank` to `Dataset.filter()`, i get this:

`Dataset.filter() got an unexpected keyword argument 'with_rank'`

### Steps to reproduce the bug

Run notebook:

https://colab.research.google.com/drive/1WUNKph8BdP0on5ve3gQnh_PE0cFLQqTn?usp=sharing

### Expected behavior

Should work?

### Environment info

NVIDIA RTX 4090

Comments:
Thanks for reporting! I've opened a PR with a fix

@mariosasko thank you very much :)
Issue: datasets.DownloadMode.FORCE_REDOWNLOAD use cache to download dataset features with load_dataset function
State: open
Author: LsTam91

### Describe the bug

I have updated my dataset by adding a new feature, and push it to the hub. When I want to download it on my machine which contain the old version by using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` I get an error (paste bellow).

Seems that the load_dataset function still use the old features schema instead of downloading everything new from the HUB.

I find a way to go around this issue by manually deleting the old dataset cache. But from my understanding of `datasets.DownloadMode.FORCE_REDOWNLOAD` option, the dataset cache should be ignored.

### Steps to reproduce the bug

1. Download your dataset in your machine using `datasets.load_dataset`
2. Create a new feature in your dataset and push it to the hub
3. On the same machine redownload your dataset using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)`

### Expected behavior

`
ValueError: Couldn't cast
id: string
level: string
context: list<element: string>
  child 0, element: string
type: string
answer: string
question: string
supporting_facts: list<element: string>
  child 0, element: string
fra_answer: string
fra_question: string
-- schema metadata --
huggingface: '{"info": {"features": {"id": {"dtype": "string", "_type": "' + 490
to
{'id': Value(dtype='string', id=None), 'level': Value(dtype='string', id=None), 'context': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'type': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'supporting_facts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

DatasetGenerationError
...
DatasetGenerationError: An error occurred while generating the dataset`

### Environment info

datasets-2.16.1 huggingface-hub-0.20.2

Comments:
Issue: Document YAML configuration with "data_dir"
State: open
Author: severo

See https://huggingface.co/datasets/uonlp/CulturaX/discussions/15#6597e83f185db94370d6bf50 for reference

Comments:
In particular, I would like to have an example of how to replace the following configuration (from https://huggingface.co/docs/hub/datasets-manual-configuration#splits)

```
---
configs:
- config_name: default
  data_files:
  - split: train
    path: "data/*.csv"
  - split: test
    path: "holdout/*.csv"
---
```

with the `data_dir` field.
Issue: Fix imagefolder with one image
State: closed
Author: lhoestq

A dataset repository with one image and one metadata file was considered a JSON dataset instead of an ImageFolder dataset. This is because we pick the dataset type with the most compatible data file extensions present in the repository and it results in a tie in this case.

e.g. for https://huggingface.co/datasets/multimodalart/repro_1_image

I fixed this by deprioritizing metadata files in the count.

fix https://github.com/huggingface/datasets/issues/6545

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6556). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Fixed in dataset viewer: https://huggingface.co/datasets/multimodalart/repro_1_image

<img width="682" alt="Capture d’écran 2024-02-12 à 22 57 08" src="https://github.com/huggingface/datasets/assets/1676121/be9a8dbc-2d78-4ffc-aed4-293a7c57bc0d">
Issue: Cannot import name 'load_dataset' from .... module ‘datasets’
State: closed
Author: ciaoyizhen

### Describe the bug


use python -m pip install datasets to install

### Steps to reproduce the bug

from datasets import load_dataset

### Expected behavior

it doesn't work

### Environment info

datasets version==2.15.0
python == 3.10.12
linux  version I don't know??

Comments:
I don't know  My conpany conputer cannot work. but in my computer, it work?

Do you have a folder in your working directory called datasets?
Issue: Loading a dataset from Google Colab hangs at "Resolving data files".
State: closed
Author: KelSolaar

### Describe the bug

Hello,

I'm trying to load a dataset from Google Colab but the process hangs at `Resolving data files`:

![image](https://github.com/huggingface/datasets/assets/99779/7175ad85-e571-46ed-9f87-92653985777d)

It is happening when the `_get_origin_metadata` definition is invoked:

```python
def _get_origin_metadata(
    data_files: List[str],
    max_workers=64,
    download_config: Optional[DownloadConfig] = None,
) -> Tuple[str]:
    return thread_map(
        partial(_get_single_origin_metadata, download_config=download_config),
        data_files,
        max_workers=max_workers,
        tqdm_class=hf_tqdm,
        desc="Resolving data files",
        disable=len(data_files) <= 16,
```

The thread is then stuck at `waiter.acquire()` in the builtin `threading.py` file.

I can load the dataset just fine on my machine.

Cheers,

Thomas


### Steps to reproduce the bug

In Google Colab:

```python
!pip install datasets
from datasets import load_dataset

dataset = load_dataset("colour-science/color-checker-detection-dataset")
```

### Expected behavior

The dataset should be loaded.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.1
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0

Comments:
This bug comes from the `huggingface_hub` library, see: https://github.com/huggingface/huggingface_hub/issues/1952

A fix is provided at https://github.com/huggingface/huggingface_hub/pull/1953. Feel free to install `huggingface_hub` from this PR, or wait for it to be merged and the new version of `huggingface_hub` to be released

Thanks!
Issue: Loading from hf hub with clearer error message
State: open
Author: thomwolf

### Feature request

Shouldn't this kinda work ?
```
Dataset.from_json("hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json")
```
I got an error
```
File ~/miniconda3/envs/datatrove/lib/python3.10/site-packages/datasets/data_files.py:380, in resolve_pattern(pattern, base_path, allowed_extensions, download_config)
    378     if allowed_extensions is not None:
    379         error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 380     raise FileNotFoundError(error_msg)
    381 return out

FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
(I'm logged in)
```

Fix: the correct path is
```
hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

Proposal: raise a clearer error

### Motivation

Clearer error message

### Your contribution

Can open a PR

Comments:
Maybe we can add a helper message like `Maybe try again using "hf://path/without/resolve"` if the path contains `/resolve/` ?

e.g.

```
FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
It looks like you used parts of the URL of the file from the Hugging Face website, but you should remove the "/resolve/<revision>" part to have a valid `hf://` path.
Please try again using this path instead:
    hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

and suggest `f"hf://datasets/HuggingFaceTB/eval_data@{revision}/eval_data_context_and_answers.json"` if revision != "main"

EDIT: I think this message should also be raised from the  `huggingface_hub`'s `HfFileSystem` implementation
Issue: Skip if a dataset has issues
State: open
Author: hadianasliwa

### Describe the bug

Hello everyone,
I'm using **load_datasets** from **huggingface** to download the datasets and I'm facing an issue, the download starts but it reaches some state and then  fails with the following error:
Couldn't reach https://huggingface.co/datasets/wikimedia/wikipedia/resolve/4cb9b0d719291f1a10f96f67d609c5d442980dc9/20231101.ext/train-00000-of-00001.parquet

Failed to resolve \'huggingface.co\' ([Errno -3] Temporary failure in name resolution)"))')))


![image](https://github.com/huggingface/datasets/assets/143214684/8847d9cb-529e-4eda-9c76-282713dfa3af)

so I was wondering is there a parameter to be passed to load_dataset() to skip files that can't be downloaded??

### Steps to reproduce the bug

Parameter to be passed to load_dataset() of huggingface to skip files that can't be downloaded??

### Expected behavior

load_dataset() finishes without error

### Environment info

None

Comments:
It looks like a transient DNS issue. It should work fine now if you try again.

There is no parameter in load_dataset to skip failed downloads. In your case it would have skipped every single subsequent download until the DNS issue was resolved anyway.
Issue: `image` column not automatically inferred if image dataset only contains 1 image
State: closed
Author: apolinario

### Describe the bug

By default, the standard Image Dataset maps out `file_name` to `image` when loading an Image Dataset. 

However, if the dataset contains only 1 image, this does not take place 

### Steps to reproduce the bug

Input
(dataset with one image `multimodalart/repro_1_image`) 
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_1_image")
dataset
```

Output: 
```py
DatasetDict({
    train: Dataset({
        features: ['file_name', 'prompt'],
        num_rows: 1
    })
})
```

Input 
(dataset with 2+ images `multimodalart/repro_2_image`) 
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_2_image")
dataset
```

Output:
```py
DatasetDict({
    train: Dataset({
        features: ['image', 'prompt'],
        num_rows: 2
    })
})
```

### Expected behavior

Expected to map `file_name`  → `image` for all dataset sizes, including 1. 

### Environment info

Both latest main and 2.16.0

Comments:
Issue: Datasets : wikipedia 20220301.en error 
State: closed
Author: ppx666

### Describe the bug

When I used load_dataset to download this data set, the following error occurred. The main problem was that the target data did not exist.

### Steps to reproduce the bug

1.I tried downloading directly.
```python
wiki_dataset = load_dataset("wikipedia", "20220301.en")
```
An exception occurred
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`
```
2.I modified the code as prompted.
```python
wiki_dataset = load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')
```
An exception occurred:
```
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
```


### Expected behavior

I searched in the parent directory of the corresponding URL, but there was no corresponding "20220301" directory.
I really need this data set and hope to provide a download method.

### Environment info

python 3.8
datasets 2.16.0
apache-beam 2.52.0
dill 0.3.7


Comments:
Hi ! We now recommend using the `wikimedia/wikipedia` dataset, can you try loading this one instead ?

```python
wiki_dataset  = load_dataset("wikimedia/wikipedia", "20231101.en")
```

This bug has been fixed in `2.16.1` thanks to https://github.com/huggingface/datasets/pull/6544, feel free to update `datasets` and re-run your code :)

```
pip install -U datasets
```
Issue: Extreme inefficiency for `save_to_disk` when merging datasets
State: open
Author: KatarinaYuan

### Describe the bug

Hi, I tried to merge in total 22M sequences of data, where each sequence is of maximum length 2000. I found that merging these datasets and then `save_to_disk` is extremely slow because of flattening the indices. Wondering if you have any suggestions or guidance on this. Thank you very much! 

### Steps to reproduce the bug

The source data is too big to demonstrate

### Expected behavior

The source data is too big to demonstrate

### Environment info

python                     3.9.0
datasets                  2.7.0
pytorch                   2.0.0 
tokenizers                0.13.1
transformers              4.31.0

Comments:
Concatenating datasets doesn't create any indices mapping - so flattening indices is not needed (unless you shuffle the dataset).
Can you share the snippet of code you are using to merge your datasets and save them to disk ?
Issue: 'Repo card metadata block was not found' when loading a pragmeval dataset
State: open
Author: lambdaofgod

### Describe the bug

I can't load dataset subsets of 'pragmeval'.

The funny thing is I ran the dataset author's [colab notebook](https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing) and it works just fine. I tried to install exactly the same packages that are installed on colab using poetry, so my environment info only differs from the one from colab in linux version - I still get the same bug outside colab.

### Steps to reproduce the bug

Install dependencies with poetry

pyproject.toml
```
[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
datasets = "2.16.0"
pandas = "1.5.3"
pyarrow = "10.0.1"
huggingface-hub = "0.19.4"
fsspec = "2023.6.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

`poetry run python -c "import datasets; print(datasets.get_dataset_config_names('pragmeval'))`

prints ['default']

### Expected behavior

The command should print

```
['emergent',
 'emobank-arousal',
 'emobank-dominance',
 'emobank-valence',
 'gum',
 'mrda',
 'pdtb',
 'persuasiveness-claimtype',
 'persuasiveness-eloquence',
 'persuasiveness-premisetype',
 'persuasiveness-relevance',
 'persuasiveness-specificity',
 'persuasiveness-strength',
 'sarcasm',
 'squinky-formality',
 'squinky-implicature',
 'squinky-informativeness',
 'stac',
 'switchboard',
 'verifiability']
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0

Comments:
Issue: Adding support for netCDF (*.nc) files
State: open
Author: shermansiu

### Feature request

netCDF (*.nc) is a file format for storing multidimensional scientific data, which is used by packages like `xarray` (labelled multi-dimensional arrays in Python). It would be nice to have native support for netCDF in `datasets`.

### Motivation

When uploading *.nc files onto Huggingface Hub through the `datasets` API, I would like to be able to preview the dataset without converting it to another format.

### Your contribution

I can submit a PR, provided I have the time.

Comments:
Related to #3113 

Conceptually, we can use xarray to load the netCDF file, then xarray -> pandas -> pyarrow.

I'd still need to verify that such a conversion would be lossless, especially for multi-dimensional data.