jamesLeeeeeee/datasets-github-issues huggingface dataset:
AttributeError: 'str' object has no attribute 'to'
Author: andysingal | Pull request: no | State: closed

### Describe the bug

```
--------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-6-80c6086794e8>](https://localhost:8080/#) in <cell line: 10>()
      8 		  report_to="wandb")
      9 
---> 10 trainer = Trainer(
     11     model=model,
     12     args=training_args,

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _move_model_to_device(self, model, device)
    688 
    689     def _move_model_to_device(self, model, device):
--> 690         model = model.to(device)
    691         # Moving a model to an XLA device disconnects the tied weights, so we have to retie them.
    692         if self.args.parallel_mode == ParallelMode.TPU and hasattr(model, "tie_weights"):

AttributeError: 'str' object has no attribute 'to'
```

### Steps to reproduce the bug

here is the notebook:
```
https://colab.research.google.com/drive/10JDBNsLlYrQdnI2FWfDK3F5M8wvVUDXG?usp=sharing

```


### Expected behavior

run the Training 

### Environment info

Colab Notebook , T4
I train controlnet_sdxl in bf16 datatype, got unsupported ERROR in datasets
Author: HelloWorldBeginner | Pull request: no | State: open

### Describe the bug

```
Traceback (most recent call last):
  File "train_controlnet_sdxl.py", line 1252, in <module>
    main(args)
  File "train_controlnet_sdxl.py", line 1013, in main
    train_dataset = train_dataset.map(compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 592, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3093, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3489, in _map_single
    writer.write_batch(batch)
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_writer.py", line 557, in write_batch
    arrays.append(pa.array(typed_sequence))
  File "pyarrow/array.pxi", line 248, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 113, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/arrow_writer.py", line 191, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True))
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/features/features.py", line 447, in cast_to_python_objects
    return _cast_to_python_objects(
  File "/home/miniconda3/envs/mhh_df/lib/python3.8/site-packages/datasets/features/features.py", line 324, in _cast_to_python_objects
    for x in obj.detach().cpu().numpy()
TypeError: Got unsupported ScalarType BFloat16
```

### Steps to reproduce the bug

Here is my train script I use BF16 type，I use diffusers train my model
```
export MODEL_DIR="/home/mhh/sd_models/stable-diffusion-xl-base-1.0"
export OUTPUT_DIR="./control_net"
export VAE_NAME="/home/mhh/sd_models/sdxl-vae-fp16-fix"

accelerate launch train_controlnet_sdxl.py \
 --pretrained_model_name_or_path=$MODEL_DIR \
 --output_dir=$OUTPUT_DIR \
 --pretrained_vae_model_name_or_path=$VAE_NAME \
 --dataset_name=/home/mhh/sd_datasets/fusing/fill50k \
 --mixed_precision="bf16" \
 --resolution=1024 \
 --learning_rate=1e-5 \
 --max_train_steps=200 \
 --validation_image "/home/mhh/sd_datasets/controlnet_image/conditioning_image_1.png" "/home/mhh/sd_datasets/controlnet_image/conditioning_image_2.png" \
 --validation_prompt "red circle with blue background" "cyan circle with brown floral background" \
 --validation_steps=50 \
 --train_batch_size=1 \
 --gradient_accumulation_steps=4 \
 --report_to="wandb" \
 --seed=42 \
```

### Expected behavior

When I changed the data type to fp16, it worked.

### Environment info

datasets                2.16.1
numpy                   1.24.4
`drop_last_batch=True` for IterableDataset map function is ignored with multiprocessing DataLoader 
Author: naba89 | Pull request: no | State: closed

### Describe the bug

Scenario:
- Interleaving two iterable datasets of unequal lengths (`all_exhausted`), followed by a batch mapping with batch size 2 to effectively merge the two datasets and get a sample from each dataset in a single batch, with `drop_last_batch=True` to skip the last batch in case it doesn't have two samples.

What works:
- Using DataLoader with `num_workers=0`

What does not work:
- Using DataLoader with `num_workers=1`, errors in the last batch.

Basically, `drop_last_batch=True` is ignored when using multiple dataloading workers.

Please take a look at the minimal repro script below.

### Steps to reproduce the bug

```python
from datasets import Dataset, interleave_datasets
from torch.utils.data import DataLoader


def merge_samples(batch):
    assert len(batch['a']) == 2, "Batch size must be 2"
    batch['c'] = [batch['a'][0]]
    batch['d'] = [batch['a'][1]]
    return batch


def gen1():
    for ii in range(1, 8385):
        yield {"a": ii}


def gen2():
    for ii in range(1, 5302):
        yield {"a": ii}


if __name__ == '__main__':

    dataset1 = Dataset.from_generator(gen1).to_iterable_dataset(num_shards=1024)
    dataset2 = Dataset.from_generator(gen2).to_iterable_dataset(num_shards=1024)

    interleaved = interleave_datasets([dataset1, dataset2], stopping_strategy="all_exhausted")
    mapped = interleaved.map(merge_samples, batched=True, batch_size=2, remove_columns=interleaved.column_names,
                             drop_last_batch=True)

    # Works
    loader = DataLoader(mapped, batch_size=32, num_workers=0)
    i = 0
    for b in loader:
        print(i, b['c'].shape, b['d'].shape)
        i += 1

    print("DataLoader with num_workers=0 works")

    # Doesn't work
    loader = DataLoader(mapped, batch_size=32, num_workers=1)
    i = 0
    for b in loader:
        print(i, b['c'].shape, b['d'].shape)
        i += 1


```

### Expected behavior

 `drop_last_batch=True` should have same behaviour for `num_workers=0` and `num_workers>=1`

### Environment info

- `datasets` version: 2.16.1
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.6.0

I have also tested on Linux and got the same behavior.
`Dataset.filter` missing `with_rank` parameter
Author: kopyl | Pull request: no | State: closed

### Describe the bug

The issue shall be open: https://github.com/huggingface/datasets/issues/6435

When i try to pass `with_rank` to `Dataset.filter()`, i get this:

`Dataset.filter() got an unexpected keyword argument 'with_rank'`

### Steps to reproduce the bug

Run notebook:

https://colab.research.google.com/drive/1WUNKph8BdP0on5ve3gQnh_PE0cFLQqTn?usp=sharing

### Expected behavior

Should work?

### Environment info

NVIDIA RTX 4090
datasets.DownloadMode.FORCE_REDOWNLOAD use cache to download dataset features with load_dataset function
Author: LsTam91 | Pull request: no | State: open

### Describe the bug

I have updated my dataset by adding a new feature, and push it to the hub. When I want to download it on my machine which contain the old version by using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` I get an error (paste bellow).

Seems that the load_dataset function still use the old features schema instead of downloading everything new from the HUB.

I find a way to go around this issue by manually deleting the old dataset cache. But from my understanding of `datasets.DownloadMode.FORCE_REDOWNLOAD` option, the dataset cache should be ignored.

### Steps to reproduce the bug

1. Download your dataset in your machine using `datasets.load_dataset`
2. Create a new feature in your dataset and push it to the hub
3. On the same machine redownload your dataset using `datasets.load_dataset("your_dataset_name", download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)`

### Expected behavior

`
ValueError: Couldn't cast
id: string
level: string
context: list<element: string>
  child 0, element: string
type: string
answer: string
question: string
supporting_facts: list<element: string>
  child 0, element: string
fra_answer: string
fra_question: string
-- schema metadata --
huggingface: '{"info": {"features": {"id": {"dtype": "string", "_type": "' + 490
to
{'id': Value(dtype='string', id=None), 'level': Value(dtype='string', id=None), 'context': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 'type': Value(dtype='string', id=None), 'answer': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'supporting_facts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
because column names don't match

The above exception was the direct cause of the following exception:

DatasetGenerationError
...
DatasetGenerationError: An error occurred while generating the dataset`

### Environment info

datasets-2.16.1 huggingface-hub-0.20.2
Latest version 2.16.1, when load dataset error occurs. ValueError: BuilderConfig 'allenai--c4' not found. Available: ['default']
Author: zhulinJulia24 | Pull request: no | State: closed

### Describe the bug

python script is:
```

from datasets import load_dataset
cache_dir = 'path/to/your/cache/directory'
dataset = load_dataset('allenai/c4','allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train',  use_auth_token=False, cache_dir=cache_dir)
```


the script success when datasets version is 2.14.7. 
when using 2.16.1, error occurs
`
ValueError: BuilderConfig 'allenai--c4' not found. Available: ['default']`

### Steps to reproduce the bug

1. pip install datasets==2.16.1
2. run python script:
```

from datasets import load_dataset
cache_dir = 'path/to/your/cache/directory'
dataset = load_dataset('allenai/c4','allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train',  use_auth_token=False, cache_dir=cache_dir)
```

### Expected behavior

the dataset should be loaded successful in the latest version. 

### Environment info

datasets 2.16.1
Fix imagefolder with one image
Author: lhoestq | Pull request: yes | State: closed

A dataset repository with one image and one metadata file was considered a JSON dataset instead of an ImageFolder dataset. This is because we pick the dataset type with the most compatible data file extensions present in the repository and it results in a tie in this case.

e.g. for https://huggingface.co/datasets/multimodalart/repro_1_image

I fixed this by deprioritizing metadata files in the count.

fix https://github.com/huggingface/datasets/issues/6545
Cannot import name 'load_dataset' from .... module ‘datasets’
Author: ciaoyizhen | Pull request: no | State: closed

### Describe the bug


use python -m pip install datasets to install

### Steps to reproduce the bug

from datasets import load_dataset

### Expected behavior

it doesn't work

### Environment info

datasets version==2.15.0
python == 3.10.12
linux  version I don't know??
Loading a dataset from Google Colab hangs at "Resolving data files".
Author: KelSolaar | Pull request: no | State: closed

### Describe the bug

Hello,

I'm trying to load a dataset from Google Colab but the process hangs at `Resolving data files`:

![image](https://github.com/huggingface/datasets/assets/99779/7175ad85-e571-46ed-9f87-92653985777d)

It is happening when the `_get_origin_metadata` definition is invoked:

```python
def _get_origin_metadata(
    data_files: List[str],
    max_workers=64,
    download_config: Optional[DownloadConfig] = None,
) -> Tuple[str]:
    return thread_map(
        partial(_get_single_origin_metadata, download_config=download_config),
        data_files,
        max_workers=max_workers,
        tqdm_class=hf_tqdm,
        desc="Resolving data files",
        disable=len(data_files) <= 16,
```

The thread is then stuck at `waiter.acquire()` in the builtin `threading.py` file.

I can load the dataset just fine on my machine.

Cheers,

Thomas


### Steps to reproduce the bug

In Google Colab:

```python
!pip install datasets
from datasets import load_dataset

dataset = load_dataset("colour-science/color-checker-detection-dataset")
```

### Expected behavior

The dataset should be loaded.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.1
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Fix parallel downloads for datasets without scripts
Author: lhoestq | Pull request: yes | State: closed

Enable parallel downloads using multiprocessing when `num_proc` is passed to `load_dataset`.

It was enabled for datasets with scripts already (if they passed lists to `dl_manager.download`) but not for no-script datasets (we pass dicts {split: [list of files]} to `dl_manager.download` for those ones).

I fixed this by parallelising on the lists contained in the data files dicts when possible.

I also added a context manager `stack_multiprocessing_download_progress_bars` in `DownloadManager` to stack the progress bard of the downloads (from `cached_path(...)` calls). Otherwise the progress bars overlap each other with an annoying flickering effect.
Loading from hf hub with clearer error message
Author: thomwolf | Pull request: no | State: open

### Feature request

Shouldn't this kinda work ?
```
Dataset.from_json("hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json")
```
I got an error
```
File ~/miniconda3/envs/datatrove/lib/python3.10/site-packages/datasets/data_files.py:380, in resolve_pattern(pattern, base_path, allowed_extensions, download_config)
    378     if allowed_extensions is not None:
    379         error_msg += f" with any supported extension {list(allowed_extensions)}"
--> 380     raise FileNotFoundError(error_msg)
    381 return out

FileNotFoundError: Unable to find 'hf://datasets/HuggingFaceTB/eval_data/resolve/main/eval_data_context_and_answers.json'
(I'm logged in)
```

Fix: the correct path is
```
hf://datasets/HuggingFaceTB/eval_data/eval_data_context_and_answers.json
```

Proposal: raise a clearer error

### Motivation

Clearer error message

### Your contribution

Can open a PR
Skip if a dataset has issues
Author: hadianasliwa | Pull request: no | State: open

### Describe the bug

Hello everyone,
I'm using **load_datasets** from **huggingface** to download the datasets and I'm facing an issue, the download starts but it reaches some state and then  fails with the following error:
Couldn't reach https://huggingface.co/datasets/wikimedia/wikipedia/resolve/4cb9b0d719291f1a10f96f67d609c5d442980dc9/20231101.ext/train-00000-of-00001.parquet

Failed to resolve \'huggingface.co\' ([Errno -3] Temporary failure in name resolution)"))')))


![image](https://github.com/huggingface/datasets/assets/143214684/8847d9cb-529e-4eda-9c76-282713dfa3af)

so I was wondering is there a parameter to be passed to load_dataset() to skip files that can't be downloaded??

### Steps to reproduce the bug

Parameter to be passed to load_dataset() of huggingface to skip files that can't be downloaded??

### Expected behavior

load_dataset() finishes without error

### Environment info

None
`image` column not automatically inferred if image dataset only contains 1 image
Author: apolinario | Pull request: no | State: closed

### Describe the bug

By default, the standard Image Dataset maps out `file_name` to `image` when loading an Image Dataset. 

However, if the dataset contains only 1 image, this does not take place 

### Steps to reproduce the bug

Input
(dataset with one image `multimodalart/repro_1_image`) 
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_1_image")
dataset
```

Output: 
```py
DatasetDict({
    train: Dataset({
        features: ['file_name', 'prompt'],
        num_rows: 1
    })
})
```

Input 
(dataset with 2+ images `multimodalart/repro_2_image`) 
```py
from datasets import load_dataset
dataset = load_dataset("multimodalart/repro_2_image")
dataset
```

Output:
```py
DatasetDict({
    train: Dataset({
        features: ['image', 'prompt'],
        num_rows: 2
    })
})
```

### Expected behavior

Expected to map `file_name`  → `image` for all dataset sizes, including 1. 

### Environment info

Both latest main and 2.16.0
Datasets : wikipedia 20220301.en error 
Author: ppx666 | Pull request: no | State: closed

### Describe the bug

When I used load_dataset to download this data set, the following error occurred. The main problem was that the target data did not exist.

### Steps to reproduce the bug

1.I tried downloading directly.
```python
wiki_dataset = load_dataset("wikipedia", "20220301.en")
```
An exception occurred
```
MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
	`load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')`
```
2.I modified the code as prompted.
```python
wiki_dataset = load_dataset('wikipedia', '20220301.en', beam_runner='DirectRunner')
```
An exception occurred:
```
FileNotFoundError: Couldn't find file at https://dumps.wikimedia.org/enwiki/20220301/dumpstatus.json
```


### Expected behavior

I searched in the parent directory of the corresponding URL, but there was no corresponding "20220301" directory.
I really need this data set and hope to provide a download method.

### Environment info

python 3.8
datasets 2.16.0
apache-beam 2.52.0
dill 0.3.7
Extreme inefficiency for `save_to_disk` when merging datasets
Author: KatarinaYuan | Pull request: no | State: open

### Describe the bug

Hi, I tried to merge in total 22M sequences of data, where each sequence is of maximum length 2000. I found that merging these datasets and then `save_to_disk` is extremely slow because of flattening the indices. Wondering if you have any suggestions or guidance on this. Thank you very much! 

### Steps to reproduce the bug

The source data is too big to demonstrate

### Expected behavior

The source data is too big to demonstrate

### Environment info

python                     3.9.0
datasets                  2.7.0
pytorch                   2.0.0 
tokenizers                0.13.1
transformers              4.31.0
'Repo card metadata block was not found' when loading a pragmeval dataset
Author: lambdaofgod | Pull request: no | State: open

### Describe the bug

I can't load dataset subsets of 'pragmeval'.

The funny thing is I ran the dataset author's [colab notebook](https://colab.research.google.com/drive/1sg--LF4z7XR1wxAOfp0-3d4J6kQ9nj_A?usp=sharing) and it works just fine. I tried to install exactly the same packages that are installed on colab using poetry, so my environment info only differs from the one from colab in linux version - I still get the same bug outside colab.

### Steps to reproduce the bug

Install dependencies with poetry

pyproject.toml
```
[tool.poetry]
name = "project"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
datasets = "2.16.0"
pandas = "1.5.3"
pyarrow = "10.0.1"
huggingface-hub = "0.19.4"
fsspec = "2023.6.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
```

`poetry run python -c "import datasets; print(datasets.get_dataset_config_names('pragmeval'))`

prints ['default']

### Expected behavior

The command should print

```
['emergent',
 'emobank-arousal',
 'emobank-dominance',
 'emobank-valence',
 'gum',
 'mrda',
 'pdtb',
 'persuasiveness-claimtype',
 'persuasiveness-eloquence',
 'persuasiveness-premisetype',
 'persuasiveness-relevance',
 'persuasiveness-specificity',
 'persuasiveness-strength',
 'sarcasm',
 'squinky-formality',
 'squinky-implicature',
 'squinky-informativeness',
 'stac',
 'switchboard',
 'verifiability']
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.19.4
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Adding support for netCDF (*.nc) files
Author: shermansiu | Pull request: no | State: open

### Feature request

netCDF (*.nc) is a file format for storing multidimensional scientific data, which is used by packages like `xarray` (labelled multi-dimensional arrays in Python). It would be nice to have native support for netCDF in `datasets`.

### Motivation

When uploading *.nc files onto Huggingface Hub through the `datasets` API, I would like to be able to preview the dataset without converting it to another format.

### Your contribution

I can submit a PR, provided I have the time.
ted_talks_iwslt | Error: Config name is missing
Author: rayliuca | Pull request: no | State: closed

### Describe the bug

Running load_dataset using the newest `datasets` library like below on the ted_talks_iwslt using year pair data will throw an error "Config name is missing"

see also:
https://huggingface.co/datasets/ted_talks_iwslt/discussions/3

likely caused by #6493, where the `and not config_kwargs` part in the if logic was removed 
 https://github.com/huggingface/datasets/blob/ef3b5dd3633995c95d77f35fb17f89ff44990bc4/src/datasets/builder.py#L512

### Steps to reproduce the bug

run: 
```python
load_dataset("ted_talks_iwslt", language_pair=("ja", "en"), year="2015")
```

### Expected behavior

Load the data without error

### Environment info

datasets 2.16.0