SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Multiple dataloader memory error

Comment:
Hi!

200+ data loaders is a lot. Have you tried to reduce the number of datasets by concatenating/interleaving the ones with the same structure/task (the API is `{concatenate_datasets/interleave_datasets}([dset1, ..., dset_N])`)?
Issue: Multiple dataloader memory error

Comment:
Hi @mariosasko, thank you for your reply. I tried pre-concatenating different datasets into one, but one key need is to keep each batch the same data type. Considering that the concatenate-then-segment operation for prefetched samples may span across different data types after concatenating/interleaving (cuz different data sources are mixed), any solution to remain the same data source for each batch?
Issue: Multiple dataloader memory error

Comment:
@cyk1337 have you found any solutions to it?
@mariosasko I tried with interleave_datasets to sample batches from two large datasets (wikipedia alike) and it results in out-of-memory error during data loading (16gpus, >1TB physical memory). Do you have any idea about it?
Issue: Dataset Viewer issue for MoritzLaurer/multilingual_nli

Comment:
Thanks for reporting  @MoritzLaurer.

Currently, the dataset preview is working properly: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

Please note that when a dataset is modified, it might take some time until the preview is completely updated.

@severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?
Issue: Dataset Viewer issue for MoritzLaurer/multilingual_nli

Comment:
Thanks for your response. You are right, its now working well. I had waited for 30 min or so and refreshed several times and thought there was some other error. Yeah, a different error message sounds like a good idea to avoid confusion.
Issue: Dataset Viewer issue for MoritzLaurer/multilingual_nli

Comment:
> @severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Yes, it's a known issue, and we're about to ship a better version
Issue: Allow pathlib PoxisPath in Dataset.read_json

Comment:
@cccntu I want to make a quick fix for this, but I am struggling to find where the json dataset builder is. Do you know?
Issue: Allow pathlib PoxisPath in Dataset.read_json

Comment:
@vvvm23 I think you mean think:
```python
ds = datasets.load_dataset('json', data_files=[Path('test.jsonl')])
```
And the place you want to modify is here:
```
utils/file_utils.py:64, in is_remote_url(url_or_filename)
     63 def is_remote_url(url_or_filename: str) -> bool:
---> 64     parsed = urlparse(url_or_filename)
     65     return parsed.scheme in ("http", "https", "s3", "gs", "hdfs", "ftp")
```

Probably just need to check first if `url_or_filename` is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Btw, I tried installing from main, and ran my code above and got a different error. Probably because the API have changed.
`AttributeError: module 'datasets' has no attribute 'read_json'`
Issue: Allow pathlib PoxisPath in Dataset.read_json

Comment:
> @vvvm23 I think you mean think:

You are correct, thanks!

> Probably just need to check first if url_or_filename is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Is PathLike sufficient, or should I check the file exists here? Or is that handled later?
Issue: Allow pathlib PoxisPath in Dataset.read_json

Comment:
I think here we just want to avoid passing Path to urlparse. A simpler solution is to add a str() call and convert the input to string before passing to the next step. No need to check anything.
Issue: Allow pathlib PoxisPath in Dataset.read_json

Comment:
Above PR should do your first suggestion. Hope that works for you, as I am going on holiday and won't be able to change much :wink:
Issue: TFDS wiki_dialog dataset to Huggingface dataset

Comment:
@albertvillanova any help ? The linked dataset is in beam format which is similar to wikipedia dataset in huggingface that you  scripted..
Issue: TFDS wiki_dialog dataset to Huggingface dataset

Comment:
Nvm, I was able to port it to huggingface datasets, will upload to the hub soon
Issue: Got "AttributeError: 'xPath' object has no attribute 'read'" when loading an excel dataset with my own code

Comment:
Hi hi, instead of using `download_and_extract` function, I only use `download` function: `base_dir = Path(dl_manager.download(urls))`. It turns out that the code works for `datasets==2.2.2`, however, it doesn't work with `datasets==2.4.0`.
Issue: Got "AttributeError: 'xPath' object has no attribute 'read'" when loading an excel dataset with my own code

Comment:
Hi @yana-xuyan, thanks for reporting.

Indeed you already found the answer: an Excel file should be just downloaded and not downloaded-and-extracted.

The reason why is that if you call also extract, our library will try to infer the compression format (and extract it). And Excel files are viewed as ZIP files and extracted as so (into a directory). This is because the Office Open XML is indeed a zipped file under the hood): https://en.wikipedia.org/wiki/Office_Open_XML
> Office Open XML (also informally known as OOXML) is a **zipped**, XML-based file format
```python
import zipfile

zipfile.is_zipfile("filename.xlsx")
```
returns `True`.
Issue: Got "AttributeError: 'xPath' object has no attribute 'read'" when loading an excel dataset with my own code

Comment:
Hi @albertvillanova, thank you for your reply! Do you have any clue on why the same error still exists with `datasets==2.4.0` even after I don't extract the downloaded file? FYI, if I downgrade to `datasets==2.2.2`, the code works fine.
Issue: Got "AttributeError: 'xPath' object has no attribute 'read'" when loading an excel dataset with my own code

Comment:
I guess this has to do with the cache: you should remove the previously-wrongly generated directory from the cache; otherwise `datasets` tries to re-use it.
Issue: Using disk for memory with the method `from_dict`

Comment:
This issue was also causing an OOM in @nateraw 's workflow and shows again that behavior is confusing - we should definitely switch to using the disk IMO
Issue: map() function removes columns when input_columns is not None

Comment:
Hi! Thanks for reporting! This looks like a bug. I've just opened a PR with the fix.
Issue: No preprocessed wikipedia is working on huggingface/datasets

Comment:
Thanks for reporting @aninrusimha.

Please, note that the preprocessed datasets are still available, as described in the dataset card, e.g.: https://huggingface.co/datasets/wikipedia
```python
ds = load_dataset("wikipedia", "20220301.en")
```
Issue: No preprocessed wikipedia is working on huggingface/datasets

Comment:
This is working now, but I was getting an error a few days ago when running an existing script. Unfortunately I did not do a proper bug report, but for some reason I was unable to load the dataset due to a request being made to the wikimedia website. However, its working now. Thanks for the reply!
Issue: file missing when load_dataset with openwebtext on windows

Comment:
I have tried to extract ```0015896-b1054262f7da52a0518521e29c8e352c.txt``` from ```17ecf461bfccd469a1fbc264ccb03731f8606eea7b3e2e8b86e13d18040bf5b3/urlsf_subset00-16_data.xz``` with 7-zip
and put the file into cache_path ```F://huggingface/datasets/downloads/extracted/0901d27f43b7e9ac0577da0d0061c8c632ba0b70ecd1b4bfb21562d9b7486faa```
there is still raise the same error and I  find the file  was removed from cache_path after I run the run_mlm.py with ```python run_mlm.py --model_type roberta --tokenizer_name roberta-base --dataset_name openwebtext --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --do_train --do_eval --output_dir F:/model/roberta-base```.
Issue: Dataset Viewer issue for super_glue

Comment:
Thanks for reporting @wzsxxa.

However the "super_glue" dataset is rendered properly by the Dataset preview: https://huggingface.co/datasets/super_glue
Issue: Bug in multilingual_with_para config of exams dataset and checksums error

Comment:
Hi @albertvillanova. Unfortunately I still get this error. Is this because the merge has yet to be released? Is there a way to track the release?
Issue: Bug in multilingual_with_para config of exams dataset and checksums error

Comment:
Hi @thesofakillers, yes you are right: the fix will be available after next release (it was planned for today; Monday at the latest).

In the meantime, you can use the version of the `exams` on our main branch by passing `revision` to `load_dataset`:
```python
ds = load_dataset("exams", revision="main")
```
Issue: Dataset Viewer issue for darragh/demo_data_raw3

Comment:
do you have an idea of why it can occur @huggingface/datasets? The dataset consists of a single parquet file.
Issue: Dataset Viewer issue for darragh/demo_data_raw3

Comment:
Thanks for reporting @severo.

I'm not able to reproduce that error. I get instead:
```
FileNotFoundError: [Errno 2] No such file or directory: 'orix/data/ChiSig/唐合乐-9-3.jpg'
```

Which pyarrow version are you using? Mine is 6.0.1.
Issue: Dataset Viewer issue for darragh/demo_data_raw3

Comment:
Apparently, there is something weird with that Parquet file: its schema is:
```
images: extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>>
```

I have forced a right schema:
```python
from datasets import Features, Image, load_dataset

features = Features({"images": Image()})
ds = datasets.load_dataset("parquet", split="train", data_files="train-00000-of-00001.parquet", features=features)
```
and then recreated a new Parquet file:
```python
ds.to_parquet("train.parquet")
```

Now this Parquet file has the right schema:
```
images: struct<bytes: binary, path: string>
  child 0, bytes: binary
  child 1, path: string
```
and can be loaded normally:
```python
In [26]: ds = load_dataset("parquet", split="train", data_files="dataset.parquet")
n [27]: ds
Out[27]: 
Dataset({
    features: ['images'],
    num_rows: 20
})
```
Issue: Misalignment between card tag validation and docs

Comment:
(Note that the doc is aligned with the hub validation rules, and the "ground truth" is the hub validation rules given that they apply to all datasets, not just the canonical ones)
Issue: Misalignment between card tag validation and docs

Comment:
Instead of our own implementation, we now use `huggingface_hub`'s `DatasetCardData`, which has the correct type hint, so I think we can close this issue.
Issue: streaming dataset with concatenating splits raises an error

Comment:
Hi! Only the name of a particular split ("train", "test", ...) is supported as a split pattern if `streaming=True`. We plan to address this limitation soon.
Issue: streaming dataset with concatenating splits raises an error

Comment:
yes, same error occurs.
```python
from datasets import load_dataset

# error
repo = "nateraw/ade20k-tiny"
dataset = load_dataset(repo, split="train+validation", streaming=True)
```

```python
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-3-a6ae02d63899>](https://localhost:8080/#) in <cell line: 5>()
      3 # error
      4 repo = "nateraw/ade20k-tiny"
----> 5 dataset = load_dataset(repo, split="train+validation", streaming=True)

1 frames
[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in as_streaming_dataset(self, split, base_path)
   1265             splits_generator = splits_generators[split]
   1266         else:
-> 1267             raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
   1268 
   1269         # Create a dataset for each of the given splits

ValueError: Bad split: train+validation. Available splits: ['train', 'validation']
```

google colab, `datasets==2.12.0`
```
- huggingface_hub version: 0.14.1
- Platform: Linux-5.10.147+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /root/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: 2.7.12
- Tensorflow: 2.12.0
- Torch: 2.0.0+cu118
- Jinja2: 3.1.2
- Graphviz: 0.20.1
- Pydot: 1.4.2
- Pillow: 8.4.0
- hf_transfer: N/A
- gradio: N/A
- ENDPOINT: https://huggingface.co/
- HUGGINGFACE_HUB_CACHE: /root/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /root/.cache/huggingface/assets
- HF_TOKEN_PATH: /root/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
```
Issue: `with_format` behavior is inconsistent on different datasets

Comment:
Hi! You can get a `torch.Tensor` if you do the following:
```python
raw = load_dataset("beans", split="train")
raw = raw.select(range(100))

preprocessor = AutoFeatureExtractor.from_pretrained("nateraw/vit-base-beans")

from datasets import Array3D
features = raw.features.copy()
features["pixel_values"] = datasets.Array3D(shape=(3, 224, 224), dtype="float32")

def preprocess_func(examples):
    imgs = [img.convert("RGB") for img in examples["image"]]
    return preprocessor(imgs)

data = raw.map(preprocess_func, batched=True, features=features)

print(type(data[0]["pixel_values"]))

data = data.with_format("torch", columns=["pixel_values"])

print(type(data[0]["pixel_values"]))
```

The reason for this "inconsistency" in the default case is the way PyArrow infers the type of multi-dim arrays (in this case, the `pixel_values` column). If the type is not specified manually, PyArrow assumes it is a dynamic-length sequence (it needs to know the type before writing the first batch to a cache file, and it can't be sure the array is fixed ahead of time; `ArrayXD` is our way of telling that the dims are fixed), so it already fails to convert the corresponding array to NumPy properly (you get an array of `np.object` arrays). And `with_format("torch")` replaces NumPy arrays with Torch tensors, so this bad formatting propagates.
Issue: video dataset loader/parser

Comment:
Hi! We've just started discussing the video support in `datasets` (decoding backends, video feature type, etc.), so I believe we should have something tangible by the end of this year.

Also, if you have additional video features in mind that you would like to see, feel free to let us know
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
@mariosasko I'm getting a similar issue when creating a Dataset from a Pandas dataframe, like so:

```
from datasets import Dataset, Features, Image, Value
import pandas as pd
import requests
import PIL

# we need to define the features ourselves
features = Features({
    'a': Value(dtype='int32'),
    'b': Image(),
})

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = PIL.Image.open(requests.get(url, stream=True).raw)

df = pd.DataFrame({"a": [1, 2], 
                   "b": [image, image]})

dataset = Dataset.from_pandas(df, features=features) 
```
results in 

```
ArrowInvalid: ('Could not convert <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x480 at 0x7F7991A15C10> with type JpegImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column b with type object')
```

Will the PR linked above also fix that?
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
I would expect this to work, but it doesn't. Shouldn't be too hard to fix tho (in a subsequent PR).
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
Hi @mariosasko just wanted to check in if there is a PR to follow for this. I was looking to create a demo app using this. If it's not working I can just use byte encoded images in the dataset which are not displayed.
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
I was just pointed here by @mariosasko, meanwhile I found a workaround using `encode_example` like so:

```
from datasets import load_from_disk, Dataset
DATASET_PATH = "/hf/m4-master/data/cm4/cm4-10000-v0.1"
ds1 = load_from_disk(DATASET_PATH)
ds2 = Dataset.from_dict(mapping={k: [] for k in ds1[99].keys()},
                       features=ds1.features
)
for i in range(2):
    # could add several representative items here
    row = ds1[99]
    row_encoded = ds2.features.encode_example(row)
    ds2 = ds2.add_item(row_encoded)
```
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
Hmm, interesting. If I create the dataset on the fly:

```
from datasets import load_from_disk, Dataset
DATASET_PATH = "/hf/m4-master/data/cm4/cm4-10000-v0.1"
ds1 = load_from_disk(DATASET_PATH)
ds2 = Dataset.from_dict(mapping={k: [v]*2 for k, v in ds1[99].items()},
                        features=ds1.features)
```

it doesn't fail with the error in the OP, as `from_dict` performs `encode_batch`.

However if I try to use this dataset it fails now with:

```
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 557, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 524, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2775, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2655, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2347, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "debug_leak2.py", line 235, in split_pack_and_pad
    images.append(image_transform(image.convert("RGB")))
AttributeError: 'dict' object has no attribute 'convert'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "debug_leak2.py", line 418, in <module>
    train_loader, val_loader = get_dataloaders()
  File "debug_leak2.py", line 348, in get_dataloaders
    dataset = dataset.map(mapper, batch_size=32, batched=True, remove_columns=dataset.column_names, num_proc=4)
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2500, in map
    transformed_shards[index] = async_result.get()
  File "/home/stas/anaconda3/envs/py38-pt112/lib/python3.8/site-packages/multiprocess/pool.py", line 771, in get
    raise self._value
AttributeError: 'dict' object has no attribute 'convert'
```

but if I create that same dataset one item at a time as in the previous comment's code snippet it doesn't fail.

The features of this dataset are set to:

```
{'texts': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None), 
'images': Sequence(feature=Image(decode=True, id=None), length=-1, id=None)}
```
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
> @mariosasko I'm getting a similar issue when creating a Dataset from a Pandas dataframe, like so:
> 
> ```
> from datasets import Dataset, Features, Image, Value
> import pandas as pd
> import requests
> import PIL
> 
> # we need to define the features ourselves
> features = Features({
>     'a': Value(dtype='int32'),
>     'b': Image(),
> })
> 
> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
> image = PIL.Image.open(requests.get(url, stream=True).raw)
> 
> df = pd.DataFrame({"a": [1, 2], 
>                    "b": [image, image]})
> 
> dataset = Dataset.from_pandas(df, features=features) 
> ```
> 
> results in
> 
> ```
> ArrowInvalid: ('Could not convert <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x480 at 0x7F7991A15C10> with type JpegImageFile: did not recognize Python value type when inferring an Arrow data type', 'Conversion failed for column b with type object')
> ```
> 
> Will the PR linked above also fix that?

It looks like the problem still exists.
Any news ? Any good workaround ?

Thank you
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
There is a workaround: 
Create a loader python scrypt and upload the dataset to huggingface.

Here is an example how to do that:

https://huggingface.co/datasets/jamescalam/image-text-demo/tree/main

and Here are videos with explanations:

https://www.youtube.com/watch?v=lqK4ocAKveE and https://www.youtube.com/watch?v=ODdKC30dT8c
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
> Any update on this? I'm still facing this issure. Any workaround?

I was able to resolve my issue with a quick workaround: 

```
from collections import defaultdict
from datasets import Dataset
  
data = defaultdict(list)
for idx in tqdm(range( len(dataloader)),desc="Captioning..."):
    img = dataloader[idx]
    data['image'].append(img)
    data['text'].append(f"{img_{idx}})
  
dataset = Dataset.from_dict(data)
dataset = dataset.filter(lambda example: example['image'] is not None)
dataset = dataset.filter(lambda example: example['text'] is not None)
  
dataset.push_to_hub(path-to-repo', private=False)
```

Hope it helps!
Happy coding
Issue: ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

Comment:
> > Any update on this? I'm still facing this issure. Any workaround?
> 
> I was able to resolve my issue with a quick workaround:
> 
> ```
> from collections import defaultdict
> from datasets import Dataset
>   
> data = defaultdict(list)
> for idx in tqdm(range( len(dataloader)),desc="Captioning..."):
>     img = dataloader[idx]
>     data['image'].append(img)
>     data['text'].append(f"{img_{idx}})
>   
> dataset = Dataset.from_dict(data)
> dataset = dataset.filter(lambda example: example['image'] is not None)
> dataset = dataset.filter(lambda example: example['text'] is not None)
>   
> dataset.push_to_hub(path-to-repo', private=False)
> ```
> 
> Hope it helps! Happy coding

It works!!
Issue: Missing MBPP splits

Comment:
Thanks for reporting this as well, @stadlerb.

I suggest waiting for the answer of the data owners...
Issue: Missing MBPP splits

Comment:
@albertvillanova The first author of the paper responded to the upstream issue:
> Task IDs 11-510 are the 500 test problems. We use 90 problems (511-600) for validation and then remaining 374 for fine-tuning (601-974). The other problems can be used as desired, either for training or few-shot prompting (although this should be specified).
Issue: Missing MBPP splits

Comment:
Thanks for the follow-up, @stadlerb.

Would you be willing to open a Pull Request to address this issue? :wink:
Issue: Add DocVQA

Comment:
Thanks for proposing, @NielsRogge.

Please, note this dataset requires registering in their website and their Terms and Conditions state we cannot distribute their URL:
```
1. You will NOT distribute the download URLs
...
```