SergeiGKS/huggingface-dataset-issues huggingface dataset:
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
> As I've already said in other discussions, I would not recommend pushing files saved with save_to_disk to the Hub but save to parquet shards and upload them instead. The Hub does not support datasets saved with save_to_disk, which is meant for disk only.

Well noted, thanks. That part was not clear to me :)
Issue: Pushing a large dataset on the hub consistently hangs

Comment:
If the any shard is missing on the Hub, it will re-upload it. It looks like the 30th shard was missing on the Hub in your case. 

It also means that the other files up to the 77th that were successfully uploaded won't be uploaded again.

cc @mariosasko who might know better
Issue: Sequence of array not supported for most dtype

Comment:
Related, `float16` is the only dtype not supported by `Array2D` (probably by every `ArrayND`):

```python
from datasets import Array2D, Features, Dataset

import numpy as np

for dtype in [
    "bool",  # ok
    "int8",  # ok
    "int16",  # ok
    "int32",  # ok
    "int64",  # ok
    "uint8",  # ok
    "uint16",  # ok
    "uint32",  # ok
    "uint64",  # ok
    "float16",  # failed
    "float32",  # ok
    "float64",  # ok
]:
    features = Features({"foo": Array2D(dtype=dtype, shape=(3, 4))})
    array = np.zeros((3, 4), dtype=dtype)
    try:
        dataset = Dataset.from_dict({"foo": [array]}, features=features)
    except Exception as e:
        print(f"Failed for dtype={dtype}")
```
Issue: Sequence of array not supported for most dtype

Comment:
Here's something I can't explain:

When an array is encoded in the `from_dict` method, the numpy array is converted to a list (thus losing the original dtype, which is transfromed to the nearest builtin Python type)

https://github.com/huggingface/datasets/blob/6ee61e6e695b1df9f232d47faf3a5e2b30b33737/src/datasets/features/features.py#L524-L525

However, later on, this same data is written to memory, and it seems authorized that the data is an array (or in this case, a list of arrays). 

https://github.com/huggingface/datasets/blob/6ee61e6e695b1df9f232d47faf3a5e2b30b33737/src/datasets/arrow_writer.py#L185-L186

So the question is: why convert it to a Python list? This seems to be quite expensive both in terms of write time (all data is copied) and memory (e.g., an int8 is converted to an int64).

Finally, if I try to remove this step, it solves all the previous problems, and it seems to me that it doesn't break anything (the CI passes without problem).
Issue: Sequence of array not supported for most dtype

Comment:
Arrow only support 1d numpy arrays, so we convert multidim arrays to lists of 1s arrays (and keep the dtype).

Though you noticed that it's concerting to lists and lose the dtype. If it's the case then it's a bug.
Issue: Sequence of array not supported for most dtype

Comment:
Ok the conversion to list shouldn't be there indeed ! Could you open a PR to remove it ?
Issue: `datasets.map` not reusing cached copy by default

Comment:
This can happen when a map transform cannot be hashed deterministically (e.g., an object referenced by the transform changes its state after the first call - an issue with fast tokenizers). The solution is to provide `cache_file_name` in the `map` call to check this file for the cached result instead of relying on the default caching mechanism.
Issue: Importing PyTorch reduces multiprocessing performance for map

Comment:
Hi! The times match when I run this code locally or on Colab.

Also, we use `multiprocess`, not `multiprocessing`, for parallelization, and torch's `__init__.py` (executed on `import torch` ) slightly modifies the latter.
Issue: Importing PyTorch reduces multiprocessing performance for map

Comment:
Hey Mariosasko,

Thanks for looking into it. We further did some investigations after your comment and figured out it's only affecting some hardware/software configurations with the `pytorch` installation of `conda-forge`. Based on this we found the following issue in PyTorch: https://github.com/pytorch/pytorch/issues/102269 with a quick fix for now.

Since it seems to be a deeper issue with forking processes, the difference between`multiprocess` and `multiprocessing` didn't make a difference.

Closing this, since the issue comes from `pytorch` not `dataset`.
Issue: `IndexError` when indexing `Sequence` of `Array2D` with `None` values

Comment:
Easy fix would be to add:

```python
null_indices -= np.arange(len(null_indices))
```

before L279, but I'm not sure it's the most intuitive way to fix it.
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
Based on https://github.com/rapidsai/cudf/issues/10187, this probably means your `pyarrow` installation is not compatible with `datasets`.

Can you please execute the following commands in the terminal and paste the output here?
```
conda list | grep arrow
``` 
```
python -c "import pyarrow; print(pyarrow.__file__)"
```
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
> Based on [rapidsai/cudf#10187](https://github.com/rapidsai/cudf/issues/10187), this probably means your `pyarrow` installation is not compatible with `datasets`.
> 
> Can you please execute the following commands in the terminal and paste the output here?
> 
> ```
> conda list | grep arrow
> ```
> 
> ```
> python -c "import pyarrow; print(pyarrow.__file__)"
> ```


Here is the output to the first command:
```
arrow-cpp                 11.0.0           py39h7f74497_0  
pyarrow                   12.0.0                   pypi_0    pypi
```
and the second:
```
/Users/edward/opt/anaconda3/envs/cs235/lib/python3.9/site-packages/pyarrow/__init__.py
```
Thanks!
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.

Do we need to update dependencies?
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
Please note that our CI properly passes all tests with `pyarrow-12.0.0`, for Python 3.7 and Python 3.10, for Ubuntu and Windows: see for example https://github.com/huggingface/datasets/actions/runs/5157324334/jobs/9289582291
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
For conda with python3.8.16 this solved my problem! thanks!

> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
> 
> Do we need to update dependencies? I can work on that if no one else is working on it.
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
Thanks for replying. I am not sure about those environments but it seems like pyarrow-12.0.0 does not work for conda with python 3.8.16. 

> Please note that our CI properly passes all tests with `pyarrow-12.0.0`, for Python 3.7 and Python 3.10, for Ubuntu and Windows: see for example https://github.com/huggingface/datasets/actions/runs/5157324334/jobs/9289582291
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
Got the same error with:

```
arrow-cpp                 11.0.0          py310h7516544_0  
pyarrow                   12.0.0                   pypi_0    pypi

python                    3.10.11              h7a1cb2a_2  

datasets                  2.13.0             pyhd8ed1ab_0    conda-forge
```
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
> 
> Do we need to update dependencies?

This solved the issue for me as well.
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
> I got the same error, pyarrow 12.0.0 released May/2023 (https://pypi.org/project/pyarrow/) is not compatible, running `pip install pyarrow==11.0.0` to force install the previous version solved the problem.
> 
> Do we need to update dependencies?

Solved it for me also
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
> 基于 [rapidsai/cudf#10187](https://github.com/rapidsai/cudf/issues/10187)，这可能意味着您的安装与 不兼容。`pyarrow``datasets`
> 
> 您能否在终端中执行以下命令并将输出粘贴到此处？
> 
> ```
> conda list | grep arrow
> ```
> 
> ```
> python -c "import pyarrow; print(pyarrow.__file__)"
> ```

arrow-cpp                 11.0.0          py310h7516544_0  
pyarrow                   12.0.1                   pypi_0    pypi

/root/miniconda3/lib/python3.10/site-packages/pyarrow/__init__.py
Issue: Cannot import datasets - ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility

Comment:
Got the same problem with

arrow-cpp                 11.0.0          py310h1fc3239_0  
pyarrow                   12.0.1                   pypi_0    pypi

miniforge3/envs/mlp/lib/python3.10/site-packages/pyarrow/__init__.py

Reverting back to pyarrow 11 solved the problem.
Issue: Length of table does not accurately reflect the split

Comment:
As already replied by @lhoestq (private channel):
> `.train_test_split` (as well as `.shard`, `.select`) doesn't create a new arrow table to save time and disk space. Instead, it uses an indices mapping on top of the table that locate which examples are part of train or test.
Issue: I tried to load a custom dataset using the following statement: dataset = load_dataset('json', data_files=data_files). The dataset contains 50 million text-image pairs, but an error occurred.

Comment:
Thanks for reporting, @cjt222.

What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead.
Issue: I tried to load a custom dataset using the following statement: dataset = load_dataset('json', data_files=data_files). The dataset contains 50 million text-image pairs, but an error occurred.

Comment:
> Thanks for reporting, @cjt222.
> 
> What is the structure of your JSON files. Please note that it is normally simpler if the data file format is JSON-Lines instead.

Thanks! I have encountered similar problems. I modify the json format from list to line and works!
Issue: Missing elements in `map` a batched dataset

Comment:
Hi ! in your code batching is **only used within** `map`, to process examples in batch. The dataset itself however is not batched and returns elements one by one.

To iterate on batches, you can do
```python
for batch in dataset.iter(batch_size=8):
    ...
```
Issue: Cannot use both set_format and set_transform

Comment:
Currently, it's not possible to chain `set_format`/`set_transform` calls (plus, this is a breaking change if we decide to implement it), so I see two possible solutions:
* using `set_format`/`set_transform` for the 1st transform and then passing the transformed example/batch to the 2nd transform
* implementing and registering a custom formatter (the relevant code is [here](https://github.com/huggingface/datasets/tree/main/src/datasets/formatting))

Btw, your example requires a single `set_format` call:
```python
ds.set_format("torch", columns=["image"], output_all_columns=True, dtype=torch.double)
```
Issue: Cannot use both set_format and set_transform

Comment:
Hey Mario,
Thanks, for getting back to me. the toDouble was just an example my real life case requires many more transforms.

What do you mean by:
> using set_format/set_transform for the 1st transform and then passing the transformed example/batch to the 2nd transform

How would that go, I thought you can't chain them?

As for the custom formatter, is it possible to reference an existing formatter, in my case `torch_formatter` inside of my custom formatter?

maybe I can inherit from it and just call `super.recursive_tensorize()`?
Issue: Cannot use both set_format and set_transform

Comment:
> How would that go, I thought you can't chain them?

Yes, they cannot be chained. This is what I meant:
```python
ds.set_transform(first_transform)
# calling the 2nd transform on each accessed batch
second_transform(ds[2:3])
```

> As for the custom formatter, is it possible to reference an existing formatter, in my case torch_formatter inside of my custom formatter?
>
>maybe I can inherit from it and just call super.recursive_tensorize()?

Yes, subclassing makes the most sense.
Issue: Unbearably slow sorting on big mapped datasets

Comment:
Hi ! `shard` currently returns a slow dataset by default, with examples evenly distributed in the dataset.

You can get a fast dataset using `contiguous=True` (which should be the default imo):

```python
dataset = dataset.shard(10, 0, contiguous=True)
```

This way you don't need to flatten_indices() and sort should be fast as well
Issue: Unbearably slow sorting on big mapped datasets

Comment:
@lhoestq 

> contiguous=True (which should be the default imo)

For `IterableDataset`, it's not possible to implement contiguous sharding without knowing the number of examples in advance, so setting the default value to `contiguous=True` would result in an inconsistency between `Dataset` and `IterableDataset` (when we add `IterableDataset.shard`)
Issue: Unbearably slow sorting on big mapped datasets

Comment:
Actually sharded iterable datasets are made of sub iterables that generally yield contiguous data no ? So in a way it's possible to shard an iterable dataset contiguously.

If the dataset is made of one shard it's indeed not possible to shard it contiguously though
Issue: Unbearably slow sorting on big mapped datasets

Comment:
> Actually sharded iterable datasets are made of sub iterables that generally yield contiguous data no ? So in a way it's possible to shard an iterable dataset contiguously.

But sharding an iterable dataset by sharding its `gen_kwargs` would still yield approximate shards(not equal to `Dataset.shard`), no?
Issue: Unbearably slow sorting on big mapped datasets

Comment:
I understand the issue doesn't exist with non-mapped datasets, but if flattening is so much more efficient than sorting the indices, that's an issue in itself.

There are plenty of issues people posted for which the root cause turns out to be the same. It seems like mapped datasets are terribly inefficient. I think I saw some issue like that somewhere (about the mapped datasets in general), but can't find it now.

Maybe indices should be flattened before any additional processing, then.
Issue: Offer an alternative to Iterable Dataset that allows lazy loading and processing while skipping batches efficiently

Comment:
We plan to improve this eventually (see https://github.com/huggingface/datasets/issues/5454 and https://github.com/huggingface/datasets/issues/5380).

> Is it possible to lazily load samples of a mapped dataset ? I'm used to [dataset scripts](https://huggingface.co/docs/datasets/dataset_script), maybe something can be done there.
If not, I could do it using a plain Pytorch dataset. Then I would need to convert it to a datasets' dataset to get all the features of datasets. Is it something possible ?

Yes, by creating a mapped dataset that stores audio URLs. Indexing a dataset in such format only downloads and decodes the bytes of the accessed samples (without storing them on disk).

You can do the following to create this dataset:
```python

def gen():
    # Generator that yields (audio URL, text) pairs as dict
    ...
    yield {"audio": "audio_url", "text": "some text"}

features = Features({"audio": datasets.Audio(), "text": datasets.Value("string")})
ds = Dataset.from_generator(gen, features=features)
ds[2:5] # downloads and decodes the samples each time they are accessed
```
Issue: The dir name and split strings are confused when loading ArmelR/stack-exchange-instruction dataset

Comment:
Thanks for reporting, @DongHande.

I think the issue is caused by the metadata in the dataset card: in the header of the `README.md`, they state that the dataset has 4 splits ("finetune", "reward", "rl", "evaluation"). 
```yaml
  splits:
    - name: finetune
      num_bytes: 6674567576
      num_examples: 3000000
    - name: reward
      num_bytes: 6674341521
      num_examples: 3000000
    - name: rl
      num_bytes: 6679279968
      num_examples: 3000000
    - name: evaluation
      num_bytes: 4022714493
      num_examples: 1807695
```


I guess the user wanted to define these as configs, instead of splits. This is not yet supported for no-script datasets, but will be soon supported. See:
- #5331

I think we should contact the dataset author to inform about the issue with the split names, as you already did: https://huggingface.co/datasets/ArmelR/stack-exchange-instruction/discussions/1
Let's continue the discussion there!
Issue: User access requests with manual review do not notify the dataset owner

Comment:
I think this has been addressed.

Please open a new issue if you are still not getting notified.
Issue: HuggingsFace dataset example give error

Comment:
Nice catch @donhuvy, that's because some models don't need the `token_type_ids`, as in this case, as the example is using `distilbert-base-cased`, and according to the DistilBert documentation at https://huggingface.co/transformers/v3.0.2/model_doc/distilbert.html, `DistilBert doesn’t have token_type_ids, you don’t need to indicate which token belongs to which segment. Just separate your segments with the separation token tokenizer.sep_token (or [SEP])`. `token_type_ids` are neither required in some other well known models such as RoBERTa. 

Here the issue comes due to a mismatch between the tokenizer and the model, as the Colab is using a BERT tokenizer (`bert-base-cased`), while the model is a DistilBERT (`distilbert-base-cased`), so aligning the tokenizer and the model solves it!
Issue: HuggingsFace dataset example give error

Comment:
This has been addressed in #5902.

The Quicktour notebook is deprecated now - please use the notebook version of the [Quickstart doc page](https://huggingface.co/docs/datasets/main/en/quickstart) instead ("Open in Colab" button).
Issue: Use work-stealing algorithm when parallel computing

Comment:
Alternatively we could set the number of shards to be a factor than the number of processes (current they're equal) - this way it will be less likely to end up with a shard that is significantly slower than all the other ones.
Issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset

Comment:
Hi! 

You want to use `push_to_hub` (creates Parquet files) instead of `save_to_disk` (creates Arrow files) when creating a Hub dataset. Parquet is designed for long-term storage and takes less space than the Arrow format, and, most importantly, `load_dataset` can parse it, which should fix the viewer. 

Regarding the dataset generation, `Dataset.from_generator` with the video data represented as `datasets.Value("binary")` followed by `push_to_hub` should work (if the `push_to_hub` step times out, restart it to resume uploading)

PS: Once the dataset is uploaded, to make working with the dataset easier, it's a good idea to add a [transform](https://huggingface.co/docs/datasets/main/en/process#format-transform) to the README that shows how to decode the binary video data into something a model can understand. Also, if you get an `ArrowInvalid` error (can happen when working with large binary data) in `Dataset.from_generator`, reduce the value of `writer_batch_size` (the default is 1000) to fix it.
Issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset

Comment:
One issue here is that Dataset.from_generator can work well for the non 'infinite sampling' version of the dataset. The training set for example is often sampled dynamically given the video files that I have uploaded. I worry that storing the video data as binary means that I'll end up duplicating a lot of the data. Furthermore, storing video data as anything but .mp4 would quickly make the dataset size from 1.9TB to 1PB.
Issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset

Comment:
> storing video data as anything but .mp4

What I mean by storing as `datasets.Value("binary")` is embedding raw MP4 bytes in the Arrow table, but, indeed, this would waste a lot of space if there are duplicates.

So I see two options:
* if one video is not mapped to too many samples, you can embed the video bytes and do "group by" on the rest of the columns (this would turn them into lists) to avoid duplicating them (then, it should be easy to define a `map` in the README that samples the video data to "unpack" the samples)
* you can create a dataset script that downloads the video files and embeds their file paths into the Arrow file

Also, I misread MP4 as MP3. We need to add a `Video` feature to the `datasets` lib to support MP4 files in the viewer (a bit trickier to implement than the `Image` feature due to the Arrow limitations).
Issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset

Comment:
@mariosasko Right. If I want my dataset to be streamable, what are the necessary requirements to achieve that within the context of .mp4 binaries like we have here? I guess your second point here would not support that right?
Issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset

Comment:
The streaming would work, but the video paths would require using `fsspec.open` to get the content.
Issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset

Comment:
Not yet. The (open source) tooling for video is not great in terms of ease of use/performance, so we are discussing internally the best way to support it (one option is creating a new library for video IO, but this will require a lot of work)
Issue: A way to upload and visualize .mp4 files (millions of them) as part of a dataset

Comment:
True. I spend a good 4 months just mixing and matching existing solutions so I could get performance that would not IO bound my model training. 

This is what I ended up with, in case it's useful

https://github.com/AntreasAntoniou/TALI/blob/045cf9e5aa75b1bf2c6d5351fb910fa10e3ff32c/tali/data/data_plus.py#L85
Issue: Split dataset by node: index error when sharding iterable dataset

Comment:
cc @lhoestq in case you have any ideas here! Might need a multi-host set-up to debug (can give you access to a JAX one if you need)
Issue: load_dataset from s3 file system through streaming can't not iterate data 

Comment:
This sounds related to #5281.

Can you try passing `storage_options=s3_client.storage_options` instead passing it to `use_auth_token=` ?