Dataset file size on disk is very large with 3D Array
Author: jblemoine
State: open

Hi, 

I have created my own dataset using the provided dataset loading script. It is an image dataset where images are stored as 3D Array with dtype=uint8. 

The actual size on disk is surprisingly large. It takes 520 MB. Here is some info from `dataset_info.json`. 

`{
    "description": "",
    "citation": "",
    "homepage": "",
    "license": "",
    "features": {
        "image": {
            "shape": [224, 224, 3],
            "dtype": "uint8",
            "id": null,
            "_type": "Array3D",
        }
    },
    "post_processed": null,
    "supervised_keys": null,
    "builder_name": "shot_type_image_dataset",
    "config_name": "default",
    "version": {
        "version_str": "0.0.0",
        "description": null,
        "major": 0,
        "minor": 0,
        "patch": 0,
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 520803408,
            "num_examples": 1479,
            "dataset_name": "shot_type_image_dataset",
        }
    },
    "download_checksums": {
        "": {
            "num_bytes": 16940447118,
            "checksum": "5854035705efe08b0ed8f3cf3da7b4d29cba9055c2d2d702c79785350d72ee03",
        }
    },
    "download_size": 16940447118,
    "post_processing_size": null,
    "dataset_size": 520803408,
    "size_in_bytes": 17461250526,
}`

I have created the same dataset with tensorflow_dataset and it takes only 125MB on disk.

I am wondering, is it normal behavior ? I understand `Datasets` uses Arrow for serialization wheres tf uses TF Records.

This might be a problem for large dataset. 

Thanks for your help. 


Comments: Hi ! In the arrow file we store all the integers as uint8.
So your arrow file should weigh around `height x width x n_channels x n_images` bytes.

What feature type do your TFDS dataset have ?

If it uses a `tfds.features.Image` type, then what is stored is the encoded data (as png or jpg for example). Since these encodings are made for compression, the resulting tfrecord is smaller that the arrow file.

We are working on adding a similar feature in `datasets`: the ability to store the encoded data instead of the raw integers for images, but also for audio data. This way, arrow files will have similar sizes as tfrecords for images.

Thanks for the prompt response. You're right about the encoding, I have the `tfds.features.Image` feature type you mentioned.
However, as described in the `dataset_info.json`, my dataset is made of 1479 (224x224x3) images. 1479 x 224 x 224 x 3 = 222630912 bytes which is far from the actual size 520803408 bytes. 

Anyway I look forward to the Image feature type in `datasets`. 

@lhoestq I changed the data structure so I have a  2D Array feature type instead of a 3D Array by grouping the two last dimensions ( a 224x672 2D Array instead of a 224x224x3 3D Array). The file size is now  223973964 bytes, nearly half the previous size! Which is around of what I would expect.
I found similar behavior in existing `datasets` collection, when comparing black and white vs color image, for example MNIST vs CIFAR. 

Interesting !
This may be because of the offsets that are stored with the array data.

Currently the offsets are stored even if the `shape` of the arrays is fixed. This was needed because of some issues with pyarrow a few months ago. I think these issues have been addressed now, so we can probably try to remove them to make the file lighter.

Ideally in your case the floats data should be 220 MB for both Array2D and Array3D

Yeah for sure, can you be a bit more specific about where the offset is stored in the code base ? And any reference to pyarrow issues if you have some. I would be very interested in contributing to `datasets` by trying to fix this issue. 

Pyarrow has two types of lists: variable length lists and fixed size lists.
Currently we store the ArrayXD data as variable length lists. They take more disk space because they must store both actual data and offsets.
In the `datasets` code this is done here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L346-L352

To use a fixed length list, one should use the `list_size` argument of `pyarrow.list_()`.
I believe this would work directly modulo some changes in the numpy conversion here:

https://github.com/huggingface/nlp/blob/dbac87c8a083f806467f5afc4ec9b401a7e4c15c/src/datasets/features.py#L381-L395