GitHub issue #5534: map() breaks at certain dataset size when using Array3D
State: open | Created: 2023-02-15T16:34:25Z

### Describe the bug

`map()` magically breaks when using a `Array3D` feature and mapping it. I created a very simple dummy dataset (see below). When filtering it down to 95 elements I can apply map, but it breaks when filtering it down to just 96 entries with the following exception:
```
Traceback (most recent call last):
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3255, in _map_single
    writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 581, in finalize
    self.write_examples_on_file()
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 440, in write_examples_on_file
    batch_examples[col] = array_concat(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1931, in array_concat
    return _concat_arrays(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1901, in _concat_arrays
    return array_type.wrap_array(_concat_arrays([array.storage for array in arrays]))
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1920, in _concat_arrays
    return pa.ListArray.from_arrays(
  File "pyarrow/array.pxi", line 1997, in pyarrow.lib.ListArray.from_arrays
  File "pyarrow/array.pxi", line 1527, in pyarrow.lib.Array.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Negative offsets in list array

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 2815, in map
    return self._map_single(
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 546, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 513, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/fingerprint.py", line 480, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3259, in _map_single
    writer.finalize()
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 581, in finalize
    self.write_examples_on_file()
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/arrow_writer.py", line 440, in write_examples_on_file
    batch_examples[col] = array_concat(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1931, in array_concat
    return _concat_arrays(arrays)
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1901, in _concat_arrays
    return array_type.wrap_array(_concat_arrays([array.storage for array in arrays]))
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1922, in _concat_arrays
    _concat_arrays([array.values for array in arrays]),
  File "/home/arbi01/miniconda3/envs/tmp9/lib/python3.9/site-packages/datasets/table.py", line 1920, in _concat_arrays
    return pa.ListArray.from_arrays(
  File "pyarrow/array.pxi", line 1997, in pyarrow.lib.ListArray.from_arrays
  File "pyarrow/array.pxi", line 1527, in pyarrow.lib.Array.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Negative offsets in list array

```


### Steps to reproduce the bug

1. put following dataset loading script into: debug/debug.py
```python
import datasets
import numpy as np


class DEBUG(datasets.GeneratorBasedBuilder):
    """DEBUG dataset."""

    def _info(self):
        return datasets.DatasetInfo(
            features=datasets.Features(
                {
                    "id": datasets.Value("uint8"),
                    "img_data": datasets.Array3D(shape=(3, 224, 224), dtype="uint8"),
                },
            ),
            supervised_keys=None,
        )

    def _split_generators(self, dl_manager):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN)]

    def _generate_examples(self):
        for i in range(149):
            image_np = np.zeros(shape=(3, 224, 224), dtype=np.int8).tolist()
            yield f"id_{i}", {"id": i, "img_data": image_np}

```

2. try the following code:

```python
import datasets

def add_dummy_col(ex):
    ex["dummy"] = "test"
    return ex


ds = datasets.load_dataset(path="debug", split="train")

# works
ds_filtered_works = ds.filter(lambda example: example["id"] < 95)
print(f"filtered result size: {len(ds_filtered_works)}")
# output:
# filtered result size: 95
ds_mapped_works = ds_filtered_works.map(add_dummy_col)

# fails
ds_filtered_error = ds.filter(lambda example: example["id"] < 96)
print(f"filtered result size: {len(ds_filtered_error)}")
# output:
# filtered result size: 96
ds_mapped_error = ds_filtered_error.map(add_dummy_col)
```


### Expected behavior

The example code does not fail.

### Environment info

Python 3.9.16 (main, Jan 11 2023, 16:05:54); [GCC 11.2.0] :: Anaconda, Inc. on linux
datasets 2.9.0