`None` replaced by `[]` after first batch in map
Opened by lhoestq (closed)

Sometimes `None` can be replaced by `[]` when running map:

```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
```

This issue has been experienced when running the `run_qa.py` example from `transformers` (see issue https://github.com/huggingface/transformers/issues/15401)

This can be due to a bug in when casting `None` in nested lists. Casting only happens after the first batch, since the first batch is used to infer the feature types.

cc @sgugger 

Comments:
It looks like this is because of this behavior in pyarrow:
```python
import pyarrow as pa

arr = pa.array([None, [0]])
reconstructed_arr = pa.ListArray.from_arrays(arr.offsets, arr.values)
print(reconstructed_arr.to_pylist())
# [[], [0]]
```

It seems that `arr.offsets` can reconstruct the array properly, but an offsets array with null values can:
```python
fixed_offsets = pa.array([None, 0, 1])
fixed_arr = pa.ListArray.from_arrays(fixed_offsets, arr.values)
print(fixed_arr.to_pylist())
# [None, [0]]

print(arr.offsets.to_pylist())
# [0, 0, 1]
print(fixed_offsets.to_pylist())
# [None, 0, 1]
```
EDIT: this is because `arr.offsets` is not enough to reconstruct the array, we also need the validity bitmap

The offsets don't have nulls because they don't include the validity bitmap from `arr.buffers()[0]`, which is used to say which values are null and which values are non-null.

Though the validity bitmap also seems to be wrong:
```python
bin(int(arr.buffers()[0].hex(), 16))
# '0b10'
# it should be 0b110 - 1 corresponds to non-null and 0 corresponds to null, if you take the bits in reverse order
```

So apparently I can't even create the fixed offsets array using this.

If I understand correctly it's always missing the 1 on the left, so I can add it manually as a hack to fix the issue until this is fixed in pyarrow EDIT: actually it may be more complicated than that

EDIT2: actuall it's right, it corresponds to the validity bitmap of the array of logical length 2. So if we use the offsets array, the values array, and this validity bitmap it should be possible to reconstruct the array properly

I created an issue on Apache Arrow's JIRA: https://issues.apache.org/jira/browse/ARROW-15837

And another one: https://issues.apache.org/jira/browse/ARROW-15839

FYI the behavior is the same with:
- `datasets` version: 1.18.3
- Platform: Linux-5.8.0-50-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.11
- PyArrow version: 6.0.1


but not with:
- `datasets` version: 1.8.0
- Platform: Linux-4.18.0-305.40.2.el8_4.x86_64-x86_64-with-redhat-8.4-Ootpa
- Python version: 3.7.11
- PyArrow version: 3.0.0

i.e. it outputs:
```py
0  [None, [0]]
1  [None, [0]]
2  [None, [0]]
3  [None, [0]]
```


Thanks for the insights @PaulLerner !

I found a way to workaround this issue for the code example presented in this issue.

Note that empty lists will still appear when you explicitly `cast` a list of lists that contain None values like [None, [0]] to a new feature type (e.g. to change the integer precision). In this case it will show a warning that it happened. If you don't cast anything, then the None values will be kept as expected.

Let me know what you think !

Hi! I feel like I’m missing something in your answer, *what* is the workaround? Is it fixed in some `datasets` version?

`pa.ListArray.from_arrays` returns empty lists instead of None values. The workaround I added inside `datasets` simply consists in not using `pa.ListArray.from_arrays` :)

Once this PR [here ](https://github.com/huggingface/datasets/pull/4282)is merged, we'll release a new version of `datasets` that currectly returns the None values in the case described in this issue

EDIT: released :) but let's keep this issue open because it might happen again if users change the integer precision for example