support LargeListArray in pyarrow
State: closed

```python
import numpy as np
import datasets
a = np.zeros((5000000, 768))
res = datasets.Dataset.from_dict({'embedding': a})

'''
  File '/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/arrow_writer.py', line 178, in __arrow_array__
    out = numpy_to_pyarrow_listarray(data)
  File "/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/features/features.py", line 1173, in numpy_to_pyarrow_listarray
    offsets = pa.array(np.arange(n_offsets + 1) * step_offsets, type=pa.int32())
  File "pyarrow/array.pxi", line 312, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 2147483904 not in range: -2147483648 to 2147483647
'''
```


Loading a large numpy array currently raises the error above as the type of offsets is `int32`. 
And pyarrow has supported [LargeListArray](https://arrow.apache.org/docs/python/generated/pyarrow.LargeListArray.html) for this case.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4800). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this! Can you run `make style` at the repo root to fix the code quality error in CI and add a test?

Hi, I have fixed the code quality error and added a test

It seems that CI fails due to the lack of memory for allocating a large array, while I pass the test locally.

Also, the current implementation of the NumPy-to-PyArrow conversion creates a lot of copies, which is not ideal for large arrays.

We can improve performance significantly if we rewrite this part:
https://github.com/huggingface/datasets/blob/83f695c14507a3a38e9f4d84612cf49e5f50c153/src/datasets/features/features.py#L1322-L1323

as
```python
 values = pa.array(arr.ravel(), type=type) 
```

@xwwwwww Feel free to ignore https://github.com/huggingface/datasets/pull/4800#issuecomment-1212280549 and revert the changes you've made to address it. 

Without copying the array, this would be possible:
```python
arr = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

dset = Dataset.from_dict({"data": [arr]})

arr[0][0] = 100 # this change would be reflected in dset's PyArrow table -> a breaking change and also probably unexpected by the user 
```

> @xwwwwww Feel free to ignore [#4800 (comment)](https://github.com/huggingface/datasets/pull/4800#issuecomment-1212280549) and revert the changes you've made to address it.
> 
> Without copying the array, this would be possible:
> 
> ```python
> arr = np.array([
>     [1, 2, 3],
>     [4, 5, 6]
> ])
> 
> dset = Dataset.from_dict({"data": [arr]})
> 
> arr[0][0] = 100 # this change would be reflected in dset's PyArrow table -> a breaking change and also probably unexpected by the user 
> ```

Oh, that makes sense.

passed tests in ubuntu while failed in windows

@mariosasko Hi, do you have any clue about this failure in windows?

Perhaps we can skip the added test on Windows then.

Not sure if this can help, but the ERR tool available on Windows outputs the following for the returned error code `-1073741819`:
```
# for decimal -1073741819 / hex 0xc0000005
  ISCSI_ERR_SETUP_NETWORK_NODE                                   iscsilog.h
# Failed to setup initiator portal. Error status is given in
# the dump data.
  STATUS_ACCESS_VIOLATION                                        ntstatus.h
# The instruction at 0x%p referenced memory at 0x%p. The
# memory could not be %s.
  USBD_STATUS_DEV_NOT_RESPONDING                                 usb.h
# as an HRESULT: Severity: FAILURE (1), FACILITY_NONE (0x0), Code 0x5
# for decimal 5 / hex 0x5
  WINBIO_FP_TOO_FAST                                             winbio_err.h
# Move your finger more slowly on the fingerprint reader.
# as an HRESULT: Severity: FAILURE (1), FACILITY_NULL (0x0), Code 0x5
  ERROR_ACCESS_DENIED                                            winerror.h
# Access is denied.
# 5 matches found for "-1073741819"
```

What's the proper way to skip the added test in windows?
I tried `if platform.system() == 'Linux'`, but the CI test seems stuck

@mariosasko Hi, any idea about this :)

Hi again! We want to skip the test on Windows but not on Linux. You can use this decorator to do so: 
```python
@pytest.mark.skipif(os.name == "nt" and (os.getenv("CIRCLECI") == "true" or os.getenv("GITHUB_ACTIONS") == "true"), reason="The Windows CI runner does not have enough RAM to run this test")
@pytest.mark.parametrize(...)
def test_large_array_xd_with_np(...):
    ...
```

> Hi again! We want to skip the test on Windows but not on Linux. You can use this decorator to do so:
> 
> ```python
> @pytest.mark.skipif(os.name == "nt" and (os.getenv("CIRCLECI") == "true" or os.getenv("GITHUB_ACTIONS") == "true"), reason="The Windows CI runner does not have enough RAM to run this test")
> @pytest.mark.parametrize(...)
> def test_large_array_xd_with_np(...):
>     ...
> ```

CI on windows still stucks :(

@mariosasko Hi, could you please take a look at this issue

@mariosasko Hi, all checks have passed, and we are finally ready to merge this PR :)

@lhoestq @albertvillanova Perhaps other maintainers can take a look and merge this PR :)

same issus come from pyarrow.Is there a solution for this?
file parquet:50GB
datasets version: 2.14.4
pyarrow :12.0.1

Generating train split: 0 examples [01:22, ? examples/s]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/datasets/builder.py", line 1925, in _prepare_split_single
    for _, table in generator:
  File "/opt/conda/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 79, in _generate_tables
    for batch_idx, record_batch in enumerate(
  File "pyarrow/_parquet.pyx", line 1315, in iter_batches
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: List index overflow.

when this feature adds to the newest version?

LargeListArray support is not ready yet, there is one remaining change:

> I think the key is to add the large parameter to Sequence and update the functions you modified in this PR to use pa.list_() if large is False, and pa.large_list otherwise

Gents, any move on this. Convert largse list of dicts to Datasets is a nightmare and took all RAM possible. Is there any other alternative?

Thanks,
Steve


Arrow large_list is supported since datasets 2.21.0. See: https://github.com/huggingface/datasets/releases/tag/2.21.0
- #7019