Fix Image feature to dynamically select pyarrow storage type for large binary/string inputs

Add dynamic pyarrow storage type selection to the Image feature so that pa.large_binary() and pa.large_string() inputs are preserved natively instead of being cast down to pa.binary() / pa.string()
Introduce _get_target_pa_type() to detect large types from incoming storage and _to_struct_storage() to build struct arrays with the correct field types
Both cast_storage and embed_storage now adapt to the input storage type

# Motivation
When PyArrow internally promotes columns to large_binary or large_string (which happens for large datasets), the Image feature would either attempt to cast them back down, failing for data exceeding 2 GB, or produce type mismatches that cause TypeError: Mask must be a pyarrow. Array of type boolean during embed_storage. This change makes the Image feature dynamically match the storage type of the input, avoiding both cast errors and type mismatches.

# Related Issues
Closes https://github.com/huggingface/datasets/issues/5717



Hi ! if the goal to fix the linked issue maybe we can find a simpler approach.

While your PR seems to solve it, it needs extra careful attention to not add regressions for image datasets, in particular when it comes to data with mixes of both binary and large_binary

e.g.
- concatenating image datasets should always work without copies
- having a stateful feature type should not break stateless functions like the ones that append new record batches or row groups to arrow or parquet files

I agree. I am currently performing additional tests, and I found that my implementation introduced a bug where the first batch in an Arrow shard is written before we know it's too large, which introduces a schema mismatch error.

I will continue to work on a solution that takes your notes into account. 

Thank you! 

I have been giving this PR a thorough consideration. The underlying errors that I see when I have arrow chunks larger than 2GB are a type mismatch between Array and ChunkedArray, which is caused by chunking the binaries

I think a proper solution is to make table.py chunk-aware, so that we allow embed_storage to return either pa.StructArray or pa.ChunkedArray of StructArray, and follow arrows recommendations: https://github.com/apache/arrow/blob/main/docs/source/cpp/arrays.rst#size-limitations-and-recommendations
This will result in embed_storage and table.py becoming quite a lot more complex, though. 

Alternatively, we can allow for parsing a large_binary bool save_to_disk() / push_to_hub(), and make the user opt in for the large_binary type at creation. This will require a binary to large_binary promotion when concatenating the datasets. This might be an alright middle ground without too many code changes, though it will break compatibility when opting in on large_binaries. 


I am unsure of all the edge cases that can happen, so I am not confident in commenting on the backward compatibility of these approaches. Let me know what you think @lhoestq. 

Hmm even if we return ChunkedArray, I think the ArrowWriter will try to combine_chunks() before writing the record batch

The original issue seems to be about save_to_disk specifically, maybe we can focus on that one. In particular it seems it currently uses a fixed batch size:

https://github.com/huggingface/datasets/blob/0e267a7c50f3618ae94bc5e9095a00b82570f197/src/datasets/arrow_dataset.py#L1941-L1951

This explains why in some cases a record batch can be >2GB. I guess a simple fix can be to adapt the batch size automatically using the same method as for push_to_hub:

https://github.com/huggingface/datasets/blob/0e267a7c50f3618ae94bc5e9095a00b82570f197/src/datasets/arrow_dataset.py#L5930-L5933

Okay, I will revert the current changes, implement the changes, and test whether they solve my problem. 

Your suggestion seems to fix the issue where the write batches get too large, but it will not fix the issue when a single row is larger than 2GB. This is likely an extreme edge case. 

I will reopen and mark ready for review the PR when it is ready.