Fix `array.values` handling in array cast/embed

Fix #6280, fix #6311, fix #6360


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006278 / 0.011353 (-0.005075) | 0.003692 / 0.011008 (-0.007316) | 0.080464 / 0.038508 (0.041956) | 0.064751 / 0.023109 (0.041642) | 0.318586 / 0.275898 (0.042688) | 0.351435 / 0.323480 (0.027955) | 0.005044 / 0.007986 (-0.002942) | 0.003034 / 0.004328 (-0.001295) | 0.063710 / 0.004250 (0.059460) | 0.050607 / 0.037052 (0.013555) | 0.318491 / 0.258489 (0.060001) | 0.365688 / 0.293841 (0.071847) | 0.027818 / 0.128546 (-0.100729) | 0.008119 / 0.075646 (-0.067527) | 0.262141 / 0.419271 (-0.157131) | 0.044710 / 0.043533 (0.001177) | 0.318875 / 0.255139 (0.063736) | 0.344559 / 0.283200 (0.061360) | 0.022861 / 0.141683 (-0.118822) | 1.452402 / 1.452155 (0.000247) | 1.502340 / 1.492716 (0.009624) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219355 / 0.018006 (0.201349) | 0.433311 / 0.000490 (0.432822) | 0.006545 / 0.000200 (0.006345) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024538 / 0.037411 (-0.012874) | 0.073346 / 0.014526 (0.058821) | 0.083824 / 0.176557 (-0.092733) | 0.145176 / 0.737135 (-0.591959) | 0.085941 / 0.296338 (-0.210397) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395153 / 0.215209 (0.179944) | 3.944734 / 2.077655 (1.867080) | 1.883910 / 1.504120 (0.379790) | 1.690560 / 1.541195 (0.149365) | 1.775180 / 1.468490 (0.306690) | 0.506873 / 4.584777 (-4.077904) | 3.111095 / 3.745712 (-0.634617) | 2.915358 / 5.269862 (-2.354504) | 1.892886 / 4.565676 (-2.672791) | 0.058690 / 0.424275 (-0.365585) | 0.006550 / 0.007607 (-0.001057) | 0.463372 / 0.226044 (0.237328) | 4.640511 / 2.268929 (2.371583) | 2.321051 / 55.444624 (-53.123573) | 1.986330 / 6.876477 (-4.890147) | 2.160046 / 2.142072 (0.017973) | 0.597833 / 4.805227 (-4.207394) | 0.127946 / 6.500664 (-6.372718) | 0.059709 / 0.075469 (-0.015760) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.278966 / 1.841788 (-0.562822) | 17.863102 / 8.074308 (9.788794) | 13.896057 / 10.191392 (3.704665) | 0.147512 / 0.680424 (-0.532912) | 0.016771 / 0.534201 (-0.517430) | 0.335260 / 0.579283 (-0.244024) | 0.383019 / 0.434364 (-0.051345) | 0.384821 / 0.540337 (-0.155516) | 0.550143 / 1.386936 (-0.836793) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006234 / 0.011353 (-0.005118) | 0.003695 / 0.011008 (-0.007313) | 0.062654 / 0.038508 (0.024146) | 0.059397 / 0.023109 (0.036287) | 0.458375 / 0.275898 (0.182477) | 0.488951 / 0.323480 (0.165471) | 0.004971 / 0.007986 (-0.003014) | 0.002914 / 0.004328 (-0.001415) | 0.061184 / 0.004250 (0.056934) | 0.051246 / 0.037052 (0.014194) | 0.458035 / 0.258489 (0.199546) | 0.490838 / 0.293841 (0.196997) | 0.028746 / 0.128546 (-0.099800) | 0.008167 / 0.075646 (-0.067480) | 0.068006 / 0.419271 (-0.351265) | 0.041809 / 0.043533 (-0.001724) | 0.453896 / 0.255139 (0.198757) | 0.477583 / 0.283200 (0.194383) | 0.020906 / 0.141683 (-0.120777) | 1.443275 / 1.452155 (-0.008879) | 1.493431 / 1.492716 (0.000714) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219903 / 0.018006 (0.201896) | 0.410275 / 0.000490 (0.409785) | 0.003919 / 0.000200 (0.003719) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027850 / 0.037411 (-0.009561) | 0.080444 / 0.014526 (0.065918) | 0.089943 / 0.176557 (-0.086614) | 0.145810 / 0.737135 (-0.591326) | 0.090908 / 0.296338 (-0.205430) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.464386 / 0.215209 (0.249177) | 4.633787 / 2.077655 (2.556133) | 2.581658 / 1.504120 (1.077538) | 2.408486 / 1.541195 (0.867291) | 2.460491 / 1.468490 (0.992001) | 0.507512 / 4.584777 (-4.077265) | 3.190363 / 3.745712 (-0.555349) | 2.895581 / 5.269862 (-2.374280) | 1.871506 / 4.565676 (-2.694171) | 0.058469 / 0.424275 (-0.365806) | 0.006526 / 0.007607 (-0.001082) | 0.537641 / 0.226044 (0.311596) | 5.396660 / 2.268929 (3.127731) | 3.027028 / 55.444624 (-52.417596) | 2.703771 / 6.876477 (-4.172705) | 2.865576 / 2.142072 (0.723503) | 0.600103 / 4.805227 (-4.205124) | 0.127109 / 6.500664 (-6.373555) | 0.060985 / 0.075469 (-0.014484) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.365030 / 1.841788 (-0.476758) | 17.988218 / 8.074308 (9.913909) | 14.900796 / 10.191392 (4.709404) | 0.158211 / 0.680424 (-0.522213) | 0.018291 / 0.534201 (-0.515910) | 0.337437 / 0.579283 (-0.241846) | 0.383710 / 0.434364 (-0.050654) | 0.392341 / 0.540337 (-0.147997) | 0.561584 / 1.386936 (-0.825352) |

</details>
</details>

![](https://cml.dev/watermark.png#b7571ab4b0d9b67b767c55db400b4ffac0f752f1 "CML watermark")


CI failures are unrelated

I also plan to address https://github.com/huggingface/datasets/issues/6280#issuecomment-1749310065 in this PR :).

Oh ok, ping me again whenever you want another review :)

Have you had a chance to continue this ? I can also take a look if you want

Yes, I'll finish it next week :).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6283). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq Feel free to review this again. I've bumped PyArrow to 12.0.0 to simplify the implementation (no need for custom `array_concat` and less `pa.Array.from_buffers`). However, this makes `apache-beam` complain as it only supports `<12.0.0`. The next `apache-beam` release will set this boundary to `<15.0.0.`, so I think the only solution is to wait for it to be published.