Support pyarrow 14.0.0
Opened by albertvillanova (closed)

Support `pyarrow` 14.0.0.

Fix #6377 and fix #6374 (root cause).

This fix is analog to a previous one:
- #6175

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007561 / 0.011353 (-0.003792) | 0.004824 / 0.011008 (-0.006184) | 0.110372 / 0.038508 (0.071864) | 0.076767 / 0.023109 (0.053657) | 0.357094 / 0.275898 (0.081196) | 0.420566 / 0.323480 (0.097086) | 0.004753 / 0.007986 (-0.003232) | 0.004734 / 0.004328 (0.000405) | 0.072926 / 0.004250 (0.068675) | 0.058045 / 0.037052 (0.020992) | 0.401109 / 0.258489 (0.142620) | 0.444585 / 0.293841 (0.150744) | 0.046492 / 0.128546 (-0.082055) | 0.013948 / 0.075646 (-0.061698) | 0.305188 / 0.419271 (-0.114083) | 0.063112 / 0.043533 (0.019579) | 0.384711 / 0.255139 (0.129572) | 0.411375 / 0.283200 (0.128175) | 0.048147 / 0.141683 (-0.093536) | 1.632357 / 1.452155 (0.180202) | 1.661021 / 1.492716 (0.168304) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.281104 / 0.018006 (0.263098) | 0.567152 / 0.000490 (0.566662) | 0.007178 / 0.000200 (0.006978) | 0.000121 / 0.000054 (0.000066) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029337 / 0.037411 (-0.008075) | 0.081644 / 0.014526 (0.067118) | 0.103326 / 0.176557 (-0.073230) | 0.155299 / 0.737135 (-0.581836) | 0.093518 / 0.296338 (-0.202821) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.517979 / 0.215209 (0.302769) | 5.250052 / 2.077655 (3.172397) | 2.220543 / 1.504120 (0.716424) | 1.901087 / 1.541195 (0.359892) | 1.920564 / 1.468490 (0.452073) | 0.766289 / 4.584777 (-3.818488) | 5.130968 / 3.745712 (1.385256) | 4.561874 / 5.269862 (-0.707988) | 2.702808 / 4.565676 (-1.862868) | 0.078929 / 0.424275 (-0.345346) | 0.007834 / 0.007607 (0.000226) | 0.636628 / 0.226044 (0.410583) | 6.309391 / 2.268929 (4.040463) | 2.942180 / 55.444624 (-52.502445) | 2.369557 / 6.876477 (-4.506920) | 2.347528 / 2.142072 (0.205456) | 0.911110 / 4.805227 (-3.894117) | 0.189102 / 6.500664 (-6.311562) | 0.068012 / 0.075469 (-0.007457) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.494431 / 1.841788 (-0.347356) | 22.161476 / 8.074308 (14.087168) | 19.426403 / 10.191392 (9.235011) | 0.211154 / 0.680424 (-0.469270) | 0.030655 / 0.534201 (-0.503546) | 0.440449 / 0.579283 (-0.138834) | 0.526522 / 0.434364 (0.092158) | 0.517494 / 0.540337 (-0.022844) | 0.727387 / 1.386936 (-0.659549) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008354 / 0.011353 (-0.002999) | 0.006108 / 0.011008 (-0.004900) | 0.069079 / 0.038508 (0.030571) | 0.080402 / 0.023109 (0.057292) | 0.452166 / 0.275898 (0.176268) | 0.440264 / 0.323480 (0.116784) | 0.005942 / 0.007986 (-0.002043) | 0.003397 / 0.004328 (-0.000932) | 0.079856 / 0.004250 (0.075606) | 0.056329 / 0.037052 (0.019276) | 0.424261 / 0.258489 (0.165772) | 0.464362 / 0.293841 (0.170521) | 0.051968 / 0.128546 (-0.076578) | 0.015204 / 0.075646 (-0.060442) | 0.085940 / 0.419271 (-0.333332) | 0.066673 / 0.043533 (0.023140) | 0.436481 / 0.255139 (0.181342) | 0.445285 / 0.283200 (0.162085) | 0.035188 / 0.141683 (-0.106495) | 1.579442 / 1.452155 (0.127288) | 1.686120 / 1.492716 (0.193404) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.319039 / 0.018006 (0.301032) | 0.655080 / 0.000490 (0.654591) | 0.005445 / 0.000200 (0.005245) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028566 / 0.037411 (-0.008845) | 0.092131 / 0.014526 (0.077605) | 0.103654 / 0.176557 (-0.072902) | 0.158082 / 0.737135 (-0.579054) | 0.107520 / 0.296338 (-0.188819) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.573479 / 0.215209 (0.358270) | 5.629751 / 2.077655 (3.552096) | 2.501722 / 1.504120 (0.997602) | 2.156255 / 1.541195 (0.615061) | 2.251296 / 1.468490 (0.782805) | 0.767686 / 4.584777 (-3.817091) | 5.080866 / 3.745712 (1.335154) | 4.353351 / 5.269862 (-0.916510) | 2.818707 / 4.565676 (-1.746970) | 0.082617 / 0.424275 (-0.341658) | 0.008045 / 0.007607 (0.000438) | 0.665462 / 0.226044 (0.439417) | 6.961380 / 2.268929 (4.692452) | 3.308717 / 55.444624 (-52.135907) | 2.664239 / 6.876477 (-4.212238) | 2.782790 / 2.142072 (0.640718) | 0.919567 / 4.805227 (-3.885660) | 0.186731 / 6.500664 (-6.313933) | 0.063437 / 0.075469 (-0.012032) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.668076 / 1.841788 (-0.173712) | 22.720187 / 8.074308 (14.645879) | 19.803359 / 10.191392 (9.611967) | 0.237201 / 0.680424 (-0.443223) | 0.041156 / 0.534201 (-0.493045) | 0.458974 / 0.579283 (-0.120309) | 0.620276 / 0.434364 (0.185912) | 0.544079 / 0.540337 (0.003741) | 0.722715 / 1.386936 (-0.664221) |

</details>
</details>

![](https://cml.dev/watermark.png#1ed9306b6c512befb721b681fba3222221c8468e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006882 / 0.011353 (-0.004471) | 0.004238 / 0.011008 (-0.006770) | 0.084042 / 0.038508 (0.045534) | 0.074175 / 0.023109 (0.051065) | 0.308771 / 0.275898 (0.032873) | 0.346300 / 0.323480 (0.022820) | 0.005455 / 0.007986 (-0.002530) | 0.003638 / 0.004328 (-0.000690) | 0.065326 / 0.004250 (0.061076) | 0.056080 / 0.037052 (0.019028) | 0.326324 / 0.258489 (0.067834) | 0.360133 / 0.293841 (0.066292) | 0.031577 / 0.128546 (-0.096969) | 0.008675 / 0.075646 (-0.066971) | 0.288051 / 0.419271 (-0.131221) | 0.052769 / 0.043533 (0.009236) | 0.308689 / 0.255139 (0.053550) | 0.328270 / 0.283200 (0.045070) | 0.025028 / 0.141683 (-0.116655) | 1.520670 / 1.452155 (0.068515) | 1.585229 / 1.492716 (0.092513) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.284078 / 0.018006 (0.266072) | 0.558134 / 0.000490 (0.557644) | 0.015042 / 0.000200 (0.014842) | 0.000429 / 0.000054 (0.000375) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028747 / 0.037411 (-0.008664) | 0.083816 / 0.014526 (0.069290) | 0.207467 / 0.176557 (0.030911) | 0.163527 / 0.737135 (-0.573608) | 0.100148 / 0.296338 (-0.196190) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.376109 / 0.215209 (0.160900) | 3.749639 / 2.077655 (1.671984) | 1.827081 / 1.504120 (0.322961) | 1.662021 / 1.541195 (0.120827) | 1.734655 / 1.468490 (0.266165) | 0.483701 / 4.584777 (-4.101075) | 3.454772 / 3.745712 (-0.290941) | 3.465079 / 5.269862 (-1.804783) | 2.070874 / 4.565676 (-2.494802) | 0.056714 / 0.424275 (-0.367561) | 0.007786 / 0.007607 (0.000179) | 0.455980 / 0.226044 (0.229936) | 4.530612 / 2.268929 (2.261683) | 2.345757 / 55.444624 (-53.098867) | 2.030289 / 6.876477 (-4.846188) | 2.068440 / 2.142072 (-0.073632) | 0.576502 / 4.805227 (-4.228725) | 0.131787 / 6.500664 (-6.368878) | 0.060038 / 0.075469 (-0.015431) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272225 / 1.841788 (-0.569563) | 19.373635 / 8.074308 (11.299327) | 14.167831 / 10.191392 (3.976439) | 0.166336 / 0.680424 (-0.514088) | 0.018420 / 0.534201 (-0.515781) | 0.387878 / 0.579283 (-0.191405) | 0.413105 / 0.434364 (-0.021259) | 0.458618 / 0.540337 (-0.081720) | 0.639031 / 1.386936 (-0.747905) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007122 / 0.011353 (-0.004230) | 0.004193 / 0.011008 (-0.006815) | 0.066194 / 0.038508 (0.027686) | 0.077775 / 0.023109 (0.054666) | 0.349780 / 0.275898 (0.073882) | 0.383417 / 0.323480 (0.059937) | 0.006416 / 0.007986 (-0.001570) | 0.003651 / 0.004328 (-0.000677) | 0.064837 / 0.004250 (0.060587) | 0.058012 / 0.037052 (0.020959) | 0.351085 / 0.258489 (0.092596) | 0.387302 / 0.293841 (0.093462) | 0.032447 / 0.128546 (-0.096099) | 0.008636 / 0.075646 (-0.067011) | 0.071962 / 0.419271 (-0.347309) | 0.047839 / 0.043533 (0.004306) | 0.349508 / 0.255139 (0.094369) | 0.361892 / 0.283200 (0.078693) | 0.024129 / 0.141683 (-0.117554) | 1.523828 / 1.452155 (0.071673) | 1.607371 / 1.492716 (0.114655) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245928 / 0.018006 (0.227922) | 0.567708 / 0.000490 (0.567218) | 0.003789 / 0.000200 (0.003589) | 0.000092 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034107 / 0.037411 (-0.003304) | 0.092539 / 0.014526 (0.078014) | 0.110735 / 0.176557 (-0.065821) | 0.163251 / 0.737135 (-0.573884) | 0.110353 / 0.296338 (-0.185985) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399992 / 0.215209 (0.184783) | 3.976526 / 2.077655 (1.898872) | 2.056182 / 1.504120 (0.552062) | 1.856624 / 1.541195 (0.315429) | 1.941540 / 1.468490 (0.473050) | 0.484662 / 4.584777 (-4.100115) | 3.548228 / 3.745712 (-0.197484) | 3.352900 / 5.269862 (-1.916962) | 2.056310 / 4.565676 (-2.509366) | 0.056952 / 0.424275 (-0.367323) | 0.007284 / 0.007607 (-0.000323) | 0.473749 / 0.226044 (0.247704) | 4.736510 / 2.268929 (2.467581) | 2.570711 / 55.444624 (-52.873913) | 2.204237 / 6.876477 (-4.672239) | 2.438512 / 2.142072 (0.296439) | 0.575542 / 4.805227 (-4.229685) | 0.129260 / 6.500664 (-6.371404) | 0.057704 / 0.075469 (-0.017765) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.316659 / 1.841788 (-0.525128) | 20.103340 / 8.074308 (12.029032) | 14.488385 / 10.191392 (4.296993) | 0.171841 / 0.680424 (-0.508583) | 0.020148 / 0.534201 (-0.514053) | 0.398456 / 0.579283 (-0.180828) | 0.443516 / 0.434364 (0.009152) | 0.479597 / 0.540337 (-0.060741) | 0.643665 / 1.386936 (-0.743271) |

</details>
</details>

![](https://cml.dev/watermark.png#370be814b0c18769ea8e699e3647fadcf431e6df "CML watermark")