Issue: Fix ArrayXD YAML conversion
State: closed
Author: mariosasko

Replace the `shape` tuple with a list in the `ArrayXD` YAML conversion. 

Fix #6112 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6168). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009350 / 0.011353 (-0.002003) | 0.005658 / 0.011008 (-0.005350) | 0.123173 / 0.038508 (0.084664) | 0.096354 / 0.023109 (0.073244) | 0.464398 / 0.275898 (0.188500) | 0.544455 / 0.323480 (0.220975) | 0.007337 / 0.007986 (-0.000648) | 0.004424 / 0.004328 (0.000096) | 0.089715 / 0.004250 (0.085465) | 0.072462 / 0.037052 (0.035410) | 0.460601 / 0.258489 (0.202112) | 0.544384 / 0.293841 (0.250543) | 0.052994 / 0.128546 (-0.075552) | 0.014459 / 0.075646 (-0.061187) | 0.464368 / 0.419271 (0.045096) | 0.072889 / 0.043533 (0.029356) | 0.471387 / 0.255139 (0.216248) | 0.560982 / 0.283200 (0.277783) | 0.041398 / 0.141683 (-0.100285) | 1.964688 / 1.452155 (0.512533) | 2.240727 / 1.492716 (0.748011) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.308524 / 0.018006 (0.290518) | 0.669306 / 0.000490 (0.668816) | 0.006644 / 0.000200 (0.006444) | 0.000108 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037395 / 0.037411 (-0.000016) | 0.111303 / 0.014526 (0.096777) | 0.158988 / 0.176557 (-0.017569) | 0.236155 / 0.737135 (-0.500980) | 0.134775 / 0.296338 (-0.161564) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.648830 / 0.215209 (0.433621) | 6.614794 / 2.077655 (4.537139) | 2.867526 / 1.504120 (1.363407) | 2.472967 / 1.541195 (0.931772) | 2.488419 / 1.468490 (1.019929) | 0.915785 / 4.584777 (-3.668992) | 6.010754 / 3.745712 (2.265042) | 5.468873 / 5.269862 (0.199011) | 3.446535 / 4.565676 (-1.119141) | 0.118592 / 0.424275 (-0.305684) | 0.012005 / 0.007607 (0.004398) | 0.808467 / 0.226044 (0.582423) | 8.152122 / 2.268929 (5.883193) | 3.751282 / 55.444624 (-51.693342) | 3.009569 / 6.876477 (-3.866908) | 3.282613 / 2.142072 (1.140540) | 1.152727 / 4.805227 (-3.652500) | 0.240224 / 6.500664 (-6.260440) | 0.097871 / 0.075469 (0.022402) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.824944 / 1.841788 (-0.016843) | 27.840842 / 8.074308 (19.766533) | 24.368669 / 10.191392 (14.177277) | 0.260621 / 0.680424 (-0.419803) | 0.033730 / 0.534201 (-0.500471) | 0.552494 / 0.579283 (-0.026789) | 0.666921 / 0.434364 (0.232557) | 0.648812 / 0.540337 (0.108475) | 0.912602 / 1.386936 (-0.474334) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011688 / 0.011353 (0.000335) | 0.005794 / 0.011008 (-0.005215) | 0.093466 / 0.038508 (0.054958) | 0.102583 / 0.023109 (0.079474) | 0.593572 / 0.275898 (0.317674) | 0.614351 / 0.323480 (0.290871) | 0.007006 / 0.007986 (-0.000980) | 0.005557 / 0.004328 (0.001229) | 0.087779 / 0.004250 (0.083529) | 0.072639 / 0.037052 (0.035586) | 0.577464 / 0.258489 (0.318975) | 0.628240 / 0.293841 (0.334399) | 0.053876 / 0.128546 (-0.074670) | 0.015383 / 0.075646 (-0.060263) | 0.110633 / 0.419271 (-0.308639) | 0.067467 / 0.043533 (0.023934) | 0.613457 / 0.255139 (0.358318) | 0.604939 / 0.283200 (0.321739) | 0.041738 / 0.141683 (-0.099945) | 1.967167 / 1.452155 (0.515012) | 2.121009 / 1.492716 (0.628293) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.449937 / 0.018006 (0.431930) | 0.694410 / 0.000490 (0.693921) | 0.064051 / 0.000200 (0.063851) | 0.000810 / 0.000054 (0.000756) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.045138 / 0.037411 (0.007727) | 0.116831 / 0.014526 (0.102306) | 0.131906 / 0.176557 (-0.044651) | 0.202421 / 0.737135 (-0.534714) | 0.132568 / 0.296338 (-0.163770) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.698046 / 0.215209 (0.482837) | 7.112591 / 2.077655 (5.034936) | 3.332679 / 1.504120 (1.828559) | 2.946384 / 1.541195 (1.405189) | 3.074484 / 1.468490 (1.605994) | 0.970917 / 4.584777 (-3.613859) | 6.143506 / 3.745712 (2.397794) | 5.572496 / 5.269862 (0.302634) | 3.602673 / 4.565676 (-0.963004) | 0.115068 / 0.424275 (-0.309207) | 0.009971 / 0.007607 (0.002364) | 0.891090 / 0.226044 (0.665046) | 8.761788 / 2.268929 (6.492859) | 4.362685 / 55.444624 (-51.081939) | 3.612893 / 6.876477 (-3.263583) | 3.797948 / 2.142072 (1.655876) | 1.202890 / 4.805227 (-3.602337) | 0.238120 / 6.500664 (-6.262544) | 0.095612 / 0.075469 (0.020143) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.958880 / 1.841788 (0.117092) | 28.216454 / 8.074308 (20.142146) | 25.361424 / 10.191392 (15.170032) | 0.308203 / 0.680424 (-0.372221) | 0.032903 / 0.534201 (-0.501298) | 0.539714 / 0.579283 (-0.039569) | 0.688278 / 0.434364 (0.253914) | 0.644818 / 0.540337 (0.104481) | 0.905694 / 1.386936 (-0.481242) |

</details>
</details>

![](https://cml.dev/watermark.png#a5289345e5b23548fee680a0bbc047c0b9a5ee8c "CML watermark")


Maybe convert all the tuples by default instead of hardcoding a logic specific to ArrayXD ?

@mariosasko Have you been able to fix this issue ? we're having quite a rough time updating our dataset lately

@lhoestq Does it look good now?

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005519 / 0.011353 (-0.005834) | 0.003482 / 0.011008 (-0.007527) | 0.064904 / 0.038508 (0.026396) | 0.052399 / 0.023109 (0.029289) | 0.247238 / 0.275898 (-0.028660) | 0.273426 / 0.323480 (-0.050054) | 0.003102 / 0.007986 (-0.004884) | 0.003420 / 0.004328 (-0.000908) | 0.048029 / 0.004250 (0.043779) | 0.039378 / 0.037052 (0.002326) | 0.253809 / 0.258489 (-0.004680) | 0.287483 / 0.293841 (-0.006358) | 0.028096 / 0.128546 (-0.100450) | 0.010806 / 0.075646 (-0.064841) | 0.207799 / 0.419271 (-0.211472) | 0.035861 / 0.043533 (-0.007672) | 0.251912 / 0.255139 (-0.003227) | 0.278877 / 0.283200 (-0.004323) | 0.019498 / 0.141683 (-0.122185) | 1.104916 / 1.452155 (-0.347238) | 1.157376 / 1.492716 (-0.335340) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093051 / 0.018006 (0.075045) | 0.303331 / 0.000490 (0.302841) | 0.000209 / 0.000200 (0.000009) | 0.000044 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018052 / 0.037411 (-0.019359) | 0.060597 / 0.014526 (0.046071) | 0.074033 / 0.176557 (-0.102524) | 0.120966 / 0.737135 (-0.616169) | 0.075012 / 0.296338 (-0.221326) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.283922 / 0.215209 (0.068713) | 2.805445 / 2.077655 (0.727791) | 1.498292 / 1.504120 (-0.005828) | 1.371686 / 1.541195 (-0.169509) | 1.402074 / 1.468490 (-0.066416) | 0.567231 / 4.584777 (-4.017546) | 2.393291 / 3.745712 (-1.352422) | 2.800329 / 5.269862 (-2.469533) | 1.789197 / 4.565676 (-2.776479) | 0.063620 / 0.424275 (-0.360655) | 0.005008 / 0.007607 (-0.002600) | 0.338929 / 0.226044 (0.112884) | 3.292122 / 2.268929 (1.023193) | 1.813313 / 55.444624 (-53.631311) | 1.557122 / 6.876477 (-5.319354) | 1.576395 / 2.142072 (-0.565677) | 0.666714 / 4.805227 (-4.138513) | 0.118253 / 6.500664 (-6.382411) | 0.042633 / 0.075469 (-0.032836) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.950678 / 1.841788 (-0.891110) | 11.589806 / 8.074308 (3.515498) | 10.436701 / 10.191392 (0.245309) | 0.141048 / 0.680424 (-0.539376) | 0.014766 / 0.534201 (-0.519435) | 0.298359 / 0.579283 (-0.280924) | 0.268850 / 0.434364 (-0.165514) | 0.340242 / 0.540337 (-0.200095) | 0.451447 / 1.386936 (-0.935489) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005405 / 0.011353 (-0.005948) | 0.003545 / 0.011008 (-0.007464) | 0.048959 / 0.038508 (0.010451) | 0.056565 / 0.023109 (0.033455) | 0.274289 / 0.275898 (-0.001609) | 0.296565 / 0.323480 (-0.026915) | 0.004790 / 0.007986 (-0.003196) | 0.002772 / 0.004328 (-0.001557) | 0.048605 / 0.004250 (0.044354) | 0.040676 / 0.037052 (0.003624) | 0.279949 / 0.258489 (0.021460) | 0.312816 / 0.293841 (0.018976) | 0.029605 / 0.128546 (-0.098941) | 0.010799 / 0.075646 (-0.064848) | 0.056941 / 0.419271 (-0.362331) | 0.034518 / 0.043533 (-0.009014) | 0.277193 / 0.255139 (0.022054) | 0.292334 / 0.283200 (0.009134) | 0.018836 / 0.141683 (-0.122847) | 1.145228 / 1.452155 (-0.306927) | 1.198958 / 1.492716 (-0.293758) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093618 / 0.018006 (0.075612) | 0.303687 / 0.000490 (0.303197) | 0.000234 / 0.000200 (0.000034) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021347 / 0.037411 (-0.016065) | 0.067811 / 0.014526 (0.053286) | 0.080631 / 0.176557 (-0.095926) | 0.119289 / 0.737135 (-0.617846) | 0.082085 / 0.296338 (-0.214254) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.293374 / 0.215209 (0.078165) | 2.864516 / 2.077655 (0.786861) | 1.611042 / 1.504120 (0.106922) | 1.466124 / 1.541195 (-0.075071) | 1.480509 / 1.468490 (0.012019) | 0.569463 / 4.584777 (-4.015314) | 2.448181 / 3.745712 (-1.297531) | 2.841732 / 5.269862 (-2.428130) | 1.754458 / 4.565676 (-2.811219) | 0.063771 / 0.424275 (-0.360505) | 0.004976 / 0.007607 (-0.002631) | 0.346094 / 0.226044 (0.120050) | 3.440090 / 2.268929 (1.171162) | 1.961862 / 55.444624 (-53.482763) | 1.675780 / 6.876477 (-5.200697) | 1.679676 / 2.142072 (-0.462396) | 0.641063 / 4.805227 (-4.164164) | 0.116268 / 6.500664 (-6.384396) | 0.041767 / 0.075469 (-0.033702) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980286 / 1.841788 (-0.861502) | 12.055227 / 8.074308 (3.980919) | 10.685417 / 10.191392 (0.494025) | 0.140842 / 0.680424 (-0.539582) | 0.015413 / 0.534201 (-0.518788) | 0.286939 / 0.579283 (-0.292344) | 0.278796 / 0.434364 (-0.155568) | 0.326740 / 0.540337 (-0.213597) | 0.574516 / 1.386936 (-0.812421) |

</details>
</details>

![](https://cml.dev/watermark.png#53b8c2d9d3714f4a56d74637c0840a90cbfb7aa0 "CML watermark")