Add Arrow builder docs
State: closed | Author: lhoestq (MEMBER)

following https://github.com/huggingface/datasets/pull/5944

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006522 / 0.011353 (-0.004831) | 0.004319 / 0.011008 (-0.006690) | 0.099280 / 0.038508 (0.060772) | 0.033117 / 0.023109 (0.010007) | 0.339392 / 0.275898 (0.063494) | 0.366219 / 0.323480 (0.042739) | 0.003896 / 0.007986 (-0.004090) | 0.003412 / 0.004328 (-0.000916) | 0.076655 / 0.004250 (0.072404) | 0.045203 / 0.037052 (0.008150) | 0.355800 / 0.258489 (0.097311) | 0.372533 / 0.293841 (0.078692) | 0.032318 / 0.128546 (-0.096229) | 0.009030 / 0.075646 (-0.066616) | 0.328701 / 0.419271 (-0.090571) | 0.052891 / 0.043533 (0.009358) | 0.341131 / 0.255139 (0.085992) | 0.351593 / 0.283200 (0.068393) | 0.105136 / 0.141683 (-0.036546) | 1.475953 / 1.452155 (0.023798) | 1.566074 / 1.492716 (0.073357) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216671 / 0.018006 (0.198664) | 0.446952 / 0.000490 (0.446462) | 0.006340 / 0.000200 (0.006140) | 0.000096 / 0.000054 (0.000042) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028293 / 0.037411 (-0.009118) | 0.112298 / 0.014526 (0.097773) | 0.118634 / 0.176557 (-0.057923) | 0.175542 / 0.737135 (-0.561593) | 0.124773 / 0.296338 (-0.171565) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435209 / 0.215209 (0.220000) | 4.344361 / 2.077655 (2.266706) | 2.128943 / 1.504120 (0.624823) | 1.945465 / 1.541195 (0.404271) | 2.049932 / 1.468490 (0.581442) | 0.547126 / 4.584777 (-4.037651) | 3.768698 / 3.745712 (0.022986) | 1.924441 / 5.269862 (-3.345420) | 1.146364 / 4.565676 (-3.419312) | 0.067466 / 0.424275 (-0.356809) | 0.011175 / 0.007607 (0.003568) | 0.540978 / 0.226044 (0.314933) | 5.393120 / 2.268929 (3.124191) | 2.639027 / 55.444624 (-52.805597) | 2.327216 / 6.876477 (-4.549261) | 2.500532 / 2.142072 (0.358460) | 0.679120 / 4.805227 (-4.126107) | 0.148824 / 6.500664 (-6.351840) | 0.064195 / 0.075469 (-0.011274) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.158387 / 1.841788 (-0.683401) | 14.880751 / 8.074308 (6.806443) | 14.725249 / 10.191392 (4.533857) | 0.149785 / 0.680424 (-0.530639) | 0.017338 / 0.534201 (-0.516863) | 0.390980 / 0.579283 (-0.188303) | 0.425611 / 0.434364 (-0.008753) | 0.458851 / 0.540337 (-0.081487) | 0.559209 / 1.386936 (-0.827727) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006835 / 0.011353 (-0.004518) | 0.004318 / 0.011008 (-0.006690) | 0.076715 / 0.038508 (0.038207) | 0.033528 / 0.023109 (0.010419) | 0.411986 / 0.275898 (0.136087) | 0.438752 / 0.323480 (0.115272) | 0.004039 / 0.007986 (-0.003947) | 0.003509 / 0.004328 (-0.000819) | 0.077924 / 0.004250 (0.073673) | 0.049519 / 0.037052 (0.012467) | 0.420595 / 0.258489 (0.162106) | 0.450536 / 0.293841 (0.156695) | 0.032817 / 0.128546 (-0.095729) | 0.008963 / 0.075646 (-0.066684) | 0.083818 / 0.419271 (-0.335454) | 0.057591 / 0.043533 (0.014058) | 0.404605 / 0.255139 (0.149466) | 0.423661 / 0.283200 (0.140462) | 0.110698 / 0.141683 (-0.030984) | 1.512515 / 1.452155 (0.060361) | 1.569207 / 1.492716 (0.076490) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.200795 / 0.018006 (0.182789) | 0.448853 / 0.000490 (0.448363) | 0.003657 / 0.000200 (0.003457) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031612 / 0.037411 (-0.005799) | 0.116712 / 0.014526 (0.102186) | 0.126162 / 0.176557 (-0.050395) | 0.180522 / 0.737135 (-0.556614) | 0.129768 / 0.296338 (-0.166570) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.433797 / 0.215209 (0.218588) | 4.353099 / 2.077655 (2.275444) | 2.117582 / 1.504120 (0.613462) | 1.934487 / 1.541195 (0.393292) | 2.016988 / 1.468490 (0.548498) | 0.531387 / 4.584777 (-4.053390) | 3.843520 / 3.745712 (0.097807) | 1.879560 / 5.269862 (-3.390301) | 1.129445 / 4.565676 (-3.436231) | 0.065952 / 0.424275 (-0.358323) | 0.011566 / 0.007607 (0.003959) | 0.533949 / 0.226044 (0.307904) | 5.327447 / 2.268929 (3.058518) | 2.572202 / 55.444624 (-52.872422) | 2.240723 / 6.876477 (-4.635753) | 2.329290 / 2.142072 (0.187217) | 0.662162 / 4.805227 (-4.143066) | 0.143191 / 6.500664 (-6.357473) | 0.065273 / 0.075469 (-0.010196) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.274945 / 1.841788 (-0.566843) | 15.444511 / 8.074308 (7.370203) | 14.793524 / 10.191392 (4.602132) | 0.175607 / 0.680424 (-0.504817) | 0.017324 / 0.534201 (-0.516877) | 0.396172 / 0.579283 (-0.183111) | 0.437334 / 0.434364 (0.002970) | 0.472621 / 0.540337 (-0.067716) | 0.574888 / 1.386936 (-0.812048) |

</details>
</details>

![](https://cml.dev/watermark.png#b4ab1b3ed7257b0e0ad075d7271a51835f320a5e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006976 / 0.011353 (-0.004377) | 0.004541 / 0.011008 (-0.006467) | 0.106085 / 0.038508 (0.067577) | 0.029148 / 0.023109 (0.006039) | 0.306386 / 0.275898 (0.030488) | 0.351474 / 0.323480 (0.027994) | 0.003924 / 0.007986 (-0.004062) | 0.004588 / 0.004328 (0.000260) | 0.090479 / 0.004250 (0.086229) | 0.041195 / 0.037052 (0.004142) | 0.346020 / 0.258489 (0.087531) | 0.362526 / 0.293841 (0.068685) | 0.041020 / 0.128546 (-0.087526) | 0.012536 / 0.075646 (-0.063110) | 0.333247 / 0.419271 (-0.086024) | 0.059786 / 0.043533 (0.016253) | 0.318094 / 0.255139 (0.062955) | 0.343879 / 0.283200 (0.060679) | 0.110083 / 0.141683 (-0.031600) | 1.514027 / 1.452155 (0.061872) | 1.551435 / 1.492716 (0.058719) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235401 / 0.018006 (0.217395) | 0.544292 / 0.000490 (0.543803) | 0.005284 / 0.000200 (0.005084) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025008 / 0.037411 (-0.012403) | 0.102235 / 0.014526 (0.087709) | 0.105523 / 0.176557 (-0.071034) | 0.180846 / 0.737135 (-0.556289) | 0.107078 / 0.296338 (-0.189261) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.502374 / 0.215209 (0.287165) | 5.224254 / 2.077655 (3.146600) | 1.987193 / 1.504120 (0.483073) | 1.694680 / 1.541195 (0.153485) | 1.663907 / 1.468490 (0.195417) | 0.786470 / 4.584777 (-3.798307) | 4.977895 / 3.745712 (1.232183) | 4.713451 / 5.269862 (-0.556410) | 2.298763 / 4.565676 (-2.266913) | 0.090225 / 0.424275 (-0.334051) | 0.011427 / 0.007607 (0.003820) | 0.640686 / 0.226044 (0.414641) | 6.351727 / 2.268929 (4.082798) | 2.636912 / 55.444624 (-52.807712) | 2.075566 / 6.876477 (-4.800911) | 2.080260 / 2.142072 (-0.061812) | 0.952727 / 4.805227 (-3.852500) | 0.188651 / 6.500664 (-6.312013) | 0.068997 / 0.075469 (-0.006472) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.258878 / 1.841788 (-0.582910) | 15.444724 / 8.074308 (7.370416) | 17.521918 / 10.191392 (7.330526) | 0.189732 / 0.680424 (-0.490692) | 0.031084 / 0.534201 (-0.503117) | 0.445150 / 0.579283 (-0.134133) | 0.575844 / 0.434364 (0.141480) | 0.498162 / 0.540337 (-0.042176) | 0.635885 / 1.386936 (-0.751051) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007402 / 0.011353 (-0.003951) | 0.005058 / 0.011008 (-0.005950) | 0.077659 / 0.038508 (0.039151) | 0.034934 / 0.023109 (0.011825) | 0.373139 / 0.275898 (0.097241) | 0.411857 / 0.323480 (0.088377) | 0.003751 / 0.007986 (-0.004235) | 0.003634 / 0.004328 (-0.000695) | 0.075914 / 0.004250 (0.071663) | 0.037555 / 0.037052 (0.000503) | 0.387482 / 0.258489 (0.128993) | 0.434407 / 0.293841 (0.140566) | 0.040540 / 0.128546 (-0.088006) | 0.013458 / 0.075646 (-0.062189) | 0.096129 / 0.419271 (-0.323143) | 0.055369 / 0.043533 (0.011836) | 0.386564 / 0.255139 (0.131425) | 0.410417 / 0.283200 (0.127218) | 0.093265 / 0.141683 (-0.048418) | 1.432841 / 1.452155 (-0.019314) | 1.533180 / 1.492716 (0.040463) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.281051 / 0.018006 (0.263045) | 0.547635 / 0.000490 (0.547146) | 0.004434 / 0.000200 (0.004234) | 0.000105 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026409 / 0.037411 (-0.011002) | 0.098586 / 0.014526 (0.084060) | 0.109223 / 0.176557 (-0.067334) | 0.165958 / 0.737135 (-0.571177) | 0.111751 / 0.296338 (-0.184587) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.542717 / 0.215209 (0.327508) | 5.530075 / 2.077655 (3.452420) | 2.351141 / 1.504120 (0.847022) | 2.021659 / 1.541195 (0.480464) | 1.964900 / 1.468490 (0.496410) | 0.819698 / 4.584777 (-3.765079) | 4.917412 / 3.745712 (1.171700) | 2.425149 / 5.269862 (-2.844712) | 1.561953 / 4.565676 (-3.003724) | 0.098417 / 0.424275 (-0.325858) | 0.012594 / 0.007607 (0.004986) | 0.717212 / 0.226044 (0.491168) | 6.994833 / 2.268929 (4.725904) | 2.997347 / 55.444624 (-52.447277) | 2.388366 / 6.876477 (-4.488111) | 2.502913 / 2.142072 (0.360841) | 1.030545 / 4.805227 (-3.774682) | 0.184844 / 6.500664 (-6.315820) | 0.076889 / 0.075469 (0.001420) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.371647 / 1.841788 (-0.470141) | 15.522995 / 8.074308 (7.448687) | 17.349823 / 10.191392 (7.158431) | 0.229709 / 0.680424 (-0.450714) | 0.023303 / 0.534201 (-0.510898) | 0.413874 / 0.579283 (-0.165409) | 0.567552 / 0.434364 (0.133188) | 0.491722 / 0.540337 (-0.048615) | 0.590640 / 1.386936 (-0.796296) |

</details>
</details>

![](https://cml.dev/watermark.png#f1911ffa5d1f58f509d04fe1ddeb9d00a63f94d5 "CML watermark")