Raise error in `DatasetBuilder.as_dataset` when `file_format` is not `"arrow"`
Opened by mariosasko (closed)

Raise an error in  `DatasetBuilder.as_dataset` when `file_format != "arrow"` (and fix the docstring)

Fix #5874 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006416 / 0.011353 (-0.004937) | 0.004278 / 0.011008 (-0.006731) | 0.097562 / 0.038508 (0.059054) | 0.029488 / 0.023109 (0.006379) | 0.308648 / 0.275898 (0.032750) | 0.339879 / 0.323480 (0.016399) | 0.005288 / 0.007986 (-0.002697) | 0.005033 / 0.004328 (0.000704) | 0.074666 / 0.004250 (0.070416) | 0.034888 / 0.037052 (-0.002164) | 0.309960 / 0.258489 (0.051471) | 0.344276 / 0.293841 (0.050435) | 0.025564 / 0.128546 (-0.102982) | 0.008579 / 0.075646 (-0.067067) | 0.319796 / 0.419271 (-0.099476) | 0.044786 / 0.043533 (0.001253) | 0.308888 / 0.255139 (0.053749) | 0.334001 / 0.283200 (0.050802) | 0.089917 / 0.141683 (-0.051766) | 1.456696 / 1.452155 (0.004541) | 1.542273 / 1.492716 (0.049557) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.213236 / 0.018006 (0.195230) | 0.425139 / 0.000490 (0.424650) | 0.008831 / 0.000200 (0.008631) | 0.000209 / 0.000054 (0.000155) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023990 / 0.037411 (-0.013421) | 0.096787 / 0.014526 (0.082261) | 0.105783 / 0.176557 (-0.070774) | 0.167182 / 0.737135 (-0.569954) | 0.108896 / 0.296338 (-0.187442) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419844 / 0.215209 (0.204635) | 4.201909 / 2.077655 (2.124254) | 1.910784 / 1.504120 (0.406664) | 1.685183 / 1.541195 (0.143988) | 1.716927 / 1.468490 (0.248437) | 0.548261 / 4.584777 (-4.036516) | 3.414168 / 3.745712 (-0.331544) | 1.695446 / 5.269862 (-3.574415) | 0.989668 / 4.565676 (-3.576008) | 0.067328 / 0.424275 (-0.356948) | 0.012084 / 0.007607 (0.004477) | 0.523799 / 0.226044 (0.297754) | 5.240589 / 2.268929 (2.971661) | 2.331618 / 55.444624 (-53.113007) | 1.996094 / 6.876477 (-4.880383) | 2.105450 / 2.142072 (-0.036623) | 0.654614 / 4.805227 (-4.150613) | 0.134721 / 6.500664 (-6.365943) | 0.066227 / 0.075469 (-0.009242) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196266 / 1.841788 (-0.645521) | 13.990045 / 8.074308 (5.915737) | 13.928126 / 10.191392 (3.736734) | 0.142600 / 0.680424 (-0.537824) | 0.016462 / 0.534201 (-0.517739) | 0.363113 / 0.579283 (-0.216170) | 0.428590 / 0.434364 (-0.005773) | 0.452594 / 0.540337 (-0.087743) | 0.551678 / 1.386936 (-0.835258) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005992 / 0.011353 (-0.005361) | 0.004161 / 0.011008 (-0.006847) | 0.076098 / 0.038508 (0.037589) | 0.028559 / 0.023109 (0.005450) | 0.411696 / 0.275898 (0.135798) | 0.444519 / 0.323480 (0.121040) | 0.004965 / 0.007986 (-0.003021) | 0.003452 / 0.004328 (-0.000876) | 0.075107 / 0.004250 (0.070857) | 0.037305 / 0.037052 (0.000252) | 0.429728 / 0.258489 (0.171239) | 0.444313 / 0.293841 (0.150472) | 0.025278 / 0.128546 (-0.103268) | 0.008527 / 0.075646 (-0.067120) | 0.081502 / 0.419271 (-0.337770) | 0.041237 / 0.043533 (-0.002296) | 0.417848 / 0.255139 (0.162709) | 0.426615 / 0.283200 (0.143415) | 0.094641 / 0.141683 (-0.047041) | 1.525141 / 1.452155 (0.072987) | 1.615608 / 1.492716 (0.122892) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.192867 / 0.018006 (0.174861) | 0.414979 / 0.000490 (0.414490) | 0.000815 / 0.000200 (0.000615) | 0.000068 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025354 / 0.037411 (-0.012058) | 0.102085 / 0.014526 (0.087559) | 0.107930 / 0.176557 (-0.068626) | 0.160483 / 0.737135 (-0.576652) | 0.112341 / 0.296338 (-0.183997) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.446938 / 0.215209 (0.231728) | 4.480057 / 2.077655 (2.402402) | 2.154825 / 1.504120 (0.650705) | 1.942774 / 1.541195 (0.401580) | 1.996418 / 1.468490 (0.527928) | 0.556728 / 4.584777 (-4.028049) | 3.441228 / 3.745712 (-0.304484) | 3.004179 / 5.269862 (-2.265683) | 1.314104 / 4.565676 (-3.251573) | 0.068670 / 0.424275 (-0.355606) | 0.011972 / 0.007607 (0.004365) | 0.556604 / 0.226044 (0.330560) | 5.561783 / 2.268929 (3.292855) | 2.631262 / 55.444624 (-52.813363) | 2.262143 / 6.876477 (-4.614333) | 2.364243 / 2.142072 (0.222170) | 0.660621 / 4.805227 (-4.144607) | 0.137371 / 6.500664 (-6.363293) | 0.069104 / 0.075469 (-0.006365) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.305706 / 1.841788 (-0.536081) | 14.015932 / 8.074308 (5.941624) | 14.353580 / 10.191392 (4.162187) | 0.146172 / 0.680424 (-0.534251) | 0.016699 / 0.534201 (-0.517502) | 0.357970 / 0.579283 (-0.221313) | 0.389067 / 0.434364 (-0.045297) | 0.415470 / 0.540337 (-0.124867) | 0.501359 / 1.386936 (-0.885577) |

</details>
</details>

![](https://cml.dev/watermark.png#b2b837b4e7267db9e32d2613d8bf8d70d2ce0b47 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006800 / 0.011353 (-0.004552) | 0.004721 / 0.011008 (-0.006287) | 0.097760 / 0.038508 (0.059252) | 0.034192 / 0.023109 (0.011083) | 0.298240 / 0.275898 (0.022342) | 0.331119 / 0.323480 (0.007639) | 0.005826 / 0.007986 (-0.002160) | 0.003968 / 0.004328 (-0.000360) | 0.073833 / 0.004250 (0.069582) | 0.046288 / 0.037052 (0.009236) | 0.303018 / 0.258489 (0.044529) | 0.342163 / 0.293841 (0.048322) | 0.028504 / 0.128546 (-0.100042) | 0.009031 / 0.075646 (-0.066615) | 0.331617 / 0.419271 (-0.087655) | 0.060911 / 0.043533 (0.017379) | 0.304044 / 0.255139 (0.048905) | 0.328959 / 0.283200 (0.045759) | 0.113174 / 0.141683 (-0.028509) | 1.424652 / 1.452155 (-0.027502) | 1.531392 / 1.492716 (0.038676) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206175 / 0.018006 (0.188169) | 0.435916 / 0.000490 (0.435426) | 0.002587 / 0.000200 (0.002387) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026996 / 0.037411 (-0.010415) | 0.106722 / 0.014526 (0.092196) | 0.117655 / 0.176557 (-0.058902) | 0.176969 / 0.737135 (-0.560166) | 0.122577 / 0.296338 (-0.173762) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.396086 / 0.215209 (0.180877) | 3.972465 / 2.077655 (1.894811) | 1.800798 / 1.504120 (0.296678) | 1.616747 / 1.541195 (0.075552) | 1.680711 / 1.468490 (0.212221) | 0.526479 / 4.584777 (-4.058298) | 3.791528 / 3.745712 (0.045816) | 2.989518 / 5.269862 (-2.280344) | 1.463221 / 4.565676 (-3.102455) | 0.065649 / 0.424275 (-0.358626) | 0.012155 / 0.007607 (0.004548) | 0.500241 / 0.226044 (0.274197) | 5.008895 / 2.268929 (2.739966) | 2.315288 / 55.444624 (-53.129336) | 1.959409 / 6.876477 (-4.917067) | 2.102371 / 2.142072 (-0.039701) | 0.639611 / 4.805227 (-4.165617) | 0.140101 / 6.500664 (-6.360563) | 0.063599 / 0.075469 (-0.011870) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.206729 / 1.841788 (-0.635059) | 15.127250 / 8.074308 (7.052942) | 14.397228 / 10.191392 (4.205836) | 0.148802 / 0.680424 (-0.531622) | 0.017628 / 0.534201 (-0.516573) | 0.396150 / 0.579283 (-0.183133) | 0.435826 / 0.434364 (0.001462) | 0.471215 / 0.540337 (-0.069122) | 0.559413 / 1.386936 (-0.827523) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006479 / 0.011353 (-0.004874) | 0.004520 / 0.011008 (-0.006488) | 0.074395 / 0.038508 (0.035887) | 0.033400 / 0.023109 (0.010291) | 0.388411 / 0.275898 (0.112513) | 0.396714 / 0.323480 (0.073234) | 0.005736 / 0.007986 (-0.002250) | 0.004038 / 0.004328 (-0.000291) | 0.073595 / 0.004250 (0.069345) | 0.045207 / 0.037052 (0.008155) | 0.378096 / 0.258489 (0.119607) | 0.417830 / 0.293841 (0.123989) | 0.028365 / 0.128546 (-0.100181) | 0.008887 / 0.075646 (-0.066760) | 0.080766 / 0.419271 (-0.338505) | 0.046923 / 0.043533 (0.003390) | 0.376190 / 0.255139 (0.121051) | 0.385875 / 0.283200 (0.102675) | 0.107542 / 0.141683 (-0.034141) | 1.409257 / 1.452155 (-0.042898) | 1.518475 / 1.492716 (0.025759) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223299 / 0.018006 (0.205292) | 0.440640 / 0.000490 (0.440150) | 0.000397 / 0.000200 (0.000197) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031388 / 0.037411 (-0.006024) | 0.113078 / 0.014526 (0.098552) | 0.124398 / 0.176557 (-0.052159) | 0.173802 / 0.737135 (-0.563333) | 0.129555 / 0.296338 (-0.166783) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440220 / 0.215209 (0.225011) | 4.398052 / 2.077655 (2.320398) | 2.188396 / 1.504120 (0.684276) | 1.997811 / 1.541195 (0.456616) | 2.093338 / 1.468490 (0.624847) | 0.519597 / 4.584777 (-4.065180) | 3.885795 / 3.745712 (0.140083) | 2.896327 / 5.269862 (-2.373534) | 1.245785 / 4.565676 (-3.319891) | 0.065675 / 0.424275 (-0.358600) | 0.011729 / 0.007607 (0.004121) | 0.541526 / 0.226044 (0.315482) | 5.406763 / 2.268929 (3.137834) | 2.722914 / 55.444624 (-52.721711) | 2.471111 / 6.876477 (-4.405366) | 2.541488 / 2.142072 (0.399415) | 0.633566 / 4.805227 (-4.171661) | 0.139622 / 6.500664 (-6.361042) | 0.064220 / 0.075469 (-0.011249) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.296097 / 1.841788 (-0.545690) | 15.095320 / 8.074308 (7.021012) | 14.300821 / 10.191392 (4.109429) | 0.145470 / 0.680424 (-0.534954) | 0.017496 / 0.534201 (-0.516705) | 0.400589 / 0.579283 (-0.178694) | 0.423091 / 0.434364 (-0.011273) | 0.468258 / 0.540337 (-0.072079) | 0.570873 / 1.386936 (-0.816063) |

</details>
</details>

![](https://cml.dev/watermark.png#aee6c67034d6ff298b2153a2fcdab97f14ee6d66 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005918 / 0.011353 (-0.005435) | 0.004393 / 0.011008 (-0.006615) | 0.091677 / 0.038508 (0.053169) | 0.033546 / 0.023109 (0.010437) | 0.344682 / 0.275898 (0.068784) | 0.388906 / 0.323480 (0.065426) | 0.005412 / 0.007986 (-0.002574) | 0.004909 / 0.004328 (0.000580) | 0.082589 / 0.004250 (0.078339) | 0.045242 / 0.037052 (0.008190) | 0.339191 / 0.258489 (0.080702) | 0.349673 / 0.293841 (0.055832) | 0.026805 / 0.128546 (-0.101742) | 0.007529 / 0.075646 (-0.068117) | 0.319108 / 0.419271 (-0.100164) | 0.049482 / 0.043533 (0.005949) | 0.320013 / 0.255139 (0.064874) | 0.342059 / 0.283200 (0.058859) | 0.096623 / 0.141683 (-0.045060) | 1.458204 / 1.452155 (0.006049) | 1.571172 / 1.492716 (0.078455) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235171 / 0.018006 (0.217165) | 0.479678 / 0.000490 (0.479188) | 0.006627 / 0.000200 (0.006427) | 0.000257 / 0.000054 (0.000202) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025716 / 0.037411 (-0.011696) | 0.107730 / 0.014526 (0.093204) | 0.111595 / 0.176557 (-0.064962) | 0.171316 / 0.737135 (-0.565819) | 0.118962 / 0.296338 (-0.177377) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.376318 / 0.215209 (0.161109) | 4.039484 / 2.077655 (1.961829) | 1.811548 / 1.504120 (0.307428) | 1.646728 / 1.541195 (0.105533) | 1.688071 / 1.468490 (0.219581) | 0.551256 / 4.584777 (-4.033520) | 4.153931 / 3.745712 (0.408218) | 3.424154 / 5.269862 (-1.845707) | 1.734860 / 4.565676 (-2.830816) | 0.067753 / 0.424275 (-0.356522) | 0.012699 / 0.007607 (0.005092) | 0.505722 / 0.226044 (0.279677) | 4.997321 / 2.268929 (2.728392) | 2.258755 / 55.444624 (-53.185869) | 1.954382 / 6.876477 (-4.922095) | 1.967545 / 2.142072 (-0.174527) | 0.630489 / 4.805227 (-4.174738) | 0.138738 / 6.500664 (-6.361926) | 0.064907 / 0.075469 (-0.010562) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.209634 / 1.841788 (-0.632154) | 15.055062 / 8.074308 (6.980754) | 12.721606 / 10.191392 (2.530214) | 0.164908 / 0.680424 (-0.515516) | 0.019528 / 0.534201 (-0.514673) | 0.400136 / 0.579283 (-0.179147) | 0.451640 / 0.434364 (0.017276) | 0.466272 / 0.540337 (-0.074065) | 0.553258 / 1.386936 (-0.833679) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006341 / 0.011353 (-0.005011) | 0.004617 / 0.011008 (-0.006391) | 0.077953 / 0.038508 (0.039445) | 0.031104 / 0.023109 (0.007995) | 0.360328 / 0.275898 (0.084430) | 0.408403 / 0.323480 (0.084923) | 0.005704 / 0.007986 (-0.002282) | 0.003588 / 0.004328 (-0.000741) | 0.071441 / 0.004250 (0.067190) | 0.043520 / 0.037052 (0.006468) | 0.375798 / 0.258489 (0.117309) | 0.400955 / 0.293841 (0.107114) | 0.028166 / 0.128546 (-0.100381) | 0.008578 / 0.075646 (-0.067068) | 0.086673 / 0.419271 (-0.332598) | 0.046424 / 0.043533 (0.002891) | 0.367276 / 0.255139 (0.112137) | 0.414550 / 0.283200 (0.131351) | 0.097355 / 0.141683 (-0.044328) | 1.465191 / 1.452155 (0.013036) | 1.555028 / 1.492716 (0.062312) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196642 / 0.018006 (0.178636) | 0.464221 / 0.000490 (0.463731) | 0.002726 / 0.000200 (0.002526) | 0.000110 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028078 / 0.037411 (-0.009333) | 0.110762 / 0.014526 (0.096236) | 0.122212 / 0.176557 (-0.054344) | 0.164758 / 0.737135 (-0.572377) | 0.133969 / 0.296338 (-0.162370) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448134 / 0.215209 (0.232925) | 4.339335 / 2.077655 (2.261680) | 2.129209 / 1.504120 (0.625089) | 1.957805 / 1.541195 (0.416611) | 1.994038 / 1.468490 (0.525548) | 0.497101 / 4.584777 (-4.087676) | 4.114432 / 3.745712 (0.368720) | 3.437305 / 5.269862 (-1.832556) | 1.692810 / 4.565676 (-2.872866) | 0.071077 / 0.424275 (-0.353198) | 0.012735 / 0.007607 (0.005128) | 0.534393 / 0.226044 (0.308348) | 5.217445 / 2.268929 (2.948517) | 2.594858 / 55.444624 (-52.849766) | 2.317464 / 6.876477 (-4.559012) | 2.337974 / 2.142072 (0.195902) | 0.622291 / 4.805227 (-4.182936) | 0.144934 / 6.500664 (-6.355730) | 0.068524 / 0.075469 (-0.006945) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.310601 / 1.841788 (-0.531187) | 15.771527 / 8.074308 (7.697219) | 13.952032 / 10.191392 (3.760640) | 0.212473 / 0.680424 (-0.467951) | 0.017963 / 0.534201 (-0.516238) | 0.400755 / 0.579283 (-0.178528) | 0.439817 / 0.434364 (0.005453) | 0.472614 / 0.540337 (-0.067724) | 0.558410 / 1.386936 (-0.828526) |

</details>
</details>

![](https://cml.dev/watermark.png#1b51429d02a0da1ff798873afe655309136c5689 "CML watermark")