Better error message when combining dataset dicts instead of datasets
State: closed | Author: lhoestq (MEMBER)

close https://github.com/huggingface/datasets/issues/5851

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007167 / 0.011353 (-0.004185) | 0.004914 / 0.011008 (-0.006094) | 0.096858 / 0.038508 (0.058350) | 0.033468 / 0.023109 (0.010359) | 0.297276 / 0.275898 (0.021378) | 0.344289 / 0.323480 (0.020809) | 0.005703 / 0.007986 (-0.002282) | 0.003972 / 0.004328 (-0.000357) | 0.075191 / 0.004250 (0.070940) | 0.046247 / 0.037052 (0.009194) | 0.317857 / 0.258489 (0.059368) | 0.347263 / 0.293841 (0.053422) | 0.035017 / 0.128546 (-0.093529) | 0.012036 / 0.075646 (-0.063611) | 0.332522 / 0.419271 (-0.086750) | 0.050188 / 0.043533 (0.006655) | 0.296627 / 0.255139 (0.041488) | 0.319196 / 0.283200 (0.035997) | 0.101100 / 0.141683 (-0.040583) | 1.484536 / 1.452155 (0.032382) | 1.606364 / 1.492716 (0.113648) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.203954 / 0.018006 (0.185948) | 0.436505 / 0.000490 (0.436015) | 0.003853 / 0.000200 (0.003654) | 0.000079 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025834 / 0.037411 (-0.011578) | 0.105759 / 0.014526 (0.091233) | 0.114289 / 0.176557 (-0.062268) | 0.174388 / 0.737135 (-0.562748) | 0.122248 / 0.296338 (-0.174090) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404218 / 0.215209 (0.189009) | 4.027900 / 2.077655 (1.950245) | 1.854757 / 1.504120 (0.350637) | 1.668882 / 1.541195 (0.127687) | 1.731451 / 1.468490 (0.262961) | 0.707843 / 4.584777 (-3.876934) | 3.756386 / 3.745712 (0.010674) | 2.067751 / 5.269862 (-3.202110) | 1.313039 / 4.565676 (-3.252638) | 0.086442 / 0.424275 (-0.337833) | 0.012329 / 0.007607 (0.004722) | 0.505964 / 0.226044 (0.279919) | 5.050788 / 2.268929 (2.781860) | 2.353936 / 55.444624 (-53.090688) | 2.055560 / 6.876477 (-4.820917) | 2.162948 / 2.142072 (0.020876) | 0.850532 / 4.805227 (-3.954696) | 0.168560 / 6.500664 (-6.332104) | 0.063143 / 0.075469 (-0.012326) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.182723 / 1.841788 (-0.659065) | 14.779342 / 8.074308 (6.705034) | 14.461572 / 10.191392 (4.270180) | 0.163120 / 0.680424 (-0.517303) | 0.017978 / 0.534201 (-0.516223) | 0.419168 / 0.579283 (-0.160115) | 0.420955 / 0.434364 (-0.013409) | 0.509710 / 0.540337 (-0.030628) | 0.619586 / 1.386936 (-0.767350) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006804 / 0.011353 (-0.004549) | 0.005136 / 0.011008 (-0.005872) | 0.074910 / 0.038508 (0.036402) | 0.032552 / 0.023109 (0.009443) | 0.374998 / 0.275898 (0.099100) | 0.399219 / 0.323480 (0.075739) | 0.005615 / 0.007986 (-0.002371) | 0.004118 / 0.004328 (-0.000210) | 0.074219 / 0.004250 (0.069969) | 0.045924 / 0.037052 (0.008871) | 0.383228 / 0.258489 (0.124739) | 0.407195 / 0.293841 (0.113354) | 0.035460 / 0.128546 (-0.093086) | 0.012460 / 0.075646 (-0.063187) | 0.087077 / 0.419271 (-0.332195) | 0.050507 / 0.043533 (0.006974) | 0.369001 / 0.255139 (0.113862) | 0.385761 / 0.283200 (0.102561) | 0.106999 / 0.141683 (-0.034684) | 1.465456 / 1.452155 (0.013302) | 1.556962 / 1.492716 (0.064246) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.214926 / 0.018006 (0.196920) | 0.436893 / 0.000490 (0.436403) | 0.003388 / 0.000200 (0.003188) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029919 / 0.037411 (-0.007492) | 0.110859 / 0.014526 (0.096333) | 0.120617 / 0.176557 (-0.055939) | 0.171781 / 0.737135 (-0.565355) | 0.125627 / 0.296338 (-0.170712) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436024 / 0.215209 (0.220815) | 4.359167 / 2.077655 (2.281512) | 2.188399 / 1.504120 (0.684279) | 2.001196 / 1.541195 (0.460001) | 2.023710 / 1.468490 (0.555220) | 0.713799 / 4.584777 (-3.870978) | 3.832217 / 3.745712 (0.086504) | 3.269351 / 5.269862 (-2.000510) | 1.534608 / 4.565676 (-3.031068) | 0.088505 / 0.424275 (-0.335770) | 0.012345 / 0.007607 (0.004738) | 0.542446 / 0.226044 (0.316401) | 5.377757 / 2.268929 (3.108828) | 2.659837 / 55.444624 (-52.784787) | 2.272356 / 6.876477 (-4.604120) | 2.297289 / 2.142072 (0.155217) | 0.855276 / 4.805227 (-3.949952) | 0.170666 / 6.500664 (-6.329998) | 0.064549 / 0.075469 (-0.010920) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255938 / 1.841788 (-0.585850) | 15.151471 / 8.074308 (7.077163) | 12.905762 / 10.191392 (2.714370) | 0.162425 / 0.680424 (-0.517999) | 0.017504 / 0.534201 (-0.516697) | 0.448671 / 0.579283 (-0.130612) | 0.422424 / 0.434364 (-0.011940) | 0.551772 / 0.540337 (0.011434) | 0.649115 / 1.386936 (-0.737821) |

</details>
</details>

![](https://cml.dev/watermark.png#be73d9f192149727c5542ff257df81b03024fa39 "CML watermark")


Having those different checks helps providing an appropriate error message.

If the input is a dict, we suggest to select a split. If the input lists is a mix of iterable and non-iterable, we mention that it must be one or the other.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006559 / 0.011353 (-0.004794) | 0.004569 / 0.011008 (-0.006439) | 0.104503 / 0.038508 (0.065995) | 0.028220 / 0.023109 (0.005111) | 0.365507 / 0.275898 (0.089609) | 0.400238 / 0.323480 (0.076758) | 0.004968 / 0.007986 (-0.003017) | 0.003271 / 0.004328 (-0.001057) | 0.082804 / 0.004250 (0.078554) | 0.036299 / 0.037052 (-0.000754) | 0.361201 / 0.258489 (0.102712) | 0.410962 / 0.293841 (0.117121) | 0.030423 / 0.128546 (-0.098123) | 0.011612 / 0.075646 (-0.064034) | 0.331820 / 0.419271 (-0.087452) | 0.043822 / 0.043533 (0.000289) | 0.356242 / 0.255139 (0.101103) | 0.393035 / 0.283200 (0.109836) | 0.088426 / 0.141683 (-0.053257) | 1.484139 / 1.452155 (0.031984) | 1.566712 / 1.492716 (0.073995) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.195887 / 0.018006 (0.177880) | 0.402720 / 0.000490 (0.402231) | 0.003516 / 0.000200 (0.003316) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023270 / 0.037411 (-0.014141) | 0.095834 / 0.014526 (0.081308) | 0.102924 / 0.176557 (-0.073632) | 0.161397 / 0.737135 (-0.575738) | 0.105225 / 0.296338 (-0.191114) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.451701 / 0.215209 (0.236491) | 4.495171 / 2.077655 (2.417517) | 2.223203 / 1.504120 (0.719083) | 2.035533 / 1.541195 (0.494338) | 2.076182 / 1.468490 (0.607692) | 0.697317 / 4.584777 (-3.887460) | 3.406309 / 3.745712 (-0.339403) | 1.847179 / 5.269862 (-3.422683) | 1.158762 / 4.565676 (-3.406914) | 0.083067 / 0.424275 (-0.341208) | 0.012453 / 0.007607 (0.004846) | 0.546502 / 0.226044 (0.320458) | 5.455712 / 2.268929 (3.186784) | 2.654142 / 55.444624 (-52.790483) | 2.298722 / 6.876477 (-4.577755) | 2.383467 / 2.142072 (0.241395) | 0.805950 / 4.805227 (-3.999278) | 0.152479 / 6.500664 (-6.348185) | 0.066784 / 0.075469 (-0.008685) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.239129 / 1.841788 (-0.602659) | 13.603707 / 8.074308 (5.529398) | 14.062004 / 10.191392 (3.870612) | 0.130928 / 0.680424 (-0.549495) | 0.016907 / 0.534201 (-0.517294) | 0.381614 / 0.579283 (-0.197670) | 0.386770 / 0.434364 (-0.047594) | 0.455792 / 0.540337 (-0.084545) | 0.526092 / 1.386936 (-0.860844) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006202 / 0.011353 (-0.005151) | 0.004478 / 0.011008 (-0.006531) | 0.076492 / 0.038508 (0.037984) | 0.026703 / 0.023109 (0.003594) | 0.355134 / 0.275898 (0.079236) | 0.391207 / 0.323480 (0.067727) | 0.004852 / 0.007986 (-0.003133) | 0.003271 / 0.004328 (-0.001057) | 0.075080 / 0.004250 (0.070830) | 0.038803 / 0.037052 (0.001750) | 0.359530 / 0.258489 (0.101041) | 0.409044 / 0.293841 (0.115203) | 0.030366 / 0.128546 (-0.098180) | 0.011544 / 0.075646 (-0.064102) | 0.084849 / 0.419271 (-0.334423) | 0.040076 / 0.043533 (-0.003457) | 0.357359 / 0.255139 (0.102220) | 0.384075 / 0.283200 (0.100875) | 0.089130 / 0.141683 (-0.052552) | 1.520400 / 1.452155 (0.068246) | 1.604403 / 1.492716 (0.111687) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.257127 / 0.018006 (0.239121) | 0.403691 / 0.000490 (0.403202) | 0.006894 / 0.000200 (0.006694) | 0.000088 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024653 / 0.037411 (-0.012758) | 0.098834 / 0.014526 (0.084309) | 0.107276 / 0.176557 (-0.069281) | 0.158256 / 0.737135 (-0.578879) | 0.111339 / 0.296338 (-0.184999) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445006 / 0.215209 (0.229797) | 4.452953 / 2.077655 (2.375299) | 2.168291 / 1.504120 (0.664171) | 1.969457 / 1.541195 (0.428262) | 2.003505 / 1.468490 (0.535015) | 0.695857 / 4.584777 (-3.888920) | 3.433424 / 3.745712 (-0.312288) | 2.466977 / 5.269862 (-2.802885) | 1.528167 / 4.565676 (-3.037509) | 0.082425 / 0.424275 (-0.341850) | 0.012470 / 0.007607 (0.004863) | 0.559039 / 0.226044 (0.332995) | 5.609496 / 2.268929 (3.340568) | 2.602898 / 55.444624 (-52.841726) | 2.273971 / 6.876477 (-4.602506) | 2.303370 / 2.142072 (0.161298) | 0.803875 / 4.805227 (-4.001352) | 0.151069 / 6.500664 (-6.349595) | 0.067956 / 0.075469 (-0.007513) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.334443 / 1.841788 (-0.507345) | 13.773252 / 8.074308 (5.698944) | 13.007042 / 10.191392 (2.815650) | 0.127939 / 0.680424 (-0.552485) | 0.016412 / 0.534201 (-0.517789) | 0.374744 / 0.579283 (-0.204539) | 0.396912 / 0.434364 (-0.037452) | 0.443197 / 0.540337 (-0.097140) | 0.528338 / 1.386936 (-0.858598) |

</details>
</details>

![](https://cml.dev/watermark.png#51d9f2a3064aa89a780e3d02c6cc34000c51c4fb "CML watermark")


Just modified it to use only one loop. I think I managed to keep it readable as well

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007382 / 0.011353 (-0.003971) | 0.005143 / 0.011008 (-0.005865) | 0.097635 / 0.038508 (0.059127) | 0.034726 / 0.023109 (0.011616) | 0.315556 / 0.275898 (0.039658) | 0.355951 / 0.323480 (0.032472) | 0.006055 / 0.007986 (-0.001931) | 0.004264 / 0.004328 (-0.000065) | 0.073636 / 0.004250 (0.069386) | 0.050480 / 0.037052 (0.013428) | 0.316031 / 0.258489 (0.057542) | 0.363933 / 0.293841 (0.070092) | 0.035138 / 0.128546 (-0.093408) | 0.012407 / 0.075646 (-0.063239) | 0.333677 / 0.419271 (-0.085595) | 0.050586 / 0.043533 (0.007053) | 0.309507 / 0.255139 (0.054369) | 0.327043 / 0.283200 (0.043844) | 0.108975 / 0.141683 (-0.032708) | 1.447778 / 1.452155 (-0.004377) | 1.519971 / 1.492716 (0.027255) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.248770 / 0.018006 (0.230764) | 0.603036 / 0.000490 (0.602546) | 0.000383 / 0.000200 (0.000183) | 0.000058 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027094 / 0.037411 (-0.010317) | 0.104427 / 0.014526 (0.089901) | 0.120627 / 0.176557 (-0.055929) | 0.178790 / 0.737135 (-0.558346) | 0.124877 / 0.296338 (-0.171461) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.414442 / 0.215209 (0.199233) | 4.138009 / 2.077655 (2.060355) | 1.964642 / 1.504120 (0.460523) | 1.775940 / 1.541195 (0.234745) | 1.899719 / 1.468490 (0.431228) | 0.695406 / 4.584777 (-3.889371) | 3.760470 / 3.745712 (0.014758) | 3.906958 / 5.269862 (-1.362904) | 2.028164 / 4.565676 (-2.537513) | 0.086704 / 0.424275 (-0.337571) | 0.012465 / 0.007607 (0.004857) | 0.512336 / 0.226044 (0.286292) | 5.108587 / 2.268929 (2.839659) | 2.435273 / 55.444624 (-53.009352) | 2.142387 / 6.876477 (-4.734090) | 2.258234 / 2.142072 (0.116162) | 0.854035 / 4.805227 (-3.951193) | 0.170443 / 6.500664 (-6.330222) | 0.065762 / 0.075469 (-0.009707) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.187529 / 1.841788 (-0.654259) | 15.151164 / 8.074308 (7.076856) | 14.577545 / 10.191392 (4.386153) | 0.166973 / 0.680424 (-0.513450) | 0.017883 / 0.534201 (-0.516318) | 0.427607 / 0.579283 (-0.151676) | 0.417050 / 0.434364 (-0.017314) | 0.508116 / 0.540337 (-0.032221) | 0.590173 / 1.386936 (-0.796763) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007499 / 0.011353 (-0.003854) | 0.005195 / 0.011008 (-0.005813) | 0.073600 / 0.038508 (0.035091) | 0.033574 / 0.023109 (0.010464) | 0.377506 / 0.275898 (0.101608) | 0.432752 / 0.323480 (0.109272) | 0.006042 / 0.007986 (-0.001944) | 0.006427 / 0.004328 (0.002098) | 0.071666 / 0.004250 (0.067416) | 0.053243 / 0.037052 (0.016190) | 0.363972 / 0.258489 (0.105483) | 0.454988 / 0.293841 (0.161147) | 0.035118 / 0.128546 (-0.093428) | 0.012395 / 0.075646 (-0.063251) | 0.084308 / 0.419271 (-0.334963) | 0.048589 / 0.043533 (0.005057) | 0.368036 / 0.255139 (0.112897) | 0.399414 / 0.283200 (0.116215) | 0.109043 / 0.141683 (-0.032640) | 1.462972 / 1.452155 (0.010817) | 1.574443 / 1.492716 (0.081726) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215107 / 0.018006 (0.197101) | 0.550255 / 0.000490 (0.549765) | 0.004630 / 0.000200 (0.004430) | 0.000104 / 0.000054 (0.000049) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029948 / 0.037411 (-0.007463) | 0.111866 / 0.014526 (0.097340) | 0.126559 / 0.176557 (-0.049997) | 0.181443 / 0.737135 (-0.555693) | 0.130559 / 0.296338 (-0.165779) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.441410 / 0.215209 (0.226201) | 4.403406 / 2.077655 (2.325752) | 2.180276 / 1.504120 (0.676156) | 2.003729 / 1.541195 (0.462534) | 2.079394 / 1.468490 (0.610904) | 0.706061 / 4.584777 (-3.878716) | 3.805668 / 3.745712 (0.059956) | 3.864941 / 5.269862 (-1.404921) | 1.970468 / 4.565676 (-2.595208) | 0.086033 / 0.424275 (-0.338242) | 0.012261 / 0.007607 (0.004654) | 0.550427 / 0.226044 (0.324383) | 5.542270 / 2.268929 (3.273342) | 2.717047 / 55.444624 (-52.727577) | 2.449022 / 6.876477 (-4.427455) | 2.549567 / 2.142072 (0.407495) | 0.854981 / 4.805227 (-3.950247) | 0.169756 / 6.500664 (-6.330908) | 0.067082 / 0.075469 (-0.008387) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.281369 / 1.841788 (-0.560419) | 15.445090 / 8.074308 (7.370781) | 13.205652 / 10.191392 (3.014260) | 0.170070 / 0.680424 (-0.510354) | 0.017815 / 0.534201 (-0.516385) | 0.425193 / 0.579283 (-0.154090) | 0.425205 / 0.434364 (-0.009159) | 0.493561 / 0.540337 (-0.046776) | 0.588994 / 1.386936 (-0.797942) |

</details>
</details>

![](https://cml.dev/watermark.png#e427105fc68fce04d0f3c74efb942cbf3a65d166 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006345 / 0.011353 (-0.005008) | 0.004330 / 0.011008 (-0.006678) | 0.096327 / 0.038508 (0.057819) | 0.032964 / 0.023109 (0.009855) | 0.335600 / 0.275898 (0.059702) | 0.365635 / 0.323480 (0.042155) | 0.005435 / 0.007986 (-0.002551) | 0.005005 / 0.004328 (0.000677) | 0.071107 / 0.004250 (0.066856) | 0.044363 / 0.037052 (0.007311) | 0.339988 / 0.258489 (0.081498) | 0.375575 / 0.293841 (0.081734) | 0.028343 / 0.128546 (-0.100203) | 0.008587 / 0.075646 (-0.067059) | 0.324349 / 0.419271 (-0.094922) | 0.050105 / 0.043533 (0.006573) | 0.327398 / 0.255139 (0.072259) | 0.348479 / 0.283200 (0.065279) | 0.102357 / 0.141683 (-0.039326) | 1.419905 / 1.452155 (-0.032250) | 1.534887 / 1.492716 (0.042171) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.212418 / 0.018006 (0.194412) | 0.433183 / 0.000490 (0.432693) | 0.000595 / 0.000200 (0.000395) | 0.000062 / 0.000054 (0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027520 / 0.037411 (-0.009891) | 0.109503 / 0.014526 (0.094977) | 0.118202 / 0.176557 (-0.058355) | 0.177236 / 0.737135 (-0.559899) | 0.123736 / 0.296338 (-0.172602) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405734 / 0.215209 (0.190525) | 4.039566 / 2.077655 (1.961911) | 1.838211 / 1.504120 (0.334091) | 1.652650 / 1.541195 (0.111456) | 1.753488 / 1.468490 (0.284998) | 0.525258 / 4.584777 (-4.059519) | 3.704509 / 3.745712 (-0.041203) | 1.826794 / 5.269862 (-3.443067) | 1.236361 / 4.565676 (-3.329315) | 0.065619 / 0.424275 (-0.358656) | 0.011606 / 0.007607 (0.003999) | 0.505954 / 0.226044 (0.279910) | 5.054140 / 2.268929 (2.785211) | 2.352587 / 55.444624 (-53.092037) | 2.050601 / 6.876477 (-4.825875) | 2.097222 / 2.142072 (-0.044850) | 0.641044 / 4.805227 (-4.164183) | 0.140676 / 6.500664 (-6.359988) | 0.063217 / 0.075469 (-0.012253) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.177750 / 1.841788 (-0.664038) | 14.819346 / 8.074308 (6.745038) | 14.085937 / 10.191392 (3.894545) | 0.168618 / 0.680424 (-0.511806) | 0.017189 / 0.534201 (-0.517011) | 0.393415 / 0.579283 (-0.185868) | 0.422879 / 0.434364 (-0.011485) | 0.477289 / 0.540337 (-0.063048) | 0.569078 / 1.386936 (-0.817858) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006502 / 0.011353 (-0.004850) | 0.004640 / 0.011008 (-0.006368) | 0.073272 / 0.038508 (0.034764) | 0.033225 / 0.023109 (0.010116) | 0.359165 / 0.275898 (0.083267) | 0.391659 / 0.323480 (0.068179) | 0.005684 / 0.007986 (-0.002302) | 0.004045 / 0.004328 (-0.000284) | 0.072880 / 0.004250 (0.068629) | 0.046260 / 0.037052 (0.009208) | 0.361772 / 0.258489 (0.103283) | 0.402905 / 0.293841 (0.109064) | 0.027732 / 0.128546 (-0.100814) | 0.008864 / 0.075646 (-0.066783) | 0.081961 / 0.419271 (-0.337310) | 0.046170 / 0.043533 (0.002637) | 0.364198 / 0.255139 (0.109059) | 0.387468 / 0.283200 (0.104269) | 0.105456 / 0.141683 (-0.036227) | 1.457176 / 1.452155 (0.005021) | 1.564899 / 1.492716 (0.072183) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.179129 / 0.018006 (0.161123) | 0.439699 / 0.000490 (0.439209) | 0.002882 / 0.000200 (0.002682) | 0.000090 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029123 / 0.037411 (-0.008288) | 0.112046 / 0.014526 (0.097520) | 0.122773 / 0.176557 (-0.053784) | 0.178404 / 0.737135 (-0.558732) | 0.127904 / 0.296338 (-0.168434) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440413 / 0.215209 (0.225204) | 4.407334 / 2.077655 (2.329680) | 2.112932 / 1.504120 (0.608812) | 1.911034 / 1.541195 (0.369840) | 2.057168 / 1.468490 (0.588677) | 0.525472 / 4.584777 (-4.059305) | 3.738894 / 3.745712 (-0.006818) | 1.807592 / 5.269862 (-3.462270) | 1.053837 / 4.565676 (-3.511839) | 0.066203 / 0.424275 (-0.358072) | 0.011965 / 0.007607 (0.004358) | 0.541137 / 0.226044 (0.315093) | 5.415040 / 2.268929 (3.146112) | 2.580476 / 55.444624 (-52.864148) | 2.234144 / 6.876477 (-4.642333) | 2.306014 / 2.142072 (0.163942) | 0.644221 / 4.805227 (-4.161006) | 0.142870 / 6.500664 (-6.357794) | 0.065015 / 0.075469 (-0.010454) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.303465 / 1.841788 (-0.538323) | 14.949683 / 8.074308 (6.875375) | 14.370871 / 10.191392 (4.179478) | 0.142714 / 0.680424 (-0.537710) | 0.017372 / 0.534201 (-0.516829) | 0.403898 / 0.579283 (-0.175385) | 0.424781 / 0.434364 (-0.009583) | 0.465984 / 0.540337 (-0.074353) | 0.570863 / 1.386936 (-0.816074) |

</details>
</details>

![](https://cml.dev/watermark.png#22d1d533e8ab831b1aa1aab3e7d3c72ba42a83e8 "CML watermark")