Support returning dataframe in map transform
Opened by mariosasko (closed)

Allow returning Pandas DataFrames in `map` transforms.

(Plus, raise an error in the non-batched mode if a returned PyArrow table/Pandas DataFrame has more than one row)



Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009725 / 0.011353 (-0.001628) | 0.006014 / 0.011008 (-0.004994) | 0.136039 / 0.038508 (0.097531) | 0.049685 / 0.023109 (0.026576) | 0.492967 / 0.275898 (0.217068) | 0.553775 / 0.323480 (0.230295) | 0.007421 / 0.007986 (-0.000564) | 0.004686 / 0.004328 (0.000357) | 0.106639 / 0.004250 (0.102389) | 0.073483 / 0.037052 (0.036431) | 0.507194 / 0.258489 (0.248705) | 0.535760 / 0.293841 (0.241919) | 0.049666 / 0.128546 (-0.078880) | 0.014139 / 0.075646 (-0.061507) | 0.435459 / 0.419271 (0.016188) | 0.076026 / 0.043533 (0.032493) | 0.454542 / 0.255139 (0.199403) | 0.512724 / 0.283200 (0.229524) | 0.034969 / 0.141683 (-0.106713) | 1.881048 / 1.452155 (0.428893) | 1.959915 / 1.492716 (0.467199) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.265322 / 0.018006 (0.247316) | 0.573963 / 0.000490 (0.573474) | 0.017493 / 0.000200 (0.017293) | 0.000637 / 0.000054 (0.000582) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028712 / 0.037411 (-0.008699) | 0.149554 / 0.014526 (0.135029) | 0.130013 / 0.176557 (-0.046544) | 0.203408 / 0.737135 (-0.533727) | 0.144778 / 0.296338 (-0.151561) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.664198 / 0.215209 (0.448989) | 6.418054 / 2.077655 (4.340399) | 2.602338 / 1.504120 (1.098219) | 2.212992 / 1.541195 (0.671797) | 2.214309 / 1.468490 (0.745819) | 0.914772 / 4.584777 (-3.670005) | 5.824831 / 3.745712 (2.079119) | 2.865381 / 5.269862 (-2.404481) | 1.906020 / 4.565676 (-2.659657) | 0.106947 / 0.424275 (-0.317328) | 0.013467 / 0.007607 (0.005860) | 0.834556 / 0.226044 (0.608512) | 8.237078 / 2.268929 (5.968150) | 3.380919 / 55.444624 (-52.063705) | 2.656713 / 6.876477 (-4.219764) | 2.834941 / 2.142072 (0.692869) | 1.151241 / 4.805227 (-3.653986) | 0.220860 / 6.500664 (-6.279804) | 0.080781 / 0.075469 (0.005312) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.655128 / 1.841788 (-0.186660) | 18.696108 / 8.074308 (10.621800) | 22.882108 / 10.191392 (12.690716) | 0.236041 / 0.680424 (-0.444383) | 0.031073 / 0.534201 (-0.503128) | 0.525263 / 0.579283 (-0.054021) | 0.632933 / 0.434364 (0.198569) | 0.707228 / 0.540337 (0.166890) | 0.753508 / 1.386936 (-0.633428) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009875 / 0.011353 (-0.001478) | 0.005135 / 0.011008 (-0.005873) | 0.101307 / 0.038508 (0.062799) | 0.044895 / 0.023109 (0.021786) | 0.497824 / 0.275898 (0.221926) | 0.573098 / 0.323480 (0.249618) | 0.006669 / 0.007986 (-0.001317) | 0.004289 / 0.004328 (-0.000039) | 0.105824 / 0.004250 (0.101573) | 0.061002 / 0.037052 (0.023950) | 0.510127 / 0.258489 (0.251638) | 0.581387 / 0.293841 (0.287546) | 0.052843 / 0.128546 (-0.075703) | 0.015506 / 0.075646 (-0.060140) | 0.116057 / 0.419271 (-0.303215) | 0.063444 / 0.043533 (0.019912) | 0.479366 / 0.255139 (0.224227) | 0.518419 / 0.283200 (0.235220) | 0.034876 / 0.141683 (-0.106806) | 2.018446 / 1.452155 (0.566292) | 1.960755 / 1.492716 (0.468039) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.269077 / 0.018006 (0.251070) | 0.606059 / 0.000490 (0.605569) | 0.000488 / 0.000200 (0.000288) | 0.000093 / 0.000054 (0.000038) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032465 / 0.037411 (-0.004946) | 0.136517 / 0.014526 (0.121991) | 0.147740 / 0.176557 (-0.028816) | 0.193802 / 0.737135 (-0.543334) | 0.151876 / 0.296338 (-0.144462) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.709866 / 0.215209 (0.494657) | 6.848193 / 2.077655 (4.770538) | 3.310853 / 1.504120 (1.806733) | 2.940813 / 1.541195 (1.399619) | 2.934934 / 1.468490 (1.466444) | 0.927104 / 4.584777 (-3.657673) | 5.921607 / 3.745712 (2.175895) | 4.926558 / 5.269862 (-0.343303) | 2.853269 / 4.565676 (-1.712407) | 0.120278 / 0.424275 (-0.303998) | 0.015468 / 0.007607 (0.007861) | 0.820509 / 0.226044 (0.594464) | 8.263136 / 2.268929 (5.994208) | 3.780214 / 55.444624 (-51.664410) | 3.108482 / 6.876477 (-3.767995) | 3.101544 / 2.142072 (0.959471) | 1.165539 / 4.805227 (-3.639688) | 0.229215 / 6.500664 (-6.271449) | 0.079862 / 0.075469 (0.004393) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.775071 / 1.841788 (-0.066717) | 19.327621 / 8.074308 (11.253313) | 23.057537 / 10.191392 (12.866145) | 0.250649 / 0.680424 (-0.429775) | 0.029767 / 0.534201 (-0.504434) | 0.554774 / 0.579283 (-0.024509) | 0.651919 / 0.434364 (0.217555) | 0.651641 / 0.540337 (0.111304) | 0.762386 / 1.386936 (-0.624550) |

</details>
</details>

![](https://cml.dev/watermark.png#fdc3ce7060366f480621e8640903c9ab476164e7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005997 / 0.011353 (-0.005356) | 0.003892 / 0.011008 (-0.007116) | 0.098020 / 0.038508 (0.059512) | 0.042584 / 0.023109 (0.019475) | 0.317909 / 0.275898 (0.042011) | 0.395042 / 0.323480 (0.071563) | 0.005358 / 0.007986 (-0.002628) | 0.003266 / 0.004328 (-0.001062) | 0.076698 / 0.004250 (0.072447) | 0.062331 / 0.037052 (0.025279) | 0.334900 / 0.258489 (0.076411) | 0.379355 / 0.293841 (0.085514) | 0.030815 / 0.128546 (-0.097731) | 0.008596 / 0.075646 (-0.067050) | 0.327739 / 0.419271 (-0.091533) | 0.054061 / 0.043533 (0.010528) | 0.311044 / 0.255139 (0.055905) | 0.336705 / 0.283200 (0.053506) | 0.022785 / 0.141683 (-0.118898) | 1.516793 / 1.452155 (0.064639) | 1.590435 / 1.492716 (0.097719) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289157 / 0.018006 (0.271151) | 0.531074 / 0.000490 (0.530585) | 0.004672 / 0.000200 (0.004472) | 0.000095 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026173 / 0.037411 (-0.011238) | 0.105723 / 0.014526 (0.091197) | 0.118010 / 0.176557 (-0.058547) | 0.178062 / 0.737135 (-0.559073) | 0.120059 / 0.296338 (-0.176279) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.410870 / 0.215209 (0.195661) | 4.042183 / 2.077655 (1.964528) | 1.830059 / 1.504120 (0.325939) | 1.638996 / 1.541195 (0.097802) | 1.701368 / 1.468490 (0.232878) | 0.529915 / 4.584777 (-4.054861) | 3.693308 / 3.745712 (-0.052404) | 1.827875 / 5.269862 (-3.441986) | 1.063237 / 4.565676 (-3.502440) | 0.065368 / 0.424275 (-0.358907) | 0.010986 / 0.007607 (0.003379) | 0.509399 / 0.226044 (0.283354) | 5.092739 / 2.268929 (2.823810) | 2.293490 / 55.444624 (-53.151135) | 1.958742 / 6.876477 (-4.917735) | 2.024985 / 2.142072 (-0.117088) | 0.646978 / 4.805227 (-4.158249) | 0.138616 / 6.500664 (-6.362048) | 0.062101 / 0.075469 (-0.013368) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.202016 / 1.841788 (-0.639772) | 14.493204 / 8.074308 (6.418896) | 12.992160 / 10.191392 (2.800768) | 0.188922 / 0.680424 (-0.491502) | 0.017594 / 0.534201 (-0.516606) | 0.399917 / 0.579283 (-0.179367) | 0.429760 / 0.434364 (-0.004604) | 0.497906 / 0.540337 (-0.042431) | 0.608745 / 1.386936 (-0.778191) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006164 / 0.011353 (-0.005189) | 0.003980 / 0.011008 (-0.007028) | 0.074676 / 0.038508 (0.036168) | 0.041337 / 0.023109 (0.018228) | 0.400981 / 0.275898 (0.125083) | 0.448791 / 0.323480 (0.125312) | 0.004063 / 0.007986 (-0.003923) | 0.004443 / 0.004328 (0.000114) | 0.075011 / 0.004250 (0.070760) | 0.056494 / 0.037052 (0.019441) | 0.402054 / 0.258489 (0.143565) | 0.446122 / 0.293841 (0.152281) | 0.031752 / 0.128546 (-0.096794) | 0.008835 / 0.075646 (-0.066811) | 0.081226 / 0.419271 (-0.338046) | 0.051501 / 0.043533 (0.007969) | 0.383674 / 0.255139 (0.128535) | 0.405524 / 0.283200 (0.122325) | 0.025929 / 0.141683 (-0.115754) | 1.492985 / 1.452155 (0.040830) | 1.541601 / 1.492716 (0.048885) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305149 / 0.018006 (0.287142) | 0.497259 / 0.000490 (0.496770) | 0.000420 / 0.000200 (0.000220) | 0.000056 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027933 / 0.037411 (-0.009479) | 0.111900 / 0.014526 (0.097374) | 0.124879 / 0.176557 (-0.051678) | 0.178952 / 0.737135 (-0.558184) | 0.127698 / 0.296338 (-0.168640) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.448525 / 0.215209 (0.233316) | 4.486791 / 2.077655 (2.409137) | 2.256687 / 1.504120 (0.752567) | 2.061078 / 1.541195 (0.519884) | 2.078924 / 1.468490 (0.610434) | 0.534412 / 4.584777 (-4.050365) | 3.721098 / 3.745712 (-0.024614) | 1.818735 / 5.269862 (-3.451127) | 1.104198 / 4.565676 (-3.461479) | 0.066277 / 0.424275 (-0.357998) | 0.011441 / 0.007607 (0.003834) | 0.550140 / 0.226044 (0.324095) | 5.498079 / 2.268929 (3.229150) | 2.717398 / 55.444624 (-52.727227) | 2.410194 / 6.876477 (-4.466283) | 2.405304 / 2.142072 (0.263231) | 0.665432 / 4.805227 (-4.139796) | 0.141488 / 6.500664 (-6.359177) | 0.064051 / 0.075469 (-0.011419) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272334 / 1.841788 (-0.569454) | 14.901608 / 8.074308 (6.827300) | 14.287857 / 10.191392 (4.096465) | 0.165337 / 0.680424 (-0.515086) | 0.017402 / 0.534201 (-0.516799) | 0.398120 / 0.579283 (-0.181163) | 0.416539 / 0.434364 (-0.017825) | 0.463890 / 0.540337 (-0.076447) | 0.567909 / 1.386936 (-0.819027) |

</details>
</details>

![](https://cml.dev/watermark.png#504ec0f2e00ee38e0993ed1e4f1e10f1eefaea0d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009434 / 0.011353 (-0.001919) | 0.005567 / 0.011008 (-0.005441) | 0.122652 / 0.038508 (0.084144) | 0.050177 / 0.023109 (0.027067) | 0.384292 / 0.275898 (0.108394) | 0.446608 / 0.323480 (0.123128) | 0.006502 / 0.007986 (-0.001484) | 0.004523 / 0.004328 (0.000194) | 0.100581 / 0.004250 (0.096331) | 0.073615 / 0.037052 (0.036563) | 0.420179 / 0.258489 (0.161690) | 0.474631 / 0.293841 (0.180790) | 0.047942 / 0.128546 (-0.080604) | 0.013864 / 0.075646 (-0.061783) | 0.419384 / 0.419271 (0.000112) | 0.088317 / 0.043533 (0.044784) | 0.379620 / 0.255139 (0.124481) | 0.412639 / 0.283200 (0.129440) | 0.048947 / 0.141683 (-0.092736) | 1.823498 / 1.452155 (0.371343) | 1.966629 / 1.492716 (0.473913) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.300669 / 0.018006 (0.282663) | 0.593499 / 0.000490 (0.593009) | 0.007247 / 0.000200 (0.007047) | 0.000114 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030556 / 0.037411 (-0.006856) | 0.119252 / 0.014526 (0.104726) | 0.131403 / 0.176557 (-0.045153) | 0.201845 / 0.737135 (-0.535291) | 0.139350 / 0.296338 (-0.156989) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.652400 / 0.215209 (0.437191) | 6.536540 / 2.077655 (4.458886) | 2.644565 / 1.504120 (1.140445) | 2.245181 / 1.541195 (0.703986) | 2.316030 / 1.468490 (0.847540) | 0.922535 / 4.584777 (-3.662242) | 5.469065 / 3.745712 (1.723353) | 2.800489 / 5.269862 (-2.469373) | 1.749042 / 4.565676 (-2.816635) | 0.108444 / 0.424275 (-0.315831) | 0.015651 / 0.007607 (0.008044) | 0.846085 / 0.226044 (0.620041) | 8.018460 / 2.268929 (5.749531) | 3.338710 / 55.444624 (-52.105914) | 2.675998 / 6.876477 (-4.200479) | 2.918550 / 2.142072 (0.776478) | 1.135145 / 4.805227 (-3.670082) | 0.215165 / 6.500664 (-6.285499) | 0.082066 / 0.075469 (0.006597) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.561661 / 1.841788 (-0.280127) | 18.519035 / 8.074308 (10.444727) | 19.046300 / 10.191392 (8.854908) | 0.236890 / 0.680424 (-0.443534) | 0.027681 / 0.534201 (-0.506520) | 0.511998 / 0.579283 (-0.067285) | 0.591627 / 0.434364 (0.157264) | 0.562021 / 0.540337 (0.021683) | 0.679354 / 1.386936 (-0.707582) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009643 / 0.011353 (-0.001710) | 0.005768 / 0.011008 (-0.005241) | 0.104430 / 0.038508 (0.065922) | 0.050044 / 0.023109 (0.026935) | 0.464117 / 0.275898 (0.188219) | 0.518439 / 0.323480 (0.194959) | 0.006935 / 0.007986 (-0.001051) | 0.004316 / 0.004328 (-0.000013) | 0.094330 / 0.004250 (0.090080) | 0.071451 / 0.037052 (0.034399) | 0.492248 / 0.258489 (0.233759) | 0.555740 / 0.293841 (0.261899) | 0.047836 / 0.128546 (-0.080711) | 0.014788 / 0.075646 (-0.060859) | 0.107590 / 0.419271 (-0.311682) | 0.064396 / 0.043533 (0.020863) | 0.451529 / 0.255139 (0.196390) | 0.475025 / 0.283200 (0.191826) | 0.040006 / 0.141683 (-0.101677) | 1.797107 / 1.452155 (0.344953) | 1.879261 / 1.492716 (0.386545) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.298458 / 0.018006 (0.280451) | 0.613022 / 0.000490 (0.612532) | 0.003582 / 0.000200 (0.003382) | 0.000106 / 0.000054 (0.000052) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030179 / 0.037411 (-0.007232) | 0.123286 / 0.014526 (0.108760) | 0.132070 / 0.176557 (-0.044486) | 0.190883 / 0.737135 (-0.546252) | 0.138526 / 0.296338 (-0.157812) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.666908 / 0.215209 (0.451699) | 6.489035 / 2.077655 (4.411381) | 2.897027 / 1.504120 (1.392907) | 2.565150 / 1.541195 (1.023956) | 2.504827 / 1.468490 (1.036336) | 0.916112 / 4.584777 (-3.668665) | 5.651751 / 3.745712 (1.906039) | 2.743382 / 5.269862 (-2.526479) | 1.773338 / 4.565676 (-2.792338) | 0.128764 / 0.424275 (-0.295511) | 0.013140 / 0.007607 (0.005533) | 0.803281 / 0.226044 (0.577236) | 8.258874 / 2.268929 (5.989945) | 3.633260 / 55.444624 (-51.811364) | 2.878827 / 6.876477 (-3.997649) | 2.977178 / 2.142072 (0.835106) | 1.130467 / 4.805227 (-3.674760) | 0.226381 / 6.500664 (-6.274283) | 0.081550 / 0.075469 (0.006081) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.842927 / 1.841788 (0.001139) | 18.411520 / 8.074308 (10.337212) | 21.118228 / 10.191392 (10.926836) | 0.231526 / 0.680424 (-0.448898) | 0.029300 / 0.534201 (-0.504901) | 0.527450 / 0.579283 (-0.051834) | 0.618873 / 0.434364 (0.184509) | 0.593314 / 0.540337 (0.052976) | 0.734430 / 1.386936 (-0.652506) |

</details>
</details>

![](https://cml.dev/watermark.png#0d2b8854c265b4dc202e480427890f472b34ea15 "CML watermark")