Dill 3.7 support
Opened by mariosasko (closed)

Adds support for dill 3.7.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007700 / 0.011353 (-0.003653) | 0.004680 / 0.011008 (-0.006328) | 0.098812 / 0.038508 (0.060304) | 0.085062 / 0.023109 (0.061952) | 0.371472 / 0.275898 (0.095574) | 0.412552 / 0.323480 (0.089072) | 0.004700 / 0.007986 (-0.003285) | 0.003765 / 0.004328 (-0.000564) | 0.074267 / 0.004250 (0.070017) | 0.063003 / 0.037052 (0.025951) | 0.391842 / 0.258489 (0.133353) | 0.436955 / 0.293841 (0.143114) | 0.035291 / 0.128546 (-0.093255) | 0.009309 / 0.075646 (-0.066338) | 0.313097 / 0.419271 (-0.106174) | 0.060098 / 0.043533 (0.016565) | 0.350726 / 0.255139 (0.095587) | 0.402692 / 0.283200 (0.119493) | 0.029321 / 0.141683 (-0.112361) | 1.671806 / 1.452155 (0.219651) | 1.743760 / 1.492716 (0.251044) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242281 / 0.018006 (0.224275) | 0.505054 / 0.000490 (0.504564) | 0.006595 / 0.000200 (0.006395) | 0.000091 / 0.000054 (0.000037) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032174 / 0.037411 (-0.005238) | 0.094483 / 0.014526 (0.079957) | 0.108527 / 0.176557 (-0.068030) | 0.178983 / 0.737135 (-0.558152) | 0.113766 / 0.296338 (-0.182572) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419764 / 0.215209 (0.204555) | 4.282650 / 2.077655 (2.204995) | 2.075325 / 1.504120 (0.571205) | 1.897668 / 1.541195 (0.356473) | 2.027109 / 1.468490 (0.558619) | 0.519983 / 4.584777 (-4.064794) | 4.134603 / 3.745712 (0.388891) | 6.586711 / 5.269862 (1.316849) | 3.811726 / 4.565676 (-0.753951) | 0.058628 / 0.424275 (-0.365647) | 0.007586 / 0.007607 (-0.000021) | 0.502180 / 0.226044 (0.276136) | 5.101588 / 2.268929 (2.832660) | 2.534295 / 55.444624 (-52.910330) | 2.220170 / 6.876477 (-4.656307) | 2.441110 / 2.142072 (0.299038) | 0.644775 / 4.805227 (-4.160452) | 0.144716 / 6.500664 (-6.355948) | 0.067018 / 0.075469 (-0.008451) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.431279 / 1.841788 (-0.410508) | 21.947814 / 8.074308 (13.873506) | 15.548236 / 10.191392 (5.356844) | 0.174774 / 0.680424 (-0.505650) | 0.021182 / 0.534201 (-0.513019) | 0.441320 / 0.579283 (-0.137963) | 0.476685 / 0.434364 (0.042321) | 0.506277 / 0.540337 (-0.034060) | 0.809943 / 1.386936 (-0.576993) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007172 / 0.011353 (-0.004181) | 0.004358 / 0.011008 (-0.006650) | 0.068604 / 0.038508 (0.030096) | 0.083956 / 0.023109 (0.060847) | 0.402579 / 0.275898 (0.126681) | 0.444714 / 0.323480 (0.121235) | 0.005940 / 0.007986 (-0.002046) | 0.003607 / 0.004328 (-0.000722) | 0.073134 / 0.004250 (0.068883) | 0.061722 / 0.037052 (0.024669) | 0.410957 / 0.258489 (0.152468) | 0.458819 / 0.293841 (0.164978) | 0.033710 / 0.128546 (-0.094836) | 0.010230 / 0.075646 (-0.065417) | 0.084678 / 0.419271 (-0.334593) | 0.058203 / 0.043533 (0.014670) | 0.444972 / 0.255139 (0.189833) | 0.470962 / 0.283200 (0.187763) | 0.029222 / 0.141683 (-0.112461) | 1.671460 / 1.452155 (0.219306) | 1.759471 / 1.492716 (0.266754) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.238894 / 0.018006 (0.220888) | 0.493605 / 0.000490 (0.493115) | 0.001979 / 0.000200 (0.001780) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036498 / 0.037411 (-0.000913) | 0.095245 / 0.014526 (0.080719) | 0.112147 / 0.176557 (-0.064409) | 0.171128 / 0.737135 (-0.566007) | 0.115295 / 0.296338 (-0.181044) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.461067 / 0.215209 (0.245858) | 4.723932 / 2.077655 (2.646277) | 2.432697 / 1.504120 (0.928578) | 2.237302 / 1.541195 (0.696107) | 2.351320 / 1.468490 (0.882830) | 0.509963 / 4.584777 (-4.074813) | 4.194817 / 3.745712 (0.449105) | 6.689529 / 5.269862 (1.419667) | 3.351198 / 4.565676 (-1.214478) | 0.064563 / 0.424275 (-0.359712) | 0.008605 / 0.007607 (0.000998) | 0.575590 / 0.226044 (0.349546) | 5.644179 / 2.268929 (3.375250) | 3.021375 / 55.444624 (-52.423249) | 2.595305 / 6.876477 (-4.281172) | 2.839228 / 2.142072 (0.697156) | 0.657148 / 4.805227 (-4.148079) | 0.144831 / 6.500664 (-6.355834) | 0.067882 / 0.075469 (-0.007587) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.595580 / 1.841788 (-0.246208) | 22.431609 / 8.074308 (14.357301) | 15.700845 / 10.191392 (5.509453) | 0.164675 / 0.680424 (-0.515749) | 0.021322 / 0.534201 (-0.512879) | 0.455270 / 0.579283 (-0.124013) | 0.451547 / 0.434364 (0.017183) | 0.520955 / 0.540337 (-0.019383) | 0.687803 / 1.386936 (-0.699133) |

</details>
</details>

![](https://cml.dev/watermark.png#7d19574e9f44bd3b59a3e47ca7c4ea66305a8e6b "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008171 / 0.011353 (-0.003182) | 0.005563 / 0.011008 (-0.005445) | 0.102265 / 0.038508 (0.063757) | 0.074755 / 0.023109 (0.051646) | 0.431317 / 0.275898 (0.155419) | 0.472179 / 0.323480 (0.148699) | 0.006153 / 0.007986 (-0.001833) | 0.003832 / 0.004328 (-0.000496) | 0.078480 / 0.004250 (0.074230) | 0.056250 / 0.037052 (0.019197) | 0.432938 / 0.258489 (0.174449) | 0.480983 / 0.293841 (0.187142) | 0.048861 / 0.128546 (-0.079685) | 0.016252 / 0.075646 (-0.059394) | 0.343508 / 0.419271 (-0.075763) | 0.065057 / 0.043533 (0.021524) | 0.468418 / 0.255139 (0.213279) | 0.463692 / 0.283200 (0.180492) | 0.032912 / 0.141683 (-0.108771) | 1.795194 / 1.452155 (0.343039) | 1.833047 / 1.492716 (0.340331) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.197980 / 0.018006 (0.179974) | 0.500662 / 0.000490 (0.500172) | 0.007380 / 0.000200 (0.007181) | 0.000110 / 0.000054 (0.000055) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028323 / 0.037411 (-0.009089) | 0.089817 / 0.014526 (0.075291) | 0.102923 / 0.176557 (-0.073633) | 0.173851 / 0.737135 (-0.563284) | 0.104006 / 0.296338 (-0.192333) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.580277 / 0.215209 (0.365068) | 5.878739 / 2.077655 (3.801085) | 2.404673 / 1.504120 (0.900553) | 2.071765 / 1.541195 (0.530571) | 2.106024 / 1.468490 (0.637534) | 0.855217 / 4.584777 (-3.729560) | 4.918602 / 3.745712 (1.172890) | 5.354984 / 5.269862 (0.085122) | 3.141288 / 4.565676 (-1.424389) | 0.099553 / 0.424275 (-0.324723) | 0.008152 / 0.007607 (0.000545) | 0.709857 / 0.226044 (0.483813) | 7.144602 / 2.268929 (4.875673) | 3.137637 / 55.444624 (-52.306987) | 2.379851 / 6.876477 (-4.496626) | 2.346426 / 2.142072 (0.204353) | 1.033416 / 4.805227 (-3.771811) | 0.213120 / 6.500664 (-6.287544) | 0.076037 / 0.075469 (0.000568) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.597742 / 1.841788 (-0.244046) | 21.745366 / 8.074308 (13.671058) | 20.830698 / 10.191392 (10.639306) | 0.238727 / 0.680424 (-0.441697) | 0.027923 / 0.534201 (-0.506278) | 0.466073 / 0.579283 (-0.113210) | 0.548647 / 0.434364 (0.114283) | 0.549245 / 0.540337 (0.008908) | 0.977148 / 1.386936 (-0.409788) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008252 / 0.011353 (-0.003101) | 0.004653 / 0.011008 (-0.006356) | 0.084012 / 0.038508 (0.045504) | 0.077418 / 0.023109 (0.054309) | 0.440748 / 0.275898 (0.164850) | 0.464279 / 0.323480 (0.140799) | 0.005762 / 0.007986 (-0.002224) | 0.004909 / 0.004328 (0.000581) | 0.086441 / 0.004250 (0.082190) | 0.057883 / 0.037052 (0.020831) | 0.466655 / 0.258489 (0.208166) | 0.479751 / 0.293841 (0.185910) | 0.047166 / 0.128546 (-0.081380) | 0.014480 / 0.075646 (-0.061166) | 0.092599 / 0.419271 (-0.326672) | 0.062454 / 0.043533 (0.018921) | 0.449753 / 0.255139 (0.194614) | 0.461876 / 0.283200 (0.178676) | 0.034828 / 0.141683 (-0.106855) | 1.752249 / 1.452155 (0.300095) | 1.865449 / 1.492716 (0.372732) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.245028 / 0.018006 (0.227022) | 0.509564 / 0.000490 (0.509074) | 0.003930 / 0.000200 (0.003730) | 0.000110 / 0.000054 (0.000056) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034746 / 0.037411 (-0.002665) | 0.096563 / 0.014526 (0.082037) | 0.107581 / 0.176557 (-0.068975) | 0.184952 / 0.737135 (-0.552184) | 0.108747 / 0.296338 (-0.187591) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.613091 / 0.215209 (0.397882) | 5.994985 / 2.077655 (3.917330) | 2.711276 / 1.504120 (1.207156) | 2.415862 / 1.541195 (0.874668) | 2.391055 / 1.468490 (0.922565) | 0.868723 / 4.584777 (-3.716054) | 4.953992 / 3.745712 (1.208280) | 4.606542 / 5.269862 (-0.663319) | 2.942162 / 4.565676 (-1.623515) | 0.102737 / 0.424275 (-0.321538) | 0.008634 / 0.007607 (0.001027) | 0.722122 / 0.226044 (0.496078) | 7.245097 / 2.268929 (4.976168) | 3.428232 / 55.444624 (-52.016393) | 2.709539 / 6.876477 (-4.166938) | 2.857956 / 2.142072 (0.715884) | 1.045594 / 4.805227 (-3.759634) | 0.213344 / 6.500664 (-6.287320) | 0.073601 / 0.075469 (-0.001868) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.651954 / 1.841788 (-0.189834) | 22.458646 / 8.074308 (14.384338) | 19.583203 / 10.191392 (9.391811) | 0.246932 / 0.680424 (-0.433492) | 0.025730 / 0.534201 (-0.508471) | 0.473475 / 0.579283 (-0.105808) | 0.521411 / 0.434364 (0.087047) | 0.562038 / 0.540337 (0.021700) | 0.767673 / 1.386936 (-0.619263) |

</details>
</details>

![](https://cml.dev/watermark.png#3869d99628329c696f6975377f65e625dd8ef3e0 "CML watermark")


The CI error is unrelated.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006649 / 0.011353 (-0.004703) | 0.003963 / 0.011008 (-0.007045) | 0.084564 / 0.038508 (0.046056) | 0.075668 / 0.023109 (0.052559) | 0.314233 / 0.275898 (0.038335) | 0.343320 / 0.323480 (0.019841) | 0.005405 / 0.007986 (-0.002581) | 0.003356 / 0.004328 (-0.000973) | 0.065094 / 0.004250 (0.060844) | 0.058774 / 0.037052 (0.021722) | 0.320772 / 0.258489 (0.062283) | 0.353546 / 0.293841 (0.059705) | 0.030921 / 0.128546 (-0.097625) | 0.008463 / 0.075646 (-0.067184) | 0.287490 / 0.419271 (-0.131781) | 0.053188 / 0.043533 (0.009656) | 0.324023 / 0.255139 (0.068884) | 0.337828 / 0.283200 (0.054628) | 0.024764 / 0.141683 (-0.116918) | 1.458028 / 1.452155 (0.005873) | 1.521615 / 1.492716 (0.028899) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209360 / 0.018006 (0.191353) | 0.461331 / 0.000490 (0.460841) | 0.000386 / 0.000200 (0.000186) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028405 / 0.037411 (-0.009006) | 0.081074 / 0.014526 (0.066548) | 0.094868 / 0.176557 (-0.081689) | 0.151050 / 0.737135 (-0.586085) | 0.095854 / 0.296338 (-0.200484) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.393957 / 0.215209 (0.178748) | 3.938649 / 2.077655 (1.860994) | 1.938190 / 1.504120 (0.434070) | 1.766458 / 1.541195 (0.225263) | 1.818028 / 1.468490 (0.349538) | 0.483926 / 4.584777 (-4.100851) | 3.641957 / 3.745712 (-0.103755) | 4.883845 / 5.269862 (-0.386016) | 2.960300 / 4.565676 (-1.605377) | 0.057227 / 0.424275 (-0.367048) | 0.007285 / 0.007607 (-0.000322) | 0.475928 / 0.226044 (0.249884) | 4.756757 / 2.268929 (2.487828) | 2.502659 / 55.444624 (-52.941966) | 2.178067 / 6.876477 (-4.698410) | 2.378298 / 2.142072 (0.236226) | 0.578639 / 4.805227 (-4.226588) | 0.132512 / 6.500664 (-6.368152) | 0.059656 / 0.075469 (-0.015813) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.272673 / 1.841788 (-0.569115) | 19.266884 / 8.074308 (11.192576) | 14.272930 / 10.191392 (4.081538) | 0.165897 / 0.680424 (-0.514527) | 0.018436 / 0.534201 (-0.515765) | 0.395177 / 0.579283 (-0.184107) | 0.420134 / 0.434364 (-0.014229) | 0.460781 / 0.540337 (-0.079557) | 0.645376 / 1.386936 (-0.741560) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006504 / 0.011353 (-0.004849) | 0.003942 / 0.011008 (-0.007066) | 0.064936 / 0.038508 (0.026428) | 0.075015 / 0.023109 (0.051905) | 0.396871 / 0.275898 (0.120973) | 0.423448 / 0.323480 (0.099968) | 0.005239 / 0.007986 (-0.002747) | 0.003265 / 0.004328 (-0.001063) | 0.064910 / 0.004250 (0.060660) | 0.055006 / 0.037052 (0.017953) | 0.392818 / 0.258489 (0.134329) | 0.429735 / 0.293841 (0.135894) | 0.031847 / 0.128546 (-0.096699) | 0.008626 / 0.075646 (-0.067021) | 0.071591 / 0.419271 (-0.347681) | 0.049006 / 0.043533 (0.005473) | 0.384913 / 0.255139 (0.129774) | 0.408969 / 0.283200 (0.125769) | 0.023573 / 0.141683 (-0.118110) | 1.490271 / 1.452155 (0.038117) | 1.564620 / 1.492716 (0.071904) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225917 / 0.018006 (0.207911) | 0.450369 / 0.000490 (0.449880) | 0.000375 / 0.000200 (0.000175) | 0.000055 / 0.000054 (0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031196 / 0.037411 (-0.006215) | 0.090486 / 0.014526 (0.075960) | 0.102326 / 0.176557 (-0.074231) | 0.157483 / 0.737135 (-0.579653) | 0.103670 / 0.296338 (-0.192668) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417577 / 0.215209 (0.202368) | 4.170798 / 2.077655 (2.093143) | 2.123689 / 1.504120 (0.619569) | 1.948231 / 1.541195 (0.407037) | 2.040277 / 1.468490 (0.571787) | 0.497919 / 4.584777 (-4.086858) | 3.633270 / 3.745712 (-0.112442) | 4.851698 / 5.269862 (-0.418164) | 2.691992 / 4.565676 (-1.873684) | 0.058641 / 0.424275 (-0.365634) | 0.007719 / 0.007607 (0.000112) | 0.500652 / 0.226044 (0.274607) | 4.988657 / 2.268929 (2.719728) | 2.604488 / 55.444624 (-52.840136) | 2.329829 / 6.876477 (-4.546648) | 2.468239 / 2.142072 (0.326167) | 0.598724 / 4.805227 (-4.206503) | 0.135959 / 6.500664 (-6.364706) | 0.061088 / 0.075469 (-0.014381) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.352107 / 1.841788 (-0.489681) | 19.973976 / 8.074308 (11.899668) | 14.292812 / 10.191392 (4.101420) | 0.163855 / 0.680424 (-0.516568) | 0.018402 / 0.534201 (-0.515799) | 0.393128 / 0.579283 (-0.186155) | 0.407379 / 0.434364 (-0.026985) | 0.462324 / 0.540337 (-0.078013) | 0.607501 / 1.386936 (-0.779435) |

</details>
</details>

![](https://cml.dev/watermark.png#ae126ac974cad3050f90106e5909232140786811 "CML watermark")