Set dev version
Opened by lhoestq (closed)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008959 / 0.011353 (-0.002394) | 0.004549 / 0.011008 (-0.006460) | 0.102012 / 0.038508 (0.063504) | 0.030122 / 0.023109 (0.007013) | 0.303731 / 0.275898 (0.027833) | 0.344418 / 0.323480 (0.020938) | 0.007199 / 0.007986 (-0.000787) | 0.003415 / 0.004328 (-0.000913) | 0.079784 / 0.004250 (0.075534) | 0.034894 / 0.037052 (-0.002158) | 0.304739 / 0.258489 (0.046250) | 0.359457 / 0.293841 (0.065616) | 0.034194 / 0.128546 (-0.094352) | 0.011348 / 0.075646 (-0.064298) | 0.324340 / 0.419271 (-0.094931) | 0.041071 / 0.043533 (-0.002461) | 0.304437 / 0.255139 (0.049298) | 0.335517 / 0.283200 (0.052317) | 0.087787 / 0.141683 (-0.053895) | 1.467293 / 1.452155 (0.015138) | 1.543529 / 1.492716 (0.050813) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.187654 / 0.018006 (0.169648) | 0.426558 / 0.000490 (0.426068) | 0.003585 / 0.000200 (0.003385) | 0.000076 / 0.000054 (0.000022) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023410 / 0.037411 (-0.014001) | 0.097065 / 0.014526 (0.082539) | 0.105358 / 0.176557 (-0.071198) | 0.140941 / 0.737135 (-0.596195) | 0.109484 / 0.296338 (-0.186855) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420334 / 0.215209 (0.205125) | 4.223235 / 2.077655 (2.145581) | 1.866213 / 1.504120 (0.362093) | 1.673829 / 1.541195 (0.132634) | 1.757828 / 1.468490 (0.289337) | 0.702203 / 4.584777 (-3.882574) | 3.426192 / 3.745712 (-0.319521) | 1.950392 / 5.269862 (-3.319470) | 1.286139 / 4.565676 (-3.279538) | 0.082858 / 0.424275 (-0.341417) | 0.012587 / 0.007607 (0.004980) | 0.531920 / 0.226044 (0.305876) | 5.344425 / 2.268929 (3.075497) | 2.337875 / 55.444624 (-53.106749) | 1.967713 / 6.876477 (-4.908764) | 2.022075 / 2.142072 (-0.119997) | 0.829267 / 4.805227 (-3.975961) | 0.151712 / 6.500664 (-6.348952) | 0.066617 / 0.075469 (-0.008852) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.251867 / 1.841788 (-0.589921) | 13.861756 / 8.074308 (5.787448) | 14.236309 / 10.191392 (4.044917) | 0.138215 / 0.680424 (-0.542209) | 0.028600 / 0.534201 (-0.505601) | 0.395890 / 0.579283 (-0.183393) | 0.403971 / 0.434364 (-0.030393) | 0.479033 / 0.540337 (-0.061305) | 0.564019 / 1.386936 (-0.822917) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006845 / 0.011353 (-0.004508) | 0.004544 / 0.011008 (-0.006464) | 0.098719 / 0.038508 (0.060211) | 0.029082 / 0.023109 (0.005973) | 0.426011 / 0.275898 (0.150113) | 0.447185 / 0.323480 (0.123705) | 0.005203 / 0.007986 (-0.002783) | 0.004790 / 0.004328 (0.000462) | 0.076446 / 0.004250 (0.072196) | 0.040649 / 0.037052 (0.003596) | 0.414810 / 0.258489 (0.156321) | 0.452082 / 0.293841 (0.158241) | 0.031842 / 0.128546 (-0.096704) | 0.011575 / 0.075646 (-0.064071) | 0.320710 / 0.419271 (-0.098561) | 0.044994 / 0.043533 (0.001461) | 0.415645 / 0.255139 (0.160506) | 0.435235 / 0.283200 (0.152035) | 0.091756 / 0.141683 (-0.049927) | 1.493900 / 1.452155 (0.041746) | 1.592353 / 1.492716 (0.099637) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.264710 / 0.018006 (0.246703) | 0.410553 / 0.000490 (0.410064) | 0.024497 / 0.000200 (0.024297) | 0.000232 / 0.000054 (0.000178) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024452 / 0.037411 (-0.012959) | 0.102673 / 0.014526 (0.088147) | 0.107787 / 0.176557 (-0.068770) | 0.147368 / 0.737135 (-0.589767) | 0.112127 / 0.296338 (-0.184211) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.471294 / 0.215209 (0.256085) | 4.711638 / 2.077655 (2.633983) | 2.436819 / 1.504120 (0.932699) | 2.238540 / 1.541195 (0.697345) | 2.334134 / 1.468490 (0.865644) | 0.697668 / 4.584777 (-3.887108) | 3.414332 / 3.745712 (-0.331380) | 2.783248 / 5.269862 (-2.486614) | 1.529599 / 4.565676 (-3.036078) | 0.082626 / 0.424275 (-0.341649) | 0.012385 / 0.007607 (0.004778) | 0.580486 / 0.226044 (0.354441) | 5.837914 / 2.268929 (3.568986) | 2.915129 / 55.444624 (-52.529495) | 2.606254 / 6.876477 (-4.270223) | 2.659031 / 2.142072 (0.516958) | 0.810431 / 4.805227 (-3.994796) | 0.151666 / 6.500664 (-6.348998) | 0.066873 / 0.075469 (-0.008596) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.259933 / 1.841788 (-0.581855) | 14.052388 / 8.074308 (5.978080) | 13.356141 / 10.191392 (3.164749) | 0.138416 / 0.680424 (-0.542008) | 0.016582 / 0.534201 (-0.517619) | 0.378110 / 0.579283 (-0.201173) | 0.385089 / 0.434364 (-0.049275) | 0.465299 / 0.540337 (-0.075038) | 0.559780 / 1.386936 (-0.827156) |

</details>
</details>

![](https://cml.dev/watermark.png#d2859fd4d4beca33f21539a6e1df9a7f012cbd10 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011945 / 0.011353 (0.000592) | 0.006128 / 0.011008 (-0.004880) | 0.128926 / 0.038508 (0.090418) | 0.037708 / 0.023109 (0.014599) | 0.373449 / 0.275898 (0.097551) | 0.423567 / 0.323480 (0.100088) | 0.009848 / 0.007986 (0.001863) | 0.006097 / 0.004328 (0.001769) | 0.098275 / 0.004250 (0.094024) | 0.043199 / 0.037052 (0.006147) | 0.376848 / 0.258489 (0.118359) | 0.441819 / 0.293841 (0.147978) | 0.055094 / 0.128546 (-0.073453) | 0.019704 / 0.075646 (-0.055942) | 0.422746 / 0.419271 (0.003474) | 0.061764 / 0.043533 (0.018231) | 0.381056 / 0.255139 (0.125917) | 0.419343 / 0.283200 (0.136144) | 0.116720 / 0.141683 (-0.024963) | 1.763913 / 1.452155 (0.311759) | 1.872306 / 1.492716 (0.379589) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198651 / 0.018006 (0.180645) | 0.560565 / 0.000490 (0.560075) | 0.004269 / 0.000200 (0.004069) | 0.000114 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027307 / 0.037411 (-0.010104) | 0.128276 / 0.014526 (0.113750) | 0.129015 / 0.176557 (-0.047542) | 0.167269 / 0.737135 (-0.569866) | 0.143955 / 0.296338 (-0.152384) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.564954 / 0.215209 (0.349745) | 5.810570 / 2.077655 (3.732916) | 2.456382 / 1.504120 (0.952262) | 2.115809 / 1.541195 (0.574614) | 2.097363 / 1.468490 (0.628873) | 1.189712 / 4.584777 (-3.395065) | 5.318287 / 3.745712 (1.572575) | 2.965763 / 5.269862 (-2.304099) | 2.177958 / 4.565676 (-2.387719) | 0.144135 / 0.424275 (-0.280140) | 0.014348 / 0.007607 (0.006741) | 0.781715 / 0.226044 (0.555670) | 7.688349 / 2.268929 (5.419421) | 3.189260 / 55.444624 (-52.255365) | 2.552340 / 6.876477 (-4.324137) | 2.559312 / 2.142072 (0.417240) | 1.490755 / 4.805227 (-3.314473) | 0.257908 / 6.500664 (-6.242756) | 0.082016 / 0.075469 (0.006547) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.565735 / 1.841788 (-0.276053) | 17.660338 / 8.074308 (9.586030) | 19.493573 / 10.191392 (9.302181) | 0.241310 / 0.680424 (-0.439114) | 0.043485 / 0.534201 (-0.490716) | 0.557397 / 0.579283 (-0.021886) | 0.624385 / 0.434364 (0.190021) | 0.634601 / 0.540337 (0.094264) | 0.743140 / 1.386936 (-0.643796) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.010134 / 0.011353 (-0.001219) | 0.005858 / 0.011008 (-0.005150) | 0.128741 / 0.038508 (0.090232) | 0.036769 / 0.023109 (0.013660) | 0.470894 / 0.275898 (0.194996) | 0.524302 / 0.323480 (0.200822) | 0.006830 / 0.007986 (-0.001156) | 0.006166 / 0.004328 (0.001838) | 0.094875 / 0.004250 (0.090625) | 0.051201 / 0.037052 (0.014148) | 0.493992 / 0.258489 (0.235503) | 0.510540 / 0.293841 (0.216699) | 0.056354 / 0.128546 (-0.072192) | 0.020512 / 0.075646 (-0.055134) | 0.417809 / 0.419271 (-0.001463) | 0.061941 / 0.043533 (0.018408) | 0.498883 / 0.255139 (0.243744) | 0.480762 / 0.283200 (0.197563) | 0.110753 / 0.141683 (-0.030930) | 1.914096 / 1.452155 (0.461941) | 1.941338 / 1.492716 (0.448622) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.237955 / 0.018006 (0.219949) | 0.518136 / 0.000490 (0.517647) | 0.000475 / 0.000200 (0.000275) | 0.000095 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032947 / 0.037411 (-0.004465) | 0.127857 / 0.014526 (0.113331) | 0.133911 / 0.176557 (-0.042646) | 0.188406 / 0.737135 (-0.548729) | 0.143939 / 0.296338 (-0.152400) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.787553 / 0.215209 (0.572344) | 6.976572 / 2.077655 (4.898918) | 2.897964 / 1.504120 (1.393844) | 2.545906 / 1.541195 (1.004711) | 2.622111 / 1.468490 (1.153620) | 1.278283 / 4.584777 (-3.306494) | 5.650447 / 3.745712 (1.904734) | 4.955835 / 5.269862 (-0.314027) | 2.767946 / 4.565676 (-1.797731) | 0.149385 / 0.424275 (-0.274890) | 0.014340 / 0.007607 (0.006733) | 0.861774 / 0.226044 (0.635730) | 8.660985 / 2.268929 (6.392057) | 3.685611 / 55.444624 (-51.759014) | 2.963087 / 6.876477 (-3.913390) | 3.020746 / 2.142072 (0.878673) | 1.538908 / 4.805227 (-3.266319) | 0.285875 / 6.500664 (-6.214789) | 0.080337 / 0.075469 (0.004867) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.575155 / 1.841788 (-0.266633) | 17.548946 / 8.074308 (9.474638) | 19.954104 / 10.191392 (9.762712) | 0.242025 / 0.680424 (-0.438398) | 0.025586 / 0.534201 (-0.508615) | 0.515676 / 0.579283 (-0.063607) | 0.607035 / 0.434364 (0.172671) | 0.633597 / 0.540337 (0.093259) | 0.744577 / 1.386936 (-0.642359) |

</details>
</details>

![](https://cml.dev/watermark.png#6529cada7879496bf18dd686e4d281de81d6203c "CML watermark")