Set dev version
State: closed | Author: albertvillanova (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6420). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004536 / 0.011353 (-0.006816) | 0.002979 / 0.011008 (-0.008030) | 0.061984 / 0.038508 (0.023476) | 0.029382 / 0.023109 (0.006273) | 0.245237 / 0.275898 (-0.030661) | 0.270571 / 0.323480 (-0.052909) | 0.003956 / 0.007986 (-0.004029) | 0.002453 / 0.004328 (-0.001876) | 0.047967 / 0.004250 (0.043717) | 0.043695 / 0.037052 (0.006643) | 0.248457 / 0.258489 (-0.010032) | 0.283293 / 0.293841 (-0.010548) | 0.023603 / 0.128546 (-0.104943) | 0.007225 / 0.075646 (-0.068422) | 0.200533 / 0.419271 (-0.218739) | 0.055310 / 0.043533 (0.011777) | 0.245152 / 0.255139 (-0.009987) | 0.267187 / 0.283200 (-0.016012) | 0.018158 / 0.141683 (-0.123525) | 1.126079 / 1.452155 (-0.326075) | 1.185137 / 1.492716 (-0.307580) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092436 / 0.018006 (0.074430) | 0.300132 / 0.000490 (0.299642) | 0.000206 / 0.000200 (0.000006) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018476 / 0.037411 (-0.018935) | 0.062827 / 0.014526 (0.048301) | 0.074605 / 0.176557 (-0.101952) | 0.119768 / 0.737135 (-0.617368) | 0.076044 / 0.296338 (-0.220294) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279717 / 0.215209 (0.064508) | 2.752308 / 2.077655 (0.674654) | 1.434954 / 1.504120 (-0.069166) | 1.314700 / 1.541195 (-0.226495) | 1.347689 / 1.468490 (-0.120802) | 0.400332 / 4.584777 (-4.184445) | 2.383024 / 3.745712 (-1.362689) | 2.583130 / 5.269862 (-2.686732) | 1.567670 / 4.565676 (-2.998007) | 0.045446 / 0.424275 (-0.378829) | 0.004813 / 0.007607 (-0.002794) | 0.336191 / 0.226044 (0.110147) | 3.319837 / 2.268929 (1.050909) | 1.816808 / 55.444624 (-53.627817) | 1.539052 / 6.876477 (-5.337424) | 1.550765 / 2.142072 (-0.591307) | 0.484253 / 4.805227 (-4.320974) | 0.100494 / 6.500664 (-6.400170) | 0.041614 / 0.075469 (-0.033855) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.940857 / 1.841788 (-0.900931) | 11.784946 / 8.074308 (3.710638) | 10.397038 / 10.191392 (0.205646) | 0.141458 / 0.680424 (-0.538965) | 0.014193 / 0.534201 (-0.520008) | 0.268304 / 0.579283 (-0.310979) | 0.267059 / 0.434364 (-0.167305) | 0.309389 / 0.540337 (-0.230949) | 0.420628 / 1.386936 (-0.966308) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004776 / 0.011353 (-0.006577) | 0.002941 / 0.011008 (-0.008067) | 0.048659 / 0.038508 (0.010151) | 0.053334 / 0.023109 (0.030225) | 0.273342 / 0.275898 (-0.002556) | 0.302278 / 0.323480 (-0.021202) | 0.004001 / 0.007986 (-0.003984) | 0.002414 / 0.004328 (-0.001914) | 0.047504 / 0.004250 (0.043254) | 0.038581 / 0.037052 (0.001529) | 0.277768 / 0.258489 (0.019279) | 0.306772 / 0.293841 (0.012931) | 0.024146 / 0.128546 (-0.104400) | 0.007233 / 0.075646 (-0.068413) | 0.053308 / 0.419271 (-0.365964) | 0.032617 / 0.043533 (-0.010916) | 0.277390 / 0.255139 (0.022251) | 0.296015 / 0.283200 (0.012816) | 0.018733 / 0.141683 (-0.122950) | 1.124895 / 1.452155 (-0.327260) | 1.182579 / 1.492716 (-0.310137) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093375 / 0.018006 (0.075369) | 0.301555 / 0.000490 (0.301066) | 0.000217 / 0.000200 (0.000017) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021284 / 0.037411 (-0.016127) | 0.070158 / 0.014526 (0.055632) | 0.080187 / 0.176557 (-0.096370) | 0.119282 / 0.737135 (-0.617854) | 0.081672 / 0.296338 (-0.214666) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.314396 / 0.215209 (0.099187) | 2.975114 / 2.077655 (0.897459) | 1.724658 / 1.504120 (0.220539) | 1.604464 / 1.541195 (0.063269) | 1.652736 / 1.468490 (0.184246) | 0.395064 / 4.584777 (-4.189713) | 2.412768 / 3.745712 (-1.332944) | 2.564427 / 5.269862 (-2.705435) | 1.507627 / 4.565676 (-3.058050) | 0.045463 / 0.424275 (-0.378812) | 0.004797 / 0.007607 (-0.002810) | 0.383115 / 0.226044 (0.157071) | 3.501976 / 2.268929 (1.233048) | 2.087512 / 55.444624 (-53.357113) | 1.793132 / 6.876477 (-5.083345) | 1.804178 / 2.142072 (-0.337895) | 0.468287 / 4.805227 (-4.336940) | 0.097247 / 6.500664 (-6.403417) | 0.041139 / 0.075469 (-0.034330) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.976034 / 1.841788 (-0.865754) | 12.431248 / 8.074308 (4.356940) | 10.896064 / 10.191392 (0.704672) | 0.129137 / 0.680424 (-0.551287) | 0.015636 / 0.534201 (-0.518565) | 0.268219 / 0.579283 (-0.311064) | 0.278345 / 0.434364 (-0.156019) | 0.302696 / 0.540337 (-0.237642) | 0.408465 / 1.386936 (-0.978471) |

</details>
</details>

![](https://cml.dev/watermark.png#51c53e94acd7a273c24899c045446df021314cd2 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007703 / 0.011353 (-0.003650) | 0.004614 / 0.011008 (-0.006394) | 0.101425 / 0.038508 (0.062917) | 0.040122 / 0.023109 (0.017013) | 0.398890 / 0.275898 (0.122992) | 0.424392 / 0.323480 (0.100912) | 0.005411 / 0.007986 (-0.002575) | 0.003747 / 0.004328 (-0.000582) | 0.080494 / 0.004250 (0.076243) | 0.059392 / 0.037052 (0.022340) | 0.398025 / 0.258489 (0.139536) | 0.454293 / 0.293841 (0.160452) | 0.043662 / 0.128546 (-0.084884) | 0.013726 / 0.075646 (-0.061920) | 0.352910 / 0.419271 (-0.066362) | 0.088572 / 0.043533 (0.045039) | 0.401677 / 0.255139 (0.146538) | 0.421774 / 0.283200 (0.138575) | 0.033377 / 0.141683 (-0.108305) | 1.728499 / 1.452155 (0.276344) | 1.821557 / 1.492716 (0.328841) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230744 / 0.018006 (0.212738) | 0.496188 / 0.000490 (0.495698) | 0.010315 / 0.000200 (0.010115) | 0.000402 / 0.000054 (0.000348) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028859 / 0.037411 (-0.008552) | 0.089688 / 0.014526 (0.075163) | 0.111697 / 0.176557 (-0.064860) | 0.183238 / 0.737135 (-0.553898) | 0.112407 / 0.296338 (-0.183931) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.558394 / 0.215209 (0.343185) | 5.643048 / 2.077655 (3.565393) | 2.454622 / 1.504120 (0.950502) | 2.183338 / 1.541195 (0.642143) | 2.324793 / 1.468490 (0.856303) | 0.859482 / 4.584777 (-3.725295) | 4.959346 / 3.745712 (1.213634) | 4.599224 / 5.269862 (-0.670638) | 2.764382 / 4.565676 (-1.801295) | 0.089976 / 0.424275 (-0.334299) | 0.008144 / 0.007607 (0.000537) | 0.634675 / 0.226044 (0.408631) | 6.555693 / 2.268929 (4.286765) | 3.080252 / 55.444624 (-52.364373) | 2.442715 / 6.876477 (-4.433762) | 2.475126 / 2.142072 (0.333053) | 0.986459 / 4.805227 (-3.818768) | 0.193859 / 6.500664 (-6.306805) | 0.063652 / 0.075469 (-0.011817) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.545318 / 1.841788 (-0.296469) | 21.928751 / 8.074308 (13.854442) | 20.598229 / 10.191392 (10.406837) | 0.234046 / 0.680424 (-0.446377) | 0.025947 / 0.534201 (-0.508254) | 0.459773 / 0.579283 (-0.119510) | 0.598026 / 0.434364 (0.163662) | 0.555260 / 0.540337 (0.014922) | 0.782767 / 1.386936 (-0.604169) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009322 / 0.011353 (-0.002030) | 0.004650 / 0.011008 (-0.006358) | 0.079326 / 0.038508 (0.040818) | 0.079112 / 0.023109 (0.056003) | 0.428708 / 0.275898 (0.152810) | 0.481647 / 0.323480 (0.158168) | 0.006419 / 0.007986 (-0.001566) | 0.003878 / 0.004328 (-0.000450) | 0.079013 / 0.004250 (0.074762) | 0.058107 / 0.037052 (0.021055) | 0.436967 / 0.258489 (0.178478) | 0.501120 / 0.293841 (0.207279) | 0.052972 / 0.128546 (-0.075574) | 0.014414 / 0.075646 (-0.061232) | 0.098587 / 0.419271 (-0.320685) | 0.061626 / 0.043533 (0.018093) | 0.451623 / 0.255139 (0.196484) | 0.468893 / 0.283200 (0.185693) | 0.032479 / 0.141683 (-0.109203) | 1.911743 / 1.452155 (0.459588) | 1.969024 / 1.492716 (0.476308) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.232015 / 0.018006 (0.214009) | 0.508637 / 0.000490 (0.508147) | 0.005470 / 0.000200 (0.005270) | 0.000131 / 0.000054 (0.000076) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035345 / 0.037411 (-0.002066) | 0.106319 / 0.014526 (0.091794) | 0.117205 / 0.176557 (-0.059352) | 0.176527 / 0.737135 (-0.560608) | 0.121566 / 0.296338 (-0.174773) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.584920 / 0.215209 (0.369711) | 5.745688 / 2.077655 (3.668034) | 2.519875 / 1.504120 (1.015755) | 2.197593 / 1.541195 (0.656398) | 2.296670 / 1.468490 (0.828180) | 0.831938 / 4.584777 (-3.752839) | 5.130594 / 3.745712 (1.384882) | 4.581385 / 5.269862 (-0.688476) | 2.829516 / 4.565676 (-1.736161) | 0.099015 / 0.424275 (-0.325260) | 0.011468 / 0.007607 (0.003861) | 0.702717 / 0.226044 (0.476672) | 6.856099 / 2.268929 (4.587170) | 3.372966 / 55.444624 (-52.071658) | 2.567664 / 6.876477 (-4.308812) | 2.699200 / 2.142072 (0.557127) | 0.992316 / 4.805227 (-3.812911) | 0.190463 / 6.500664 (-6.310201) | 0.063305 / 0.075469 (-0.012165) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.591491 / 1.841788 (-0.250296) | 21.696492 / 8.074308 (13.622184) | 19.695404 / 10.191392 (9.504012) | 0.222853 / 0.680424 (-0.457571) | 0.032936 / 0.534201 (-0.501265) | 0.431209 / 0.579283 (-0.148074) | 0.543101 / 0.434364 (0.108737) | 0.543427 / 0.540337 (0.003089) | 0.742102 / 1.386936 (-0.644834) |

</details>
</details>

![](https://cml.dev/watermark.png#534a227179265df9093230885613c95390325705 "CML watermark")