Set dev version
State: closed | Author: albertvillanova (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6826). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004893 / 0.011353 (-0.006460) | 0.003238 / 0.011008 (-0.007771) | 0.063143 / 0.038508 (0.024635) | 0.029770 / 0.023109 (0.006661) | 0.229052 / 0.275898 (-0.046846) | 0.254534 / 0.323480 (-0.068945) | 0.003083 / 0.007986 (-0.004903) | 0.002615 / 0.004328 (-0.001714) | 0.049684 / 0.004250 (0.045434) | 0.043745 / 0.037052 (0.006693) | 0.248985 / 0.258489 (-0.009504) | 0.275957 / 0.293841 (-0.017884) | 0.027323 / 0.128546 (-0.101223) | 0.010372 / 0.075646 (-0.065275) | 0.206494 / 0.419271 (-0.212778) | 0.035230 / 0.043533 (-0.008303) | 0.234235 / 0.255139 (-0.020904) | 0.252395 / 0.283200 (-0.030805) | 0.019442 / 0.141683 (-0.122240) | 1.130677 / 1.452155 (-0.321478) | 1.161721 / 1.492716 (-0.330996) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091659 / 0.018006 (0.073653) | 0.301323 / 0.000490 (0.300833) | 0.000212 / 0.000200 (0.000012) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018360 / 0.037411 (-0.019051) | 0.061101 / 0.014526 (0.046575) | 0.072383 / 0.176557 (-0.104174) | 0.117656 / 0.737135 (-0.619479) | 0.073903 / 0.296338 (-0.222436) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.272768 / 0.215209 (0.057558) | 2.655714 / 2.077655 (0.578059) | 1.446254 / 1.504120 (-0.057866) | 1.330543 / 1.541195 (-0.210652) | 1.352527 / 1.468490 (-0.115964) | 0.561428 / 4.584777 (-4.023349) | 2.368182 / 3.745712 (-1.377530) | 2.746508 / 5.269862 (-2.523353) | 1.713972 / 4.565676 (-2.851705) | 0.062046 / 0.424275 (-0.362229) | 0.005427 / 0.007607 (-0.002180) | 0.321652 / 0.226044 (0.095607) | 3.181812 / 2.268929 (0.912883) | 1.766778 / 55.444624 (-53.677846) | 1.492502 / 6.876477 (-5.383975) | 1.534658 / 2.142072 (-0.607415) | 0.640372 / 4.805227 (-4.164856) | 0.118180 / 6.500664 (-6.382484) | 0.042698 / 0.075469 (-0.032771) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.993262 / 1.841788 (-0.848525) | 11.512827 / 8.074308 (3.438518) | 9.602140 / 10.191392 (-0.589252) | 0.144723 / 0.680424 (-0.535701) | 0.014122 / 0.534201 (-0.520079) | 0.302211 / 0.579283 (-0.277072) | 0.268026 / 0.434364 (-0.166338) | 0.326524 / 0.540337 (-0.213813) | 0.423781 / 1.386936 (-0.963155) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005388 / 0.011353 (-0.005965) | 0.003535 / 0.011008 (-0.007473) | 0.050139 / 0.038508 (0.011631) | 0.031813 / 0.023109 (0.008704) | 0.269501 / 0.275898 (-0.006397) | 0.294355 / 0.323480 (-0.029125) | 0.004128 / 0.007986 (-0.003858) | 0.002684 / 0.004328 (-0.001644) | 0.049295 / 0.004250 (0.045045) | 0.040129 / 0.037052 (0.003077) | 0.282406 / 0.258489 (0.023917) | 0.309822 / 0.293841 (0.015981) | 0.028506 / 0.128546 (-0.100040) | 0.010434 / 0.075646 (-0.065213) | 0.057890 / 0.419271 (-0.361382) | 0.032487 / 0.043533 (-0.011046) | 0.270631 / 0.255139 (0.015492) | 0.288734 / 0.283200 (0.005534) | 0.018710 / 0.141683 (-0.122973) | 1.151571 / 1.452155 (-0.300583) | 1.195222 / 1.492716 (-0.297494) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090939 / 0.018006 (0.072932) | 0.300278 / 0.000490 (0.299788) | 0.000202 / 0.000200 (0.000002) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022036 / 0.037411 (-0.015376) | 0.075131 / 0.014526 (0.060605) | 0.087775 / 0.176557 (-0.088782) | 0.125719 / 0.737135 (-0.611416) | 0.088491 / 0.296338 (-0.207848) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300363 / 0.215209 (0.085154) | 2.931852 / 2.077655 (0.854197) | 1.633688 / 1.504120 (0.129568) | 1.512641 / 1.541195 (-0.028554) | 1.527703 / 1.468490 (0.059213) | 0.572781 / 4.584777 (-4.011996) | 2.445950 / 3.745712 (-1.299762) | 2.883667 / 5.269862 (-2.386195) | 1.761396 / 4.565676 (-2.804280) | 0.064422 / 0.424275 (-0.359853) | 0.005332 / 0.007607 (-0.002275) | 0.346730 / 0.226044 (0.120686) | 3.443815 / 2.268929 (1.174886) | 1.988677 / 55.444624 (-53.455948) | 1.707688 / 6.876477 (-5.168789) | 1.694216 / 2.142072 (-0.447856) | 0.634834 / 4.805227 (-4.170393) | 0.115044 / 6.500664 (-6.385620) | 0.040853 / 0.075469 (-0.034616) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.009382 / 1.841788 (-0.832405) | 12.327511 / 8.074308 (4.253203) | 10.123296 / 10.191392 (-0.068097) | 0.130770 / 0.680424 (-0.549654) | 0.015548 / 0.534201 (-0.518653) | 0.286650 / 0.579283 (-0.292633) | 0.270267 / 0.434364 (-0.164097) | 0.333485 / 0.540337 (-0.206852) | 0.428288 / 1.386936 (-0.958648) |

</details>
</details>

![](https://cml.dev/watermark.png#f96e74d5c633cd5435dd526adb4a74631eb05c43 "CML watermark")