Set dev version
State: closed | Author: albertvillanova (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6220). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005950 / 0.011353 (-0.005403) | 0.003578 / 0.011008 (-0.007431) | 0.079327 / 0.038508 (0.040819) | 0.057862 / 0.023109 (0.034752) | 0.317288 / 0.275898 (0.041390) | 0.358210 / 0.323480 (0.034730) | 0.004685 / 0.007986 (-0.003301) | 0.002879 / 0.004328 (-0.001450) | 0.062355 / 0.004250 (0.058105) | 0.045093 / 0.037052 (0.008041) | 0.322520 / 0.258489 (0.064031) | 0.367114 / 0.293841 (0.073273) | 0.027233 / 0.128546 (-0.101313) | 0.007941 / 0.075646 (-0.067705) | 0.260511 / 0.419271 (-0.158761) | 0.044355 / 0.043533 (0.000822) | 0.332993 / 0.255139 (0.077854) | 0.351363 / 0.283200 (0.068163) | 0.020784 / 0.141683 (-0.120899) | 1.429044 / 1.452155 (-0.023111) | 1.489355 / 1.492716 (-0.003362) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.180903 / 0.018006 (0.162897) | 0.421566 / 0.000490 (0.421077) | 0.003259 / 0.000200 (0.003059) | 0.000068 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023765 / 0.037411 (-0.013646) | 0.072815 / 0.014526 (0.058289) | 0.084592 / 0.176557 (-0.091965) | 0.143556 / 0.737135 (-0.593579) | 0.083591 / 0.296338 (-0.212748) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401896 / 0.215209 (0.186687) | 4.006344 / 2.077655 (1.928689) | 2.092280 / 1.504120 (0.588160) | 1.937828 / 1.541195 (0.396633) | 2.026901 / 1.468490 (0.558411) | 0.499999 / 4.584777 (-4.084778) | 3.008715 / 3.745712 (-0.736997) | 2.789735 / 5.269862 (-2.480127) | 1.827319 / 4.565676 (-2.738358) | 0.057413 / 0.424275 (-0.366862) | 0.006716 / 0.007607 (-0.000891) | 0.473061 / 0.226044 (0.247016) | 4.733256 / 2.268929 (2.464327) | 2.403922 / 55.444624 (-53.040702) | 2.017466 / 6.876477 (-4.859011) | 2.209710 / 2.142072 (0.067638) | 0.590813 / 4.805227 (-4.214414) | 0.124760 / 6.500664 (-6.375904) | 0.060976 / 0.075469 (-0.014494) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.229172 / 1.841788 (-0.612616) | 17.924644 / 8.074308 (9.850336) | 13.697347 / 10.191392 (3.505955) | 0.128258 / 0.680424 (-0.552166) | 0.016780 / 0.534201 (-0.517421) | 0.329301 / 0.579283 (-0.249982) | 0.344527 / 0.434364 (-0.089837) | 0.379482 / 0.540337 (-0.160855) | 0.513851 / 1.386936 (-0.873085) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005962 / 0.011353 (-0.005391) | 0.003613 / 0.011008 (-0.007396) | 0.062428 / 0.038508 (0.023920) | 0.058151 / 0.023109 (0.035042) | 0.452926 / 0.275898 (0.177027) | 0.489740 / 0.323480 (0.166260) | 0.006137 / 0.007986 (-0.001848) | 0.002890 / 0.004328 (-0.001438) | 0.062880 / 0.004250 (0.058629) | 0.046175 / 0.037052 (0.009123) | 0.452416 / 0.258489 (0.193927) | 0.486047 / 0.293841 (0.192206) | 0.028517 / 0.128546 (-0.100029) | 0.008102 / 0.075646 (-0.067544) | 0.068251 / 0.419271 (-0.351020) | 0.040569 / 0.043533 (-0.002964) | 0.461306 / 0.255139 (0.206167) | 0.477675 / 0.283200 (0.194475) | 0.020944 / 0.141683 (-0.120739) | 1.414300 / 1.452155 (-0.037855) | 1.502108 / 1.492716 (0.009391) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.217786 / 0.018006 (0.199780) | 0.410757 / 0.000490 (0.410267) | 0.002981 / 0.000200 (0.002781) | 0.000081 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026846 / 0.037411 (-0.010565) | 0.080098 / 0.014526 (0.065572) | 0.090591 / 0.176557 (-0.085965) | 0.144674 / 0.737135 (-0.592461) | 0.091287 / 0.296338 (-0.205052) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458224 / 0.215209 (0.243015) | 4.590541 / 2.077655 (2.512886) | 2.511251 / 1.504120 (1.007131) | 2.329165 / 1.541195 (0.787970) | 2.379187 / 1.468490 (0.910696) | 0.507485 / 4.584777 (-4.077292) | 3.135011 / 3.745712 (-0.610701) | 2.805913 / 5.269862 (-2.463948) | 1.851382 / 4.565676 (-2.714295) | 0.057981 / 0.424275 (-0.366294) | 0.006557 / 0.007607 (-0.001050) | 0.532496 / 0.226044 (0.306452) | 5.348802 / 2.268929 (3.079874) | 2.993379 / 55.444624 (-52.451245) | 2.636372 / 6.876477 (-4.240104) | 2.753219 / 2.142072 (0.611147) | 0.591989 / 4.805227 (-4.213238) | 0.126691 / 6.500664 (-6.373973) | 0.062359 / 0.075469 (-0.013110) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.345498 / 1.841788 (-0.496290) | 18.335767 / 8.074308 (10.261458) | 15.115449 / 10.191392 (4.924057) | 0.147382 / 0.680424 (-0.533041) | 0.017729 / 0.534201 (-0.516472) | 0.334337 / 0.579283 (-0.244946) | 0.359035 / 0.434364 (-0.075329) | 0.386319 / 0.540337 (-0.154019) | 0.536378 / 1.386936 (-0.850558) |

</details>
</details>

![](https://cml.dev/watermark.png#f2b028fd83d74e7701e7b8f2d87e740a989505a7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009136 / 0.011353 (-0.002216) | 0.005567 / 0.011008 (-0.005442) | 0.120320 / 0.038508 (0.081812) | 0.078082 / 0.023109 (0.054973) | 0.405579 / 0.275898 (0.129681) | 0.459714 / 0.323480 (0.136234) | 0.006327 / 0.007986 (-0.001659) | 0.007187 / 0.004328 (0.002859) | 0.084373 / 0.004250 (0.080122) | 0.059727 / 0.037052 (0.022675) | 0.418918 / 0.258489 (0.160429) | 0.486767 / 0.293841 (0.192927) | 0.047715 / 0.128546 (-0.080831) | 0.014417 / 0.075646 (-0.061229) | 0.379847 / 0.419271 (-0.039425) | 0.067472 / 0.043533 (0.023939) | 0.419304 / 0.255139 (0.164166) | 0.466260 / 0.283200 (0.183060) | 0.036872 / 0.141683 (-0.104811) | 1.876273 / 1.452155 (0.424119) | 2.043856 / 1.492716 (0.551140) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.296266 / 0.018006 (0.278260) | 0.601843 / 0.000490 (0.601354) | 0.005663 / 0.000200 (0.005463) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033272 / 0.037411 (-0.004139) | 0.098839 / 0.014526 (0.084313) | 0.124658 / 0.176557 (-0.051899) | 0.190226 / 0.737135 (-0.546909) | 0.119288 / 0.296338 (-0.177051) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.600878 / 0.215209 (0.385668) | 6.011749 / 2.077655 (3.934095) | 2.611809 / 1.504120 (1.107689) | 2.314985 / 1.541195 (0.773790) | 2.398988 / 1.468490 (0.930498) | 0.835577 / 4.584777 (-3.749200) | 5.482848 / 3.745712 (1.737136) | 4.965393 / 5.269862 (-0.304469) | 3.082420 / 4.565676 (-1.483256) | 0.098048 / 0.424275 (-0.326227) | 0.009148 / 0.007607 (0.001541) | 0.725721 / 0.226044 (0.499676) | 7.297429 / 2.268929 (5.028501) | 3.558050 / 55.444624 (-51.886575) | 2.815884 / 6.876477 (-4.060593) | 3.094103 / 2.142072 (0.952031) | 1.023617 / 4.805227 (-3.781610) | 0.222453 / 6.500664 (-6.278211) | 0.081707 / 0.075469 (0.006238) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.788327 / 1.841788 (-0.053461) | 25.285829 / 8.074308 (17.211521) | 21.878811 / 10.191392 (11.687419) | 0.215494 / 0.680424 (-0.464930) | 0.032050 / 0.534201 (-0.502151) | 0.505210 / 0.579283 (-0.074073) | 0.623545 / 0.434364 (0.189181) | 0.583342 / 0.540337 (0.043005) | 0.826497 / 1.386936 (-0.560439) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009640 / 0.011353 (-0.001713) | 0.005479 / 0.011008 (-0.005529) | 0.088940 / 0.038508 (0.050432) | 0.084186 / 0.023109 (0.061077) | 0.552290 / 0.275898 (0.276392) | 0.583296 / 0.323480 (0.259816) | 0.006999 / 0.007986 (-0.000987) | 0.004597 / 0.004328 (0.000269) | 0.089407 / 0.004250 (0.085157) | 0.067210 / 0.037052 (0.030157) | 0.554968 / 0.258489 (0.296479) | 0.595635 / 0.293841 (0.301794) | 0.052245 / 0.128546 (-0.076301) | 0.015914 / 0.075646 (-0.059733) | 0.097037 / 0.419271 (-0.322235) | 0.063954 / 0.043533 (0.020421) | 0.533752 / 0.255139 (0.278614) | 0.573789 / 0.283200 (0.290589) | 0.036526 / 0.141683 (-0.105157) | 1.867713 / 1.452155 (0.415558) | 1.996901 / 1.492716 (0.504185) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.414967 / 0.018006 (0.396961) | 0.632367 / 0.000490 (0.631877) | 0.064061 / 0.000200 (0.063861) | 0.000565 / 0.000054 (0.000510) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035953 / 0.037411 (-0.001458) | 0.112603 / 0.014526 (0.098077) | 0.126227 / 0.176557 (-0.050330) | 0.196881 / 0.737135 (-0.540255) | 0.127635 / 0.296338 (-0.168704) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.674735 / 0.215209 (0.459526) | 6.614578 / 2.077655 (4.536923) | 3.208198 / 1.504120 (1.704078) | 2.870412 / 1.541195 (1.329217) | 2.979358 / 1.468490 (1.510868) | 0.872589 / 4.584777 (-3.712187) | 5.501771 / 3.745712 (1.756059) | 4.865191 / 5.269862 (-0.404671) | 3.075281 / 4.565676 (-1.490396) | 0.098048 / 0.424275 (-0.326227) | 0.009121 / 0.007607 (0.001514) | 0.801639 / 0.226044 (0.575595) | 8.062040 / 2.268929 (5.793111) | 3.996693 / 55.444624 (-51.447931) | 3.343770 / 6.876477 (-3.532706) | 3.555977 / 2.142072 (1.413904) | 1.035050 / 4.805227 (-3.770177) | 0.227552 / 6.500664 (-6.273112) | 0.097733 / 0.075469 (0.022264) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.897210 / 1.841788 (0.055422) | 25.762459 / 8.074308 (17.688151) | 22.771290 / 10.191392 (12.579898) | 0.252650 / 0.680424 (-0.427773) | 0.032534 / 0.534201 (-0.501667) | 0.521047 / 0.579283 (-0.058236) | 0.620850 / 0.434364 (0.186486) | 0.612750 / 0.540337 (0.072413) | 0.837486 / 1.386936 (-0.549451) |

</details>
</details>

![](https://cml.dev/watermark.png#1f522e5bdd73c45f7ba0a03f2ecd4e7de7351f2e "CML watermark")