set dev version
State: closed | Author: lhoestq (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6709). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005081 / 0.011353 (-0.006272) | 0.004182 / 0.011008 (-0.006826) | 0.063377 / 0.038508 (0.024869) | 0.027880 / 0.023109 (0.004770) | 0.247260 / 0.275898 (-0.028638) | 0.273580 / 0.323480 (-0.049900) | 0.002995 / 0.007986 (-0.004991) | 0.002804 / 0.004328 (-0.001524) | 0.049669 / 0.004250 (0.045418) | 0.042469 / 0.037052 (0.005417) | 0.268606 / 0.258489 (0.010117) | 0.292867 / 0.293841 (-0.000973) | 0.028077 / 0.128546 (-0.100469) | 0.011031 / 0.075646 (-0.064615) | 0.210225 / 0.419271 (-0.209047) | 0.035723 / 0.043533 (-0.007810) | 0.252131 / 0.255139 (-0.003008) | 0.272895 / 0.283200 (-0.010304) | 0.019809 / 0.141683 (-0.121874) | 1.138500 / 1.452155 (-0.313655) | 1.167752 / 1.492716 (-0.324964) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094881 / 0.018006 (0.076875) | 0.300168 / 0.000490 (0.299678) | 0.000207 / 0.000200 (0.000007) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.017917 / 0.037411 (-0.019494) | 0.061854 / 0.014526 (0.047328) | 0.074481 / 0.176557 (-0.102075) | 0.120075 / 0.737135 (-0.617061) | 0.074627 / 0.296338 (-0.221711) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287888 / 0.215209 (0.072679) | 2.770165 / 2.077655 (0.692510) | 1.500071 / 1.504120 (-0.004049) | 1.374857 / 1.541195 (-0.166338) | 1.427291 / 1.468490 (-0.041200) | 0.558431 / 4.584777 (-4.026346) | 2.439352 / 3.745712 (-1.306361) | 2.787471 / 5.269862 (-2.482391) | 1.742636 / 4.565676 (-2.823041) | 0.061716 / 0.424275 (-0.362559) | 0.004961 / 0.007607 (-0.002646) | 0.345209 / 0.226044 (0.119164) | 3.360253 / 2.268929 (1.091325) | 1.847945 / 55.444624 (-53.596680) | 1.595733 / 6.876477 (-5.280744) | 1.642350 / 2.142072 (-0.499723) | 0.638639 / 4.805227 (-4.166588) | 0.116918 / 6.500664 (-6.383746) | 0.042132 / 0.075469 (-0.033338) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.980602 / 1.841788 (-0.861185) | 11.545402 / 8.074308 (3.471094) | 9.452471 / 10.191392 (-0.738921) | 0.129930 / 0.680424 (-0.550494) | 0.014143 / 0.534201 (-0.520058) | 0.290302 / 0.579283 (-0.288981) | 0.263785 / 0.434364 (-0.170579) | 0.339580 / 0.540337 (-0.200758) | 0.450355 / 1.386936 (-0.936581) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005565 / 0.011353 (-0.005788) | 0.003764 / 0.011008 (-0.007244) | 0.050082 / 0.038508 (0.011574) | 0.030354 / 0.023109 (0.007245) | 0.250609 / 0.275898 (-0.025289) | 0.277200 / 0.323480 (-0.046280) | 0.004276 / 0.007986 (-0.003710) | 0.002805 / 0.004328 (-0.001523) | 0.048765 / 0.004250 (0.044514) | 0.045477 / 0.037052 (0.008425) | 0.267704 / 0.258489 (0.009215) | 0.303214 / 0.293841 (0.009373) | 0.029393 / 0.128546 (-0.099153) | 0.010623 / 0.075646 (-0.065023) | 0.058201 / 0.419271 (-0.361070) | 0.053131 / 0.043533 (0.009599) | 0.258682 / 0.255139 (0.003543) | 0.276069 / 0.283200 (-0.007131) | 0.018260 / 0.141683 (-0.123423) | 1.141542 / 1.452155 (-0.310613) | 1.185780 / 1.492716 (-0.306936) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096857 / 0.018006 (0.078850) | 0.300656 / 0.000490 (0.300167) | 0.000450 / 0.000200 (0.000250) | 0.000059 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022416 / 0.037411 (-0.014995) | 0.074781 / 0.014526 (0.060255) | 0.087299 / 0.176557 (-0.089257) | 0.127616 / 0.737135 (-0.609519) | 0.088382 / 0.296338 (-0.207957) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.298639 / 0.215209 (0.083430) | 2.940002 / 2.077655 (0.862347) | 1.709707 / 1.504120 (0.205587) | 1.556502 / 1.541195 (0.015307) | 1.592841 / 1.468490 (0.124351) | 0.570237 / 4.584777 (-4.014539) | 2.467576 / 3.745712 (-1.278137) | 2.741021 / 5.269862 (-2.528840) | 1.776526 / 4.565676 (-2.789151) | 0.063999 / 0.424275 (-0.360276) | 0.005068 / 0.007607 (-0.002539) | 0.360727 / 0.226044 (0.134682) | 3.535404 / 2.268929 (1.266476) | 2.035345 / 55.444624 (-53.409279) | 1.755916 / 6.876477 (-5.120561) | 1.889281 / 2.142072 (-0.252791) | 0.649025 / 4.805227 (-4.156202) | 0.118210 / 6.500664 (-6.382454) | 0.040815 / 0.075469 (-0.034654) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.005650 / 1.841788 (-0.836138) | 12.228314 / 8.074308 (4.154006) | 10.147363 / 10.191392 (-0.044029) | 0.159258 / 0.680424 (-0.521166) | 0.015288 / 0.534201 (-0.518913) | 0.288144 / 0.579283 (-0.291139) | 0.281319 / 0.434364 (-0.153045) | 0.323380 / 0.540337 (-0.216958) | 0.426887 / 1.386936 (-0.960049) |

</details>
</details>

![](https://cml.dev/watermark.png#8b04ccb486f3831b4b0d2474119823efa3815709 "CML watermark")