set dev version
State: closed | Author: lhoestq (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6547). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004855 / 0.011353 (-0.006498) | 0.003552 / 0.011008 (-0.007456) | 0.062328 / 0.038508 (0.023820) | 0.031142 / 0.023109 (0.008032) | 0.247726 / 0.275898 (-0.028172) | 0.270951 / 0.323480 (-0.052528) | 0.002887 / 0.007986 (-0.005099) | 0.002663 / 0.004328 (-0.001665) | 0.047888 / 0.004250 (0.043638) | 0.042932 / 0.037052 (0.005880) | 0.253660 / 0.258489 (-0.004829) | 0.274997 / 0.293841 (-0.018844) | 0.027200 / 0.128546 (-0.101347) | 0.010851 / 0.075646 (-0.064796) | 0.206566 / 0.419271 (-0.212706) | 0.035311 / 0.043533 (-0.008222) | 0.254146 / 0.255139 (-0.000993) | 0.269074 / 0.283200 (-0.014126) | 0.019221 / 0.141683 (-0.122462) | 1.101986 / 1.452155 (-0.350169) | 1.155541 / 1.492716 (-0.337175) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.004749 / 0.018006 (-0.013257) | 0.301627 / 0.000490 (0.301138) | 0.000208 / 0.000200 (0.000008) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018205 / 0.037411 (-0.019206) | 0.060420 / 0.014526 (0.045894) | 0.072533 / 0.176557 (-0.104023) | 0.119807 / 0.737135 (-0.617328) | 0.073249 / 0.296338 (-0.223089) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284947 / 0.215209 (0.069738) | 2.796939 / 2.077655 (0.719285) | 1.486076 / 1.504120 (-0.018043) | 1.358247 / 1.541195 (-0.182948) | 1.383680 / 1.468490 (-0.084811) | 0.550253 / 4.584777 (-4.034524) | 2.364783 / 3.745712 (-1.380929) | 2.765631 / 5.269862 (-2.504230) | 1.695694 / 4.565676 (-2.869983) | 0.061519 / 0.424275 (-0.362756) | 0.004914 / 0.007607 (-0.002693) | 0.340370 / 0.226044 (0.114325) | 3.313175 / 2.268929 (1.044247) | 1.805421 / 55.444624 (-53.639203) | 1.532151 / 6.876477 (-5.344325) | 1.541195 / 2.142072 (-0.600878) | 0.625266 / 4.805227 (-4.179961) | 0.119980 / 6.500664 (-6.380684) | 0.042334 / 0.075469 (-0.033135) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.952893 / 1.841788 (-0.888895) | 11.322232 / 8.074308 (3.247924) | 9.982108 / 10.191392 (-0.209284) | 0.130034 / 0.680424 (-0.550389) | 0.013192 / 0.534201 (-0.521009) | 0.286041 / 0.579283 (-0.293243) | 0.269802 / 0.434364 (-0.164562) | 0.323582 / 0.540337 (-0.216755) | 0.428641 / 1.386936 (-0.958295) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005071 / 0.011353 (-0.006282) | 0.003368 / 0.011008 (-0.007640) | 0.049003 / 0.038508 (0.010495) | 0.029507 / 0.023109 (0.006398) | 0.271859 / 0.275898 (-0.004039) | 0.294660 / 0.323480 (-0.028820) | 0.004218 / 0.007986 (-0.003767) | 0.002686 / 0.004328 (-0.001642) | 0.047947 / 0.004250 (0.043696) | 0.044499 / 0.037052 (0.007447) | 0.273982 / 0.258489 (0.015493) | 0.303393 / 0.293841 (0.009552) | 0.029649 / 0.128546 (-0.098898) | 0.010555 / 0.075646 (-0.065091) | 0.057553 / 0.419271 (-0.361718) | 0.051686 / 0.043533 (0.008153) | 0.274079 / 0.255139 (0.018940) | 0.292535 / 0.283200 (0.009335) | 0.019211 / 0.141683 (-0.122472) | 1.130629 / 1.452155 (-0.321526) | 1.196791 / 1.492716 (-0.295925) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093617 / 0.018006 (0.075611) | 0.302698 / 0.000490 (0.302209) | 0.000222 / 0.000200 (0.000022) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022830 / 0.037411 (-0.014581) | 0.077061 / 0.014526 (0.062535) | 0.089464 / 0.176557 (-0.087092) | 0.127487 / 0.737135 (-0.609649) | 0.092133 / 0.296338 (-0.204205) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.295362 / 0.215209 (0.080153) | 2.902251 / 2.077655 (0.824596) | 1.600508 / 1.504120 (0.096388) | 1.477763 / 1.541195 (-0.063431) | 1.492242 / 1.468490 (0.023752) | 0.569347 / 4.584777 (-4.015430) | 2.449873 / 3.745712 (-1.295839) | 2.787207 / 5.269862 (-2.482655) | 1.723852 / 4.565676 (-2.841825) | 0.063076 / 0.424275 (-0.361199) | 0.005060 / 0.007607 (-0.002547) | 0.349614 / 0.226044 (0.123569) | 3.429735 / 2.268929 (1.160806) | 1.953883 / 55.444624 (-53.490741) | 1.664232 / 6.876477 (-5.212245) | 1.648864 / 2.142072 (-0.493209) | 0.640295 / 4.805227 (-4.164932) | 0.117053 / 6.500664 (-6.383611) | 0.041314 / 0.075469 (-0.034156) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.970663 / 1.841788 (-0.871125) | 12.144810 / 8.074308 (4.070502) | 10.938985 / 10.191392 (0.747593) | 0.140502 / 0.680424 (-0.539922) | 0.015522 / 0.534201 (-0.518679) | 0.286629 / 0.579283 (-0.292654) | 0.283695 / 0.434364 (-0.150669) | 0.327298 / 0.540337 (-0.213039) | 0.424635 / 1.386936 (-0.962301) |

</details>
</details>

![](https://cml.dev/watermark.png#e23a59ef7ba2b50d4e5588825c41212a3cfd1331 "CML watermark")