set dev version
State: closed | Author: lhoestq (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6064). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006704 / 0.011353 (-0.004649) | 0.004208 / 0.011008 (-0.006800) | 0.085895 / 0.038508 (0.047387) | 0.079303 / 0.023109 (0.056193) | 0.353430 / 0.275898 (0.077532) | 0.390814 / 0.323480 (0.067334) | 0.006565 / 0.007986 (-0.001420) | 0.003588 / 0.004328 (-0.000740) | 0.065249 / 0.004250 (0.060999) | 0.059772 / 0.037052 (0.022720) | 0.356315 / 0.258489 (0.097826) | 0.404812 / 0.293841 (0.110971) | 0.031127 / 0.128546 (-0.097419) | 0.008656 / 0.075646 (-0.066991) | 0.288734 / 0.419271 (-0.130537) | 0.053157 / 0.043533 (0.009625) | 0.354651 / 0.255139 (0.099512) | 0.370590 / 0.283200 (0.087391) | 0.024944 / 0.141683 (-0.116738) | 1.472393 / 1.452155 (0.020238) | 1.548946 / 1.492716 (0.056229) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223430 / 0.018006 (0.205424) | 0.567359 / 0.000490 (0.566870) | 0.006744 / 0.000200 (0.006544) | 0.000094 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030174 / 0.037411 (-0.007237) | 0.084865 / 0.014526 (0.070339) | 0.098986 / 0.176557 (-0.077571) | 0.161458 / 0.737135 (-0.575678) | 0.099198 / 0.296338 (-0.197141) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404324 / 0.215209 (0.189115) | 4.043744 / 2.077655 (1.966090) | 1.972834 / 1.504120 (0.468714) | 1.801634 / 1.541195 (0.260439) | 1.891198 / 1.468490 (0.422708) | 0.488511 / 4.584777 (-4.096266) | 3.566890 / 3.745712 (-0.178823) | 3.369415 / 5.269862 (-1.900447) | 2.054995 / 4.565676 (-2.510682) | 0.057225 / 0.424275 (-0.367050) | 0.007360 / 0.007607 (-0.000247) | 0.471813 / 0.226044 (0.245769) | 4.734397 / 2.268929 (2.465468) | 2.526585 / 55.444624 (-52.918039) | 2.230535 / 6.876477 (-4.645942) | 2.434403 / 2.142072 (0.292330) | 0.630090 / 4.805227 (-4.175137) | 0.138544 / 6.500664 (-6.362120) | 0.060099 / 0.075469 (-0.015370) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.260951 / 1.841788 (-0.580837) | 20.051513 / 8.074308 (11.977204) | 14.675938 / 10.191392 (4.484546) | 0.169535 / 0.680424 (-0.510889) | 0.018574 / 0.534201 (-0.515627) | 0.394255 / 0.579283 (-0.185028) | 0.412713 / 0.434364 (-0.021651) | 0.475891 / 0.540337 (-0.064446) | 0.658223 / 1.386936 (-0.728713) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006969 / 0.011353 (-0.004384) | 0.004417 / 0.011008 (-0.006591) | 0.064399 / 0.038508 (0.025891) | 0.082928 / 0.023109 (0.059819) | 0.402285 / 0.275898 (0.126387) | 0.440032 / 0.323480 (0.116552) | 0.005896 / 0.007986 (-0.002090) | 0.003580 / 0.004328 (-0.000749) | 0.065340 / 0.004250 (0.061090) | 0.060363 / 0.037052 (0.023311) | 0.417413 / 0.258489 (0.158924) | 0.448527 / 0.293841 (0.154686) | 0.032238 / 0.128546 (-0.096308) | 0.008820 / 0.075646 (-0.066826) | 0.071516 / 0.419271 (-0.347755) | 0.050614 / 0.043533 (0.007081) | 0.406565 / 0.255139 (0.151426) | 0.422527 / 0.283200 (0.139328) | 0.025866 / 0.141683 (-0.115817) | 1.512256 / 1.452155 (0.060101) | 1.568433 / 1.492716 (0.075717) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.266521 / 0.018006 (0.248515) | 0.564524 / 0.000490 (0.564034) | 0.005236 / 0.000200 (0.005036) | 0.000085 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031998 / 0.037411 (-0.005413) | 0.090754 / 0.014526 (0.076229) | 0.105954 / 0.176557 (-0.070602) | 0.164506 / 0.737135 (-0.572629) | 0.108792 / 0.296338 (-0.187546) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422044 / 0.215209 (0.206835) | 4.204449 / 2.077655 (2.126795) | 2.232060 / 1.504120 (0.727940) | 2.060389 / 1.541195 (0.519194) | 2.152723 / 1.468490 (0.684233) | 0.488456 / 4.584777 (-4.096321) | 3.591102 / 3.745712 (-0.154611) | 5.250401 / 5.269862 (-0.019461) | 3.060259 / 4.565676 (-1.505417) | 0.057558 / 0.424275 (-0.366717) | 0.007881 / 0.007607 (0.000274) | 0.508631 / 0.226044 (0.282587) | 5.064857 / 2.268929 (2.795928) | 2.719068 / 55.444624 (-52.725556) | 2.389992 / 6.876477 (-4.486485) | 2.595073 / 2.142072 (0.453000) | 0.590179 / 4.805227 (-4.215048) | 0.136149 / 6.500664 (-6.364515) | 0.062546 / 0.075469 (-0.012923) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.369252 / 1.841788 (-0.472535) | 20.637580 / 8.074308 (12.563272) | 14.217129 / 10.191392 (4.025737) | 0.195464 / 0.680424 (-0.484960) | 0.018452 / 0.534201 (-0.515749) | 0.397044 / 0.579283 (-0.182239) | 0.401127 / 0.434364 (-0.033237) | 0.465033 / 0.540337 (-0.075305) | 0.613484 / 1.386936 (-0.773452) |

</details>
</details>

![](https://cml.dev/watermark.png#d9f1651128e50e7887f5e8eaaf6b55fe4cd84fdc "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006793 / 0.011353 (-0.004559) | 0.004374 / 0.011008 (-0.006635) | 0.084958 / 0.038508 (0.046450) | 0.080440 / 0.023109 (0.057331) | 0.317951 / 0.275898 (0.042053) | 0.376133 / 0.323480 (0.052653) | 0.005775 / 0.007986 (-0.002211) | 0.003644 / 0.004328 (-0.000684) | 0.064823 / 0.004250 (0.060573) | 0.059442 / 0.037052 (0.022390) | 0.319636 / 0.258489 (0.061147) | 0.389668 / 0.293841 (0.095827) | 0.031181 / 0.128546 (-0.097365) | 0.008725 / 0.075646 (-0.066921) | 0.288514 / 0.419271 (-0.130757) | 0.053466 / 0.043533 (0.009933) | 0.323131 / 0.255139 (0.067992) | 0.345276 / 0.283200 (0.062076) | 0.025046 / 0.141683 (-0.116637) | 1.491659 / 1.452155 (0.039504) | 1.562105 / 1.492716 (0.069389) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.286325 / 0.018006 (0.268319) | 0.578021 / 0.000490 (0.577531) | 0.007240 / 0.000200 (0.007040) | 0.000095 / 0.000054 (0.000040) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030163 / 0.037411 (-0.007248) | 0.082100 / 0.014526 (0.067574) | 0.098331 / 0.176557 (-0.078225) | 0.160517 / 0.737135 (-0.576618) | 0.098479 / 0.296338 (-0.197859) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401782 / 0.215209 (0.186573) | 4.006330 / 2.077655 (1.928675) | 2.033841 / 1.504120 (0.529721) | 1.853248 / 1.541195 (0.312053) | 1.980046 / 1.468490 (0.511556) | 0.480636 / 4.584777 (-4.104141) | 3.684482 / 3.745712 (-0.061231) | 5.601940 / 5.269862 (0.332079) | 3.369683 / 4.565676 (-1.195993) | 0.057105 / 0.424275 (-0.367170) | 0.007462 / 0.007607 (-0.000145) | 0.474860 / 0.226044 (0.248815) | 4.749624 / 2.268929 (2.480695) | 2.492084 / 55.444624 (-52.952540) | 2.157985 / 6.876477 (-4.718491) | 2.420997 / 2.142072 (0.278925) | 0.574718 / 4.805227 (-4.230509) | 0.134672 / 6.500664 (-6.365992) | 0.061677 / 0.075469 (-0.013792) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.284151 / 1.841788 (-0.557637) | 20.186823 / 8.074308 (12.112515) | 14.247024 / 10.191392 (4.055632) | 0.171606 / 0.680424 (-0.508818) | 0.018619 / 0.534201 (-0.515582) | 0.394156 / 0.579283 (-0.185127) | 0.424684 / 0.434364 (-0.009679) | 0.476056 / 0.540337 (-0.064281) | 0.668751 / 1.386936 (-0.718185) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006807 / 0.011353 (-0.004546) | 0.004142 / 0.011008 (-0.006867) | 0.065503 / 0.038508 (0.026995) | 0.083232 / 0.023109 (0.060122) | 0.378278 / 0.275898 (0.102380) | 0.410191 / 0.323480 (0.086711) | 0.005660 / 0.007986 (-0.002326) | 0.003486 / 0.004328 (-0.000842) | 0.066109 / 0.004250 (0.061859) | 0.059654 / 0.037052 (0.022601) | 0.375965 / 0.258489 (0.117476) | 0.420046 / 0.293841 (0.126205) | 0.031587 / 0.128546 (-0.096959) | 0.008693 / 0.075646 (-0.066953) | 0.071121 / 0.419271 (-0.348151) | 0.049468 / 0.043533 (0.005935) | 0.373785 / 0.255139 (0.118646) | 0.395577 / 0.283200 (0.112377) | 0.024138 / 0.141683 (-0.117545) | 1.465451 / 1.452155 (0.013297) | 1.547565 / 1.492716 (0.054849) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.325241 / 0.018006 (0.307234) | 0.532415 / 0.000490 (0.531925) | 0.004755 / 0.000200 (0.004555) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033472 / 0.037411 (-0.003939) | 0.090574 / 0.014526 (0.076048) | 0.106712 / 0.176557 (-0.069845) | 0.164353 / 0.737135 (-0.572783) | 0.109344 / 0.296338 (-0.186994) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.420161 / 0.215209 (0.204952) | 4.192334 / 2.077655 (2.114679) | 2.178181 / 1.504120 (0.674061) | 2.017405 / 1.541195 (0.476211) | 2.182783 / 1.468490 (0.714293) | 0.484037 / 4.584777 (-4.100740) | 3.641911 / 3.745712 (-0.103801) | 5.543874 / 5.269862 (0.274013) | 3.440084 / 4.565676 (-1.125593) | 0.056662 / 0.424275 (-0.367614) | 0.007773 / 0.007607 (0.000166) | 0.498357 / 0.226044 (0.272313) | 4.951315 / 2.268929 (2.682386) | 2.656732 / 55.444624 (-52.787892) | 2.370566 / 6.876477 (-4.505910) | 2.682289 / 2.142072 (0.540217) | 0.598479 / 4.805227 (-4.206749) | 0.151546 / 6.500664 (-6.349118) | 0.063278 / 0.075469 (-0.012191) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.385897 / 1.841788 (-0.455891) | 20.961851 / 8.074308 (12.887543) | 14.465688 / 10.191392 (4.274296) | 0.166156 / 0.680424 (-0.514268) | 0.018848 / 0.534201 (-0.515353) | 0.401712 / 0.579283 (-0.177571) | 0.416674 / 0.434364 (-0.017690) | 0.471834 / 0.540337 (-0.068503) | 0.622463 / 1.386936 (-0.764473) |

</details>
</details>

![](https://cml.dev/watermark.png#7e3ab9bc6ae8cc42f7e7d01afbd2637d51c3faf6 "CML watermark")