set dev version
State: closed | Author: lhoestq (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6083). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006049 / 0.011353 (-0.005304) | 0.003698 / 0.011008 (-0.007310) | 0.080614 / 0.038508 (0.042106) | 0.060955 / 0.023109 (0.037846) | 0.337119 / 0.275898 (0.061221) | 0.369544 / 0.323480 (0.046064) | 0.004681 / 0.007986 (-0.003305) | 0.002892 / 0.004328 (-0.001436) | 0.062907 / 0.004250 (0.058657) | 0.049235 / 0.037052 (0.012183) | 0.338842 / 0.258489 (0.080353) | 0.371172 / 0.293841 (0.077331) | 0.027016 / 0.128546 (-0.101530) | 0.007940 / 0.075646 (-0.067706) | 0.260902 / 0.419271 (-0.158369) | 0.044566 / 0.043533 (0.001034) | 0.342354 / 0.255139 (0.087215) | 0.359829 / 0.283200 (0.076629) | 0.020801 / 0.141683 (-0.120881) | 1.444111 / 1.452155 (-0.008044) | 1.515595 / 1.492716 (0.022879) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.183446 / 0.018006 (0.165439) | 0.437071 / 0.000490 (0.436581) | 0.003124 / 0.000200 (0.002924) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023760 / 0.037411 (-0.013651) | 0.072812 / 0.014526 (0.058286) | 0.082790 / 0.176557 (-0.093766) | 0.146330 / 0.737135 (-0.590805) | 0.084469 / 0.296338 (-0.211870) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395215 / 0.215209 (0.180006) | 3.953023 / 2.077655 (1.875369) | 1.914268 / 1.504120 (0.410148) | 1.710195 / 1.541195 (0.169001) | 1.782594 / 1.468490 (0.314104) | 0.503651 / 4.584777 (-4.081126) | 3.039656 / 3.745712 (-0.706056) | 4.364691 / 5.269862 (-0.905171) | 2.597762 / 4.565676 (-1.967915) | 0.057384 / 0.424275 (-0.366891) | 0.006419 / 0.007607 (-0.001188) | 0.467214 / 0.226044 (0.241169) | 4.661425 / 2.268929 (2.392497) | 2.341957 / 55.444624 (-53.102667) | 1.977598 / 6.876477 (-4.898878) | 2.178005 / 2.142072 (0.035933) | 0.588492 / 4.805227 (-4.216735) | 0.124972 / 6.500664 (-6.375692) | 0.060902 / 0.075469 (-0.014567) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243092 / 1.841788 (-0.598695) | 18.369971 / 8.074308 (10.295663) | 13.939700 / 10.191392 (3.748308) | 0.149275 / 0.680424 (-0.531149) | 0.016873 / 0.534201 (-0.517328) | 0.334245 / 0.579283 (-0.245038) | 0.353832 / 0.434364 (-0.080532) | 0.382720 / 0.540337 (-0.157617) | 0.534634 / 1.386936 (-0.852302) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005933 / 0.011353 (-0.005420) | 0.003695 / 0.011008 (-0.007313) | 0.063457 / 0.038508 (0.024949) | 0.062347 / 0.023109 (0.039238) | 0.412370 / 0.275898 (0.136472) | 0.450399 / 0.323480 (0.126920) | 0.004627 / 0.007986 (-0.003358) | 0.002822 / 0.004328 (-0.001507) | 0.063819 / 0.004250 (0.059569) | 0.049154 / 0.037052 (0.012101) | 0.428196 / 0.258489 (0.169707) | 0.464109 / 0.293841 (0.170268) | 0.026967 / 0.128546 (-0.101579) | 0.007876 / 0.075646 (-0.067770) | 0.068479 / 0.419271 (-0.350793) | 0.041080 / 0.043533 (-0.002453) | 0.399817 / 0.255139 (0.144678) | 0.426900 / 0.283200 (0.143701) | 0.019931 / 0.141683 (-0.121752) | 1.461642 / 1.452155 (0.009487) | 1.529314 / 1.492716 (0.036598) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230256 / 0.018006 (0.212249) | 0.423442 / 0.000490 (0.422952) | 0.002492 / 0.000200 (0.002292) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025798 / 0.037411 (-0.011613) | 0.077361 / 0.014526 (0.062836) | 0.088454 / 0.176557 (-0.088102) | 0.142137 / 0.737135 (-0.594998) | 0.088213 / 0.296338 (-0.208125) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.417656 / 0.215209 (0.202447) | 4.157095 / 2.077655 (2.079440) | 2.132863 / 1.504120 (0.628743) | 1.967220 / 1.541195 (0.426025) | 2.020505 / 1.468490 (0.552015) | 0.496835 / 4.584777 (-4.087942) | 2.989251 / 3.745712 (-0.756462) | 2.849315 / 5.269862 (-2.420546) | 1.848941 / 4.565676 (-2.716736) | 0.057307 / 0.424275 (-0.366968) | 0.006825 / 0.007607 (-0.000782) | 0.489103 / 0.226044 (0.263059) | 4.904776 / 2.268929 (2.635847) | 2.593914 / 55.444624 (-52.850710) | 2.253384 / 6.876477 (-4.623093) | 2.426384 / 2.142072 (0.284312) | 0.592467 / 4.805227 (-4.212760) | 0.126122 / 6.500664 (-6.374542) | 0.063160 / 0.075469 (-0.012309) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.313020 / 1.841788 (-0.528768) | 18.343984 / 8.074308 (10.269676) | 13.763060 / 10.191392 (3.571668) | 0.146312 / 0.680424 (-0.534111) | 0.016980 / 0.534201 (-0.517221) | 0.339572 / 0.579283 (-0.239711) | 0.351310 / 0.434364 (-0.083054) | 0.397616 / 0.540337 (-0.142721) | 0.536879 / 1.386936 (-0.850057) |

</details>
</details>

![](https://cml.dev/watermark.png#73bed12ecda17d1573fd3bf73ed5db24d3622f86 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009979 / 0.011353 (-0.001374) | 0.005024 / 0.011008 (-0.005984) | 0.096566 / 0.038508 (0.058058) | 0.081181 / 0.023109 (0.058072) | 0.398415 / 0.275898 (0.122517) | 0.513971 / 0.323480 (0.190491) | 0.006716 / 0.007986 (-0.001269) | 0.004350 / 0.004328 (0.000022) | 0.071418 / 0.004250 (0.067168) | 0.065002 / 0.037052 (0.027949) | 0.424791 / 0.258489 (0.166302) | 0.442369 / 0.293841 (0.148528) | 0.054540 / 0.128546 (-0.074007) | 0.014067 / 0.075646 (-0.061580) | 0.368930 / 0.419271 (-0.050341) | 0.082468 / 0.043533 (0.038935) | 0.419875 / 0.255139 (0.164736) | 0.508308 / 0.283200 (0.225108) | 0.050411 / 0.141683 (-0.091272) | 1.582271 / 1.452155 (0.130116) | 1.842033 / 1.492716 (0.349317) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.290427 / 0.018006 (0.272420) | 0.594736 / 0.000490 (0.594246) | 0.007058 / 0.000200 (0.006858) | 0.000149 / 0.000054 (0.000095) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027085 / 0.037411 (-0.010326) | 0.087626 / 0.014526 (0.073101) | 0.094299 / 0.176557 (-0.082257) | 0.160169 / 0.737135 (-0.576966) | 0.101474 / 0.296338 (-0.194864) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.545845 / 0.215209 (0.330636) | 5.674389 / 2.077655 (3.596734) | 2.489065 / 1.504120 (0.984945) | 2.166674 / 1.541195 (0.625479) | 2.166925 / 1.468490 (0.698434) | 0.791244 / 4.584777 (-3.793533) | 4.944878 / 3.745712 (1.199165) | 4.121628 / 5.269862 (-1.148234) | 2.701262 / 4.565676 (-1.864415) | 0.087609 / 0.424275 (-0.336666) | 0.006945 / 0.007607 (-0.000662) | 0.668478 / 0.226044 (0.442434) | 6.552813 / 2.268929 (4.283885) | 3.164698 / 55.444624 (-52.279927) | 2.447333 / 6.876477 (-4.429144) | 2.608271 / 2.142072 (0.466198) | 0.954202 / 4.805227 (-3.851025) | 0.187730 / 6.500664 (-6.312934) | 0.063229 / 0.075469 (-0.012240) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.461042 / 1.841788 (-0.380746) | 21.601409 / 8.074308 (13.527101) | 18.553604 / 10.191392 (8.362212) | 0.234571 / 0.680424 (-0.445853) | 0.027119 / 0.534201 (-0.507082) | 0.423448 / 0.579283 (-0.155835) | 0.556397 / 0.434364 (0.122033) | 0.493958 / 0.540337 (-0.046379) | 0.711345 / 1.386936 (-0.675591) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008637 / 0.011353 (-0.002716) | 0.014450 / 0.011008 (0.003442) | 0.084135 / 0.038508 (0.045627) | 0.080513 / 0.023109 (0.057403) | 0.557941 / 0.275898 (0.282042) | 0.563199 / 0.323480 (0.239719) | 0.006475 / 0.007986 (-0.001510) | 0.004407 / 0.004328 (0.000078) | 0.088537 / 0.004250 (0.084287) | 0.060871 / 0.037052 (0.023819) | 0.593077 / 0.258489 (0.334588) | 0.615572 / 0.293841 (0.321732) | 0.050157 / 0.128546 (-0.078389) | 0.014313 / 0.075646 (-0.061333) | 0.091784 / 0.419271 (-0.327487) | 0.065649 / 0.043533 (0.022116) | 0.532569 / 0.255139 (0.277430) | 0.580775 / 0.283200 (0.297575) | 0.036434 / 0.141683 (-0.105249) | 2.080051 / 1.452155 (0.627896) | 1.907430 / 1.492716 (0.414713) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.297763 / 0.018006 (0.279757) | 0.670408 / 0.000490 (0.669918) | 0.000467 / 0.000200 (0.000267) | 0.000082 / 0.000054 (0.000027) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030297 / 0.037411 (-0.007114) | 0.100310 / 0.014526 (0.085784) | 0.113158 / 0.176557 (-0.063398) | 0.149599 / 0.737135 (-0.587536) | 0.102620 / 0.296338 (-0.193718) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.616588 / 0.215209 (0.401379) | 6.572262 / 2.077655 (4.494608) | 2.830748 / 1.504120 (1.326628) | 2.478441 / 1.541195 (0.937246) | 2.573017 / 1.468490 (1.104527) | 0.844154 / 4.584777 (-3.740623) | 5.161625 / 3.745712 (1.415913) | 4.541114 / 5.269862 (-0.728748) | 2.907804 / 4.565676 (-1.657872) | 0.097044 / 0.424275 (-0.327231) | 0.008692 / 0.007607 (0.001085) | 0.806640 / 0.226044 (0.580595) | 7.620521 / 2.268929 (5.351593) | 3.587100 / 55.444624 (-51.857524) | 2.901319 / 6.876477 (-3.975157) | 3.091288 / 2.142072 (0.949215) | 1.056109 / 4.805227 (-3.749118) | 0.209860 / 6.500664 (-6.290804) | 0.079575 / 0.075469 (0.004106) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.966194 / 1.841788 (0.124407) | 28.040515 / 8.074308 (19.966207) | 25.848647 / 10.191392 (15.657255) | 0.255472 / 0.680424 (-0.424951) | 0.036154 / 0.534201 (-0.498046) | 0.515168 / 0.579283 (-0.064115) | 0.696092 / 0.434364 (0.261728) | 0.602712 / 0.540337 (0.062374) | 0.781091 / 1.386936 (-0.605845) |

</details>
</details>

![](https://cml.dev/watermark.png#6f641aca7fbb1f21da48c087a5c10e76f4c6be35 "CML watermark")