Set dev version
State: closed | Author: lhoestq (MEMBER)


Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5564). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008810 / 0.011353 (-0.002543) | 0.004583 / 0.011008 (-0.006425) | 0.100787 / 0.038508 (0.062279) | 0.030170 / 0.023109 (0.007061) | 0.301749 / 0.275898 (0.025851) | 0.386958 / 0.323480 (0.063478) | 0.007211 / 0.007986 (-0.000775) | 0.004939 / 0.004328 (0.000611) | 0.078046 / 0.004250 (0.073796) | 0.035672 / 0.037052 (-0.001380) | 0.314403 / 0.258489 (0.055914) | 0.348547 / 0.293841 (0.054706) | 0.034242 / 0.128546 (-0.094304) | 0.011599 / 0.075646 (-0.064047) | 0.321936 / 0.419271 (-0.097336) | 0.043214 / 0.043533 (-0.000319) | 0.298782 / 0.255139 (0.043643) | 0.334513 / 0.283200 (0.051313) | 0.091630 / 0.141683 (-0.050053) | 1.518194 / 1.452155 (0.066039) | 1.553665 / 1.492716 (0.060949) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.196322 / 0.018006 (0.178316) | 0.427280 / 0.000490 (0.426790) | 0.001933 / 0.000200 (0.001733) | 0.000069 / 0.000054 (0.000015) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023190 / 0.037411 (-0.014221) | 0.097387 / 0.014526 (0.082862) | 0.104532 / 0.176557 (-0.072024) | 0.166670 / 0.737135 (-0.570465) | 0.108787 / 0.296338 (-0.187552) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.415776 / 0.215209 (0.200567) | 4.135899 / 2.077655 (2.058244) | 1.857600 / 1.504120 (0.353480) | 1.654099 / 1.541195 (0.112904) | 1.729102 / 1.468490 (0.260612) | 0.695946 / 4.584777 (-3.888831) | 3.352776 / 3.745712 (-0.392936) | 2.754443 / 5.269862 (-2.515418) | 1.517181 / 4.565676 (-3.048495) | 0.082782 / 0.424275 (-0.341493) | 0.012431 / 0.007607 (0.004824) | 0.526593 / 0.226044 (0.300548) | 5.263051 / 2.268929 (2.994123) | 2.290713 / 55.444624 (-53.153911) | 1.953017 / 6.876477 (-4.923460) | 1.998419 / 2.142072 (-0.143653) | 0.817055 / 4.805227 (-3.988173) | 0.148213 / 6.500664 (-6.352451) | 0.065527 / 0.075469 (-0.009942) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.254275 / 1.841788 (-0.587513) | 13.618962 / 8.074308 (5.544654) | 14.057134 / 10.191392 (3.865742) | 0.137180 / 0.680424 (-0.543244) | 0.028460 / 0.534201 (-0.505741) | 0.393836 / 0.579283 (-0.185447) | 0.406665 / 0.434364 (-0.027699) | 0.476812 / 0.540337 (-0.063526) | 0.561047 / 1.386936 (-0.825889) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006483 / 0.011353 (-0.004870) | 0.004525 / 0.011008 (-0.006483) | 0.075696 / 0.038508 (0.037188) | 0.027306 / 0.023109 (0.004197) | 0.359141 / 0.275898 (0.083243) | 0.394595 / 0.323480 (0.071115) | 0.004907 / 0.007986 (-0.003079) | 0.003403 / 0.004328 (-0.000925) | 0.074473 / 0.004250 (0.070223) | 0.037801 / 0.037052 (0.000749) | 0.359350 / 0.258489 (0.100861) | 0.411902 / 0.293841 (0.118061) | 0.032280 / 0.128546 (-0.096267) | 0.011728 / 0.075646 (-0.063918) | 0.085692 / 0.419271 (-0.333580) | 0.047779 / 0.043533 (0.004246) | 0.348820 / 0.255139 (0.093681) | 0.389396 / 0.283200 (0.106197) | 0.094923 / 0.141683 (-0.046760) | 1.507137 / 1.452155 (0.054982) | 1.556873 / 1.492716 (0.064157) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.197510 / 0.018006 (0.179504) | 0.413885 / 0.000490 (0.413395) | 0.002527 / 0.000200 (0.002327) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024571 / 0.037411 (-0.012840) | 0.099845 / 0.014526 (0.085319) | 0.108130 / 0.176557 (-0.068426) | 0.176153 / 0.737135 (-0.560982) | 0.111907 / 0.296338 (-0.184432) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436393 / 0.215209 (0.221184) | 4.343296 / 2.077655 (2.265642) | 2.056062 / 1.504120 (0.551942) | 1.855372 / 1.541195 (0.314177) | 1.946429 / 1.468490 (0.477939) | 0.701862 / 4.584777 (-3.882915) | 3.337115 / 3.745712 (-0.408597) | 2.755416 / 5.269862 (-2.514446) | 1.335596 / 4.565676 (-3.230081) | 0.083938 / 0.424275 (-0.340337) | 0.012914 / 0.007607 (0.005307) | 0.530272 / 0.226044 (0.304228) | 5.307739 / 2.268929 (3.038810) | 2.506435 / 55.444624 (-52.938189) | 2.170830 / 6.876477 (-4.705646) | 2.224641 / 2.142072 (0.082568) | 0.804416 / 4.805227 (-4.000811) | 0.151594 / 6.500664 (-6.349070) | 0.067221 / 0.075469 (-0.008248) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.257063 / 1.841788 (-0.584725) | 14.054346 / 8.074308 (5.980038) | 13.490649 / 10.191392 (3.299257) | 0.139320 / 0.680424 (-0.541104) | 0.016501 / 0.534201 (-0.517700) | 0.382655 / 0.579283 (-0.196629) | 0.383305 / 0.434364 (-0.051059) | 0.465091 / 0.540337 (-0.075247) | 0.552552 / 1.386936 (-0.834384) |

</details>
</details>

![](https://cml.dev/watermark.png#c480083958126c40bb7bdba8e1eeb3945a8fe6ea "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.011278 / 0.011353 (-0.000075) | 0.007351 / 0.011008 (-0.003657) | 0.131145 / 0.038508 (0.092637) | 0.041585 / 0.023109 (0.018476) | 0.410230 / 0.275898 (0.134332) | 0.464069 / 0.323480 (0.140589) | 0.010228 / 0.007986 (0.002242) | 0.005324 / 0.004328 (0.000996) | 0.102680 / 0.004250 (0.098430) | 0.041644 / 0.037052 (0.004592) | 0.439127 / 0.258489 (0.180638) | 0.467828 / 0.293841 (0.173987) | 0.054373 / 0.128546 (-0.074173) | 0.019495 / 0.075646 (-0.056152) | 0.432425 / 0.419271 (0.013153) | 0.056863 / 0.043533 (0.013331) | 0.405883 / 0.255139 (0.150744) | 0.452786 / 0.283200 (0.169586) | 0.109888 / 0.141683 (-0.031795) | 1.797015 / 1.452155 (0.344860) | 1.985937 / 1.492716 (0.493221) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.275121 / 0.018006 (0.257115) | 0.587585 / 0.000490 (0.587095) | 0.005557 / 0.000200 (0.005357) | 0.000118 / 0.000054 (0.000064) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032968 / 0.037411 (-0.004443) | 0.135886 / 0.014526 (0.121360) | 0.154000 / 0.176557 (-0.022557) | 0.233345 / 0.737135 (-0.503790) | 0.144125 / 0.296338 (-0.152214) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.613056 / 0.215209 (0.397847) | 6.206135 / 2.077655 (4.128480) | 2.686989 / 1.504120 (1.182869) | 2.389946 / 1.541195 (0.848751) | 2.437506 / 1.468490 (0.969016) | 1.255900 / 4.584777 (-3.328877) | 5.654803 / 3.745712 (1.909091) | 5.467693 / 5.269862 (0.197832) | 2.872397 / 4.565676 (-1.693279) | 0.145658 / 0.424275 (-0.278617) | 0.016883 / 0.007607 (0.009276) | 0.793820 / 0.226044 (0.567775) | 7.961881 / 2.268929 (5.692952) | 3.617422 / 55.444624 (-51.827203) | 2.795185 / 6.876477 (-4.081292) | 2.881726 / 2.142072 (0.739653) | 1.434543 / 4.805227 (-3.370685) | 0.252206 / 6.500664 (-6.248458) | 0.094694 / 0.075469 (0.019225) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.552401 / 1.841788 (-0.289386) | 18.436068 / 8.074308 (10.361760) | 22.539049 / 10.191392 (12.347657) | 0.269471 / 0.680424 (-0.410953) | 0.053242 / 0.534201 (-0.480959) | 0.568325 / 0.579283 (-0.010958) | 0.660339 / 0.434364 (0.225975) | 0.689507 / 0.540337 (0.149169) | 0.836785 / 1.386936 (-0.550151) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009853 / 0.011353 (-0.001500) | 0.009752 / 0.011008 (-0.001256) | 0.095422 / 0.038508 (0.056914) | 0.037760 / 0.023109 (0.014651) | 0.450898 / 0.275898 (0.175000) | 0.501671 / 0.323480 (0.178191) | 0.006748 / 0.007986 (-0.001237) | 0.005054 / 0.004328 (0.000725) | 0.099382 / 0.004250 (0.095131) | 0.058078 / 0.037052 (0.021026) | 0.447606 / 0.258489 (0.189116) | 0.503887 / 0.293841 (0.210046) | 0.054579 / 0.128546 (-0.073967) | 0.026150 / 0.075646 (-0.049496) | 0.113042 / 0.419271 (-0.306230) | 0.061049 / 0.043533 (0.017516) | 0.437831 / 0.255139 (0.182692) | 0.480830 / 0.283200 (0.197630) | 0.121199 / 0.141683 (-0.020484) | 1.795409 / 1.452155 (0.343254) | 1.911207 / 1.492716 (0.418491) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.311774 / 0.018006 (0.293768) | 0.602027 / 0.000490 (0.601537) | 0.000651 / 0.000200 (0.000451) | 0.000136 / 0.000054 (0.000081) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035185 / 0.037411 (-0.002227) | 0.149574 / 0.014526 (0.135048) | 0.153672 / 0.176557 (-0.022884) | 0.241720 / 0.737135 (-0.495416) | 0.153543 / 0.296338 (-0.142795) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.678508 / 0.215209 (0.463299) | 6.535313 / 2.077655 (4.457658) | 2.840175 / 1.504120 (1.336055) | 2.458141 / 1.541195 (0.916947) | 2.551369 / 1.468490 (1.082879) | 1.339117 / 4.584777 (-3.245660) | 5.844429 / 3.745712 (2.098717) | 3.221100 / 5.269862 (-2.048762) | 2.114844 / 4.565676 (-2.450833) | 0.149263 / 0.424275 (-0.275012) | 0.016101 / 0.007607 (0.008494) | 0.830650 / 0.226044 (0.604605) | 8.096655 / 2.268929 (5.827727) | 3.445947 / 55.444624 (-51.998677) | 2.826874 / 6.876477 (-4.049603) | 2.812765 / 2.142072 (0.670693) | 1.453789 / 4.805227 (-3.351438) | 0.263911 / 6.500664 (-6.236753) | 0.082609 / 0.075469 (0.007139) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.651624 / 1.841788 (-0.190163) | 18.703020 / 8.074308 (10.628712) | 21.360445 / 10.191392 (11.169053) | 0.249718 / 0.680424 (-0.430706) | 0.028373 / 0.534201 (-0.505828) | 0.576237 / 0.579283 (-0.003046) | 0.620574 / 0.434364 (0.186210) | 0.684155 / 0.540337 (0.143817) | 0.758950 / 1.386936 (-0.627986) |

</details>
</details>

![](https://cml.dev/watermark.png#f51ef325602bb297a18a75680575cbe9b940b1d9 "CML watermark")