Bump dev version
State: closed | Author: lhoestq (MEMBER)

workaround to fix an issue with transformers CI

https://github.com/huggingface/transformers/pull/24867#discussion_r1266519626

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6047). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006384 / 0.011353 (-0.004969) | 0.003872 / 0.011008 (-0.007136) | 0.083454 / 0.038508 (0.044946) | 0.069120 / 0.023109 (0.046011) | 0.312573 / 0.275898 (0.036675) | 0.345814 / 0.323480 (0.022334) | 0.005729 / 0.007986 (-0.002257) | 0.003225 / 0.004328 (-0.001103) | 0.063950 / 0.004250 (0.059700) | 0.053998 / 0.037052 (0.016946) | 0.316492 / 0.258489 (0.058003) | 0.350738 / 0.293841 (0.056897) | 0.030770 / 0.128546 (-0.097776) | 0.008474 / 0.075646 (-0.067173) | 0.286989 / 0.419271 (-0.132282) | 0.052473 / 0.043533 (0.008940) | 0.314361 / 0.255139 (0.059222) | 0.335170 / 0.283200 (0.051970) | 0.022885 / 0.141683 (-0.118798) | 1.465430 / 1.452155 (0.013275) | 1.527799 / 1.492716 (0.035083) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209377 / 0.018006 (0.191371) | 0.455583 / 0.000490 (0.455094) | 0.003352 / 0.000200 (0.003152) | 0.000080 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026284 / 0.037411 (-0.011127) | 0.080710 / 0.014526 (0.066185) | 0.091741 / 0.176557 (-0.084816) | 0.147602 / 0.737135 (-0.589534) | 0.091173 / 0.296338 (-0.205166) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386592 / 0.215209 (0.171383) | 3.856665 / 2.077655 (1.779011) | 1.835745 / 1.504120 (0.331625) | 1.671814 / 1.541195 (0.130619) | 1.711224 / 1.468490 (0.242734) | 0.484704 / 4.584777 (-4.100073) | 3.649239 / 3.745712 (-0.096473) | 3.784051 / 5.269862 (-1.485810) | 2.241195 / 4.565676 (-2.324482) | 0.056613 / 0.424275 (-0.367662) | 0.007140 / 0.007607 (-0.000467) | 0.464585 / 0.226044 (0.238540) | 4.616537 / 2.268929 (2.347609) | 2.371969 / 55.444624 (-53.072656) | 1.977754 / 6.876477 (-4.898723) | 2.083385 / 2.142072 (-0.058687) | 0.582330 / 4.805227 (-4.222897) | 0.132744 / 6.500664 (-6.367920) | 0.059822 / 0.075469 (-0.015647) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.259566 / 1.841788 (-0.582221) | 18.990166 / 8.074308 (10.915858) | 13.992069 / 10.191392 (3.800677) | 0.160001 / 0.680424 (-0.520423) | 0.018622 / 0.534201 (-0.515579) | 0.392921 / 0.579283 (-0.186362) | 0.418225 / 0.434364 (-0.016139) | 0.471252 / 0.540337 (-0.069086) | 0.653227 / 1.386936 (-0.733709) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006641 / 0.011353 (-0.004712) | 0.003738 / 0.011008 (-0.007271) | 0.064053 / 0.038508 (0.025545) | 0.069467 / 0.023109 (0.046357) | 0.360625 / 0.275898 (0.084727) | 0.394291 / 0.323480 (0.070811) | 0.005236 / 0.007986 (-0.002750) | 0.003304 / 0.004328 (-0.001024) | 0.064078 / 0.004250 (0.059827) | 0.054605 / 0.037052 (0.017552) | 0.374567 / 0.258489 (0.116078) | 0.411227 / 0.293841 (0.117386) | 0.031614 / 0.128546 (-0.096933) | 0.008323 / 0.075646 (-0.067324) | 0.070616 / 0.419271 (-0.348656) | 0.050077 / 0.043533 (0.006544) | 0.362229 / 0.255139 (0.107090) | 0.388310 / 0.283200 (0.105110) | 0.024053 / 0.141683 (-0.117630) | 1.508913 / 1.452155 (0.056759) | 1.562140 / 1.492716 (0.069423) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230172 / 0.018006 (0.212165) | 0.449363 / 0.000490 (0.448873) | 0.002374 / 0.000200 (0.002174) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029813 / 0.037411 (-0.007598) | 0.087298 / 0.014526 (0.072772) | 0.096712 / 0.176557 (-0.079845) | 0.152864 / 0.737135 (-0.584271) | 0.098204 / 0.296338 (-0.198135) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408664 / 0.215209 (0.193455) | 4.075068 / 2.077655 (1.997414) | 2.096365 / 1.504120 (0.592245) | 1.936096 / 1.541195 (0.394901) | 1.961872 / 1.468490 (0.493382) | 0.483383 / 4.584777 (-4.101394) | 3.686926 / 3.745712 (-0.058787) | 4.798824 / 5.269862 (-0.471037) | 2.652279 / 4.565676 (-1.913398) | 0.056695 / 0.424275 (-0.367580) | 0.007592 / 0.007607 (-0.000016) | 0.484710 / 0.226044 (0.258665) | 4.842153 / 2.268929 (2.573225) | 2.636828 / 55.444624 (-52.807796) | 2.243666 / 6.876477 (-4.632811) | 2.375972 / 2.142072 (0.233899) | 0.578544 / 4.805227 (-4.226683) | 0.132579 / 6.500664 (-6.368085) | 0.061287 / 0.075469 (-0.014182) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.360287 / 1.841788 (-0.481501) | 19.464110 / 8.074308 (11.389802) | 14.530875 / 10.191392 (4.339483) | 0.149479 / 0.680424 (-0.530944) | 0.018471 / 0.534201 (-0.515730) | 0.395399 / 0.579283 (-0.183884) | 0.412897 / 0.434364 (-0.021467) | 0.465194 / 0.540337 (-0.075144) | 0.611752 / 1.386936 (-0.775184) |

</details>
</details>

![](https://cml.dev/watermark.png#79a535de98b590da7bc223a6498c59790882f14a "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008986 / 0.011353 (-0.002367) | 0.005104 / 0.011008 (-0.005905) | 0.108371 / 0.038508 (0.069863) | 0.091655 / 0.023109 (0.068546) | 0.430183 / 0.275898 (0.154285) | 0.481387 / 0.323480 (0.157907) | 0.006662 / 0.007986 (-0.001324) | 0.004681 / 0.004328 (0.000353) | 0.089325 / 0.004250 (0.085075) | 0.065096 / 0.037052 (0.028044) | 0.435021 / 0.258489 (0.176532) | 0.478635 / 0.293841 (0.184794) | 0.047628 / 0.128546 (-0.080918) | 0.013496 / 0.075646 (-0.062150) | 0.389661 / 0.419271 (-0.029611) | 0.082260 / 0.043533 (0.038727) | 0.474165 / 0.255139 (0.219026) | 0.464877 / 0.283200 (0.181677) | 0.039784 / 0.141683 (-0.101899) | 1.874694 / 1.452155 (0.422539) | 1.980183 / 1.492716 (0.487467) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.254044 / 0.018006 (0.236038) | 0.631495 / 0.000490 (0.631005) | 0.000628 / 0.000200 (0.000428) | 0.000086 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.038773 / 0.037411 (0.001362) | 0.103681 / 0.014526 (0.089156) | 0.125081 / 0.176557 (-0.051476) | 0.198345 / 0.737135 (-0.538790) | 0.122217 / 0.296338 (-0.174121) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.611677 / 0.215209 (0.396468) | 6.220790 / 2.077655 (4.143135) | 2.729858 / 1.504120 (1.225739) | 2.351944 / 1.541195 (0.810749) | 2.449137 / 1.468490 (0.980647) | 0.896842 / 4.584777 (-3.687935) | 5.537491 / 3.745712 (1.791778) | 8.480182 / 5.269862 (3.210320) | 5.251404 / 4.565676 (0.685728) | 0.100449 / 0.424275 (-0.323826) | 0.009008 / 0.007607 (0.001401) | 0.750060 / 0.226044 (0.524016) | 7.390940 / 2.268929 (5.122011) | 3.478256 / 55.444624 (-51.966369) | 2.883597 / 6.876477 (-3.992880) | 3.082256 / 2.142072 (0.940183) | 1.114339 / 4.805227 (-3.690889) | 0.225389 / 6.500664 (-6.275275) | 0.083972 / 0.075469 (0.008503) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.741522 / 1.841788 (-0.100266) | 25.674700 / 8.074308 (17.600392) | 24.324412 / 10.191392 (14.133020) | 0.257878 / 0.680424 (-0.422546) | 0.038384 / 0.534201 (-0.495817) | 0.508302 / 0.579283 (-0.070981) | 0.612979 / 0.434364 (0.178615) | 0.584366 / 0.540337 (0.044029) | 0.881115 / 1.386936 (-0.505821) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009114 / 0.011353 (-0.002239) | 0.005333 / 0.011008 (-0.005675) | 0.094944 / 0.038508 (0.056436) | 0.099178 / 0.023109 (0.076068) | 0.529813 / 0.275898 (0.253915) | 0.551282 / 0.323480 (0.227802) | 0.006442 / 0.007986 (-0.001543) | 0.004283 / 0.004328 (-0.000045) | 0.084257 / 0.004250 (0.080007) | 0.067557 / 0.037052 (0.030504) | 0.514733 / 0.258489 (0.256244) | 0.568200 / 0.293841 (0.274359) | 0.050969 / 0.128546 (-0.077577) | 0.014495 / 0.075646 (-0.061151) | 0.097089 / 0.419271 (-0.322182) | 0.063142 / 0.043533 (0.019609) | 0.513327 / 0.255139 (0.258188) | 0.520593 / 0.283200 (0.237394) | 0.036824 / 0.141683 (-0.104859) | 1.954875 / 1.452155 (0.502720) | 1.976307 / 1.492716 (0.483591) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.304070 / 0.018006 (0.286063) | 0.611073 / 0.000490 (0.610583) | 0.005027 / 0.000200 (0.004827) | 0.000113 / 0.000054 (0.000059) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.037993 / 0.037411 (0.000582) | 0.115876 / 0.014526 (0.101350) | 0.118087 / 0.176557 (-0.058469) | 0.186437 / 0.737135 (-0.550699) | 0.129883 / 0.296338 (-0.166456) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.658292 / 0.215209 (0.443083) | 6.618257 / 2.077655 (4.540602) | 3.203786 / 1.504120 (1.699667) | 2.858714 / 1.541195 (1.317519) | 2.940974 / 1.468490 (1.472484) | 0.856238 / 4.584777 (-3.728538) | 5.427708 / 3.745712 (1.681996) | 4.810048 / 5.269862 (-0.459813) | 3.120006 / 4.565676 (-1.445671) | 0.098098 / 0.424275 (-0.326177) | 0.010077 / 0.007607 (0.002470) | 0.790890 / 0.226044 (0.564845) | 7.956679 / 2.268929 (5.687750) | 3.955710 / 55.444624 (-51.488914) | 3.446419 / 6.876477 (-3.430057) | 3.541228 / 2.142072 (1.399156) | 1.013420 / 4.805227 (-3.791808) | 0.213741 / 6.500664 (-6.286923) | 0.080857 / 0.075469 (0.005388) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.813265 / 1.841788 (-0.028522) | 25.965199 / 8.074308 (17.890891) | 21.892761 / 10.191392 (11.701369) | 0.257843 / 0.680424 (-0.422580) | 0.029388 / 0.534201 (-0.504813) | 0.510609 / 0.579283 (-0.068674) | 0.626579 / 0.434364 (0.192215) | 0.576865 / 0.540337 (0.036528) | 0.826610 / 1.386936 (-0.560326) |

</details>
</details>

![](https://cml.dev/watermark.png#a1a9c00249b330f97f66ceb86c2939261091f4fe "CML watermark")