Check builder cls default config name in inspect
Opened by lhoestq (closed)

Fix https://github.com/huggingface/datasets-server/issues/1812

this was causing this issue:

```ipython
In [1]: from datasets import *

In [2]: inspect.get_dataset_config_names("aakanksha/udpos")
Out[2]: ['default']

In [3]: load_dataset_builder("aakanksha/udpos").config.name
Out[3]: 'en'
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006591 / 0.011353 (-0.004762) | 0.003991 / 0.011008 (-0.007017) | 0.085197 / 0.038508 (0.046689) | 0.080312 / 0.023109 (0.057202) | 0.342026 / 0.275898 (0.066128) | 0.370749 / 0.323480 (0.047269) | 0.004124 / 0.007986 (-0.003861) | 0.003413 / 0.004328 (-0.000916) | 0.064363 / 0.004250 (0.060113) | 0.055920 / 0.037052 (0.018868) | 0.340667 / 0.258489 (0.082178) | 0.380138 / 0.293841 (0.086297) | 0.031115 / 0.128546 (-0.097431) | 0.008511 / 0.075646 (-0.067135) | 0.289065 / 0.419271 (-0.130207) | 0.052266 / 0.043533 (0.008734) | 0.343808 / 0.255139 (0.088669) | 0.353578 / 0.283200 (0.070378) | 0.024006 / 0.141683 (-0.117676) | 1.490322 / 1.452155 (0.038168) | 1.591133 / 1.492716 (0.098417) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.234718 / 0.018006 (0.216712) | 0.447023 / 0.000490 (0.446533) | 0.009343 / 0.000200 (0.009143) | 0.000259 / 0.000054 (0.000204) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030466 / 0.037411 (-0.006945) | 0.083367 / 0.014526 (0.068841) | 0.100532 / 0.176557 (-0.076024) | 0.158018 / 0.737135 (-0.579117) | 0.098280 / 0.296338 (-0.198059) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408501 / 0.215209 (0.193292) | 4.066937 / 2.077655 (1.989282) | 2.034029 / 1.504120 (0.529909) | 1.842982 / 1.541195 (0.301788) | 1.987319 / 1.468490 (0.518829) | 0.492126 / 4.584777 (-4.092651) | 3.554027 / 3.745712 (-0.191685) | 3.289023 / 5.269862 (-1.980839) | 2.069796 / 4.565676 (-2.495880) | 0.057930 / 0.424275 (-0.366346) | 0.007308 / 0.007607 (-0.000299) | 0.482596 / 0.226044 (0.256552) | 4.830714 / 2.268929 (2.561785) | 2.506787 / 55.444624 (-52.937838) | 2.163498 / 6.876477 (-4.712979) | 2.389135 / 2.142072 (0.247062) | 0.597538 / 4.805227 (-4.207689) | 0.134268 / 6.500664 (-6.366396) | 0.061189 / 0.075469 (-0.014280) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.245328 / 1.841788 (-0.596460) | 19.145151 / 8.074308 (11.070843) | 14.742121 / 10.191392 (4.550729) | 0.144749 / 0.680424 (-0.535675) | 0.018433 / 0.534201 (-0.515768) | 0.391867 / 0.579283 (-0.187416) | 0.416555 / 0.434364 (-0.017809) | 0.454341 / 0.540337 (-0.085997) | 0.646833 / 1.386936 (-0.740103) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006669 / 0.011353 (-0.004684) | 0.004031 / 0.011008 (-0.006978) | 0.064347 / 0.038508 (0.025839) | 0.076857 / 0.023109 (0.053748) | 0.415864 / 0.275898 (0.139966) | 0.468615 / 0.323480 (0.145135) | 0.005383 / 0.007986 (-0.002603) | 0.003314 / 0.004328 (-0.001015) | 0.064829 / 0.004250 (0.060578) | 0.057182 / 0.037052 (0.020129) | 0.417055 / 0.258489 (0.158566) | 0.472725 / 0.293841 (0.178884) | 0.031938 / 0.128546 (-0.096608) | 0.008564 / 0.075646 (-0.067082) | 0.070649 / 0.419271 (-0.348623) | 0.047439 / 0.043533 (0.003906) | 0.409589 / 0.255139 (0.154450) | 0.433700 / 0.283200 (0.150500) | 0.024132 / 0.141683 (-0.117551) | 1.500825 / 1.452155 (0.048670) | 1.592059 / 1.492716 (0.099343) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225652 / 0.018006 (0.207646) | 0.444188 / 0.000490 (0.443698) | 0.004581 / 0.000200 (0.004381) | 0.000104 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033272 / 0.037411 (-0.004139) | 0.096833 / 0.014526 (0.082307) | 0.107134 / 0.176557 (-0.069422) | 0.159299 / 0.737135 (-0.577836) | 0.107533 / 0.296338 (-0.188806) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429100 / 0.215209 (0.213890) | 4.281051 / 2.077655 (2.203396) | 2.318713 / 1.504120 (0.814593) | 2.165645 / 1.541195 (0.624451) | 2.250224 / 1.468490 (0.781734) | 0.495791 / 4.584777 (-4.088986) | 3.591953 / 3.745712 (-0.153760) | 3.303426 / 5.269862 (-1.966436) | 2.076861 / 4.565676 (-2.488816) | 0.058369 / 0.424275 (-0.365906) | 0.007387 / 0.007607 (-0.000220) | 0.501270 / 0.226044 (0.275225) | 5.014987 / 2.268929 (2.746059) | 2.800951 / 55.444624 (-52.643673) | 2.464316 / 6.876477 (-4.412161) | 2.685259 / 2.142072 (0.543187) | 0.584797 / 4.805227 (-4.220430) | 0.131889 / 6.500664 (-6.368775) | 0.061021 / 0.075469 (-0.014448) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.366982 / 1.841788 (-0.474806) | 19.820376 / 8.074308 (11.746068) | 14.968664 / 10.191392 (4.777272) | 0.165344 / 0.680424 (-0.515080) | 0.019956 / 0.534201 (-0.514245) | 0.395843 / 0.579283 (-0.183441) | 0.420854 / 0.434364 (-0.013510) | 0.465065 / 0.540337 (-0.075272) | 0.651531 / 1.386936 (-0.735405) |

</details>
</details>

![](https://cml.dev/watermark.png#980ca0e13300f5392cd87189d5afd5942927afc7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005974 / 0.011353 (-0.005379) | 0.003714 / 0.011008 (-0.007294) | 0.080049 / 0.038508 (0.041541) | 0.061233 / 0.023109 (0.038124) | 0.317187 / 0.275898 (0.041289) | 0.352725 / 0.323480 (0.029245) | 0.004867 / 0.007986 (-0.003119) | 0.002953 / 0.004328 (-0.001376) | 0.063156 / 0.004250 (0.058905) | 0.046752 / 0.037052 (0.009700) | 0.320171 / 0.258489 (0.061682) | 0.367572 / 0.293841 (0.073731) | 0.027253 / 0.128546 (-0.101293) | 0.008100 / 0.075646 (-0.067546) | 0.261206 / 0.419271 (-0.158066) | 0.044581 / 0.043533 (0.001048) | 0.331169 / 0.255139 (0.076030) | 0.348719 / 0.283200 (0.065519) | 0.021397 / 0.141683 (-0.120286) | 1.528315 / 1.452155 (0.076160) | 1.533789 / 1.492716 (0.041073) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.233336 / 0.018006 (0.215329) | 0.416866 / 0.000490 (0.416376) | 0.008805 / 0.000200 (0.008605) | 0.000240 / 0.000054 (0.000186) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024754 / 0.037411 (-0.012657) | 0.073311 / 0.014526 (0.058785) | 0.085419 / 0.176557 (-0.091138) | 0.146380 / 0.737135 (-0.590756) | 0.085545 / 0.296338 (-0.210793) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431426 / 0.215209 (0.216217) | 4.315899 / 2.077655 (2.238244) | 2.232492 / 1.504120 (0.728372) | 2.064174 / 1.541195 (0.522979) | 2.158982 / 1.468490 (0.690492) | 0.499375 / 4.584777 (-4.085402) | 3.093259 / 3.745712 (-0.652454) | 2.848260 / 5.269862 (-2.421601) | 1.853097 / 4.565676 (-2.712579) | 0.057143 / 0.424275 (-0.367132) | 0.006349 / 0.007607 (-0.001258) | 0.507747 / 0.226044 (0.281702) | 5.078872 / 2.268929 (2.809944) | 2.717697 / 55.444624 (-52.726927) | 2.363564 / 6.876477 (-4.512913) | 2.485756 / 2.142072 (0.343684) | 0.595888 / 4.805227 (-4.209340) | 0.127285 / 6.500664 (-6.373379) | 0.060639 / 0.075469 (-0.014830) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.219287 / 1.841788 (-0.622501) | 17.300038 / 8.074308 (9.225730) | 13.747230 / 10.191392 (3.555838) | 0.144841 / 0.680424 (-0.535583) | 0.016587 / 0.534201 (-0.517614) | 0.336891 / 0.579283 (-0.242392) | 0.376128 / 0.434364 (-0.058236) | 0.385749 / 0.540337 (-0.154588) | 0.552218 / 1.386936 (-0.834718) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006477 / 0.011353 (-0.004876) | 0.003709 / 0.011008 (-0.007299) | 0.064708 / 0.038508 (0.026200) | 0.062627 / 0.023109 (0.039518) | 0.444721 / 0.275898 (0.168823) | 0.477825 / 0.323480 (0.154345) | 0.004890 / 0.007986 (-0.003096) | 0.002896 / 0.004328 (-0.001432) | 0.063781 / 0.004250 (0.059530) | 0.050488 / 0.037052 (0.013436) | 0.453466 / 0.258489 (0.194977) | 0.483303 / 0.293841 (0.189462) | 0.028814 / 0.128546 (-0.099732) | 0.008207 / 0.075646 (-0.067440) | 0.070140 / 0.419271 (-0.349131) | 0.041487 / 0.043533 (-0.002045) | 0.454599 / 0.255139 (0.199460) | 0.468374 / 0.283200 (0.185174) | 0.019758 / 0.141683 (-0.121925) | 1.437542 / 1.452155 (-0.014613) | 1.507965 / 1.492716 (0.015249) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223358 / 0.018006 (0.205352) | 0.413824 / 0.000490 (0.413334) | 0.004593 / 0.000200 (0.004393) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026278 / 0.037411 (-0.011134) | 0.081992 / 0.014526 (0.067466) | 0.089969 / 0.176557 (-0.086587) | 0.143668 / 0.737135 (-0.593467) | 0.091273 / 0.296338 (-0.205066) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.461198 / 0.215209 (0.245989) | 4.615398 / 2.077655 (2.537743) | 2.552291 / 1.504120 (1.048171) | 2.373789 / 1.541195 (0.832595) | 2.431591 / 1.468490 (0.963101) | 0.507683 / 4.584777 (-4.077094) | 3.148771 / 3.745712 (-0.596941) | 2.849118 / 5.269862 (-2.420744) | 1.883001 / 4.565676 (-2.682675) | 0.059423 / 0.424275 (-0.364852) | 0.006463 / 0.007607 (-0.001144) | 0.535129 / 0.226044 (0.309085) | 5.362870 / 2.268929 (3.093941) | 3.016548 / 55.444624 (-52.428076) | 2.666205 / 6.876477 (-4.210271) | 2.821396 / 2.142072 (0.679324) | 0.606596 / 4.805227 (-4.198631) | 0.125991 / 6.500664 (-6.374673) | 0.063566 / 0.075469 (-0.011903) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.364771 / 1.841788 (-0.477017) | 18.000713 / 8.074308 (9.926404) | 14.840330 / 10.191392 (4.648937) | 0.144770 / 0.680424 (-0.535653) | 0.018060 / 0.534201 (-0.516141) | 0.334470 / 0.579283 (-0.244813) | 0.387386 / 0.434364 (-0.046978) | 0.398743 / 0.540337 (-0.141595) | 0.555437 / 1.386936 (-0.831499) |

</details>
</details>

![](https://cml.dev/watermark.png#5b974c9af6b45b6ebdbbf4b3418f25506c1c0618 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006491 / 0.011353 (-0.004862) | 0.004058 / 0.011008 (-0.006950) | 0.084462 / 0.038508 (0.045954) | 0.072310 / 0.023109 (0.049201) | 0.352458 / 0.275898 (0.076560) | 0.385829 / 0.323480 (0.062350) | 0.003978 / 0.007986 (-0.004007) | 0.003455 / 0.004328 (-0.000873) | 0.064070 / 0.004250 (0.059819) | 0.055577 / 0.037052 (0.018525) | 0.361288 / 0.258489 (0.102799) | 0.400147 / 0.293841 (0.106306) | 0.030785 / 0.128546 (-0.097761) | 0.008676 / 0.075646 (-0.066971) | 0.287481 / 0.419271 (-0.131791) | 0.052643 / 0.043533 (0.009110) | 0.354670 / 0.255139 (0.099531) | 0.382322 / 0.283200 (0.099122) | 0.025657 / 0.141683 (-0.116026) | 1.486798 / 1.452155 (0.034643) | 1.588439 / 1.492716 (0.095723) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240881 / 0.018006 (0.222875) | 0.463997 / 0.000490 (0.463507) | 0.009688 / 0.000200 (0.009488) | 0.000601 / 0.000054 (0.000546) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029071 / 0.037411 (-0.008340) | 0.083077 / 0.014526 (0.068551) | 0.119857 / 0.176557 (-0.056699) | 0.153387 / 0.737135 (-0.583749) | 0.132162 / 0.296338 (-0.164177) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.383822 / 0.215209 (0.168613) | 3.828572 / 2.077655 (1.750918) | 1.877629 / 1.504120 (0.373509) | 1.708757 / 1.541195 (0.167562) | 1.771658 / 1.468490 (0.303168) | 0.482439 / 4.584777 (-4.102338) | 3.496247 / 3.745712 (-0.249466) | 3.282055 / 5.269862 (-1.987807) | 2.053069 / 4.565676 (-2.512607) | 0.056626 / 0.424275 (-0.367649) | 0.007338 / 0.007607 (-0.000269) | 0.461257 / 0.226044 (0.235213) | 4.605326 / 2.268929 (2.336397) | 2.408365 / 55.444624 (-53.036260) | 1.986550 / 6.876477 (-4.889926) | 2.225220 / 2.142072 (0.083148) | 0.601301 / 4.805227 (-4.203927) | 0.132217 / 6.500664 (-6.368447) | 0.061217 / 0.075469 (-0.014252) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.268706 / 1.841788 (-0.573081) | 18.892026 / 8.074308 (10.817717) | 14.093892 / 10.191392 (3.902500) | 0.162483 / 0.680424 (-0.517941) | 0.018372 / 0.534201 (-0.515829) | 0.391901 / 0.579283 (-0.187382) | 0.401578 / 0.434364 (-0.032786) | 0.456741 / 0.540337 (-0.083596) | 0.646760 / 1.386936 (-0.740176) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006657 / 0.011353 (-0.004696) | 0.003981 / 0.011008 (-0.007027) | 0.066126 / 0.038508 (0.027617) | 0.072673 / 0.023109 (0.049564) | 0.409970 / 0.275898 (0.134072) | 0.430797 / 0.323480 (0.107317) | 0.005477 / 0.007986 (-0.002508) | 0.003362 / 0.004328 (-0.000966) | 0.065532 / 0.004250 (0.061282) | 0.056018 / 0.037052 (0.018966) | 0.406676 / 0.258489 (0.148187) | 0.438516 / 0.293841 (0.144675) | 0.032795 / 0.128546 (-0.095751) | 0.008580 / 0.075646 (-0.067066) | 0.072692 / 0.419271 (-0.346579) | 0.048110 / 0.043533 (0.004577) | 0.396826 / 0.255139 (0.141687) | 0.418442 / 0.283200 (0.135242) | 0.023269 / 0.141683 (-0.118414) | 1.499438 / 1.452155 (0.047283) | 1.568842 / 1.492716 (0.076126) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218729 / 0.018006 (0.200723) | 0.450771 / 0.000490 (0.450281) | 0.004996 / 0.000200 (0.004796) | 0.000086 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031484 / 0.037411 (-0.005928) | 0.092927 / 0.014526 (0.078401) | 0.107849 / 0.176557 (-0.068707) | 0.156658 / 0.737135 (-0.580478) | 0.106373 / 0.296338 (-0.189965) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434658 / 0.215209 (0.219449) | 4.336386 / 2.077655 (2.258731) | 2.322577 / 1.504120 (0.818457) | 2.149505 / 1.541195 (0.608310) | 2.201967 / 1.468490 (0.733476) | 0.496994 / 4.584777 (-4.087783) | 3.533065 / 3.745712 (-0.212647) | 3.235750 / 5.269862 (-2.034112) | 2.034854 / 4.565676 (-2.530823) | 0.058258 / 0.424275 (-0.366017) | 0.007260 / 0.007607 (-0.000347) | 0.509115 / 0.226044 (0.283071) | 5.088427 / 2.268929 (2.819499) | 2.793551 / 55.444624 (-52.651073) | 2.430588 / 6.876477 (-4.445889) | 2.625998 / 2.142072 (0.483926) | 0.611676 / 4.805227 (-4.193552) | 0.133343 / 6.500664 (-6.367321) | 0.059888 / 0.075469 (-0.015581) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.377292 / 1.841788 (-0.464496) | 19.214299 / 8.074308 (11.139991) | 14.629146 / 10.191392 (4.437754) | 0.171283 / 0.680424 (-0.509141) | 0.020348 / 0.534201 (-0.513853) | 0.397823 / 0.579283 (-0.181461) | 0.411590 / 0.434364 (-0.022774) | 0.470850 / 0.540337 (-0.069487) | 0.658667 / 1.386936 (-0.728269) |

</details>
</details>

![](https://cml.dev/watermark.png#a1e1867e932f14233244fb25713f3c94c46ff50a "CML watermark")