Remove unused allowed_extensions param
Opened by albertvillanova (closed)

This PR removes unused `allowed_extensions` parameter from `create_builder_configs_from_metadata_configs`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009055 / 0.011353 (-0.002298) | 0.008835 / 0.011008 (-0.002173) | 0.117048 / 0.038508 (0.078540) | 0.096268 / 0.023109 (0.073159) | 0.474678 / 0.275898 (0.198780) | 0.550509 / 0.323480 (0.227029) | 0.005552 / 0.007986 (-0.002434) | 0.004315 / 0.004328 (-0.000013) | 0.094336 / 0.004250 (0.090086) | 0.061945 / 0.037052 (0.024892) | 0.461422 / 0.258489 (0.202933) | 0.521271 / 0.293841 (0.227430) | 0.049116 / 0.128546 (-0.079430) | 0.015007 / 0.075646 (-0.060639) | 0.414351 / 0.419271 (-0.004920) | 0.137520 / 0.043533 (0.093987) | 0.465627 / 0.255139 (0.210488) | 0.537244 / 0.283200 (0.254044) | 0.068577 / 0.141683 (-0.073106) | 1.921373 / 1.452155 (0.469219) | 2.506653 / 1.492716 (1.013937) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.273970 / 0.018006 (0.255963) | 0.750295 / 0.000490 (0.749805) | 0.004241 / 0.000200 (0.004041) | 0.000128 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033793 / 0.037411 (-0.003618) | 0.105562 / 0.014526 (0.091037) | 0.131771 / 0.176557 (-0.044786) | 0.196890 / 0.737135 (-0.540245) | 0.119842 / 0.296338 (-0.176496) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.634881 / 0.215209 (0.419672) | 6.069221 / 2.077655 (3.991566) | 2.678765 / 1.504120 (1.174646) | 2.460309 / 1.541195 (0.919114) | 2.517579 / 1.468490 (1.049089) | 0.869558 / 4.584777 (-3.715219) | 5.407686 / 3.745712 (1.661974) | 4.920687 / 5.269862 (-0.349175) | 3.130066 / 4.565676 (-1.435611) | 0.100337 / 0.424275 (-0.323938) | 0.009615 / 0.007607 (0.002008) | 0.745275 / 0.226044 (0.519231) | 7.577890 / 2.268929 (5.308962) | 3.607887 / 55.444624 (-51.836738) | 2.922211 / 6.876477 (-3.954266) | 3.205592 / 2.142072 (1.063519) | 1.052298 / 4.805227 (-3.752929) | 0.218798 / 6.500664 (-6.281866) | 0.082137 / 0.075469 (0.006667) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.696551 / 1.841788 (-0.145237) | 24.946074 / 8.074308 (16.871766) | 23.114202 / 10.191392 (12.922810) | 0.220498 / 0.680424 (-0.459925) | 0.029388 / 0.534201 (-0.504813) | 0.494721 / 0.579283 (-0.084562) | 0.603085 / 0.434364 (0.168722) | 0.573093 / 0.540337 (0.032756) | 0.784937 / 1.386936 (-0.601999) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009642 / 0.011353 (-0.001711) | 0.007551 / 0.011008 (-0.003457) | 0.085224 / 0.038508 (0.046716) | 0.099493 / 0.023109 (0.076384) | 0.503824 / 0.275898 (0.227926) | 0.546583 / 0.323480 (0.223103) | 0.006385 / 0.007986 (-0.001601) | 0.004751 / 0.004328 (0.000423) | 0.084699 / 0.004250 (0.080449) | 0.067875 / 0.037052 (0.030823) | 0.485313 / 0.258489 (0.226824) | 0.535808 / 0.293841 (0.241967) | 0.049935 / 0.128546 (-0.078611) | 0.014427 / 0.075646 (-0.061219) | 0.095531 / 0.419271 (-0.323741) | 0.068487 / 0.043533 (0.024954) | 0.502204 / 0.255139 (0.247065) | 0.514393 / 0.283200 (0.231193) | 0.037350 / 0.141683 (-0.104333) | 1.849380 / 1.452155 (0.397226) | 1.920151 / 1.492716 (0.427434) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.298363 / 0.018006 (0.280357) | 0.651555 / 0.000490 (0.651065) | 0.005910 / 0.000200 (0.005710) | 0.000103 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039170 / 0.037411 (0.001758) | 0.106436 / 0.014526 (0.091910) | 0.129880 / 0.176557 (-0.046677) | 0.185401 / 0.737135 (-0.551734) | 0.125732 / 0.296338 (-0.170607) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.643248 / 0.215209 (0.428039) | 6.374807 / 2.077655 (4.297152) | 3.057296 / 1.504120 (1.553176) | 2.779534 / 1.541195 (1.238340) | 2.790165 / 1.468490 (1.321675) | 0.841580 / 4.584777 (-3.743197) | 5.371478 / 3.745712 (1.625766) | 4.973251 / 5.269862 (-0.296610) | 3.235817 / 4.565676 (-1.329860) | 0.097276 / 0.424275 (-0.326999) | 0.008840 / 0.007607 (0.001233) | 0.728678 / 0.226044 (0.502634) | 7.526382 / 2.268929 (5.257454) | 3.792550 / 55.444624 (-51.652074) | 3.439134 / 6.876477 (-3.437342) | 3.466626 / 2.142072 (1.324553) | 1.035894 / 4.805227 (-3.769333) | 0.211670 / 6.500664 (-6.288994) | 0.087596 / 0.075469 (0.012127) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.782755 / 1.841788 (-0.059033) | 25.704407 / 8.074308 (17.630099) | 23.799672 / 10.191392 (13.608280) | 0.233952 / 0.680424 (-0.446472) | 0.030810 / 0.534201 (-0.503391) | 0.505857 / 0.579283 (-0.073426) | 0.629331 / 0.434364 (0.194967) | 0.608530 / 0.540337 (0.068192) | 0.813688 / 1.386936 (-0.573248) |

</details>
</details>

![](https://cml.dev/watermark.png#ed4d6bb5f1331576c41b04acd9872a5349a0915c "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006401 / 0.011353 (-0.004952) | 0.003916 / 0.011008 (-0.007092) | 0.083976 / 0.038508 (0.045468) | 0.072583 / 0.023109 (0.049474) | 0.322747 / 0.275898 (0.046849) | 0.345159 / 0.323480 (0.021679) | 0.005366 / 0.007986 (-0.002620) | 0.003399 / 0.004328 (-0.000930) | 0.064232 / 0.004250 (0.059982) | 0.053313 / 0.037052 (0.016261) | 0.353127 / 0.258489 (0.094638) | 0.361398 / 0.293841 (0.067557) | 0.030604 / 0.128546 (-0.097942) | 0.008615 / 0.075646 (-0.067031) | 0.285806 / 0.419271 (-0.133466) | 0.050887 / 0.043533 (0.007354) | 0.312293 / 0.255139 (0.057154) | 0.349716 / 0.283200 (0.066516) | 0.024546 / 0.141683 (-0.117137) | 1.472318 / 1.452155 (0.020163) | 1.536063 / 1.492716 (0.043347) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.280012 / 0.018006 (0.262006) | 0.593574 / 0.000490 (0.593085) | 0.004083 / 0.000200 (0.003883) | 0.000195 / 0.000054 (0.000141) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027715 / 0.037411 (-0.009696) | 0.081392 / 0.014526 (0.066866) | 0.096445 / 0.176557 (-0.080112) | 0.152131 / 0.737135 (-0.585004) | 0.094825 / 0.296338 (-0.201514) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.380749 / 0.215209 (0.165540) | 3.806994 / 2.077655 (1.729339) | 1.842544 / 1.504120 (0.338424) | 1.682829 / 1.541195 (0.141635) | 1.701679 / 1.468490 (0.233189) | 0.484830 / 4.584777 (-4.099947) | 3.517359 / 3.745712 (-0.228353) | 3.231211 / 5.269862 (-2.038651) | 2.029371 / 4.565676 (-2.536306) | 0.057199 / 0.424275 (-0.367077) | 0.007653 / 0.007607 (0.000046) | 0.458572 / 0.226044 (0.232528) | 4.579835 / 2.268929 (2.310907) | 2.326467 / 55.444624 (-53.118157) | 1.939646 / 6.876477 (-4.936831) | 2.133150 / 2.142072 (-0.008922) | 0.596251 / 4.805227 (-4.208976) | 0.131979 / 6.500664 (-6.368686) | 0.059226 / 0.075469 (-0.016243) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.234833 / 1.841788 (-0.606955) | 19.475522 / 8.074308 (11.401214) | 14.102760 / 10.191392 (3.911368) | 0.159657 / 0.680424 (-0.520767) | 0.018292 / 0.534201 (-0.515909) | 0.391079 / 0.579283 (-0.188204) | 0.406736 / 0.434364 (-0.027628) | 0.459159 / 0.540337 (-0.081178) | 0.618159 / 1.386936 (-0.768777) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006592 / 0.011353 (-0.004761) | 0.004052 / 0.011008 (-0.006957) | 0.064536 / 0.038508 (0.026028) | 0.075051 / 0.023109 (0.051942) | 0.379596 / 0.275898 (0.103698) | 0.412413 / 0.323480 (0.088933) | 0.005377 / 0.007986 (-0.002608) | 0.003466 / 0.004328 (-0.000863) | 0.064958 / 0.004250 (0.060708) | 0.055265 / 0.037052 (0.018213) | 0.391505 / 0.258489 (0.133016) | 0.425345 / 0.293841 (0.131504) | 0.030750 / 0.128546 (-0.097796) | 0.008652 / 0.075646 (-0.066994) | 0.072107 / 0.419271 (-0.347165) | 0.048340 / 0.043533 (0.004807) | 0.387714 / 0.255139 (0.132575) | 0.402602 / 0.283200 (0.119402) | 0.023492 / 0.141683 (-0.118191) | 1.528377 / 1.452155 (0.076222) | 1.574827 / 1.492716 (0.082110) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316999 / 0.018006 (0.298993) | 0.528391 / 0.000490 (0.527901) | 0.005183 / 0.000200 (0.004983) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029670 / 0.037411 (-0.007741) | 0.087130 / 0.014526 (0.072604) | 0.099897 / 0.176557 (-0.076660) | 0.154074 / 0.737135 (-0.583062) | 0.104309 / 0.296338 (-0.192030) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.408804 / 0.215209 (0.193595) | 4.072248 / 2.077655 (1.994593) | 2.103333 / 1.504120 (0.599213) | 1.931972 / 1.541195 (0.390777) | 1.980132 / 1.468490 (0.511642) | 0.482623 / 4.584777 (-4.102154) | 3.532789 / 3.745712 (-0.212923) | 3.304962 / 5.269862 (-1.964899) | 2.036672 / 4.565676 (-2.529004) | 0.056944 / 0.424275 (-0.367331) | 0.007190 / 0.007607 (-0.000417) | 0.490650 / 0.226044 (0.264606) | 4.903604 / 2.268929 (2.634675) | 2.586247 / 55.444624 (-52.858377) | 2.227631 / 6.876477 (-4.648846) | 2.397286 / 2.142072 (0.255214) | 0.579167 / 4.805227 (-4.226060) | 0.132037 / 6.500664 (-6.368627) | 0.059971 / 0.075469 (-0.015498) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.336430 / 1.841788 (-0.505358) | 19.915846 / 8.074308 (11.841538) | 14.102781 / 10.191392 (3.911389) | 0.147956 / 0.680424 (-0.532468) | 0.018192 / 0.534201 (-0.516009) | 0.397949 / 0.579283 (-0.181334) | 0.408529 / 0.434364 (-0.025835) | 0.479382 / 0.540337 (-0.060955) | 0.659735 / 1.386936 (-0.727201) |

</details>
</details>

![](https://cml.dev/watermark.png#98074122449bc031f7269f298f1c55f20e39b975 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005880 / 0.011353 (-0.005473) | 0.003677 / 0.011008 (-0.007332) | 0.080022 / 0.038508 (0.041514) | 0.055554 / 0.023109 (0.032445) | 0.397449 / 0.275898 (0.121551) | 0.428346 / 0.323480 (0.104867) | 0.004613 / 0.007986 (-0.003373) | 0.002873 / 0.004328 (-0.001455) | 0.062226 / 0.004250 (0.057976) | 0.044721 / 0.037052 (0.007669) | 0.404792 / 0.258489 (0.146303) | 0.437467 / 0.293841 (0.143626) | 0.027166 / 0.128546 (-0.101381) | 0.008077 / 0.075646 (-0.067569) | 0.260469 / 0.419271 (-0.158803) | 0.043551 / 0.043533 (0.000018) | 0.401712 / 0.255139 (0.146573) | 0.427294 / 0.283200 (0.144094) | 0.021243 / 0.141683 (-0.120440) | 1.464553 / 1.452155 (0.012398) | 1.507112 / 1.492716 (0.014396) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198415 / 0.018006 (0.180408) | 0.427940 / 0.000490 (0.427450) | 0.004236 / 0.000200 (0.004036) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023759 / 0.037411 (-0.013652) | 0.073262 / 0.014526 (0.058736) | 0.677113 / 0.176557 (0.500557) | 0.194964 / 0.737135 (-0.542172) | 0.086121 / 0.296338 (-0.210217) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.401176 / 0.215209 (0.185967) | 4.028688 / 2.077655 (1.951034) | 2.026804 / 1.504120 (0.522685) | 1.887964 / 1.541195 (0.346770) | 2.008991 / 1.468490 (0.540501) | 0.498847 / 4.584777 (-4.085930) | 3.015920 / 3.745712 (-0.729792) | 2.837019 / 5.269862 (-2.432843) | 1.849976 / 4.565676 (-2.715701) | 0.057545 / 0.424275 (-0.366730) | 0.006645 / 0.007607 (-0.000962) | 0.470225 / 0.226044 (0.244180) | 4.720910 / 2.268929 (2.451982) | 2.473693 / 55.444624 (-52.970931) | 2.177525 / 6.876477 (-4.698952) | 2.374702 / 2.142072 (0.232630) | 0.588253 / 4.805227 (-4.216974) | 0.125512 / 6.500664 (-6.375152) | 0.061247 / 0.075469 (-0.014222) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255829 / 1.841788 (-0.585959) | 18.251689 / 8.074308 (10.177381) | 13.690373 / 10.191392 (3.498981) | 0.146928 / 0.680424 (-0.533496) | 0.016534 / 0.534201 (-0.517667) | 0.335249 / 0.579283 (-0.244034) | 0.338940 / 0.434364 (-0.095424) | 0.382170 / 0.540337 (-0.158168) | 0.529570 / 1.386936 (-0.857366) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005920 / 0.011353 (-0.005433) | 0.003557 / 0.011008 (-0.007451) | 0.062776 / 0.038508 (0.024267) | 0.058473 / 0.023109 (0.035364) | 0.358780 / 0.275898 (0.082882) | 0.394161 / 0.323480 (0.070682) | 0.004636 / 0.007986 (-0.003349) | 0.002865 / 0.004328 (-0.001463) | 0.062033 / 0.004250 (0.057782) | 0.047154 / 0.037052 (0.010101) | 0.367718 / 0.258489 (0.109229) | 0.400814 / 0.293841 (0.106973) | 0.026919 / 0.128546 (-0.101628) | 0.008071 / 0.075646 (-0.067575) | 0.067802 / 0.419271 (-0.351469) | 0.040894 / 0.043533 (-0.002638) | 0.358757 / 0.255139 (0.103618) | 0.384971 / 0.283200 (0.101771) | 0.020019 / 0.141683 (-0.121664) | 1.458578 / 1.452155 (0.006423) | 1.525059 / 1.492716 (0.032342) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.207795 / 0.018006 (0.189789) | 0.413201 / 0.000490 (0.412712) | 0.005199 / 0.000200 (0.004999) | 0.000085 / 0.000054 (0.000031) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025716 / 0.037411 (-0.011696) | 0.078434 / 0.014526 (0.063908) | 0.086920 / 0.176557 (-0.089637) | 0.138327 / 0.737135 (-0.598808) | 0.088120 / 0.296338 (-0.208219) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434344 / 0.215209 (0.219135) | 4.343114 / 2.077655 (2.265459) | 2.384439 / 1.504120 (0.880319) | 2.253929 / 1.541195 (0.712735) | 2.306811 / 1.468490 (0.838321) | 0.497572 / 4.584777 (-4.087205) | 3.028794 / 3.745712 (-0.716919) | 2.833484 / 5.269862 (-2.436377) | 1.878918 / 4.565676 (-2.686759) | 0.057133 / 0.424275 (-0.367143) | 0.006357 / 0.007607 (-0.001251) | 0.508019 / 0.226044 (0.281975) | 5.076935 / 2.268929 (2.808007) | 2.745784 / 55.444624 (-52.698841) | 2.476291 / 6.876477 (-4.400186) | 2.677264 / 2.142072 (0.535191) | 0.587173 / 4.805227 (-4.218054) | 0.126373 / 6.500664 (-6.374291) | 0.062815 / 0.075469 (-0.012654) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.355482 / 1.841788 (-0.486305) | 18.818227 / 8.074308 (10.743919) | 13.954289 / 10.191392 (3.762896) | 0.143413 / 0.680424 (-0.537011) | 0.016844 / 0.534201 (-0.517357) | 0.338334 / 0.579283 (-0.240949) | 0.344559 / 0.434364 (-0.089805) | 0.400669 / 0.540337 (-0.139669) | 0.563835 / 1.386936 (-0.823101) |

</details>
</details>

![](https://cml.dev/watermark.png#c02a44715c036b5261686669727394b1308a3a4b "CML watermark")