Issue: Fix archive fs test
State: closed
Author: lhoestq


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008664 / 0.011353 (-0.002689) | 0.004622 / 0.011008 (-0.006387) | 0.101716 / 0.038508 (0.063208) | 0.030044 / 0.023109 (0.006935) | 0.298476 / 0.275898 (0.022578) | 0.360873 / 0.323480 (0.037393) | 0.007012 / 0.007986 (-0.000974) | 0.003409 / 0.004328 (-0.000919) | 0.077731 / 0.004250 (0.073480) | 0.035493 / 0.037052 (-0.001560) | 0.311474 / 0.258489 (0.052985) | 0.357276 / 0.293841 (0.063435) | 0.033909 / 0.128546 (-0.094638) | 0.011315 / 0.075646 (-0.064332) | 0.323149 / 0.419271 (-0.096122) | 0.040678 / 0.043533 (-0.002855) | 0.298487 / 0.255139 (0.043348) | 0.323107 / 0.283200 (0.039907) | 0.086641 / 0.141683 (-0.055042) | 1.452905 / 1.452155 (0.000750) | 1.510953 / 1.492716 (0.018237) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.190607 / 0.018006 (0.172601) | 0.409786 / 0.000490 (0.409297) | 0.000818 / 0.000200 (0.000618) | 0.000075 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023267 / 0.037411 (-0.014144) | 0.095390 / 0.014526 (0.080864) | 0.104381 / 0.176557 (-0.072175) | 0.150735 / 0.737135 (-0.586401) | 0.106876 / 0.296338 (-0.189462) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434259 / 0.215209 (0.219050) | 4.326978 / 2.077655 (2.249323) | 2.036690 / 1.504120 (0.532570) | 1.836459 / 1.541195 (0.295264) | 1.904003 / 1.468490 (0.435513) | 0.697265 / 4.584777 (-3.887512) | 3.435911 / 3.745712 (-0.309802) | 3.240918 / 5.269862 (-2.028944) | 1.629220 / 4.565676 (-2.936456) | 0.083158 / 0.424275 (-0.341117) | 0.012604 / 0.007607 (0.004997) | 0.539818 / 0.226044 (0.313773) | 5.397860 / 2.268929 (3.128932) | 2.483890 / 55.444624 (-52.960735) | 2.132404 / 6.876477 (-4.744072) | 2.162583 / 2.142072 (0.020510) | 0.817773 / 4.805227 (-3.987454) | 0.151677 / 6.500664 (-6.348987) | 0.066569 / 0.075469 (-0.008900) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243449 / 1.841788 (-0.598339) | 13.699854 / 8.074308 (5.625546) | 13.930979 / 10.191392 (3.739587) | 0.165344 / 0.680424 (-0.515079) | 0.028910 / 0.534201 (-0.505291) | 0.396201 / 0.579283 (-0.183082) | 0.404448 / 0.434364 (-0.029916) | 0.482031 / 0.540337 (-0.058306) | 0.570023 / 1.386936 (-0.816913) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006785 / 0.011353 (-0.004568) | 0.004643 / 0.011008 (-0.006365) | 0.076755 / 0.038508 (0.038247) | 0.027893 / 0.023109 (0.004783) | 0.342539 / 0.275898 (0.066641) | 0.379103 / 0.323480 (0.055623) | 0.005107 / 0.007986 (-0.002879) | 0.003413 / 0.004328 (-0.000915) | 0.075779 / 0.004250 (0.071528) | 0.039251 / 0.037052 (0.002199) | 0.343425 / 0.258489 (0.084935) | 0.385292 / 0.293841 (0.091451) | 0.032229 / 0.128546 (-0.096317) | 0.011666 / 0.075646 (-0.063980) | 0.086452 / 0.419271 (-0.332819) | 0.042918 / 0.043533 (-0.000615) | 0.343145 / 0.255139 (0.088006) | 0.367916 / 0.283200 (0.084717) | 0.090810 / 0.141683 (-0.050873) | 1.471679 / 1.452155 (0.019524) | 1.566683 / 1.492716 (0.073966) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220343 / 0.018006 (0.202336) | 0.396155 / 0.000490 (0.395665) | 0.003831 / 0.000200 (0.003631) | 0.000080 / 0.000054 (0.000025) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024990 / 0.037411 (-0.012421) | 0.101270 / 0.014526 (0.086744) | 0.110115 / 0.176557 (-0.066442) | 0.161770 / 0.737135 (-0.575365) | 0.112187 / 0.296338 (-0.184151) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436199 / 0.215209 (0.220989) | 4.329084 / 2.077655 (2.251429) | 2.043335 / 1.504120 (0.539215) | 1.836799 / 1.541195 (0.295604) | 1.908362 / 1.468490 (0.439872) | 0.700518 / 4.584777 (-3.884259) | 3.418003 / 3.745712 (-0.327710) | 1.860621 / 5.269862 (-3.409241) | 1.171343 / 4.565676 (-3.394334) | 0.083150 / 0.424275 (-0.341125) | 0.012543 / 0.007607 (0.004936) | 0.533528 / 0.226044 (0.307483) | 5.339660 / 2.268929 (3.070732) | 2.499494 / 55.444624 (-52.945131) | 2.154773 / 6.876477 (-4.721704) | 2.198734 / 2.142072 (0.056661) | 0.803383 / 4.805227 (-4.001844) | 0.150980 / 6.500664 (-6.349684) | 0.068050 / 0.075469 (-0.007419) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.309487 / 1.841788 (-0.532301) | 14.177068 / 8.074308 (6.102760) | 13.218912 / 10.191392 (3.027520) | 0.156857 / 0.680424 (-0.523567) | 0.016534 / 0.534201 (-0.517667) | 0.383986 / 0.579283 (-0.195297) | 0.395264 / 0.434364 (-0.039100) | 0.442310 / 0.540337 (-0.098027) | 0.535535 / 1.386936 (-0.851401) |

</details>
</details>

![](https://cml.dev/watermark.png#64e24bca88be711f4fdcb9c18edaddc1db0bbe2e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009446 / 0.011353 (-0.001907) | 0.005061 / 0.011008 (-0.005948) | 0.099783 / 0.038508 (0.061275) | 0.036379 / 0.023109 (0.013270) | 0.296769 / 0.275898 (0.020871) | 0.368990 / 0.323480 (0.045510) | 0.007891 / 0.007986 (-0.000094) | 0.003940 / 0.004328 (-0.000389) | 0.076284 / 0.004250 (0.072034) | 0.044390 / 0.037052 (0.007337) | 0.313373 / 0.258489 (0.054884) | 0.361118 / 0.293841 (0.067277) | 0.039058 / 0.128546 (-0.089488) | 0.012016 / 0.075646 (-0.063631) | 0.334239 / 0.419271 (-0.085033) | 0.047028 / 0.043533 (0.003495) | 0.297766 / 0.255139 (0.042627) | 0.312853 / 0.283200 (0.029653) | 0.099117 / 0.141683 (-0.042566) | 1.475487 / 1.452155 (0.023332) | 1.557487 / 1.492716 (0.064771) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206243 / 0.018006 (0.188237) | 0.443920 / 0.000490 (0.443430) | 0.001404 / 0.000200 (0.001205) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026347 / 0.037411 (-0.011065) | 0.105880 / 0.014526 (0.091354) | 0.116227 / 0.176557 (-0.060330) | 0.157404 / 0.737135 (-0.579732) | 0.121668 / 0.296338 (-0.174671) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.398614 / 0.215209 (0.183405) | 3.970657 / 2.077655 (1.893002) | 1.778899 / 1.504120 (0.274779) | 1.591806 / 1.541195 (0.050611) | 1.687717 / 1.468490 (0.219227) | 0.695399 / 4.584777 (-3.889378) | 3.829281 / 3.745712 (0.083569) | 2.140856 / 5.269862 (-3.129006) | 1.355027 / 4.565676 (-3.210650) | 0.085714 / 0.424275 (-0.338561) | 0.012130 / 0.007607 (0.004523) | 0.505807 / 0.226044 (0.279762) | 5.053098 / 2.268929 (2.784170) | 2.321694 / 55.444624 (-53.122931) | 2.015909 / 6.876477 (-4.860568) | 2.100862 / 2.142072 (-0.041210) | 0.855689 / 4.805227 (-3.949539) | 0.167192 / 6.500664 (-6.333472) | 0.062376 / 0.075469 (-0.013093) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196647 / 1.841788 (-0.645141) | 14.971356 / 8.074308 (6.897048) | 13.897184 / 10.191392 (3.705792) | 0.193267 / 0.680424 (-0.487157) | 0.029252 / 0.534201 (-0.504949) | 0.444885 / 0.579283 (-0.134398) | 0.452792 / 0.434364 (0.018429) | 0.550157 / 0.540337 (0.009819) | 0.658524 / 1.386936 (-0.728412) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007774 / 0.011353 (-0.003579) | 0.005304 / 0.011008 (-0.005704) | 0.075530 / 0.038508 (0.037022) | 0.034930 / 0.023109 (0.011821) | 0.343879 / 0.275898 (0.067981) | 0.386487 / 0.323480 (0.063008) | 0.005998 / 0.007986 (-0.001987) | 0.005619 / 0.004328 (0.001291) | 0.075865 / 0.004250 (0.071614) | 0.050499 / 0.037052 (0.013446) | 0.345503 / 0.258489 (0.087014) | 0.392081 / 0.293841 (0.098240) | 0.037118 / 0.128546 (-0.091429) | 0.012540 / 0.075646 (-0.063107) | 0.086202 / 0.419271 (-0.333069) | 0.050672 / 0.043533 (0.007139) | 0.343622 / 0.255139 (0.088483) | 0.353853 / 0.283200 (0.070653) | 0.105408 / 0.141683 (-0.036274) | 1.460695 / 1.452155 (0.008540) | 1.524270 / 1.492716 (0.031554) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219356 / 0.018006 (0.201350) | 0.440740 / 0.000490 (0.440251) | 0.014313 / 0.000200 (0.014114) | 0.000103 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030297 / 0.037411 (-0.007115) | 0.108723 / 0.014526 (0.094197) | 0.125085 / 0.176557 (-0.051471) | 0.176664 / 0.737135 (-0.560471) | 0.126659 / 0.296338 (-0.169680) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445790 / 0.215209 (0.230581) | 4.241046 / 2.077655 (2.163391) | 2.027381 / 1.504120 (0.523261) | 1.821070 / 1.541195 (0.279876) | 1.934417 / 1.468490 (0.465927) | 0.710897 / 4.584777 (-3.873880) | 3.840397 / 3.745712 (0.094685) | 3.959196 / 5.269862 (-1.310666) | 1.646069 / 4.565676 (-2.919608) | 0.088615 / 0.424275 (-0.335660) | 0.012321 / 0.007607 (0.004714) | 0.523463 / 0.226044 (0.297418) | 5.240147 / 2.268929 (2.971218) | 2.521639 / 55.444624 (-52.922986) | 2.246535 / 6.876477 (-4.629942) | 2.365913 / 2.142072 (0.223841) | 0.851288 / 4.805227 (-3.953939) | 0.170179 / 6.500664 (-6.330485) | 0.064732 / 0.075469 (-0.010737) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255505 / 1.841788 (-0.586283) | 15.305457 / 8.074308 (7.231148) | 13.214186 / 10.191392 (3.022794) | 0.188971 / 0.680424 (-0.491453) | 0.018972 / 0.534201 (-0.515229) | 0.429621 / 0.579283 (-0.149662) | 0.428738 / 0.434364 (-0.005626) | 0.536241 / 0.540337 (-0.004096) | 0.632998 / 1.386936 (-0.753938) |

</details>
</details>

![](https://cml.dev/watermark.png#b64fae9509f6e9da9cabf0ce677966598fc61e38 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008435 / 0.011353 (-0.002918) | 0.004454 / 0.011008 (-0.006554) | 0.099091 / 0.038508 (0.060583) | 0.028890 / 0.023109 (0.005781) | 0.297450 / 0.275898 (0.021551) | 0.329025 / 0.323480 (0.005545) | 0.006584 / 0.007986 (-0.001401) | 0.004669 / 0.004328 (0.000340) | 0.077387 / 0.004250 (0.073137) | 0.033701 / 0.037052 (-0.003352) | 0.301272 / 0.258489 (0.042783) | 0.345401 / 0.293841 (0.051560) | 0.033473 / 0.128546 (-0.095073) | 0.011244 / 0.075646 (-0.064402) | 0.321941 / 0.419271 (-0.097330) | 0.040646 / 0.043533 (-0.002887) | 0.306686 / 0.255139 (0.051547) | 0.321868 / 0.283200 (0.038668) | 0.084281 / 0.141683 (-0.057401) | 1.491414 / 1.452155 (0.039259) | 1.542799 / 1.492716 (0.050083) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.188368 / 0.018006 (0.170362) | 0.398595 / 0.000490 (0.398105) | 0.000805 / 0.000200 (0.000605) | 0.000075 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022690 / 0.037411 (-0.014721) | 0.096795 / 0.014526 (0.082269) | 0.104037 / 0.176557 (-0.072520) | 0.149409 / 0.737135 (-0.587727) | 0.108022 / 0.296338 (-0.188317) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.419316 / 0.215209 (0.204107) | 4.186850 / 2.077655 (2.109196) | 1.920182 / 1.504120 (0.416062) | 1.715493 / 1.541195 (0.174298) | 1.757767 / 1.468490 (0.289277) | 0.692296 / 4.584777 (-3.892480) | 3.342330 / 3.745712 (-0.403382) | 1.842063 / 5.269862 (-3.427798) | 1.150190 / 4.565676 (-3.415487) | 0.082792 / 0.424275 (-0.341483) | 0.012540 / 0.007607 (0.004933) | 0.528867 / 0.226044 (0.302822) | 5.297818 / 2.268929 (3.028890) | 2.313173 / 55.444624 (-53.131451) | 1.941723 / 6.876477 (-4.934754) | 1.982948 / 2.142072 (-0.159125) | 0.808951 / 4.805227 (-3.996276) | 0.149338 / 6.500664 (-6.351326) | 0.064838 / 0.075469 (-0.010631) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.187865 / 1.841788 (-0.653923) | 13.381918 / 8.074308 (5.307610) | 13.730627 / 10.191392 (3.539234) | 0.149976 / 0.680424 (-0.530447) | 0.028249 / 0.534201 (-0.505952) | 0.392591 / 0.579283 (-0.186692) | 0.403451 / 0.434364 (-0.030912) | 0.467484 / 0.540337 (-0.072853) | 0.560296 / 1.386936 (-0.826640) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006440 / 0.011353 (-0.004913) | 0.004488 / 0.011008 (-0.006521) | 0.077875 / 0.038508 (0.039367) | 0.027284 / 0.023109 (0.004174) | 0.341625 / 0.275898 (0.065727) | 0.374960 / 0.323480 (0.051480) | 0.005581 / 0.007986 (-0.002405) | 0.003326 / 0.004328 (-0.001003) | 0.076928 / 0.004250 (0.072677) | 0.038205 / 0.037052 (0.001153) | 0.345933 / 0.258489 (0.087444) | 0.383675 / 0.293841 (0.089834) | 0.031908 / 0.128546 (-0.096638) | 0.011724 / 0.075646 (-0.063922) | 0.086974 / 0.419271 (-0.332298) | 0.043084 / 0.043533 (-0.000449) | 0.339663 / 0.255139 (0.084524) | 0.363782 / 0.283200 (0.080582) | 0.090934 / 0.141683 (-0.050749) | 1.459718 / 1.452155 (0.007563) | 1.541104 / 1.492716 (0.048388) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224005 / 0.018006 (0.205998) | 0.400727 / 0.000490 (0.400238) | 0.000427 / 0.000200 (0.000227) | 0.000061 / 0.000054 (0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024604 / 0.037411 (-0.012807) | 0.099813 / 0.014526 (0.085287) | 0.104034 / 0.176557 (-0.072523) | 0.156245 / 0.737135 (-0.580890) | 0.108739 / 0.296338 (-0.187600) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440500 / 0.215209 (0.225291) | 4.379934 / 2.077655 (2.302279) | 2.075826 / 1.504120 (0.571706) | 1.867635 / 1.541195 (0.326441) | 1.919035 / 1.468490 (0.450545) | 0.696613 / 4.584777 (-3.888164) | 3.334993 / 3.745712 (-0.410720) | 1.857139 / 5.269862 (-3.412723) | 1.160598 / 4.565676 (-3.405079) | 0.083120 / 0.424275 (-0.341155) | 0.012475 / 0.007607 (0.004868) | 0.544607 / 0.226044 (0.318563) | 5.436808 / 2.268929 (3.167879) | 2.518562 / 55.444624 (-52.926063) | 2.158434 / 6.876477 (-4.718042) | 2.170691 / 2.142072 (0.028618) | 0.811297 / 4.805227 (-3.993930) | 0.150675 / 6.500664 (-6.349990) | 0.065655 / 0.075469 (-0.009814) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.277627 / 1.841788 (-0.564160) | 13.833501 / 8.074308 (5.759193) | 13.038718 / 10.191392 (2.847325) | 0.148837 / 0.680424 (-0.531587) | 0.016440 / 0.534201 (-0.517761) | 0.379147 / 0.579283 (-0.200136) | 0.379753 / 0.434364 (-0.054611) | 0.460197 / 0.540337 (-0.080141) | 0.544152 / 1.386936 (-0.842784) |

</details>
</details>

![](https://cml.dev/watermark.png#6e2a235cbab1c91dc5eca0cb123f9c9d9f743461 "CML watermark")