Add trust_remote_code argument
State: closed

Draft about adding `trust_remote_code` to `load_dataset`.

```python
ds = load_dataset(..., trust_remote_code=True)  # run remote code (current default)
```

It would default to `True` (current behavior) and in the next major release it will prompt the user to check the code before running it (we'll communicate on this before doing it of course).

```python
# in the future
ds = load_dataset(...)  # prompt the user to check the code before running it (future default)
ds = load_dataset(..., trust_remote_code=True)  # run remote code
ds = load_dataset(..., trust_remote_code=False)  # disallow remote code
```

Related to https://github.com/huggingface/datasets/issues/6400

Will do a separate PR to use the parquet export when possible

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004947 / 0.011353 (-0.006405) | 0.002961 / 0.011008 (-0.008047) | 0.063474 / 0.038508 (0.024966) | 0.030162 / 0.023109 (0.007053) | 0.232388 / 0.275898 (-0.043511) | 0.257654 / 0.323480 (-0.065826) | 0.002969 / 0.007986 (-0.005017) | 0.002336 / 0.004328 (-0.001993) | 0.049724 / 0.004250 (0.045473) | 0.045608 / 0.037052 (0.008555) | 0.236079 / 0.258489 (-0.022410) | 0.267809 / 0.293841 (-0.026032) | 0.023805 / 0.128546 (-0.104741) | 0.007177 / 0.075646 (-0.068470) | 0.202167 / 0.419271 (-0.217104) | 0.056181 / 0.043533 (0.012648) | 0.256464 / 0.255139 (0.001325) | 0.271908 / 0.283200 (-0.011292) | 0.020211 / 0.141683 (-0.121472) | 1.114112 / 1.452155 (-0.338042) | 1.174879 / 1.492716 (-0.317837) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093457 / 0.018006 (0.075451) | 0.307643 / 0.000490 (0.307154) | 0.000212 / 0.000200 (0.000012) | 0.000047 / 0.000054 (-0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018635 / 0.037411 (-0.018777) | 0.062099 / 0.014526 (0.047573) | 0.073619 / 0.176557 (-0.102938) | 0.119986 / 0.737135 (-0.617149) | 0.075439 / 0.296338 (-0.220899) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280142 / 0.215209 (0.064933) | 2.733790 / 2.077655 (0.656136) | 1.457633 / 1.504120 (-0.046487) | 1.336288 / 1.541195 (-0.204907) | 1.363191 / 1.468490 (-0.105299) | 0.399331 / 4.584777 (-4.185446) | 2.343099 / 3.745712 (-1.402614) | 2.617059 / 5.269862 (-2.652802) | 1.575912 / 4.565676 (-2.989765) | 0.045621 / 0.424275 (-0.378655) | 0.004825 / 0.007607 (-0.002782) | 0.346669 / 0.226044 (0.120625) | 3.225982 / 2.268929 (0.957054) | 1.787067 / 55.444624 (-53.657557) | 1.503883 / 6.876477 (-5.372593) | 1.527593 / 2.142072 (-0.614479) | 0.466806 / 4.805227 (-4.338421) | 0.098537 / 6.500664 (-6.402127) | 0.042028 / 0.075469 (-0.033441) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.945040 / 1.841788 (-0.896748) | 11.970022 / 8.074308 (3.895714) | 10.261176 / 10.191392 (0.069784) | 0.138231 / 0.680424 (-0.542193) | 0.013933 / 0.534201 (-0.520268) | 0.270640 / 0.579283 (-0.308643) | 0.263185 / 0.434364 (-0.171178) | 0.306686 / 0.540337 (-0.233651) | 0.423164 / 1.386936 (-0.963772) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004765 / 0.011353 (-0.006588) | 0.003158 / 0.011008 (-0.007850) | 0.047813 / 0.038508 (0.009305) | 0.053363 / 0.023109 (0.030254) | 0.278570 / 0.275898 (0.002671) | 0.291500 / 0.323480 (-0.031980) | 0.003987 / 0.007986 (-0.003998) | 0.002430 / 0.004328 (-0.001898) | 0.048059 / 0.004250 (0.043809) | 0.038595 / 0.037052 (0.001542) | 0.276383 / 0.258489 (0.017894) | 0.304234 / 0.293841 (0.010393) | 0.024402 / 0.128546 (-0.104144) | 0.007303 / 0.075646 (-0.068343) | 0.055091 / 0.419271 (-0.364180) | 0.032735 / 0.043533 (-0.010797) | 0.270905 / 0.255139 (0.015766) | 0.287181 / 0.283200 (0.003981) | 0.018919 / 0.141683 (-0.122764) | 1.153814 / 1.452155 (-0.298341) | 1.197009 / 1.492716 (-0.295707) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093743 / 0.018006 (0.075737) | 0.302877 / 0.000490 (0.302387) | 0.000223 / 0.000200 (0.000023) | 0.000052 / 0.000054 (-0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021279 / 0.037411 (-0.016133) | 0.070886 / 0.014526 (0.056360) | 0.081628 / 0.176557 (-0.094928) | 0.119721 / 0.737135 (-0.617414) | 0.083093 / 0.296338 (-0.213245) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.297788 / 0.215209 (0.082579) | 2.915235 / 2.077655 (0.837580) | 1.587580 / 1.504120 (0.083460) | 1.461699 / 1.541195 (-0.079495) | 1.520609 / 1.468490 (0.052119) | 0.398363 / 4.584777 (-4.186413) | 2.408415 / 3.745712 (-1.337297) | 2.552776 / 5.269862 (-2.717086) | 1.508219 / 4.565676 (-3.057457) | 0.045884 / 0.424275 (-0.378391) | 0.004842 / 0.007607 (-0.002765) | 0.341376 / 0.226044 (0.115331) | 3.420192 / 2.268929 (1.151264) | 1.974938 / 55.444624 (-53.469686) | 1.678283 / 6.876477 (-5.198194) | 1.702439 / 2.142072 (-0.439633) | 0.467056 / 4.805227 (-4.338172) | 0.098684 / 6.500664 (-6.401980) | 0.041052 / 0.075469 (-0.034417) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.990145 / 1.841788 (-0.851643) | 12.143198 / 8.074308 (4.068890) | 10.911039 / 10.191392 (0.719647) | 0.130384 / 0.680424 (-0.550040) | 0.015602 / 0.534201 (-0.518599) | 0.270799 / 0.579283 (-0.308484) | 0.279060 / 0.434364 (-0.155304) | 0.315108 / 0.540337 (-0.225230) | 0.413576 / 1.386936 (-0.973360) |

</details>
</details>

![](https://cml.dev/watermark.png#d99b8225e28cca88ed9c2d9b1d8e0342762c4ece "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004911 / 0.011353 (-0.006442) | 0.002808 / 0.011008 (-0.008200) | 0.061367 / 0.038508 (0.022859) | 0.050154 / 0.023109 (0.027045) | 0.250403 / 0.275898 (-0.025495) | 0.273831 / 0.323480 (-0.049649) | 0.002914 / 0.007986 (-0.005071) | 0.002493 / 0.004328 (-0.001836) | 0.048288 / 0.004250 (0.044037) | 0.039219 / 0.037052 (0.002167) | 0.260043 / 0.258489 (0.001554) | 0.288177 / 0.293841 (-0.005664) | 0.023123 / 0.128546 (-0.105423) | 0.006981 / 0.075646 (-0.068666) | 0.201306 / 0.419271 (-0.217965) | 0.035670 / 0.043533 (-0.007863) | 0.255237 / 0.255139 (0.000098) | 0.283701 / 0.283200 (0.000502) | 0.019349 / 0.141683 (-0.122334) | 1.100963 / 1.452155 (-0.351192) | 1.152725 / 1.492716 (-0.339992) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.106350 / 0.018006 (0.088344) | 0.300577 / 0.000490 (0.300087) | 0.000206 / 0.000200 (0.000006) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019028 / 0.037411 (-0.018384) | 0.062643 / 0.014526 (0.048118) | 0.072771 / 0.176557 (-0.103786) | 0.119873 / 0.737135 (-0.617263) | 0.074470 / 0.296338 (-0.221869) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287032 / 0.215209 (0.071823) | 2.826134 / 2.077655 (0.748480) | 1.507362 / 1.504120 (0.003242) | 1.382929 / 1.541195 (-0.158266) | 1.385361 / 1.468490 (-0.083129) | 0.412081 / 4.584777 (-4.172696) | 2.384289 / 3.745712 (-1.361423) | 2.551316 / 5.269862 (-2.718546) | 1.562954 / 4.565676 (-3.002722) | 0.046669 / 0.424275 (-0.377606) | 0.004804 / 0.007607 (-0.002803) | 0.337751 / 0.226044 (0.111707) | 3.378894 / 2.268929 (1.109965) | 1.848817 / 55.444624 (-53.595807) | 1.564560 / 6.876477 (-5.311917) | 1.579577 / 2.142072 (-0.562496) | 0.484531 / 4.805227 (-4.320697) | 0.101157 / 6.500664 (-6.399507) | 0.042272 / 0.075469 (-0.033197) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.948289 / 1.841788 (-0.893498) | 11.490877 / 8.074308 (3.416569) | 10.492787 / 10.191392 (0.301395) | 0.128575 / 0.680424 (-0.551849) | 0.013716 / 0.534201 (-0.520485) | 0.271075 / 0.579283 (-0.308208) | 0.269749 / 0.434364 (-0.164615) | 0.306378 / 0.540337 (-0.233959) | 0.400204 / 1.386936 (-0.986732) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004821 / 0.011353 (-0.006532) | 0.002773 / 0.011008 (-0.008235) | 0.048934 / 0.038508 (0.010426) | 0.049490 / 0.023109 (0.026380) | 0.271107 / 0.275898 (-0.004791) | 0.291472 / 0.323480 (-0.032008) | 0.004734 / 0.007986 (-0.003252) | 0.002437 / 0.004328 (-0.001892) | 0.048840 / 0.004250 (0.044590) | 0.039757 / 0.037052 (0.002704) | 0.276037 / 0.258489 (0.017548) | 0.298220 / 0.293841 (0.004379) | 0.024595 / 0.128546 (-0.103952) | 0.007320 / 0.075646 (-0.068327) | 0.054693 / 0.419271 (-0.364578) | 0.032672 / 0.043533 (-0.010861) | 0.271555 / 0.255139 (0.016416) | 0.287685 / 0.283200 (0.004485) | 0.017159 / 0.141683 (-0.124524) | 1.118496 / 1.452155 (-0.333659) | 1.177389 / 1.492716 (-0.315327) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090469 / 0.018006 (0.072463) | 0.306014 / 0.000490 (0.305525) | 0.000218 / 0.000200 (0.000018) | 0.000044 / 0.000054 (-0.000010) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021452 / 0.037411 (-0.015960) | 0.070014 / 0.014526 (0.055488) | 0.081917 / 0.176557 (-0.094639) | 0.120615 / 0.737135 (-0.616520) | 0.081745 / 0.296338 (-0.214593) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.294049 / 0.215209 (0.078840) | 2.886802 / 2.077655 (0.809147) | 1.607817 / 1.504120 (0.103697) | 1.474172 / 1.541195 (-0.067023) | 1.474744 / 1.468490 (0.006254) | 0.398178 / 4.584777 (-4.186599) | 2.455908 / 3.745712 (-1.289804) | 2.463003 / 5.269862 (-2.806858) | 1.560402 / 4.565676 (-3.005275) | 0.046208 / 0.424275 (-0.378067) | 0.004862 / 0.007607 (-0.002745) | 0.350862 / 0.226044 (0.124817) | 3.463958 / 2.268929 (1.195030) | 1.934696 / 55.444624 (-53.509928) | 1.660090 / 6.876477 (-5.216387) | 1.770920 / 2.142072 (-0.371153) | 0.468409 / 4.805227 (-4.336819) | 0.096812 / 6.500664 (-6.403852) | 0.040580 / 0.075469 (-0.034889) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.978102 / 1.841788 (-0.863686) | 11.943265 / 8.074308 (3.868957) | 10.684995 / 10.191392 (0.493603) | 0.131554 / 0.680424 (-0.548870) | 0.015608 / 0.534201 (-0.518593) | 0.271449 / 0.579283 (-0.307834) | 0.282485 / 0.434364 (-0.151879) | 0.302376 / 0.540337 (-0.237962) | 0.524908 / 1.386936 (-0.862028) |

</details>
</details>

![](https://cml.dev/watermark.png#2bb0b21e37a57257a7d428f8744c862ca92c0c7e "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004926 / 0.011353 (-0.006427) | 0.003020 / 0.011008 (-0.007988) | 0.061899 / 0.038508 (0.023391) | 0.063836 / 0.023109 (0.040726) | 0.239252 / 0.275898 (-0.036646) | 0.268320 / 0.323480 (-0.055160) | 0.003939 / 0.007986 (-0.004046) | 0.002557 / 0.004328 (-0.001772) | 0.048469 / 0.004250 (0.044219) | 0.038707 / 0.037052 (0.001655) | 0.247563 / 0.258489 (-0.010926) | 0.281171 / 0.293841 (-0.012670) | 0.023564 / 0.128546 (-0.104983) | 0.007699 / 0.075646 (-0.067948) | 0.207561 / 0.419271 (-0.211710) | 0.036362 / 0.043533 (-0.007171) | 0.248324 / 0.255139 (-0.006814) | 0.269673 / 0.283200 (-0.013527) | 0.018841 / 0.141683 (-0.122842) | 1.123407 / 1.452155 (-0.328748) | 1.170422 / 1.492716 (-0.322295) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096278 / 0.018006 (0.078272) | 0.311477 / 0.000490 (0.310988) | 0.000217 / 0.000200 (0.000017) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019470 / 0.037411 (-0.017942) | 0.071888 / 0.014526 (0.057362) | 0.074264 / 0.176557 (-0.102292) | 0.124413 / 0.737135 (-0.612723) | 0.075602 / 0.296338 (-0.220737) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284755 / 0.215209 (0.069546) | 2.770789 / 2.077655 (0.693135) | 1.478276 / 1.504120 (-0.025843) | 1.375287 / 1.541195 (-0.165907) | 1.398032 / 1.468490 (-0.070458) | 0.420457 / 4.584777 (-4.164320) | 2.445929 / 3.745712 (-1.299783) | 2.819548 / 5.269862 (-2.450313) | 1.628506 / 4.565676 (-2.937171) | 0.047687 / 0.424275 (-0.376588) | 0.004861 / 0.007607 (-0.002746) | 0.340173 / 0.226044 (0.114129) | 3.340703 / 2.268929 (1.071774) | 1.882803 / 55.444624 (-53.561821) | 1.587206 / 6.876477 (-5.289271) | 1.645298 / 2.142072 (-0.496774) | 0.490957 / 4.805227 (-4.314270) | 0.102779 / 6.500664 (-6.397885) | 0.048372 / 0.075469 (-0.027098) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.958311 / 1.841788 (-0.883477) | 12.354981 / 8.074308 (4.280673) | 10.864826 / 10.191392 (0.673434) | 0.149053 / 0.680424 (-0.531371) | 0.015078 / 0.534201 (-0.519123) | 0.270117 / 0.579283 (-0.309166) | 0.274495 / 0.434364 (-0.159869) | 0.307584 / 0.540337 (-0.232753) | 0.405603 / 1.386936 (-0.981333) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004996 / 0.011353 (-0.006357) | 0.002995 / 0.011008 (-0.008014) | 0.047897 / 0.038508 (0.009389) | 0.056413 / 0.023109 (0.033303) | 0.277669 / 0.275898 (0.001771) | 0.300679 / 0.323480 (-0.022801) | 0.004094 / 0.007986 (-0.003892) | 0.002519 / 0.004328 (-0.001810) | 0.049536 / 0.004250 (0.045285) | 0.042341 / 0.037052 (0.005288) | 0.281533 / 0.258489 (0.023044) | 0.306771 / 0.293841 (0.012930) | 0.025379 / 0.128546 (-0.103167) | 0.007495 / 0.075646 (-0.068152) | 0.054453 / 0.419271 (-0.364818) | 0.032616 / 0.043533 (-0.010917) | 0.277844 / 0.255139 (0.022705) | 0.296265 / 0.283200 (0.013065) | 0.019462 / 0.141683 (-0.122221) | 1.115841 / 1.452155 (-0.336313) | 1.169662 / 1.492716 (-0.323054) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095459 / 0.018006 (0.077453) | 0.301590 / 0.000490 (0.301100) | 0.000230 / 0.000200 (0.000030) | 0.000061 / 0.000054 (0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022182 / 0.037411 (-0.015229) | 0.085367 / 0.014526 (0.070842) | 0.084006 / 0.176557 (-0.092550) | 0.121260 / 0.737135 (-0.615876) | 0.084137 / 0.296338 (-0.212202) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.310335 / 0.215209 (0.095126) | 3.002531 / 2.077655 (0.924876) | 1.642282 / 1.504120 (0.138162) | 1.573044 / 1.541195 (0.031849) | 1.572076 / 1.468490 (0.103586) | 0.422037 / 4.584777 (-4.162740) | 2.495295 / 3.745712 (-1.250417) | 2.523707 / 5.269862 (-2.746155) | 1.725824 / 4.565676 (-2.839853) | 0.047814 / 0.424275 (-0.376461) | 0.004868 / 0.007607 (-0.002739) | 0.352833 / 0.226044 (0.126789) | 3.477241 / 2.268929 (1.208313) | 1.983888 / 55.444624 (-53.460736) | 1.696883 / 6.876477 (-5.179594) | 1.831665 / 2.142072 (-0.310407) | 0.502976 / 4.805227 (-4.302251) | 0.101264 / 6.500664 (-6.399400) | 0.041779 / 0.075469 (-0.033690) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.981629 / 1.841788 (-0.860159) | 12.550634 / 8.074308 (4.476326) | 11.113382 / 10.191392 (0.921990) | 0.136565 / 0.680424 (-0.543859) | 0.016742 / 0.534201 (-0.517459) | 0.274316 / 0.579283 (-0.304967) | 0.284687 / 0.434364 (-0.149676) | 0.309966 / 0.540337 (-0.230372) | 0.557990 / 1.386936 (-0.828946) |

</details>
</details>

![](https://cml.dev/watermark.png#b0c30facb87af83107a645eeffcd18c0775afe11 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004980 / 0.011353 (-0.006373) | 0.002786 / 0.011008 (-0.008222) | 0.062460 / 0.038508 (0.023952) | 0.051811 / 0.023109 (0.028702) | 0.231734 / 0.275898 (-0.044164) | 0.254075 / 0.323480 (-0.069405) | 0.002884 / 0.007986 (-0.005102) | 0.002317 / 0.004328 (-0.002011) | 0.049044 / 0.004250 (0.044793) | 0.038984 / 0.037052 (0.001931) | 0.241193 / 0.258489 (-0.017296) | 0.272091 / 0.293841 (-0.021750) | 0.023098 / 0.128546 (-0.105448) | 0.007190 / 0.075646 (-0.068456) | 0.201409 / 0.419271 (-0.217863) | 0.036100 / 0.043533 (-0.007433) | 0.238185 / 0.255139 (-0.016954) | 0.257127 / 0.283200 (-0.026072) | 0.019542 / 0.141683 (-0.122141) | 1.127925 / 1.452155 (-0.324230) | 1.174354 / 1.492716 (-0.318362) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.099608 / 0.018006 (0.081601) | 0.315046 / 0.000490 (0.314556) | 0.000282 / 0.000200 (0.000082) | 0.000042 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018710 / 0.037411 (-0.018701) | 0.062557 / 0.014526 (0.048031) | 0.074021 / 0.176557 (-0.102536) | 0.119670 / 0.737135 (-0.617465) | 0.076491 / 0.296338 (-0.219847) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282940 / 0.215209 (0.067731) | 2.788542 / 2.077655 (0.710887) | 1.496039 / 1.504120 (-0.008080) | 1.367542 / 1.541195 (-0.173653) | 1.393705 / 1.468490 (-0.074785) | 0.405910 / 4.584777 (-4.178867) | 2.422544 / 3.745712 (-1.323168) | 2.602822 / 5.269862 (-2.667039) | 1.586853 / 4.565676 (-2.978823) | 0.045440 / 0.424275 (-0.378836) | 0.004792 / 0.007607 (-0.002815) | 0.342059 / 0.226044 (0.116015) | 3.366880 / 2.268929 (1.097952) | 1.810566 / 55.444624 (-53.634058) | 1.527112 / 6.876477 (-5.349364) | 1.548906 / 2.142072 (-0.593166) | 0.479491 / 4.805227 (-4.325736) | 0.099807 / 6.500664 (-6.400857) | 0.041951 / 0.075469 (-0.033518) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.953723 / 1.841788 (-0.888065) | 11.837240 / 8.074308 (3.762932) | 10.562979 / 10.191392 (0.371587) | 0.145064 / 0.680424 (-0.535360) | 0.014285 / 0.534201 (-0.519916) | 0.270605 / 0.579283 (-0.308678) | 0.264086 / 0.434364 (-0.170278) | 0.308000 / 0.540337 (-0.232337) | 0.403916 / 1.386936 (-0.983020) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004796 / 0.011353 (-0.006557) | 0.002997 / 0.011008 (-0.008011) | 0.048702 / 0.038508 (0.010193) | 0.053377 / 0.023109 (0.030267) | 0.271852 / 0.275898 (-0.004046) | 0.293366 / 0.323480 (-0.030114) | 0.004041 / 0.007986 (-0.003945) | 0.002459 / 0.004328 (-0.001869) | 0.048197 / 0.004250 (0.043947) | 0.040094 / 0.037052 (0.003042) | 0.275837 / 0.258489 (0.017348) | 0.301174 / 0.293841 (0.007333) | 0.024433 / 0.128546 (-0.104113) | 0.007203 / 0.075646 (-0.068444) | 0.054080 / 0.419271 (-0.365192) | 0.033237 / 0.043533 (-0.010295) | 0.271177 / 0.255139 (0.016038) | 0.293062 / 0.283200 (0.009862) | 0.018399 / 0.141683 (-0.123284) | 1.149527 / 1.452155 (-0.302628) | 1.202717 / 1.492716 (-0.290000) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093168 / 0.018006 (0.075162) | 0.290536 / 0.000490 (0.290046) | 0.000290 / 0.000200 (0.000090) | 0.000074 / 0.000054 (0.000020) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021191 / 0.037411 (-0.016221) | 0.069990 / 0.014526 (0.055465) | 0.080636 / 0.176557 (-0.095920) | 0.120151 / 0.737135 (-0.616984) | 0.082944 / 0.296338 (-0.213395) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289673 / 0.215209 (0.074463) | 2.828419 / 2.077655 (0.750764) | 1.590741 / 1.504120 (0.086621) | 1.480969 / 1.541195 (-0.060226) | 1.512761 / 1.468490 (0.044271) | 0.398328 / 4.584777 (-4.186449) | 2.441134 / 3.745712 (-1.304578) | 2.487606 / 5.269862 (-2.782256) | 1.586604 / 4.565676 (-2.979073) | 0.045578 / 0.424275 (-0.378697) | 0.004842 / 0.007607 (-0.002766) | 0.344556 / 0.226044 (0.118512) | 3.395982 / 2.268929 (1.127053) | 1.963354 / 55.444624 (-53.481271) | 1.680496 / 6.876477 (-5.195980) | 1.827916 / 2.142072 (-0.314157) | 0.476203 / 4.805227 (-4.329024) | 0.098016 / 6.500664 (-6.402648) | 0.041234 / 0.075469 (-0.034235) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977820 / 1.841788 (-0.863968) | 12.139614 / 8.074308 (4.065306) | 10.643071 / 10.191392 (0.451679) | 0.130928 / 0.680424 (-0.549496) | 0.015341 / 0.534201 (-0.518860) | 0.271304 / 0.579283 (-0.307979) | 0.284671 / 0.434364 (-0.149693) | 0.306210 / 0.540337 (-0.234128) | 0.546498 / 1.386936 (-0.840438) |

</details>
</details>

![](https://cml.dev/watermark.png#1bf7408a171db4a744d1760a9e32ba21deb8d41d "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004748 / 0.011353 (-0.006605) | 0.002942 / 0.011008 (-0.008066) | 0.061298 / 0.038508 (0.022790) | 0.052873 / 0.023109 (0.029764) | 0.250573 / 0.275898 (-0.025325) | 0.270636 / 0.323480 (-0.052844) | 0.002925 / 0.007986 (-0.005061) | 0.003126 / 0.004328 (-0.001203) | 0.047340 / 0.004250 (0.043090) | 0.038662 / 0.037052 (0.001609) | 0.252151 / 0.258489 (-0.006338) | 0.284700 / 0.293841 (-0.009141) | 0.025145 / 0.128546 (-0.103402) | 0.007075 / 0.075646 (-0.068572) | 0.200501 / 0.419271 (-0.218771) | 0.035623 / 0.043533 (-0.007910) | 0.249657 / 0.255139 (-0.005482) | 0.272384 / 0.283200 (-0.010815) | 0.018331 / 0.141683 (-0.123351) | 1.095064 / 1.452155 (-0.357091) | 1.145304 / 1.492716 (-0.347412) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092548 / 0.018006 (0.074542) | 0.299338 / 0.000490 (0.298848) | 0.000212 / 0.000200 (0.000012) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018723 / 0.037411 (-0.018688) | 0.062226 / 0.014526 (0.047700) | 0.072840 / 0.176557 (-0.103717) | 0.120073 / 0.737135 (-0.617063) | 0.074536 / 0.296338 (-0.221802) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.284862 / 0.215209 (0.069653) | 2.791842 / 2.077655 (0.714188) | 1.506481 / 1.504120 (0.002361) | 1.368952 / 1.541195 (-0.172243) | 1.372555 / 1.468490 (-0.095935) | 0.408292 / 4.584777 (-4.176485) | 2.381155 / 3.745712 (-1.364558) | 2.613617 / 5.269862 (-2.656244) | 1.575892 / 4.565676 (-2.989785) | 0.047526 / 0.424275 (-0.376749) | 0.004792 / 0.007607 (-0.002815) | 0.344818 / 0.226044 (0.118773) | 3.344965 / 2.268929 (1.076036) | 1.883659 / 55.444624 (-53.560965) | 1.596039 / 6.876477 (-5.280437) | 1.584410 / 2.142072 (-0.557662) | 0.486672 / 4.805227 (-4.318555) | 0.101464 / 6.500664 (-6.399200) | 0.041824 / 0.075469 (-0.033645) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.930491 / 1.841788 (-0.911296) | 11.636526 / 8.074308 (3.562218) | 10.371829 / 10.191392 (0.180437) | 0.138181 / 0.680424 (-0.542243) | 0.014307 / 0.534201 (-0.519894) | 0.268322 / 0.579283 (-0.310961) | 0.264173 / 0.434364 (-0.170191) | 0.303649 / 0.540337 (-0.236688) | 0.399958 / 1.386936 (-0.986978) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004802 / 0.011353 (-0.006551) | 0.002861 / 0.011008 (-0.008147) | 0.048843 / 0.038508 (0.010335) | 0.053887 / 0.023109 (0.030778) | 0.278690 / 0.275898 (0.002792) | 0.302729 / 0.323480 (-0.020751) | 0.003929 / 0.007986 (-0.004057) | 0.002376 / 0.004328 (-0.001953) | 0.048146 / 0.004250 (0.043896) | 0.039842 / 0.037052 (0.002790) | 0.281595 / 0.258489 (0.023106) | 0.305813 / 0.293841 (0.011972) | 0.024214 / 0.128546 (-0.104333) | 0.007201 / 0.075646 (-0.068446) | 0.053604 / 0.419271 (-0.365667) | 0.032841 / 0.043533 (-0.010691) | 0.276168 / 0.255139 (0.021029) | 0.293869 / 0.283200 (0.010669) | 0.017550 / 0.141683 (-0.124132) | 1.121508 / 1.452155 (-0.330647) | 1.177694 / 1.492716 (-0.315022) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091805 / 0.018006 (0.073799) | 0.299026 / 0.000490 (0.298536) | 0.000219 / 0.000200 (0.000019) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021094 / 0.037411 (-0.016318) | 0.069769 / 0.014526 (0.055243) | 0.081191 / 0.176557 (-0.095366) | 0.118884 / 0.737135 (-0.618252) | 0.081955 / 0.296338 (-0.214383) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292159 / 0.215209 (0.076950) | 2.874473 / 2.077655 (0.796819) | 1.614695 / 1.504120 (0.110575) | 1.492123 / 1.541195 (-0.049071) | 1.505293 / 1.468490 (0.036803) | 0.394498 / 4.584777 (-4.190279) | 2.455539 / 3.745712 (-1.290173) | 2.458184 / 5.269862 (-2.811677) | 1.569108 / 4.565676 (-2.996569) | 0.046576 / 0.424275 (-0.377699) | 0.005093 / 0.007607 (-0.002514) | 0.346142 / 0.226044 (0.120098) | 3.398171 / 2.268929 (1.129242) | 1.971953 / 55.444624 (-53.472672) | 1.695275 / 6.876477 (-5.181201) | 1.840511 / 2.142072 (-0.301562) | 0.465932 / 4.805227 (-4.339295) | 0.098578 / 6.500664 (-6.402086) | 0.040456 / 0.075469 (-0.035013) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977636 / 1.841788 (-0.864152) | 12.083585 / 8.074308 (4.009277) | 10.509082 / 10.191392 (0.317690) | 0.130717 / 0.680424 (-0.549707) | 0.015958 / 0.534201 (-0.518243) | 0.273504 / 0.579283 (-0.305780) | 0.276498 / 0.434364 (-0.157866) | 0.306139 / 0.540337 (-0.234199) | 0.522521 / 1.386936 (-0.864415) |

</details>
</details>

![](https://cml.dev/watermark.png#6e17dd8acec9a958ba82a5f753276b842eaadf52 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004859 / 0.011353 (-0.006493) | 0.002423 / 0.011008 (-0.008585) | 0.060969 / 0.038508 (0.022461) | 0.048758 / 0.023109 (0.025649) | 0.245400 / 0.275898 (-0.030498) | 0.263686 / 0.323480 (-0.059794) | 0.002852 / 0.007986 (-0.005134) | 0.002273 / 0.004328 (-0.002055) | 0.047648 / 0.004250 (0.043398) | 0.038310 / 0.037052 (0.001258) | 0.249849 / 0.258489 (-0.008640) | 0.279305 / 0.293841 (-0.014536) | 0.022897 / 0.128546 (-0.105649) | 0.006882 / 0.075646 (-0.068764) | 0.202793 / 0.419271 (-0.216478) | 0.034557 / 0.043533 (-0.008976) | 0.252147 / 0.255139 (-0.002992) | 0.267414 / 0.283200 (-0.015785) | 0.019956 / 0.141683 (-0.121727) | 1.106181 / 1.452155 (-0.345973) | 1.158423 / 1.492716 (-0.334293) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.086848 / 0.018006 (0.068842) | 0.295235 / 0.000490 (0.294745) | 0.000211 / 0.000200 (0.000011) | 0.000041 / 0.000054 (-0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018209 / 0.037411 (-0.019203) | 0.061967 / 0.014526 (0.047441) | 0.071551 / 0.176557 (-0.105005) | 0.117525 / 0.737135 (-0.619611) | 0.073401 / 0.296338 (-0.222937) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.272388 / 0.215209 (0.057179) | 2.689797 / 2.077655 (0.612143) | 1.440897 / 1.504120 (-0.063223) | 1.334689 / 1.541195 (-0.206505) | 1.356395 / 1.468490 (-0.112095) | 0.387201 / 4.584777 (-4.197576) | 2.342908 / 3.745712 (-1.402804) | 2.480156 / 5.269862 (-2.789706) | 1.512342 / 4.565676 (-3.053335) | 0.042324 / 0.424275 (-0.381951) | 0.004744 / 0.007607 (-0.002863) | 0.323568 / 0.226044 (0.097523) | 3.190021 / 2.268929 (0.921093) | 1.765046 / 55.444624 (-53.679578) | 1.513958 / 6.876477 (-5.362519) | 1.504943 / 2.142072 (-0.637129) | 0.452302 / 4.805227 (-4.352925) | 0.094728 / 6.500664 (-6.405936) | 0.038641 / 0.075469 (-0.036828) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.939721 / 1.841788 (-0.902067) | 11.174180 / 8.074308 (3.099872) | 10.046717 / 10.191392 (-0.144675) | 0.124877 / 0.680424 (-0.555547) | 0.013687 / 0.534201 (-0.520514) | 0.261002 / 0.579283 (-0.318282) | 0.267349 / 0.434364 (-0.167015) | 0.306545 / 0.540337 (-0.233792) | 0.389322 / 1.386936 (-0.997614) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004702 / 0.011353 (-0.006651) | 0.002431 / 0.011008 (-0.008577) | 0.046138 / 0.038508 (0.007630) | 0.048356 / 0.023109 (0.025246) | 0.272154 / 0.275898 (-0.003744) | 0.292676 / 0.323480 (-0.030804) | 0.003870 / 0.007986 (-0.004115) | 0.002294 / 0.004328 (-0.002035) | 0.048129 / 0.004250 (0.043879) | 0.039026 / 0.037052 (0.001974) | 0.273900 / 0.258489 (0.015411) | 0.295927 / 0.293841 (0.002086) | 0.024044 / 0.128546 (-0.104502) | 0.006906 / 0.075646 (-0.068740) | 0.053268 / 0.419271 (-0.366004) | 0.032360 / 0.043533 (-0.011173) | 0.273470 / 0.255139 (0.018331) | 0.286207 / 0.283200 (0.003007) | 0.017580 / 0.141683 (-0.124103) | 1.091064 / 1.452155 (-0.361091) | 1.159645 / 1.492716 (-0.333071) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.087149 / 0.018006 (0.069143) | 0.293489 / 0.000490 (0.293000) | 0.000217 / 0.000200 (0.000017) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021779 / 0.037411 (-0.015632) | 0.066453 / 0.014526 (0.051928) | 0.078517 / 0.176557 (-0.098039) | 0.117317 / 0.737135 (-0.619819) | 0.079828 / 0.296338 (-0.216511) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287605 / 0.215209 (0.072396) | 2.811094 / 2.077655 (0.733439) | 1.572474 / 1.504120 (0.068354) | 1.450294 / 1.541195 (-0.090900) | 1.456052 / 1.468490 (-0.012438) | 0.402095 / 4.584777 (-4.182682) | 2.444709 / 3.745712 (-1.301003) | 2.390837 / 5.269862 (-2.879024) | 1.530519 / 4.565676 (-3.035157) | 0.043520 / 0.424275 (-0.380755) | 0.004788 / 0.007607 (-0.002819) | 0.337436 / 0.226044 (0.111391) | 3.326111 / 2.268929 (1.057182) | 1.889273 / 55.444624 (-53.555352) | 1.624423 / 6.876477 (-5.252054) | 1.715766 / 2.142072 (-0.426307) | 0.484570 / 4.805227 (-4.320657) | 0.091691 / 6.500664 (-6.408973) | 0.038278 / 0.075469 (-0.037191) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.961708 / 1.841788 (-0.880079) | 11.496471 / 8.074308 (3.422162) | 10.211589 / 10.191392 (0.020197) | 0.127584 / 0.680424 (-0.552840) | 0.015178 / 0.534201 (-0.519023) | 0.267290 / 0.579283 (-0.311993) | 0.259305 / 0.434364 (-0.175059) | 0.303433 / 0.540337 (-0.236905) | 0.508016 / 1.386936 (-0.878920) |

</details>
</details>

![](https://cml.dev/watermark.png#72880aa8a3e4b49438db72b13fb9a2541331820b "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004558 / 0.011353 (-0.006795) | 0.002563 / 0.011008 (-0.008445) | 0.061314 / 0.038508 (0.022806) | 0.049312 / 0.023109 (0.026203) | 0.240988 / 0.275898 (-0.034910) | 0.260548 / 0.323480 (-0.062932) | 0.002817 / 0.007986 (-0.005169) | 0.002904 / 0.004328 (-0.001425) | 0.048515 / 0.004250 (0.044264) | 0.037511 / 0.037052 (0.000459) | 0.244880 / 0.258489 (-0.013609) | 0.276118 / 0.293841 (-0.017723) | 0.022636 / 0.128546 (-0.105910) | 0.006694 / 0.075646 (-0.068953) | 0.201336 / 0.419271 (-0.217936) | 0.035228 / 0.043533 (-0.008305) | 0.242424 / 0.255139 (-0.012715) | 0.260178 / 0.283200 (-0.023022) | 0.017836 / 0.141683 (-0.123847) | 1.122296 / 1.452155 (-0.329859) | 1.189024 / 1.492716 (-0.303692) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090051 / 0.018006 (0.072045) | 0.298562 / 0.000490 (0.298073) | 0.000216 / 0.000200 (0.000016) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018228 / 0.037411 (-0.019184) | 0.062379 / 0.014526 (0.047853) | 0.073482 / 0.176557 (-0.103075) | 0.120341 / 0.737135 (-0.616794) | 0.073868 / 0.296338 (-0.222470) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.280195 / 0.215209 (0.064986) | 2.743333 / 2.077655 (0.665678) | 1.470078 / 1.504120 (-0.034042) | 1.335874 / 1.541195 (-0.205321) | 1.342961 / 1.468490 (-0.125529) | 0.409203 / 4.584777 (-4.175574) | 2.392217 / 3.745712 (-1.353495) | 2.544161 / 5.269862 (-2.725701) | 1.544016 / 4.565676 (-3.021660) | 0.059485 / 0.424275 (-0.364790) | 0.004833 / 0.007607 (-0.002775) | 0.335114 / 0.226044 (0.109070) | 3.289009 / 2.268929 (1.020080) | 1.854666 / 55.444624 (-53.589959) | 1.566282 / 6.876477 (-5.310195) | 1.561287 / 2.142072 (-0.580786) | 0.484961 / 4.805227 (-4.320267) | 0.099651 / 6.500664 (-6.401013) | 0.041408 / 0.075469 (-0.034061) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.941743 / 1.841788 (-0.900044) | 11.165692 / 8.074308 (3.091383) | 10.236693 / 10.191392 (0.045301) | 0.129694 / 0.680424 (-0.550730) | 0.014879 / 0.534201 (-0.519322) | 0.275120 / 0.579283 (-0.304163) | 0.263822 / 0.434364 (-0.170542) | 0.306429 / 0.540337 (-0.233909) | 0.397611 / 1.386936 (-0.989325) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004714 / 0.011353 (-0.006639) | 0.002430 / 0.011008 (-0.008578) | 0.047644 / 0.038508 (0.009136) | 0.049710 / 0.023109 (0.026601) | 0.271950 / 0.275898 (-0.003948) | 0.290996 / 0.323480 (-0.032483) | 0.003888 / 0.007986 (-0.004097) | 0.002367 / 0.004328 (-0.001962) | 0.047623 / 0.004250 (0.043372) | 0.039574 / 0.037052 (0.002522) | 0.274540 / 0.258489 (0.016051) | 0.298065 / 0.293841 (0.004224) | 0.024677 / 0.128546 (-0.103869) | 0.006844 / 0.075646 (-0.068802) | 0.053180 / 0.419271 (-0.366091) | 0.032391 / 0.043533 (-0.011141) | 0.273222 / 0.255139 (0.018083) | 0.290336 / 0.283200 (0.007136) | 0.017911 / 0.141683 (-0.123772) | 1.105879 / 1.452155 (-0.346276) | 1.176979 / 1.492716 (-0.315737) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089563 / 0.018006 (0.071557) | 0.296392 / 0.000490 (0.295903) | 0.000214 / 0.000200 (0.000014) | 0.000053 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021588 / 0.037411 (-0.015824) | 0.069951 / 0.014526 (0.055425) | 0.080397 / 0.176557 (-0.096160) | 0.118772 / 0.737135 (-0.618363) | 0.080356 / 0.296338 (-0.215983) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288492 / 0.215209 (0.073283) | 2.839553 / 2.077655 (0.761898) | 1.597504 / 1.504120 (0.093384) | 1.475001 / 1.541195 (-0.066193) | 1.481561 / 1.468490 (0.013071) | 0.411851 / 4.584777 (-4.172926) | 2.397322 / 3.745712 (-1.348390) | 2.444078 / 5.269862 (-2.825784) | 1.557106 / 4.565676 (-3.008571) | 0.047159 / 0.424275 (-0.377116) | 0.004842 / 0.007607 (-0.002765) | 0.346221 / 0.226044 (0.120177) | 3.387900 / 2.268929 (1.118972) | 1.962167 / 55.444624 (-53.482457) | 1.675017 / 6.876477 (-5.201460) | 1.788745 / 2.142072 (-0.353328) | 0.488063 / 4.805227 (-4.317164) | 0.098878 / 6.500664 (-6.401786) | 0.040369 / 0.075469 (-0.035100) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.977999 / 1.841788 (-0.863789) | 11.671558 / 8.074308 (3.597250) | 10.327847 / 10.191392 (0.136455) | 0.129317 / 0.680424 (-0.551107) | 0.015600 / 0.534201 (-0.518601) | 0.267967 / 0.579283 (-0.311316) | 0.273811 / 0.434364 (-0.160553) | 0.301749 / 0.540337 (-0.238588) | 0.515493 / 1.386936 (-0.871443) |

</details>
</details>

![](https://cml.dev/watermark.png#5394939b0b3d124674f938e1f1cd9e8de3cbdbf7 "CML watermark")


I added tests and docs @mariosasko @albertvillanova let le know what you think !

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004867 / 0.011353 (-0.006486) | 0.002952 / 0.011008 (-0.008056) | 0.062008 / 0.038508 (0.023500) | 0.055279 / 0.023109 (0.032170) | 0.248160 / 0.275898 (-0.027738) | 0.276173 / 0.323480 (-0.047307) | 0.003945 / 0.007986 (-0.004041) | 0.002371 / 0.004328 (-0.001958) | 0.048385 / 0.004250 (0.044134) | 0.038997 / 0.037052 (0.001945) | 0.257465 / 0.258489 (-0.001024) | 0.286920 / 0.293841 (-0.006921) | 0.023031 / 0.128546 (-0.105515) | 0.007075 / 0.075646 (-0.068571) | 0.201897 / 0.419271 (-0.217375) | 0.035637 / 0.043533 (-0.007896) | 0.252050 / 0.255139 (-0.003089) | 0.272580 / 0.283200 (-0.010620) | 0.018578 / 0.141683 (-0.123105) | 1.129427 / 1.452155 (-0.322727) | 1.172182 / 1.492716 (-0.320534) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091806 / 0.018006 (0.073800) | 0.298632 / 0.000490 (0.298143) | 0.000202 / 0.000200 (0.000002) | 0.000047 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019123 / 0.037411 (-0.018288) | 0.062603 / 0.014526 (0.048077) | 0.074352 / 0.176557 (-0.102205) | 0.120431 / 0.737135 (-0.616704) | 0.074622 / 0.296338 (-0.221717) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276019 / 0.215209 (0.060810) | 2.701610 / 2.077655 (0.623955) | 1.398388 / 1.504120 (-0.105732) | 1.270902 / 1.541195 (-0.270292) | 1.307992 / 1.468490 (-0.160499) | 0.396350 / 4.584777 (-4.188427) | 2.351064 / 3.745712 (-1.394648) | 2.606229 / 5.269862 (-2.663632) | 1.591075 / 4.565676 (-2.974601) | 0.046429 / 0.424275 (-0.377846) | 0.004832 / 0.007607 (-0.002775) | 0.327887 / 0.226044 (0.101843) | 3.277847 / 2.268929 (1.008918) | 1.767210 / 55.444624 (-53.677414) | 1.483997 / 6.876477 (-5.392479) | 1.515689 / 2.142072 (-0.626383) | 0.471326 / 4.805227 (-4.333902) | 0.098821 / 6.500664 (-6.401843) | 0.041914 / 0.075469 (-0.033555) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.956278 / 1.841788 (-0.885510) | 11.924373 / 8.074308 (3.850065) | 10.493926 / 10.191392 (0.302534) | 0.140214 / 0.680424 (-0.540210) | 0.013679 / 0.534201 (-0.520522) | 0.270304 / 0.579283 (-0.308979) | 0.266518 / 0.434364 (-0.167846) | 0.310113 / 0.540337 (-0.230224) | 0.399811 / 1.386936 (-0.987125) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004793 / 0.011353 (-0.006560) | 0.002879 / 0.011008 (-0.008130) | 0.048632 / 0.038508 (0.010124) | 0.051413 / 0.023109 (0.028304) | 0.272704 / 0.275898 (-0.003194) | 0.291541 / 0.323480 (-0.031939) | 0.003913 / 0.007986 (-0.004072) | 0.002387 / 0.004328 (-0.001941) | 0.049045 / 0.004250 (0.044795) | 0.040164 / 0.037052 (0.003112) | 0.273052 / 0.258489 (0.014563) | 0.300139 / 0.293841 (0.006298) | 0.024225 / 0.128546 (-0.104321) | 0.007060 / 0.075646 (-0.068587) | 0.054360 / 0.419271 (-0.364911) | 0.032882 / 0.043533 (-0.010650) | 0.270295 / 0.255139 (0.015157) | 0.312253 / 0.283200 (0.029054) | 0.017413 / 0.141683 (-0.124270) | 1.137306 / 1.452155 (-0.314849) | 1.203705 / 1.492716 (-0.289011) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091083 / 0.018006 (0.073077) | 0.301607 / 0.000490 (0.301117) | 0.000219 / 0.000200 (0.000019) | 0.000052 / 0.000054 (-0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021753 / 0.037411 (-0.015658) | 0.069693 / 0.014526 (0.055167) | 0.080481 / 0.176557 (-0.096075) | 0.118581 / 0.737135 (-0.618555) | 0.082231 / 0.296338 (-0.214108) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.300014 / 0.215209 (0.084805) | 2.885934 / 2.077655 (0.808279) | 1.594120 / 1.504120 (0.090000) | 1.472312 / 1.541195 (-0.068883) | 1.491663 / 1.468490 (0.023173) | 0.412946 / 4.584777 (-4.171831) | 2.494168 / 3.745712 (-1.251544) | 2.527987 / 5.269862 (-2.741875) | 1.589187 / 4.565676 (-2.976490) | 0.046594 / 0.424275 (-0.377681) | 0.004810 / 0.007607 (-0.002797) | 0.345496 / 0.226044 (0.119452) | 3.428850 / 2.268929 (1.159921) | 1.952696 / 55.444624 (-53.491929) | 1.663285 / 6.876477 (-5.213191) | 1.822187 / 2.142072 (-0.319885) | 0.483798 / 4.805227 (-4.321430) | 0.101403 / 6.500664 (-6.399261) | 0.041773 / 0.075469 (-0.033696) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.974247 / 1.841788 (-0.867541) | 12.459980 / 8.074308 (4.385672) | 10.354792 / 10.191392 (0.163400) | 0.129083 / 0.680424 (-0.551341) | 0.015225 / 0.534201 (-0.518976) | 0.267673 / 0.579283 (-0.311610) | 0.281011 / 0.434364 (-0.153352) | 0.303054 / 0.540337 (-0.237283) | 0.405719 / 1.386936 (-0.981217) |

</details>
</details>

![](https://cml.dev/watermark.png#33dc51fc1a8122b842bb7839ff0eda32f173c325 "CML watermark")


I switched to using `deepmind/code_contests` in examples in the docs to avoid having to pass trust_remote_code, and remove the DEFAULT naming stuff :)

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005169 / 0.011353 (-0.006184) | 0.003066 / 0.011008 (-0.007942) | 0.068884 / 0.038508 (0.030376) | 0.060345 / 0.023109 (0.037236) | 0.243050 / 0.275898 (-0.032848) | 0.265523 / 0.323480 (-0.057957) | 0.002918 / 0.007986 (-0.005067) | 0.002495 / 0.004328 (-0.001834) | 0.051538 / 0.004250 (0.047288) | 0.040010 / 0.037052 (0.002957) | 0.249603 / 0.258489 (-0.008886) | 0.287955 / 0.293841 (-0.005886) | 0.024003 / 0.128546 (-0.104543) | 0.007111 / 0.075646 (-0.068535) | 0.205041 / 0.419271 (-0.214231) | 0.036296 / 0.043533 (-0.007237) | 0.246135 / 0.255139 (-0.009004) | 0.268801 / 0.283200 (-0.014399) | 0.018451 / 0.141683 (-0.123232) | 1.130387 / 1.452155 (-0.321767) | 1.162041 / 1.492716 (-0.330675) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.096370 / 0.018006 (0.078364) | 0.309867 / 0.000490 (0.309377) | 0.000229 / 0.000200 (0.000029) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018688 / 0.037411 (-0.018723) | 0.062859 / 0.014526 (0.048333) | 0.076383 / 0.176557 (-0.100173) | 0.120385 / 0.737135 (-0.616750) | 0.080192 / 0.296338 (-0.216147) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.282994 / 0.215209 (0.067785) | 2.742341 / 2.077655 (0.664686) | 1.432041 / 1.504120 (-0.072079) | 1.303282 / 1.541195 (-0.237913) | 1.347198 / 1.468490 (-0.121292) | 0.399145 / 4.584777 (-4.185632) | 2.359766 / 3.745712 (-1.385947) | 2.753577 / 5.269862 (-2.516285) | 1.639953 / 4.565676 (-2.925724) | 0.047111 / 0.424275 (-0.377164) | 0.004946 / 0.007607 (-0.002661) | 0.338857 / 0.226044 (0.112813) | 3.328709 / 2.268929 (1.059781) | 1.794729 / 55.444624 (-53.649895) | 1.508514 / 6.876477 (-5.367963) | 1.550737 / 2.142072 (-0.591335) | 0.484227 / 4.805227 (-4.321000) | 0.101001 / 6.500664 (-6.399663) | 0.042792 / 0.075469 (-0.032677) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.956471 / 1.841788 (-0.885317) | 12.031362 / 8.074308 (3.957054) | 10.512914 / 10.191392 (0.321522) | 0.141841 / 0.680424 (-0.538583) | 0.014343 / 0.534201 (-0.519858) | 0.273916 / 0.579283 (-0.305367) | 0.266150 / 0.434364 (-0.168214) | 0.312020 / 0.540337 (-0.228317) | 0.410465 / 1.386936 (-0.976471) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004945 / 0.011353 (-0.006408) | 0.003288 / 0.011008 (-0.007720) | 0.048247 / 0.038508 (0.009739) | 0.057892 / 0.023109 (0.034783) | 0.269741 / 0.275898 (-0.006157) | 0.293728 / 0.323480 (-0.029752) | 0.004789 / 0.007986 (-0.003197) | 0.002477 / 0.004328 (-0.001852) | 0.047825 / 0.004250 (0.043575) | 0.040780 / 0.037052 (0.003727) | 0.273355 / 0.258489 (0.014865) | 0.300057 / 0.293841 (0.006216) | 0.024481 / 0.128546 (-0.104066) | 0.007285 / 0.075646 (-0.068361) | 0.053046 / 0.419271 (-0.366226) | 0.032342 / 0.043533 (-0.011190) | 0.272293 / 0.255139 (0.017154) | 0.290842 / 0.283200 (0.007642) | 0.017546 / 0.141683 (-0.124137) | 1.155816 / 1.452155 (-0.296339) | 1.195839 / 1.492716 (-0.296878) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.094177 / 0.018006 (0.076170) | 0.305122 / 0.000490 (0.304632) | 0.000237 / 0.000200 (0.000037) | 0.000063 / 0.000054 (0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021817 / 0.037411 (-0.015595) | 0.070711 / 0.014526 (0.056185) | 0.084028 / 0.176557 (-0.092528) | 0.120160 / 0.737135 (-0.616975) | 0.083085 / 0.296338 (-0.213254) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289127 / 0.215209 (0.073918) | 2.826365 / 2.077655 (0.748710) | 1.582910 / 1.504120 (0.078790) | 1.472796 / 1.541195 (-0.068399) | 1.497491 / 1.468490 (0.029000) | 0.412276 / 4.584777 (-4.172501) | 2.430692 / 3.745712 (-1.315020) | 2.556444 / 5.269862 (-2.713418) | 1.625782 / 4.565676 (-2.939895) | 0.047921 / 0.424275 (-0.376354) | 0.004809 / 0.007607 (-0.002798) | 0.345569 / 0.226044 (0.119524) | 3.417785 / 2.268929 (1.148856) | 1.959223 / 55.444624 (-53.485401) | 1.672765 / 6.876477 (-5.203712) | 1.852444 / 2.142072 (-0.289628) | 0.489225 / 4.805227 (-4.316002) | 0.100624 / 6.500664 (-6.400040) | 0.041242 / 0.075469 (-0.034227) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.971130 / 1.841788 (-0.870658) | 12.652204 / 8.074308 (4.577896) | 10.661821 / 10.191392 (0.470429) | 0.147636 / 0.680424 (-0.532787) | 0.015738 / 0.534201 (-0.518463) | 0.272763 / 0.579283 (-0.306520) | 0.282623 / 0.434364 (-0.151741) | 0.341303 / 0.540337 (-0.199035) | 0.412149 / 1.386936 (-0.974787) |

</details>
</details>

![](https://cml.dev/watermark.png#9499908c97ceef1792f69b71e93e36602880a4ae "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004589 / 0.011353 (-0.006764) | 0.002730 / 0.011008 (-0.008279) | 0.061862 / 0.038508 (0.023353) | 0.050945 / 0.023109 (0.027836) | 0.240776 / 0.275898 (-0.035122) | 0.266000 / 0.323480 (-0.057480) | 0.003823 / 0.007986 (-0.004162) | 0.002345 / 0.004328 (-0.001983) | 0.047821 / 0.004250 (0.043571) | 0.037813 / 0.037052 (0.000761) | 0.251075 / 0.258489 (-0.007415) | 0.279430 / 0.293841 (-0.014411) | 0.022957 / 0.128546 (-0.105590) | 0.007294 / 0.075646 (-0.068353) | 0.206092 / 0.419271 (-0.213180) | 0.035308 / 0.043533 (-0.008225) | 0.247197 / 0.255139 (-0.007942) | 0.264988 / 0.283200 (-0.018212) | 0.017588 / 0.141683 (-0.124095) | 1.093291 / 1.452155 (-0.358864) | 1.165477 / 1.492716 (-0.327240) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.104057 / 0.018006 (0.086051) | 0.303424 / 0.000490 (0.302934) | 0.000223 / 0.000200 (0.000023) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019040 / 0.037411 (-0.018371) | 0.063161 / 0.014526 (0.048635) | 0.085333 / 0.176557 (-0.091224) | 0.155973 / 0.737135 (-0.581162) | 0.077528 / 0.296338 (-0.218810) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276104 / 0.215209 (0.060895) | 2.738174 / 2.077655 (0.660519) | 1.479484 / 1.504120 (-0.024636) | 1.354094 / 1.541195 (-0.187100) | 1.385312 / 1.468490 (-0.083178) | 0.401398 / 4.584777 (-4.183379) | 2.368503 / 3.745712 (-1.377209) | 2.586405 / 5.269862 (-2.683457) | 1.573978 / 4.565676 (-2.991699) | 0.046969 / 0.424275 (-0.377306) | 0.004874 / 0.007607 (-0.002733) | 0.334028 / 0.226044 (0.107984) | 3.269645 / 2.268929 (1.000717) | 1.834528 / 55.444624 (-53.610096) | 1.559883 / 6.876477 (-5.316594) | 1.581380 / 2.142072 (-0.560693) | 0.479580 / 4.805227 (-4.325647) | 0.099077 / 6.500664 (-6.401587) | 0.041166 / 0.075469 (-0.034303) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.918810 / 1.841788 (-0.922978) | 11.505017 / 8.074308 (3.430709) | 10.331934 / 10.191392 (0.140542) | 0.128079 / 0.680424 (-0.552345) | 0.013716 / 0.534201 (-0.520485) | 0.271567 / 0.579283 (-0.307716) | 0.264846 / 0.434364 (-0.169518) | 0.305245 / 0.540337 (-0.235092) | 0.401391 / 1.386936 (-0.985546) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.004860 / 0.011353 (-0.006493) | 0.002854 / 0.011008 (-0.008155) | 0.048327 / 0.038508 (0.009819) | 0.051377 / 0.023109 (0.028268) | 0.264344 / 0.275898 (-0.011554) | 0.286800 / 0.323480 (-0.036680) | 0.003969 / 0.007986 (-0.004016) | 0.002415 / 0.004328 (-0.001914) | 0.048498 / 0.004250 (0.044247) | 0.040399 / 0.037052 (0.003347) | 0.267254 / 0.258489 (0.008765) | 0.292049 / 0.293841 (-0.001792) | 0.024730 / 0.128546 (-0.103817) | 0.007275 / 0.075646 (-0.068371) | 0.053725 / 0.419271 (-0.365546) | 0.033142 / 0.043533 (-0.010391) | 0.265418 / 0.255139 (0.010279) | 0.286242 / 0.283200 (0.003042) | 0.017824 / 0.141683 (-0.123859) | 1.135978 / 1.452155 (-0.316176) | 1.192506 / 1.492716 (-0.300210) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091907 / 0.018006 (0.073900) | 0.307152 / 0.000490 (0.306663) | 0.000223 / 0.000200 (0.000023) | 0.000046 / 0.000054 (-0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021909 / 0.037411 (-0.015502) | 0.070676 / 0.014526 (0.056150) | 0.081651 / 0.176557 (-0.094906) | 0.120915 / 0.737135 (-0.616220) | 0.085882 / 0.296338 (-0.210456) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.288008 / 0.215209 (0.072799) | 2.861352 / 2.077655 (0.783697) | 1.539045 / 1.504120 (0.034925) | 1.412175 / 1.541195 (-0.129019) | 1.421236 / 1.468490 (-0.047254) | 0.404921 / 4.584777 (-4.179856) | 2.480211 / 3.745712 (-1.265501) | 2.473083 / 5.269862 (-2.796779) | 1.558894 / 4.565676 (-3.006783) | 0.046692 / 0.424275 (-0.377584) | 0.004802 / 0.007607 (-0.002805) | 0.346046 / 0.226044 (0.120001) | 3.464387 / 2.268929 (1.195459) | 1.937298 / 55.444624 (-53.507326) | 1.593701 / 6.876477 (-5.282776) | 1.730688 / 2.142072 (-0.411385) | 0.481069 / 4.805227 (-4.324158) | 0.098991 / 6.500664 (-6.401673) | 0.040491 / 0.075469 (-0.034978) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.967809 / 1.841788 (-0.873979) | 11.952335 / 8.074308 (3.878027) | 10.616711 / 10.191392 (0.425319) | 0.128938 / 0.680424 (-0.551486) | 0.015455 / 0.534201 (-0.518746) | 0.272100 / 0.579283 (-0.307183) | 0.278275 / 0.434364 (-0.156089) | 0.309711 / 0.540337 (-0.230627) | 0.411026 / 1.386936 (-0.975910) |

</details>
</details>

![](https://cml.dev/watermark.png#495bc04226a67983f523d12d42b680172f8d4893 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008470 / 0.011353 (-0.002883) | 0.003201 / 0.011008 (-0.007808) | 0.063193 / 0.038508 (0.024685) | 0.064174 / 0.023109 (0.041064) | 0.248316 / 0.275898 (-0.027582) | 0.281598 / 0.323480 (-0.041882) | 0.004076 / 0.007986 (-0.003909) | 0.002397 / 0.004328 (-0.001932) | 0.048834 / 0.004250 (0.044584) | 0.056517 / 0.037052 (0.019465) | 0.254164 / 0.258489 (-0.004326) | 0.289800 / 0.293841 (-0.004041) | 0.031092 / 0.128546 (-0.097454) | 0.010885 / 0.075646 (-0.064762) | 0.219198 / 0.419271 (-0.200073) | 0.040087 / 0.043533 (-0.003446) | 0.250900 / 0.255139 (-0.004239) | 0.267787 / 0.283200 (-0.015413) | 0.019666 / 0.141683 (-0.122017) | 1.114960 / 1.452155 (-0.337194) | 1.266675 / 1.492716 (-0.226041) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091429 / 0.018006 (0.073422) | 0.301804 / 0.000490 (0.301314) | 0.000212 / 0.000200 (0.000012) | 0.000064 / 0.000054 (0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021053 / 0.037411 (-0.016358) | 0.062407 / 0.014526 (0.047881) | 0.073166 / 0.176557 (-0.103391) | 0.119642 / 0.737135 (-0.617493) | 0.074771 / 0.296338 (-0.221567) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.278582 / 0.215209 (0.063373) | 2.773023 / 2.077655 (0.695368) | 1.459977 / 1.504120 (-0.044143) | 1.330453 / 1.541195 (-0.210742) | 1.372797 / 1.468490 (-0.095693) | 0.628845 / 4.584777 (-3.955932) | 3.428779 / 3.745712 (-0.316933) | 3.138967 / 5.269862 (-2.130895) | 2.126891 / 4.565676 (-2.438785) | 0.062340 / 0.424275 (-0.361935) | 0.004939 / 0.007607 (-0.002668) | 0.336058 / 0.226044 (0.110014) | 3.463741 / 2.268929 (1.194813) | 1.847504 / 55.444624 (-53.597120) | 1.984173 / 6.876477 (-4.892304) | 1.602962 / 2.142072 (-0.539110) | 0.637683 / 4.805227 (-4.167545) | 0.117898 / 6.500664 (-6.382766) | 0.043308 / 0.075469 (-0.032161) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.087773 / 1.841788 (-0.754014) | 14.959526 / 8.074308 (6.885218) | 10.886003 / 10.191392 (0.694611) | 0.163385 / 0.680424 (-0.517039) | 0.016679 / 0.534201 (-0.517522) | 0.351913 / 0.579283 (-0.227370) | 0.359007 / 0.434364 (-0.075357) | 0.323824 / 0.540337 (-0.216513) | 0.549268 / 1.386936 (-0.837668) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005265 / 0.011353 (-0.006088) | 0.003367 / 0.011008 (-0.007641) | 0.062741 / 0.038508 (0.024233) | 0.068463 / 0.023109 (0.045354) | 0.258497 / 0.275898 (-0.017401) | 0.355360 / 0.323480 (0.031880) | 0.003910 / 0.007986 (-0.004075) | 0.002399 / 0.004328 (-0.001929) | 0.055564 / 0.004250 (0.051313) | 0.039644 / 0.037052 (0.002591) | 0.258313 / 0.258489 (-0.000176) | 0.328927 / 0.293841 (0.035086) | 0.035634 / 0.128546 (-0.092912) | 0.010378 / 0.075646 (-0.065268) | 0.073109 / 0.419271 (-0.346163) | 0.039752 / 0.043533 (-0.003781) | 0.258237 / 0.255139 (0.003098) | 0.330329 / 0.283200 (0.047129) | 0.023924 / 0.141683 (-0.117759) | 1.198639 / 1.452155 (-0.253515) | 1.202307 / 1.492716 (-0.290409) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.091297 / 0.018006 (0.073290) | 0.298729 / 0.000490 (0.298240) | 0.000210 / 0.000200 (0.000010) | 0.000049 / 0.000054 (-0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022381 / 0.037411 (-0.015030) | 0.070226 / 0.014526 (0.055700) | 0.080549 / 0.176557 (-0.096007) | 0.119677 / 0.737135 (-0.617458) | 0.082612 / 0.296338 (-0.213727) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.289270 / 0.215209 (0.074061) | 2.853830 / 2.077655 (0.776175) | 1.528938 / 1.504120 (0.024818) | 1.398429 / 1.541195 (-0.142766) | 1.472465 / 1.468490 (0.003975) | 0.779015 / 4.584777 (-3.805762) | 3.287724 / 3.745712 (-0.457988) | 3.020908 / 5.269862 (-2.248953) | 1.926094 / 4.565676 (-2.639583) | 0.063163 / 0.424275 (-0.361112) | 0.005175 / 0.007607 (-0.002432) | 0.342884 / 0.226044 (0.116840) | 3.476837 / 2.268929 (1.207908) | 1.880683 / 55.444624 (-53.563942) | 1.613845 / 6.876477 (-5.262632) | 1.624734 / 2.142072 (-0.517338) | 0.626220 / 4.805227 (-4.179007) | 0.114976 / 6.500664 (-6.385689) | 0.040670 / 0.075469 (-0.034799) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.116815 / 1.841788 (-0.724973) | 15.388426 / 8.074308 (7.314118) | 10.825276 / 10.191392 (0.633884) | 0.172659 / 0.680424 (-0.507765) | 0.015468 / 0.534201 (-0.518733) | 0.285552 / 0.579283 (-0.293731) | 0.346886 / 0.434364 (-0.087478) | 0.348696 / 0.540337 (-0.191641) | 0.729335 / 1.386936 (-0.657601) |

</details>
</details>

![](https://cml.dev/watermark.png#d7bbf346dc268b8084dee406b2a6e2b96d44bc3b "CML watermark")