Issue: Always set nullable fields in the writer
State: closed
Author: lhoestq

This fixes loading of e.g. parquet data with non-nullable fields.

Indeed `datasets.Features` doesn't support non-nullable fields, which can lead to data not concatenable due to arrow schema mismatch.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006640 / 0.011353 (-0.004713) | 0.004606 / 0.011008 (-0.006402) | 0.098870 / 0.038508 (0.060362) | 0.028201 / 0.023109 (0.005092) | 0.304396 / 0.275898 (0.028498) | 0.339804 / 0.323480 (0.016324) | 0.005011 / 0.007986 (-0.002974) | 0.003530 / 0.004328 (-0.000799) | 0.075223 / 0.004250 (0.070973) | 0.037922 / 0.037052 (0.000870) | 0.310273 / 0.258489 (0.051784) | 0.348324 / 0.293841 (0.054483) | 0.030181 / 0.128546 (-0.098365) | 0.011584 / 0.075646 (-0.064062) | 0.322637 / 0.419271 (-0.096635) | 0.043119 / 0.043533 (-0.000414) | 0.314514 / 0.255139 (0.059375) | 0.334384 / 0.283200 (0.051185) | 0.092551 / 0.141683 (-0.049132) | 1.496694 / 1.452155 (0.044539) | 1.555426 / 1.492716 (0.062710) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.205078 / 0.018006 (0.187072) | 0.399200 / 0.000490 (0.398710) | 0.004881 / 0.000200 (0.004681) | 0.000200 / 0.000054 (0.000146) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025042 / 0.037411 (-0.012369) | 0.101501 / 0.014526 (0.086975) | 0.107430 / 0.176557 (-0.069127) | 0.170107 / 0.737135 (-0.567028) | 0.111253 / 0.296338 (-0.185086) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.460358 / 0.215209 (0.245149) | 4.592037 / 2.077655 (2.514383) | 2.222612 / 1.504120 (0.718493) | 2.022804 / 1.541195 (0.481610) | 2.040824 / 1.468490 (0.572334) | 0.700485 / 4.584777 (-3.884292) | 3.427847 / 3.745712 (-0.317866) | 2.836916 / 5.269862 (-2.432946) | 1.505055 / 4.565676 (-3.060621) | 0.083206 / 0.424275 (-0.341069) | 0.046492 / 0.007607 (0.038885) | 0.555562 / 0.226044 (0.329518) | 5.563574 / 2.268929 (3.294645) | 2.635273 / 55.444624 (-52.809351) | 2.299377 / 6.876477 (-4.577100) | 2.394512 / 2.142072 (0.252440) | 0.809541 / 4.805227 (-3.995686) | 0.151814 / 6.500664 (-6.348850) | 0.067241 / 0.075469 (-0.008228) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.188396 / 1.841788 (-0.653392) | 13.714596 / 8.074308 (5.640288) | 14.076906 / 10.191392 (3.885514) | 0.143447 / 0.680424 (-0.536977) | 0.016514 / 0.534201 (-0.517687) | 0.383075 / 0.579283 (-0.196209) | 0.386997 / 0.434364 (-0.047367) | 0.441941 / 0.540337 (-0.098396) | 0.522145 / 1.386936 (-0.864791) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006266 / 0.011353 (-0.005086) | 0.004562 / 0.011008 (-0.006446) | 0.077472 / 0.038508 (0.038964) | 0.027596 / 0.023109 (0.004486) | 0.400498 / 0.275898 (0.124600) | 0.406728 / 0.323480 (0.083248) | 0.004745 / 0.007986 (-0.003241) | 0.003375 / 0.004328 (-0.000954) | 0.076645 / 0.004250 (0.072394) | 0.037756 / 0.037052 (0.000703) | 0.415183 / 0.258489 (0.156694) | 0.413758 / 0.293841 (0.119917) | 0.030624 / 0.128546 (-0.097922) | 0.011525 / 0.075646 (-0.064121) | 0.086033 / 0.419271 (-0.333238) | 0.039307 / 0.043533 (-0.004226) | 0.418192 / 0.255139 (0.163053) | 0.403152 / 0.283200 (0.119952) | 0.094141 / 0.141683 (-0.047542) | 1.459012 / 1.452155 (0.006857) | 1.546493 / 1.492716 (0.053777) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239494 / 0.018006 (0.221488) | 0.420918 / 0.000490 (0.420428) | 0.000411 / 0.000200 (0.000211) | 0.000057 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024525 / 0.037411 (-0.012886) | 0.099793 / 0.014526 (0.085267) | 0.105888 / 0.176557 (-0.070669) | 0.155912 / 0.737135 (-0.581223) | 0.109937 / 0.296338 (-0.186401) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470108 / 0.215209 (0.254899) | 4.696390 / 2.077655 (2.618735) | 2.467841 / 1.504120 (0.963721) | 2.275012 / 1.541195 (0.733818) | 2.430736 / 1.468490 (0.962245) | 0.700442 / 4.584777 (-3.884335) | 3.458451 / 3.745712 (-0.287261) | 1.921120 / 5.269862 (-3.348742) | 1.183292 / 4.565676 (-3.382384) | 0.083985 / 0.424275 (-0.340290) | 0.012510 / 0.007607 (0.004903) | 0.589066 / 0.226044 (0.363022) | 5.896070 / 2.268929 (3.627141) | 2.935379 / 55.444624 (-52.509245) | 2.599524 / 6.876477 (-4.276953) | 2.663426 / 2.142072 (0.521354) | 0.812096 / 4.805227 (-3.993131) | 0.152559 / 6.500664 (-6.348105) | 0.066906 / 0.075469 (-0.008563) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.333341 / 1.841788 (-0.508446) | 14.441667 / 8.074308 (6.367359) | 14.754069 / 10.191392 (4.562677) | 0.155707 / 0.680424 (-0.524716) | 0.016983 / 0.534201 (-0.517218) | 0.389386 / 0.579283 (-0.189897) | 0.394106 / 0.434364 (-0.040258) | 0.447355 / 0.540337 (-0.092982) | 0.533142 / 1.386936 (-0.853794) |

</details>
</details>

![](https://cml.dev/watermark.png#99ee4467ce77f8f718159a535e237dd8790b5bed "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007801 / 0.011353 (-0.003552) | 0.004884 / 0.011008 (-0.006124) | 0.114754 / 0.038508 (0.076245) | 0.040427 / 0.023109 (0.017318) | 0.402064 / 0.275898 (0.126166) | 0.428830 / 0.323480 (0.105350) | 0.006429 / 0.007986 (-0.001556) | 0.004394 / 0.004328 (0.000066) | 0.087681 / 0.004250 (0.083431) | 0.053684 / 0.037052 (0.016632) | 0.399967 / 0.258489 (0.141478) | 0.445298 / 0.293841 (0.151457) | 0.033194 / 0.128546 (-0.095352) | 0.010288 / 0.075646 (-0.065359) | 0.390719 / 0.419271 (-0.028552) | 0.059311 / 0.043533 (0.015778) | 0.393651 / 0.255139 (0.138512) | 0.418395 / 0.283200 (0.135196) | 0.121494 / 0.141683 (-0.020189) | 1.735470 / 1.452155 (0.283315) | 1.820485 / 1.492716 (0.327769) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.012887 / 0.018006 (-0.005119) | 0.491652 / 0.000490 (0.491162) | 0.005481 / 0.000200 (0.005281) | 0.000127 / 0.000054 (0.000073) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030931 / 0.037411 (-0.006480) | 0.125212 / 0.014526 (0.110686) | 0.136004 / 0.176557 (-0.040552) | 0.201686 / 0.737135 (-0.535449) | 0.140181 / 0.296338 (-0.156157) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.475003 / 0.215209 (0.259794) | 4.743918 / 2.077655 (2.666263) | 2.149422 / 1.504120 (0.645302) | 1.925016 / 1.541195 (0.383821) | 2.061441 / 1.468490 (0.592951) | 0.619845 / 4.584777 (-3.964932) | 4.534691 / 3.745712 (0.788979) | 2.248198 / 5.269862 (-3.021664) | 1.409868 / 4.565676 (-3.155808) | 0.080265 / 0.424275 (-0.344010) | 0.014455 / 0.007607 (0.006848) | 0.597810 / 0.226044 (0.371765) | 5.845492 / 2.268929 (3.576564) | 2.729139 / 55.444624 (-52.715486) | 2.313879 / 6.876477 (-4.562598) | 2.418763 / 2.142072 (0.276690) | 0.748687 / 4.805227 (-4.056540) | 0.165278 / 6.500664 (-6.335387) | 0.076848 / 0.075469 (0.001379) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.416349 / 1.841788 (-0.425439) | 17.440903 / 8.074308 (9.366595) | 17.025733 / 10.191392 (6.834341) | 0.167428 / 0.680424 (-0.512995) | 0.020484 / 0.534201 (-0.513717) | 0.470273 / 0.579283 (-0.109010) | 0.494380 / 0.434364 (0.060016) | 0.566131 / 0.540337 (0.025794) | 0.690444 / 1.386936 (-0.696492) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007695 / 0.011353 (-0.003657) | 0.005551 / 0.011008 (-0.005457) | 0.087812 / 0.038508 (0.049304) | 0.039107 / 0.023109 (0.015998) | 0.436461 / 0.275898 (0.160563) | 0.465116 / 0.323480 (0.141636) | 0.006590 / 0.007986 (-0.001396) | 0.004672 / 0.004328 (0.000343) | 0.087109 / 0.004250 (0.082858) | 0.054227 / 0.037052 (0.017175) | 0.442660 / 0.258489 (0.184171) | 0.484296 / 0.293841 (0.190455) | 0.033308 / 0.128546 (-0.095238) | 0.010780 / 0.075646 (-0.064866) | 0.095255 / 0.419271 (-0.324016) | 0.054399 / 0.043533 (0.010866) | 0.431734 / 0.255139 (0.176595) | 0.453583 / 0.283200 (0.170383) | 0.116067 / 0.141683 (-0.025616) | 1.780701 / 1.452155 (0.328546) | 1.851077 / 1.492716 (0.358360) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.228000 / 0.018006 (0.209994) | 0.485733 / 0.000490 (0.485243) | 0.003955 / 0.000200 (0.003755) | 0.000109 / 0.000054 (0.000054) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033974 / 0.037411 (-0.003437) | 0.134504 / 0.014526 (0.119978) | 0.144421 / 0.176557 (-0.032135) | 0.202171 / 0.737135 (-0.534964) | 0.152015 / 0.296338 (-0.144323) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.520462 / 0.215209 (0.305253) | 5.233339 / 2.077655 (3.155684) | 2.575013 / 1.504120 (1.070893) | 2.384119 / 1.541195 (0.842924) | 2.403856 / 1.468490 (0.935366) | 0.618656 / 4.584777 (-3.966121) | 4.663582 / 3.745712 (0.917870) | 3.738594 / 5.269862 (-1.531268) | 1.794903 / 4.565676 (-2.770773) | 0.077903 / 0.424275 (-0.346372) | 0.014681 / 0.007607 (0.007074) | 0.648615 / 0.226044 (0.422570) | 6.503721 / 2.268929 (4.234792) | 3.326239 / 55.444624 (-52.118386) | 2.989791 / 6.876477 (-3.886685) | 2.995479 / 2.142072 (0.853407) | 0.765483 / 4.805227 (-4.039744) | 0.169783 / 6.500664 (-6.330882) | 0.077533 / 0.075469 (0.002064) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.518736 / 1.841788 (-0.323051) | 17.989119 / 8.074308 (9.914811) | 15.484365 / 10.191392 (5.292973) | 0.168507 / 0.680424 (-0.511917) | 0.020289 / 0.534201 (-0.513912) | 0.467491 / 0.579283 (-0.111793) | 0.501714 / 0.434364 (0.067350) | 0.553418 / 0.540337 (0.013081) | 0.662199 / 1.386936 (-0.724737) |

</details>
</details>

![](https://cml.dev/watermark.png#5ebda17e4362bd5f6123543a14fa526a3b54481a "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007044 / 0.011353 (-0.004309) | 0.004750 / 0.011008 (-0.006258) | 0.096694 / 0.038508 (0.058186) | 0.035682 / 0.023109 (0.012573) | 0.300613 / 0.275898 (0.024715) | 0.334831 / 0.323480 (0.011351) | 0.006428 / 0.007986 (-0.001558) | 0.004456 / 0.004328 (0.000128) | 0.075060 / 0.004250 (0.070810) | 0.053166 / 0.037052 (0.016114) | 0.299601 / 0.258489 (0.041112) | 0.359521 / 0.293841 (0.065680) | 0.028072 / 0.128546 (-0.100474) | 0.009216 / 0.075646 (-0.066430) | 0.328895 / 0.419271 (-0.090377) | 0.050881 / 0.043533 (0.007349) | 0.298265 / 0.255139 (0.043126) | 0.318095 / 0.283200 (0.034896) | 0.116046 / 0.141683 (-0.025637) | 1.491312 / 1.452155 (0.039157) | 1.556053 / 1.492716 (0.063337) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.014248 / 0.018006 (-0.003758) | 0.551455 / 0.000490 (0.550965) | 0.006096 / 0.000200 (0.005897) | 0.000145 / 0.000054 (0.000091) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030598 / 0.037411 (-0.006813) | 0.109549 / 0.014526 (0.095023) | 0.123207 / 0.176557 (-0.053350) | 0.181940 / 0.737135 (-0.555195) | 0.128965 / 0.296338 (-0.167374) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404552 / 0.215209 (0.189343) | 4.030674 / 2.077655 (1.953020) | 1.841819 / 1.504120 (0.337699) | 1.650055 / 1.541195 (0.108860) | 1.763208 / 1.468490 (0.294718) | 0.532715 / 4.584777 (-4.052062) | 3.774810 / 3.745712 (0.029098) | 3.221927 / 5.269862 (-2.047934) | 1.607974 / 4.565676 (-2.957702) | 0.067160 / 0.424275 (-0.357116) | 0.012479 / 0.007607 (0.004872) | 0.498801 / 0.226044 (0.272757) | 4.980567 / 2.268929 (2.711638) | 2.356017 / 55.444624 (-53.088608) | 2.018975 / 6.876477 (-4.857502) | 2.218343 / 2.142072 (0.076270) | 0.645714 / 4.805227 (-4.159514) | 0.145470 / 6.500664 (-6.355195) | 0.065666 / 0.075469 (-0.009803) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.205756 / 1.841788 (-0.636031) | 15.682779 / 8.074308 (7.608470) | 14.748987 / 10.191392 (4.557595) | 0.167105 / 0.680424 (-0.513319) | 0.017554 / 0.534201 (-0.516647) | 0.393924 / 0.579283 (-0.185359) | 0.432659 / 0.434364 (-0.001705) | 0.502033 / 0.540337 (-0.038304) | 0.602244 / 1.386936 (-0.784692) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007077 / 0.011353 (-0.004276) | 0.004911 / 0.011008 (-0.006097) | 0.075120 / 0.038508 (0.036612) | 0.035460 / 0.023109 (0.012351) | 0.362569 / 0.275898 (0.086671) | 0.398995 / 0.323480 (0.075515) | 0.006587 / 0.007986 (-0.001398) | 0.004571 / 0.004328 (0.000242) | 0.074647 / 0.004250 (0.070397) | 0.057331 / 0.037052 (0.020279) | 0.365123 / 0.258489 (0.106634) | 0.408617 / 0.293841 (0.114776) | 0.028911 / 0.128546 (-0.099635) | 0.009533 / 0.075646 (-0.066113) | 0.081566 / 0.419271 (-0.337705) | 0.048841 / 0.043533 (0.005308) | 0.367245 / 0.255139 (0.112106) | 0.375975 / 0.283200 (0.092776) | 0.123211 / 0.141683 (-0.018472) | 1.471588 / 1.452155 (0.019433) | 1.569342 / 1.492716 (0.076625) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.328443 / 0.018006 (0.310436) | 0.541402 / 0.000490 (0.540912) | 0.000440 / 0.000200 (0.000240) | 0.000058 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030772 / 0.037411 (-0.006639) | 0.115833 / 0.014526 (0.101307) | 0.127837 / 0.176557 (-0.048719) | 0.180897 / 0.737135 (-0.556238) | 0.132458 / 0.296338 (-0.163881) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445979 / 0.215209 (0.230770) | 4.453101 / 2.077655 (2.375447) | 2.276625 / 1.504120 (0.772505) | 2.102167 / 1.541195 (0.560972) | 2.181583 / 1.468490 (0.713093) | 0.525069 / 4.584777 (-4.059708) | 3.803446 / 3.745712 (0.057734) | 1.954173 / 5.269862 (-3.315688) | 1.088734 / 4.565676 (-3.476942) | 0.066020 / 0.424275 (-0.358255) | 0.012158 / 0.007607 (0.004551) | 0.546828 / 0.226044 (0.320783) | 5.454060 / 2.268929 (3.185132) | 2.756154 / 55.444624 (-52.688470) | 2.476501 / 6.876477 (-4.399976) | 2.525875 / 2.142072 (0.383803) | 0.647515 / 4.805227 (-4.157712) | 0.144511 / 6.500664 (-6.356153) | 0.067060 / 0.075469 (-0.008409) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.306456 / 1.841788 (-0.535332) | 15.822623 / 8.074308 (7.748315) | 14.929114 / 10.191392 (4.737721) | 0.168650 / 0.680424 (-0.511773) | 0.018043 / 0.534201 (-0.516158) | 0.396712 / 0.579283 (-0.182572) | 0.425800 / 0.434364 (-0.008564) | 0.466452 / 0.540337 (-0.073885) | 0.564370 / 1.386936 (-0.822566) |

</details>
</details>

![](https://cml.dev/watermark.png#5ebda17e4362bd5f6123543a14fa526a3b54481a "CML watermark")