Replace deprecated license_file in setup.cfg
State: closed

Replace deprecated license_file in `setup.cfg`.

See: https://github.com/huggingface/datasets/actions/runs/6610930650/job/17953825724?pr=6331
```
      /tmp/pip-build-env-a51hls20/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006884 / 0.011353 (-0.004469) | 0.004132 / 0.011008 (-0.006877) | 0.085993 / 0.038508 (0.047485) | 0.084049 / 0.023109 (0.060940) | 0.346194 / 0.275898 (0.070296) | 0.386999 / 0.323480 (0.063519) | 0.004185 / 0.007986 (-0.003801) | 0.004354 / 0.004328 (0.000026) | 0.065137 / 0.004250 (0.060886) | 0.057629 / 0.037052 (0.020577) | 0.353639 / 0.258489 (0.095150) | 0.400815 / 0.293841 (0.106974) | 0.031370 / 0.128546 (-0.097176) | 0.008719 / 0.075646 (-0.066927) | 0.289579 / 0.419271 (-0.129693) | 0.052826 / 0.043533 (0.009293) | 0.351110 / 0.255139 (0.095971) | 0.375663 / 0.283200 (0.092464) | 0.025892 / 0.141683 (-0.115791) | 1.481943 / 1.452155 (0.029789) | 1.541494 / 1.492716 (0.048778) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.240007 / 0.018006 (0.222000) | 0.456216 / 0.000490 (0.455726) | 0.009348 / 0.000200 (0.009148) | 0.000370 / 0.000054 (0.000315) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029541 / 0.037411 (-0.007870) | 0.088394 / 0.014526 (0.073868) | 0.098460 / 0.176557 (-0.078096) | 0.154053 / 0.737135 (-0.583083) | 0.098821 / 0.296338 (-0.197518) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.386751 / 0.215209 (0.171542) | 3.809818 / 2.077655 (1.732164) | 1.833439 / 1.504120 (0.329319) | 1.686924 / 1.541195 (0.145729) | 1.796882 / 1.468490 (0.328392) | 0.488853 / 4.584777 (-4.095924) | 3.606369 / 3.745712 (-0.139343) | 3.460003 / 5.269862 (-1.809858) | 2.087493 / 4.565676 (-2.478184) | 0.056838 / 0.424275 (-0.367437) | 0.007679 / 0.007607 (0.000072) | 0.455080 / 0.226044 (0.229036) | 4.539227 / 2.268929 (2.270299) | 2.337245 / 55.444624 (-53.107379) | 1.988195 / 6.876477 (-4.888281) | 2.067473 / 2.142072 (-0.074600) | 0.576640 / 4.805227 (-4.228587) | 0.132140 / 6.500664 (-6.368525) | 0.060737 / 0.075469 (-0.014732) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.268866 / 1.841788 (-0.572922) | 19.695296 / 8.074308 (11.620988) | 14.431254 / 10.191392 (4.239862) | 0.166779 / 0.680424 (-0.513645) | 0.018262 / 0.534201 (-0.515939) | 0.390406 / 0.579283 (-0.188877) | 0.411284 / 0.434364 (-0.023080) | 0.456696 / 0.540337 (-0.083642) | 0.629660 / 1.386936 (-0.757276) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007210 / 0.011353 (-0.004143) | 0.004124 / 0.011008 (-0.006884) | 0.065877 / 0.038508 (0.027368) | 0.086242 / 0.023109 (0.063133) | 0.420087 / 0.275898 (0.144189) | 0.454327 / 0.323480 (0.130847) | 0.005586 / 0.007986 (-0.002399) | 0.003465 / 0.004328 (-0.000863) | 0.065153 / 0.004250 (0.060902) | 0.059337 / 0.037052 (0.022285) | 0.420913 / 0.258489 (0.162424) | 0.458552 / 0.293841 (0.164711) | 0.032335 / 0.128546 (-0.096211) | 0.008672 / 0.075646 (-0.066974) | 0.072029 / 0.419271 (-0.347242) | 0.048148 / 0.043533 (0.004615) | 0.423334 / 0.255139 (0.168196) | 0.440616 / 0.283200 (0.157416) | 0.023761 / 0.141683 (-0.117922) | 1.487022 / 1.452155 (0.034868) | 1.554028 / 1.492716 (0.061312) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216693 / 0.018006 (0.198687) | 0.446359 / 0.000490 (0.445869) | 0.005294 / 0.000200 (0.005094) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034655 / 0.037411 (-0.002756) | 0.099479 / 0.014526 (0.084953) | 0.111822 / 0.176557 (-0.064735) | 0.160675 / 0.737135 (-0.576461) | 0.108718 / 0.296338 (-0.187621) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440270 / 0.215209 (0.225061) | 4.389013 / 2.077655 (2.311358) | 2.408007 / 1.504120 (0.903887) | 2.237233 / 1.541195 (0.696038) | 2.344131 / 1.468490 (0.875641) | 0.493143 / 4.584777 (-4.091634) | 3.620024 / 3.745712 (-0.125688) | 3.335810 / 5.269862 (-1.934052) | 2.079256 / 4.565676 (-2.486420) | 0.058324 / 0.424275 (-0.365951) | 0.007410 / 0.007607 (-0.000197) | 0.512057 / 0.226044 (0.286013) | 5.120629 / 2.268929 (2.851701) | 2.913268 / 55.444624 (-52.531356) | 2.558214 / 6.876477 (-4.318262) | 2.784146 / 2.142072 (0.642074) | 0.593308 / 4.805227 (-4.211920) | 0.134941 / 6.500664 (-6.365723) | 0.062292 / 0.075469 (-0.013177) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351795 / 1.841788 (-0.489993) | 20.489559 / 8.074308 (12.415251) | 15.046116 / 10.191392 (4.854724) | 0.166339 / 0.680424 (-0.514085) | 0.020449 / 0.534201 (-0.513752) | 0.406570 / 0.579283 (-0.172713) | 0.423405 / 0.434364 (-0.010959) | 0.474541 / 0.540337 (-0.065796) | 0.653280 / 1.386936 (-0.733656) |

</details>
</details>

![](https://cml.dev/watermark.png#3bde0f0f0e556e55b95c72b0f83bdcf7145c813c "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006362 / 0.011353 (-0.004991) | 0.003990 / 0.011008 (-0.007018) | 0.084020 / 0.038508 (0.045512) | 0.072198 / 0.023109 (0.049089) | 0.335992 / 0.275898 (0.060094) | 0.362056 / 0.323480 (0.038576) | 0.005298 / 0.007986 (-0.002688) | 0.003421 / 0.004328 (-0.000908) | 0.065343 / 0.004250 (0.061092) | 0.053310 / 0.037052 (0.016258) | 0.344855 / 0.258489 (0.086366) | 0.385524 / 0.293841 (0.091683) | 0.030209 / 0.128546 (-0.098337) | 0.008465 / 0.075646 (-0.067181) | 0.287359 / 0.419271 (-0.131912) | 0.051371 / 0.043533 (0.007838) | 0.338716 / 0.255139 (0.083577) | 0.351730 / 0.283200 (0.068530) | 0.023581 / 0.141683 (-0.118102) | 1.473772 / 1.452155 (0.021617) | 1.560594 / 1.492716 (0.067878) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.309019 / 0.018006 (0.291013) | 0.561428 / 0.000490 (0.560939) | 0.007237 / 0.000200 (0.007038) | 0.000266 / 0.000054 (0.000212) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028172 / 0.037411 (-0.009239) | 0.081050 / 0.014526 (0.066524) | 0.095952 / 0.176557 (-0.080604) | 0.151796 / 0.737135 (-0.585340) | 0.096132 / 0.296338 (-0.200206) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384287 / 0.215209 (0.169078) | 3.840797 / 2.077655 (1.763142) | 1.891120 / 1.504120 (0.387000) | 1.743498 / 1.541195 (0.202303) | 1.821037 / 1.468490 (0.352547) | 0.484946 / 4.584777 (-4.099831) | 3.586053 / 3.745712 (-0.159659) | 3.446215 / 5.269862 (-1.823647) | 2.054352 / 4.565676 (-2.511325) | 0.057315 / 0.424275 (-0.366960) | 0.007541 / 0.007607 (-0.000066) | 0.464088 / 0.226044 (0.238044) | 4.634005 / 2.268929 (2.365076) | 2.355818 / 55.444624 (-53.088806) | 2.045584 / 6.876477 (-4.830893) | 2.039455 / 2.142072 (-0.102617) | 0.576137 / 4.805227 (-4.229090) | 0.132071 / 6.500664 (-6.368593) | 0.059611 / 0.075469 (-0.015858) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.280078 / 1.841788 (-0.561710) | 19.054079 / 8.074308 (10.979771) | 14.291090 / 10.191392 (4.099698) | 0.170607 / 0.680424 (-0.509817) | 0.018489 / 0.534201 (-0.515712) | 0.391802 / 0.579283 (-0.187481) | 0.418945 / 0.434364 (-0.015419) | 0.464084 / 0.540337 (-0.076254) | 0.638099 / 1.386936 (-0.748837) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006735 / 0.011353 (-0.004618) | 0.004133 / 0.011008 (-0.006876) | 0.064620 / 0.038508 (0.026112) | 0.076395 / 0.023109 (0.053286) | 0.399659 / 0.275898 (0.123761) | 0.426821 / 0.323480 (0.103341) | 0.006407 / 0.007986 (-0.001578) | 0.003472 / 0.004328 (-0.000857) | 0.064922 / 0.004250 (0.060671) | 0.058312 / 0.037052 (0.021260) | 0.403286 / 0.258489 (0.144797) | 0.437772 / 0.293841 (0.143931) | 0.032323 / 0.128546 (-0.096223) | 0.008727 / 0.075646 (-0.066919) | 0.071344 / 0.419271 (-0.347927) | 0.048673 / 0.043533 (0.005141) | 0.400693 / 0.255139 (0.145554) | 0.418668 / 0.283200 (0.135468) | 0.022871 / 0.141683 (-0.118812) | 1.517691 / 1.452155 (0.065536) | 1.552021 / 1.492716 (0.059305) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.305279 / 0.018006 (0.287272) | 0.520054 / 0.000490 (0.519564) | 0.007247 / 0.000200 (0.007047) | 0.000098 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032001 / 0.037411 (-0.005410) | 0.091273 / 0.014526 (0.076747) | 0.106480 / 0.176557 (-0.070077) | 0.163122 / 0.737135 (-0.574014) | 0.105244 / 0.296338 (-0.191094) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.432207 / 0.215209 (0.216998) | 4.304856 / 2.077655 (2.227202) | 2.326790 / 1.504120 (0.822670) | 2.150081 / 1.541195 (0.608886) | 2.150558 / 1.468490 (0.682068) | 0.488808 / 4.584777 (-4.095969) | 3.690435 / 3.745712 (-0.055277) | 3.302625 / 5.269862 (-1.967236) | 2.044193 / 4.565676 (-2.521483) | 0.057520 / 0.424275 (-0.366755) | 0.007281 / 0.007607 (-0.000326) | 0.521078 / 0.226044 (0.295034) | 5.162620 / 2.268929 (2.893691) | 2.744041 / 55.444624 (-52.700583) | 2.407211 / 6.876477 (-4.469266) | 2.606290 / 2.142072 (0.464217) | 0.586412 / 4.805227 (-4.218815) | 0.132152 / 6.500664 (-6.368512) | 0.059424 / 0.075469 (-0.016045) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.351879 / 1.841788 (-0.489908) | 19.460608 / 8.074308 (11.386299) | 14.643413 / 10.191392 (4.452021) | 0.168062 / 0.680424 (-0.512362) | 0.020396 / 0.534201 (-0.513805) | 0.395885 / 0.579283 (-0.183398) | 0.439551 / 0.434364 (0.005187) | 0.473051 / 0.540337 (-0.067286) | 0.644614 / 1.386936 (-0.742322) |

</details>
</details>

![](https://cml.dev/watermark.png#732b2ed47728fffc8d74f92691c21de8ac7423fe "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.014708 / 0.011353 (0.003355) | 0.008309 / 0.011008 (-0.002699) | 0.138986 / 0.038508 (0.100478) | 0.121781 / 0.023109 (0.098671) | 0.495536 / 0.275898 (0.219637) | 0.565195 / 0.323480 (0.241715) | 0.008018 / 0.007986 (0.000032) | 0.004904 / 0.004328 (0.000575) | 0.080622 / 0.004250 (0.076371) | 0.078917 / 0.037052 (0.041865) | 0.489424 / 0.258489 (0.230935) | 0.540496 / 0.293841 (0.246656) | 0.061110 / 0.128546 (-0.067437) | 0.021443 / 0.075646 (-0.054203) | 0.395789 / 0.419271 (-0.023482) | 0.076727 / 0.043533 (0.033194) | 0.427808 / 0.255139 (0.172669) | 0.519672 / 0.283200 (0.236473) | 0.041607 / 0.141683 (-0.100076) | 2.098675 / 1.452155 (0.646520) | 2.175123 / 1.492716 (0.682407) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.275784 / 0.018006 (0.257777) | 0.707103 / 0.000490 (0.706613) | 0.011524 / 0.000200 (0.011324) | 0.000390 / 0.000054 (0.000336) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032897 / 0.037411 (-0.004514) | 0.123239 / 0.014526 (0.108713) | 0.151815 / 0.176557 (-0.024741) | 0.214790 / 0.737135 (-0.522345) | 0.139166 / 0.296338 (-0.157173) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.740662 / 0.215209 (0.525453) | 7.540376 / 2.077655 (5.462721) | 3.168207 / 1.504120 (1.664087) | 2.745663 / 1.541195 (1.204468) | 2.714020 / 1.468490 (1.245530) | 1.182632 / 4.584777 (-3.402145) | 6.365807 / 3.745712 (2.620095) | 6.317228 / 5.269862 (1.047366) | 4.061107 / 4.565676 (-0.504569) | 0.146939 / 0.424275 (-0.277336) | 0.011765 / 0.007607 (0.004158) | 0.910564 / 0.226044 (0.684519) | 9.020618 / 2.268929 (6.751689) | 4.180748 / 55.444624 (-51.263876) | 3.290257 / 6.876477 (-3.586220) | 3.363172 / 2.142072 (1.221099) | 1.239142 / 4.805227 (-3.566086) | 0.294965 / 6.500664 (-6.205699) | 0.088520 / 0.075469 (0.013051) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.867528 / 1.841788 (0.025741) | 29.494058 / 8.074308 (21.419750) | 31.386703 / 10.191392 (21.195311) | 0.302488 / 0.680424 (-0.377936) | 0.036116 / 0.534201 (-0.498085) | 0.622112 / 0.579283 (0.042829) | 0.775658 / 0.434364 (0.341294) | 0.632452 / 0.540337 (0.092115) | 0.909424 / 1.386936 (-0.477512) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.016002 / 0.011353 (0.004649) | 0.007007 / 0.011008 (-0.004002) | 0.100463 / 0.038508 (0.061955) | 0.124423 / 0.023109 (0.101314) | 0.556014 / 0.275898 (0.280116) | 0.600909 / 0.323480 (0.277429) | 0.007272 / 0.007986 (-0.000714) | 0.006743 / 0.004328 (0.002415) | 0.088575 / 0.004250 (0.084324) | 0.066003 / 0.037052 (0.028951) | 0.580080 / 0.258489 (0.321591) | 0.655567 / 0.293841 (0.361726) | 0.065295 / 0.128546 (-0.063252) | 0.021105 / 0.075646 (-0.054541) | 0.120044 / 0.419271 (-0.299227) | 0.081133 / 0.043533 (0.037600) | 0.570322 / 0.255139 (0.315183) | 0.581134 / 0.283200 (0.297934) | 0.046298 / 0.141683 (-0.095385) | 2.113200 / 1.452155 (0.661045) | 2.344187 / 1.492716 (0.851471) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.284517 / 0.018006 (0.266511) | 0.611834 / 0.000490 (0.611345) | 0.005581 / 0.000200 (0.005381) | 0.000153 / 0.000054 (0.000098) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.042162 / 0.037411 (0.004750) | 0.114496 / 0.014526 (0.099970) | 0.134034 / 0.176557 (-0.042523) | 0.201649 / 0.737135 (-0.535486) | 0.143235 / 0.296338 (-0.153103) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.764863 / 0.215209 (0.549654) | 7.603076 / 2.077655 (5.525421) | 3.318911 / 1.504120 (1.814791) | 2.939815 / 1.541195 (1.398620) | 2.870911 / 1.468490 (1.402421) | 1.171978 / 4.584777 (-3.412799) | 6.479933 / 3.745712 (2.734221) | 5.944387 / 5.269862 (0.674526) | 4.282625 / 4.565676 (-0.283051) | 0.123672 / 0.424275 (-0.300603) | 0.009666 / 0.007607 (0.002059) | 0.870683 / 0.226044 (0.644638) | 9.187788 / 2.268929 (6.918859) | 4.431818 / 55.444624 (-51.012807) | 3.460457 / 6.876477 (-3.416020) | 3.708198 / 2.142072 (1.566126) | 1.353673 / 4.805227 (-3.451554) | 0.264274 / 6.500664 (-6.236390) | 0.074943 / 0.075469 (-0.000526) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 2.073810 / 1.841788 (0.232023) | 29.182464 / 8.074308 (21.108156) | 30.527040 / 10.191392 (20.335648) | 0.307561 / 0.680424 (-0.372863) | 0.047384 / 0.534201 (-0.486817) | 0.662760 / 0.579283 (0.083477) | 0.768321 / 0.434364 (0.333957) | 0.692296 / 0.540337 (0.151959) | 0.955197 / 1.386936 (-0.431739) |

</details>
</details>

![](https://cml.dev/watermark.png#1e82d6f017c7fc0ab6b65847c1e34772c880d3b7 "CML watermark")