Preserve split order in DataFilesDict
Opened by albertvillanova (closed)

After investigation, I have found that this copy forces the splits to be sorted alphabetically: https://github.com/huggingface/datasets/blob/029227a116c14720afca71b9b22e78eb2a1c09a6/src/datasets/builder.py#L556

This PR removes the alphabetically sort of `DataFilesDict` keys.
- Note that for a `dict`, the order of keys is relevant when hashing:
  ```python
  hash1 = Hasher.hash({'train': 'train.csv', 'test': 'test.csv'})
  hash2 = Hasher.hash({'test': 'test.csv', 'train': 'train.csv'})
  assert hash1 != hash2
  ```
- The `DataFilesDict` is a subclass of `dict`, thus the order should be relevant as well
  ```python
  hash1 = Hasher.hash(DataFilesDict({'train': 'train.csv', 'test': 'test.csv'}))
  hash2 = Hasher.hash(DataFilesDict({'test': 'test.csv', 'train': 'train.csv'}))
  assert hash1 != hash2
  ```

Fix #6196.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007621 / 0.011353 (-0.003732) | 0.004534 / 0.011008 (-0.006475) | 0.099834 / 0.038508 (0.061326) | 0.083029 / 0.023109 (0.059920) | 0.387559 / 0.275898 (0.111661) | 0.422453 / 0.323480 (0.098973) | 0.006070 / 0.007986 (-0.001916) | 0.003725 / 0.004328 (-0.000604) | 0.075923 / 0.004250 (0.071672) | 0.060578 / 0.037052 (0.023525) | 0.403569 / 0.258489 (0.145079) | 0.444991 / 0.293841 (0.151150) | 0.035847 / 0.128546 (-0.092699) | 0.009872 / 0.075646 (-0.065774) | 0.335506 / 0.419271 (-0.083766) | 0.060509 / 0.043533 (0.016976) | 0.381034 / 0.255139 (0.125895) | 0.426938 / 0.283200 (0.143738) | 0.027662 / 0.141683 (-0.114021) | 1.729565 / 1.452155 (0.277410) | 1.842082 / 1.492716 (0.349366) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230371 / 0.018006 (0.212365) | 0.518216 / 0.000490 (0.517726) | 0.003897 / 0.000200 (0.003697) | 0.000087 / 0.000054 (0.000033) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031942 / 0.037411 (-0.005470) | 0.096609 / 0.014526 (0.082083) | 0.112707 / 0.176557 (-0.063850) | 0.178849 / 0.737135 (-0.558286) | 0.112793 / 0.296338 (-0.183546) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.445896 / 0.215209 (0.230687) | 4.451173 / 2.077655 (2.373519) | 2.183380 / 1.504120 (0.679260) | 1.991583 / 1.541195 (0.450388) | 2.096219 / 1.468490 (0.627729) | 0.566692 / 4.584777 (-4.018085) | 4.078278 / 3.745712 (0.332566) | 3.787950 / 5.269862 (-1.481911) | 2.372651 / 4.565676 (-2.193025) | 0.065500 / 0.424275 (-0.358775) | 0.008918 / 0.007607 (0.001311) | 0.535589 / 0.226044 (0.309545) | 5.364130 / 2.268929 (3.095201) | 2.805381 / 55.444624 (-52.639244) | 2.350769 / 6.876477 (-4.525708) | 2.592887 / 2.142072 (0.450814) | 0.675475 / 4.805227 (-4.129752) | 0.153907 / 6.500664 (-6.346757) | 0.071138 / 0.075469 (-0.004331) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.498236 / 1.841788 (-0.343552) | 22.810460 / 8.074308 (14.736152) | 16.275035 / 10.191392 (6.083643) | 0.200242 / 0.680424 (-0.480182) | 0.021553 / 0.534201 (-0.512648) | 0.469437 / 0.579283 (-0.109846) | 0.477752 / 0.434364 (0.043388) | 0.537411 / 0.540337 (-0.002927) | 0.741730 / 1.386936 (-0.645206) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008009 / 0.011353 (-0.003344) | 0.004626 / 0.011008 (-0.006382) | 0.074871 / 0.038508 (0.036363) | 0.085214 / 0.023109 (0.062105) | 0.478057 / 0.275898 (0.202159) | 0.522038 / 0.323480 (0.198558) | 0.007055 / 0.007986 (-0.000931) | 0.003813 / 0.004328 (-0.000515) | 0.076238 / 0.004250 (0.071988) | 0.065738 / 0.037052 (0.028686) | 0.484391 / 0.258489 (0.225902) | 0.524425 / 0.293841 (0.230584) | 0.038375 / 0.128546 (-0.090171) | 0.009964 / 0.075646 (-0.065682) | 0.084027 / 0.419271 (-0.335245) | 0.056979 / 0.043533 (0.013447) | 0.486910 / 0.255139 (0.231771) | 0.501185 / 0.283200 (0.217985) | 0.027000 / 0.141683 (-0.114683) | 1.767378 / 1.452155 (0.315224) | 1.870511 / 1.492716 (0.377795) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.267067 / 0.018006 (0.249061) | 0.501714 / 0.000490 (0.501224) | 0.012379 / 0.000200 (0.012179) | 0.000129 / 0.000054 (0.000075) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036706 / 0.037411 (-0.000706) | 0.110064 / 0.014526 (0.095538) | 0.124896 / 0.176557 (-0.051660) | 0.186730 / 0.737135 (-0.550405) | 0.123501 / 0.296338 (-0.172837) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.510793 / 0.215209 (0.295583) | 5.133056 / 2.077655 (3.055401) | 2.776456 / 1.504120 (1.272336) | 2.595557 / 1.541195 (1.054362) | 2.717922 / 1.468490 (1.249432) | 0.578333 / 4.584777 (-4.006444) | 4.169935 / 3.745712 (0.424223) | 3.800078 / 5.269862 (-1.469784) | 2.385866 / 4.565676 (-2.179810) | 0.068114 / 0.424275 (-0.356161) | 0.008771 / 0.007607 (0.001164) | 0.597894 / 0.226044 (0.371850) | 5.970293 / 2.268929 (3.701364) | 3.352715 / 55.444624 (-52.091909) | 2.972062 / 6.876477 (-3.904415) | 3.179232 / 2.142072 (1.037160) | 0.689838 / 4.805227 (-4.115389) | 0.154890 / 6.500664 (-6.345774) | 0.072321 / 0.075469 (-0.003148) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.613666 / 1.841788 (-0.228121) | 23.441538 / 8.074308 (15.367230) | 17.105417 / 10.191392 (6.914025) | 0.171449 / 0.680424 (-0.508975) | 0.023257 / 0.534201 (-0.510944) | 0.466724 / 0.579283 (-0.112559) | 0.470835 / 0.434364 (0.036471) | 0.561860 / 0.540337 (0.021523) | 0.759048 / 1.386936 (-0.627888) |

</details>
</details>

![](https://cml.dev/watermark.png#0f3b6eaf69d3352394d3bf3c4d6ed01dd2af5860 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007557 / 0.011353 (-0.003796) | 0.004211 / 0.011008 (-0.006797) | 0.096243 / 0.038508 (0.057735) | 0.083603 / 0.023109 (0.060493) | 0.367114 / 0.275898 (0.091216) | 0.415182 / 0.323480 (0.091702) | 0.005796 / 0.007986 (-0.002189) | 0.003791 / 0.004328 (-0.000537) | 0.073505 / 0.004250 (0.069254) | 0.060335 / 0.037052 (0.023283) | 0.392182 / 0.258489 (0.133693) | 0.421315 / 0.293841 (0.127474) | 0.036128 / 0.128546 (-0.092419) | 0.009953 / 0.075646 (-0.065693) | 0.338965 / 0.419271 (-0.080307) | 0.061006 / 0.043533 (0.017473) | 0.372317 / 0.255139 (0.117178) | 0.414367 / 0.283200 (0.131167) | 0.026970 / 0.141683 (-0.114713) | 1.730381 / 1.452155 (0.278227) | 1.808340 / 1.492716 (0.315624) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222622 / 0.018006 (0.204615) | 0.474064 / 0.000490 (0.473574) | 0.004817 / 0.000200 (0.004617) | 0.000089 / 0.000054 (0.000034) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032528 / 0.037411 (-0.004883) | 0.097457 / 0.014526 (0.082931) | 0.112273 / 0.176557 (-0.064283) | 0.177953 / 0.737135 (-0.559182) | 0.112358 / 0.296338 (-0.183981) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.442601 / 0.215209 (0.227392) | 4.442065 / 2.077655 (2.364410) | 2.156813 / 1.504120 (0.652694) | 1.970289 / 1.541195 (0.429094) | 2.052878 / 1.468490 (0.584388) | 0.562661 / 4.584777 (-4.022116) | 4.255529 / 3.745712 (0.509817) | 3.767650 / 5.269862 (-1.502212) | 2.431078 / 4.565676 (-2.134598) | 0.065624 / 0.424275 (-0.358651) | 0.008738 / 0.007607 (0.001131) | 0.546839 / 0.226044 (0.320795) | 5.362863 / 2.268929 (3.093934) | 2.695924 / 55.444624 (-52.748701) | 2.334589 / 6.876477 (-4.541888) | 2.530757 / 2.142072 (0.388685) | 0.675991 / 4.805227 (-4.129236) | 0.153852 / 6.500664 (-6.346813) | 0.069189 / 0.075469 (-0.006280) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.522916 / 1.841788 (-0.318872) | 21.515907 / 8.074308 (13.441599) | 16.411708 / 10.191392 (6.220316) | 0.168245 / 0.680424 (-0.512179) | 0.021165 / 0.534201 (-0.513036) | 0.461838 / 0.579283 (-0.117446) | 0.488867 / 0.434364 (0.054503) | 0.536278 / 0.540337 (-0.004059) | 0.766690 / 1.386936 (-0.620246) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007683 / 0.011353 (-0.003670) | 0.004401 / 0.011008 (-0.006608) | 0.075463 / 0.038508 (0.036955) | 0.081737 / 0.023109 (0.058628) | 0.466469 / 0.275898 (0.190571) | 0.514909 / 0.323480 (0.191429) | 0.006106 / 0.007986 (-0.001880) | 0.003936 / 0.004328 (-0.000393) | 0.076773 / 0.004250 (0.072523) | 0.061025 / 0.037052 (0.023973) | 0.473348 / 0.258489 (0.214858) | 0.525326 / 0.293841 (0.231485) | 0.038224 / 0.128546 (-0.090322) | 0.009559 / 0.075646 (-0.066087) | 0.080847 / 0.419271 (-0.338424) | 0.056738 / 0.043533 (0.013205) | 0.475116 / 0.255139 (0.219977) | 0.494689 / 0.283200 (0.211490) | 0.029364 / 0.141683 (-0.112319) | 1.796681 / 1.452155 (0.344527) | 1.850600 / 1.492716 (0.357884) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.327126 / 0.018006 (0.309119) | 0.469186 / 0.000490 (0.468696) | 0.050600 / 0.000200 (0.050400) | 0.000439 / 0.000054 (0.000385) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036710 / 0.037411 (-0.000701) | 0.108669 / 0.014526 (0.094143) | 0.119808 / 0.176557 (-0.056748) | 0.181501 / 0.737135 (-0.555634) | 0.121487 / 0.296338 (-0.174852) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.509076 / 0.215209 (0.293867) | 5.056970 / 2.077655 (2.979316) | 2.775958 / 1.504120 (1.271838) | 2.592548 / 1.541195 (1.051353) | 2.654381 / 1.468490 (1.185890) | 0.557407 / 4.584777 (-4.027370) | 4.418232 / 3.745712 (0.672519) | 3.698072 / 5.269862 (-1.571790) | 2.380607 / 4.565676 (-2.185069) | 0.066242 / 0.424275 (-0.358034) | 0.008350 / 0.007607 (0.000743) | 0.572354 / 0.226044 (0.346309) | 5.857637 / 2.268929 (3.588709) | 3.242512 / 55.444624 (-52.202112) | 2.891144 / 6.876477 (-3.985332) | 3.217987 / 2.142072 (1.075915) | 0.676049 / 4.805227 (-4.129178) | 0.155515 / 6.500664 (-6.345149) | 0.068616 / 0.075469 (-0.006853) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.670048 / 1.841788 (-0.171740) | 22.629573 / 8.074308 (14.555265) | 16.887676 / 10.191392 (6.696284) | 0.168571 / 0.680424 (-0.511853) | 0.023361 / 0.534201 (-0.510840) | 0.463358 / 0.579283 (-0.115925) | 0.463278 / 0.434364 (0.028914) | 0.602397 / 0.540337 (0.062060) | 0.793249 / 1.386936 (-0.593687) |

</details>
</details>

![](https://cml.dev/watermark.png#eee318573aba6574a43d457aa0347348c1f3e4aa "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006693 / 0.011353 (-0.004660) | 0.004100 / 0.011008 (-0.006908) | 0.084166 / 0.038508 (0.045658) | 0.074469 / 0.023109 (0.051360) | 0.356092 / 0.275898 (0.080194) | 0.392389 / 0.323480 (0.068909) | 0.003996 / 0.007986 (-0.003990) | 0.004020 / 0.004328 (-0.000308) | 0.064997 / 0.004250 (0.060747) | 0.053897 / 0.037052 (0.016845) | 0.362942 / 0.258489 (0.104453) | 0.408694 / 0.293841 (0.114854) | 0.031656 / 0.128546 (-0.096890) | 0.008713 / 0.075646 (-0.066933) | 0.289306 / 0.419271 (-0.129966) | 0.053067 / 0.043533 (0.009534) | 0.358740 / 0.255139 (0.103601) | 0.393347 / 0.283200 (0.110147) | 0.025430 / 0.141683 (-0.116253) | 1.486114 / 1.452155 (0.033959) | 1.572698 / 1.492716 (0.079981) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.215423 / 0.018006 (0.197417) | 0.467694 / 0.000490 (0.467204) | 0.003965 / 0.000200 (0.003765) | 0.000112 / 0.000054 (0.000057) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027936 / 0.037411 (-0.009475) | 0.084235 / 0.014526 (0.069709) | 0.136275 / 0.176557 (-0.040282) | 0.151154 / 0.737135 (-0.585982) | 0.185592 / 0.296338 (-0.110747) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.393784 / 0.215209 (0.178575) | 3.927878 / 2.077655 (1.850223) | 1.961216 / 1.504120 (0.457096) | 1.802264 / 1.541195 (0.261069) | 1.971186 / 1.468490 (0.502696) | 0.487981 / 4.584777 (-4.096796) | 3.649046 / 3.745712 (-0.096666) | 3.302471 / 5.269862 (-1.967391) | 2.058075 / 4.565676 (-2.507602) | 0.057072 / 0.424275 (-0.367203) | 0.007624 / 0.007607 (0.000017) | 0.470139 / 0.226044 (0.244095) | 4.697711 / 2.268929 (2.428783) | 2.494813 / 55.444624 (-52.949811) | 2.133084 / 6.876477 (-4.743393) | 2.329740 / 2.142072 (0.187667) | 0.585857 / 4.805227 (-4.219371) | 0.134442 / 6.500664 (-6.366223) | 0.060860 / 0.075469 (-0.014609) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.248504 / 1.841788 (-0.593283) | 19.448427 / 8.074308 (11.374119) | 14.446139 / 10.191392 (4.254747) | 0.168081 / 0.680424 (-0.512342) | 0.018028 / 0.534201 (-0.516173) | 0.395061 / 0.579283 (-0.184222) | 0.418777 / 0.434364 (-0.015587) | 0.454509 / 0.540337 (-0.085828) | 0.628488 / 1.386936 (-0.758448) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006946 / 0.011353 (-0.004406) | 0.004096 / 0.011008 (-0.006912) | 0.065322 / 0.038508 (0.026813) | 0.074336 / 0.023109 (0.051227) | 0.405327 / 0.275898 (0.129429) | 0.436878 / 0.323480 (0.113398) | 0.006083 / 0.007986 (-0.001902) | 0.003345 / 0.004328 (-0.000984) | 0.065725 / 0.004250 (0.061474) | 0.056398 / 0.037052 (0.019345) | 0.406906 / 0.258489 (0.148417) | 0.443330 / 0.293841 (0.149489) | 0.033036 / 0.128546 (-0.095510) | 0.008503 / 0.075646 (-0.067144) | 0.071865 / 0.419271 (-0.347406) | 0.048956 / 0.043533 (0.005423) | 0.404579 / 0.255139 (0.149440) | 0.424904 / 0.283200 (0.141704) | 0.021786 / 0.141683 (-0.119897) | 1.491868 / 1.452155 (0.039713) | 1.565252 / 1.492716 (0.072536) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.231363 / 0.018006 (0.213357) | 0.454962 / 0.000490 (0.454472) | 0.004680 / 0.000200 (0.004480) | 0.000100 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032569 / 0.037411 (-0.004843) | 0.094928 / 0.014526 (0.080402) | 0.108096 / 0.176557 (-0.068461) | 0.158727 / 0.737135 (-0.578409) | 0.106951 / 0.296338 (-0.189387) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.431469 / 0.215209 (0.216260) | 4.283929 / 2.077655 (2.206274) | 2.283891 / 1.504120 (0.779771) | 2.118172 / 1.541195 (0.576977) | 2.192628 / 1.468490 (0.724138) | 0.492026 / 4.584777 (-4.092751) | 3.692126 / 3.745712 (-0.053587) | 3.269827 / 5.269862 (-2.000035) | 2.028948 / 4.565676 (-2.536728) | 0.057932 / 0.424275 (-0.366344) | 0.007301 / 0.007607 (-0.000306) | 0.508411 / 0.226044 (0.282367) | 5.072803 / 2.268929 (2.803875) | 2.756532 / 55.444624 (-52.688092) | 2.432192 / 6.876477 (-4.444285) | 2.654864 / 2.142072 (0.512791) | 0.589458 / 4.805227 (-4.215769) | 0.133924 / 6.500664 (-6.366740) | 0.060764 / 0.075469 (-0.014705) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.350737 / 1.841788 (-0.491051) | 20.265217 / 8.074308 (12.190909) | 14.969039 / 10.191392 (4.777647) | 0.164226 / 0.680424 (-0.516198) | 0.020090 / 0.534201 (-0.514111) | 0.397010 / 0.579283 (-0.182273) | 0.412927 / 0.434364 (-0.021437) | 0.473931 / 0.540337 (-0.066406) | 0.653462 / 1.386936 (-0.733474) |

</details>
</details>

![](https://cml.dev/watermark.png#00cb5cc57337cdff338d7a54396bf25c5c5abd67 "CML watermark")