Flatten dataset on the fly in `save_to_disk`
Opened by mariosasko (closed)

Flatten a dataset on the fly in `save_to_disk` instead of doing it with `flatten_indices`  to avoid creating an additional cache file.

(this is one of the sub-tasks in https://github.com/huggingface/datasets/issues/5507)

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009866 / 0.011353 (-0.001487) | 0.005334 / 0.011008 (-0.005675) | 0.101771 / 0.038508 (0.063263) | 0.037722 / 0.023109 (0.014613) | 0.301026 / 0.275898 (0.025128) | 0.336618 / 0.323480 (0.013138) | 0.008679 / 0.007986 (0.000693) | 0.005640 / 0.004328 (0.001312) | 0.077076 / 0.004250 (0.072825) | 0.045068 / 0.037052 (0.008016) | 0.302570 / 0.258489 (0.044081) | 0.359093 / 0.293841 (0.065252) | 0.038865 / 0.128546 (-0.089681) | 0.012318 / 0.075646 (-0.063328) | 0.334819 / 0.419271 (-0.084452) | 0.047980 / 0.043533 (0.004447) | 0.296999 / 0.255139 (0.041860) | 0.318855 / 0.283200 (0.035656) | 0.110633 / 0.141683 (-0.031050) | 1.464326 / 1.452155 (0.012172) | 1.537386 / 1.492716 (0.044670) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.282906 / 0.018006 (0.264900) | 0.498418 / 0.000490 (0.497928) | 0.001507 / 0.000200 (0.001307) | 0.000087 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029948 / 0.037411 (-0.007463) | 0.114385 / 0.014526 (0.099859) | 0.125783 / 0.176557 (-0.050774) | 0.193458 / 0.737135 (-0.543678) | 0.129725 / 0.296338 (-0.166614) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.403822 / 0.215209 (0.188613) | 4.034180 / 2.077655 (1.956525) | 1.768206 / 1.504120 (0.264086) | 1.579267 / 1.541195 (0.038072) | 1.725077 / 1.468490 (0.256587) | 0.698743 / 4.584777 (-3.886034) | 3.723481 / 3.745712 (-0.022231) | 2.302374 / 5.269862 (-2.967488) | 1.497954 / 4.565676 (-3.067723) | 0.087360 / 0.424275 (-0.336915) | 0.012453 / 0.007607 (0.004846) | 0.523374 / 0.226044 (0.297329) | 5.244962 / 2.268929 (2.976033) | 2.272874 / 55.444624 (-53.171750) | 1.935570 / 6.876477 (-4.940907) | 2.043151 / 2.142072 (-0.098921) | 0.866298 / 4.805227 (-3.938929) | 0.169376 / 6.500664 (-6.331288) | 0.064578 / 0.075469 (-0.010892) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.217372 / 1.841788 (-0.624416) | 15.896050 / 8.074308 (7.821742) | 15.165190 / 10.191392 (4.973798) | 0.171168 / 0.680424 (-0.509256) | 0.029770 / 0.534201 (-0.504431) | 0.449030 / 0.579283 (-0.130253) | 0.454704 / 0.434364 (0.020340) | 0.550689 / 0.540337 (0.010351) | 0.651182 / 1.386936 (-0.735754) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008072 / 0.011353 (-0.003281) | 0.005533 / 0.011008 (-0.005475) | 0.076343 / 0.038508 (0.037835) | 0.037997 / 0.023109 (0.014888) | 0.350465 / 0.275898 (0.074567) | 0.391168 / 0.323480 (0.067688) | 0.006475 / 0.007986 (-0.001511) | 0.004299 / 0.004328 (-0.000029) | 0.074867 / 0.004250 (0.070617) | 0.055256 / 0.037052 (0.018204) | 0.363919 / 0.258489 (0.105430) | 0.396521 / 0.293841 (0.102680) | 0.037746 / 0.128546 (-0.090801) | 0.012556 / 0.075646 (-0.063091) | 0.087974 / 0.419271 (-0.331297) | 0.050850 / 0.043533 (0.007317) | 0.345857 / 0.255139 (0.090718) | 0.361019 / 0.283200 (0.077820) | 0.111007 / 0.141683 (-0.030676) | 1.444014 / 1.452155 (-0.008140) | 1.533154 / 1.492716 (0.040438) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.332114 / 0.018006 (0.314108) | 0.517232 / 0.000490 (0.516742) | 0.004459 / 0.000200 (0.004259) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033147 / 0.037411 (-0.004264) | 0.119983 / 0.014526 (0.105457) | 0.125970 / 0.176557 (-0.050586) | 0.196375 / 0.737135 (-0.540760) | 0.133849 / 0.296338 (-0.162489) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429477 / 0.215209 (0.214267) | 4.263750 / 2.077655 (2.186096) | 2.079409 / 1.504120 (0.575289) | 1.899831 / 1.541195 (0.358636) | 2.048472 / 1.468490 (0.579982) | 0.720945 / 4.584777 (-3.863832) | 3.813195 / 3.745712 (0.067483) | 2.250353 / 5.269862 (-3.019508) | 1.401496 / 4.565676 (-3.164181) | 0.090052 / 0.424275 (-0.334223) | 0.012552 / 0.007607 (0.004945) | 0.536839 / 0.226044 (0.310794) | 5.361089 / 2.268929 (3.092161) | 2.559710 / 55.444624 (-52.884914) | 2.226963 / 6.876477 (-4.649513) | 2.341898 / 2.142072 (0.199825) | 0.872115 / 4.805227 (-3.933112) | 0.173776 / 6.500664 (-6.326888) | 0.068567 / 0.075469 (-0.006902) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.294583 / 1.841788 (-0.547205) | 16.624099 / 8.074308 (8.549791) | 13.698509 / 10.191392 (3.507117) | 0.161917 / 0.680424 (-0.518506) | 0.017744 / 0.534201 (-0.516457) | 0.428547 / 0.579283 (-0.150736) | 0.424687 / 0.434364 (-0.009677) | 0.525812 / 0.540337 (-0.014525) | 0.629075 / 1.386936 (-0.757861) |

</details>
</details>

![](https://cml.dev/watermark.png#33e4d6af919db17bf9a1eac544a0501b5972393b "CML watermark")


_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008667 / 0.011353 (-0.002686) | 0.004921 / 0.011008 (-0.006087) | 0.098352 / 0.038508 (0.059844) | 0.033983 / 0.023109 (0.010873) | 0.291640 / 0.275898 (0.015742) | 0.323388 / 0.323480 (-0.000092) | 0.007943 / 0.007986 (-0.000043) | 0.003922 / 0.004328 (-0.000407) | 0.075861 / 0.004250 (0.071610) | 0.042606 / 0.037052 (0.005554) | 0.298571 / 0.258489 (0.040081) | 0.345496 / 0.293841 (0.051655) | 0.037443 / 0.128546 (-0.091103) | 0.012114 / 0.075646 (-0.063532) | 0.333269 / 0.419271 (-0.086003) | 0.047762 / 0.043533 (0.004229) | 0.295452 / 0.255139 (0.040313) | 0.319641 / 0.283200 (0.036441) | 0.101083 / 0.141683 (-0.040600) | 1.432179 / 1.452155 (-0.019976) | 1.523976 / 1.492716 (0.031260) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.241327 / 0.018006 (0.223321) | 0.538315 / 0.000490 (0.537825) | 0.003479 / 0.000200 (0.003279) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025857 / 0.037411 (-0.011554) | 0.104833 / 0.014526 (0.090307) | 0.116826 / 0.176557 (-0.059730) | 0.183460 / 0.737135 (-0.553675) | 0.119595 / 0.296338 (-0.176743) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.397533 / 0.215209 (0.182324) | 3.968664 / 2.077655 (1.891010) | 1.774025 / 1.504120 (0.269905) | 1.577424 / 1.541195 (0.036229) | 1.623049 / 1.468490 (0.154559) | 0.701008 / 4.584777 (-3.883769) | 3.753278 / 3.745712 (0.007565) | 2.078313 / 5.269862 (-3.191549) | 1.335639 / 4.565676 (-3.230037) | 0.085216 / 0.424275 (-0.339059) | 0.012087 / 0.007607 (0.004480) | 0.513219 / 0.226044 (0.287174) | 5.097693 / 2.268929 (2.828765) | 2.275030 / 55.444624 (-53.169594) | 1.928037 / 6.876477 (-4.948439) | 1.941216 / 2.142072 (-0.200856) | 0.856720 / 4.805227 (-3.948507) | 0.166723 / 6.500664 (-6.333941) | 0.062263 / 0.075469 (-0.013206) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.196054 / 1.841788 (-0.645734) | 14.190526 / 8.074308 (6.116218) | 14.053768 / 10.191392 (3.862376) | 0.179982 / 0.680424 (-0.500442) | 0.029024 / 0.534201 (-0.505177) | 0.440391 / 0.579283 (-0.138892) | 0.445627 / 0.434364 (0.011264) | 0.543098 / 0.540337 (0.002761) | 0.640577 / 1.386936 (-0.746359) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007008 / 0.011353 (-0.004345) | 0.005015 / 0.011008 (-0.005993) | 0.073783 / 0.038508 (0.035274) | 0.032401 / 0.023109 (0.009292) | 0.343382 / 0.275898 (0.067484) | 0.358317 / 0.323480 (0.034837) | 0.005548 / 0.007986 (-0.002437) | 0.005188 / 0.004328 (0.000859) | 0.072867 / 0.004250 (0.068617) | 0.048555 / 0.037052 (0.011502) | 0.334516 / 0.258489 (0.076027) | 0.390263 / 0.293841 (0.096422) | 0.036343 / 0.128546 (-0.092203) | 0.012243 / 0.075646 (-0.063404) | 0.087067 / 0.419271 (-0.332205) | 0.049025 / 0.043533 (0.005492) | 0.333977 / 0.255139 (0.078838) | 0.354427 / 0.283200 (0.071227) | 0.104771 / 0.141683 (-0.036912) | 1.434588 / 1.452155 (-0.017567) | 1.519788 / 1.492716 (0.027072) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.264002 / 0.018006 (0.245996) | 0.547902 / 0.000490 (0.547412) | 0.000461 / 0.000200 (0.000261) | 0.000062 / 0.000054 (0.000008) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028916 / 0.037411 (-0.008496) | 0.110267 / 0.014526 (0.095741) | 0.119190 / 0.176557 (-0.057367) | 0.188599 / 0.737135 (-0.548537) | 0.126948 / 0.296338 (-0.169391) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422777 / 0.215209 (0.207568) | 4.209813 / 2.077655 (2.132158) | 2.001360 / 1.504120 (0.497240) | 1.802651 / 1.541195 (0.261456) | 1.860357 / 1.468490 (0.391867) | 0.695006 / 4.584777 (-3.889771) | 3.741917 / 3.745712 (-0.003795) | 3.313071 / 5.269862 (-1.956791) | 1.726366 / 4.565676 (-2.839311) | 0.086185 / 0.424275 (-0.338090) | 0.012256 / 0.007607 (0.004649) | 0.536874 / 0.226044 (0.310830) | 5.253008 / 2.268929 (2.984079) | 2.457189 / 55.444624 (-52.987436) | 2.112199 / 6.876477 (-4.764278) | 2.117867 / 2.142072 (-0.024205) | 0.831914 / 4.805227 (-3.973314) | 0.168238 / 6.500664 (-6.332426) | 0.065075 / 0.075469 (-0.010394) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.280795 / 1.841788 (-0.560993) | 14.606608 / 8.074308 (6.532299) | 13.317597 / 10.191392 (3.126205) | 0.166590 / 0.680424 (-0.513834) | 0.017520 / 0.534201 (-0.516681) | 0.420978 / 0.579283 (-0.158305) | 0.415708 / 0.434364 (-0.018656) | 0.523619 / 0.540337 (-0.016718) | 0.625299 / 1.386936 (-0.761637) |

</details>
</details>

![](https://cml.dev/watermark.png#a2a83a8ea4b3a87a925ef44b787e87b59bf68225 "CML watermark")