Support streaming datasets with numpy.load
Opened by albertvillanova (closed)

Support streaming datasets with `numpy.load`.

See: https://huggingface.co/datasets/qgallouedec/gia_dataset/discussions/1

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006607 / 0.011353 (-0.004746) | 0.004610 / 0.011008 (-0.006398) | 0.100673 / 0.038508 (0.062165) | 0.027739 / 0.023109 (0.004630) | 0.326290 / 0.275898 (0.050392) | 0.344296 / 0.323480 (0.020816) | 0.005021 / 0.007986 (-0.002964) | 0.003327 / 0.004328 (-0.001002) | 0.077779 / 0.004250 (0.073529) | 0.040237 / 0.037052 (0.003185) | 0.308992 / 0.258489 (0.050503) | 0.355017 / 0.293841 (0.061176) | 0.031203 / 0.128546 (-0.097343) | 0.011749 / 0.075646 (-0.063898) | 0.327431 / 0.419271 (-0.091840) | 0.043033 / 0.043533 (-0.000500) | 0.309713 / 0.255139 (0.054574) | 0.336550 / 0.283200 (0.053351) | 0.084891 / 0.141683 (-0.056792) | 1.555641 / 1.452155 (0.103487) | 1.613214 / 1.492716 (0.120497) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.216269 / 0.018006 (0.198262) | 0.422066 / 0.000490 (0.421576) | 0.004055 / 0.000200 (0.003855) | 0.000073 / 0.000054 (0.000019) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023759 / 0.037411 (-0.013652) | 0.096937 / 0.014526 (0.082411) | 0.105312 / 0.176557 (-0.071244) | 0.167840 / 0.737135 (-0.569295) | 0.107998 / 0.296338 (-0.188340) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.458315 / 0.215209 (0.243106) | 4.584803 / 2.077655 (2.507148) | 2.193641 / 1.504120 (0.689521) | 1.981494 / 1.541195 (0.440299) | 2.020358 / 1.468490 (0.551868) | 0.696763 / 4.584777 (-3.888014) | 3.388432 / 3.745712 (-0.357280) | 3.335038 / 5.269862 (-1.934823) | 1.648551 / 4.565676 (-2.917126) | 0.083753 / 0.424275 (-0.340522) | 0.012855 / 0.007607 (0.005248) | 0.562331 / 0.226044 (0.336286) | 5.649259 / 2.268929 (3.380330) | 2.680309 / 55.444624 (-52.764315) | 2.319297 / 6.876477 (-4.557180) | 2.444016 / 2.142072 (0.301943) | 0.809821 / 4.805227 (-3.995407) | 0.152855 / 6.500664 (-6.347809) | 0.067756 / 0.075469 (-0.007713) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.213318 / 1.841788 (-0.628470) | 13.887822 / 8.074308 (5.813514) | 14.276325 / 10.191392 (4.084933) | 0.156227 / 0.680424 (-0.524197) | 0.016377 / 0.534201 (-0.517824) | 0.377080 / 0.579283 (-0.202203) | 0.386561 / 0.434364 (-0.047803) | 0.435631 / 0.540337 (-0.104707) | 0.520863 / 1.386936 (-0.866073) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006740 / 0.011353 (-0.004613) | 0.004704 / 0.011008 (-0.006304) | 0.076840 / 0.038508 (0.038331) | 0.027519 / 0.023109 (0.004409) | 0.343219 / 0.275898 (0.067321) | 0.376810 / 0.323480 (0.053330) | 0.005048 / 0.007986 (-0.002938) | 0.003356 / 0.004328 (-0.000972) | 0.077098 / 0.004250 (0.072848) | 0.038601 / 0.037052 (0.001548) | 0.345723 / 0.258489 (0.087233) | 0.388635 / 0.293841 (0.094794) | 0.033612 / 0.128546 (-0.094934) | 0.011689 / 0.075646 (-0.063957) | 0.086446 / 0.419271 (-0.332825) | 0.044390 / 0.043533 (0.000857) | 0.343763 / 0.255139 (0.088624) | 0.368591 / 0.283200 (0.085392) | 0.091605 / 0.141683 (-0.050078) | 1.478615 / 1.452155 (0.026461) | 1.580858 / 1.492716 (0.088142) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.223547 / 0.018006 (0.205541) | 0.411243 / 0.000490 (0.410753) | 0.000916 / 0.000200 (0.000716) | 0.000070 / 0.000054 (0.000016) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025223 / 0.037411 (-0.012189) | 0.100970 / 0.014526 (0.086445) | 0.108178 / 0.176557 (-0.068378) | 0.156827 / 0.737135 (-0.580308) | 0.111431 / 0.296338 (-0.184907) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.434168 / 0.215209 (0.218959) | 4.361874 / 2.077655 (2.284219) | 2.060735 / 1.504120 (0.556615) | 1.861100 / 1.541195 (0.319906) | 1.920692 / 1.468490 (0.452202) | 0.697909 / 4.584777 (-3.886868) | 3.477036 / 3.745712 (-0.268676) | 3.002469 / 5.269862 (-2.267392) | 1.449325 / 4.565676 (-3.116351) | 0.083034 / 0.424275 (-0.341241) | 0.012805 / 0.007607 (0.005198) | 0.531391 / 0.226044 (0.305347) | 5.323015 / 2.268929 (3.054086) | 2.488605 / 55.444624 (-52.956020) | 2.158254 / 6.876477 (-4.718222) | 2.189633 / 2.142072 (0.047560) | 0.805972 / 4.805227 (-3.999256) | 0.153105 / 6.500664 (-6.347559) | 0.068909 / 0.075469 (-0.006561) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.276851 / 1.841788 (-0.564937) | 14.431510 / 8.074308 (6.357202) | 14.544788 / 10.191392 (4.353396) | 0.146589 / 0.680424 (-0.533835) | 0.016890 / 0.534201 (-0.517311) | 0.379897 / 0.579283 (-0.199387) | 0.389153 / 0.434364 (-0.045211) | 0.440097 / 0.540337 (-0.100241) | 0.524191 / 1.386936 (-0.862745) |

</details>
</details>

![](https://cml.dev/watermark.png#e1af108015e43f9df8734a1faeeaeb9eafce3971 "CML watermark")