[WebDataset] Audio support and bug fixes
State: closed | Author: lhoestq (MEMBER)

- Add audio support
- Fix an issue where user-provided features with additional fields are not taken into account

Close https://github.com/huggingface/datasets/issues/6569

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6573). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005421 / 0.011353 (-0.005932) | 0.003915 / 0.011008 (-0.007094) | 0.065109 / 0.038508 (0.026601) | 0.031274 / 0.023109 (0.008165) | 0.248702 / 0.275898 (-0.027196) | 0.275688 / 0.323480 (-0.047792) | 0.003007 / 0.007986 (-0.004978) | 0.002942 / 0.004328 (-0.001387) | 0.050928 / 0.004250 (0.046678) | 0.043751 / 0.037052 (0.006699) | 0.263860 / 0.258489 (0.005371) | 0.291499 / 0.293841 (-0.002342) | 0.028268 / 0.128546 (-0.100278) | 0.011467 / 0.075646 (-0.064180) | 0.210531 / 0.419271 (-0.208740) | 0.036302 / 0.043533 (-0.007231) | 0.251565 / 0.255139 (-0.003574) | 0.272001 / 0.283200 (-0.011199) | 0.020370 / 0.141683 (-0.121313) | 1.175493 / 1.452155 (-0.276662) | 1.229167 / 1.492716 (-0.263550) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.095713 / 0.018006 (0.077707) | 0.308912 / 0.000490 (0.308422) | 0.000231 / 0.000200 (0.000031) | 0.000057 / 0.000054 (0.000002) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019080 / 0.037411 (-0.018332) | 0.062043 / 0.014526 (0.047517) | 0.075642 / 0.176557 (-0.100915) | 0.122789 / 0.737135 (-0.614347) | 0.077507 / 0.296338 (-0.218831) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.279929 / 0.215209 (0.064720) | 2.773336 / 2.077655 (0.695682) | 1.481740 / 1.504120 (-0.022379) | 1.357207 / 1.541195 (-0.183987) | 1.414314 / 1.468490 (-0.054176) | 0.573776 / 4.584777 (-4.011000) | 2.399273 / 3.745712 (-1.346439) | 2.918885 / 5.269862 (-2.350977) | 1.798867 / 4.565676 (-2.766809) | 0.064352 / 0.424275 (-0.359923) | 0.005164 / 0.007607 (-0.002443) | 0.337141 / 0.226044 (0.111097) | 3.402291 / 2.268929 (1.133362) | 1.854308 / 55.444624 (-53.590317) | 1.555789 / 6.876477 (-5.320687) | 1.625873 / 2.142072 (-0.516199) | 0.658589 / 4.805227 (-4.146638) | 0.122273 / 6.500664 (-6.378391) | 0.043910 / 0.075469 (-0.031560) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.933127 / 1.841788 (-0.908660) | 12.436657 / 8.074308 (4.362348) | 10.891750 / 10.191392 (0.700358) | 0.143236 / 0.680424 (-0.537187) | 0.014636 / 0.534201 (-0.519565) | 0.290375 / 0.579283 (-0.288908) | 0.275473 / 0.434364 (-0.158891) | 0.327007 / 0.540337 (-0.213331) | 0.425888 / 1.386936 (-0.961048) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005820 / 0.011353 (-0.005533) | 0.003684 / 0.011008 (-0.007324) | 0.050234 / 0.038508 (0.011726) | 0.030744 / 0.023109 (0.007634) | 0.279560 / 0.275898 (0.003662) | 0.305829 / 0.323480 (-0.017651) | 0.004053 / 0.007986 (-0.003933) | 0.002743 / 0.004328 (-0.001585) | 0.051087 / 0.004250 (0.046836) | 0.047601 / 0.037052 (0.010549) | 0.290441 / 0.258489 (0.031951) | 0.326719 / 0.293841 (0.032878) | 0.030245 / 0.128546 (-0.098301) | 0.011508 / 0.075646 (-0.064139) | 0.058436 / 0.419271 (-0.360835) | 0.059235 / 0.043533 (0.015702) | 0.278978 / 0.255139 (0.023839) | 0.298146 / 0.283200 (0.014946) | 0.020926 / 0.141683 (-0.120757) | 1.205608 / 1.452155 (-0.246547) | 1.224920 / 1.492716 (-0.267796) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.098384 / 0.018006 (0.080378) | 0.307975 / 0.000490 (0.307485) | 0.000233 / 0.000200 (0.000033) | 0.000054 / 0.000054 (-0.000000) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023012 / 0.037411 (-0.014399) | 0.077450 / 0.014526 (0.062924) | 0.089314 / 0.176557 (-0.087242) | 0.128610 / 0.737135 (-0.608526) | 0.091521 / 0.296338 (-0.204818) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.310620 / 0.215209 (0.095411) | 3.030374 / 2.077655 (0.952720) | 1.653468 / 1.504120 (0.149348) | 1.526860 / 1.541195 (-0.014334) | 1.605328 / 1.468490 (0.136838) | 0.585444 / 4.584777 (-3.999333) | 2.471700 / 3.745712 (-1.274012) | 2.791268 / 5.269862 (-2.478594) | 1.815965 / 4.565676 (-2.749712) | 0.064713 / 0.424275 (-0.359562) | 0.005095 / 0.007607 (-0.002512) | 0.364843 / 0.226044 (0.138799) | 3.601633 / 2.268929 (1.332705) | 2.022642 / 55.444624 (-53.421982) | 1.737164 / 6.876477 (-5.139312) | 1.923636 / 2.142072 (-0.218437) | 0.670673 / 4.805227 (-4.134554) | 0.121547 / 6.500664 (-6.379117) | 0.042880 / 0.075469 (-0.032589) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.987873 / 1.841788 (-0.853914) | 13.279323 / 8.074308 (5.205015) | 11.480806 / 10.191392 (1.289414) | 0.142118 / 0.680424 (-0.538306) | 0.016486 / 0.534201 (-0.517715) | 0.291617 / 0.579283 (-0.287667) | 0.284639 / 0.434364 (-0.149725) | 0.329596 / 0.540337 (-0.210742) | 0.430168 / 1.386936 (-0.956768) |

</details>
</details>

![](https://cml.dev/watermark.png#4a5b7d9562231c9fbb36e30c1cf0ac54133d1e77 "CML watermark")