Allow direct cast from binary to Audio/Image
State: closed | Author: mariosasko (COLLABORATOR)

To address https://github.com/huggingface/datasets/discussions/5593.


Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008337 / 0.011353 (-0.003016) | 0.005588 / 0.011008 (-0.005421) | 0.110259 / 0.038508 (0.071751) | 0.038928 / 0.023109 (0.015819) | 0.350441 / 0.275898 (0.074543) | 0.378473 / 0.323480 (0.054993) | 0.006369 / 0.007986 (-0.001616) | 0.005730 / 0.004328 (0.001401) | 0.083042 / 0.004250 (0.078792) | 0.048686 / 0.037052 (0.011634) | 0.367561 / 0.258489 (0.109072) | 0.398073 / 0.293841 (0.104232) | 0.043247 / 0.128546 (-0.085299) | 0.013862 / 0.075646 (-0.061785) | 0.386745 / 0.419271 (-0.032527) | 0.060107 / 0.043533 (0.016574) | 0.345450 / 0.255139 (0.090311) | 0.371269 / 0.283200 (0.088069) | 0.117508 / 0.141683 (-0.024175) | 1.689345 / 1.452155 (0.237191) | 1.777119 / 1.492716 (0.284402) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.248248 / 0.018006 (0.230242) | 0.505200 / 0.000490 (0.504710) | 0.015354 / 0.000200 (0.015155) | 0.000794 / 0.000054 (0.000740) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030179 / 0.037411 (-0.007232) | 0.118583 / 0.014526 (0.104057) | 0.131546 / 0.176557 (-0.045010) | 0.196173 / 0.737135 (-0.540962) | 0.140532 / 0.296338 (-0.155807) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.470733 / 0.215209 (0.255524) | 4.758868 / 2.077655 (2.681213) | 2.246731 / 1.504120 (0.742611) | 1.995232 / 1.541195 (0.454037) | 2.057596 / 1.468490 (0.589106) | 0.819227 / 4.584777 (-3.765550) | 4.472093 / 3.745712 (0.726381) | 2.428154 / 5.269862 (-2.841708) | 1.748023 / 4.565676 (-2.817654) | 0.101965 / 0.424275 (-0.322310) | 0.014706 / 0.007607 (0.007098) | 0.600593 / 0.226044 (0.374548) | 5.869565 / 2.268929 (3.600637) | 2.764890 / 55.444624 (-52.679735) | 2.332112 / 6.876477 (-4.544364) | 2.486190 / 2.142072 (0.344118) | 0.979123 / 4.805227 (-3.826104) | 0.199543 / 6.500664 (-6.301121) | 0.075906 / 0.075469 (0.000436) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.397694 / 1.841788 (-0.444094) | 16.910500 / 8.074308 (8.836192) | 16.174131 / 10.191392 (5.982739) | 0.173975 / 0.680424 (-0.506449) | 0.021403 / 0.534201 (-0.512798) | 0.496187 / 0.579283 (-0.083096) | 0.487369 / 0.434364 (0.053005) | 0.565924 / 0.540337 (0.025587) | 0.684965 / 1.386936 (-0.701971) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008253 / 0.011353 (-0.003100) | 0.005745 / 0.011008 (-0.005263) | 0.085848 / 0.038508 (0.047340) | 0.038753 / 0.023109 (0.015644) | 0.401278 / 0.275898 (0.125379) | 0.433132 / 0.323480 (0.109652) | 0.006112 / 0.007986 (-0.001874) | 0.005973 / 0.004328 (0.001644) | 0.085339 / 0.004250 (0.081088) | 0.053297 / 0.037052 (0.016244) | 0.400265 / 0.258489 (0.141776) | 0.455155 / 0.293841 (0.161314) | 0.043116 / 0.128546 (-0.085430) | 0.013957 / 0.075646 (-0.061689) | 0.099507 / 0.419271 (-0.319764) | 0.058858 / 0.043533 (0.015325) | 0.398030 / 0.255139 (0.142891) | 0.418171 / 0.283200 (0.134971) | 0.114392 / 0.141683 (-0.027291) | 1.683102 / 1.452155 (0.230947) | 1.801427 / 1.492716 (0.308711) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242271 / 0.018006 (0.224265) | 0.494920 / 0.000490 (0.494430) | 0.007328 / 0.000200 (0.007128) | 0.000144 / 0.000054 (0.000090) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034061 / 0.037411 (-0.003351) | 0.146417 / 0.014526 (0.131891) | 0.161079 / 0.176557 (-0.015477) | 0.213999 / 0.737135 (-0.523137) | 0.166704 / 0.296338 (-0.129634) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.491214 / 0.215209 (0.276005) | 4.846946 / 2.077655 (2.769291) | 2.352595 / 1.504120 (0.848475) | 2.114055 / 1.541195 (0.572860) | 2.213537 / 1.468490 (0.745047) | 0.799625 / 4.584777 (-3.785152) | 4.440519 / 3.745712 (0.694807) | 4.476103 / 5.269862 (-0.793758) | 2.249384 / 4.565676 (-2.316292) | 0.098807 / 0.424275 (-0.325468) | 0.014463 / 0.007607 (0.006856) | 0.611793 / 0.226044 (0.385748) | 6.045710 / 2.268929 (3.776782) | 2.865957 / 55.444624 (-52.578667) | 2.454052 / 6.876477 (-4.422425) | 2.606153 / 2.142072 (0.464080) | 0.969057 / 4.805227 (-3.836170) | 0.198499 / 6.500664 (-6.302166) | 0.077012 / 0.075469 (0.001543) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.497020 / 1.841788 (-0.344767) | 17.834277 / 8.074308 (9.759969) | 16.413792 / 10.191392 (6.222400) | 0.201979 / 0.680424 (-0.478445) | 0.020627 / 0.534201 (-0.513574) | 0.499767 / 0.579283 (-0.079516) | 0.496982 / 0.434364 (0.062618) | 0.579554 / 0.540337 (0.039216) | 0.693287 / 1.386936 (-0.693649) |

</details>
</details>

![](https://cml.dev/watermark.png#a1a3fee942ae159ff6cfe6a23b343605e7e12f55 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007461 / 0.011353 (-0.003892) | 0.005341 / 0.011008 (-0.005668) | 0.099252 / 0.038508 (0.060744) | 0.034723 / 0.023109 (0.011614) | 0.300980 / 0.275898 (0.025082) | 0.353860 / 0.323480 (0.030380) | 0.006100 / 0.007986 (-0.001885) | 0.004149 / 0.004328 (-0.000180) | 0.074765 / 0.004250 (0.070514) | 0.052226 / 0.037052 (0.015174) | 0.305098 / 0.258489 (0.046609) | 0.357445 / 0.293841 (0.063604) | 0.036129 / 0.128546 (-0.092417) | 0.012482 / 0.075646 (-0.063165) | 0.333321 / 0.419271 (-0.085951) | 0.050489 / 0.043533 (0.006956) | 0.294728 / 0.255139 (0.039589) | 0.322722 / 0.283200 (0.039523) | 0.101226 / 0.141683 (-0.040456) | 1.436787 / 1.452155 (-0.015367) | 1.515784 / 1.492716 (0.023068) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.291836 / 0.018006 (0.273830) | 0.550735 / 0.000490 (0.550245) | 0.003828 / 0.000200 (0.003628) | 0.000113 / 0.000054 (0.000058) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028490 / 0.037411 (-0.008922) | 0.109543 / 0.014526 (0.095017) | 0.119451 / 0.176557 (-0.057105) | 0.176721 / 0.737135 (-0.560415) | 0.126711 / 0.296338 (-0.169628) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418863 / 0.215209 (0.203654) | 4.179167 / 2.077655 (2.101512) | 1.965126 / 1.504120 (0.461006) | 1.775544 / 1.541195 (0.234349) | 1.882667 / 1.468490 (0.414177) | 0.709201 / 4.584777 (-3.875576) | 3.754780 / 3.745712 (0.009068) | 2.175324 / 5.269862 (-3.094538) | 1.477454 / 4.565676 (-3.088223) | 0.085527 / 0.424275 (-0.338748) | 0.012685 / 0.007607 (0.005078) | 0.514276 / 0.226044 (0.288231) | 5.140518 / 2.268929 (2.871589) | 2.436011 / 55.444624 (-53.008614) | 2.114355 / 6.876477 (-4.762122) | 2.278893 / 2.142072 (0.136821) | 0.847825 / 4.805227 (-3.957402) | 0.169579 / 6.500664 (-6.331086) | 0.065306 / 0.075469 (-0.010163) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.190376 / 1.841788 (-0.651411) | 14.756581 / 8.074308 (6.682272) | 14.622610 / 10.191392 (4.431218) | 0.168186 / 0.680424 (-0.512238) | 0.017527 / 0.534201 (-0.516674) | 0.427808 / 0.579283 (-0.151475) | 0.437278 / 0.434364 (0.002914) | 0.509242 / 0.540337 (-0.031095) | 0.602500 / 1.386936 (-0.784436) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007331 / 0.011353 (-0.004022) | 0.005703 / 0.011008 (-0.005305) | 0.074992 / 0.038508 (0.036484) | 0.034069 / 0.023109 (0.010960) | 0.343513 / 0.275898 (0.067615) | 0.369061 / 0.323480 (0.045582) | 0.006034 / 0.007986 (-0.001951) | 0.004344 / 0.004328 (0.000016) | 0.074678 / 0.004250 (0.070428) | 0.052262 / 0.037052 (0.015210) | 0.364758 / 0.258489 (0.106269) | 0.401130 / 0.293841 (0.107289) | 0.037635 / 0.128546 (-0.090912) | 0.012599 / 0.075646 (-0.063047) | 0.086935 / 0.419271 (-0.332337) | 0.058161 / 0.043533 (0.014628) | 0.338727 / 0.255139 (0.083589) | 0.355957 / 0.283200 (0.072757) | 0.111607 / 0.141683 (-0.030076) | 1.454357 / 1.452155 (0.002202) | 1.591529 / 1.492716 (0.098813) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.284379 / 0.018006 (0.266373) | 0.550720 / 0.000490 (0.550230) | 0.002868 / 0.000200 (0.002668) | 0.000102 / 0.000054 (0.000048) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028876 / 0.037411 (-0.008535) | 0.110892 / 0.014526 (0.096366) | 0.122519 / 0.176557 (-0.054038) | 0.169774 / 0.737135 (-0.567361) | 0.129381 / 0.296338 (-0.166957) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429181 / 0.215209 (0.213972) | 4.251016 / 2.077655 (2.173361) | 2.056778 / 1.504120 (0.552658) | 1.860458 / 1.541195 (0.319264) | 1.958923 / 1.468490 (0.490432) | 0.712667 / 4.584777 (-3.872110) | 3.856910 / 3.745712 (0.111198) | 3.374535 / 5.269862 (-1.895327) | 1.846744 / 4.565676 (-2.718932) | 0.087238 / 0.424275 (-0.337037) | 0.012718 / 0.007607 (0.005111) | 0.524654 / 0.226044 (0.298609) | 5.209756 / 2.268929 (2.940827) | 2.494882 / 55.444624 (-52.949743) | 2.201150 / 6.876477 (-4.675327) | 2.274189 / 2.142072 (0.132117) | 0.844728 / 4.805227 (-3.960499) | 0.167467 / 6.500664 (-6.333197) | 0.064018 / 0.075469 (-0.011451) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.273284 / 1.841788 (-0.568503) | 15.104413 / 8.074308 (7.030105) | 15.134025 / 10.191392 (4.942633) | 0.147568 / 0.680424 (-0.532856) | 0.017429 / 0.534201 (-0.516772) | 0.422052 / 0.579283 (-0.157231) | 0.425786 / 0.434364 (-0.008578) | 0.491753 / 0.540337 (-0.048584) | 0.585091 / 1.386936 (-0.801845) |

</details>
</details>

![](https://cml.dev/watermark.png#f3d26e74898e0a9dc0d78490104e2e173269ef5b "CML watermark")