Issue: Fix dump of bfloat16 torch tensor
State: closed
Author: lhoestq

close https://github.com/huggingface/datasets/issues/7000

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7002). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005321 / 0.011353 (-0.006032) | 0.003495 / 0.011008 (-0.007514) | 0.065577 / 0.038508 (0.027069) | 0.030876 / 0.023109 (0.007767) | 0.255216 / 0.275898 (-0.020682) | 0.265111 / 0.323480 (-0.058368) | 0.003149 / 0.007986 (-0.004837) | 0.004062 / 0.004328 (-0.000267) | 0.051142 / 0.004250 (0.046891) | 0.042460 / 0.037052 (0.005408) | 0.270692 / 0.258489 (0.012203) | 0.284957 / 0.293841 (-0.008884) | 0.030143 / 0.128546 (-0.098403) | 0.012148 / 0.075646 (-0.063498) | 0.203706 / 0.419271 (-0.215565) | 0.035948 / 0.043533 (-0.007584) | 0.251391 / 0.255139 (-0.003748) | 0.270908 / 0.283200 (-0.012292) | 0.018496 / 0.141683 (-0.123187) | 1.118567 / 1.452155 (-0.333587) | 1.157695 / 1.492716 (-0.335021) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.135649 / 0.018006 (0.117643) | 0.281489 / 0.000490 (0.281000) | 0.000244 / 0.000200 (0.000044) | 0.000042 / 0.000054 (-0.000012) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018700 / 0.037411 (-0.018711) | 0.062305 / 0.014526 (0.047779) | 0.074968 / 0.176557 (-0.101589) | 0.121490 / 0.737135 (-0.615645) | 0.075585 / 0.296338 (-0.220754) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.276929 / 0.215209 (0.061720) | 2.733543 / 2.077655 (0.655888) | 1.414585 / 1.504120 (-0.089535) | 1.301975 / 1.541195 (-0.239220) | 1.336698 / 1.468490 (-0.131792) | 0.720650 / 4.584777 (-3.864127) | 2.374796 / 3.745712 (-1.370917) | 2.866534 / 5.269862 (-2.403327) | 1.819607 / 4.565676 (-2.746069) | 0.077914 / 0.424275 (-0.346361) | 0.005146 / 0.007607 (-0.002461) | 0.331722 / 0.226044 (0.105678) | 3.290875 / 2.268929 (1.021946) | 1.799806 / 55.444624 (-53.644818) | 1.476816 / 6.876477 (-5.399660) | 1.511441 / 2.142072 (-0.630631) | 0.798043 / 4.805227 (-4.007185) | 0.134577 / 6.500664 (-6.366087) | 0.042055 / 0.075469 (-0.033415) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.967908 / 1.841788 (-0.873880) | 11.215688 / 8.074308 (3.141380) | 9.486403 / 10.191392 (-0.704989) | 0.141864 / 0.680424 (-0.538560) | 0.013462 / 0.534201 (-0.520739) | 0.302601 / 0.579283 (-0.276682) | 0.266870 / 0.434364 (-0.167494) | 0.336963 / 0.540337 (-0.203375) | 0.425374 / 1.386936 (-0.961562) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005549 / 0.011353 (-0.005803) | 0.003464 / 0.011008 (-0.007544) | 0.051421 / 0.038508 (0.012913) | 0.032320 / 0.023109 (0.009211) | 0.269591 / 0.275898 (-0.006307) | 0.292015 / 0.323480 (-0.031465) | 0.004351 / 0.007986 (-0.003634) | 0.002772 / 0.004328 (-0.001556) | 0.048836 / 0.004250 (0.044586) | 0.039501 / 0.037052 (0.002449) | 0.282419 / 0.258489 (0.023930) | 0.312289 / 0.293841 (0.018448) | 0.031788 / 0.128546 (-0.096759) | 0.012074 / 0.075646 (-0.063572) | 0.060457 / 0.419271 (-0.358814) | 0.033106 / 0.043533 (-0.010427) | 0.270323 / 0.255139 (0.015184) | 0.287855 / 0.283200 (0.004655) | 0.017865 / 0.141683 (-0.123818) | 1.130406 / 1.452155 (-0.321749) | 1.178679 / 1.492716 (-0.314038) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.093606 / 0.018006 (0.075600) | 0.297328 / 0.000490 (0.296838) | 0.000211 / 0.000200 (0.000011) | 0.000043 / 0.000054 (-0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022498 / 0.037411 (-0.014913) | 0.076927 / 0.014526 (0.062401) | 0.088013 / 0.176557 (-0.088544) | 0.127279 / 0.737135 (-0.609857) | 0.089424 / 0.296338 (-0.206914) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296441 / 0.215209 (0.081232) | 2.913051 / 2.077655 (0.835396) | 1.581816 / 1.504120 (0.077696) | 1.451575 / 1.541195 (-0.089620) | 1.458968 / 1.468490 (-0.009522) | 0.727191 / 4.584777 (-3.857586) | 0.954607 / 3.745712 (-2.791106) | 2.824357 / 5.269862 (-2.445505) | 1.886779 / 4.565676 (-2.678898) | 0.079397 / 0.424275 (-0.344878) | 0.005566 / 0.007607 (-0.002041) | 0.351655 / 0.226044 (0.125611) | 3.395790 / 2.268929 (1.126862) | 1.886238 / 55.444624 (-53.558387) | 1.615413 / 6.876477 (-5.261064) | 1.723922 / 2.142072 (-0.418150) | 0.807858 / 4.805227 (-3.997369) | 0.132998 / 6.500664 (-6.367667) | 0.040396 / 0.075469 (-0.035073) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.008527 / 1.841788 (-0.833261) | 11.736104 / 8.074308 (3.661796) | 10.283367 / 10.191392 (0.091975) | 0.141386 / 0.680424 (-0.539038) | 0.015722 / 0.534201 (-0.518479) | 0.301785 / 0.579283 (-0.277498) | 0.123073 / 0.434364 (-0.311291) | 0.340478 / 0.540337 (-0.199859) | 0.462936 / 1.386936 (-0.924000) |

</details>
</details>

![](https://cml.dev/watermark.png#bfb0a414d68e945addf95a9419a8314c372e19ba "CML watermark")