Issue: Remove py.typed
State: closed
Author: mariosasko

Fix https://github.com/huggingface/datasets/issues/3841

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008283 / 0.011353 (-0.003070) | 0.004450 / 0.011008 (-0.006558) | 0.099773 / 0.038508 (0.061265) | 0.029068 / 0.023109 (0.005959) | 0.296799 / 0.275898 (0.020901) | 0.350946 / 0.323480 (0.027466) | 0.007331 / 0.007986 (-0.000655) | 0.004550 / 0.004328 (0.000222) | 0.077603 / 0.004250 (0.073352) | 0.034307 / 0.037052 (-0.002746) | 0.313174 / 0.258489 (0.054685) | 0.342270 / 0.293841 (0.048429) | 0.033463 / 0.128546 (-0.095083) | 0.011421 / 0.075646 (-0.064225) | 0.317188 / 0.419271 (-0.102083) | 0.040985 / 0.043533 (-0.002548) | 0.300800 / 0.255139 (0.045661) | 0.360171 / 0.283200 (0.076972) | 0.086702 / 0.141683 (-0.054981) | 1.474679 / 1.452155 (0.022525) | 1.518319 / 1.492716 (0.025603) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198059 / 0.018006 (0.180052) | 0.403502 / 0.000490 (0.403012) | 0.002663 / 0.000200 (0.002463) | 0.000218 / 0.000054 (0.000164) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022946 / 0.037411 (-0.014465) | 0.096466 / 0.014526 (0.081940) | 0.104092 / 0.176557 (-0.072465) | 0.138499 / 0.737135 (-0.598636) | 0.106941 / 0.296338 (-0.189397) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416000 / 0.215209 (0.200791) | 4.153120 / 2.077655 (2.075465) | 1.843957 / 1.504120 (0.339837) | 1.650391 / 1.541195 (0.109197) | 1.684765 / 1.468490 (0.216275) | 0.688917 / 4.584777 (-3.895860) | 3.442797 / 3.745712 (-0.302916) | 1.834685 / 5.269862 (-3.435176) | 1.148046 / 4.565676 (-3.417631) | 0.082299 / 0.424275 (-0.341976) | 0.012399 / 0.007607 (0.004792) | 0.521099 / 0.226044 (0.295054) | 5.223695 / 2.268929 (2.954767) | 2.270970 / 55.444624 (-53.173654) | 1.921321 / 6.876477 (-4.955156) | 1.954675 / 2.142072 (-0.187398) | 0.809383 / 4.805227 (-3.995845) | 0.148562 / 6.500664 (-6.352102) | 0.064764 / 0.075469 (-0.010705) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.212687 / 1.841788 (-0.629101) | 13.491641 / 8.074308 (5.417333) | 12.972926 / 10.191392 (2.781534) | 0.137036 / 0.680424 (-0.543388) | 0.028591 / 0.534201 (-0.505610) | 0.391980 / 0.579283 (-0.187303) | 0.394474 / 0.434364 (-0.039889) | 0.456582 / 0.540337 (-0.083755) | 0.535984 / 1.386936 (-0.850952) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006419 / 0.011353 (-0.004934) | 0.004295 / 0.011008 (-0.006713) | 0.077702 / 0.038508 (0.039194) | 0.027368 / 0.023109 (0.004259) | 0.336713 / 0.275898 (0.060815) | 0.370074 / 0.323480 (0.046594) | 0.004657 / 0.007986 (-0.003328) | 0.003308 / 0.004328 (-0.001021) | 0.075747 / 0.004250 (0.071496) | 0.037323 / 0.037052 (0.000271) | 0.342382 / 0.258489 (0.083893) | 0.381109 / 0.293841 (0.087269) | 0.031804 / 0.128546 (-0.096742) | 0.011761 / 0.075646 (-0.063885) | 0.086818 / 0.419271 (-0.332454) | 0.042058 / 0.043533 (-0.001475) | 0.346295 / 0.255139 (0.091156) | 0.366857 / 0.283200 (0.083658) | 0.088666 / 0.141683 (-0.053016) | 1.533711 / 1.452155 (0.081556) | 1.537422 / 1.492716 (0.044705) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220416 / 0.018006 (0.202410) | 0.387393 / 0.000490 (0.386903) | 0.003739 / 0.000200 (0.003539) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024083 / 0.037411 (-0.013329) | 0.098036 / 0.014526 (0.083510) | 0.102908 / 0.176557 (-0.073648) | 0.139512 / 0.737135 (-0.597623) | 0.107703 / 0.296338 (-0.188635) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.437615 / 0.215209 (0.222406) | 4.373140 / 2.077655 (2.295486) | 2.065063 / 1.504120 (0.560943) | 1.863938 / 1.541195 (0.322743) | 1.907955 / 1.468490 (0.439465) | 0.695830 / 4.584777 (-3.888947) | 3.394248 / 3.745712 (-0.351464) | 1.842794 / 5.269862 (-3.427068) | 1.156928 / 4.565676 (-3.408748) | 0.082505 / 0.424275 (-0.341771) | 0.012405 / 0.007607 (0.004798) | 0.538041 / 0.226044 (0.311997) | 5.363508 / 2.268929 (3.094579) | 2.509383 / 55.444624 (-52.935241) | 2.160416 / 6.876477 (-4.716061) | 2.162054 / 2.142072 (0.019982) | 0.802419 / 4.805227 (-4.002809) | 0.150529 / 6.500664 (-6.350135) | 0.066418 / 0.075469 (-0.009051) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.257221 / 1.841788 (-0.584567) | 13.748839 / 8.074308 (5.674531) | 13.310555 / 10.191392 (3.119163) | 0.152997 / 0.680424 (-0.527427) | 0.016618 / 0.534201 (-0.517583) | 0.375443 / 0.579283 (-0.203840) | 0.374942 / 0.434364 (-0.059422) | 0.466704 / 0.540337 (-0.073633) | 0.553563 / 1.386936 (-0.833373) |

</details>
</details>

![](https://cml.dev/watermark.png#1ac8343af4e2dc6fe0771d0be70eaf8a6e5a8fbc "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009260 / 0.011353 (-0.002092) | 0.005213 / 0.011008 (-0.005795) | 0.102151 / 0.038508 (0.063643) | 0.035619 / 0.023109 (0.012510) | 0.296266 / 0.275898 (0.020368) | 0.359884 / 0.323480 (0.036404) | 0.008176 / 0.007986 (0.000190) | 0.005031 / 0.004328 (0.000703) | 0.077178 / 0.004250 (0.072927) | 0.041898 / 0.037052 (0.004846) | 0.305640 / 0.258489 (0.047151) | 0.346275 / 0.293841 (0.052434) | 0.037684 / 0.128546 (-0.090863) | 0.011816 / 0.075646 (-0.063831) | 0.334853 / 0.419271 (-0.084419) | 0.046535 / 0.043533 (0.003002) | 0.291544 / 0.255139 (0.036405) | 0.317194 / 0.283200 (0.033994) | 0.103212 / 0.141683 (-0.038471) | 1.424994 / 1.452155 (-0.027161) | 1.486216 / 1.492716 (-0.006501) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.011816 / 0.018006 (-0.006190) | 0.442092 / 0.000490 (0.441602) | 0.001297 / 0.000200 (0.001097) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028277 / 0.037411 (-0.009134) | 0.110431 / 0.014526 (0.095905) | 0.118456 / 0.176557 (-0.058100) | 0.156778 / 0.737135 (-0.580357) | 0.123036 / 0.296338 (-0.173302) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.399006 / 0.215209 (0.183797) | 3.990367 / 2.077655 (1.912712) | 1.798739 / 1.504120 (0.294620) | 1.607133 / 1.541195 (0.065938) | 1.748897 / 1.468490 (0.280407) | 0.690666 / 4.584777 (-3.894111) | 3.795892 / 3.745712 (0.050180) | 3.479317 / 5.269862 (-1.790545) | 1.861268 / 4.565676 (-2.704409) | 0.085235 / 0.424275 (-0.339040) | 0.012997 / 0.007607 (0.005390) | 0.512489 / 0.226044 (0.286445) | 5.039515 / 2.268929 (2.770587) | 2.258079 / 55.444624 (-53.186545) | 1.907178 / 6.876477 (-4.969299) | 1.985953 / 2.142072 (-0.156119) | 0.843595 / 4.805227 (-3.961633) | 0.165286 / 6.500664 (-6.335378) | 0.063026 / 0.075469 (-0.012443) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.186680 / 1.841788 (-0.655108) | 14.976016 / 8.074308 (6.901708) | 14.436941 / 10.191392 (4.245549) | 0.172620 / 0.680424 (-0.507804) | 0.028760 / 0.534201 (-0.505441) | 0.443505 / 0.579283 (-0.135778) | 0.435665 / 0.434364 (0.001301) | 0.520164 / 0.540337 (-0.020174) | 0.608348 / 1.386936 (-0.778588) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007510 / 0.011353 (-0.003842) | 0.005012 / 0.011008 (-0.005996) | 0.077865 / 0.038508 (0.039357) | 0.033610 / 0.023109 (0.010500) | 0.365996 / 0.275898 (0.090098) | 0.416393 / 0.323480 (0.092913) | 0.005672 / 0.007986 (-0.002314) | 0.005334 / 0.004328 (0.001006) | 0.074948 / 0.004250 (0.070698) | 0.045962 / 0.037052 (0.008909) | 0.362209 / 0.258489 (0.103719) | 0.410522 / 0.293841 (0.116681) | 0.036247 / 0.128546 (-0.092299) | 0.012432 / 0.075646 (-0.063214) | 0.088754 / 0.419271 (-0.330517) | 0.048848 / 0.043533 (0.005315) | 0.370994 / 0.255139 (0.115855) | 0.382476 / 0.283200 (0.099277) | 0.103443 / 0.141683 (-0.038240) | 1.483127 / 1.452155 (0.030972) | 1.573366 / 1.492716 (0.080650) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.224163 / 0.018006 (0.206157) | 0.475136 / 0.000490 (0.474646) | 0.000394 / 0.000200 (0.000194) | 0.000057 / 0.000054 (0.000003) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030612 / 0.037411 (-0.006799) | 0.113983 / 0.014526 (0.099457) | 0.121835 / 0.176557 (-0.054722) | 0.160092 / 0.737135 (-0.577043) | 0.127431 / 0.296338 (-0.168908) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.421389 / 0.215209 (0.206179) | 4.207638 / 2.077655 (2.129984) | 2.040265 / 1.504120 (0.536145) | 1.868617 / 1.541195 (0.327422) | 1.979016 / 1.468490 (0.510526) | 0.712499 / 4.584777 (-3.872278) | 3.783091 / 3.745712 (0.037379) | 2.124293 / 5.269862 (-3.145569) | 1.382028 / 4.565676 (-3.183649) | 0.087133 / 0.424275 (-0.337142) | 0.012634 / 0.007607 (0.005027) | 0.518965 / 0.226044 (0.292920) | 5.188330 / 2.268929 (2.919401) | 2.556593 / 55.444624 (-52.888031) | 2.243081 / 6.876477 (-4.633396) | 2.340420 / 2.142072 (0.198347) | 0.858010 / 4.805227 (-3.947218) | 0.169165 / 6.500664 (-6.331499) | 0.065177 / 0.075469 (-0.010292) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.297350 / 1.841788 (-0.544438) | 15.404241 / 8.074308 (7.329933) | 13.806039 / 10.191392 (3.614647) | 0.182055 / 0.680424 (-0.498369) | 0.017789 / 0.534201 (-0.516412) | 0.422828 / 0.579283 (-0.156455) | 0.418269 / 0.434364 (-0.016095) | 0.521561 / 0.540337 (-0.018777) | 0.642526 / 1.386936 (-0.744410) |

</details>
</details>

![](https://cml.dev/watermark.png#0009eea6819c32a888f65b0fdb5889b6d311c436 "CML watermark")