Issue: Fix protocol prefix for Beam
State: closed
Author: mariosasko

Fix #6147

Comments:
<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006736 / 0.011353 (-0.004617) | 0.004099 / 0.011008 (-0.006909) | 0.084339 / 0.038508 (0.045831) | 0.073715 / 0.023109 (0.050605) | 0.311962 / 0.275898 (0.036064) | 0.356108 / 0.323480 (0.032628) | 0.005321 / 0.007986 (-0.002665) | 0.003390 / 0.004328 (-0.000939) | 0.064622 / 0.004250 (0.060372) | 0.053978 / 0.037052 (0.016926) | 0.328967 / 0.258489 (0.070478) | 0.370506 / 0.293841 (0.076665) | 0.031123 / 0.128546 (-0.097423) | 0.008465 / 0.075646 (-0.067181) | 0.288136 / 0.419271 (-0.131136) | 0.052909 / 0.043533 (0.009376) | 0.325189 / 0.255139 (0.070050) | 0.360112 / 0.283200 (0.076912) | 0.023389 / 0.141683 (-0.118294) | 1.492899 / 1.452155 (0.040744) | 1.586449 / 1.492716 (0.093733) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.219708 / 0.018006 (0.201702) | 0.469550 / 0.000490 (0.469060) | 0.002776 / 0.000200 (0.002576) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028985 / 0.037411 (-0.008427) | 0.083487 / 0.014526 (0.068961) | 0.096938 / 0.176557 (-0.079619) | 0.152886 / 0.737135 (-0.584249) | 0.096242 / 0.296338 (-0.200096) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.381959 / 0.215209 (0.166750) | 3.800033 / 2.077655 (1.722378) | 1.831903 / 1.504120 (0.327783) | 1.663207 / 1.541195 (0.122012) | 1.747282 / 1.468490 (0.278792) | 0.481671 / 4.584777 (-4.103106) | 3.653725 / 3.745712 (-0.091987) | 3.253058 / 5.269862 (-2.016804) | 2.022014 / 4.565676 (-2.543663) | 0.056651 / 0.424275 (-0.367624) | 0.007640 / 0.007607 (0.000033) | 0.461795 / 0.226044 (0.235750) | 4.625535 / 2.268929 (2.356606) | 2.356341 / 55.444624 (-53.088283) | 1.977437 / 6.876477 (-4.899040) | 2.179672 / 2.142072 (0.037599) | 0.582875 / 4.805227 (-4.222353) | 0.132964 / 6.500664 (-6.367700) | 0.060398 / 0.075469 (-0.015071) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.309567 / 1.841788 (-0.532220) | 19.856306 / 8.074308 (11.781997) | 14.074350 / 10.191392 (3.882958) | 0.149615 / 0.680424 (-0.530809) | 0.018487 / 0.534201 (-0.515714) | 0.393995 / 0.579283 (-0.185288) | 0.409057 / 0.434364 (-0.025307) | 0.459551 / 0.540337 (-0.080787) | 0.644594 / 1.386936 (-0.742342) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006824 / 0.011353 (-0.004529) | 0.004099 / 0.011008 (-0.006909) | 0.064415 / 0.038508 (0.025907) | 0.077983 / 0.023109 (0.054874) | 0.359351 / 0.275898 (0.083453) | 0.395168 / 0.323480 (0.071688) | 0.005384 / 0.007986 (-0.002602) | 0.003298 / 0.004328 (-0.001030) | 0.065041 / 0.004250 (0.060791) | 0.056717 / 0.037052 (0.019664) | 0.366882 / 0.258489 (0.108393) | 0.401337 / 0.293841 (0.107496) | 0.032273 / 0.128546 (-0.096273) | 0.008666 / 0.075646 (-0.066981) | 0.071442 / 0.419271 (-0.347829) | 0.049999 / 0.043533 (0.006466) | 0.365001 / 0.255139 (0.109862) | 0.379579 / 0.283200 (0.096379) | 0.023357 / 0.141683 (-0.118326) | 1.476839 / 1.452155 (0.024684) | 1.541703 / 1.492716 (0.048987) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239014 / 0.018006 (0.221008) | 0.460678 / 0.000490 (0.460188) | 0.003368 / 0.000200 (0.003168) | 0.000089 / 0.000054 (0.000035) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030981 / 0.037411 (-0.006430) | 0.088287 / 0.014526 (0.073761) | 0.102459 / 0.176557 (-0.074098) | 0.154695 / 0.737135 (-0.582441) | 0.103479 / 0.296338 (-0.192860) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.416084 / 0.215209 (0.200874) | 4.128365 / 2.077655 (2.050710) | 2.113053 / 1.504120 (0.608934) | 1.948993 / 1.541195 (0.407798) | 2.035609 / 1.468490 (0.567119) | 0.481705 / 4.584777 (-4.103072) | 3.630366 / 3.745712 (-0.115346) | 3.340837 / 5.269862 (-1.929024) | 2.052573 / 4.565676 (-2.513104) | 0.056805 / 0.424275 (-0.367470) | 0.007294 / 0.007607 (-0.000313) | 0.489597 / 0.226044 (0.263553) | 4.892728 / 2.268929 (2.623799) | 2.564692 / 55.444624 (-52.879932) | 2.251964 / 6.876477 (-4.624513) | 2.457912 / 2.142072 (0.315839) | 0.588433 / 4.805227 (-4.216794) | 0.133588 / 6.500664 (-6.367076) | 0.062298 / 0.075469 (-0.013171) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.328566 / 1.841788 (-0.513222) | 20.145568 / 8.074308 (12.071260) | 14.231306 / 10.191392 (4.039914) | 0.168356 / 0.680424 (-0.512067) | 0.018333 / 0.534201 (-0.515868) | 0.390901 / 0.579283 (-0.188382) | 0.415005 / 0.434364 (-0.019359) | 0.477282 / 0.540337 (-0.063055) | 0.652085 / 1.386936 (-0.734851) |

</details>
</details>

![](https://cml.dev/watermark.png#341a41880a70b29f030caa0d36f1e297535ba5f9 "CML watermark")


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6161). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006388 / 0.011353 (-0.004965) | 0.003917 / 0.011008 (-0.007092) | 0.087397 / 0.038508 (0.048889) | 0.068522 / 0.023109 (0.045412) | 0.313299 / 0.275898 (0.037401) | 0.342884 / 0.323480 (0.019405) | 0.005216 / 0.007986 (-0.002770) | 0.003293 / 0.004328 (-0.001035) | 0.067474 / 0.004250 (0.063224) | 0.051122 / 0.037052 (0.014070) | 0.326443 / 0.258489 (0.067954) | 0.355744 / 0.293841 (0.061903) | 0.031130 / 0.128546 (-0.097416) | 0.008617 / 0.075646 (-0.067029) | 0.291201 / 0.419271 (-0.128070) | 0.052050 / 0.043533 (0.008517) | 0.312135 / 0.255139 (0.056996) | 0.347233 / 0.283200 (0.064034) | 0.023775 / 0.141683 (-0.117907) | 1.478807 / 1.452155 (0.026652) | 1.581239 / 1.492716 (0.088522) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208252 / 0.018006 (0.190246) | 0.466314 / 0.000490 (0.465824) | 0.004439 / 0.000200 (0.004239) | 0.000104 / 0.000054 (0.000050) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027918 / 0.037411 (-0.009494) | 0.082410 / 0.014526 (0.067884) | 0.094231 / 0.176557 (-0.082326) | 0.150189 / 0.737135 (-0.586946) | 0.095404 / 0.296338 (-0.200935) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382026 / 0.215209 (0.166817) | 3.822213 / 2.077655 (1.744559) | 1.833716 / 1.504120 (0.329596) | 1.666250 / 1.541195 (0.125055) | 1.703350 / 1.468490 (0.234860) | 0.477918 / 4.584777 (-4.106859) | 3.629304 / 3.745712 (-0.116408) | 3.199672 / 5.269862 (-2.070190) | 1.977855 / 4.565676 (-2.587821) | 0.056275 / 0.424275 (-0.368000) | 0.007538 / 0.007607 (-0.000070) | 0.455995 / 0.226044 (0.229950) | 4.559234 / 2.268929 (2.290305) | 2.333819 / 55.444624 (-53.110805) | 2.006851 / 6.876477 (-4.869625) | 2.150683 / 2.142072 (0.008611) | 0.576786 / 4.805227 (-4.228441) | 0.132352 / 6.500664 (-6.368312) | 0.059359 / 0.075469 (-0.016110) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.261525 / 1.841788 (-0.580262) | 19.174957 / 8.074308 (11.100649) | 14.286796 / 10.191392 (4.095404) | 0.144610 / 0.680424 (-0.535813) | 0.018213 / 0.534201 (-0.515988) | 0.390404 / 0.579283 (-0.188879) | 0.404678 / 0.434364 (-0.029686) | 0.455636 / 0.540337 (-0.084701) | 0.620801 / 1.386936 (-0.766135) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006383 / 0.011353 (-0.004970) | 0.003852 / 0.011008 (-0.007156) | 0.064116 / 0.038508 (0.025607) | 0.068920 / 0.023109 (0.045810) | 0.359439 / 0.275898 (0.083541) | 0.388904 / 0.323480 (0.065425) | 0.005192 / 0.007986 (-0.002794) | 0.003233 / 0.004328 (-0.001095) | 0.064589 / 0.004250 (0.060339) | 0.054496 / 0.037052 (0.017444) | 0.368699 / 0.258489 (0.110210) | 0.400420 / 0.293841 (0.106579) | 0.030869 / 0.128546 (-0.097677) | 0.008424 / 0.075646 (-0.067222) | 0.071015 / 0.419271 (-0.348257) | 0.048333 / 0.043533 (0.004801) | 0.360652 / 0.255139 (0.105513) | 0.393534 / 0.283200 (0.110334) | 0.022685 / 0.141683 (-0.118998) | 1.495565 / 1.452155 (0.043410) | 1.537947 / 1.492716 (0.045230) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.232911 / 0.018006 (0.214905) | 0.454191 / 0.000490 (0.453702) | 0.005711 / 0.000200 (0.005511) | 0.000117 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029486 / 0.037411 (-0.007925) | 0.087249 / 0.014526 (0.072724) | 0.100104 / 0.176557 (-0.076453) | 0.151556 / 0.737135 (-0.585580) | 0.100853 / 0.296338 (-0.195485) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.415134 / 0.215209 (0.199925) | 4.139068 / 2.077655 (2.061413) | 2.121079 / 1.504120 (0.616959) | 1.945616 / 1.541195 (0.404421) | 1.988188 / 1.468490 (0.519698) | 0.483994 / 4.584777 (-4.100783) | 3.640366 / 3.745712 (-0.105347) | 3.218896 / 5.269862 (-2.050966) | 2.015527 / 4.565676 (-2.550149) | 0.056946 / 0.424275 (-0.367329) | 0.007262 / 0.007607 (-0.000345) | 0.486075 / 0.226044 (0.260031) | 4.864191 / 2.268929 (2.595262) | 2.590853 / 55.444624 (-52.853772) | 2.315359 / 6.876477 (-4.561118) | 2.418733 / 2.142072 (0.276661) | 0.582378 / 4.805227 (-4.222849) | 0.134097 / 6.500664 (-6.366568) | 0.060797 / 0.075469 (-0.014672) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.337021 / 1.841788 (-0.504766) | 19.468907 / 8.074308 (11.394599) | 14.348874 / 10.191392 (4.157482) | 0.170408 / 0.680424 (-0.510016) | 0.018414 / 0.534201 (-0.515787) | 0.394551 / 0.579283 (-0.184732) | 0.404750 / 0.434364 (-0.029613) | 0.471972 / 0.540337 (-0.068365) | 0.650607 / 1.386936 (-0.736329) |

</details>
</details>

![](https://cml.dev/watermark.png#ab4d978e2d5c246dc91e2fed041b06a38190be3b "CML watermark")


The CI errors are unrelated to the changes

The Beam API is now deprecated, so I think closing this PR is also fine :)