Use auth to get parquet export
Opened by lhoestq (closed)

added `token` to the `_datasets_server` functions

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6468). All of your documentation changes will be reflected on that endpoint.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005076 / 0.011353 (-0.006277) | 0.003510 / 0.011008 (-0.007499) | 0.062939 / 0.038508 (0.024431) | 0.049191 / 0.023109 (0.026082) | 0.259088 / 0.275898 (-0.016810) | 0.273523 / 0.323480 (-0.049957) | 0.003902 / 0.007986 (-0.004083) | 0.002699 / 0.004328 (-0.001630) | 0.049077 / 0.004250 (0.044827) | 0.037174 / 0.037052 (0.000121) | 0.256467 / 0.258489 (-0.002022) | 0.291235 / 0.293841 (-0.002606) | 0.028119 / 0.128546 (-0.100427) | 0.010404 / 0.075646 (-0.065243) | 0.205825 / 0.419271 (-0.213446) | 0.035741 / 0.043533 (-0.007792) | 0.253219 / 0.255139 (-0.001920) | 0.274986 / 0.283200 (-0.008214) | 0.018379 / 0.141683 (-0.123304) | 1.131139 / 1.452155 (-0.321016) | 1.175875 / 1.492716 (-0.316841) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.090717 / 0.018006 (0.072710) | 0.299285 / 0.000490 (0.298796) | 0.000217 / 0.000200 (0.000017) | 0.000050 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018678 / 0.037411 (-0.018733) | 0.060558 / 0.014526 (0.046032) | 0.073828 / 0.176557 (-0.102728) | 0.119302 / 0.737135 (-0.617833) | 0.075261 / 0.296338 (-0.221078) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.277018 / 0.215209 (0.061809) | 2.713255 / 2.077655 (0.635601) | 1.427512 / 1.504120 (-0.076608) | 1.311374 / 1.541195 (-0.229821) | 1.348756 / 1.468490 (-0.119734) | 0.561777 / 4.584777 (-4.023000) | 2.393578 / 3.745712 (-1.352134) | 2.798109 / 5.269862 (-2.471753) | 1.754808 / 4.565676 (-2.810869) | 0.062302 / 0.424275 (-0.361973) | 0.004948 / 0.007607 (-0.002659) | 0.328468 / 0.226044 (0.102423) | 3.246558 / 2.268929 (0.977629) | 1.786816 / 55.444624 (-53.657808) | 1.482937 / 6.876477 (-5.393540) | 1.516109 / 2.142072 (-0.625963) | 0.634457 / 4.805227 (-4.170770) | 0.116505 / 6.500664 (-6.384159) | 0.042162 / 0.075469 (-0.033308) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.935312 / 1.841788 (-0.906476) | 11.540599 / 8.074308 (3.466291) | 10.512593 / 10.191392 (0.321201) | 0.129638 / 0.680424 (-0.550786) | 0.013994 / 0.534201 (-0.520207) | 0.291490 / 0.579283 (-0.287793) | 0.263641 / 0.434364 (-0.170722) | 0.328718 / 0.540337 (-0.211619) | 0.437598 / 1.386936 (-0.949338) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005192 / 0.011353 (-0.006161) | 0.003454 / 0.011008 (-0.007554) | 0.049448 / 0.038508 (0.010940) | 0.050968 / 0.023109 (0.027859) | 0.273702 / 0.275898 (-0.002196) | 0.296934 / 0.323480 (-0.026545) | 0.004066 / 0.007986 (-0.003920) | 0.002611 / 0.004328 (-0.001718) | 0.048284 / 0.004250 (0.044034) | 0.041399 / 0.037052 (0.004346) | 0.283000 / 0.258489 (0.024511) | 0.302553 / 0.293841 (0.008712) | 0.029086 / 0.128546 (-0.099460) | 0.010510 / 0.075646 (-0.065137) | 0.058097 / 0.419271 (-0.361175) | 0.032992 / 0.043533 (-0.010541) | 0.271752 / 0.255139 (0.016613) | 0.293535 / 0.283200 (0.010335) | 0.016958 / 0.141683 (-0.124725) | 1.130126 / 1.452155 (-0.322028) | 1.187228 / 1.492716 (-0.305488) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.092321 / 0.018006 (0.074315) | 0.302599 / 0.000490 (0.302109) | 0.000215 / 0.000200 (0.000015) | 0.000051 / 0.000054 (-0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021837 / 0.037411 (-0.015574) | 0.071148 / 0.014526 (0.056622) | 0.082448 / 0.176557 (-0.094108) | 0.128083 / 0.737135 (-0.609053) | 0.090864 / 0.296338 (-0.205474) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.296248 / 0.215209 (0.081039) | 2.881130 / 2.077655 (0.803476) | 1.580360 / 1.504120 (0.076240) | 1.454642 / 1.541195 (-0.086553) | 1.461453 / 1.468490 (-0.007037) | 0.567500 / 4.584777 (-4.017277) | 2.493708 / 3.745712 (-1.252004) | 2.756623 / 5.269862 (-2.513239) | 1.771319 / 4.565676 (-2.794358) | 0.062287 / 0.424275 (-0.361988) | 0.004917 / 0.007607 (-0.002691) | 0.348034 / 0.226044 (0.121990) | 3.426938 / 2.268929 (1.158010) | 1.954190 / 55.444624 (-53.490435) | 1.660870 / 6.876477 (-5.215607) | 1.675118 / 2.142072 (-0.466955) | 0.636843 / 4.805227 (-4.168384) | 0.115028 / 6.500664 (-6.385636) | 0.040702 / 0.075469 (-0.034767) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.988076 / 1.841788 (-0.853711) | 11.890867 / 8.074308 (3.816559) | 10.621169 / 10.191392 (0.429777) | 0.131568 / 0.680424 (-0.548856) | 0.014994 / 0.534201 (-0.519207) | 0.288900 / 0.579283 (-0.290384) | 0.272092 / 0.434364 (-0.162272) | 0.329397 / 0.540337 (-0.210940) | 0.569337 / 1.386936 (-0.817599) |

</details>
</details>

![](https://cml.dev/watermark.png#ae3b4a2268adc2f21568ff63891e9a83530c7e29 "CML watermark")