Add push_to_hub with multiple configs docs
Opened by lhoestq (closed)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005920 / 0.011353 (-0.005433) | 0.003623 / 0.011008 (-0.007385) | 0.079283 / 0.038508 (0.040775) | 0.058325 / 0.023109 (0.035216) | 0.313733 / 0.275898 (0.037835) | 0.360790 / 0.323480 (0.037310) | 0.004653 / 0.007986 (-0.003332) | 0.002876 / 0.004328 (-0.001452) | 0.062137 / 0.004250 (0.057886) | 0.045084 / 0.037052 (0.008031) | 0.328569 / 0.258489 (0.070079) | 0.368965 / 0.293841 (0.075124) | 0.027085 / 0.128546 (-0.101461) | 0.008051 / 0.075646 (-0.067595) | 0.260222 / 0.419271 (-0.159050) | 0.045477 / 0.043533 (0.001944) | 0.315344 / 0.255139 (0.060205) | 0.348215 / 0.283200 (0.065015) | 0.021352 / 0.141683 (-0.120331) | 1.432200 / 1.452155 (-0.019955) | 1.509217 / 1.492716 (0.016501) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.199843 / 0.018006 (0.181837) | 0.427925 / 0.000490 (0.427435) | 0.002903 / 0.000200 (0.002703) | 0.000067 / 0.000054 (0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023121 / 0.037411 (-0.014291) | 0.072451 / 0.014526 (0.057925) | 0.083260 / 0.176557 (-0.093296) | 0.142879 / 0.737135 (-0.594257) | 0.084053 / 0.296338 (-0.212286) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.394922 / 0.215209 (0.179713) | 3.956111 / 2.077655 (1.878456) | 1.926411 / 1.504120 (0.422291) | 1.743840 / 1.541195 (0.202646) | 1.776957 / 1.468490 (0.308467) | 0.502134 / 4.584777 (-4.082643) | 3.001721 / 3.745712 (-0.743991) | 2.852496 / 5.269862 (-2.417365) | 1.862794 / 4.565676 (-2.702883) | 0.057544 / 0.424275 (-0.366731) | 0.006751 / 0.007607 (-0.000856) | 0.470619 / 0.226044 (0.244575) | 4.696674 / 2.268929 (2.427746) | 2.326545 / 55.444624 (-53.118080) | 1.980888 / 6.876477 (-4.895589) | 2.139172 / 2.142072 (-0.002901) | 0.590256 / 4.805227 (-4.214971) | 0.125815 / 6.500664 (-6.374849) | 0.061000 / 0.075469 (-0.014469) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.261948 / 1.841788 (-0.579839) | 18.317473 / 8.074308 (10.243165) | 13.810883 / 10.191392 (3.619491) | 0.146180 / 0.680424 (-0.534244) | 0.016701 / 0.534201 (-0.517500) | 0.330731 / 0.579283 (-0.248552) | 0.345103 / 0.434364 (-0.089261) | 0.374449 / 0.540337 (-0.165889) | 0.522463 / 1.386936 (-0.864473) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006217 / 0.011353 (-0.005136) | 0.003678 / 0.011008 (-0.007331) | 0.062321 / 0.038508 (0.023813) | 0.059256 / 0.023109 (0.036147) | 0.444501 / 0.275898 (0.168603) | 0.475881 / 0.323480 (0.152401) | 0.004863 / 0.007986 (-0.003123) | 0.002916 / 0.004328 (-0.001412) | 0.062197 / 0.004250 (0.057946) | 0.048449 / 0.037052 (0.011396) | 0.443680 / 0.258489 (0.185191) | 0.484570 / 0.293841 (0.190729) | 0.028694 / 0.128546 (-0.099852) | 0.008096 / 0.075646 (-0.067550) | 0.068347 / 0.419271 (-0.350924) | 0.041031 / 0.043533 (-0.002502) | 0.443907 / 0.255139 (0.188768) | 0.469888 / 0.283200 (0.186689) | 0.020237 / 0.141683 (-0.121445) | 1.438484 / 1.452155 (-0.013671) | 1.512652 / 1.492716 (0.019936) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.243118 / 0.018006 (0.225111) | 0.416797 / 0.000490 (0.416308) | 0.010421 / 0.000200 (0.010221) | 0.000082 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026191 / 0.037411 (-0.011220) | 0.080881 / 0.014526 (0.066355) | 0.093207 / 0.176557 (-0.083349) | 0.146708 / 0.737135 (-0.590428) | 0.091676 / 0.296338 (-0.204663) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.461475 / 0.215209 (0.246266) | 4.617351 / 2.077655 (2.539696) | 2.564369 / 1.504120 (1.060249) | 2.393263 / 1.541195 (0.852068) | 2.447343 / 1.468490 (0.978853) | 0.508764 / 4.584777 (-4.076013) | 3.075460 / 3.745712 (-0.670252) | 2.884683 / 5.269862 (-2.385179) | 1.866432 / 4.565676 (-2.699244) | 0.058759 / 0.424275 (-0.365516) | 0.006591 / 0.007607 (-0.001016) | 0.537718 / 0.226044 (0.311674) | 5.378709 / 2.268929 (3.109781) | 3.006751 / 55.444624 (-52.437873) | 2.666653 / 6.876477 (-4.209824) | 2.847559 / 2.142072 (0.705486) | 0.596878 / 4.805227 (-4.208350) | 0.125073 / 6.500664 (-6.375591) | 0.061345 / 0.075469 (-0.014124) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.349066 / 1.841788 (-0.492721) | 18.684735 / 8.074308 (10.610427) | 15.128142 / 10.191392 (4.936750) | 0.149254 / 0.680424 (-0.531170) | 0.017911 / 0.534201 (-0.516290) | 0.344057 / 0.579283 (-0.235226) | 0.363474 / 0.434364 (-0.070890) | 0.399425 / 0.540337 (-0.140912) | 0.549329 / 1.386936 (-0.837607) |

</details>
</details>

![](https://cml.dev/watermark.png#e675a2396efb5204a4553721001f3b46aa4cc334 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005843 / 0.011353 (-0.005510) | 0.003549 / 0.011008 (-0.007460) | 0.082318 / 0.038508 (0.043810) | 0.056835 / 0.023109 (0.033726) | 0.312968 / 0.275898 (0.037070) | 0.345918 / 0.323480 (0.022438) | 0.003239 / 0.007986 (-0.004747) | 0.002762 / 0.004328 (-0.001567) | 0.062362 / 0.004250 (0.058111) | 0.045934 / 0.037052 (0.008882) | 0.317035 / 0.258489 (0.058546) | 0.358473 / 0.293841 (0.064632) | 0.027311 / 0.128546 (-0.101235) | 0.007994 / 0.075646 (-0.067652) | 0.261565 / 0.419271 (-0.157706) | 0.044942 / 0.043533 (0.001410) | 0.313092 / 0.255139 (0.057953) | 0.339021 / 0.283200 (0.055821) | 0.021555 / 0.141683 (-0.120127) | 1.421232 / 1.452155 (-0.030923) | 1.487597 / 1.492716 (-0.005119) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.206432 / 0.018006 (0.188425) | 0.421932 / 0.000490 (0.421442) | 0.002825 / 0.000200 (0.002625) | 0.000065 / 0.000054 (0.000011) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022795 / 0.037411 (-0.014616) | 0.072666 / 0.014526 (0.058140) | 0.082779 / 0.176557 (-0.093778) | 0.142320 / 0.737135 (-0.594815) | 0.083343 / 0.296338 (-0.212995) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.394227 / 0.215209 (0.179018) | 3.931858 / 2.077655 (1.854203) | 1.909953 / 1.504120 (0.405833) | 1.711298 / 1.541195 (0.170104) | 1.745816 / 1.468490 (0.277326) | 0.503670 / 4.584777 (-4.081107) | 3.053677 / 3.745712 (-0.692035) | 2.802597 / 5.269862 (-2.467264) | 1.825315 / 4.565676 (-2.740362) | 0.057741 / 0.424275 (-0.366534) | 0.006581 / 0.007607 (-0.001027) | 0.463597 / 0.226044 (0.237552) | 4.638821 / 2.268929 (2.369893) | 2.301266 / 55.444624 (-53.143358) | 1.967111 / 6.876477 (-4.909365) | 2.097756 / 2.142072 (-0.044317) | 0.589840 / 4.805227 (-4.215387) | 0.125538 / 6.500664 (-6.375126) | 0.061203 / 0.075469 (-0.014266) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.291815 / 1.841788 (-0.549973) | 17.997040 / 8.074308 (9.922732) | 13.616252 / 10.191392 (3.424860) | 0.137349 / 0.680424 (-0.543075) | 0.016626 / 0.534201 (-0.517575) | 0.329611 / 0.579283 (-0.249672) | 0.346592 / 0.434364 (-0.087772) | 0.379521 / 0.540337 (-0.160817) | 0.528058 / 1.386936 (-0.858878) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006073 / 0.011353 (-0.005280) | 0.003594 / 0.011008 (-0.007414) | 0.062537 / 0.038508 (0.024029) | 0.057503 / 0.023109 (0.034394) | 0.449427 / 0.275898 (0.173529) | 0.482729 / 0.323480 (0.159249) | 0.004690 / 0.007986 (-0.003295) | 0.002901 / 0.004328 (-0.001428) | 0.062421 / 0.004250 (0.058171) | 0.046405 / 0.037052 (0.009353) | 0.456578 / 0.258489 (0.198089) | 0.492268 / 0.293841 (0.198427) | 0.028283 / 0.128546 (-0.100263) | 0.008028 / 0.075646 (-0.067618) | 0.067885 / 0.419271 (-0.351387) | 0.041273 / 0.043533 (-0.002260) | 0.449870 / 0.255139 (0.194731) | 0.472305 / 0.283200 (0.189106) | 0.018556 / 0.141683 (-0.123127) | 1.449016 / 1.452155 (-0.003138) | 1.490839 / 1.492716 (-0.001877) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.226569 / 0.018006 (0.208563) | 0.417106 / 0.000490 (0.416616) | 0.002784 / 0.000200 (0.002584) | 0.000072 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025803 / 0.037411 (-0.011608) | 0.081084 / 0.014526 (0.066559) | 0.091851 / 0.176557 (-0.084706) | 0.143982 / 0.737135 (-0.593153) | 0.090511 / 0.296338 (-0.205827) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463664 / 0.215209 (0.248454) | 4.634528 / 2.077655 (2.556874) | 2.574739 / 1.504120 (1.070619) | 2.412857 / 1.541195 (0.871662) | 2.442858 / 1.468490 (0.974368) | 0.511990 / 4.584777 (-4.072787) | 3.070345 / 3.745712 (-0.675367) | 2.842290 / 5.269862 (-2.427571) | 1.846727 / 4.565676 (-2.718950) | 0.058852 / 0.424275 (-0.365424) | 0.006624 / 0.007607 (-0.000983) | 0.539616 / 0.226044 (0.313571) | 5.410784 / 2.268929 (3.141856) | 3.065593 / 55.444624 (-52.379031) | 2.677930 / 6.876477 (-4.198547) | 2.817548 / 2.142072 (0.675476) | 0.602672 / 4.805227 (-4.202555) | 0.125689 / 6.500664 (-6.374975) | 0.062007 / 0.075469 (-0.013462) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.335336 / 1.841788 (-0.506452) | 18.310099 / 8.074308 (10.235791) | 14.818452 / 10.191392 (4.627060) | 0.154001 / 0.680424 (-0.526423) | 0.017892 / 0.534201 (-0.516309) | 0.345989 / 0.579283 (-0.233294) | 0.352108 / 0.434364 (-0.082256) | 0.394333 / 0.540337 (-0.146004) | 0.547680 / 1.386936 (-0.839256) |

</details>
</details>

![](https://cml.dev/watermark.png#d058d6e9b849acb5bc61d7df597a94253b487eb6 "CML watermark")