Issue: docs: Update num_shards docs to mention num_proc on Dataset and DatasetDict
State: closed
Author: connor-henderson

Closes #5653

@mariosasko

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007351 / 0.011353 (-0.004002) | 0.005025 / 0.011008 (-0.005983) | 0.095978 / 0.038508 (0.057470) | 0.033486 / 0.023109 (0.010377) | 0.294427 / 0.275898 (0.018529) | 0.325157 / 0.323480 (0.001677) | 0.005671 / 0.007986 (-0.002315) | 0.005284 / 0.004328 (0.000955) | 0.073159 / 0.004250 (0.068909) | 0.045162 / 0.037052 (0.008110) | 0.294004 / 0.258489 (0.035515) | 0.343545 / 0.293841 (0.049704) | 0.036857 / 0.128546 (-0.091689) | 0.012245 / 0.075646 (-0.063401) | 0.332258 / 0.419271 (-0.087014) | 0.051909 / 0.043533 (0.008377) | 0.295701 / 0.255139 (0.040562) | 0.315247 / 0.283200 (0.032048) | 0.102363 / 0.141683 (-0.039320) | 1.441944 / 1.452155 (-0.010211) | 1.527161 / 1.492716 (0.034445) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211769 / 0.018006 (0.193763) | 0.452015 / 0.000490 (0.451525) | 0.004041 / 0.000200 (0.003841) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027396 / 0.037411 (-0.010015) | 0.108318 / 0.014526 (0.093793) | 0.116851 / 0.176557 (-0.059706) | 0.172658 / 0.737135 (-0.564478) | 0.122876 / 0.296338 (-0.173462) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.406484 / 0.215209 (0.191275) | 4.053849 / 2.077655 (1.976194) | 1.842947 / 1.504120 (0.338827) | 1.649473 / 1.541195 (0.108278) | 1.728629 / 1.468490 (0.260139) | 0.699519 / 4.584777 (-3.885258) | 3.730823 / 3.745712 (-0.014889) | 2.139624 / 5.269862 (-3.130237) | 1.487839 / 4.565676 (-3.077837) | 0.086699 / 0.424275 (-0.337576) | 0.012815 / 0.007607 (0.005208) | 0.514014 / 0.226044 (0.287969) | 5.153315 / 2.268929 (2.884387) | 2.324431 / 55.444624 (-53.120193) | 1.971533 / 6.876477 (-4.904944) | 2.074480 / 2.142072 (-0.067592) | 0.842419 / 4.805227 (-3.962808) | 0.169140 / 6.500664 (-6.331524) | 0.065206 / 0.075469 (-0.010263) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.180887 / 1.841788 (-0.660901) | 14.627401 / 8.074308 (6.553093) | 14.382699 / 10.191392 (4.191307) | 0.143986 / 0.680424 (-0.536438) | 0.017460 / 0.534201 (-0.516741) | 0.422100 / 0.579283 (-0.157183) | 0.417474 / 0.434364 (-0.016890) | 0.493712 / 0.540337 (-0.046625) | 0.589744 / 1.386936 (-0.797193) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007538 / 0.011353 (-0.003815) | 0.005122 / 0.011008 (-0.005887) | 0.073858 / 0.038508 (0.035350) | 0.034561 / 0.023109 (0.011451) | 0.341250 / 0.275898 (0.065352) | 0.373063 / 0.323480 (0.049583) | 0.005785 / 0.007986 (-0.002200) | 0.005393 / 0.004328 (0.001065) | 0.072354 / 0.004250 (0.068104) | 0.047005 / 0.037052 (0.009953) | 0.341179 / 0.258489 (0.082690) | 0.386299 / 0.293841 (0.092458) | 0.038315 / 0.128546 (-0.090231) | 0.012200 / 0.075646 (-0.063446) | 0.086132 / 0.419271 (-0.333140) | 0.049873 / 0.043533 (0.006340) | 0.337985 / 0.255139 (0.082846) | 0.354806 / 0.283200 (0.071607) | 0.103557 / 0.141683 (-0.038126) | 1.445682 / 1.452155 (-0.006473) | 1.551008 / 1.492716 (0.058291) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.235873 / 0.018006 (0.217867) | 0.448445 / 0.000490 (0.447955) | 0.001307 / 0.000200 (0.001108) | 0.000087 / 0.000054 (0.000032) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029809 / 0.037411 (-0.007603) | 0.108833 / 0.014526 (0.094307) | 0.123289 / 0.176557 (-0.053268) | 0.176516 / 0.737135 (-0.560620) | 0.127186 / 0.296338 (-0.169153) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.422037 / 0.215209 (0.206828) | 4.188073 / 2.077655 (2.110418) | 1.999295 / 1.504120 (0.495175) | 1.809229 / 1.541195 (0.268034) | 1.930798 / 1.468490 (0.462308) | 0.694371 / 4.584777 (-3.890406) | 3.833432 / 3.745712 (0.087719) | 3.235600 / 5.269862 (-2.034262) | 1.867822 / 4.565676 (-2.697854) | 0.085734 / 0.424275 (-0.338541) | 0.012727 / 0.007607 (0.005120) | 0.542261 / 0.226044 (0.316217) | 5.289366 / 2.268929 (3.020437) | 2.469636 / 55.444624 (-52.974988) | 2.139392 / 6.876477 (-4.737084) | 2.193305 / 2.142072 (0.051233) | 0.846747 / 4.805227 (-3.958481) | 0.168965 / 6.500664 (-6.331699) | 0.064463 / 0.075469 (-0.011006) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.263818 / 1.841788 (-0.577970) | 15.254642 / 8.074308 (7.180334) | 14.428111 / 10.191392 (4.236719) | 0.164770 / 0.680424 (-0.515654) | 0.017476 / 0.534201 (-0.516725) | 0.420198 / 0.579283 (-0.159085) | 0.443250 / 0.434364 (0.008886) | 0.496904 / 0.540337 (-0.043434) | 0.596541 / 1.386936 (-0.790395) |

</details>
</details>

![](https://cml.dev/watermark.png#4db8e33eb9cf6cd4453cdfa246c065e0eedf170c "CML watermark")