Doc readme improvements
State: closed | Author: mariosasko (COLLABORATOR)

Changes in the doc READMe:
* adds two new sections (to be aligned with `transformers` and `hfh`): "Previewing the documentation" and "Writing documentation examples"
* replaces the mentions of `transformers` with `datasets`
* fixes some dead links

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006761 / 0.011353 (-0.004592) | 0.004307 / 0.011008 (-0.006701) | 0.084682 / 0.038508 (0.046174) | 0.083994 / 0.023109 (0.060885) | 0.316612 / 0.275898 (0.040714) | 0.346157 / 0.323480 (0.022678) | 0.004490 / 0.007986 (-0.003495) | 0.003699 / 0.004328 (-0.000629) | 0.066144 / 0.004250 (0.061894) | 0.057958 / 0.037052 (0.020906) | 0.319018 / 0.258489 (0.060529) | 0.367597 / 0.293841 (0.073756) | 0.031146 / 0.128546 (-0.097401) | 0.008814 / 0.075646 (-0.066832) | 0.290971 / 0.419271 (-0.128301) | 0.052769 / 0.043533 (0.009236) | 0.313125 / 0.255139 (0.057986) | 0.330473 / 0.283200 (0.047273) | 0.025922 / 0.141683 (-0.115760) | 1.494989 / 1.452155 (0.042834) | 1.556140 / 1.492716 (0.063423) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.310580 / 0.018006 (0.292574) | 0.563600 / 0.000490 (0.563110) | 0.012344 / 0.000200 (0.012144) | 0.000382 / 0.000054 (0.000328) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031468 / 0.037411 (-0.005943) | 0.084856 / 0.014526 (0.070331) | 0.101371 / 0.176557 (-0.075186) | 0.158735 / 0.737135 (-0.578400) | 0.102451 / 0.296338 (-0.193888) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.402288 / 0.215209 (0.187079) | 4.001351 / 2.077655 (1.923696) | 2.022710 / 1.504120 (0.518590) | 1.850236 / 1.541195 (0.309041) | 1.946779 / 1.468490 (0.478289) | 0.485828 / 4.584777 (-4.098949) | 3.584925 / 3.745712 (-0.160787) | 3.400815 / 5.269862 (-1.869046) | 2.123187 / 4.565676 (-2.442490) | 0.057373 / 0.424275 (-0.366902) | 0.007383 / 0.007607 (-0.000224) | 0.479773 / 0.226044 (0.253729) | 4.805342 / 2.268929 (2.536414) | 2.530151 / 55.444624 (-52.914473) | 2.190136 / 6.876477 (-4.686341) | 2.463666 / 2.142072 (0.321593) | 0.583512 / 4.805227 (-4.221715) | 0.134205 / 6.500664 (-6.366459) | 0.062021 / 0.075469 (-0.013448) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.239532 / 1.841788 (-0.602255) | 20.252941 / 8.074308 (12.178633) | 14.265697 / 10.191392 (4.074305) | 0.158745 / 0.680424 (-0.521679) | 0.018605 / 0.534201 (-0.515596) | 0.394246 / 0.579283 (-0.185037) | 0.415260 / 0.434364 (-0.019104) | 0.462636 / 0.540337 (-0.077701) | 0.645318 / 1.386936 (-0.741618) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007063 / 0.011353 (-0.004290) | 0.004388 / 0.011008 (-0.006621) | 0.064997 / 0.038508 (0.026489) | 0.085135 / 0.023109 (0.062026) | 0.424349 / 0.275898 (0.148451) | 0.456033 / 0.323480 (0.132553) | 0.005745 / 0.007986 (-0.002241) | 0.003705 / 0.004328 (-0.000624) | 0.065835 / 0.004250 (0.061585) | 0.058366 / 0.037052 (0.021314) | 0.421654 / 0.258489 (0.163165) | 0.460334 / 0.293841 (0.166493) | 0.032828 / 0.128546 (-0.095718) | 0.008974 / 0.075646 (-0.066673) | 0.072524 / 0.419271 (-0.346747) | 0.048558 / 0.043533 (0.005025) | 0.413546 / 0.255139 (0.158407) | 0.435765 / 0.283200 (0.152565) | 0.023754 / 0.141683 (-0.117929) | 1.476884 / 1.452155 (0.024730) | 1.560011 / 1.492716 (0.067294) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.318279 / 0.018006 (0.300272) | 0.544990 / 0.000490 (0.544501) | 0.007118 / 0.000200 (0.006918) | 0.000097 / 0.000054 (0.000043) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033352 / 0.037411 (-0.004059) | 0.092921 / 0.014526 (0.078395) | 0.109028 / 0.176557 (-0.067528) | 0.161433 / 0.737135 (-0.575703) | 0.108445 / 0.296338 (-0.187893) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.438925 / 0.215209 (0.223716) | 4.400714 / 2.077655 (2.323059) | 2.403727 / 1.504120 (0.899607) | 2.236472 / 1.541195 (0.695277) | 2.319219 / 1.468490 (0.850729) | 0.490159 / 4.584777 (-4.094618) | 3.647474 / 3.745712 (-0.098238) | 3.433144 / 5.269862 (-1.836718) | 2.145367 / 4.565676 (-2.420310) | 0.057994 / 0.424275 (-0.366281) | 0.007452 / 0.007607 (-0.000155) | 0.513808 / 0.226044 (0.287763) | 5.130792 / 2.268929 (2.861863) | 2.861691 / 55.444624 (-52.582934) | 2.473292 / 6.876477 (-4.403185) | 2.756445 / 2.142072 (0.614372) | 0.586783 / 4.805227 (-4.218444) | 0.134170 / 6.500664 (-6.366494) | 0.061149 / 0.075469 (-0.014320) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.350144 / 1.841788 (-0.491644) | 21.003528 / 8.074308 (12.929220) | 15.174314 / 10.191392 (4.982922) | 0.186535 / 0.680424 (-0.493888) | 0.020821 / 0.534201 (-0.513380) | 0.399210 / 0.579283 (-0.180073) | 0.431942 / 0.434364 (-0.002422) | 0.475395 / 0.540337 (-0.064942) | 0.677457 / 1.386936 (-0.709479) |

</details>
</details>

![](https://cml.dev/watermark.png#6aa5fc278324a253eab43ad1bc048e822e4ae5c7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007062 / 0.011353 (-0.004291) | 0.004299 / 0.011008 (-0.006710) | 0.086019 / 0.038508 (0.047511) | 0.085166 / 0.023109 (0.062057) | 0.355804 / 0.275898 (0.079906) | 0.381056 / 0.323480 (0.057577) | 0.005500 / 0.007986 (-0.002486) | 0.003496 / 0.004328 (-0.000833) | 0.064866 / 0.004250 (0.060615) | 0.057399 / 0.037052 (0.020346) | 0.357914 / 0.258489 (0.099425) | 0.395387 / 0.293841 (0.101546) | 0.031763 / 0.128546 (-0.096784) | 0.008665 / 0.075646 (-0.066981) | 0.290097 / 0.419271 (-0.129175) | 0.053297 / 0.043533 (0.009765) | 0.355659 / 0.255139 (0.100520) | 0.378232 / 0.283200 (0.095032) | 0.026015 / 0.141683 (-0.115668) | 1.437121 / 1.452155 (-0.015034) | 1.538798 / 1.492716 (0.046082) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.243518 / 0.018006 (0.225511) | 0.461361 / 0.000490 (0.460871) | 0.009529 / 0.000200 (0.009329) | 0.000473 / 0.000054 (0.000419) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030379 / 0.037411 (-0.007032) | 0.089851 / 0.014526 (0.075325) | 0.098278 / 0.176557 (-0.078278) | 0.157077 / 0.737135 (-0.580058) | 0.098997 / 0.296338 (-0.197341) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.382415 / 0.215209 (0.167206) | 3.801964 / 2.077655 (1.724309) | 1.887680 / 1.504120 (0.383560) | 1.775903 / 1.541195 (0.234709) | 1.851338 / 1.468490 (0.382848) | 0.483616 / 4.584777 (-4.101161) | 3.612977 / 3.745712 (-0.132736) | 3.397700 / 5.269862 (-1.872162) | 2.114572 / 4.565676 (-2.451105) | 0.057250 / 0.424275 (-0.367025) | 0.007362 / 0.007607 (-0.000245) | 0.456873 / 0.226044 (0.230829) | 4.567319 / 2.268929 (2.298391) | 2.399476 / 55.444624 (-53.045148) | 2.054542 / 6.876477 (-4.821935) | 2.343432 / 2.142072 (0.201359) | 0.582319 / 4.805227 (-4.222908) | 0.134045 / 6.500664 (-6.366619) | 0.062726 / 0.075469 (-0.012743) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283390 / 1.841788 (-0.558398) | 20.358511 / 8.074308 (12.284202) | 14.933989 / 10.191392 (4.742597) | 0.164960 / 0.680424 (-0.515464) | 0.018625 / 0.534201 (-0.515576) | 0.394087 / 0.579283 (-0.185196) | 0.416761 / 0.434364 (-0.017603) | 0.466669 / 0.540337 (-0.073669) | 0.643161 / 1.386936 (-0.743775) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007141 / 0.011353 (-0.004212) | 0.004185 / 0.011008 (-0.006824) | 0.066097 / 0.038508 (0.027588) | 0.088436 / 0.023109 (0.065327) | 0.401189 / 0.275898 (0.125291) | 0.440402 / 0.323480 (0.116922) | 0.005729 / 0.007986 (-0.002257) | 0.003527 / 0.004328 (-0.000801) | 0.065278 / 0.004250 (0.061027) | 0.060866 / 0.037052 (0.023813) | 0.407035 / 0.258489 (0.148546) | 0.443923 / 0.293841 (0.150083) | 0.032922 / 0.128546 (-0.095625) | 0.008739 / 0.075646 (-0.066907) | 0.071800 / 0.419271 (-0.347472) | 0.048994 / 0.043533 (0.005461) | 0.403736 / 0.255139 (0.148597) | 0.419566 / 0.283200 (0.136366) | 0.025369 / 0.141683 (-0.116314) | 1.474980 / 1.452155 (0.022825) | 1.553500 / 1.492716 (0.060784) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.225224 / 0.018006 (0.207218) | 0.462891 / 0.000490 (0.462401) | 0.006958 / 0.000200 (0.006758) | 0.000163 / 0.000054 (0.000108) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.034431 / 0.037411 (-0.002980) | 0.100021 / 0.014526 (0.085495) | 0.108339 / 0.176557 (-0.068217) | 0.162762 / 0.737135 (-0.574374) | 0.108951 / 0.296338 (-0.187388) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.435966 / 0.215209 (0.220757) | 4.351744 / 2.077655 (2.274089) | 2.372307 / 1.504120 (0.868187) | 2.192146 / 1.541195 (0.650951) | 2.326839 / 1.468490 (0.858349) | 0.488292 / 4.584777 (-4.096485) | 3.745227 / 3.745712 (-0.000485) | 3.456306 / 5.269862 (-1.813556) | 2.159771 / 4.565676 (-2.405906) | 0.057953 / 0.424275 (-0.366322) | 0.007469 / 0.007607 (-0.000138) | 0.515116 / 0.226044 (0.289071) | 5.162871 / 2.268929 (2.893942) | 2.850336 / 55.444624 (-52.594288) | 2.514700 / 6.876477 (-4.361777) | 2.748843 / 2.142072 (0.606770) | 0.587687 / 4.805227 (-4.217540) | 0.134333 / 6.500664 (-6.366331) | 0.062097 / 0.075469 (-0.013372) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.377082 / 1.841788 (-0.464705) | 21.103127 / 8.074308 (13.028819) | 15.325275 / 10.191392 (5.133883) | 0.166225 / 0.680424 (-0.514199) | 0.020472 / 0.534201 (-0.513729) | 0.395866 / 0.579283 (-0.183417) | 0.444964 / 0.434364 (0.010600) | 0.475367 / 0.540337 (-0.064970) | 0.693325 / 1.386936 (-0.693611) |

</details>
</details>

![](https://cml.dev/watermark.png#79b5bbbd52ffd90dd958c05b333d7c90a03756cc "CML watermark")