Add code examples to API docs
State: closed | Author: stevhliu (MEMBER)
Labels: name: documentation

This PR adds code examples for functions related to the base Datasets class to highlight usage. Most of the examples use the `rotten_tomatoes` dataset since it is nice and small. Several things I would appreciate feedback on:

- Do you think it is clearer to make every code example fully reproducible so when users copy the code they can actually run it and get an output? This seems quite repetitive - maybe even unnecessary - but it is definitely clearer. Personally, I think we might be able to get away with not including this since users probably want to try the function on their own dataset. For example:

   ```py
   >>> from datasets import load_dataset
   >>> ds = load_dataset("rotten_tomatoes", split="validation")
   >>> code example goes here
   ```

- Should we showcase a function with more than one parameter to highlight different use-cases (it's pretty basic right now, but I'd be happy to add more)?
- For the `class_encode_column` function, let me know if there is a simpler dataset with fewer columns (currently using `winograd_wsc`) so it is easier for users to see what changed.
- Where possible, I try to show the input before and the output after using a function like `flatten` for example. Do you think this is too much and just showing the usage (ie, `>>> ds.flatten()`) will be sufficient?

Thanks :)

Comments: _The documentation is not available anymore as the PR was closed or merged._

> Do you think it is clearer to make every code example fully reproducible so when users copy the code they can actually run it and get an output? This seems quite repetitive - maybe even unnecessary - but it is definitely clearer.

I think it's ok to be repetitive to get more clarity. Many users come from `transformers` and may have little experience with some processing methods (especially torch users).

> Should we showcase a function with more than one parameter to highlight different use-cases (it's pretty basic right now, but I'd be happy to add more)?

Maybe let's do it case by case, depending on whether there are parameters that are likely to be used often ?

> For the class_encode_column function, let me know if there is a simpler dataset with fewer columns (currently using winograd_wsc) so it is easier for users to see what changed.

You can try with `boolq`, it has a boolean column that can be converted to labels

> Where possible, I try to show the input before and the output after using a function like flatten for example. Do you think this is too much and just showing the usage (ie, >>> ds.flatten()) will be sufficient?

No I don't think it's too much, it's nice this way thanks :)

Updated each code example so they are fully reproducible (where applicable)! The next step will be to identify some functions where we can show off some parameters that are useful or commonly used. Some useful parameters can be:

- use `map(batched=True)` to process batches of examples.
- set a seed in `shuffle`.
- set `shuffle` and `seed` in `train_test_split`.

Let me know if you think of anything else related to the functions in `arrow_dataset.py`!

Cool thanks ! I think you can also do `num_proc` for `map`