Extend text to support yielding lines, paragraphs or documents
Opened by albertvillanova (closed)

Add `config.row` option to `text` module to allow yielding lines (default, current case), paragraphs or documents.

Feel free to comment on the name of the config parameter `row`:
- Currently, the docs state datasets are made of rows and columns
- Other names I considered: `example`, `item`

Comments:
The parameter can also be named `split_by` with values "line", "paragraph" or "document" (no 's' at the end)

> The parameter can also be named `split_by` with values "line", "paragraph" or "document" (no 's' at the end)

@lhoestq @mariosasko I would avoid the term `split` in this context and keep it only for "train", "validation" and "test" splits.
- https://huggingface.co/docs/datasets/process.html#split
  > datasets.Dataset.train_test_split() creates train and test splits, if your dataset doesn’t already have them.
- https://huggingface.co/docs/datasets/process.html#process-multiple-splits
  > Many datasets have splits that you can process simultaneously with datasets.DatasetDict.map().

Please note that in the documentation, one of the terms more frequently used in this context is **"row"**:
- https://huggingface.co/docs/datasets/access.html#features-and-columns
  > A dataset is a table of rows and typed columns.

  > Return the number of rows and columns with the following standard attributes:
  > dataset.num_columns
  > 4
  > dataset.num_rows
  > 3668

- https://huggingface.co/docs/datasets/access.html#rows-slices-batches-and-columns
  > Get several rows of your dataset at a time with slice notation or a list of indices:
- https://huggingface.co/docs/datasets/process.html#map
  > This function can even create new rows and columns.

Other of the terms more frequently used in the docs (in the code as well) is **"example"**:
- https://huggingface.co/docs/datasets/process.html#map
  > It allows you to apply a processing function to each example in a dataset, independently or in batches.
- https://huggingface.co/docs/datasets/process.html#batch-processing
  > datasets.Dataset.map() also supports working with batches of examples.
- https://huggingface.co/docs/datasets/process.html#split-long-examples
  > When your examples are too long, you may want to split them
- https://huggingface.co/docs/datasets/process.html#data-augmentation
  > With batch processing, you can even augment your dataset with additional examples.

Less frequently used: **"item"**:
- https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.add_item
  > Add item to Dataset.

Other term used in the docs (although less frequently) is **"sample"**. The advantage of this word is that it is also a verb, so we can use the parameter: "sample_by" (if you insist on using a verb instead of a noun).

In summary, these proposals:
- config.row
- config.example
- config.item
- config.sample
- config.sample_by

I like `sample_by`. Another idea I had was `separate_by`.

It could also be `sampling`, `sampling_method`, `separation_method`.

Not a big fan of the proposed nouns alone since they are very generic, that's why I tried to have something more specific.

I also agree that we actually should avoid `split` to avoid any confusion

Thanks for the analysis of the used terms. I also like `sample_by` (`separate_by` is good too).

Thank you !! :D