GitHub issue #33: Big cleanup/refactoring for clean serialization
State: closed | Created: 2020-05-01T23:45:57Z

This PR cleans many base classes to re-build them as `dataclasses`. We can thus use a simple serialization workflow for `DatasetInfo`, including it's `Features` and `SplitDict` based on `dataclasses` `asdict()`.

The resulting code is a lot shorter, can be easily serialized/deserialized, dataset info are human-readable and we can get rid of the `dataclass_json` dependency.

The scripts have breaking changes and the conversion tool is updated.

Example of dataset info in SQuAD script now:
```python
  def _info(self):
    return nlp.DatasetInfo(
        description=_DESCRIPTION,
        features=nlp.Features({
            "id":
                nlp.Value('string'),
            "title":
                nlp.Value('string'),
            "context":
                nlp.Value('string'),
            "question":
                nlp.Value('string'),
            "answers":
                nlp.Sequence({
                    "text": nlp.Value('string'),
                    "answer_start": nlp.Value('int32'),
                }),
        }),
        # No default supervised_keys (as we have to pass both question
        # and context as input).
        supervised_keys=None,
        homepage="https://rajpurkar.github.io/SQuAD-explorer/",
        citation=_CITATION,
    )
```

Example of serialized dataset info:
```bash
{
    "description": "Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n",
    "citation": "@article{2016arXiv160605250R,\n             author = {{Rajpurkar}, Pranav and {Zhang}, Jian and {Lopyrev},\n                                 Konstantin and {Liang}, Percy},\n                title = \"{SQuAD: 100,000+ Questions for Machine Comprehension of Text}\",\n            journal = {arXiv e-prints},\n                 year = 2016,\n                    eid = {arXiv:1606.05250},\n                pages = {arXiv:1606.05250},\narchivePrefix = {arXiv},\n             eprint = {1606.05250},\n}\n",
    "homepage": "https://rajpurkar.github.io/SQuAD-explorer/",
    "license": "",
    "features": {
        "id": {
            "dtype": "string",
            "_type": "Value"
        },
        "title": {
            "dtype": "string",
            "_type": "Value"
        },
        "context": {
            "dtype": "string",
            "_type": "Value"
        },
        "question": {
            "dtype": "string",
            "_type": "Value"
        },
        "answers": {
            "feature": {
                "text": {
                    "dtype": "string",
                    "_type": "Value"
                },
                "answer_start": {
                    "dtype": "int32",
                    "_type": "Value"
                }
            },
            "length": -1,
            "_type": "Sequence"
        }
    },
    "supervised_keys": null,
    "name": "squad",
    "version": {
        "version_str": "1.0.0",
        "description": "New split API (https://tensorflow.org/datasets/splits)",
        "nlp_version_to_prepare": null,
        "major": 1,
        "minor": 0,
        "patch": 0
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 79426386,
            "num_examples": 87599,
            "dataset_name": "squad"
        },
        "validation": {
            "name": "validation",
            "num_bytes": 10491883,
            "num_examples": 10570,
            "dataset_name": "squad"
        }
    },
    "size_in_bytes": 0,
    "download_size": 35142551,
    "download_checksums": []
}
```