Repository: huggingface/datasets
Pull request: Dataset infos in yaml
To simplify the addition of new datasets, we'd like to have the dataset infos in the YAML and deprecate the dataset_infos.json file. YAML is readable and easy to edit, and the YAML metadata of the readme already contain dataset metadata so we would have everything in one place.

To be more specific, I moved these fields from DatasetInfo to the YAML:
- config_name (if there are several configs)
- download_size
- dataset_size
- features
- splits

Here is what I ended up with for `squad`:
```yaml
dataset_info:
  features:
  - name: id
    dtype: string
  - name: title
    dtype: string
  - name: context
    dtype: string
  - name: question
    dtype: string
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: answer_start
      dtype: int32
  splits:
  - name: train
    num_bytes: 79346360
    num_examples: 87599
  - name: validation
    num_bytes: 10473040
    num_examples: 10570
  config_name: plain_text
  download_size: 35142551
  dataset_size: 89819400
```

and it can be a list if there are several configs

I already did the change for `conll2000` and `crime_and_punish` as an example.

## Implementation details

### Load/Read

This is done via `DatasetInfosDict.write_to_directory/from_directory`

I had to implement a custom the YAML export logic for `SplitDict`, `Version` and `Features`.
The first two are trivial, but the logic for `Features` is more complicated, because I added a simplification step (or the YAML would be too long and less readable): it's just a formatting step to remove unnecessary nesting of YAML data.

### Other changes

I had to update the DatasetModule factories to also download the README.md alongside the dataset scripts/data files, and not just the dataset_infos.json

## YAML validation

I removed the old validation code that was in metadata.py, now we can just use the Hub YAML validation

## Datasets-cli

The `datasets-cli test --save_infos` command now creates a README.md file with the dataset_infos in it, instead of a datasets_infos.json file

## Backward compatibility

`dataset_infos.json` files are still supported and loaded if they exist to have full backward compatibility.
Though I removed the unnecessary keys when the value is the default (like all the `id: null` from the Value feature types) to make them easier to read.

## TODO

- [x] add comments
- [x] tests
- [x] document the new YAML fields 
- [x] try to reload the new dataset_infos.json file content with an old version of `datasets`

## EDITS

- removed "config_name" when there's only one config
- removed "version" for now (?), because it's not useful in general
- renamed the yaml field "dataset_info" instead of "dataset_infos", since it only has one by default (and because "infos" is not english)

Fix https://github.com/huggingface/datasets/issues/4876 and fix #2773

Problem statement:
Remove dataset_infos.json
**Is your feature request related to a problem? Please describe.**
As discussed, there are infos in the `dataset_infos.json` which are redundant and we could have them only in the README file.

Others could be migrated to the README, like: "dataset_size", "size_in_bytes", "download_size", "splits.split_name.[num_bytes, num_examples]",...

However, there are others that do not seem too meaningful in the README, like the checksums.

**Describe the solution you'd like**
Open a discussion to decide what to do with the `dataset_infos.json` files: which information to be migrated and/or which information to be kept.

cc: @julien-c @lhoestq 

Move DatasetInfo from `datasets_infos.json` to the YAML tags in `README.md`
Currently there are two places to find metadata for datasets:
- datasets_infos.json, which contains **per dataset config**
  - description
  - citation
  - license
  - splits and sizes
  - checksums of the data files
  - feature types
  - and more
- YAML tags, which contain
  - license
  - language
  - train-eval-index
  - and more

It would be nice to have a single place instead. We can rely on the YAML tags more than the JSON file for consistency with models. And it would all be indexed by our back-end directly, which is nice to have. 

One way would be to move everything to the YAML tags except the checksums (there can be tens of thousands of them). The description/citation is already in the dataset card so we probably don't need to have them in the YAML card, it would be redundant.

Here is an example for SQuAD
```yaml

download_size: 35142551
dataset_size: 89789763
version: 1.0.0
splits:
- name: train
  num_examples: 87599
  num_bytes: 79317110
- name: validation
  num_examples: 10570
  num_bytes: 10472653
features:
- name: id
  dtype: string
- name: title
  dtype: string
- name: context
  dtype: string
- name: question
  dtype: string
- name: answers
  struct:
  - name: text
    list:
      dtype: string
  - name: answer_start
    list:
      dtype: int32
```

Since there is only one configuration for SQuAD, this structure is ok. For datasets with several configs we can see in a second step, but IMO it would be ok to have these fields per config using another syntax
```yaml
configs:
- config: unlabeled
  splits:
  - name: train
    num_examples: 10000
  features:
  - name: text
    dtype: string
- config: labeled
  splits:
  - name: train
    num_examples: 100
  features:
  - name: text
    dtype: string
  - name: label
    dtype: ClassLabel
    names:
    - negative
    - positive
```
So in the end you could specify a YAML tag either at the top level (for all configs) or per config in the `configs` field

Alternatively we could keep config specific stuff in the `dataset_infos.json` as it it today

Not sure yet what's the best approach here but cc @julien-c @mariosasko @albertvillanova @polinaeterna for feedback :)

Hints from issue discussion:
also @osanseviero @Pierrci @SBrandeis potentially
Love this in principle 🚀 

Let's keep in mind users might rely on `dataset_infos.json` already.

I'm not convinced by the two-syntax solution, wouldn't it be simpler to have only one syntax with a `default` config for datasets with only one config? ie, always having the `configs` field. This makes parsing the metadata easier IMO.

Might also be good to wrap the tags under a `datasets_info` tag as follows:

```yaml
description: ...
citation: ...
dataset_infos:
  download_size: 35142551
  dataset_size: 89789763
  version: 1.0.0
  configs:
    - ...
[...]
```

Let's also keep in mind that extracting YAML metadata from a markdown readme is a bit more fastidious for users than just parsing a JSON file.
> Let's keep in mind users might rely on dataset_infos.json already.

Yea we'll full full backward compatibility

> Let's also keep in mind that extracting YAML metadata from a markdown readme is a bit more fastidious for users than just parsing a JSON file.

The main things that may use or ingest these data IMO are:
- users in the UI or IDE
- `datasets` to populate `DatasetInfo` python object
- moon landing which is already parsing YAML

Am I missing something ? If not I think it's ok to use YAML

> Might also be good to wrap the tags under a datasets_info tag as follows:

Maybe one single syntax like this then ?
```yaml
dataset_infos:
- config: unlabeled
  download_size: 35142551
  dataset_size: 89789763
  version: 1.0.0
  splits:
  - name: train
    num_examples: 10000
  features:
  - name: text
    dtype: string
- config: labeled
  download_size: 35142551
  dataset_size: 89789763
  version: 1.0.0
  splits:
  - name: train
    num_examples: 100
  features:
  - name: text
    dtype: string
  - name: label
    dtype: ClassLabel
    names:
    - negative
    - positive
```
and when you have only one config
```yaml
dataset_infos:
- config: default
  splits:
  - name: train
    num_examples: 10000
  features:
  - name: text
    dtype: string
```
love the idea, and the trend in general to move more things (like tasks) to a single place (YAML).

also, if you browse files on a dataset's page (in "Files and versions"), raw `README.md` files looks nice and readable, while `.json` files are just one long line that users need to scroll. 

> Let's also keep in mind that extracting YAML metadata from a markdown readme is a bit more fastidious for users than just parsing a JSON file.

do users often parse `datasets_infos.json` file themselves? 
> do users often parse datasets_infos.json file themselves?

Not AFAIK, but I'm sure there should be a few users.
Users that access these info via the `DatasetInfo` from `datasets` won't see the change though e.g.
```python
>> from datasets import get_datasets_infos
>>> get_datasets_infos("squad")
{'plain_text': DatasetInfo(description='Stanford Question Answering Dataset...
```
> Maybe one single syntax like this then ?

LGTM!

> The main things that may use or ingest these data IMO are:
> - users in the UI or IDE
> - datasets to populate DatasetInfo python object
> - moon landing which is already parsing YAML

Fair point!

Having dataset info in the README's YAML is great for API / `huggingface_hub` consumers as well as it will be inserted in the  `cardData` field out of the box 🔥 

Very supportive of this!

Nesting an array of configs inside `dataset_infos: ` sounds good to me. One small tweak is that `config: default` can be optional for the default config (which can be the first one by convention)

We'll be able to implement metadata validation on the Hub side so we ensure that those metadata are always in the right format (maybe for @coyotte508 ? cc @Pierrci). From a quick glance the `features` might be the harder part to validate here, any doc will be welcome.

### Other high-level points:
- as we move from mostly academic datasets to *all* datasets (which include the data inside the repos), my intuition is that more and more datasets (Hub-stored) are going to be **single-config**
- similarly, less and less datasets will have a loading script, **just the data + some metadata**
- to lower the barrier to entry to contribution, in the long term users shouldn't need to compute/update this data via a command line. It could be filled automatically on the Hub through a "bot" inside Discussions & Pull requests for instance.
re: `config: default`

Note also that the default config is not named `default`, afaiu, but create from the repo name, eg: https://huggingface.co/datasets/nbtpj/bionlp2021SAS default config is `nbtpj--bionlp2021SAS` (which is awful)
> Note also that the default config is not named default, afaiu, but create from the repo name, eg: https://huggingface.co/datasets/nbtpj/bionlp2021SAS default config is nbtpj--bionlp2021SAS (which is awful)

We can change this to `default` I think or something else
> From a quick glance the features might be the harder part to validate here, any doc will be welcome.

I dug into features validation, see:

- the OpenAPI spec: https://github.com/huggingface/datasets-server/blob/main/chart/static-files/openapi.json#L460-L697
- the node.js code: https://github.com/huggingface/moon-landing/blob/upgrade-datasets-server-client/server/lib/datasets/FeatureType.ts

Commit message: fix utf-8 yaml dump

Patch to review:
diff --git a/.github/PULL_REQUEST_TEMPLATE/add_dataset.md b/.github/PULL_REQUEST_TEMPLATE/add_dataset.md
index e569839d829..38269e2456d 100644
--- a/.github/PULL_REQUEST_TEMPLATE/add_dataset.md
+++ b/.github/PULL_REQUEST_TEMPLATE/add_dataset.md
@@ -6,11 +6,9 @@
 
 ### Checkbox
 
-- [ ] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
+- [ ] Create the dataset script `./my_dataset/my_dataset.py` using the template
 - [ ] Fill the `_DESCRIPTION` and `_CITATION` variables
 - [ ] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
 - [ ] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
-- [ ] Generate the metadata file `dataset_infos.json` for all configurations
-- [ ] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
 - [ ] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
-- [ ] Both tests for the real data and the dummy data pass.
+- [ ] Optional - test the dataset using `datasets-cli test ./dataset_name --save_infos`
diff --git a/ADD_NEW_DATASET.md b/ADD_NEW_DATASET.md
index fd4a22f4eb4..ad6f57ab6c3 100644
--- a/ADD_NEW_DATASET.md
+++ b/ADD_NEW_DATASET.md
@@ -63,8 +63,6 @@ You are now ready to start the process of adding the dataset. We will create the
 
 - a **dataset script** which contains the code to download and pre-process the dataset: e.g. `squad.py`,
 - a **dataset card** with tags and information on the dataset in a `README.md`.
-- a **metadata file** (automatically created) which contains checksums and information about the dataset to guarantee that the loading went fine: `dataset_infos.json`
-- a **dummy-data file** (automatically created) which contains small examples from the original files to test and guarantee that the script is working well in the future: `dummy_data.zip`
 
 2. Let's start by creating a new branch to hold your development changes with the name of your dataset:
 
@@ -166,7 +164,9 @@ Sometimes you need to use several *configurations* and/or *splits* (usually at l
 - if some of you dataset features are in a fixed set of classes (e.g. labels), you should use a `ClassLabel` feature.
 
 
-**Last step:** To check that your dataset works correctly and to create its `dataset_infos.json` file run the command:
+#### Tests (optional)
+
+ To check that your dataset works correctly and to create its `dataset_infos` metadata in the dataset card, run the command:
 
 ```bash
 datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
@@ -229,13 +229,13 @@ Now that your dataset script runs and create a dataset with the format you expec
 	```
 	to enable the slow tests, instead of `RUN_SLOW=1`.
 
-3. If all tests pass, your dataset works correctly. You can finally create the metadata JSON by running the command:
+3. If all tests pass, your dataset works correctly. You can finally create the metadata by running the command:
 
 	```bash
 	datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
 	```
 
-	This first command should create a `dataset_infos.json` file in your dataset folder.
+	This first command should create a `README.md` file in your dataset folder if it doesn't exist already, containing the metadata.
 
 
 You have now finished the coding part, congratulation! 🎉 You are Awesome! 😎
diff --git a/datasets/conll2000/README.md b/datasets/conll2000/README.md
index d2e0832abc6..288b3ba98f4 100644
--- a/datasets/conll2000/README.md
+++ b/datasets/conll2000/README.md
@@ -3,6 +3,96 @@ language:
 - en
 paperswithcode_id: conll-2000-1
 pretty_name: CoNLL-2000
+dataset_info:
+  features:
+  - name: id
+    dtype: string
+  - name: tokens
+    sequence: string
+  - name: pos_tags
+    sequence:
+      class_label:
+        names:
+          0: ''''''
+          1: '#'
+          2: $
+          3: (
+          4: )
+          5: ','
+          6: .
+          7: ':'
+          8: '``'
+          9: CC
+          10: CD
+          11: DT
+          12: EX
+          13: FW
+          14: IN
+          15: JJ
+          16: JJR
+          17: JJS
+          18: MD
+          19: NN
+          20: NNP
+          21: NNPS
+          22: NNS
+          23: PDT
+          24: POS
+          25: PRP
+          26: PRP$
+          27: RB
+          28: RBR
+          29: RBS
+          30: RP
+          31: SYM
+          32: TO
+          33: UH
+          34: VB
+          35: VBD
+          36: VBG
+          37: VBN
+          38: VBP
+          39: VBZ
+          40: WDT
+          41: WP
+          42: WP$
+          43: WRB
+  - name: chunk_tags
+    sequence:
+      class_label:
+        names:
+          0: O
+          1: B-ADJP
+          2: I-ADJP
+          3: B-ADVP
+          4: I-ADVP
+          5: B-CONJP
+          6: I-CONJP
+          7: B-INTJ
+          8: I-INTJ
+          9: B-LST
+          10: I-LST
+          11: B-NP
+          12: I-NP
+          13: B-PP
+          14: I-PP
+          15: B-PRT
+          16: I-PRT
+          17: B-SBAR
+          18: I-SBAR
+          19: B-UCP
+          20: I-UCP
+          21: B-VP
+          22: I-VP
+  splits:
+  - name: test
+    num_bytes: 1201151
+    num_examples: 2013
+  - name: train
+    num_bytes: 5356965
+    num_examples: 8937
+  download_size: 3481560
+  dataset_size: 6558116
 ---
 
 # Dataset Card for "conll2000"
@@ -173,4 +263,4 @@ The data fields are the same among all splits.
 
 ### Contributions
 
-Thanks to [@vblagoje](https://github.com/vblagoje), [@jplu](https://github.com/jplu) for adding this dataset.
+Thanks to [@vblagoje](https://github.com/vblagoje), [@jplu](https://github.com/jplu) for adding this dataset.
\ No newline at end of file
diff --git a/datasets/conll2000/conll2000.py b/datasets/conll2000/conll2000.py
index 996e01d46da..d349fa477cb 100644
--- a/datasets/conll2000/conll2000.py
+++ b/datasets/conll2000/conll2000.py
@@ -53,25 +53,9 @@
 _TEST_FILE = "test.txt"
 
 
-class Conll2000Config(datasets.BuilderConfig):
-    """BuilderConfig for Conll2000"""
-
-    def __init__(self, **kwargs):
-        """BuilderConfig forConll2000.
-
-        Args:
-          **kwargs: keyword arguments forwarded to super.
-        """
-        super(Conll2000Config, self).__init__(**kwargs)
-
-
 class Conll2000(datasets.GeneratorBasedBuilder):
     """Conll2000 dataset."""
 
-    BUILDER_CONFIGS = [
-        Conll2000Config(name="conll2000", version=datasets.Version("1.0.0"), description="Conll2000 dataset"),
-    ]
-
     def _info(self):
         return datasets.DatasetInfo(
             description=_DESCRIPTION,
diff --git a/datasets/conll2000/dataset_infos.json b/datasets/conll2000/dataset_infos.json
deleted file mode 100644
index 2373f4aae0c..00000000000
--- a/datasets/conll2000/dataset_infos.json
+++ /dev/null
@@ -1 +0,0 @@
-{"conll2000": {"description": " Text chunking consists of dividing a text in syntactically correlated parts of words. For example, the sentence\n He reckons the current account deficit will narrow to only # 1.8 billion in September . can be divided as follows:\n[NP He ] [VP reckons ] [NP the current account deficit ] [VP will narrow ] [PP to ] [NP only # 1.8 billion ]\n[PP in ] [NP September ] .\n\nText chunking is an intermediate step towards full parsing. It was the shared task for CoNLL-2000. Training and test\ndata for this task is available. This data consists of the same partitions of the Wall Street Journal corpus (WSJ)\nas the widely used data for noun phrase chunking: sections 15-18 as training data (211727 tokens) and section 20 as\ntest data (47377 tokens). The annotation of the data has been derived from the WSJ corpus by a program written by\nSabine Buchholz from Tilburg University, The Netherlands.\n", "citation": "@inproceedings{tksbuchholz2000conll,\n   author     = \"Tjong Kim Sang, Erik F. and Sabine Buchholz\",\n   title      = \"Introduction to the CoNLL-2000 Shared Task: Chunking\",\n   editor     = \"Claire Cardie and Walter Daelemans and Claire\n                 Nedellec and Tjong Kim Sang, Erik\",\n   booktitle  = \"Proceedings of CoNLL-2000 and LLL-2000\",\n   publisher  = \"Lisbon, Portugal\",\n   pages      = \"127--132\",\n   year       = \"2000\"\n}\n", "homepage": "https://www.clips.uantwerpen.be/conll2000/chunking/", "license": "", "features": {"id": {"dtype": "string", "id": null, "_type": "Value"}, "tokens": {"feature": {"dtype": "string", "id": null, "_type": "Value"}, "length": -1, "id": null, "_type": "Sequence"}, "pos_tags": {"feature": {"num_classes": 44, "names": ["''", "#", "$", "(", ")", ",", ".", ":", "``", "CC", "CD", "DT", "EX", "FW", "IN", "JJ", "JJR", "JJS", "MD", "NN", "NNP", "NNPS", "NNS", "PDT", "POS", "PRP", "PRP$", "RB", "RBR", "RBS", "RP", "SYM", "TO", "UH", "VB", "VBD", "VBG", "VBN", "VBP", "VBZ", "WDT", "WP", "WP$", "WRB"], "names_file": null, "id": null, "_type": "ClassLabel"}, "length": -1, "id": null, "_type": "Sequence"}, "chunk_tags": {"feature": {"num_classes": 23, "names": ["O", "B-ADJP", "I-ADJP", "B-ADVP", "I-ADVP", "B-CONJP", "I-CONJP", "B-INTJ", "I-INTJ", "B-LST", "I-LST", "B-NP", "I-NP", "B-PP", "I-PP", "B-PRT", "I-PRT", "B-SBAR", "I-SBAR", "B-UCP", "I-UCP", "B-VP", "I-VP"], "names_file": null, "id": null, "_type": "ClassLabel"}, "length": -1, "id": null, "_type": "Sequence"}}, "post_processed": null, "supervised_keys": null, "builder_name": "conll2000", "config_name": "conll2000", "version": {"version_str": "1.0.0", "description": null, "major": 1, "minor": 0, "patch": 0}, "splits": {"train": {"name": "train", "num_bytes": 5356965, "num_examples": 8937, "dataset_name": "conll2000"}, "test": {"name": "test", "num_bytes": 1201151, "num_examples": 2013, "dataset_name": "conll2000"}}, "download_checksums": {"https://github.com/teropa/nlp/raw/master/resources/corpora/conll2000/train.txt": {"num_bytes": 2842164, "checksum": "82033cd7a72b209923a98007793e8f9de3abc1c8b79d646c50648eb949b87cea"}, "https://github.com/teropa/nlp/raw/master/resources/corpora/conll2000/test.txt": {"num_bytes": 639396, "checksum": "73b7b1e565fa75a1e22fe52ecdf41b6624d6f59dacb591d44252bf4d692b1628"}}, "download_size": 3481560, "post_processing_size": null, "dataset_size": 6558116, "size_in_bytes": 10039676}}
\ No newline at end of file
diff --git a/datasets/conll2000/dummy/conll2000/1.0.0/dummy_data.zip b/datasets/conll2000/dummy/0.0.0/dummy_data.zip
similarity index 100%
rename from datasets/conll2000/dummy/conll2000/1.0.0/dummy_data.zip
rename to datasets/conll2000/dummy/0.0.0/dummy_data.zip
diff --git a/datasets/crime_and_punish/README.md b/datasets/crime_and_punish/README.md
index 6fb932d8bf9..deaffe451b3 100644
--- a/datasets/crime_and_punish/README.md
+++ b/datasets/crime_and_punish/README.md
@@ -3,6 +3,16 @@ language:
 - en
 paperswithcode_id: null
 pretty_name: CrimeAndPunish
+dataset_info:
+  dataset_size: 1270540
+  download_size: 1201735
+  features:
+  - dtype: string
+    name: line
+  splits:
+  - name: train
+    num_bytes: 1270540
+    num_examples: 21969
 ---
 
 # Dataset Card for "crime_and_punish"
@@ -144,4 +154,4 @@ The data fields are the same among all splits.
 
 ### Contributions
 
-Thanks to [@patrickvonplaten](https://github.com/patrickvonplaten), [@thomwolf](https://github.com/thomwolf) for adding this dataset.
+Thanks to [@patrickvonplaten](https://github.com/patrickvonplaten), [@thomwolf](https://github.com/thomwolf) for adding this dataset.
\ No newline at end of file
diff --git a/datasets/crime_and_punish/crime_and_punish.py b/datasets/crime_and_punish/crime_and_punish.py
index eefa7f2959d..6170741dac0 100644
--- a/datasets/crime_and_punish/crime_and_punish.py
+++ b/datasets/crime_and_punish/crime_and_punish.py
@@ -8,36 +8,7 @@
 _DATA_URL = "https://raw.githubusercontent.com/patrickvonplaten/datasets/master/crime_and_punishment.txt"
 
 
-class CrimeAndPunishConfig(datasets.BuilderConfig):
-    """BuilderConfig for Crime and Punish."""
-
-    def __init__(self, data_url, **kwargs):
-        """BuilderConfig for BlogAuthorship
-
-        Args:
-          data_url: `string`, url to the dataset (word or raw level)
-          **kwargs: keyword arguments forwarded to super.
-        """
-        super(CrimeAndPunishConfig, self).__init__(
-            version=datasets.Version(
-                "1.0.0",
-            ),
-            **kwargs,
-        )
-        self.data_url = data_url
-
-
 class CrimeAndPunish(datasets.GeneratorBasedBuilder):
-
-    VERSION = datasets.Version("0.1.0")
-    BUILDER_CONFIGS = [
-        CrimeAndPunishConfig(
-            name="crime-and-punish",
-            data_url=_DATA_URL,
-            description="word level dataset. No processing is needed other than replacing newlines with <eos> tokens.",
-        ),
-    ]
-
     def _info(self):
         return datasets.DatasetInfo(
             # This is the description that will appear on the datasets page.
@@ -58,17 +29,14 @@ def _info(self):
     def _split_generators(self, dl_manager):
         """Returns SplitGenerators."""
 
-        if self.config.name == "crime-and-punish":
-            data = dl_manager.download_and_extract(self.config.data_url)
+        data = dl_manager.download_and_extract(_DATA_URL)
 
-            return [
-                datasets.SplitGenerator(
-                    name=datasets.Split.TRAIN,
-                    gen_kwargs={"data_file": data, "split": "train"},
-                ),
-            ]
-        else:
-            raise ValueError(f"{self.config.name} does not exist")
+        return [
+            datasets.SplitGenerator(
+                name=datasets.Split.TRAIN,
+                gen_kwargs={"data_file": data, "split": "train"},
+            ),
+        ]
 
     def _generate_examples(self, data_file, split):
 
diff --git a/datasets/crime_and_punish/dataset_infos.json b/datasets/crime_and_punish/dataset_infos.json
deleted file mode 100644
index f653f0f67a9..00000000000
--- a/datasets/crime_and_punish/dataset_infos.json
+++ /dev/null
@@ -1 +0,0 @@
-{"crime-and-punish": {"description": "\n", "citation": "", "homepage": "https://www.gutenberg.org/files/2554/2554-h/2554-h.htm", "license": "", "features": {"line": {"dtype": "string", "id": null, "_type": "Value"}}, "supervised_keys": null, "builder_name": "crime_and_punish", "config_name": "crime-and-punish", "version": {"version_str": "1.0.0", "description": null, "datasets_version_to_prepare": null, "major": 1, "minor": 0, "patch": 0}, "splits": {"train": {"name": "train", "num_bytes": 1270540, "num_examples": 21969, "dataset_name": "crime_and_punish"}}, "download_checksums": {"https://raw.githubusercontent.com/patrickvonplaten/datasets/master/crime_and_punishment.txt": {"num_bytes": 1201735, "checksum": "3582bcff83e5e24ae5acb2935a191ea5ead66b11fc12fa19b0397834e8296c83"}}, "download_size": 1201735, "dataset_size": 1270540, "size_in_bytes": 2472275}}
\ No newline at end of file
diff --git a/datasets/crime_and_punish/dummy/crime-and-punish/1.0.0/dummy_data.zip b/datasets/crime_and_punish/dummy/0.0.0/dummy_data.zip
similarity index 100%
rename from datasets/crime_and_punish/dummy/crime-and-punish/1.0.0/dummy_data.zip
rename to datasets/crime_and_punish/dummy/0.0.0/dummy_data.zip
diff --git a/docs/source/dataset_card.mdx b/docs/source/dataset_card.mdx
index b62f3a73ba7..c460f453fba 100644
--- a/docs/source/dataset_card.mdx
+++ b/docs/source/dataset_card.mdx
@@ -24,4 +24,87 @@ Feel free to take a look at these dataset card examples to help you get started:
 - [CNN / DailyMail](https://huggingface.co/datasets/cnn_dailymail)
 - [Allociné](https://huggingface.co/datasets/allocine)
 
-You can also check out the (similar) documentation about [dataset cards on the Hub side](https://huggingface.co/docs/hub/datasets-cards).
\ No newline at end of file
+You can also check out the (similar) documentation about [dataset cards on the Hub side](https://huggingface.co/docs/hub/datasets-cards).
+
+## More YAML tags
+
+You can use the `dataset_info` YAML fields to define additional metadata for the dataset. Here is an example for SQuAD:
+
+```YAML
+pretty_name: SQuAD
+language:
+- en
+...
+dataset_info:
+  features:
+  - name: id
+    dtype: string
+  - name: title
+    dtype: string
+  - name: context
+    dtype: string
+  - name: question
+    dtype: string
+  - name: answers
+    sequence:
+    - name: text
+      dtype: string
+    - name: answer_start
+      dtype: int32
+  splits:
+  - name: train
+    num_bytes: 79346360
+    num_examples: 87599
+  - name: validation
+    num_bytes: 10473040
+    num_examples: 10570
+  download_size: 35142551
+  dataset_size: 89819400
+```
+
+These metadata used to be included in the `dataset_infos.json` file, which is now deprecated.
+
+### Feature types
+
+Using the `features` field you can explicitly define the feature types of your dataset.
+This is especially useful when type inference is not obvious.
+For example if there's only one non-empty example in a 1TB dataset, the type inference is not able to infer the type of each column without going through the full dataset.
+In this case, specifying the `features` field makes type inference much easier.
+
+### Split sizes
+
+Specifying the split sizes with `num_examples` enables TQDM bars (otherwise it doesn't know how many examples are left).
+It also enables integrity verifications: if the dataset doesn't have the right number of `num_examples`, an error is returned.
+
+Additionally you can add `num_bytes` to specify how big each split is.
+
+### Dataset size
+
+When [`load_dataset`] is called, it first downloads the dataset raw data files, and then it prepares the dataset in Arrow format.
+
+You can specify how many bytes are required to download the raw data files with `dataset_size`, and use `dataset_size` for the size of the dataset in Arrow format.
+
+### Multiple configurations
+
+Certain datasets like `glue` have several configurations (`cola`, `sst2`, etc.) that can be loaded using `load_dataset("glue", "cola")` for example.
+
+Each configuration can have different features, splits and sizes.
+You can specify those fields per configuration using a YAML list:
+
+```YAML
+dataset_info:
+- config_name: cola
+  features:
+    ...
+  splits:
+    ...
+  download_size: ...
+  dataset_size: ...
+- config_name: sst2
+  features:
+    ...
+  splits:
+    ...
+  download_size: ...
+  dataset_size: ...
+```
diff --git a/docs/source/dataset_script.mdx b/docs/source/dataset_script.mdx
index 03f32a06c8a..65024977218 100644
--- a/docs/source/dataset_script.mdx
+++ b/docs/source/dataset_script.mdx
@@ -274,17 +274,17 @@ def _generate_examples(self, filepath):
                     }
 ```
 
-## Generate dataset metadata
+## (Optional) Generate dataset metadata
 
-Adding dataset metadata is a great way to include information about your dataset. The metadata is stored in a `dataset_infos.json` file. It includes information like data file checksums, the number of examples required to confirm the dataset was correctly generated, and information about the dataset like its `features`.
+Adding dataset metadata is a great way to include information about your dataset. The metadata is stored in the dataset card `README.md` in YAML. It includes information like the number of examples required to confirm the dataset was correctly generated, and information about the dataset like its `features`.
 
-Run the following command to generate your dataset metadata in `dataset_infos.json` and make sure your new dataset loading script works correctly:
+Run the following command to generate your dataset metadata in `README.md` and make sure your new dataset loading script works correctly:
 
 ```
 datasets-cli test path/to/<your-dataset-loading-script> --save_infos --all_configs
 ```
 
-If your dataset loading script passed the test, you should now have a `dataset_infos.json` file in your dataset folder.
+If your dataset loading script passed the test, you should now have a `README.md` file in your dataset folder containing a `dataset_infos` field with some metadata.
 
 ## Upload to the Hub
 
diff --git a/docs/source/share.mdx b/docs/source/share.mdx
index 641889f52bc..2fc73db2253 100644
--- a/docs/source/share.mdx
+++ b/docs/source/share.mdx
@@ -81,8 +81,6 @@ Here the `namespace` is either your username or your organization name.
 
 - `your_dataset_name.py` is your dataset loading script (optional if your data files are already in the supported formats csv/jsonl/json/parquet/txt). To create a dataset script, see the [dataset script](dataset_script) page.
 
-- `dataset_infos.json` contains metadata about the dataset (required only if you have a dataset script, or if you want to specify custom feature types).
-
 ### Upload your files
 
 You can directly upload your files from your repository on the Hugging Face Hub, but this guide will show you how to upload the files from the terminal.
@@ -97,10 +95,10 @@ git add *.json
 git commit -m "add json files"
 ```
 
-6. Add the dataset loading script and metadata file:
+6. Add the dataset loading script and dataset card:
 
 ```
-cp /somewhere/data/dataset_infos.json .
+cp /somewhere/data/README.md .
 cp /somewhere/data/load_script.py .
 git add --all
 ```
diff --git a/setup.py b/setup.py
index 509042e18c6..4633d9c35e5 100644
--- a/setup.py
+++ b/setup.py
@@ -93,6 +93,8 @@
     # Utilities from PyPA to e.g., compare versions
     "packaging",
     "responses<0.19",
+    # To parse YAML metadata from dataset cards
+    "pyyaml>=5.1",
 ]
 
 AUDIO_REQUIRE = [
diff --git a/src/datasets/builder.py b/src/datasets/builder.py
index e79bd53e3fc..be4b3fb065f 100644
--- a/src/datasets/builder.py
+++ b/src/datasets/builder.py
@@ -358,7 +358,7 @@ def __init__(
         # Set download manager
         self.dl_manager = None
 
-        # Record infos even if verify_infos=False; used by "datasets-cli test" to generate dataset_infos.json
+        # Record infos even if verify_infos=False; used by "datasets-cli test" to generate file checksums for (deprecated) dataset_infos.json
         self._record_infos = False
 
         # Enable streaming (e.g. it patches "open" to work with remote files)
@@ -383,7 +383,7 @@ def manual_download_instructions(self) -> Optional[str]:
         return None
 
     @classmethod
-    def get_all_exported_dataset_infos(cls) -> dict:
+    def get_all_exported_dataset_infos(cls) -> DatasetInfosDict:
         """Empty dict if doesn't exist
 
         Example:
@@ -395,10 +395,7 @@ def get_all_exported_dataset_infos(cls) -> dict:
         {'default': DatasetInfo(description="Movie Review Dataset.\nThis is a dataset of containing 5,331 positive and 5,331 negative processed\nsentences from Rotten Tomatoes movie reviews. This data was first used in Bo\nPang and Lillian Lee, ``Seeing stars: Exploiting class relationships for\nsentiment categorization with respect to rating scales.'', Proceedings of the\nACL, 2005.\n", citation='@InProceedings{Pang+Lee:05a,\n  author =       {Bo Pang and Lillian Lee},\n  title =        {Seeing stars: Exploiting class relationships for sentiment\n                  categorization with respect to rating scales},\n  booktitle =    {Proceedings of the ACL},\n  year =         2005\n}\n', homepage='http://www.cs.cornell.edu/people/pabo/movie-review-data/', license='', features={'text': Value(dtype='string', id=None), 'label': ClassLabel(num_classes=2, names=['neg', 'pos'], id=None)}, post_processed=None, supervised_keys=SupervisedKeysData(input='', output=''), task_templates=[TextClassification(task='text-classification', text_column='text', label_column='label')], builder_name='rotten_tomatoes_movie_review', config_name='default', version=1.0.0, splits={'train': SplitInfo(name='train', num_bytes=1074810, num_examples=8530, dataset_name='rotten_tomatoes_movie_review'), 'validation': SplitInfo(name='validation', num_bytes=134679, num_examples=1066, dataset_name='rotten_tomatoes_movie_review'), 'test': SplitInfo(name='test', num_bytes=135972, num_examples=1066, dataset_name='rotten_tomatoes_movie_review')}, download_checksums={'https://storage.googleapis.com/seldon-datasets/sentence_polarity_v1/rt-polaritydata.tar.gz': {'num_bytes': 487770, 'checksum': 'a05befe52aafda71d458d188a1c54506a998b1308613ba76bbda2e5029409ce9'}}, download_size=487770, post_processing_size=None, dataset_size=1345461, size_in_bytes=1833231)}
         ```
         """
-        dset_infos_file_path = os.path.join(cls.get_imported_module_dir(), config.DATASETDICT_INFOS_FILENAME)
-        if os.path.exists(dset_infos_file_path):
-            return DatasetInfosDict.from_directory(cls.get_imported_module_dir())
-        return {}
+        return DatasetInfosDict.from_directory(cls.get_imported_module_dir())
 
     def get_exported_dataset_info(self) -> DatasetInfo:
         """Empty DatasetInfo if doesn't exist
@@ -468,11 +465,14 @@ def _create_builder_config(self, name=None, custom_features=None, **config_kwarg
             config_kwargs,
             custom_features=custom_features,
         )
-        is_custom = config_id not in self.builder_configs
+        is_custom = (config_id not in self.builder_configs) and config_id != "default"
         if is_custom:
             logger.warning(f"Using custom data configuration {config_id}")
         else:
-            if builder_config != self.builder_configs[builder_config.name]:
+            if (
+                builder_config.name in self.builder_configs
+                and builder_config != self.builder_configs[builder_config.name]
+            ):
                 raise ValueError(
                     "Cannot name a custom BuilderConfig the same as an available "
                     f"BuilderConfig. Change the name. Available BuilderConfigs: {list(self.builder_configs.keys())}"
diff --git a/src/datasets/commands/test.py b/src/datasets/commands/test.py
index 59fa9690e47..c0829993b47 100644
--- a/src/datasets/commands/test.py
+++ b/src/datasets/commands/test.py
@@ -50,7 +50,9 @@ def register_subcommand(parser: ArgumentParser):
             help="Can be used to specify a manual directory to get the files from.",
         )
         test_parser.add_argument("--all_configs", action="store_true", help="Test all dataset configurations")
-        test_parser.add_argument("--save_infos", action="store_true", help="Save the dataset infos file")
+        test_parser.add_argument(
+            "--save_infos", action="store_true", help="Save the dataset infos in the dataset card (README.md)"
+        )
         test_parser.add_argument(
             "--ignore_verifications", action="store_true", help="Run the test without checksums and splits checks"
         )
@@ -145,13 +147,13 @@ def get_builders() -> Generator[DatasetBuilder, None, None]:
             if self._save_infos:
                 builder._save_infos()
 
-            # If save_infos=True, the dataset infos file is created next to the loaded module file.
+            # If save_infos=True, the dataset card (README.md) is created next to the loaded module file.
+            # The dataset_infos are saved in the YAML part of the README.md
+
             # Let's move it to the original directory of the dataset script, to allow the user to
             # upload them on S3 at the same time afterwards.
             if self._save_infos:
-                dataset_infos_path = os.path.join(
-                    builder_cls.get_imported_module_dir(), datasets.config.DATASETDICT_INFOS_FILENAME
-                )
+                dataset_readme_path = os.path.join(builder_cls.get_imported_module_dir(), "README.md")
                 name = Path(path).name + ".py"
                 combined_path = os.path.join(path, name)
                 if os.path.isfile(path):
@@ -162,13 +164,13 @@ def get_builders() -> Generator[DatasetBuilder, None, None]:
                     dataset_dir = path
                 else:  # in case of a remote dataset
                     dataset_dir = None
-                    print(f"Dataset Infos file saved at {dataset_infos_path}")
+                    print(f"Dataset card saved at {dataset_readme_path}")
 
                 # Move dataset_info back to the user
                 if dataset_dir is not None:
-                    user_dataset_infos_path = os.path.join(dataset_dir, datasets.config.DATASETDICT_INFOS_FILENAME)
-                    copyfile(dataset_infos_path, user_dataset_infos_path)
-                    print(f"Dataset Infos file saved at {user_dataset_infos_path}")
+                    user_dataset_readme_path = os.path.join(dataset_dir, "README.md")
+                    copyfile(dataset_readme_path, user_dataset_readme_path)
+                    print(f"Dataset card saved at {user_dataset_readme_path}")
 
             # If clear_cache=True, the download folder and the dataset builder cache directory are deleted
             if self._clear_cache:
diff --git a/src/datasets/features/features.py b/src/datasets/features/features.py
index f9d5eb29509..0d2e310ee9a 100644
--- a/src/datasets/features/features.py
+++ b/src/datasets/features/features.py
@@ -35,6 +35,7 @@
 from pandas.api.extensions import ExtensionDtype as PandasExtensionDtype
 
 from .. import config
+from ..naming import camelcase_to_snakecase, snakecase_to_camelcase
 from ..table import array_cast
 from ..utils import logging
 from ..utils.py_utils import asdict, first_non_null_value, zip_dict
@@ -887,7 +888,7 @@ class ClassLabel:
     ```
     """
 
-    num_classes: int = None
+    num_classes: InitVar[Optional[int]] = None  # Pseudo-field: ignored by asdict/fields when converting to/from dict
     names: List[str] = None
     names_file: InitVar[Optional[str]] = None  # Pseudo-field: ignored by asdict/fields when converting to/from dict
     id: Optional[str] = None
@@ -898,7 +899,8 @@ class ClassLabel:
     _int2str: ClassVar[Dict[int, int]] = None
     _type: str = field(default="ClassLabel", init=False, repr=False)
 
-    def __post_init__(self, names_file):
+    def __post_init__(self, num_classes, names_file):
+        self.num_classes = num_classes
         self.names_file = names_file
         if self.names_file is not None and self.names is not None:
             raise ValueError("Please provide either names or names_file but not both.")
@@ -1281,7 +1283,7 @@ def generate_from_dict(obj: Any):
     class_type = globals()[obj.pop("_type")]
 
     if class_type == Sequence:
-        return Sequence(feature=generate_from_dict(obj["feature"]), length=obj["length"])
+        return Sequence(feature=generate_from_dict(obj["feature"]), length=obj.get("length", -1))
 
     field_names = {f.name for f in fields(class_type)}
     return class_type(**{k: v for k, v in obj.items() if k in field_names})
@@ -1600,6 +1602,148 @@ def from_dict(cls, dic) -> "Features":
     def to_dict(self):
         return asdict(self)
 
+    def _to_yaml_list(self) -> list:
+        # we compute the YAML list from the dict representation that is used for JSON dump
+        yaml_data = self.to_dict()
+
+        def simplify(feature: dict) -> dict:
+            if not isinstance(feature, dict):
+                raise TypeError(f"Expected a dict but got a {type(feature)}: {feature}")
+
+            #
+            # sequence:                 ->              sequence: int32
+            #   dtype: int32            ->
+            #
+            if isinstance(feature.get("sequence"), dict) and list(feature["sequence"]) == ["dtype"]:
+                feature["sequence"] = feature["sequence"]["dtype"]
+
+            #
+            # sequence:                 ->              sequence:
+            #   struct:                 ->              - name: foo
+            #   - name: foo             ->                dtype: int32
+            #     dtype: int32          ->
+            #
+            if isinstance(feature.get("sequence"), dict) and list(feature["sequence"]) == ["struct"]:
+                feature["sequence"] = feature["sequence"]["struct"]
+
+            #
+            # list:                     ->              list: int32
+            #   dtype: int32            ->
+            #
+            if isinstance(feature.get("list"), dict) and list(feature["list"]) == ["dtype"]:
+                feature["list"] = feature["list"]["dtype"]
+
+            #
+            # list:                     ->              list:
+            #   struct:                 ->              - name: foo
+            #   - name: foo             ->                dtype: int32
+            #     dtype: int32          ->
+            #
+            if isinstance(feature.get("list"), dict) and list(feature["list"]) == ["struct"]:
+                feature["list"] = feature["list"]["struct"]
+
+            #
+            # class_label:              ->              class_label:
+            #   names:                  ->                names:
+            #   - negative              ->                  0: negative
+            #   - positive              ->                  1: positive
+            #
+            if isinstance(feature.get("class_label"), dict) and isinstance(feature["class_label"].get("names"), list):
+                feature["class_label"]["names"] = dict(enumerate(feature["class_label"]["names"]))
+            return feature
+
+        def to_yaml_inner(obj: Union[dict, list]) -> dict:
+            if isinstance(obj, dict):
+                _type = obj.pop("_type", None)
+                if _type == "Sequence":
+                    _feature = obj.pop("feature")
+                    return simplify({"sequence": to_yaml_inner(_feature), **obj})
+                elif _type == "Value":
+                    return obj
+                elif _type and not obj:
+                    return {"dtype": camelcase_to_snakecase(_type)}
+                elif _type:
+                    return {"dtype": simplify({camelcase_to_snakecase(_type): obj})}
+                else:
+                    return {"struct": [{"name": name, **to_yaml_inner(_feature)} for name, _feature in obj.items()]}
+            elif isinstance(obj, list):
+                return simplify({"list": simplify(to_yaml_inner(obj[0]))})
+            else:
+                raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")
+
+        return to_yaml_inner(yaml_data)["struct"]
+
+    @classmethod
+    def _from_yaml_list(cls, yaml_data: list) -> "Features":
+        yaml_data = copy.deepcopy(yaml_data)
+
+        # we convert the list obtained from YAML data into the dict representation that is used for JSON dump
+
+        def unsimplify(feature: dict) -> dict:
+            if not isinstance(feature, dict):
+                raise TypeError(f"Expected a dict but got a {type(feature)}: {feature}")
+            #
+            # sequence: int32           ->              sequence:
+            #                           ->                dtype: int32
+            #
+            if isinstance(feature.get("sequence"), str):
+                feature["sequence"] = {"dtype": feature["sequence"]}
+            #
+            # list: int32               ->              list:
+            #                           ->                dtype: int32
+            #
+            if isinstance(feature.get("list"), str):
+                feature["list"] = {"dtype": feature["list"]}
+
+            #
+            # class_label:              ->              class_label:
+            #   names:                  ->                names:
+            #     0: negative              ->               - negative
+            #     1: positive              ->               - positive
+            #
+            if isinstance(feature.get("class_label"), dict) and isinstance(feature["class_label"].get("names"), dict):
+                label_ids = sorted(feature["class_label"]["names"])
+                if label_ids and label_ids != list(range(label_ids[-1] + 1)):
+                    raise ValueError(
+                        f"ClassLabel expected a value for all label ids [0:{label_ids[-1] + 1}] but some ids are missing."
+                    )
+                feature["class_label"]["names"] = [feature["class_label"]["names"][label_id] for label_id in label_ids]
+            return feature
+
+        def from_yaml_inner(obj: Union[dict, list]) -> Union[dict, list]:
+
+            if isinstance(obj, dict):
+                if not obj:
+                    return {}
+                _type = next(iter(obj))
+                if _type == "sequence":
+                    _feature = unsimplify(obj).pop(_type)
+                    return {"feature": from_yaml_inner(_feature), **obj, "_type": "Sequence"}
+                if _type == "list":
+                    return [from_yaml_inner(unsimplify(obj)[_type])]
+                if _type == "struct":
+                    return from_yaml_inner(obj["struct"])
+                elif _type == "dtype":
+                    if isinstance(obj["dtype"], str):
+                        # e.g. int32, float64, string, audio, image
+                        try:
+                            Value(obj["dtype"])
+                            return {**obj, "_type": "Value"}
+                        except ValueError:
+                            # for audio and image that are Audio and Image types, not Value
+                            return {"_type": snakecase_to_camelcase(obj["dtype"])}
+                    else:
+                        return from_yaml_inner(obj["dtype"])
+                else:
+                    return {"_type": snakecase_to_camelcase(_type), **unsimplify(obj)[_type]}
+            elif isinstance(obj, list):
+                names = [_feature.pop("name") for _feature in obj]
+                return {name: from_yaml_inner(_feature) for name, _feature in zip(names, obj)}
+            else:
+                raise TypeError(f"Expected a dict or a list but got {type(obj)}: {obj}")
+
+        return cls.from_dict(from_yaml_inner(yaml_data))
+
     def encode_example(self, example):
         """
         Encode example into a format for Arrow.
diff --git a/src/datasets/info.py b/src/datasets/info.py
index 2b5acfdde92..4e9c128a467 100644
--- a/src/datasets/info.py
+++ b/src/datasets/info.py
@@ -33,8 +33,9 @@
 import json
 import os
 import posixpath
-from dataclasses import dataclass, field
-from typing import Dict, List, Optional, Union
+from dataclasses import dataclass
+from pathlib import Path
+from typing import ClassVar, Dict, List, Optional, Union
 
 from fsspec.implementations.local import LocalFileSystem
 
@@ -45,6 +46,7 @@
 from .tasks import TaskTemplate, task_template_from_dict
 from .utils import Version
 from .utils.logging import get_logger
+from .utils.metadata import DatasetMetadata
 from .utils.py_utils import asdict, unique_values
 
 
@@ -118,10 +120,10 @@ class DatasetInfo:
     """
 
     # Set in the dataset scripts
-    description: str = field(default_factory=str)
-    citation: str = field(default_factory=str)
-    homepage: str = field(default_factory=str)
-    license: str = field(default_factory=str)
+    description: str = dataclasses.field(default_factory=str)
+    citation: str = dataclasses.field(default_factory=str)
+    homepage: str = dataclasses.field(default_factory=str)
+    license: str = dataclasses.field(default_factory=str)
     features: Optional[Features] = None
     post_processed: Optional[PostProcessedInfo] = None
     supervised_keys: Optional[SupervisedKeysData] = None
@@ -139,6 +141,14 @@ class DatasetInfo:
     dataset_size: Optional[int] = None
     size_in_bytes: Optional[int] = None
 
+    _INCLUDED_INFO_IN_YAML: ClassVar[List[str]] = [
+        "config_name",
+        "download_size",
+        "dataset_size",
+        "features",
+        "splits",
+    ]
+
     def __post_init__(self):
         # Convert back to the correct classes when we reload from dict
         if self.features is not None and not isinstance(self.features, Features):
@@ -301,32 +311,103 @@ def update(self, other_dataset_info: "DatasetInfo", ignore_none=True):
     def copy(self) -> "DatasetInfo":
         return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
 
+    def _to_yaml_dict(self) -> dict:
+        yaml_dict = {}
+        dataset_info_dict = asdict(self)
+        for key in dataset_info_dict:
+            if key in self._INCLUDED_INFO_IN_YAML:
+                value = getattr(self, key)
+                if hasattr(value, "_to_yaml_list"):  # Features, SplitDict
+                    yaml_dict[key] = value._to_yaml_list()
+                elif hasattr(value, "_to_yaml_string"):  # Version
+                    yaml_dict[key] = value._to_yaml_string()
+                else:
+                    yaml_dict[key] = value
+        return yaml_dict
+
+    @classmethod
+    def _from_yaml_dict(cls, yaml_data: dict) -> "DatasetInfo":
+        yaml_data = copy.deepcopy(yaml_data)
+        if yaml_data.get("features") is not None:
+            yaml_data["features"] = Features._from_yaml_list(yaml_data["features"])
+        if yaml_data.get("splits") is not None:
+            yaml_data["splits"] = SplitDict._from_yaml_list(yaml_data["splits"])
+        field_names = {f.name for f in dataclasses.fields(cls)}
+        return cls(**{k: v for k, v in yaml_data.items() if k in field_names})
+
 
 class DatasetInfosDict(Dict[str, DatasetInfo]):
     def write_to_directory(self, dataset_infos_dir, overwrite=False, pretty_print=False):
         total_dataset_infos = {}
         dataset_infos_path = os.path.join(dataset_infos_dir, config.DATASETDICT_INFOS_FILENAME)
-        if os.path.exists(dataset_infos_path) and not overwrite:
-            logger.info(f"Dataset Infos already exists in {dataset_infos_dir}. Completing it with new infos.")
+        dataset_readme_path = os.path.join(dataset_infos_dir, "README.md")
+        if not overwrite:
             total_dataset_infos = self.from_directory(dataset_infos_dir)
-        else:
-            logger.info(f"Writing new Dataset Infos in {dataset_infos_dir}")
         total_dataset_infos.update(self)
-        with open(dataset_infos_path, "w", encoding="utf-8") as f:
-            json.dump(
-                {config_name: asdict(dset_info) for config_name, dset_info in total_dataset_infos.items()},
-                f,
-                indent=4 if pretty_print else None,
-            )
+        if os.path.exists(dataset_infos_path):
+            # for backward compatibility, let's update the JSON file if it exists
+            with open(dataset_infos_path, "w", encoding="utf-8") as f:
+                dataset_infos_dict = {
+                    config_name: asdict(dset_info) for config_name, dset_info in total_dataset_infos.items()
+                }
+                json.dump(dataset_infos_dict, f, indent=4 if pretty_print else None)
+        # Dump the infos in the YAML part of the README.md file
+        if os.path.exists(dataset_readme_path):
+            dataset_metadata = DatasetMetadata.from_readme(Path(dataset_readme_path))
+        else:
+            dataset_metadata = DatasetMetadata()
+        if total_dataset_infos:
+            total_dataset_infos = {
+                config_name: dset_info._to_yaml_dict() for config_name, dset_info in total_dataset_infos.items()
+            }
+            # the config_name from the dataset_infos_dict takes over the config_name of the DatasetInfo
+            for config_name, dset_info_yaml_dict in total_dataset_infos.items():
+                dset_info_yaml_dict["config_name"] = config_name
+            if len(total_dataset_infos) == 1:
+                # use a struct instead of a list of configurations, since there's only one
+                dataset_metadata["dataset_info"] = next(iter(total_dataset_infos.values()))
+                # no need to include the configuration name when there's only one configuration and it's called "default"
+                if dataset_metadata["dataset_info"].get("config_name") == "default":
+                    dataset_metadata["dataset_info"].pop("config_name", None)
+            else:
+                dataset_metadata["dataset_info"] = []
+                for config_name, dataset_info_yaml_dict in total_dataset_infos.items():
+                    # add the config_name field in first position
+                    dataset_info_yaml_dict.pop("config_name", None)
+                    dataset_info_yaml_dict = {"config_name": config_name, **dataset_info_yaml_dict}
+                    dataset_metadata["dataset_info"].append(dataset_info_yaml_dict)
+
+            dataset_metadata.to_readme(Path(dataset_readme_path))
 
     @classmethod
     def from_directory(cls, dataset_infos_dir):
         logger.info(f"Loading Dataset Infos from {dataset_infos_dir}")
-        with open(os.path.join(dataset_infos_dir, config.DATASETDICT_INFOS_FILENAME), encoding="utf-8") as f:
-            dataset_infos_dict = {
-                config_name: DatasetInfo.from_dict(dataset_info_dict)
-                for config_name, dataset_info_dict in json.load(f).items()
-            }
+        dataset_infos_dict = {}
+        if os.path.exists(os.path.join(dataset_infos_dir, config.DATASETDICT_INFOS_FILENAME)):
+            # this is just to have backward compatibility with dataset_infos.json files
+            with open(os.path.join(dataset_infos_dir, config.DATASETDICT_INFOS_FILENAME), encoding="utf-8") as f:
+                dataset_infos_dict.update(
+                    {
+                        config_name: DatasetInfo.from_dict(dataset_info_dict)
+                        for config_name, dataset_info_dict in json.load(f).items()
+                    }
+                )
+        # Load the info from the YAML part of README.md
+        if os.path.exists(os.path.join(dataset_infos_dir, "README.md")):
+            dataset_metadata = DatasetMetadata.from_readme(Path(dataset_infos_dir) / "README.md")
+            if isinstance(dataset_metadata.get("dataset_info"), (list, dict)):
+                if isinstance(dataset_metadata["dataset_info"], list):
+                    dataset_infos_dict = {
+                        dataset_info_yaml_dict.get("config_name", "default"): DatasetInfo._from_yaml_dict(
+                            dataset_info_yaml_dict
+                        )
+                        for dataset_info_yaml_dict in dataset_metadata["dataset_info"]
+                    }
+                else:
+                    dataset_info = DatasetInfo._from_yaml_dict(dataset_metadata["dataset_info"])
+                    dataset_info.config_name = dataset_metadata["dataset_info"].get("config_name", "default")
+                    dataset_infos_dict = {dataset_info.config_name: dataset_info}
+
         return cls(**dataset_infos_dict)
 
 
@@ -344,11 +425,11 @@ class MetricInfo:
     description: str
     citation: str
     features: Features
-    inputs_description: str = field(default_factory=str)
-    homepage: str = field(default_factory=str)
-    license: str = field(default_factory=str)
-    codebase_urls: List[str] = field(default_factory=list)
-    reference_urls: List[str] = field(default_factory=list)
+    inputs_description: str = dataclasses.field(default_factory=str)
+    homepage: str = dataclasses.field(default_factory=str)
+    license: str = dataclasses.field(default_factory=str)
+    codebase_urls: List[str] = dataclasses.field(default_factory=list)
+    reference_urls: List[str] = dataclasses.field(default_factory=list)
     streamable: bool = False
     format: Optional[str] = None
 
diff --git a/src/datasets/load.py b/src/datasets/load.py
index ed4ea48a9d8..522bfd5ab4e 100644
--- a/src/datasets/load.py
+++ b/src/datasets/load.py
@@ -78,6 +78,7 @@
 from .utils.filelock import FileLock
 from .utils.info_utils import is_small_dataset
 from .utils.logging import get_logger
+from .utils.metadata import DatasetMetadata
 from .utils.py_utils import get_imports
 from .utils.version import Version
 
@@ -568,6 +569,7 @@ def __init__(
     def get_module(self) -> DatasetModule:
         # get script and other files
         dataset_infos_path = Path(self.path).parent / config.DATASETDICT_INFOS_FILENAME
+        dataset_readme_path = Path(self.path).parent / "README.md"
         imports = get_imports(self.path)
         local_imports = _download_additional_modules(
             name=self.name,
@@ -575,9 +577,11 @@ def get_module(self) -> DatasetModule:
             imports=imports,
             download_config=self.download_config,
         )
-        additional_files = (
-            [(config.DATASETDICT_INFOS_FILENAME, str(dataset_infos_path))] if dataset_infos_path.is_file() else []
-        )
+        additional_files = []
+        if dataset_infos_path.is_file():
+            additional_files.append((config.DATASETDICT_INFOS_FILENAME, str(dataset_infos_path)))
+        if dataset_readme_path.is_file():
+            additional_files.append(("README.md", dataset_readme_path))
         # copy the script and the files in an importable directory
         dynamic_modules_path = self.dynamic_modules_path if self.dynamic_modules_path else init_dynamic_modules()
         module_path, hash = _create_importable_file(
@@ -661,8 +665,16 @@ def get_module(self) -> DatasetModule:
         if os.path.isfile(os.path.join(self.path, config.DATASETDICT_INFOS_FILENAME)):
             with open(os.path.join(self.path, config.DATASETDICT_INFOS_FILENAME), encoding="utf-8") as f:
                 dataset_infos: DatasetInfosDict = json.load(f)
-            builder_kwargs["config_name"] = next(iter(dataset_infos))
-            builder_kwargs["info"] = DatasetInfo.from_dict(dataset_infos[builder_kwargs["config_name"]])
+            if dataset_infos:
+                builder_kwargs["config_name"] = next(iter(dataset_infos))
+                builder_kwargs["info"] = DatasetInfo.from_dict(next(iter(dataset_infos.values())))
+        if os.path.isfile(os.path.join(self.path, "README.md")):
+            dataset_metadata = DatasetMetadata.from_readme(Path(self.path) / "README.md")
+            if isinstance(dataset_metadata.get("dataset_info"), list) and dataset_metadata["dataset_info"]:
+                dataset_info_dict = dataset_metadata["dataset_info"][0]
+                builder_kwargs["info"] = DatasetInfo.from_dict(dataset_info_dict)
+                if "config_name" in dataset_info_dict:
+                    builder_kwargs["config_name"] = dataset_info_dict["config_name"]
         return DatasetModule(module_path, hash, builder_kwargs)
 
 
@@ -808,8 +820,25 @@ def get_module(self) -> DatasetModule:
             )
             with open(dataset_infos_path, encoding="utf-8") as f:
                 dataset_infos: DatasetInfosDict = json.load(f)
-            builder_kwargs["config_name"] = next(iter(dataset_infos))
-            builder_kwargs["info"] = DatasetInfo.from_dict(dataset_infos[builder_kwargs["config_name"]])
+            if dataset_infos:
+                builder_kwargs["config_name"] = next(iter(dataset_infos))
+                builder_kwargs["info"] = DatasetInfo.from_dict(next(iter(dataset_infos.values())))
+        except FileNotFoundError:
+            pass
+        download_config = self.download_config.copy()
+        if download_config.download_desc is None:
+            download_config.download_desc = "Downloading readme"
+        try:
+            dataset_readme_path = cached_path(
+                hf_hub_url(self.name, "README.md", revision=self.revision),
+                download_config=self.download_config,
+            )
+            dataset_metadata = DatasetMetadata.from_readme(Path(dataset_readme_path))
+            if isinstance(dataset_metadata.get("dataset_info"), list) and dataset_metadata["dataset_info"]:
+                dataset_info_dict = dataset_metadata["dataset_info"][0]
+                builder_kwargs["info"] = DatasetInfo.from_dict(dataset_info_dict)
+                if "config_name" in dataset_info_dict:
+                    builder_kwargs["config_name"] = dataset_info_dict["config_name"]
         except FileNotFoundError:
             pass
         return DatasetModule(module_path, hash, builder_kwargs)
@@ -854,10 +883,25 @@ def download_dataset_infos_file(self) -> str:
         except (FileNotFoundError, ConnectionError):
             return None
 
+    def download_dataset_readme_file(self) -> str:
+        readme_url = hf_hub_url(repo_id=self.name, path="README.md", revision=self.revision)
+        # Download the dataset infos file if available
+        download_config = self.download_config.copy()
+        if download_config.download_desc is None:
+            download_config.download_desc = "Downloading readme"
+        try:
+            return cached_path(
+                readme_url,
+                download_config=download_config,
+            )
+        except (FileNotFoundError, ConnectionError):
+            return None
+
     def get_module(self) -> DatasetModule:
         # get script and other files
         local_path = self.download_loading_script()
         dataset_infos_path = self.download_dataset_infos_file()
+        dataset_readme_path = self.download_dataset_readme_file()
         imports = get_imports(local_path)
         local_imports = _download_additional_modules(
             name=self.name,
@@ -865,7 +909,11 @@ def get_module(self) -> DatasetModule:
             imports=imports,
             download_config=self.download_config,
         )
-        additional_files = [(config.DATASETDICT_INFOS_FILENAME, dataset_infos_path)] if dataset_infos_path else []
+        additional_files = []
+        if dataset_infos_path:
+            additional_files.append((config.DATASETDICT_INFOS_FILENAME, dataset_infos_path))
+        if dataset_readme_path:
+            additional_files.append(("README.md", dataset_readme_path))
         # copy the script and the files in an importable directory
         dynamic_modules_path = self.dynamic_modules_path if self.dynamic_modules_path else init_dynamic_modules()
         module_path, hash = _create_importable_file(
diff --git a/src/datasets/splits.py b/src/datasets/splits.py
index 4825ec77872..11912f1c951 100644
--- a/src/datasets/splits.py
+++ b/src/datasets/splits.py
@@ -18,13 +18,15 @@
 
 import abc
 import collections
+import copy
+import dataclasses
 import re
-from dataclasses import dataclass, field
+from dataclasses import dataclass
 from typing import Dict, List, Optional, Union
 
 from .arrow_reader import FileInstructions, make_file_instructions
 from .naming import _split_re
-from .utils.py_utils import NonMutableDict
+from .utils.py_utils import NonMutableDict, asdict
 
 
 @dataclass
@@ -32,7 +34,14 @@ class SplitInfo:
     name: str = ""
     num_bytes: int = 0
     num_examples: int = 0
-    dataset_name: Optional[str] = None
+
+    # Deprecated
+    # For backward compatibility, this field needs to always be included in files like
+    # dataset_infos.json and dataset_info.json files
+    # To do so, we always include it in the output of datasets.utils.py_utils.asdict(split_info)
+    dataset_name: Optional[str] = dataclasses.field(
+        default=None, metadata={"include_in_asdict_even_if_is_default": True}
+    )
 
     @property
     def file_instructions(self):
@@ -545,7 +554,7 @@ def from_split_dict(cls, split_infos: Union[List, Dict], dataset_name: Optional[
             split_infos = list(split_infos.values())
 
         if dataset_name is None:
-            dataset_name = split_infos[0]["dataset_name"] if split_infos else None
+            dataset_name = split_infos[0].get("dataset_name") if split_infos else None
 
         split_dict = cls(dataset_name=dataset_name)
 
@@ -559,11 +568,27 @@ def from_split_dict(cls, split_infos: Union[List, Dict], dataset_name: Optional[
     def to_split_dict(self):
         """Returns a list of SplitInfo protos that we have."""
         # Return the SplitInfo, sorted by name
-        return sorted((s for s in self.values()), key=lambda s: s.name)
+        out = []
+        for split_name, split_info in sorted(self.items()):
+            split_info = copy.deepcopy(split_info)
+            split_info.name = split_name
+            out.append(split_info)
+        return out
 
     def copy(self):
         return SplitDict.from_split_dict(self.to_split_dict(), self.dataset_name)
 
+    def _to_yaml_list(self) -> list:
+        out = [asdict(s) for s in self.to_split_dict()]
+        # we don't need the dataset_name attribute that is deprecated
+        for split_info_dict in out:
+            split_info_dict.pop("dataset_name", None)
+        return out
+
+    @classmethod
+    def _from_yaml_list(cls, yaml_data: list) -> "SplitDict":
+        return cls.from_split_dict(yaml_data)
+
 
 @dataclass
 class SplitGenerator:
@@ -591,8 +616,8 @@ class SplitGenerator:
     """
 
     name: str
-    gen_kwargs: Dict = field(default_factory=dict)
-    split_info: SplitInfo = field(init=False)
+    gen_kwargs: Dict = dataclasses.field(default_factory=dict)
+    split_info: SplitInfo = dataclasses.field(init=False)
 
     def __post_init__(self):
         self.name = str(self.name)  # Make sure we convert NamedSplits in strings
diff --git a/src/datasets/utils/metadata.py b/src/datasets/utils/metadata.py
index 0b847250779..c4c52d29044 100644
--- a/src/datasets/utils/metadata.py
+++ b/src/datasets/utils/metadata.py
@@ -1,58 +1,10 @@
-import json
-import logging
-import re
-import warnings
 from collections import Counter
-from dataclasses import dataclass, fields
 from pathlib import Path
-from typing import Any, Callable, ClassVar, Dict, List, Optional, Tuple, Type, Union
-
-
-try:  # Python >= 3.8
-    from typing import get_args, get_origin
-except ImportError:
-
-    def get_args(tp):
-        return getattr(tp, "__args__", ())
-
-    def get_origin(tp):
-        return getattr(tp, "__origin__", None)
-
-
-# loading package files: https://stackoverflow.com/a/20885799
-try:
-    import importlib.resources as pkg_resources
-except ImportError:
-    # Try backported to PY<37 `importlib_resources`.
-    import importlib_resources as pkg_resources
 
 import yaml
 
-from . import resources
-
-
-BASE_REF_URL = "https://github.com/huggingface/datasets/tree/main/src/datasets/utils"
-this_url = f"{BASE_REF_URL}/{__file__}"
-logger = logging.getLogger(__name__)
-
-
-def load_json_resource(resource: str) -> Tuple[Any, str]:
-    content = pkg_resources.read_text(resources, resource)
-    return json.loads(content), f"{BASE_REF_URL}/resources/{resource}"
-
 
-# Source of languages.json:
-# https://datahub.io/core/language-codes/r/ietf-language-tags.csv
-# Language names were obtained with langcodes: https://github.com/LuminosoInsight/langcodes
-known_language_codes, known_language_codes_url = load_json_resource("languages.json")
-known_task_ids, known_task_ids_url = load_json_resource("tasks.json")
-known_creators, known_creators_url = load_json_resource("creators.json")
-known_size_categories, known_size_categories_url = load_json_resource("size_categories.json")
-known_multilingualities, known_multilingualities_url = load_json_resource("multilingualities.json")
-known_source_datasets, known_source_datasets_url = ["original", "extended", r"extended\|.*"], this_url
-
-
-class NoDuplicateSafeLoader(yaml.SafeLoader):
+class _NoDuplicateSafeLoader(yaml.SafeLoader):
     def _check_no_duplicates_on_constructed_node(self, node):
         keys = [self.constructed_objects[key_node] for key_node, _ in node.value]
         keys = [tuple(key) if isinstance(key, list) else key for key in keys]
@@ -67,231 +19,22 @@ def construct_mapping(self, node, deep=False):
         return mapping
 
 
-def yaml_block_from_readme(path: Path) -> Optional[str]:
+def _split_yaml_from_readme(path: Path):
     with open(path, encoding="utf-8") as readme_file:
-        content = [line.rstrip("\n") for line in readme_file]
-
-    if content[0] == "---" and "---" in content[1:]:
-        yamlblock = "\n".join(content[1 : content[1:].index("---") + 1])
-        return yamlblock
-
-    return None
-
-
-def metadata_dict_from_readme(path: Path) -> Optional[Dict[str, List[str]]]:
-    """Loads a dataset's metadata from the dataset card (REAMDE.md), as a Python dict"""
-    yaml_block = yaml_block_from_readme(path=path)
-    if yaml_block is None:
-        return None
-    metada_dict = yaml.load(yaml_block, Loader=NoDuplicateSafeLoader) or dict()
-    return metada_dict
-
-
-ValidatorOutput = Tuple[List[str], Optional[str]]
-
-
-def tagset_validator(
-    items: Union[List[str], Dict[str, List[str]]],
-    reference_values: List[str],
-    name: str,
-    url: str,
-    escape_validation_predicate_fn: Optional[Callable[[Any], bool]] = None,
-) -> ValidatorOutput:
-    reference_values = re.compile("^(?:" + "|".join(reference_values) + ")$")
-    if isinstance(items, list):
-        if escape_validation_predicate_fn is not None:
-            invalid_values = [
-                v for v in items if not reference_values.match(v) and escape_validation_predicate_fn(v) is False
-            ]
-        else:
-            invalid_values = [v for v in items if not reference_values.match(v)]
-
-    else:
-        invalid_values = []
-        if escape_validation_predicate_fn is not None:
-            for config_name, values in items.items():
-                invalid_values += [
-                    v for v in values if not reference_values.match(v) and escape_validation_predicate_fn(v) is False
-                ]
-        else:
-            for config_name, values in items.items():
-                invalid_values += [v for v in values if not reference_values.match(v)]
-
-    if len(invalid_values) > 0:
-        return [], f"{invalid_values} are not registered tags for '{name}', reference at {url}"
-    return items, None
-
-
-def validate_type(value: Any, expected_type: Type):
-    error_string = ""
-    NoneType = type(None)
-    if expected_type is NoneType:
-        if not isinstance(value, NoneType):
-            return f"Expected `{NoneType}`. Found value: `{value}` of type `{type(value)}`.\n"
-        else:
-            return error_string
-    if expected_type is str:
-        if not isinstance(value, str):
-            return f"Expected `{str}`. Found value: `{value}` of type: `{type(value)}`.\n"
-
-        elif isinstance(value, str) and len(value) == 0:
-            return (
-                f"Expected `{str}` with length > 0. Found value: `{value}` of type: `{type(value)}` with length: 0.\n"
-            )
-        else:
-            return error_string
-    # Add more `elif` statements if primitive type checking is needed
-    else:
-        expected_type_origin = get_origin(expected_type)  # typing.List[str] -> list
-        expected_type_args = get_args(expected_type)  # typing.List[str] -> (str, )
-
-        if expected_type_origin is Union:
-            for type_arg in expected_type_args:
-                temp_error_string = validate_type(value, type_arg)
-                if temp_error_string == "":  # at least one type is successfully validated
-                    return temp_error_string
-                else:
-                    if error_string == "":
-                        error_string = "(" + temp_error_string + ")"
-                    else:
-                        error_string += "\nOR\n" + "(" + temp_error_string + ")"
-        elif value is None:
-            return f"Expected `{expected_type}`. Found value: `{value}` of type: `{type(value)}`.\n"
-        else:
-            # Assuming non empty `List`/`Dict`/`Tuple`
-            if expected_type is EmptyList:
-                if len(value) == 0:
-                    return ""
-                else:
-                    return f"Expected `{expected_type}` of length 0. Found value of type: `{type(value)}`, with length: {len(value)}.\n"
+        full_content = [line.rstrip("\n") for line in readme_file]
 
-            # Assuming non empty
-            if not isinstance(value, (dict, tuple, list)) or len(value) == 0:
-                return f"Expected `{expected_type}` with length > 0. Found value of type: `{type(value)}`, with length: {len(value)}.\n"
+    if full_content[0] == "---" and "---" in full_content[1:]:
+        sep_idx = full_content[1:].index("---") + 1
+        yamlblock = "\n".join(full_content[1:sep_idx])
+        return yamlblock, "\n".join(full_content[sep_idx + 1 :])
 
-            if expected_type_origin is dict:
-                if not isinstance(value, dict):
-                    return f"Expected `{expected_type}` with length > 0. Found value of type: `{type(value)}`, with length: {len(value)}.\n"
-                if expected_type_args != get_args(Dict):  # if we specified types for keys and values
-                    key_type, value_type = expected_type_args
-                    key_error_string = ""
-                    value_error_string = ""
-                    for k, v in value.items():
-                        key_error_string += validate_type(k, key_type)
-                        value_error_string += validate_type(v, value_type)
-                    if key_error_string != "" or value_error_string != "":
-                        return f"Typing errors with keys:\n {key_error_string} and values:\n {value_error_string}"
+    return None, "\n".join(full_content)
 
-            else:  # `List`/`Tuple`
-                if not isinstance(value, (list, tuple)):
-                    return f"Expected `{expected_type}` with length > 0. Found value of type: `{type(value)}`, with length: {len(value)}.\n"
-                if expected_type_args != get_args(List):  # if we specified types for the items in the list
-                    value_type = expected_type_args[0]
-                    value_error_string = ""
-                    for v in value:
-                        value_error_string += validate_type(v, value_type)
-                    if value_error_string != "":
-                        return f"Typing errors with values:\n {value_error_string}"
 
-        return error_string
-
-
-def validate_metadata_type(metadata_dict: dict):
-    field_types = {field.name: field.type for field in fields(DatasetMetadata)}
-
-    typing_errors = {}
-    for field_name, field_value in metadata_dict.items():
-        field_type_error = validate_type(
-            metadata_dict[field_name], field_types.get(field_name, Union[List[str], Dict[str, List[str]]])
-        )
-        if field_type_error != "":
-            typing_errors[field_name] = field_type_error
-    if len(typing_errors) > 0:
-        raise TypeError(f"The following typing errors are found: {typing_errors}")
-
-
-class _nothing:
-    pass
-
-
-EmptyList = List[_nothing]
-
-
-@dataclass
-class DatasetMetadata:
-    annotations_creators: List[str]
-    language_creators: Union[EmptyList, List[str]]
-    language: Union[EmptyList, List[str]]
-    license: List[str]
-    multilinguality: List[str]
-    pretty_name: str
-    size_categories: List[str]
-    source_datasets: List[str]
-    task_categories: List[str]
-    task_ids: Union[EmptyList, List[str]]
-    language_details: Optional[str] = None
-    language_bcp47: Optional[List[str]] = None
-    paperswithcode_id: Optional[str] = None
-    train_eval_index: Optional[List[Dict]] = None
-    configs: Optional[List[str]] = None
-    extra_gated_fields: Optional[Dict] = None
-    extra_gated_prompt: Optional[str] = None
-    license_details: Optional[str] = None
-    tags: Optional[List[str]] = None
-    licenses: Optional[Union[EmptyList, List[str]]] = None  # deprecated
-    languages: Optional[Union[EmptyList, List[str]]] = None  # deprecated
+class DatasetMetadata(dict):
 
     # class attributes
-    _FIELDS_WITH_DASHES: ClassVar[set] = {"train_eval_index"}  # train-eval-index in the YAML metadata
-    _ALLOWED_YAML_KEYS: ClassVar[set] = set()  # populated later
-    _DEPRECATED_YAML_KEYS = ["licenses", "languages"]
-
-    def __post_init__(self):
-        if self.licenses is not None:
-            warnings.warning("The 'licenses' YAML field is deprecated, please use 'license' instead.")
-        if self.languages is not None:
-            warnings.warning("The 'languages' YAML field is deprecated, please use 'language' instead.")
-
-    def validate(self):
-        validate_metadata_type(metadata_dict=vars(self))
-
-        self.annotations_creators, annotations_creators_errors = self.validate_annotations_creators(
-            self.annotations_creators
-        )
-        self.language_creators, language_creators_errors = self.validate_language_creators(self.language_creators)
-        self.language, language_errors = self.validate_language_codes(self.language or self.languages)
-        self.multilinguality, multilinguality_errors = self.validate_mulitlinguality(self.multilinguality)
-        self.size_categories, size_categories_errors = self.validate_size_catgeories(self.size_categories)
-        self.source_datasets, source_datasets_errors = self.validate_source_datasets(self.source_datasets)
-        self.task_categories, task_categories_errors = self.validate_task_categories(self.task_categories)
-        self.task_ids, task_ids_errors = self.validate_task_ids(self.task_ids)
-        self.paperswithcode_id, paperswithcode_id_errors = self.validate_paperswithcode_id_errors(
-            self.paperswithcode_id
-        )
-        self.train_eval_index, train_eval_index_errors = self.validate_train_eval_index(self.train_eval_index)
-
-        errors = {
-            "annotations_creators": annotations_creators_errors,
-            "language_creators": language_creators_errors,
-            "multilinguality": multilinguality_errors,
-            "size_categories": size_categories_errors,
-            "source_datasets": source_datasets_errors,
-            "task_categories": task_categories_errors,
-            "task_ids": task_ids_errors,
-            "language": language_errors,
-            "paperswithcode_id": paperswithcode_id_errors,
-            "train_eval_index": train_eval_index_errors,
-        }
-
-        exception_msg_dict = dict()
-        for yaml_field, errs in errors.items():
-            if errs is not None:
-                exception_msg_dict[yaml_field] = errs
-        if len(exception_msg_dict) > 0:
-            raise TypeError(
-                "Could not validate the metadata, found the following errors:\n"
-                + "\n".join(f"* field '{fieldname}':\n\t{err}" for fieldname, err in exception_msg_dict.items())
-            )
+    _FIELDS_WITH_DASHES = {"train_eval_index"}  # train-eval-index in the YAML metadata
 
     @classmethod
     def from_readme(cls, path: Path) -> "DatasetMetadata":
@@ -304,172 +47,55 @@ def from_readme(cls, path: Path) -> "DatasetMetadata":
             :class:`DatasetMetadata`: The dataset's metadata
 
         Raises:
-            :obj:`TypeError`: If the dataset card has no metadata (no YAML header)
             :obj:`TypeError`: If the dataset's metadata is invalid
         """
-        yaml_string = yaml_block_from_readme(path)
+        yaml_string, _ = _split_yaml_from_readme(path)
         if yaml_string is not None:
             return cls.from_yaml_string(yaml_string)
         else:
-            raise TypeError(f"Unable to find a yaml block in '{path}'")
+            return cls()
+
+    def to_readme(self, path: Path):
+        if path.exists():
+            _, content = _split_yaml_from_readme(path)
+            full_content = "---\n" + self.to_yaml_string() + "---\n" + content
+        else:
+            full_content = "---\n" + self.to_yaml_string() + "---\n"
+        with open(path, "w", encoding="utf-8") as readme_file:
+            readme_file.write(full_content)
 
     @classmethod
-    def _metadata_dict_from_yaml_string(cls, string: str) -> dict:
-        """Loads and validates the dataset metadat from a YAML string
+    def from_yaml_string(cls, string: str) -> "DatasetMetadata":
+        """Loads and validates the dataset metadata from a YAML string
 
         Args:
             string (:obj:`str`): The YAML string
 
         Returns:
-            :class:`dict`: The dataset's metadata as a dictionary
+            :class:`DatasetMetadata`: The dataset's metadata
 
         Raises:
             :obj:`TypeError`: If the dataset's metadata is invalid
         """
-        metadata_dict = yaml.load(string, Loader=NoDuplicateSafeLoader) or dict()
-
-        # Check if the YAML keys are all correct
-        bad_keys = [k for k in metadata_dict if k not in cls._ALLOWED_YAML_KEYS]
-        if bad_keys:
-            raise ValueError(f"Bad YAML keys: {bad_keys}. Allowed fields: {cls._ALLOWED_YAML_KEYS}")
-
-        # Check if config names are valid
-        bad_keys = [k for k in metadata_dict if k not in cls._ALLOWED_YAML_KEYS]
-        if bad_keys:
-            raise ValueError(f"Bad YAML keys: {bad_keys}. Allowed fields: {cls._ALLOWED_YAML_KEYS}")
+        metadata_dict = yaml.load(string, Loader=_NoDuplicateSafeLoader) or dict()
 
         # Convert the YAML keys to DatasetMetadata fields
         metadata_dict = {
             (key.replace("-", "_") if key.replace("-", "_") in cls._FIELDS_WITH_DASHES else key): value
             for key, value in metadata_dict.items()
         }
-        return metadata_dict
-
-    @classmethod
-    def from_yaml_string(cls, string: str) -> "DatasetMetadata":
-        """Loads and validates the dataset metadata from a YAML string
-
-        Args:
-            string (:obj:`str`): The YAML string
-
-        Returns:
-            :class:`DatasetMetadata`: The dataset's metadata
-
-        Raises:
-            :obj:`TypeError`: If the dataset's metadata is invalid
-            :obj:`ValueError`: If the dataset's metadata is invalid
-        """
-        metadata_dict = cls._metadata_dict_from_yaml_string(string)
         return cls(**metadata_dict)
 
-    @staticmethod
-    def validate_annotations_creators(annotations_creators: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        return tagset_validator(
-            annotations_creators, known_creators["annotations"], "annotations_creators", known_creators_url
-        )
-
-    @staticmethod
-    def validate_language_creators(language_creators: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        return tagset_validator(language_creators, known_creators["language"], "language_creators", known_creators_url)
-
-    @staticmethod
-    def validate_language_codes(languages: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        return tagset_validator(
-            languages,
-            known_language_codes.keys(),
-            "language",
-            known_language_codes_url,
-            lambda lang: lang == "unknown",
-        )
-
-    @staticmethod
-    def validate_task_categories(task_categories: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        # TODO: we're currently ignoring all values starting with 'other' as our task taxonomy is bound to change
-        #   in the near future and we don't want to waste energy in tagging against a moving taxonomy.
-        known_set = list(known_task_ids.keys())
-        validated, error = tagset_validator(
-            task_categories, known_set, "task_categories", known_task_ids_url, lambda e: e.startswith("other-")
-        )
-        return validated, error
-
-    @staticmethod
-    def validate_task_ids(task_ids: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        # TODO: we're currently ignoring all values starting with 'other' as our task taxonomy is bound to change
-        #   in the near future and we don't want to waste energy in tagging against a moving taxonomy.
-        known_set = [tid for _cat, d in known_task_ids.items() for tid in d.get("subtasks", [])]
-        validated, error = tagset_validator(
-            task_ids,
-            known_set,
-            "task_ids",
-            known_task_ids_url,
-            lambda e: not e or "-other-" in e or e.startswith("other-"),
-        )
-        return validated, error
-
-    @staticmethod
-    def validate_mulitlinguality(multilinguality: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        validated, error = tagset_validator(
-            multilinguality,
-            list(known_multilingualities.keys()),
-            "multilinguality",
-            known_size_categories_url,
-            lambda e: e.startswith("other-"),
-        )
-        return validated, error
-
-    @staticmethod
-    def validate_size_catgeories(size_cats: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        return tagset_validator(size_cats, known_size_categories, "size_categories", known_size_categories_url)
-
-    @staticmethod
-    def validate_source_datasets(sources: Union[List[str], Dict[str, List[str]]]) -> ValidatorOutput:
-        return tagset_validator(sources, known_source_datasets, "source_datasets", known_source_datasets_url)
-
-    @staticmethod
-    def validate_paperswithcode_id_errors(paperswithcode_id: Optional[str]) -> ValidatorOutput:
-        if paperswithcode_id is None:
-            return paperswithcode_id, None
-        else:
-            if " " in paperswithcode_id or paperswithcode_id.lower() != paperswithcode_id:
-                return (
-                    None,
-                    f"The paperswithcode_id must be lower case and not contain spaces but got {paperswithcode_id}. You can find the paperswithcode_id in the URL of the dataset page on paperswithcode.com.",
-                )
-            else:
-                return paperswithcode_id, None
-
-    @staticmethod
-    def validate_pretty_name(pretty_name: Union[str, Dict[str, str]]):
-        if isinstance(pretty_name, str):
-            if len(pretty_name) == 0:
-                return None, "The pretty name must have a length greater than 0 but got an empty string."
-        else:
-            error_string = ""
-            for key, value in pretty_name.items():
-                if len(value) == 0:
-                    error_string += f"The pretty name must have a length greater than 0 but got an empty string for config: {key}.\n"
-
-            if error_string == "":
-                return None, error_string
-            else:
-                return pretty_name, None
-
-    @staticmethod
-    def validate_train_eval_index(train_eval_index: Optional[Dict]):
-        if train_eval_index is not None and not isinstance(train_eval_index, list):
-            return None, f"train-eval-index must be a list, but got {type(train_eval_index)} instead."
-        else:
-            return train_eval_index, None
-
-
-# In general the allowed YAML keys are the fields of the DatasetMetadata dataclass.
-# However it is not the case certain fields like train_eval_index,
-# for which the YAML key must use dashes and not underscores.
-# Fields that corresponds to YAML keys with dashes are defined in DatasetMetadata._FIELDS_WITH_DASHES
-DatasetMetadata._ALLOWED_YAML_KEYS = {
-    field.name.replace("_", "-") if field.name in DatasetMetadata._FIELDS_WITH_DASHES else field.name
-    for field in fields(DatasetMetadata)
-}
+    def to_yaml_string(self) -> str:
+        return yaml.safe_dump(
+            {
+                (key.replace("_", "-") if key in self._FIELDS_WITH_DASHES else key): value
+                for key, value in self.items()
+            },
+            sort_keys=False,
+            allow_unicode=True,
+            encoding="utf-8",
+        ).decode("utf-8")
 
 
 if __name__ == "__main__":
@@ -481,4 +107,5 @@ def validate_train_eval_index(train_eval_index: Optional[Dict]):
 
     readme_filepath = Path(args.readme_filepath)
     dataset_metadata = DatasetMetadata.from_readme(readme_filepath)
-    dataset_metadata.validate()
+    print(dataset_metadata)
+    dataset_metadata.to_readme(readme_filepath)
diff --git a/src/datasets/utils/py_utils.py b/src/datasets/utils/py_utils.py
index 01d0673a2f6..c4b7152307e 100644
--- a/src/datasets/utils/py_utils.py
+++ b/src/datasets/utils/py_utils.py
@@ -167,7 +167,8 @@ def _asdict_inner(obj):
             result = {}
             for f in fields(obj):
                 value = _asdict_inner(getattr(obj, f.name))
-                result[f.name] = value
+                if not f.init or value != f.default or f.metadata.get("include_in_asdict_even_if_is_default", False):
+                    result[f.name] = value
             return result
         elif isinstance(obj, tuple) and hasattr(obj, "_fields"):
             # obj is a namedtuple
diff --git a/src/datasets/utils/version.py b/src/datasets/utils/version.py
index bc2553096ae..15a2a143314 100644
--- a/src/datasets/utils/version.py
+++ b/src/datasets/utils/version.py
@@ -113,6 +113,9 @@ def from_dict(cls, dic):
         field_names = {f.name for f in dataclasses.fields(cls)}
         return cls(**{k: v for k, v in dic.items() if k in field_names})
 
+    def _to_yaml_string(self) -> str:
+        return self.version_str
+
 
 def _str_to_version(version_str, allow_wildcard=False):
     """Return the tuple (major, minor, patch) version extracted from the str."""
diff --git a/tests/commands/test_test.py b/tests/commands/test_test.py
index 80581814c63..0c4bf68ae78 100644
--- a/tests/commands/test_test.py
+++ b/tests/commands/test_test.py
@@ -1,11 +1,11 @@
-import json
 import os
 from collections import namedtuple
 
 import pytest
 
-from datasets import config
+from datasets import ClassLabel, Features, Sequence, Value
 from datasets.commands.test import TestCommand
+from datasets.info import DatasetInfo, DatasetInfosDict
 
 
 _TestCommandArgs = namedtuple(
@@ -25,126 +25,58 @@
 )
 
 
+def is_1percent_close(source, target):
+    return (abs(source - target) / target) < 0.01
+
+
 @pytest.mark.integration
 def test_test_command(dataset_loading_script_dir):
     args = _TestCommandArgs(dataset=dataset_loading_script_dir, all_configs=True, save_infos=True)
     test_command = TestCommand(*args)
     test_command.run()
-    dataset_infos_path = os.path.join(dataset_loading_script_dir, config.DATASETDICT_INFOS_FILENAME)
-    assert os.path.exists(dataset_infos_path)
-    with open(dataset_infos_path, encoding="utf-8") as f:
-        dataset_infos = json.load(f)
-    expected_dataset_infos = {
-        "default": {
-            "description": "",
-            "citation": "",
-            "homepage": "",
-            "license": "",
-            "features": {
-                "tokens": {
-                    "feature": {"dtype": "string", "id": None, "_type": "Value"},
-                    "length": -1,
-                    "id": None,
-                    "_type": "Sequence",
-                },
-                "ner_tags": {
-                    "feature": {
-                        "num_classes": 7,
-                        "names": ["O", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"],
-                        "id": None,
-                        "_type": "ClassLabel",
+    dataset_readme_path = os.path.join(dataset_loading_script_dir, "README.md")
+    assert os.path.exists(dataset_readme_path)
+    dataset_infos = DatasetInfosDict.from_directory(dataset_loading_script_dir)
+    expected_dataset_infos = DatasetInfosDict(
+        {
+            "default": DatasetInfo(
+                features=Features(
+                    {
+                        "tokens": Sequence(Value("string")),
+                        "ner_tags": Sequence(
+                            ClassLabel(names=["O", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"])
+                        ),
+                        "langs": Sequence(Value("string")),
+                        "spans": Sequence(Value("string")),
+                    }
+                ),
+                splits=[
+                    {
+                        "name": "train",
+                        "num_bytes": 2351563,
+                        "num_examples": 10000,
+                    },
+                    {
+                        "name": "validation",
+                        "num_bytes": 238418,
+                        "num_examples": 1000,
                     },
-                    "length": -1,
-                    "id": None,
-                    "_type": "Sequence",
-                },
-                "langs": {
-                    "feature": {"dtype": "string", "id": None, "_type": "Value"},
-                    "length": -1,
-                    "id": None,
-                    "_type": "Sequence",
-                },
-                "spans": {
-                    "feature": {"dtype": "string", "id": None, "_type": "Value"},
-                    "length": -1,
-                    "id": None,
-                    "_type": "Sequence",
-                },
-            },
-            "post_processed": None,
-            "supervised_keys": None,
-            "task_templates": None,
-            "builder_name": "__dummy_dataset1__",
-            "config_name": "default",
-            "version": {"version_str": "0.0.0", "description": None, "major": 0, "minor": 0, "patch": 0},
-            "splits": {
-                "train": {
-                    "name": "train",
-                    "num_bytes": 2351591,
-                    "num_examples": 10000,
-                    "dataset_name": "__dummy_dataset1__",
-                },
-                "validation": {
-                    "name": "validation",
-                    "num_bytes": 238446,
-                    "num_examples": 1000,
-                    "dataset_name": "__dummy_dataset1__",
-                },
-            },
-            "download_checksums": {
-                "https://huggingface.co/datasets/albertvillanova/tests-raw-jsonl/resolve/main/wikiann-bn-train.jsonl": {
-                    "num_bytes": 3578339,
-                    "checksum": "6fbe6dbdcb3c9c3a98b0ab4d56b1c8b73baab9293d603064a5ab5230ab4f366b",
-                },
-                "https://huggingface.co/datasets/albertvillanova/tests-raw-jsonl/resolve/main/wikiann-bn-validation.jsonl": {
-                    "num_bytes": 362341,
-                    "checksum": "2ddd0c090a8ccb721d7aa8477ed7323750683822c247015d5cfab1af1c8c8b3f",
-                },
-            },
-            "download_size": 3940680,
-            "post_processing_size": None,
-            "dataset_size": 2590037,
-            "size_in_bytes": 6530717,
+                ],
+                download_size=3940680,
+                dataset_size=2589981,
+            )
         }
-    }
+    )
     assert dataset_infos.keys() == expected_dataset_infos.keys()
-    assert dataset_infos["default"].keys() == expected_dataset_infos["default"].keys()
-    for key in dataset_infos["default"].keys():
-        if key in [
-            "description",
-            "citation",
-            "homepage",
-            "license",
-            "features",
-            "post_processed",
-            "supervised_keys",
-            "task_templates",
-            "builder_name",
-            "config_name",
-            "version",
-            "download_checksums",
-            "download_size",
-            "post_processing_size",
-        ]:
-            assert dataset_infos["default"][key] == expected_dataset_infos["default"][key]
-        elif key in ["dataset_size", "size_in_bytes"]:
-            assert round(dataset_infos["default"][key] / 10**5) == round(
-                expected_dataset_infos["default"][key] / 10**5
-            )
+    for key in DatasetInfo._INCLUDED_INFO_IN_YAML:
+        result, expected = getattr(dataset_infos["default"], key), getattr(expected_dataset_infos["default"], key)
+        if key == "num_bytes":
+            assert is_1percent_close(result, expected)
         elif key == "splits":
-            assert dataset_infos["default"]["splits"].keys() == expected_dataset_infos["default"]["splits"].keys()
-            for split in dataset_infos["default"]["splits"].keys():
-                assert (
-                    dataset_infos["default"]["splits"][split].keys()
-                    == expected_dataset_infos["default"]["splits"][split].keys()
-                )
-                for subkey in dataset_infos["default"]["splits"][split].keys():
-                    if subkey == "num_bytes":
-                        assert round(dataset_infos["default"]["splits"][split][subkey] / 10**2) == round(
-                            expected_dataset_infos["default"]["splits"][split][subkey] / 10**2
-                        )
-                    else:
-                        assert (
-                            dataset_infos["default"]["splits"][split][subkey]
-                            == expected_dataset_infos["default"]["splits"][split][subkey]
-                        )
+            assert list(result) == list(expected)
+            for split in result:
+                assert result[split].name == expected[split].name
+                assert result[split].num_examples == expected[split].num_examples
+                assert is_1percent_close(result[split].num_bytes, expected[split].num_bytes)
+        else:
+            result == expected
diff --git a/tests/features/test_features.py b/tests/features/test_features.py
index ec840f63587..e0803949032 100644
--- a/tests/features/test_features.py
+++ b/tests/features/test_features.py
@@ -7,8 +7,9 @@
 import pyarrow as pa
 import pytest
 
+from datasets import Array2D
 from datasets.arrow_dataset import Dataset
-from datasets.features import ClassLabel, Features, Image, Sequence, Value
+from datasets.features import Audio, ClassLabel, Features, Image, Sequence, Value
 from datasets.features.features import (
     _arrow_to_datasets_dtype,
     _cast_to_python_objects,
@@ -562,3 +563,65 @@ def test_dont_iterate_over_each_element_in_a_list(self, mocked_cast):
         obj = {"col_1": [[1, 2], [3, 4], [5, 6]]}
         cast_to_python_objects(obj)
         self.assertEqual(mocked_cast.call_count, 4)  # 4 = depth of obj
+
+
+SIMPLE_FEATURES = [
+    Features(),
+    Features({"a": Value("int32")}),
+    Features({"a": Value("int32", id="my feature")}),
+    Features({"a": Value("int32"), "b": Value("float64"), "c": Value("string")}),
+]
+
+CUSTOM_FEATURES = [
+    Features({"label": ClassLabel(names=["negative", "positive"])}),
+    Features({"array": Array2D(dtype="float32", shape=(4, 4))}),
+    Features({"image": Image()}),
+    Features({"audio": Audio()}),
+    Features({"image": Image(decode=False)}),
+    Features({"audio": Audio(decode=False)}),
+    Features({"translation": Translation(["en", "fr"])}),
+    Features({"translation": TranslationVariableLanguages(["en", "fr"])}),
+]
+
+NESTED_FEATURES = [
+    Features({"foo": {}}),
+    Features({"foo": {"bar": Value("int32")}}),
+    Features({"foo": {"bar1": Value("int32"), "bar2": Value("float64")}}),
+    Features({"foo": Sequence(Value("int32"))}),
+    Features({"foo": Sequence({})}),
+    Features({"foo": Sequence({"bar": Value("int32")})}),
+    Features({"foo": [Value("int32")]}),
+    Features({"foo": [{"bar": Value("int32")}]}),
+]
+
+NESTED_CUSTOM_FEATURES = [
+    Features({"foo": {"bar": ClassLabel(names=["negative", "positive"])}}),
+    Features({"foo": Sequence(ClassLabel(names=["negative", "positive"]))}),
+    Features({"foo": Sequence({"bar": ClassLabel(names=["negative", "positive"])})}),
+    Features({"foo": [ClassLabel(names=["negative", "positive"])]}),
+    Features({"foo": [{"bar": ClassLabel(names=["negative", "positive"])}]}),
+]
+
+
+@pytest.mark.parametrize("features", SIMPLE_FEATURES + CUSTOM_FEATURES + NESTED_FEATURES + NESTED_CUSTOM_FEATURES)
+def test_features_to_dict(features: Features):
+    features_dict = features.to_dict()
+    assert isinstance(features_dict, dict)
+    reloaded = Features.from_dict(features_dict)
+    assert features == reloaded
+
+
+@pytest.mark.parametrize("features", SIMPLE_FEATURES + CUSTOM_FEATURES + NESTED_FEATURES + NESTED_CUSTOM_FEATURES)
+def test_features_to_yaml_list(features: Features):
+    features_yaml_list = features._to_yaml_list()
+    assert isinstance(features_yaml_list, list)
+    reloaded = Features._from_yaml_list(features_yaml_list)
+    assert features == reloaded
+
+
+@pytest.mark.parametrize("features", SIMPLE_FEATURES + CUSTOM_FEATURES + NESTED_FEATURES + NESTED_CUSTOM_FEATURES)
+def test_features_to_arrow_schema(features: Features):
+    arrow_schema = features.arrow_schema
+    assert isinstance(arrow_schema, pa.Schema)
+    reloaded = Features.from_arrow_schema(arrow_schema)
+    assert features == reloaded
diff --git a/tests/test_dataset_cards.py b/tests/test_dataset_cards.py
index cd56c9f32c9..9462084b122 100644
--- a/tests/test_dataset_cards.py
+++ b/tests/test_dataset_cards.py
@@ -20,7 +20,7 @@
 
 from datasets.packaged_modules import _PACKAGED_DATASETS_MODULES
 from datasets.utils.logging import get_logger
-from datasets.utils.metadata import DatasetMetadata, validate_metadata_type, yaml_block_from_readme
+from datasets.utils.metadata import DatasetMetadata
 from datasets.utils.readme import ReadMe
 
 from .utils import slow
@@ -46,7 +46,11 @@ def get_changed_datasets(repo_path: Path) -> List[Path]:
 
 
 def get_all_datasets(repo_path: Path) -> List[Path]:
-    dataset_names = [path.parts[-1] for path in (repo_path / "datasets").iterdir() if path.is_dir()]
+    dataset_names = [
+        path.parts[-1]
+        for path in (repo_path / "datasets").iterdir()
+        if path.is_dir() and (path / path.name).with_suffix(".py").is_file()
+    ]
     return [dataset_name for dataset_name in dataset_names if dataset_name not in _PACKAGED_DATASETS_MODULES]
 
 
@@ -57,21 +61,20 @@ def test_changed_dataset_card(dataset_name):
     assert card_path.exists()
     error_messages = []
     try:
-        readme = ReadMe.from_readme(card_path)
+        ReadMe.from_readme(card_path)
     except Exception as readme_parsing_error:
         error_messages.append(
             f"The following issues have been found in the dataset cards:\nREADME Parsing:\n{readme_parsing_error}"
         )
     try:
-        readme = ReadMe.from_readme(card_path, suppress_parsing_errors=True)
-        readme.validate()
+        ReadMe.from_readme(card_path, suppress_parsing_errors=True)
     except Exception as readme_validation_error:
         error_messages.append(
             f"The following issues have been found in the dataset cards:\nREADME Validation:\n{readme_validation_error}"
         )
     try:
         metadata = DatasetMetadata.from_readme(card_path)
-        metadata.validate()
+        assert metadata, "empty metadata"
     except Exception as metadata_error:
         error_messages.append(
             f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
@@ -89,10 +92,8 @@ def test_dataset_card_yaml_structure(dataset_name):
     """
     card_path = repo_path / "datasets" / dataset_name / "README.md"
     assert card_path.exists()
-    yaml_string = yaml_block_from_readme(card_path)
-    metadata_dict = DatasetMetadata._metadata_dict_from_yaml_string(yaml_string)
+    metadata_dict = DatasetMetadata.from_readme(card_path)
     assert len(metadata_dict) > 0
-    validate_metadata_type(metadata_dict)
 
 
 @slow
@@ -117,7 +118,7 @@ def test_dataset_card(dataset_name):
         )
     try:
         metadata = DatasetMetadata.from_readme(card_path)
-        metadata.validate()
+        assert metadata
     except Exception as metadata_error:
         error_messages.append(
             f"The following issues have been found in the dataset cards:\nYAML tags:\n{metadata_error}"
diff --git a/tests/test_info.py b/tests/test_info.py
new file mode 100644
index 00000000000..275eaf1d4eb
--- /dev/null
+++ b/tests/test_info.py
@@ -0,0 +1,110 @@
+import os
+
+import pytest
+import yaml
+
+from datasets.features.features import Features, Value
+from datasets.info import DatasetInfo, DatasetInfosDict
+
+
+@pytest.mark.parametrize(
+    "dataset_info",
+    [
+        DatasetInfo(),
+        DatasetInfo(
+            description="foo",
+            features=Features({"a": Value("int32")}),
+            builder_name="builder",
+            config_name="config",
+            version="1.0.0",
+            splits=[{"name": "train"}],
+            download_size=42,
+        ),
+    ],
+)
+def test_dataset_info_dump_and_reload(tmp_path, dataset_info: DatasetInfo):
+    tmp_path = str(tmp_path)
+    dataset_info.write_to_directory(tmp_path)
+    reloaded = DatasetInfo.from_directory(tmp_path)
+    assert dataset_info == reloaded
+    assert os.path.exists(os.path.join(tmp_path, "dataset_info.json"))
+
+
+def test_dataset_info_to_yaml_dict():
+    dataset_info = DatasetInfo(
+        description="foo",
+        citation="bar",
+        homepage="https://foo.bar",
+        license="CC0",
+        features=Features({"a": Value("int32")}),
+        post_processed={},
+        supervised_keys=tuple(),
+        task_templates=[],
+        builder_name="builder",
+        config_name="config",
+        version="1.0.0",
+        splits=[{"name": "train", "num_examples": 42}],
+        download_checksums={},
+        download_size=1337,
+        post_processing_size=442,
+        dataset_size=1234,
+        size_in_bytes=1337 + 442 + 1234,
+    )
+    dataset_info_yaml_dict = dataset_info._to_yaml_dict()
+    assert sorted(dataset_info_yaml_dict) == sorted(DatasetInfo._INCLUDED_INFO_IN_YAML)
+    for key in DatasetInfo._INCLUDED_INFO_IN_YAML:
+        assert key in dataset_info_yaml_dict
+        assert isinstance(dataset_info_yaml_dict[key], (list, dict, int, str))
+    dataset_info_yaml = yaml.safe_dump(dataset_info_yaml_dict)
+    reloaded = yaml.safe_load(dataset_info_yaml)
+    assert dataset_info_yaml_dict == reloaded
+
+
+def test_dataset_info_to_yaml_dict_empty():
+    dataset_info = DatasetInfo()
+    dataset_info_yaml_dict = dataset_info._to_yaml_dict()
+    assert dataset_info_yaml_dict == {}
+
+
+@pytest.mark.parametrize(
+    "dataset_infos_dict",
+    [
+        DatasetInfosDict(),
+        DatasetInfosDict({"default": DatasetInfo()}),
+        DatasetInfosDict({"my_config_name": DatasetInfo()}),
+        DatasetInfosDict(
+            {
+                "default": DatasetInfo(
+                    description="foo",
+                    features=Features({"a": Value("int32")}),
+                    builder_name="builder",
+                    config_name="config",
+                    version="1.0.0",
+                    splits=[{"name": "train"}],
+                    download_size=42,
+                )
+            }
+        ),
+        DatasetInfosDict(
+            {
+                "v1": DatasetInfo(dataset_size=42),
+                "v2": DatasetInfo(dataset_size=1337),
+            }
+        ),
+    ],
+)
+def test_dataset_infos_dict_dump_and_reload(tmp_path, dataset_infos_dict: DatasetInfosDict):
+    tmp_path = str(tmp_path)
+    dataset_infos_dict.write_to_directory(tmp_path)
+    reloaded = DatasetInfosDict.from_directory(tmp_path)
+
+    # the config_name of the dataset_infos_dict take over the attribute
+    for config_name, dataset_info in dataset_infos_dict.items():
+        dataset_info.config_name = config_name
+        # the yaml representation doesn't include fields like description or citation
+        # so we just test that we can recover what we can from the yaml
+        dataset_infos_dict[config_name] = DatasetInfo._from_yaml_dict(dataset_info._to_yaml_dict())
+    assert dataset_infos_dict == reloaded
+
+    if dataset_infos_dict:
+        assert os.path.exists(os.path.join(tmp_path, "README.md"))
diff --git a/tests/test_metadata_util.py b/tests/test_metadata_util.py
index 1a96c440182..7222a963ce3 100644
--- a/tests/test_metadata_util.py
+++ b/tests/test_metadata_util.py
@@ -1,16 +1,9 @@
 import re
 import tempfile
 import unittest
-from dataclasses import _MISSING_TYPE, asdict, fields
 from pathlib import Path
 
-from datasets.utils.metadata import (
-    DatasetMetadata,
-    metadata_dict_from_readme,
-    tagset_validator,
-    validate_metadata_type,
-    yaml_block_from_readme,
-)
+from datasets.utils.metadata import DatasetMetadata
 
 
 def _dedent(string: str) -> str:
@@ -48,157 +41,26 @@ def _dedent(string: str) -> str:
 
 
 class TestMetadataUtils(unittest.TestCase):
-    def test_validate_metadata_type(self):
-        metadata_dict = {
-            "tag": ["list", "of", "values"],
-            "another tag": ["Another", {"list"}, ["of"], 0x646D46736457567A],
-        }
-        with self.assertRaises(TypeError):
-            validate_metadata_type(metadata_dict)
-
-        metadata_dict = {"tag1": []}
-        with self.assertRaises(TypeError):
-            validate_metadata_type(metadata_dict)
-
-        metadata_dict = {"tag1": None}
-        with self.assertRaises(TypeError):
-            validate_metadata_type(metadata_dict)
-
-    def test_tagset_validator(self):
-        name = "test_tag"
-        url = "https://dummy.hf.co"
-
-        items = ["tag1", "tag2", "tag2", "tag3"]
-        reference_values = ["tag1", "tag2", "tag3"]
-        returned_values, error = tagset_validator(items=items, reference_values=reference_values, name=name, url=url)
-        self.assertListEqual(returned_values, items)
-        self.assertIsNone(error)
-
-        items = []
-        reference_values = ["tag1", "tag2", "tag3"]
-        items, error = tagset_validator(items=items, reference_values=reference_values, name=name, url=url)
-        self.assertListEqual(items, [])
-        self.assertIsNone(error)
-
-        items = []
-        reference_values = []
-        returned_values, error = tagset_validator(items=items, reference_values=reference_values, name=name, url=url)
-        self.assertListEqual(returned_values, [])
-        self.assertIsNone(error)
-
-        items = ["tag1", "tag2", "tag2", "tag3", "unknown tag"]
-        reference_values = ["tag1", "tag2", "tag3"]
-        returned_values, error = tagset_validator(items=items, reference_values=reference_values, name=name, url=url)
-        self.assertListEqual(returned_values, [])
-        self.assertEqual(error, f"{['unknown tag']} are not registered tags for '{name}', reference at {url}")
-
-        def predicate_fn(string):
-            return "ignore" in string
-
-        items = ["process me", "process me too", "ignore me"]
-        reference_values = ["process me too"]
-        returned_values, error = tagset_validator(
-            items=items,
-            reference_values=reference_values,
-            name=name,
-            url=url,
-            escape_validation_predicate_fn=predicate_fn,
-        )
-        self.assertListEqual(returned_values, [])
-        self.assertEqual(error, f"{['process me']} are not registered tags for '{name}', reference at {url}")
-
-        items = ["process me", "process me too", "ignore me"]
-        reference_values = ["process me too", "process me"]
-        returned_values, error = tagset_validator(
-            items=items,
-            reference_values=reference_values,
-            name=name,
-            url=url,
-            escape_validation_predicate_fn=predicate_fn,
-        )
-        self.assertListEqual(returned_values, items)
-        self.assertIsNone(error)
-
-        items = ["ignore me too", "ignore me"]
-        reference_values = ["process me too"]
-        returned_values, error = tagset_validator(
-            items=items,
-            reference_values=reference_values,
-            name=name,
-            url=url,
-            escape_validation_predicate_fn=predicate_fn,
-        )
-        self.assertListEqual(returned_values, items)
-        self.assertIsNone(error)
-
-    def test_yaml_block_from_readme(self):
-        with tempfile.TemporaryDirectory() as tmp_dir:
-            path = Path(tmp_dir) / "README.md"
-
-            with open(path, "w+") as readme_file:
-                readme_file.write(README_YAML)
-            yaml_block = yaml_block_from_readme(path=path)
-            self.assertEqual(
-                yaml_block,
-                _dedent(
-                    """\
-                    language:
-                    - zh
-                    - en
-                    task_ids:
-                    - sentiment-classification
-                    """
-                ),
-            )
-
-            with open(path, "w+") as readme_file:
-                readme_file.write(README_EMPTY_YAML)
-            yaml_block = yaml_block_from_readme(path=path)
-            self.assertEqual(
-                yaml_block,
-                _dedent(
-                    """\
-                    """
-                ),
-            )
-
-            with open(path, "w+") as readme_file:
-                readme_file.write(README_NO_YAML)
-            yaml_block = yaml_block_from_readme(path=path)
-            self.assertIsNone(yaml_block)
-
     def test_metadata_dict_from_readme(self):
         with tempfile.TemporaryDirectory() as tmp_dir:
             path = Path(tmp_dir) / "README.md"
             with open(path, "w+") as readme_file:
                 readme_file.write(README_YAML)
-            metadata_dict = metadata_dict_from_readme(path)
+            metadata_dict = DatasetMetadata.from_readme(path)
             self.assertDictEqual(metadata_dict, {"language": ["zh", "en"], "task_ids": ["sentiment-classification"]})
 
             with open(path, "w+") as readme_file:
                 readme_file.write(README_EMPTY_YAML)
-            metadata_dict = metadata_dict_from_readme(path)
+            metadata_dict = DatasetMetadata.from_readme(path)
             self.assertDictEqual(metadata_dict, {})
 
             with open(path, "w+") as readme_file:
                 readme_file.write(README_NO_YAML)
-            metadata_dict = metadata_dict_from_readme(path)
-            self.assertIsNone(metadata_dict)
+            metadata_dict = DatasetMetadata.from_readme(path)
+            self.assertEqual(metadata_dict, {})
 
     def test_from_yaml_string(self):
 
-        default_optional_keys = {
-            field.name: field.default
-            for field in fields(DatasetMetadata)
-            if type(field.default) is not _MISSING_TYPE and field.name not in DatasetMetadata._DEPRECATED_YAML_KEYS
-        }
-
-        default_deprecated_keys = {
-            field.name: field.default
-            for field in fields(DatasetMetadata)
-            if field.name in DatasetMetadata._DEPRECATED_YAML_KEYS
-        }
-
         valid_yaml_string = _dedent(
             """\
             annotations_creators:
@@ -222,87 +84,7 @@ def test_from_yaml_string(self):
             - open-domain-qa
             """
         )
-        DatasetMetadata.from_yaml_string(valid_yaml_string)
-
-        valid_yaml_string_with_configs = _dedent(
-            """\
-            annotations_creators:
-            - found
-            language_creators:
-            - found
-            language:
-              en:
-              - en
-              fr:
-              - fr
-            license:
-            - unknown
-            multilinguality:
-            - monolingual
-            pretty_name: Test Dataset
-            size_categories:
-            - 10K<n<100K
-            source_datasets:
-            - extended|other-yahoo-webscope-l6
-            task_categories:
-            - question-answering
-            task_ids:
-            - open-domain-qa
-            """
-        )
-        DatasetMetadata.from_yaml_string(valid_yaml_string_with_configs)
-
-        invalid_tag_yaml = _dedent(
-            """\
-            annotations_creators:
-            - found
-            language_creators:
-            - some guys in Panama
-            language:
-            - en
-            license:
-            - unknown
-            multilinguality:
-            - monolingual
-            pretty_name: Test Dataset
-            size_categories:
-            - 10K<n<100K
-            source_datasets:
-            - extended|other-yahoo-webscope-l6
-            task_categories:
-            - question-answering
-            task_ids:
-            - open-domain-qa
-            """
-        )
-        with self.assertRaises(TypeError):
-            metadata = DatasetMetadata.from_yaml_string(invalid_tag_yaml)
-            metadata.validate()
-
-        missing_tag_yaml = _dedent(
-            """\
-            annotations_creators:
-            - found
-            language:
-            - en
-            license:
-            - unknown
-            multilinguality:
-            - monolingual
-            pretty_name: Test Dataset
-            size_categories:
-            - 10K<n<100K
-            source_datasets:
-            - extended|other-yahoo-webscope-l6
-            task_categories:
-            - question-answering
-            task_ids:
-            - open-domain-qa
-            """
-        )
-        with self.assertRaises(TypeError):
-            metadata = DatasetMetadata.from_yaml_string(missing_tag_yaml)
-            metadata.validate()
+        assert DatasetMetadata.from_yaml_string(valid_yaml_string)
 
         duplicate_yaml_keys = _dedent(
             """\
@@ -328,119 +110,7 @@ def test_from_yaml_string(self):
             """
         )
         with self.assertRaises(TypeError):
-            metadata = DatasetMetadata.from_yaml_string(duplicate_yaml_keys)
-            metadata.validate()
-
-        valid_yaml_string_with_duplicate_configs = _dedent(
-            """\
-            annotations_creators:
-            - found
-            language_creators:
-            - found
-            language:
-              en:
-              - en
-              en:
-              - en
-            license:
-            - unknown
-            multilinguality:
-            - monolingual
-            pretty_name: Test Dataset
-            size_categories:
-            - 10K<n<100K
-            source_datasets:
-            - extended|other-yahoo-webscope-l6
-            task_categories:
-            - question-answering
-            task_ids:
-            - open-domain-qa
-            """
-        )
-        with self.assertRaises(TypeError):
-            metadata = DatasetMetadata.from_yaml_string(valid_yaml_string_with_duplicate_configs)
-            metadata.validate()
-
-        valid_yaml_string_with_paperswithcode_id = _dedent(
-            """\
-            annotations_creators:
-            - found
-            language_creators:
-            - found
-            language:
-            - en
-            license:
-            - unknown
-            multilinguality:
-            - monolingual
-            pretty_name: Test Dataset
-            size_categories:
-            - 10K<n<100K
-            source_datasets:
-            - extended|other-yahoo-webscope-l6
-            task_categories:
-            - question-answering
-            task_ids:
-            - open-domain-qa
-            paperswithcode_id: squad
-            """
-        )
-        DatasetMetadata.from_yaml_string(valid_yaml_string_with_paperswithcode_id)
-
-        valid_yaml_string_with_null_paperswithcode_id = _dedent(
-            """\
-            annotations_creators:
-            - found
-            language_creators:
-            - found
-            language:
-            - en
-            license:
-            - unknown
-            multilinguality:
-            - monolingual
-            pretty_name: Test Dataset
-            size_categories:
-            - 10K<n<100K
-            source_datasets:
-            - extended|other-yahoo-webscope-l6
-            task_categories:
-            - question-answering
-            task_ids:
-            - open-domain-qa
-            paperswithcode_id: null
-            """
-        )
-        DatasetMetadata.from_yaml_string(valid_yaml_string_with_null_paperswithcode_id)
-
-        valid_yaml_string_with_list_paperswithcode_id = _dedent(
-            """\
-            annotations_creators:
-            - found
-            language_creators:
-            - found
-            language:
-            - en
-            license:
-            - unknown
-            multilinguality:
-            - monolingual
-            pretty_name: Test Dataset
-            size_categories:
-            - 10K<n<100K
-            source_datasets:
-            - extended|other-yahoo-webscope-l6
-            task_categories:
-            - question-answering
-            task_ids:
-            - open-domain-qa
-            paperswithcode_id:
-            - squad
-            """
-        )
-        with self.assertRaises(TypeError):
-            metadata = DatasetMetadata.from_yaml_string(valid_yaml_string_with_list_paperswithcode_id)
-            metadata.validate()
+            DatasetMetadata.from_yaml_string(duplicate_yaml_keys)
 
         valid_yaml_with_optional_keys = _dedent(
             """\
@@ -490,53 +160,4 @@ def test_from_yaml_string(self):
               I agree to use this model for non-commerical use ONLY: checkbox
             """
         )
-
-        metadata = DatasetMetadata.from_yaml_string(valid_yaml_with_optional_keys)
-        metadata_dict = asdict(metadata)
-        expected = {
-            **default_optional_keys,
-            **default_deprecated_keys,
-            "annotations_creators": ["found"],
-            "language_creators": ["found"],
-            "language": ["en"],
-            "license": ["unknown"],
-            "multilinguality": ["monolingual"],
-            "pretty_name": "Test Dataset",
-            "size_categories": ["10K<n<100K"],
-            "source_datasets": ["extended|other-yahoo-webscope-l6"],
-            "task_categories": ["text-classification"],
-            "task_ids": ["multi-class-classification"],
-            "paperswithcode_id": ["squad"],
-            "configs": ["en"],
-            "train_eval_index": [
-                {
-                    "config": "en",
-                    "task": "text-classification",
-                    "task_id": "multi_class_classification",
-                    "splits": {"train_split": "train", "eval_split": "test"},
-                    "col_mapping": {"text": "text", "label": "target"},
-                    "metrics": [{"type": "accuracy", "name": "Accuracy"}],
-                },
-            ],
-            "extra_gated_prompt": (
-                "By clicking on “Access repository” below, you also agree to ImageNet Terms of Access:\n"
-                '[RESEARCHER_FULLNAME] (the "Researcher") has requested permission to use the ImageNet database '
-                '(the "Database") at Princeton University and Stanford University. In exchange for such permission, '
-                "Researcher hereby agrees to the following terms and conditions:\n"
-                "1. Researcher shall use the Database only for non-commercial research and educational purposes.\n"
-            ),
-            "extra_gated_fields": {
-                "Company": "text",
-                "Country": "text",
-                "I agree to use this model for non-commerical use ONLY": "checkbox",
-            },
-        }
-        self.assertEqual(sorted(metadata_dict), sorted(expected))
-        for key in expected:
-            if key == "train_eval_index":
-                self.assertEqual(len(metadata_dict[key]), len(expected[key]))
-                for tei, expected_tei in zip(metadata_dict[key], expected[key]):
-                    for subkey in expected_tei:
-                        self.assertEqual(tei[subkey], expected_tei[subkey], msg=f"failed at {subkey}")
-            else:
-                self.assertEqual(metadata_dict[key], expected[key], msg=f"failed at {key}")
+        assert DatasetMetadata.from_yaml_string(valid_yaml_with_optional_keys)
diff --git a/tests/test_splits.py b/tests/test_splits.py
new file mode 100644
index 00000000000..bce980e36ab
--- /dev/null
+++ b/tests/test_splits.py
@@ -0,0 +1,36 @@
+import pytest
+
+from datasets.splits import SplitDict, SplitInfo
+from datasets.utils.py_utils import asdict
+
+
+@pytest.mark.parametrize(
+    "split_dict",
+    [
+        SplitDict(),
+        SplitDict({"train": SplitInfo(name="train", num_bytes=1337, num_examples=42, dataset_name="my_dataset")}),
+        SplitDict({"train": SplitInfo(name="train", num_bytes=1337, num_examples=42)}),
+        SplitDict({"train": SplitInfo()}),
+    ],
+)
+def test_split_dict_to_yaml_list(split_dict: SplitDict):
+    split_dict_yaml_list = split_dict._to_yaml_list()
+    assert len(split_dict_yaml_list) == len(split_dict)
+    reloaded = SplitDict._from_yaml_list(split_dict_yaml_list)
+    for split_name, split_info in split_dict.items():
+        # dataset_name field is deprecated, and is therefore not part of the YAML dump
+        split_info.dataset_name = None
+        # the split name of split_dict takes over the name of the split info object
+        split_info.name = split_name
+    assert split_dict == reloaded
+
+
+@pytest.mark.parametrize(
+    "split_info", [SplitInfo(), SplitInfo(dataset_name=None), SplitInfo(dataset_name="my_dataset")]
+)
+def test_split_dict_asdict_has_dataset_name(split_info):
+    # For backward compatibility, we need asdict(split_dict) to return split info dictrionaries with the "dataset_name"
+    # field even if it's deprecated. This way old versionso of `datasets` can still reload dataset_infos.json files
+    split_dict_asdict = asdict(SplitDict({"train": split_info}))
+    assert "dataset_name" in split_dict_asdict["train"]
+    assert split_dict_asdict["train"]["dataset_name"] == split_info.dataset_name


Code review comments:
diff_hunk: @@ -50,7 +50,9 @@ def register_subcommand(parser: ArgumentParser):
             help="Can be used to specify a manual directory to get the files from.",
         )
         test_parser.add_argument("--all_configs", action="store_true", help="Test all dataset configurations")
-        test_parser.add_argument("--save_infos", action="store_true", help="Save the dataset infos file")
+        test_parser.add_argument(
+            "--save_infos", action="store_true", help="Save the dataset infos in the dataset card (README.md)"
path: src/datasets/commands/test.py
text: @user1:
maybe change `--save_infos` to `--save_info` to be consistent with `dataset_info` instead of `dataset_infos` for users? should then be changed in documentation and docstrings and the code below too, if you agree with this change

@author:
Good idea ! I changed to --save_info and kept --save_infos as an alias

diff_hunk: @@ -229,13 +229,13 @@ Now that your dataset script runs and create a dataset with the format you expec
 	```
 	to enable the slow tests, instead of `RUN_SLOW=1`.
 
-3. If all tests pass, your dataset works correctly. You can finally create the metadata JSON by running the command:
+3. If all tests pass, your dataset works correctly. You can finally create the metadata by running the command:
 
 	```bash
 	datasets-cli test datasets/<your-dataset-folder> --save_infos --all_configs
 	```
 
-	This first command should create a `dataset_infos.json` file in your dataset folder.
+	This first command should create a `README.md` file in your dataset folder if it doesn't exist already, containing the metadata.
path: ADD_NEW_DATASET.md
text: @user1:
```suggestion
	This first command should create a `README.md` file containing the metadata if this file doesn't exist already, or add the metadata to an existing `README.md` file in your dataset folder.
```

diff_hunk: @@ -6,11 +6,9 @@
 
 ### Checkbox
 
-- [ ] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
+- [ ] Create the dataset script `./my_dataset/my_dataset.py` using the template
 - [ ] Fill the `_DESCRIPTION` and `_CITATION` variables
 - [ ] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
path: .github/PULL_REQUEST_TEMPLATE/add_dataset.md
text: @user1:
```suggestion
- [ ] Implement `_info()`, `_split_generators()` and `_generate_examples()`
```