Fix size categories in YAML Tags
State: closed | Author: gchhablani (CONTRIBUTOR)

This PR fixes several `size_categories` in YAML tags and makes them consistent. Additionally, I have added a few more categories after `1M`, up to `1T`. I would like to add that to the streamlit app also.

This PR also adds a couple of infos that I found missing.

The code for generating this:
```python
for dataset in sorted(os.listdir('./datasets/')):
    if '.' not in dataset and dataset not in ['c4', 'csv', 'downloads', 'cc100', 'ccaligned_multilingual', 'celeb_a', 'chr_en', 'emea', 'glue']:
        infos = {}
        stats = {}
        st = ''
        with open(f'datasets/{dataset}/README.md') as f:
            d = f.read()
        start_dash = d.find('---') + 3
        end_dash = d[start_dash:].find('---') + 3
        rest_text = d[end_dash + 3:]
        try:
            full_yaml = OmegaConf.create(d[start_dash:end_dash])
            readme = OmegaConf.to_container(full_yaml['size_categories'], resolve=True)
        except Exception as e:
            print(e)
            continue 
        try:
            with open(f'datasets/{dataset}/dataset_infos.json') as f:
                data = json.load(f)
        except Exception as e:
            print(e)
            continue  # Skip those without infos.
        done_set = set([])
        num_keys = len(data.keys())
        for keys in data:
            # dataset = load_dataset('opus100', f'{dirs}')
            total = 0
            for split in data[keys]['splits']:
                total = total + data[keys]['splits'][split]['num_examples']
            if total < 1000:
                st += "- n<1K" + '\n'
                infos[keys] = ["n<1K"]
            elif total >= 1000 and total < 10000:
                infos[keys] = ["1K<n<10K"]
            elif total >= 10000 and total < 100000:
                infos[keys] = ["10K<n<100K"]
            elif total >= 100000 and total < 1000000:
                infos[keys] = ["100K<n<1M"]
            elif total >= 1000000 and total < 10000000:
                infos[keys] = ["1M<n<10M"]
            elif total >= 10000000 and total < 100000000:
                infos[keys] = ["10M<n<100M"]
            elif total >= 100000000 and total < 1000000000:
                infos[keys] = ["100M<n<1B"]
            elif total >= 1000000000 and total < 10000000000:
                infos[keys] = ["1B<n<10B"]
            elif total >= 10000000000 and total < 100000000000:
                infos[keys] = ["10B<n<100B"]
            elif total >= 100000000000 and total < 1000000000000:
                infos[keys] = ["100B<n<1T"]
            else:
                infos[keys] = ["n>1T"]
            done_set = done_set.union(infos[keys])
        if (isinstance(readme, list) and list(infos.values())[0] != readme) or (isinstance(readme, dict) and readme != infos):

            print('-' * 30)
            print(done_set)
            print(f"Changing Full YAML for {dataset}")
            print(OmegaConf.to_yaml(full_yaml))

            if len(done_set) == 1:
                full_yaml['size_categories'] = list(done_set)
            else:
                full_yaml['size_categories'] = dict([(k, v) for k, v in sorted(infos.items(), key=lambda x: x[0])])

            full_yaml_string = OmegaConf.to_yaml(full_yaml)
            print('-' * 30)
            print(full_yaml_string)
            inp = input('Do you wish to continue?(Y/N)')
            if inp == 'Y':
                with open(f'./datasets/{dataset}/README.md', 'w') as f:
                    f.write('---\n')
                    f.write(full_yaml_string)
                    f.write('---')
                    f.write(rest_text)
            else:
                break
```

Note that the lower-bound is inclusive. I'm unsure if this is how it is done in the tagging app.

EDIT:
It would be great if there was a way to make the task categories consistent too. For this, the streamlit app can look into all the datasets and check for existing categories and show them in the list. This may add some consistency.

EDIT:
I understand this will not work for cases where only the infos for some of the configs are present, for example: `ccaligned_multingual` has only 5 out of several configs present, and infos has only information about them.  Hence, I have skipped a few datasets in the code, if there are more such datasets, then I'll ignore them too.

Comments: > It would be great if there was a way to make the task categories consistent too. For this, the streamlit app can look into all the datasets and check for existing categories and show them in the list. This may add some consistency.

We can also update the task lists here: https://github.com/huggingface/datasets-tagging/blob/main/task_set.json

Hi @lhoestq,

Thanks for approving.
How do I add the new categories to the tagging app? What I have added is till `1T` and not `1M`.

I'll also check the task list :)

Thanks,
Gunjan

I think you can change it here: https://github.com/huggingface/datasets-tagging/blob/main/tagging_app.py#L412-L423

Hi @lhoestq,

I have made a PR for size categories on `datasets-tagging`

For tags, I have thought of adding more tags and categories, based on what I know about the existing datasets, any list will not be exhaustive because the contributors can be very specific or very general. Hence, there could be a continuous process of evaluating existing tags and adding more and more.

```json
{
    "image-classification": {
        "description": "image classification tasks",
        "options": [
            "multi-class-classification",
            "multi-label-classification",
            "other"
        ]
    },
    "conditional-text-generation": {
        "description": "data-to-text and text transduction tasks such as translation or summarization",
        "options": [
            "machine-translation",
            "sentence-splitting-fusion",
            "extractive-and-abstractive-summarization",
            "abstractive-summarization",
            "extractive-summarization",
            "multi-document-summarization",
            "table-to-text",
            "text-simplification",
            "explanation-generation",
            "stuctured-to-text",
            "other"
        ]
    },
    "conditional-speech-generation": {
        "description": "speech generation tasks",
        "options": [
            "text-to-speech",
            "speech-translation",
            "other"
        ]
    },

    "conditional-structure-generation":{
        "description": "text or speech to structured data",
        "options":[
            "knowlege-graph-mining",
            "code-generation",
        ]
    },
    "question-answering": {
        "description": "question answering tasks",
        "options": [
            "open-domain-qa",
            "closed-domain-qa",
            "multiple-choice-qa",
            "extractive-qa",
            "abstractive-qa",
            "conversational-qa",
            "multi-document-qa",
            "other"
        ]
    },
    "speech-classification": {
        "description": "speech to label tasks",
        "options": [
            "other"
        ]
    },
    "sequence-modeling": {
        "description": "such as language, speech or dialogue modeling",
        "options": [
            "dialogue-modeling",
            "language-modeling",
            "speech-modeling",
            "multi-turn",
            "slot-filling",
            "other"
        ]
    },
    "speech-recognition": {
        "description": "speech to text tasks",
        "options": [
            "automatic-speech-recognition",
            "other"
        ]
    },
    "structure-prediction": {
        "description": "predicting structural properties of the text, such as syntax",
        "options": [
            "coreference-resolution",
            "named-entity-recognition",
            "part-of-speech-tagging",
            "parsing",
            "sentence-segmentation",
            "single-span-prediction",
            "multi-span-prediction",
            "clause-or-phrase-segmentation",
            "dependency-parsing",
            "constituency-parsing",
            "other"
        ]
    },

    "text-classification": {
        "description": "predicting a class index or boolean value",
        "options": [
            "acceptability-classification",
            "entity-linking-classification",
            "relation-extraction",
            "common-sense-reasoning",
            "fact-checking",
            "intent-classification",
            "multi-class-classification",
            "multi-label-classification",
            "natural-language-inference",
            "semantic-similarity-classification",
            "sentiment-classification",
            "topic-classification",
            "emotion-classification",
            "token-classification",
            "word-sense-disambiguation",
            "offense-classification",
            "hate-speech-classification",
            "language-classification",
            "bias-classification",
            "other"
        ]
    },
    "text-retrieval": {
        "description": "information or text retrieval tasks",
        "options": [
            "document-retrieval",
            "utterance-retrieval",
            "entity-linking-retrieval",
            "fact-checking-retrieval",
            "other"
        ]
    },
    "text-scoring": {
        "description": "text scoring tasks, predicting a real valued score for some text",
        "options": [
            "semantic-similarity-scoring",
            "sentiment-scoring",
            "other"
        ]
    },
    "other": {
        "description": "raw data or other task families",
        "options": [
            "data-mining",
            "raw-text",
            "raw-speech",
            "raw-image",
            "other"
        ]
    }
}
```
I'll sort this when adding it to the .json. Also, I'll change categories according to this if this seems okay to you and commit it to this PR.

I'll also fix spelling others, and some categories which are partially correct, for e.g. `other-machine-translation` to the correct tag.
Lastly, with the options also we can add a description to make it easier for the users to understand what we mean by each option. Example, for "emotion-classification", we can explain what kinds of data we are talking about, or what we mean by "single-span-prediction", etc.

Good idea thank you ! Can you open a PR on datasets-tagging for the tasks as well ?
Also you can update the dataset card with the new tasks categories in another PR if you don't mind

Hi @lhoestq,

Thanks, what all do I need to add to merge this PR?

We can merge this one once the PR on dataset sizes is merged on `datasets-tagging` ;)

Hi @lhoestq,

One problem with this approach is that for datasets like `ccaligned_multilingual`, the infos won't be complete because we don't have all configs. In that case, people might face trouble finding the datatset using the tag. Although, they probably won't be checking the size tag for a dataset like that.

What do you think?

CC @theo-m 

For datasets like `ccaligned_multilingual` it's important to have all the tags for users to search and find it. Currently is has the full list of tags (without the config names). So you can actually find the dataset, but you don't know what tag correspond to what configuration.