GitHub issue #2074: Fix size categories in YAML Tags
State: closed | Created: 2021-03-18T00:02:36Z

This PR fixes several `size_categories` in YAML tags and makes them consistent. Additionally, I have added a few more categories after `1M`, up to `1T`. I would like to add that to the streamlit app also.

This PR also adds a couple of infos that I found missing.

The code for generating this:
```python
for dataset in sorted(os.listdir('./datasets/')):
    if '.' not in dataset and dataset not in ['c4', 'csv', 'downloads', 'cc100', 'ccaligned_multilingual', 'celeb_a', 'chr_en', 'emea', 'glue']:
        infos = {}
        stats = {}
        st = ''
        with open(f'datasets/{dataset}/README.md') as f:
            d = f.read()
        start_dash = d.find('---') + 3
        end_dash = d[start_dash:].find('---') + 3
        rest_text = d[end_dash + 3:]
        try:
            full_yaml = OmegaConf.create(d[start_dash:end_dash])
            readme = OmegaConf.to_container(full_yaml['size_categories'], resolve=True)
        except Exception as e:
            print(e)
            continue 
        try:
            with open(f'datasets/{dataset}/dataset_infos.json') as f:
                data = json.load(f)
        except Exception as e:
            print(e)
            continue  # Skip those without infos.
        done_set = set([])
        num_keys = len(data.keys())
        for keys in data:
            # dataset = load_dataset('opus100', f'{dirs}')
            total = 0
            for split in data[keys]['splits']:
                total = total + data[keys]['splits'][split]['num_examples']
            if total < 1000:
                st += "- n<1K" + '\n'
                infos[keys] = ["n<1K"]
            elif total >= 1000 and total < 10000:
                infos[keys] = ["1K<n<10K"]
            elif total >= 10000 and total < 100000:
                infos[keys] = ["10K<n<100K"]
            elif total >= 100000 and total < 1000000:
                infos[keys] = ["100K<n<1M"]
            elif total >= 1000000 and total < 10000000:
                infos[keys] = ["1M<n<10M"]
            elif total >= 10000000 and total < 100000000:
                infos[keys] = ["10M<n<100M"]
            elif total >= 100000000 and total < 1000000000:
                infos[keys] = ["100M<n<1B"]
            elif total >= 1000000000 and total < 10000000000:
                infos[keys] = ["1B<n<10B"]
            elif total >= 10000000000 and total < 100000000000:
                infos[keys] = ["10B<n<100B"]
            elif total >= 100000000000 and total < 1000000000000:
                infos[keys] = ["100B<n<1T"]
            else:
                infos[keys] = ["n>1T"]
            done_set = done_set.union(infos[keys])
        if (isinstance(readme, list) and list(infos.values())[0] != readme) or (isinstance(readme, dict) and readme != infos):

            print('-' * 30)
            print(done_set)
            print(f"Changing Full YAML for {dataset}")
            print(OmegaConf.to_yaml(full_yaml))

            if len(done_set) == 1:
                full_yaml['size_categories'] = list(done_set)
            else:
                full_yaml['size_categories'] = dict([(k, v) for k, v in sorted(infos.items(), key=lambda x: x[0])])

            full_yaml_string = OmegaConf.to_yaml(full_yaml)
            print('-' * 30)
            print(full_yaml_string)
            inp = input('Do you wish to continue?(Y/N)')
            if inp == 'Y':
                with open(f'./datasets/{dataset}/README.md', 'w') as f:
                    f.write('---\n')
                    f.write(full_yaml_string)
                    f.write('---')
                    f.write(rest_text)
            else:
                break
```

Note that the lower-bound is inclusive. I'm unsure if this is how it is done in the tagging app.

EDIT:
It would be great if there was a way to make the task categories consistent too. For this, the streamlit app can look into all the datasets and check for existing categories and show them in the list. This may add some consistency.

EDIT:
I understand this will not work for cases where only the infos for some of the configs are present, for example: `ccaligned_multingual` has only 5 out of several configs present, and infos has only information about them.  Hence, I have skipped a few datasets in the code, if there are more such datasets, then I'll ignore them too.