Add MetaShift dataset
Opened by dnaveenr (closed)

This PR adds the MetaShift dataset.

Dataset Request  : Add MetaShift dataset [#3813](https://github.com/huggingface/datasets/issues/3813)

@lhoestq As discussed, 
- I have copied the preprocessing script and modified it as required to not create new directories and folders and instead yield the images. 
-  I do the preprocessing in _split_generators to get the required data which is then passed to _generate_examples.
-  Beyond the generated MetaShift dataset, the original preprocess script also generates the meta-graphs for each class, I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#generate-full-metashift) ]
-  There is a Bonus section, the authors share.  I have currently not included this part. [ Ref : [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- I had a basic test script which downloaded the dataset and tested the basic functionality. Things seems fine.
For real data, I performed the following test : 
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_metashift
============================================== test session starts ===============================================
platform linux -- Python 3.7.11, pytest-7.0.1, pluggy-1.0.0
rootdir: ./datasets
plugins: hydra-core-1.1.1, datadir-1.3.1, forked-1.4.0, xdist-2.5.0
collected 1 item                                                                                                 

tests/test_dataset_common.py .                                                                             [100%]

========================================= 1 passed in 4821.25s (1:20:21) =========================================
```
- I couldn't get the dummy dataset. Need some inputs here.
Error as follows : 
```
Using custom data configuration default
Dataset metashift with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

To-Do : 

- [x] Currently I am using the default _SELECTED_CLASSES. I need to use config option here as suggested
- [x] Complete fields in the Dataset Card.
- [x] Tagging the dataset using the Datasets Tagging app. 

Need your help and suggestions for improvement. Thank you


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lhoestq Please could you review this when you get time. Thank you.

Thanks a lot for your inputs @mariosasko .
> Maybe we can add the generated meta-graphs to the card as images (with attributions)?

Yes. We can do this for the default set of classes. Will add this.

> Would be cool if we could have them as additional configs. Also, maybe we could have configs that expose [image metadata](https://github.com/Weixin-Liang/MetaShift/tree/main/dataset/meta_data) from the https://nlp.stanford.edu/data/gqa/sceneGraphs.zip file (this file is downloaded in the script but not used).

I'll try adding the bonus section as additional config. 
Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?


> Regarding exposing the image metadata with a config parameter, how will we showcase/display this information ?

Oh, I forgot to mention that. Let's add a `Dataset Usage` section to the card to document the params (similar to this: https://huggingface.co/datasets/electricity_load_diagrams#dataset-usage). Also, feel free to add the constants that can be tuned as config params (e.g. `IMAGE_SUBSET_SIZE_THRESHOLD` or the `5` in `len(subject_data) <= 5`).

Okay. Got it. Will add these and constants as config parameters.

The image metadata from scene graphs looks like this : 
```json
{
    "2407890": {
        "width": 640,
        "height": 480,
        "location": "living room",
        "weather": none,
        "objects": {
            "271881": {
                "name": "chair",
                "x": 220,
                "y": 310,
                "w": 50,
                "h": 80,
                "attributes": ["brown", "wooden", "small"],
                "relations": {
                    "32452": {
                        "name": "on",
                        "object": "275312"
                    },
                    "32452": {
                        "name": "near",
                        "object": "279472"
                    }                    
                }
            }
        }
    }
}
```
``load_dataset("metashift", selected_classes=["cat", "dog", ...], image_metadata=True)``
How do we showcase/display the image metadata(json) information ?


> How do we showcase/display the image metadata(json) information ?

We can add the JSON fields as keys to the features dict:
```python
  if self.config.image_metadata:
      features.update({"width": Value("int"), "height": Value("int"), "location": Value("string"), ...}) 
```

P.S. Would rename `image_metadata`  to `with_image_metadata` 

I have added the following : 
- Added the meta-graphs to the card as images under the Section "Dataset Meta-Graphs".
- Generate the Attributes-Dataset using config parameter. [ [Link](https://github.com/Weixin-Liang/MetaShift#bonus-generate-the-metashift-attributes-dataset-subsets-defined-by-subject-attributes) ]
- Expose image metadata using config parameter.
Format of the image metadata is as follows : [Link](https://cs.stanford.edu/people/dorarad/gqa/download.html)
I have modified the "Objects" which is dict to a list of dicts with an additional parameter named object_id. 
I have defined the structure as follows : 
```
{
    "width": datasets.Value("int64"),
    "height": datasets.Value("int64"),
    "location": datasets.Value("string"),
    "weather": datasets.Value("string"),
    "objects": datasets.Sequence(
        {
            "object_id": datasets.Value("string"),
            "name": datasets.Value("string"),
            "x": datasets.Value("int64"),
            "y": datasets.Value("int64"),
            "w": datasets.Value("int64"),
            "h": datasets.Value("int64"),
            "attributes": datasets.Sequence(datasets.Value("string")),
            "relations": datasets.Sequence(
                {
                    "name": datasets.Value("string"),
                    "object": datasets.Value("string"),
                }
            ),
        }
    ),
}
```
Problem is that objects is not being shown as list of dicts. The output looks as follows : 

> metashift_dataset['train'][0]

```json 
{'image_id': '2338755', 'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x281 at 0x7F066C5A49D0>, 'label': 0, 'context': 'ground', 'width': 500, 'height': 281, 'location': None, 'weather': None, 'objects': {'object_id': ['3070704', '3070705', '3070706', '2416713', '3070702', '2790660', '3063157', '2354960', '2037127', '2392939', '2912743', '2125407', '2735257', '3260906', '2351018', '3288269', '3699852', '2734378', '3421201', '2863115'], 'name': ['bicycle', 'bicycle', 'bicycle', 'boot', 'bicycle', 'motorcycle', 'pepperoni', 'head', 'building', 'wall', 'shorts', 'people', 'wheel', 'bricks', 'man', 'cat', 'boot', 'door', 'ground', 'building'], 'x': [137, 371, 458, 215, 468, 399, 368, 245, 0, 140, 260, 284, 138, 451, 339, 187, 210, 26, 0, 313], 'y': [116, 86, 94, 150, 91, 80, 107, 22, 0, 44, 109, 69, 145, 226, 69, 22, 230, 0, 119, 0], 'w': [197, 27, 15, 73, 24, 53, 9, 37, 289, 46, 43, 30, 74, 28, 35, 116, 53, 107, 500, 55], 'h': [126, 25, 38, 128, 43, 50, 16, 44, 158, 73, 51, 52, 97, 15, 73, 252, 46, 147, 162, 77], 'attributes': [[], [], [], ['white'], [], [], [], [], [], [], [], [], [], [], [], ['white'], ['white'], ['large', 'black'], ['brick'], []], 'relations': [{'name': ['to the left of'], 'object': ['3260906']}, {'name': ['to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['3070706', '2351018', '2125407', '2790660', '2037127', '3070702', '3288269']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the right of'], 'object': ['2351018', '3070705', '3070702', '2790660', '3063157']}, {'name': ['to the right of'], 'object': ['2735257']}, {'name': ['to the right of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['2351018', '2790660', '3070706', '3070705', '3063157']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the right of', 'to the right of'], 'object': ['3070705', '2351018', '3070702', '3070706', '3063157', '2125407', '2037127', '3288269']}, {'name': ['to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the right of', 'to the left of', 'to the right of'], 'object': ['2037127', '3070706', '3070702', '2912743', '3288269', '2790660', '2125407']}, {'name': ['to the left of', 'to the right of'], 'object': ['2863115', '2734378']}, {'name': ['to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['3070705', '2351018', '3063157', '2125407', '2790660', '2863115']}, {'name': ['to the left of', 'to the right of', 'to the left of'], 'object': ['2125407', '2734378', '3288269']}, {'name': ['to the left of', 'on', 'to the left of'], 'object': ['2351018', '3288269', '3063157']}, {'name': ['to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'to the left of'], 'object': ['3063157', '2351018', '2037127', '3070705', '2392939', '2790660']}, {'name': ['to the left of', 'to the left of'], 'object': ['2416713', '3288269']}, {'name': ['to the right of'], 'object': ['3070704']}, {'name': ['to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the right of', 'to the left of', 'to the right of', 'walking down'], 'object': ['2037127', '2790660', '2125407', '3070705', '3070706', '2912743', '3070702', '3288269', '3421201']}, {'name': ['to the right of', 'to the right of', 'to the left of', 'to the right of', 'to the left of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2392939', '2734378', '2790660', '2735257', '3063157', '3070705', '2351018', '2863115']}, {'name': [], 'object': []}, {'name': ['of', 'to the left of', 'to the left of', 'to the left of'], 'object': ['2037127', '2354960', '3288269', '2392939']}, {'name': [], 'object': []}, {'name': ['to the right of', 'to the right of', 'to the right of'], 'object': ['2037127', '3288269', '2354960']}]}}
```
Expected output of image_metadata would be : 
```
{'height': 281,
 'location': None,
 'objects': [{'attributes': [],
              'h': 126,
              'name': 'bicycle',
              'object_id': '3070704',
              'relations': [{'name': 'to the left of', 'object': '3260906'}],
              'w': 197,
              'x': 137,
              'y': 116},
             {'attributes': [],
              'h': 25,
              'name': 'bicycle',
              'object_id': '3070705',
              'relations': [{'name': 'to the left of', 'object': '3070706'},
                            {'name': 'to the right of', 'object': '2351018'},
                            {'name': 'to the right of', 'object': '2125407'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '3070702'},
                            {'name': 'to the right of', 'object': '3288269'}],
              'w': 27,
              'x': 371,
              'y': 86},
             {'attributes': ['white'],
              'h': 252,
              'name': 'cat',
              'object_id': '3288269',
              'relations': [{'name': 'to the right of', 'object': '2392939'},
                            {'name': 'to the right of', 'object': '2734378'},
                            {'name': 'to the left of', 'object': '2790660'},
                            {'name': 'to the right of', 'object': '2735257'},
                            {'name': 'to the left of', 'object': '3063157'},
                            {'name': 'to the left of', 'object': '3070705'},
                            {'name': 'to the left of', 'object': '2351018'},
                            {'name': 'to the left of', 'object': '2863115'}],
              'w': 116,
              'x': 187,
              'y': 22},
             {'attributes': ['white'],
              'h': 46,
              'name': 'boot',
              'object_id': '3699852',
              'relations': [],
              'w': 53,
              'x': 210,
              'y': 230},
             .
             .
             .
             {'attributes': ['large', 'black'],
              'h': 147,
              'name': 'door',
              'object_id': '2734378',
              'relations': [{'name': 'of', 'object': '2037127'},
                            {'name': 'to the left of', 'object': '2354960'},
                            {'name': 'to the left of', 'object': '3288269'},
                            {'name': 'to the left of', 'object': '2392939'}],
              'w': 107,
              'x': 26,
              'y': 0},
             {'attributes': ['brick'],
              'h': 162,
              'name': 'ground',
              'object_id': '3421201',
              'relations': [],
              'w': 500,
              'x': 0,
              'y': 119},
             {'attributes': [],
              'h': 77,
              'name': 'building',
              'object_id': '2863115',
              'relations': [{'name': 'to the right of', 'object': '2037127'},
                            {'name': 'to the right of', 'object': '3288269'},
                            {'name': 'to the right of', 'object': '2354960'}],
              'w': 55,
              'x': 313,
              'y': 0}],
 'weather': None,
 'width': 500}

```

May I know how to get the list of dicts representation correctly ?

---
To-Do : 

- [x] Generate dataset_infos.json file.
- [x] Add “Dataset Usage” section in the cards and write about the config parameters. 
- [x] Add the constants that can be tuned as config params.


> Problem is that objects is not being shown as list of dicts. The output looks as follows :

That's expected. We convert a sequence of dictionaries to a dictionary of sequences to keep the formatting aligned with Tensorflow Datasets. You could disable this behavior by replacing `"objects": datasets.Sequence(object_fields_dict)` with `"objects": [object_fields_dict]`, but that's not what we usually do, so let's keep it like that. 

Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the `src` attribute (and specify `alt` in case the URLs go down).

I'll do a proper review again after you are finished with the dummy data.

> That's expected.

Okay. Got it. Thanks. I thought I was doing something wrong.

> Also, to limit the size of the dataset repo, please remove the pushed images and pass URLs to the images instead under the src attribute (and specify alt in case the URLs go down).

Sure. Where do we host these images ? Can I upload them to any free image hosting platform or is there any particular website you use ?

> I'll do a proper review again after you are finished with the dummy data.

Sure. Thanks. I'm working on this part. Will update you.


Update : 
- I have generated the dataset_infos.json file.

> I suggest you try to generate the dataset_infos.json file first, and then I can help with the dummy data.

I am having issues creating the dummy data. I get the following which I use the command : 

`datasets-cli dummy_data datasets/metashift`

```
Dataset metashift with config MetashiftConfig(name='metashift', version=1.0.0, data_dir=None, data_files=None, description=None) seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data/full-candidate-subsets.pkl.
Traceback (most recent call last):
  File "datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/datasets/commands/dummy_data.py", line 324, in run
    dataset_builder=dataset_builder, mock_dl_manager=mock_dl_manager
  File "/datasets/commands/dummy_data.py", line 407, in _print_dummy_data_instructions
    for split in generator_splits:
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```

> Feel free to host the images online (on imgur for example) :)

Sure. Will do that.

Thanks for the explanation regarding the dummy data zip files. I will try it out and let you know.

Instead of uploading the images to a hosting service, you can directly reference their GitHub URLs (open the image in the MetaShift repo -> click Download -> copy the image URL). For instance, this is the URL of one of the images:`https://raw.githubusercontent.com/Weixin-Liang/MetaShift/main/docs/figures/Cat-MetaGraph.jpg`. Also, feel free to replace `main` with the most recent commit hash in the copied URLs to make them more robust.

@mariosasko I've actually created metagraphs for all the default classes other than those present in the GitHub Repo and included all of them. :)  The Repo has them only for two classes.

In case we want to limit the no.of meta graphs included, we can stick to the github URLs from the repo itself.


Update : 
- I could add the dummy data and get the dummy data test to work. Since we have a preprocessing step on the dataset, one of the .pkl file size is on the higher side. This was done for the tests to pass. I hope that is okay. The dummy.zip file size is about 273K.

To-Do :
- [x] Update Dataset Structure in the data cards to include Data Instances when config parameters are used.

Please could you review when you get time. Thank you.

Thanks a lot for your suggestions, Mario. The thing I learnt from the review is that I need to make better sentence formations. I will keep this in mind. :) 

Thanks a lot for your support. @mariosasko  and @lhoestq .

> Super impressed by your work on this, congrats :)

Its my first dataset contribution to the 🤗 Datasets library, I'm super excited. Thank you. :)

Also, I think we can close this request issue now, [#3813](https://github.com/huggingface/datasets/issues/3813)