Contributing MedMCQA dataset
Opened by monk1337 (closed)

Adding MedMCQA dataset ( https://paperswithcode.com/dataset/medmcqa ) 

**Name**: MedMCQA 

**Description**: MedMCQA is a large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions. 

MedMCQA has more than 194k high-quality AIIMS & NEET PG entrance exam MCQs covering 2.4k healthcare topics and 21 medical subjects are collected with an average token length of 12.77 and high topical diversity.

The dataset contains questions about the following topics: Anesthesia, Anatomy, Biochemistry, Dental, ENT, Forensic Medicine (FM), Obstetrics and Gynecology (O&G), Medicine, Microbiology, Ophthalmology, Orthopedics Pathology, Pediatrics, Pharmacology, Physiology, 
Psychiatry, Radiology Skin, Preventive & Social Medicine (PSM), and Surgery


**Code**: https://github.com/medmcqa/medmcqa

All files are at place :

**a dataset script**  : medmcqa.py
**a dataset card with tags and information**  : README.md.
**a metadata file** : dataset_infos.json
**a dummy-data file**  : Please help to generate this file, I was facing 

` raise JSONDecodeError("Extra data", s, end)`  error 

Comments:
@lhoestq Could you please take a look?
Thank you!!

Hi, thank you for the modifications and suggestions. Please check the changes.

Can you run `make style` to fix the code formatting please ?

Oh and was wrong with the dummy_data.zip file, it must actually be placed at `datasets/medmcqa/dummy/1.1.0/dummy_data.zip` - sorry about that

Can you also set the class label names to `names=["a", "b", "c", "d"]` to make it explicit which label corresponds to each answer ? You might have to regenerate `dataset_infos.json` after that

Hi, 

1) Changed the dummy data folder

2) The labels are not ['a', 'b', 'c', 'd'] rather the labels are [1,2,3,4] where 1 represents the 1'st option, 2nd represents 2nd option so on, and its int.

I tried changing to ['a','b','c','d'] and while generating `dataset_infos.json` getting this error :

`ValueError: Class label 4 greater than configured num_classes 4`
Please check.

@lhoestq [lhoestq](https://github.com/lhoestq) Please check

You have this error because we expect the labels to start at 0, not 1. I think you just need to pass `int(data["cop"]) - 1` when generating the examples.

Sorry for the delay in responding btw

@lhoestq I corrected that but here is another issue I am facing while generating `dataset_infos.json`

I am using `" "` if it's test set and otherwise it's the correct option

https://github.com/monk1337/datasets/blob/179f81d48cdd3093302e498babce04c0bf1e33b3/datasets/medmcqa/medmcqa.py#L111
`                    "cop": "" if split == "test" else int(data["cop"]) -1,
`

but while running this command :

`datasets-cli test datasets/medmcqa --save_infos --all_configs
`

giving this error:

```
/content/datasets# datasets-cli test datasets/medmcqa --save_infos --all_configs
Using custom data configuration default
Testing builder 'default' (1/1)
Downloading and preparing dataset med_mcqa/default (download: 52.72 MiB, generated: 128.73 MiB, post-processed: Unknown size, total: 181.46 MiB) to /root/.cache/huggingface/datasets/med_mcqa/default/1.1.0/4c8e418778967b6d9603f79bbfc4fdfbcfffc389664d9aeb85e102cfde418043...
Traceback (most recent call last):                                                                  
  File "/usr/local/bin/datasets-cli", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "/content/datasets/src/datasets/commands/datasets_cli.py", line 33, in main
    service.run()
  File "/content/datasets/src/datasets/commands/test.py", line 162, in run
    try_from_hf_gcs=False,
  File "/content/datasets/src/datasets/builder.py", line 606, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/content/datasets/src/datasets/builder.py", line 1104, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "/content/datasets/src/datasets/builder.py", line 694, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/content/datasets/src/datasets/builder.py", line 1095, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/content/datasets/src/datasets/features/features.py", line 1356, in encode_example
    return encode_nested_example(self, example)
  File "/content/datasets/src/datasets/features/features.py", line 1007, in encode_nested_example
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/content/datasets/src/datasets/features/features.py", line 1007, in <dictcomp>
    return {k: encode_nested_example(sub_schema, sub_obj) for k, (sub_schema, sub_obj) in zip_dict(schema, obj)}
  File "/content/datasets/src/datasets/features/features.py", line 1052, in encode_nested_example
    return schema.encode_example(obj) if obj is not None else None
  File "/content/datasets/src/datasets/features/features.py", line 897, in encode_example
    example_data = self.str2int(example_data)
  File "/content/datasets/src/datasets/features/features.py", line 854, in str2int
    output.append(self._str2int[str(value)])
KeyError: ''
```

Hey ! You can use this instead:
`"cop": -1 if split == "test" else int(data["cop"]) -1`

@lhoestq Thank you for your assistance, and I have updated the `dataset_infos.json` without any error. All the issues are resolved. Please review and approve if it's ready to merge.

Thanks ! There are two things to fic the CI:
1. run `make style` to fix code formatting
2. fix the dummy_data.zip file. Currently it's created from a directory called "dummy" that contains the JSON file, but it should be called "dummy_data" instead

@lhoestq Please check if anything else needs to be done :) 

Let me gently remind you that you can check the CI before pinging reviewers, this way you can know if something needs to be fixed right away.

Right now, if you check the CI, you will see that you didn't fix the code formatting, and that you didn't fix the dummy data.

Let me take a look

_The documentation is not available anymore as the PR was closed or merged._

Hi @lhoestq, I am sorry if I pinged multiple times; I have already corrected the dummy_data file issues and format issue before pinging for the merge request, as you commented last time

_fix the dummy_data.zip file. Currently, it's created from a directory called "dummy" that contains the JSON file, but it should be called "dummy_data" instead._

I fixed the file name and location.

And I also ran the commands last time.

```
make style
flake8 datasets
```
Please let me know if anything else needs to be changed.

Thanks a lot @monk1337 ! :)