Add bAbI QA tasks
State: closed | Author: gchhablani (CONTRIBUTOR)

- **Name:** *The (20) QA bAbI tasks*
- **Description:** *The (20) QA bAbI tasks are a set of proxy tasks that evaluate reading comprehension via question answering. Our tasks measure understanding in several ways: whether a system is able to answer questions via chaining facts, simple induction, deduction and many more. The tasks are designed to be prerequisites for any system that aims to be capable of conversing with a human. The aim is to classify these tasks into skill sets,so that researchers can identify (and then rectify) the failings of their systems.*
- **Paper:** [arXiv](https://arxiv.org/pdf/1502.05698.pdf)
- **Data:** [Facebook Research Page](https://research.fb.com/downloads/babi/)
- **Motivation:** This is a unique dataset with story-based Question Answering. It  is a part of the `bAbI` project by Facebook Research.

**Note**: I have currently added all the 160 configs. If this seems impractical, I can keep only a few. While each `dummy_data.zip` weighs a few KBs, overall it is around 1.3MB for all configurations. This is problematic. Let me know what is to be done.

Thanks :)


### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Comments: Hi @lhoestq,

Should I remove the 160 configurations? Is it too much?

EDIT:
Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

Thanks for the changes !

> Should I remove the 160 configurations? Is it too much?

Yea 160 configuration is a lot.
Maybe this dataset can work with parameters `type` and `task_no` ?
You can just remove the configuration in BUILDER_CONFIGS to only keep a few ones.
Also feel free to add an example in the dataset card of how to load the other configurations
```
load_dataset("babi_qa", type="hn", task_no="qa1")
```
for example, and with a list of the possible combinations.

> Can you also check the task category? I'm not sure if there is an appropriate tag for the same.

It looks appropriate, thanks :)

Hi @lhoestq 

I'm unable to test it locally using:
```python
load_dataset("datasets/babi_qa", type="hn", task_no="qa1")
```
It raises an error:
```python
TypeError: __init__() got an unexpected keyword argument 'type'
```
Will this be possible only after merging? Or am I missing something here?

Can you try adding this class attribute to `BabiQa` ?
```python
BUILDER_CONFIG_CLASS = BabiQaConfig
```
This should fix the TypeError issue you got

My bad. Thanks a lot!

Hi @lhoestq 

I have added the changes. Only the "qa1" task for each category is included. Also, I haven't removed the size categories and other description because I think it will still be useful. I have updated the line in README showing the example.

Thanks,
Gunjan

Hi @lhoestq,

Does this look good now?