Deep learning NLP models for children's story understanding?
Author: jeffxtang
Pull request: no

I'm working on building NLP systems with common sense reasoning, starting with children's story understanding. I'm very interested in applying the latest pre-trained models here (and maybe Facebook's Roberta too) to a story (not one of the tested datasets like Squad 2.0 and GLUE) for QA, but am not sure how to approach it.

If the answers can be found in the text, will a modified script of run_squad.py be expected to achieve about 90% accuracy? What if the answers need commonsense knowledge and reasoning not explicitly specified in the text?

For example, if we use one of the models (Bert, GPT-2, XLNet, Roberta...) to process the Aesop's story The Fox and the Grapes, will it be able to answer questions such as:

What did the Fox gaze at when his mouth watered?
How many times did the Fox try to get the grapes?
Why did the Fox's mouth water?
Were the grapes sour or ripe?

*****
THE FOX AND THE GRAPES

A Fox one day spied a beautiful bunch of ripe grapes hanging from
a vine trained along the branches of a tree. The grapes seemed
ready to burst with juice, and the Fox's mouth watered as he
gazed longingly at them.

The bunch hung from a high branch, and the Fox had to jump for
it. The first time he jumped he missed it by a long way. So he
walked off a short distance and took a running leap at it, only
to fall short once more. Again and again he tried, but in vain.

Now he sat down and looked at the grapes in disgust.

"What a fool I am," he said. "Here I am wearing myself out to get
a bunch of sour grapes that are not worth gaping for."

And off he walked very, very scornfully.
*****

If not, what do we need to do to be able to answer the questions?

Thanks for any suggestions and thoughts!



Comments:
Hi! Your best bet is indeed to use the models that are state-of-the-art on question answering. It is currently a modified version of BERT (see SpanBERT). I cannot tell you what the accuracy would be on your dataset however, as unfortunately, these models are very sensitive to dataset changes. The SQuAD model (fine-tuned on Wikipedia) probably wouldn't get you groundbreaking results. 

You can still try it with our BERT model fine-tuned on SQuAD (`bert-large-uncased-whole-word-masking-finetuned-squad`)

If you are looking to increase the accuracy on a specific set of documents (from my understanding you’re focusing on children stories), it might be a good idea to fine-tune your model on a similar dataset. Doing so would probably yield better results on your question answering. cc @thomwolf 

Thank you @LysandreJik for your comment. Can "the models that are state-of-the-art on question answering" here answer questions which require background knowledge and reasoning not explicitly stated in the text?

Yes I'm focusing on children stories. Do you think The Children’s Book Test of the Facebook bAbi project (https://research.fb.com/downloads/babi/) might be a good dataset to fine tune the model on?

Two more questions please: Is there a tutorial on how to prepare a dataset for question answering to fine-tune the Bert model?

If such a dataset is hard to obtain or a lot more data would be needed, would a rule-based method be more practical? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@LysandreJik @thomwolf First of all, thanks a lot for your team's great work on the Swift Core ML implementations of Bert and GPT-2. I just got the chance to try out the BERT-SQuAD iOS sample and it works pretty amazingly if the answer is located in the text, although questions that require some kind of reasoning or answers that are not explicitly stated in the text like motivations or causes/effects are still tough to get right.

Do you think a hybrid approach of using rule-based common sense knowledge and reasoning with the latest deep learning NLP models would be the best way to answer questions which require background knowledge and reasoning not explicitly stated in the text?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


SQuAD is extractive question answering so will only give you spans inside the original text

By the way on common sense reasoning, you can check out this great repo by @atcbosselut: https://github.com/atcbosselut/comet-commonsense

Thanks @julien-c. I took a brief look at the paper a few months ago and will check out the repo and study the paper more carefully.