Pipeline for question generation
Author: danielduckworth
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

I can see there are pipelines for question answering, text summarisation and text generation. In my field I'm researching how question generation can be used in education research. I would love to see this pipeline added. I imagine it's a variation of question answering and text summarisation.

The paper 'Question Generation by Transformers' by Kettip Kriangchaivech, Artit Wangperawong and provides a good overview of using the SQUAD data set using questions as the output sequence with reference questions and contexts.

Artit also has an implementation called text2text https://github.com/artitw/text2text


## Motivation

It would be useful to have an official pipeline part of the hunggingface library for this use case.

## Your contribution

I'm happy to contribute some funds to pay some developers if need be but I don't have enough Python technical expertise to contribute an PRs myself.


Comments:
That would be an interesting project.

I have worked on question generation using T5. I've trained answer aware question generator on SQuAD 2.2 which achieves 41.4641 BLUE1 score and 41.0823  ROUGE_L on the dev set given gold answers.

I've also trained T5 for extracting answers from the text, and written a simple pipeline where the answer generator generates answers and then the answer-aware que generator generates questions with those answers. You can check the demo [here](https://colab.research.google.com/drive/1_2_mS5l29QHI1pXaqa4YLzAO5xm-HmH9?usp=sharing)

I've also trained T5 for direct question generation on Yahoo questions dataset. It generates single question given a context.

I would be happy to contribute to this project.

@julien-c 
Any update on this ?

We don't have any immediate plan to work on this ourselves but feel free to take a stab

@julien-c 
Okay, I just need your feedback on one thing. I'm not sure if adding a pipeline here will make sense since there are multiple ways to generate questions
1. ans aware
2. generate 1 question directly
3. generate multiple questions simultaneously

Also ans aware model will need an ans extractor/generator or the user will need to supply answers. And different models could process inputs differently. 
So does adding pipeline makes sense here ? If not I can just upload the models and provide a inference script and decide next steps with community feedback. 

Thank you!

@patil-suraj I think it makes sense to generate questions that are answer aware as this has more use cases. I also think that questions should not be so narrow that a single word from the context is the answer. Artit's [Text2Text ](https://github.com/artitw/text2text) did a pretty good job. I used this to generate 1,000 random questions from a random context and plan to have them judged by human raters. One of the main issues I saw in these questions is that sometimes the answer is in the question text. 

I'll take a look at your demo.

I also think it should generate multiple questions that can be so that some evaluation metric (BLUE1, ROUGE_L and METEOR) can be produced and have them ranked. Then the user can decide what to do with the highest metric questions. There's some research that shows the METEOR correlates better with human judgement, but this was for evaluating machine translation tasks and might not apply here.

@danielduckworth 
Yes answer aware seems to be the best chose right now. And we can generate multiple question if we have multiple answers. If you find the above demo interesting then I'll share the models so you can play with it and then we can decide how to proceed from there. 

Also the METEOR score on dev set for demo model is 26.0676

@patil-suraj Yes that would be great. How do you want to share the models?

@danielduckworth 
I've setup everything in the same [colab](https://colab.research.google.com/drive/1_2_mS5l29QHI1pXaqa4YLzAO5xm-HmH9?usp=sharing). Please have a look.

Great, thanks. I'll have a play over the next few days and get back to you.

So just to confirm, where do the models come from? Is the base model T5 and it has been tuned using the SQUAD data reference questions, contexts and answers?

@patil-suraj I've had a quick look, it's very impressive! I've only tried two passages, but the questions are sensible and the answers are more than one word which is better than the Text2Text pipeline. I definitely want to do some more work with this.

First, I'll generate a large set of questions I can get human raters to score to validate whether the quantitative metrics (METEOR etc) correlate with human judgement.

Then I would like to work on some architecture experiments to investigate the following:

1. Can questions be generated where the corresponding generated answer is not explicitly stated in the text? This would require the reader to make connections between information in the text (implied information) and make inferences. I think this could be achieved with some NLTK work but is not ideal as I think the semantics of these questions are best learned from text data rather than expert-system rules.

2. Can additional tuning of the que_gen model be done with other question/answer/context datasets that are of a different text type. For example, Wikipedia is primary factual information texts. But what about discursive texts? Or narrative texts?

Anyway, I'll continue to explore what you have with a developer I work with and maybe we can form repository we can work in with the goal of creating  a pipeline for inclusion in the Transformer package.

What do you think?

> Great, thanks. I'll have a play over the next few days and get back to you.
> 
> So just to confirm, where do the models come from? Is the base model T5 and it has been tuned using the SQUAD data reference questions, contexts and answers?

Yes both of the models are t5-base trained on SQuAD

> @patil-suraj I've had a quick look, it's very impressive! I've only tried two passages, but the questions are sensible and the answers are more than one word which is better than the Text2Text pipeline. I definitely want to do some more work with this.
> 
> First, I'll generate a large set of questions I can get human raters to score to validate whether the quantitative metrics (METEOR etc) correlate with human judgement.
> 
> Then I would like to work on some architecture experiments to investigate the following:
> 
> 1. Can questions be generated where the corresponding generated answer is not explicitly stated in the text? This would require the reader to make connections between information in the text (implied information) and make inferences. I think this could be achieved with some NLTK work but is not ideal as I think the semantics of these questions are best learned from text data rather than expert-system rules.
> 2. Can additional tuning of the que_gen model be done with other question/answer/context datasets that are of a different text type. For example, Wikipedia is primary factual information texts. But what about discursive texts? Or narrative texts?
> 
> Anyway, I'll continue to explore what you have with a developer I work with and maybe we can form repository we can work in with the goal of creating a pipeline for inclusion in the Transformer package.
> 
> What do you think?

@danielduckworth 
I'm not sure about the first, we will need to run small experiments and see if it can be achieved.

2) Yes, I do think additional fine-tuning on more diverse datasets should improve the results. My goal is to first get factual questions correct and then move to narrative texts.

And sure we can create different repo and take this forward.

@patil-suraj Thanks for the examples. Would you mind sharing your fine-tuning code used to train the model as well? 

Sure. I'm planning to release model as well as the fine-tuning code. I'll comment here once I do that.

@danai-antoniou Thanks for the wonderful suggestion. 

@patil-suraj, I also played your colab code, and it looks super great.  I look forward to the release.

@danielduckworth, I am looking into recent works in Question Generation especially using Transformer based architecture leveraging fine-tuning for small data sets. This thread is very interesting to me.

> Can questions be generated where the corresponding generated answer is not explicitly stated in the text? This would require the reader to make connections between information in the text (implied information) and make inferences. I think this could be achieved with some NLTK work but is not ideal as I think the semantics of these questions are best learned from text data rather than expert-system rules.

I have some previous experience in generating non-factoid questions from a text, with the goal of having descriptive answers rather than quiz-like. In my project, the data was not sufficient for DL models, even for fine-tuning.

We had done some user studies on generated questions as well, and found out METEOR is better correlated with human judgment on how reasonable or well-formed the questions are rather than BLEU or ROUGE scores.

For extracting relations in text, to capture more complex answers, Semantic Role Labeling (SRL) + Dependency parse tree of the text might be useful for extracting some descriptive answers. I used a tool called ClearNLP to do that.

@patil-suraj, I also checked out your Collab demo and the questions generated are looking good, much better than other models I worked with so far. Great job. Definitely looking forward to the release of the models and knowing more about the fine-tuning process.

For non-factoid questions, there is room for improvement.


Hi @emadg, @hunkim  Thank you for your interest! :)

@emadg

>For extracting relations in text, to capture more complex answers, Semantic Role Labeling (SRL) + Dependency parse tree of the text might be useful for extracting some descriptive answers. I used a tool called ClearNLP to do that.
 
This sure sounds like a good idea. My current goal is to have a end-to-end model for generating questions and answers.

> For non-factoid questions, there is room for improvement.

For those looking for less factual questions, I was actually able to get some reasonable results with a T5 pre-trained for query prediction, but there's definitely room for improvement. 

https://github.com/castorini/docTTTTTquery

Hey people, I've setup few experiments for question generation. Let me know if anyone wants to collaborate on this, I would really appreciate some help and maybe some multi-gpu compute. Everything will be open sourced after the experiments are finished.
Thank you!  

This is very interesting to me. I'm writing a master's thesis over the summer, working on transfer learning for question generation. I don't have much experience with contributing to large pre-existing frameworks like this but would definitely be happy to contribute wherever I can. 

@patil-suraj 
I had a look at your notebook. This is very impressive! Looking forward to seeing the fine-tuning process to get an idea of how this can be done using the transformer framework. 

@patil-suraj 
I would like to help. Let me know if we can collaborate. Although I have somewhat limited time to contribute.

Hi @emadg  and @vegarab, thank you for your interest,
My goal is to do open source study on que generation. Here's what I have planned 
For ans aware que generation we usually need 3 models 
first which will extract ans like spans
second model will generate question on that answer
and third will be a QA model which will take the question and produce an answer, 
then we can compare the two answers to see if the generated question is correct or not.

Having 3 models for single task is lot of complexity, so goal is to create a multi-task model which can do all of these 3 tasks
1. extract ans like spans
2. generate question based on the answer 
3. QA

Also I want to see if we can generate multiple questions end-to-end without answers.

Another experiment is generating non-factoid questions. First we need to find a right dataset for this.

I've trained t5-small model in multi-task way and its giving really good results, so now I want to train more models (t5-base, t5-large, bart-base, bart-large, bert-2-bert) and see if they improve the results.

I've also trained t5-small and t5-base for end-2-end QG and that too is giving interesting results. 

So regarding  help I'm looking for some compute to train large models, multitask t5-small took 10hrs on single V100 GPU. I also want someone to provide rigorous feedback on the work (find out mistakes, asses quality of questions etc) and help with creating a write-up for study.

Hi all! @patil-suraj really great work! I am using your approach and seems to be working very well.

I have one question though, when fine-tuning the models, did you use a case or an uncased model? Because when giving too much uppercase text as context, it's generating questions partially or totally in uppercase and are much worse. It seems to be a cased model because when lowercasing the text, the questions are much better.

Thanks in advance!

@patil-suraj,

> and third will be a QA model which will take the question and produce an answer

I think using a QA model to evaluate the generated Question is an interesting approach. Why should do this instead of evaluating with BLEU or METEOR score?

> So regarding help I'm looking for some compute to train large models, multitask t5-small took 10hrs on single V100 GPU.

Can we use GCP for the compute? if it is not going to cost a lot. I don't have GPUs myself.

> I also want someone to provide rigorous feedback on the work (find out mistakes, asses quality of questions etc) and help with creating a write-up for study.

I think I can spend some time and provide feedback about the work. Although, I need to catch up with the details related to T5 model and the fine-tuning method used here.

Interesting thread @patil-suraj I had the similar thoughts on multi task training 

- Finetune the model combining the data for both question generation & answering(one example is **context:c1 answer: a1 ---> question : q1**  & another example context:c1 question : q1 ----> answer:a1)
- Way to generate multiple questions is either using topk and topp sampling or using multiple beams.




Hey everyone,
here's a sneak peek of whats coming, everything will be available by the end of this week. stay tuned !
![ss](https://i.ibb.co/6RYFyfT/qg-ss.png)

@santhoshkolloju 
> * Way to generate multiple questions is either using topk and topp sampling or using multiple beams.

Yes, this is what I have tried in another model.


Hi @emadg 

> Why should do this instead of evaluating with BLEU or METEOR score?

BLEU and METEOR  can be used to evaluate the model when you have the original reference questions, but at inference time how can we decide if the generated question is correct (makes sense or not, has answer or not) or not without original question ? Which is why the QA model.

>I think I can spend some time and provide feedback about the work. Although, I need to catch up with the details related to T5 model and the fine-tuning method used here.

You can start your analysis once I make it available. Human feedbacks will be most valuable. Thanks ! 



Hi all, tagging everyone for notification 
@emadg , @vegarab , @gabisurita , @hunkim , @ugmSorcero , @santhoshkolloju .

Happy to finally release the project. You can find everything in [this repo](https://github.com/patil-suraj/question_generation).

[All models](https://huggingface.co/models?filter=question-generation) are available on hub with configured inference API. You can search using question-generation tag.

Here’s a [colab](https://colab.research.google.com/github/patil-suraj/question_generation/blob/master/question_generation.ipynb) if anyone wants to play more with it.

@patil-suraj Thanks!!

Any thoughts about sentence-aware question generation techniques? 
1) For each context from Squad dataset, extract the sentence where the answer is present and provide the triplet (context, sentence, question)  as model inputs for training. This would decouple our necessity  during inference to select an answer from the context since we can now randomly select an input sentence (or all the sentence) from the context and generate a question for each sentence from the context.
P(Question | Content, Sentence) would be the objective. 

Hi @kaushalshetty , you can try this with the current model as well, try this [model](https://huggingface.co/valhalla/t5-base-qg-hl) and instead of highlighting a span highlight the entire sentence and see what you get 

It's finally here! Really great work @patil-suraj !!!

I have one question:
When generating questions given answers and context, you try to find the answer within the context in order to highlight it, but there might be two problems with this approach:

- First, with the first model that you released for generating answers, sometimes I found that the generated answer was not corresponding to the original text, thus this approach would not actually find the answer within the context. But I don't know if the new model always generates answers that appear exactly in the same way than in the original text
- Second, what if an answer appears several times in the same sentence?

I don't know if you have thought about this, but maybe in these cases it would be better to take the approach to append the context after the answers.
Tell me what you think
Thanks!!

Thanks @ugmSorcero !

1. Yes, I have also observed these issues, see issue patil-suraj/question_generation#11 The initial goal of the project was to generate reading comprehension style questions like SQuAD where answers will always be in the text and I wanted to train BERT like model for answer extraction  modelled as a span extraction task. But I wanted to keep everything as simple as possible and the text-to-text approach gave good results, decided to do it using T5. Only in few cases it produces answers that are somewhat different from the context text. See the issue for details.

2. This is the exact reason I choose the highlighting approach. If you simply prepend the answer and if that answer occurs multiple times in text then the model might get confused. When extracting the answer, each sentence is highlighted and then answers are extracted only for that sentence and it's less likely that the same answer will occur several times in the same sentence, so this takes care of the issue.

I've added one model which uses the prepend approach. You can train more if you want using the prepend approach, it'll just take two commands!

Feel free to raise an issue [here](https://github.com/patil-suraj/question_generation) if you want to discuss more. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


For future reference, my pipeline using doc2query is published on https://github.com/unicamp-dl/corpus2question.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.