Evaluating Rewards for Question Generation Models

Abstract: Recent approaches to question generation have used modifications to a Seq2Seq architecture inspired by advances in machine translation. Models are trained using teacher forcing to optimise only the one-step-ahead prediction. However, at test time, the model is asked to generate a whole sequence, causing errors to propagate through the generation process (exposure bias). A number of authors have proposed countering this bias by optimising for a reward that is less tightly coupled to the training data, using reinforcement learning. We optimise directly for quality metrics, including a novel approach using a discriminator learned directly from the training data. We confirm that policy gradient methods can be used to decouple training from the ground truth, leading to increases in the metrics used as rewards. We perform a human evaluation, and show that although these metrics have previously been assumed to be good proxies for question quality, they are poorly aligned with human judgement and the model simply learns to exploit the weaknesses of the reward source.

Posing questions about a document in natural language is a crucial aspect of the effort to automatically process natural language data, enabling machines to ask clarification questions BIBREF0 , become more robust to queries BIBREF1 , and to act as automatic tutors BIBREF2 .
Recent approaches to question generation have used Seq2Seq BIBREF3 models with attention BIBREF4 and a form of copy mechanism BIBREF5 , BIBREF6 . Such models are trained to generate a plausible question, conditioned on an input document and answer span within that document BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 .
There are currently no dedicated question generation datasets, and authors have used the context-question-answer triples available in SQuAD. Only a single question is available for each context-answer pair, and models are trained using teacher forcing BIBREF11 . This lack of diverse training data combined with the one-step-ahead training procedure exacerbates the problem of exposure bias BIBREF12 . The model does not learn how to distribute probability mass over sequences that are valid but different to the ground truth; during inference, the model must predict the whole sequence, and may not be robust to mistakes during decoding.
Recent work has investigated training the models directly on a performance based objective, either by optimising for BLEU score BIBREF13 or other quality metrics BIBREF10 . By decoupling the training procedure from the ground truth data, the model is able to explore the space of possible questions and become more robust to mistakes during decoding. While the metrics used often seem to be intuitively good choices, there is an assumption that they are good proxies for question quality which has not yet been confirmed.
Our contributions are as follows. We perform fine tuning using a range of rewards, including an adversarial objective. We show that although fine tuning leads to increases in reward scores, the resulting models perform worse when evaluated by human workers. We also demonstrate that the generated questions exploit weaknesses in the reward models.
Many of the advances in natural language generation have been led by machine translation BIBREF3 , BIBREF4 , BIBREF6 .
Previous work on question generation has made extensive use of these techniques. BIBREF8 use a Seq2Seq based model to generate questions conditioned on context-answer pairs, and build on this work by preprocessing the context to resolve coreferences and adding a pointer network BIBREF9 . Similarly, BIBREF7 use a part-of-speech tagger to augment the embedding vectors. Both authors perform a human evaluation of their models, and show significant improvement over their baseline. BIBREF13 use a similar model, but apply it to the task of generating questions without conditioning on a specific answer span. BIBREF14 use a modified context encoder based on multi-perspective context matching BIBREF15 .
 BIBREF16 propose a framework for fine tuning using policy gradients, using BLEU and other automatic metrics linked to the ground truth data as the rewards. BIBREF10 describe a Seq2Seq model with attention and a pointer network, with an additional encoding layer for the answer. They also describe a method for further tuning their model on language model and question answering reward objectives using policy gradients. Unfortunately they do not perform any human evaluation to determine whether this tuning led to improved question quality.
For the related task of summarisation, BIBREF17 propose a framework for fine tuning a summarisation model using reinforcement learning, with the ROUGE similarity metric used as the reward.
The task is to generate a natural language question, conditioned on a document and answer. For example, given the input document “this paper investigates rewards for question generation" and answer “question generation", the model should produce a question such as “what is investigated in the paper?"
We use the model architecture described by BIBREF10 . Briefly, this is a Seq2Seq model BIBREF3 with attention BIBREF4 and copy mechanism BIBREF5 , BIBREF6 . BIBREF10 also add an additional answer encoder layer, and initialise the decoder with a hidden state constructed from the final state of the encoder. Beam search BIBREF18 is used to sample from the model at inference time. The model was trained using maximum likelihood before fine tuning was applied. Our implementation achieves a competitive BLEU-4 score BIBREF19 of $13.5$ on the test set used by BIBREF8 , before fine tuning.
Generated questions should be formed of language that is both fluent and relevant to the context and answer. We therefore performed fine tuning on a trained model, using rewards given either by the negative perplexity under a LSTM language model, or the F1 score attained by a question answering (QA) system, or a weighted combination of both. The language model is a standard recurrent neural network formed of a single LSTM layer. For the QA system, we use QANet BIBREF1 as implemented by BIBREF20 .
Additionally, we propose a novel approach by learning the reward directly from the training data, using a discriminator detailed in Appendix "Discriminator architecture" . We pre-trained the discriminator to predict whether an input question and associated context-answer pair were generated by our model, or originated from the training data. We then used as the reward the probability estimated by the discriminator that a generated question was in fact real. In other words, the generator was rewarded for successfully fooling the discriminator. We also experimented with interleaving updates to the discriminator within the fine tuning phase, allowing the discriminator to become adversarial and adapt alongside the generator.
These rewards $R(\hat{Y})$ were used to update the model parameters via the REINFORCE policy gradient algorithm BIBREF21 , according to $\nabla \mathcal {L} = \nabla \frac{1}{l} \sum \limits _t (\frac{R(\hat{Y})-\mu _R}{\sigma _R}) \log p(\hat{y}_t | \hat{y}_{< t}, \mathbf {D}, \mathbf {A})$ . We teacher forced the decoder with the generated sequence to reproduce the activations calculated during beam search, to enable backpropagation. All rewards were normalised with a simple form of PopArt BIBREF22 , with the running mean $\mu _R$ and standard deviation $\sigma _R$ updated online during training. We continued to apply a maximum likelihood training objective during this fine tuning.
We report the negative log-likelihood (NLL) of the test set under the different models, as well as the corpus level BLEU-4 score BIBREF19 of the generated questions compared to the ground truth. We also report the rewards achieved on the test set, as the QA, LM and discriminator scores.
For the human evaluation, we follow the standard approach in evaluating machine translation systems BIBREF23 , as used for question generation by BIBREF9 . We asked three workers to rate 300 generated questions between 1 (poor) and 5 (good) on two separate criteria: the fluency of the language used, and the relevance of the question to the context document and answer.
Table 2 shows the changes in automatic metrics for models fine tuned on various combinations of rewards, compared to the model without tuning. In all cases, the BLEU score reduced, as the training objective was no longer closely coupled to the training data. In general, models achieved better scores on the metrics on which they were fine tuned. Jointly training on a QA and LM reward resulted in better LM scores than training on only a LM reward. We conclude that fine tuning using policy gradients can be used to attain higher rewards, as expected.
Table 3 shows the human evaluation scores for a subset of the fine tuned models. The model fine tuned on a QA and LM objective is rated as significantly worse by human annotators, despite achieving higher scores in the automatic metrics. In other words, the training objective given by these reward sources does not correspond to true question quality, despite them being intuitively good choices.
The model fine tuned using an adversarial discriminator has also failed to achieve better human ratings, with the discriminator model unable to learn a useful reward source.
Table 1 shows an example where fine tuning has not only failed to improve the quality of generated questions, but has caused the model to exploit the reward source. The model fine tuned on a LM reward has degenerated into producing a loop of words that is evidently deemed probable, while the model trained on a QA reward has learned that it can simply point at the location of the answer. This observation is supported by the metrics; the model fine tuned on a QA reward has suffered a catastrophic worsening in LM score of +226.
Figure 1 shows the automatic scores against human ratings for all rated questions. The correlation coefficient between human relevance and automatic QA scores was 0.439, and between fluency and LM score was only 0.355. While the automatic scores are good indicators of whether a question will achieve the lowest human rating or not, they do not differentiate clearly between the higher ratings: training a model on these objectives will not necessarily learn to generate better questions. A good question will likely attain a high QA and LM score, but the inverse is not true; a sequence may exploit the weaknesses of the metrics and achieve a high score despite being unintelligible to a human. We conclude that fine tuning a question generation model on these rewards does not lead to better quality questions.
In this paper, we investigated the use of external reward sources for fine tuning question generation models to counteract the lack of task-specific training data. We showed that although fine tuning can be used to attain higher rewards, this does not equate to better quality questions when rated by humans. Using QA and LM rewards as a training objective causes the generator to expose the weaknesses in these models, which in turn suggests a possible use of this approach for generating adversarial training examples for QA models. The QA and LM scores are well correlated with human ratings at the lower end of the scale, suggesting they could be used as part of a reranking or filtering system.
We used an architecture based on a modified QANet as shown in Figure 2 , replacing the output layers of the model to produce a single probability. Since the discriminator is also able to consider a full context-question-answer triple as input (as opposed to a context-question pair for the QA task), we fused this information in the output layers.
Specifically, we applied max pooling over time to the output of the first two encoders, and we took the mean of the outputs of the third encoder that formed part of the answer span. These three reduced encodings were concatenated, a 64 unit hidden layer with ReLU activation applied, and the output passed through a single unit sigmoid output layer to give the estimated probability that an input context-question-answer triple originated from the ground truth dataset or was generated.

Figure and table captions:
Table 1: Example generated questions for various fine-tuning objectives. The answer is highlighted in bold. The model trained on a QA reward has learned to simply point at the answer and exploit the QA model, while the model trained on a language model objective has learned to repeat common phrase templates.
Table 2: Changes in automatic evaluation metrics after models were fine tuned on various objectives. QA refers to the F1 score obtained by a question answering system on the generated questions. LM refers to the perplexity of generated questions under a separate language model. The discriminator reward refers to the percentage of generated sequences that fooled the discriminator. Lower LM and NLL scores are better. BLEU scores decreased in all cases.
Table 3: Summary of human evaluation of selected models
Figure 1: Comparison of human and automatic metrics.
Figure 2: Discriminator architecture diagram.

Questions about this paper:
What human evaluation metrics were used in the paper?

Answers:
rating questions on a scale of 1-5 based on fluency of language used and relevance of the question to the context