SlowGuess/ACL_2020_OCR huggingface dataset:
Xin Jia, Wenjie Zhou, Xu Sun, Yunfang Wu*
MOE Key Lab of Computational Linguistics, School of EECS, Peking University
{jemmryx, wjzhou013, xusun, wuyf}@pku.edu.cn
Given a sentence and its relevant answer, how to ask good questions is a challenging task, which has many real applications. Inspired by human's paraphrasing capability to ask questions of the same meaning but with diverse expressions, we propose to incorporate paraphrase knowledge into question generation(QG) to generate human-like questions. Specifically, we present a two-hand hybrid model leveraging a self-built paraphrase resource, which is automatically conducted by a simple back-translation method. On the one hand, we conduct multi-task learning with sentence-level paraphrase generation (PG) as an auxiliary task to supplement paraphrase knowledge to the task-share encoder. On the other hand, we adopt a new loss function for diversity training to introduce more question patterns to QG. Extensive experimental results show that our proposed model obtains obvious performance gain over several strong baselines, and further human evaluation validates that our model can ask questions of high quality by leveraging paraphrase knowledge.
Question generation (QG) is an essential task for NLP, which focuses on generating grammatical questions for given paragraphs or sentences. It plays a vital role in various realistic scenarios. For educational purposes, QG can create reading comprehension materials for language learners (Heilman and Smith, 2010). For business use, QG can bring benefits to conversation systems and chat-bots for effective communication with humans (Mostafazadeh et al., 2016). Besides, automatically-generated questions can be conversely used for constructing question answering datasets to enhance reading comprehension sys
Sentence:
the next three drives of the game would end in punts.
Reference question:
what did the next three drives result in?
Question generated by the baseline model:
the next three drives of the game would end in what?
in ring theory, the notion of number is generally replaced with that of ideal.
ring theory
in what theory is the idea of a number exchanged with that of an ideal?
in what theory is the notion of number replaced with that of ideal?
Table 1: Real examples of generated questions from SQuAD. We highlight the paraphrase transitions between sentences and questions. Human creates good questions by leveraging paraphrase knowledge, while the automatically generated questions just copy the original sentence, resulting in lower evaluation scores.
tems (Tang et al., 2017; Duan et al., 2017; Xu et al., 2019; Zhang and Bansal, 2019).
Recent neural network-based methods have achieved promising results on QG, most of which are based on the seq2seq attention framework (Du et al., 2017; Zhou et al., 2017; Gao et al., 2018; Kim et al., 2018; Zhou et al., 2019b), enriched with lexical features (Zhou et al., 2017; Sun et al., 2018; Song et al., 2018) or enhanced by copy mechanism (Du and Cardie, 2018; Sun et al., 2018; Zhou et al., 2019a).
Although much progress has been made for QG, existing approaches do not explicitly model the "notorious" lexical and syntactic gaps in the generation process. That is, some parts of two texts (e.g. the input sentence and reference question, the reference question and generated question) may convey the same meaning but use different words, phrases or syntactic patterns. In real communica
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6130-6140
Figure 1: A sketch of our design to leverage paraphrase knowledge in QG.
tion, humans often paraphrase a source sentence to ask questions which are grammatical and coherent. Take SQuAD (Rajpurkar et al., 2016) as an example, which is a popular reading comprehension dataset and has been widely used for QG, there is a large percentage of questions created by paraphrasing (33.3% of the questions contain synonymy variations and 64% of questions contain syntactic variations (Rajpurkar et al., 2016)). Two examples are shown in Table 1. Due to the lack of paraphrase knowledge, the generated questions simply copy certain words from the input sequence, the quality of which is thus not competitive with human-created questions.
To address this issue, we introduce paraphrase knowledge in the QG process to generate human-like questions. The sketch of our design is illustrated in Figure 1. To make our model easy to implement and train the model in an end-to-end fashion, we do not use any extra paraphrase generation (PG) dataset but just use a simple back-translation method to automatically create paraphrases for both the input sentences and reference questions. Based on the high-quality expanded data, we propose a two-hand hybrid model. On the left hand, using the expanded sentence paraphrase as the target of PG, we perform multi-task learning with PG and QG, to optimize the task-share encoder with the paraphrase knowledge. On the right hand, with the gold reference question and question paraphrase as QG's multi-targets, we adopt a new min-loss function, to enable the QG module to learn more diverse question patterns.
We conduct extensive experiments on SQuAD and MARCO (Nguyen et al., 2016). Results show that both separate modules, the PG auxiliary task and the min-loss function, obviously improve the performances of QG task, and combing them achieves further improvements. Furthermore, human evaluation results show that our hybrid model can ask better and more human-like questions by incorporating paraphrase knowledge.
For current mainstream neural network-based methods on QG, most approaches utilize the Seq2Seq model with attention mechanism (Du et al., 2017; Zhou et al., 2017; Zhao et al., 2018b; Zhou et al., 2019a). To obtain better representations of the input sequence and answer, the answer position and token lexical features are treated as supplements for the neural encoder (Zhou et al., 2017; Song et al., 2018; Kim et al., 2018). Similar to other text generation tasks, many works on QG also employ copy or pointer mechanism to overcome the OOV problem (Du and Cardie, 2018; Sun et al., 2018; Zhang and Bansal, 2019). Recently, Zhou et al. (2019a) employ language modeling (LM) as an auxiliary task to enrich the encoder representations. In this paper, we adopt this work as one of the baseline models, since their universal model is easy to implement and achieves promising results for QG.
In order to make use of the context information of paragraphs, Zhao et al. (2018b) propose a gated self-attention network to encode context passage. Based on this, Zhang and Bansal (2019) apply reinforcement learning to deal with semantic drift in QG; Nema et al. (2019) use a passage-answer fusion mechanism to obtain answer-focused context representations; Li et al. (2019a) utilize gated attention to fuse answer-relevant relation with context sentence. Besides, Chen et al. (2019) design different passage graphs to capture structure information of passage through graph neural networks. Dong et al. (2019) propose a unified language model pre-training method to obtain better context representations for QG. All these works adopt a whole paragraph as input to generate questions. Different from this, our work only takes a sentence as input and leaves paragraph-level QG for future research.
Paraphrase generation is also a challenging task for NLP. Recent works usually obtain paraphrases by reordering or modifying the syntax or lexicon based on some paraphrase databases and rules (Fader et al., 2013; Chen et al., 2016), or by employing some neural generation methods (Prakash et al., 2016; Li et al., 2019b). In this paper, we employ a simple and effective paraphrasing method to expand both input sentences and reference questions. Our method also can be replaced with more sophisticated paraphrasing methods.
Paraphrase knowledge has been used to improve many NLP tasks, such as machine translation, ques
6131
tion answering, and text simplification. Callison-Burch et al. (2006) use paraphrase techniques to deal with unknown phrases to improve statistical machine translation. Fader et al. (2013) and Dong et al. (2017) employ paraphrase knowledge to enhance question answering models. Kriz et al. (2018) utilize paraphrase and context-based lexical substitution knowledge to improve simplification task. Similarly, Zhao et al. (2018a) combine paraphrase rules of PPDB (Ganitkevitch et al., 2013) with Transformer (Vaswani et al., 2017) to perform sentence simplification task. Guo et al. (2018a) propose a multi-task learning framework with PG and simplification. In addition, Yu et al. (2018) and Xie et al. (2019) use paraphrase as data argumentation for their primary tasks. Different from these works, we leverage paraphrase knowledge for question generation, by automatically constructing a built-in paraphrase corpus without using any external paraphrase knowledge bases.
3 Model Description
In this section, we first describe two baseline models we used: feature-enriched pointer-generator and language modeling enhanced QG. Then we explain how to obtain paraphrase resources and show the quality statistics. Furthermore, we describe in detail two modules of utilizing paraphrase knowledge: the PG auxiliary task and the min loss function, as well as their combination. The overall structure of our hybrid model is shown in Figure 2.
3.1 Baseline Models
3.1.1 Feature-enriched Pointer-generator
Sun et al. (2018) enhance pointer-generator (See et al., 2017) model with rich features proposed by Zhou et al. (2017). They adopt a bidirectional LSTM as the encoder, which takes the feature-enriched embedding  $e_i$  as input:
$$
e _ {i} = \left[ w _ {i}; a _ {i}; n _ {i}; p _ {i}; u _ {i} \right] \tag {1}
$$
where  $w_{i},a_{i},n_{i},p_{i},u_{i}$  respectively represents embeddings of word, answer position, name entity, POS and word case.
Same as the decoder used by See et al. (2017), another unidirectional LSTM with attention mechanism is used to obtain the decoder hidden state  $s_t$  and context vector  $c_t$ . Based on these, the pointer-generator model will simultaneously calculate the probabilities of generating a word from vocabulary and copying a word from the source text. The final
probability distribution is the combination of these two modes with a generation probability  $p_{g}$ :
$$
P (w) = p _ {g} P _ {v o c a b} + (1 - p _ {g}) P _ {c o p y} \tag {2}
$$
The training objective is to minimize the negative log likelihood of the target sequence  $\mathbf{q}$ :
$$
\mathcal {L} _ {q g} = - \frac {1}{T _ {q g}} \sum_ {t = 1} ^ {T _ {q g}} \log P \left(y _ {t} ^ {q g} = \mathbf {q} _ {t}\right) \tag {3}
$$
3.1.2 Language Modeling Enhanced QG
Zhou et al. (2019a) enhance QG with language modeling under a hierarchical structure of multitask learning. The language modeling aims at predicting the next and previous words in the input sequence with forward and backward LSTMs, respectively, which serves as a low-level task to provide semantic information for the high-level QG task.
In general, the input sequence will firstly be fed into the language modeling module to get the semantic hidden states, then these states will be concatenated with the input sequence to obtain the input of the feature-rich encoder:
$$
e _ {i} = \left[ w _ {i}; a _ {i}; n _ {i}; p _ {i}; u _ {i}; h _ {i} ^ {l m} \right] \tag {4}
$$
where  $h_i^{lm}$  is the semantic hidden state of LM module. The loss function of language modeling is defined as:
$$
\begin{array}{l} \mathcal {L} _ {l m} = - \frac {1}{T _ {l m} - 1} \sum_ {t = 1} ^ {T _ {l m} - 1} \log (P ^ {l m} (w _ {t + 1} | w _ {<   t + 1})) \\ - \frac {1}{T _ {l m} - 1} \sum_ {t = 2} ^ {T _ {l m}} \log \left(P ^ {l m} \left(w _ {t - 1} \mid w _ {> t - 1}\right)\right) \tag {5} \\ \end{array}
$$
where  $P^{lm}(w_{t + 1}|w_{< t + 1})$  and  $P^{lm}(w_{t - 1}|w_{>t - 1})$  represent the generation probabilities of the next word and the previous word, respectively.
As a result, the total loss of language modeling enhanced QG is formulated as:
$$
\mathcal {L} _ {l q g} = \mathcal {L} _ {q g} + \beta \mathcal {L} _ {l m} \tag {6}
$$
where  $\beta$  is a hyper-parameter to control the relative importance between language modeling and QG. Follow the work of Zhou et al. (2019a), we set  $\beta$  to 0.6. We re-implement this unified model to base our method on a strong baseline.
3.2 Paraphrase Expansion
The paraphrasing strategy is independent of the neural-based QG model, and we can use any advanced methods to generate paraphrases. In our work, we employ a simple back-translation method to automatically create paraphrases of both sentences and questions. Specially, we use a mature translation tool Google Translate, which is a free and accessible online service. We translate an original text into German and then back to English to get its paraphrase. As a result, we obtain  $\mathbf{s}'$  which is the paraphrase of the input sentence  $\mathbf{s}$ , and  $\mathbf{q}'$  which is the paraphrase of the golden reference question  $\mathbf{q}$ . In the following section, we will illustrate the way to use  $(\mathbf{s}, \mathbf{s}')$  as a training pair of the auxiliary PG task, and adopt  $(\mathbf{q}, \mathbf{q}')$  as multi-references to conduct the diversity training module. The way we expand paraphrases does not need extra PG datasets. Besides, it guarantees the PG and QG tasks share the same input  $\mathbf{s}$ , so we can optimize their sharing encoder simultaneously and train the model end-to-end.
Synonym	Syntactic	Fluency
sentence-paraphrase	74%	7%	67%
question-paraphrase	58%	44%	67%
To assess the quality of expanded paraphrases, we randomly select 100 paraphrases respectively from sentences and questions, and ask two annotators to judge the Synonym conversions and Syntactic transitions, as well as the paraphrase Fluency. As shown in Table 2,  $74\%$  sentence paraphrases and  $58\%$  question paraphrases have synonym conversions with source sequences,  $7\%$  and  $44\%$  of them have sentence pattern transitions. Besides,  $67\%$  of paraphrases have no grammar errors. Two real expansion examples are shown in Table 3. It indicates that our expansion method introduces rich and high quality paraphrasing knowledge into the original data.
3.3 Multi-task Learning with Paraphrase Generation
3.3.1 Auxiliary PG Task
The multi-task learning mechanism with PG aims at introducing paraphrase knowledge into QG. In general, we employ a parallel architecture to combine PG and QG, where QG is the main task and PG serves as an auxiliary task. To make our model
Table 2: Human evaluation of expanded paraphrases.
Input Sentence:
the current basilica of the sacred heart is located on the spot of fr.
Sentence Paraphrase:
the present basilica of the sacred heart is located in the place of fr.
Input Question:
what structure is found on the location of the original church of father sorin at notre dame?
Question Paraphrase:
what structure can be found at the location of the original church of father sorin at notre dame?
Table 3: Real examples of our paraphrase expansion on the sentences and reference questions respectively. We mark paraphrase transitions with color.
easy to implement and can be trained end-to-end, we conduct the multi-task learning in a simultaneous mode. In detail, feature-riched embeddings will first be encoded by the task-share encoder and then be fed into PG and QG decoders respectively. The PG and QG decoders both have two layers and they are identical in the structure but different in parameters.
In the auxiliary PG task, the input is the original sentence s, and the training objective is to minimize the cross-entropy loss:
$$
\mathcal {L} _ {p g} = - \frac {1}{T _ {p g}} \sum_ {t = 1} ^ {T _ {p g}} \log P \left(y _ {t} ^ {p g} = \mathbf {s} _ {t} ^ {\prime}\right) \tag {7}
$$
where  $y_{t}^{pg}$  is the generated word of PG at time step  $t$  and  $\mathbf{s}_t^\prime$  is the  $t$  th word in the expanded sentence paraphrase  $\mathbf{s}^{\prime}$ .
3.3.2 Soft Sharing Strategy
To enhance the impact of auxiliary PG task so that the paraphrase knowledge can be absorbed by the question generation process more deeply, we employ a soft sharing strategy between the first layer of PG and QG decoders. The soft sharing strategy loosely couples parameters and encourages them close to each other in representation space. Following the work of Guo et al. (2018b), we minimize the  $l_{2}$  distance between the shared layer of QG and PG decoders as a regularization. The soft sharing loss is defined as:
$$
\mathcal {L} _ {s f} = \sum_ {\mathbf {d} \in \mathcal {D}} | | \theta_ {d} - \phi_ {d} | | _ {2} \tag {8}
$$
where  $\mathcal{D}$  is the set of shared decoder parameters,  $\theta$  and  $\phi$  respectively represent the parameters of the main QG task and the auxiliary PG task.
6133
Figure 2: Illustration of our proposed hybrid model.
3.4 Diversity Training with Min-loss Function
For the QG task, a general training goal is to fit the decoded results with the reference questions. To provide more generation patterns, we adjust the training target from one golden reference question to several reference questions by using expanded paraphrase resources. We adopt a min-loss function among several references, and the loss function defined by Equation 3 can be rewritten as:
$$
\mathcal {L} _ {q g} = \min  _ {\mathbf {q} \in \mathcal {Q}} (- \frac {1}{T _ {q g}} \sum_ {t = 1} ^ {T _ {q g}} \log P (y _ {t} ^ {q g} = \mathbf {q} _ {t})) \quad (9)
$$
where  $\mathcal{Q}$  is the set of gold reference question and expanded question paraphrase  $\{q, q'\}$ . Each generated question will separately calculate the negative log-likelihood of its multiple references, and the final loss is the minimum of them. Under this training process, our model can learn multiple question expressions which are not in the original training dataset, so that the generation can be more diverse.
Besides, inspired by the work of Kovaleva et al. (2018), we have tried several loss strategies, such as minimum loss, maximum loss, and weighted loss to guide the diversity training. Among them, the minimum is the best performing strategy. By employing minimum strategy, the QG decoder fits the generated question with the most similar sequence among gold reference question and question para
phrase. In this way, more question patterns are introduced into QG process.
3.5 Hybrid Model
Combining the above modules, we get our hybrid model. During training, the feature-enriched inputs are first encoded by the task-share encoder. Then the semantic hidden states are fed into PG decoder and QG decoder, respectively. For PG decoder, it has one fitting target (expanded sentence paraphrase). For QG decoder, it calculates the cross-entropy loss with both the gold reference question and the question paraphrase and regards the minimum loss of them as the QG loss. The auxiliary PG task and diversity training strategy simultaneously optimize the question generation process. The combined training loss function can be defined as:
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {\text {l q g}} + \alpha \mathcal {L} _ {\text {p g}} + \lambda \mathcal {L} _ {\text {s f}} \tag {10}
$$
where  $\alpha$  and  $\lambda$  are both hyper-parameters. We will describe the chosen of these hyper-parameters later.
Our experiments are based on two reading comprehension datasets: SQuAD (2016) and MARCO (2016). On SQuAD, since there are two different splits that are most often used, we conduct experiments on both two splits on sentence-level. For
6134
Zhou Split	Du Split
Previous Works (conference-year)	B1	B2	B3	B4	MET	B1	B2	B3	B4	MET
s2s (ACL-2017)	-	-	-	-	-	43.09	25.96	17.50	12.28	16.62
NQG++ (NLPCC-2017)	-	-	-	13.29	-	-	-	-	-	-
M2S+cp (NAACL-2018)	-	-	-	13.91	-	-	-	-	13.98	18.77
A-P-Hybrid (EMNLP-2018)	43.02	28.14	20.51	15.64	-	-	-	-	-	-
s2sa-at-mp-gsa (EMNLP-2018)	44.51	29.07	21.06	15.82	19.67	43.47	28.23	20.40	15.32	19.29
ASs2s (AAAI-2019)	-	-	-	16.17	-	-	-	-	16.20	19.92
LM enhanced QG (EMNLP-2019)	42.80	28.43	21.08	16.23	-	-	-	-	-	-
Q-type (EMNLP-2019)	43.11	29.13	21.29	16.31	-	-	-	-	-	-
Sent-Relation (EMNLP-2019)	44.40	29.48	21.54	16.37	20.68	45.66	30.21	21.82	16.27	20.36
Our Models
baseline-1 +Data augmentation	38.16	24.35	17.60	13.28	17.73	38.91	24.80	17.83	13.36	17.97
baseline-1	41.06	26.63	19.65	14.71	19.12	41.04	27.05	19.92	15.21	19.19
baseline-1 +Min	42.03	27.61	20.27	15.48	19.61	42.97	28.52	21.02	16.06	19.93
baseline-1 + PG	42.76	28.26	20.89	16.09	20.11	43.68	28.99	21.39	16.37	20.23
baseline-1 +Min+PG (hybrid model-1)	43.61	28.67	21.09	16.23	20.29	42.66	28.68	21.39	16.55	20.44
baseline-2	42.39	28.11	20.86	16.13	19.95	42.76	28.80	21.47	16.57	20.38
baseline-2 +Min	43.38	28.92	21.49	16.61	20.40	42.94	29.06	21.73	16.88	20.60
baseline-2 +PG	43.56	28.98	21.57	16.74	20.58	43.73	29.53	22.06	17.08	20.78
baseline-2 +Min+PG (hybrid model-2)	43.63	29.21	21.79	16.93	20.58	44.32	29.88	22.28	17.21	20.96
Table 4: Experimental results of our models on SQuAD comparing with previous works and different baselines. The results of previous works are copied from their original papers. Baseline-1 and Baseline-2 refer to Feature-enriched Pointer-generator and LM enhanced QG respectively. Bn: BLEU-n, MET:METOER.
Du Split (Du et al., 2017), we use the same settings with Li et al. (2019a) and there are 74689, 10427 and 11609 sentence-question-answer triples for training, validation and test respectively. For Zhou Split (Zhou et al., 2017), we use the data shared by Zhou et al. (2017) and there are 86,635, 8,965 and 8,964 triples correspondingly. On MARCO, there are 74,097, 4,539 and 4,539 sentence-answer-question triples for train, development and test sets, respectively (Sun et al., 2018).
We expand the datasets using the paraphrase expansion approach described in Section 3.2. After that, one sample of the expanded dataset is in the form of ((sentence, sentence paraphrase), (question, question paraphrase), answer).
For fair comparison, we report the following recent works on sentence-level Du and Zhou Splits:
s2s (Du et al., 2017): an attention-based seq2seq model.
$\mathbf{NQG} + +$  (Zhou et al., 2017): a feature-enriched Seq2Seq model.
M2S+cp (Song et al., 2018): uses different matching strategies to explicitly model the information between answer and context.
A-P-Hybrid (Sun et al., 2018): generates an accurate interrogative word and focuses on important context words.
s2s-a-ct-mp-gsa (Zhao et al., 2018b): employs a gated attention encoder and a maxout pointer decoder to deal with long text inputs.
ASs2s (Kim et al., 2018): proposes an answer-separated Seq2Seq model by replacing the answer in the input sequence with some specific words.
LM enhanced QG (Zhou et al., 2019a): treats language modeling as a low-level task to provide semantic representations for the high-level QG.
Q-type (Zhou et al., 2019b): multi-task learning framework with question word prediction and QG.
Sent-Relation (Li et al., 2019a): extracts answer-relevant relations in sentence and encodes both sentence and relations to capture answer-focused representations.
We evaluate the performance of our models using BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014), which are widely used in previous works for QG.
We set the vocabulary as the most frequent 20,000 words. We use 300-dimensional GloVe word vectors as initialization of the word embeddings. Answer position and token lexical features are randomly initialized to 32-dimensional vectors through truncated normal distribution. The maximum lengths of input sequence and output sequence are 100 and 40, respectively. The hidden
6135
size of the encoder, decoder, and language modeling LSTMs are all 512. We use Adagrad optimization with learning rate 0.15 for training. The batch size is 32 and the beam search decoding size is 12. To alleviate the volatility of the training procedure, we get the average model of the 5 checkpoints closest to the best-trained model on development set.
The experimental results on two splits of SQuAD are shown in Table 4. In terms of BLEU-4 that is often regarded as the main evaluation metric for text generation, our hybrid model-2 yields the best results on both splits, with 16.93 on Zhou Split and 17.21 on Du Split. We achieve state-of-the-art results on Du Split for sentence-level QG.