SlowGuess/ACL_2020_OCR huggingface dataset:
One of the most important challenges of the QAG task is enforcing consistency between the generated question and its corresponding answer. They should be semantically consistent, such that it is possible to predict the answer given the question and the context. However, neural QG or QAG models often generate questions irrelevant to the context and the answer (Zhang and Bansal, 2019) due to the lack of the mechanism enforcing this consistency. We tackle this issue by maximizing the mutual information (MI) of a generated QA pair, assuming that an answerable QA pair will have high MI. Since an exact computation of MI is intractable, we use a neural approximation. While there exist many different approximations (Belghazi et al., 2018; Hjelm et al., 2019), we use the estimation proposed by Yeh and Chen (2019) based on Jensen-Shannon Divergence:
$$
\begin{array}{l} \operatorname {M I} (X; Y) \geq \mathbb {E} _ {\mathbf {x}, \mathbf {y} \sim \mathbb {P}} [ \log g (\mathbf {x}, \mathbf {y}) ] \\ + \frac {1}{2} \mathbb {E} _ {\tilde {\mathbf {x}}, \mathbf {y} \sim \mathbb {N}} [ \log (1 - g (\tilde {\mathbf {x}}, \mathbf {y})) ] \\ + \frac {1}{2} \mathbb {E} _ {\mathbf {x}, \tilde {\mathbf {y}} \sim \mathbb {N}} [ \log (1 - g (\mathbf {x}, \tilde {\mathbf {y}})) ] \\ =: \mathcal {L} _ {\text {I n f o}} \\ \end{array}
$$
where  $\mathbb{E}_{\mathbb{P}}$  and  $\mathbb{E}_{\mathbb{N}}$  denote expectation over positive and negative examples. We generate negative examples by shuffling the QA pairs in the minibatch, such that a question is randomly associated with an answer. Intuitively, the function  $g(\cdot)$  acts like a binary classifier that discriminates whether QA pair is from joint distribution or not. We empirically find that the following  $g(\cdot)$  effectively achieves our goal of consistent QAG:
$$
g (\mathbf {x}, \mathbf {y}) = \operatorname {s i g m o i d} (\overline {{\mathbf {x}}} ^ {T} \mathbf {W} \overline {{\mathbf {y}}})
$$
where  $\overline{\mathbf{x}} = \frac{1}{N}\sum_{i}\hat{\mathbf{d}}_i$  and  $\overline{\mathbf{y}} = \frac{1}{L}\sum_{j}\hat{\mathbf{h}}_j$  are summarized representations of question and answer, respectively. Combined with the ELBO, the final
objective of our Info-HCVAE is as follows:
$$
\max  _ {\Theta} \mathcal {L} _ {\mathrm {H C V A E}} + \lambda \mathcal {L} _ {\mathrm {I n f o}}
$$
where  $\Theta$  includes all the parameters of  $\phi, \psi, \theta$  and  $\mathbf{W}$ , and  $\lambda$  controls the effect of MI maximization. In all experiments, we always set the  $\lambda$  as 1.
Stanford Question Answering Dataset v1.1 (SQuAD) (Rajpurkar et al., 2016). This is a reading comprehension dataset consisting of questions obtained from crowdsourcing on a set of Wikipedia articles, where the answer to every question is a segment of text or a span from the corresponding reading passage. We use the same split used in Zhang and Bansal (2019) for the fair comparison.
Natural Questions (NQ) (Kwiatkowski et al., 2019). This dataset contains realistic questions from actual user queries to a search engine, using Wikipedia articles as context. We adapt the dataset provided from MRQA shared task (Fisch et al., 2019) and convert it into the extractive QA format. We split the original validation set in half, to use as validation and test for our experiments.
TriviaQA (Joshi et al., 2017). This is a reading comprehension dataset containing question-answer-evidence triples. The QA pairs and the evidence (contexts) documents are authored and uploaded by Trivia enthusiasts. Again, we only choose QA pairs of which answers are span of contexts.
HarvestingQA This dataset contains top-ranking 10K Wikipedia articles and 1M synthetic QA pairs generated from them, by the answer span extraction and QG system proposed in (Du and Cardie, 2018). We use this dataset for semi-supervised learning.
Implementation Details In all experiments, we use BERT-base ( $d = 768$ ) (Devlin et al., 2019) as the QA model, setting most of the hyperparameters as described in the original paper. For both HCVAE and Info-HCVAE, we set the hidden dimensionality of the Bi-LSTM to 300 for posterior, prior, and answer generation networks, and use the dimensionality of 450 and 900 for the encoder and the decoder of the question generation network. We set the dimensionality of  $\mathbf{z}_{\mathbf{x}}$  as 50, and define  $\mathbf{z}_{\mathbf{y}}$  to be set of
10-way categorical variables  $\mathbf{z_y} = \{\mathbf{z}_1,\dots ,\mathbf{z}_{20}\}$ . For training the QA model, we fine-tune the model for 2 epochs. We train both the QA model and Info-HCVAE with Adam optimizer (Kingma and Ba, 2015) with the batch size of 32 and the initial learning rate of  $5\cdot 10^{-5}$  and  $10^{-3}$  respectively. For semi-supervised learning, we first pre-train BERT on the synthetic data for 2 epochs and fine-tune it on the GT dataset for 2 epochs. To prevent posterior collapse, we multiply 0.1 to the KL divergence terms of question and answer (Higgins et al., 2017). For more details of the datasets and experimental setup, please see Appendix C.
Baselines We experiment two variants of our model against several baselines:
1. Harvest-QG: An attention-based neural QG model with a neural answer extraction system (Du and Cardie, 2018).
2. Maxout-QG: A neural QG model based on maxout copy mechanism with a gated self-attention (Zhao et al., 2018), which uses BERT as the word embedding as suggested by Zhang and Bansal (2019).
3. Semantic-QG: A neural QG model based on Maxout-QG with semantic-enhanced reinforcement learning (Zhang and Bansal, 2019).
4. HCVAE: Our HCVAE model without the InfoMax regularizer.
5. Info-HCVAE: Our full model with the InfoMax regularizer.
For the baselines, we use the same answer spans extracted by the answer extraction system (Du and Cardie, 2018).
QAE and R-QAE One of crucial challenges with generative models is a lack of a good quantitative evaluation metric. We adopt QA-based Evaluation (QAE) metric proposed by Zhang and Bansal (2019) to measure the quality of QA pair. QAE is obtained by first training the QA model on the synthetic data, and then evaluating the QA model with human annotated test data. However, QAE only measures how well the distribution of synthetic QA pairs matches the distribution of GT QA pairs, and does not consider the diversity of QA pairs. Thus, we propose Reverse QA-based Evaluation (R-QAE), which is the accuracy of the QA model trained on the human-annotated QA pairs, evaluated on the generated QA pairs. If the synthetic
$^{2}$ https://github.com/xinyadu/harvestingQA
Method	QAE (↑)	R-QAE (↓)
SQuAD (EM/F1)
Harvesting-QG	55.11/66.40	64.77/78.85
Maxout-QG	56.08/67.50	62.49/78.24
Semantic-QG	60.49/71.81	74.23/88.54
HCVAE	69.46/80.79	37.57/61.24
Info-HCVAE	71.18/81.51	38.80/60.73
Natural Questions (EM/F1)
Harvesting-QG	27.91/41.23	49.89/70.01
Maxout-QG	30.98/44.96	49.96/70.03
Semantic-QG	30.59/45.29	58.42/79.23
HCVAE	31.45/46.77	32.78/55.12
Info-HCVAE	37.18/51.46	29.39/53.04
TriviaQA (EM/F1)
Harvesting-QG	21.32/30.21	29.75/47.73
Maxout-QG	24.58/34.32	31.56/49.92
Semantic-QG	27.54/38.25	37.45/58.15
HCVAE	30.20/40.88	34.41/48.16
Info-HCVAE	35.45/44.11	21.65/37.65
Table 2: QAE and R-QAE results on three datasets. All results are the performances on our test set.
Harvest-QG	Maxout-QG	Semantic-QG	HCVAE	Info-HCVAE
111.74	114.58	112.94	113.89	117.41
data covers larger distribution than the human annotated training data, R-QAE will be lower. However, note that having a low R-QAE is only meaningful when the QAE is high enough since trivially invalid questions may also yield low R-QAE.
Results We compare HCVAE and Info-HCVAE with the baseline models on SQuAD, NQ, and TriviaQA. We use  $10\%$  of Wikipedia paragraphs from HarvestingQA (Du and Cardie, 2018) for evaluation. Table 2 shows that both HCVAE and Info-HCVAE significantly outperforms all baselines by large margin in QAE on all three datasets, while obtaining significantly lower R-QAE, which shows that our model generated both high-quality and diverse QA pairs from the given context. Moreover, Info-HCVAE largely outperforms HCVAE, which demonstrates the effectiveness of our InfoMax regularizer for enforcing QA-pair consistency.
Figure 3 shows the accuracy as a function of number of QA pairs. Our Info-HCVAE outperform all baselines by large margins using orders of magnitude smaller number of QA pairs. For example, Info-HCVAE achieves 61.38 points using 12K QA pairs, outperforming Semantic-QG that use 10 times larger number of QA pairs. We also report
Figure 3: QAE vs. # of QA pairs (log-scaled) on SQuAD.
Table 3: The results of mutual information estimation. The results are based on QA pairs generated from  $\mathrm{H} \times 10\%$ .
Method	QAE (↑)	R-QAE (↓)
Baseline	56.08/67.50	62.49/78.24
+Q-latent	58.66/70.54	40.00/62.02
+A-latent	69.46/80.79	37.57/61.24
+InfoMax	71.18/81.51	38.80/60.73
Table 4: QAE and R-QAE results of the ablation study on SQuAD dataset. All the results are the performances on our test set.
the score of  $\overline{\mathbf{x}}^T\mathbf{W}\overline{\mathbf{y}}$  as an approximate estimation of mutual information (MI) between QA pairs generated by each method in Table 3; our Info-HCVAE yields the largest value of MI estimation.
Ablation Study We further perform an ablation study to see the effect of each model component. We start with the model without any latent variables, which is essentially a deterministic Seq2Seq model (denoted as Baseline in Table 4). Then, we add in the question latent variable (+Q-latent) and then the answer latent variable (+A-latent), to see the effect of probabilistic latent variable modeling and hierarchical modeling respectively. The results in Table 4 shows that both are essential for improving both the quality (QAE) and diversity (R-QAE) of the generated QA pairs. Finally, adding in the InfoMax regularization (+InfoMax) further improves the performance by enhancing the consistency of the generated QA pairs.
Human Evaluation As a qualitative analysis, we first conduct a pairwise human evaluation of the QA pairs generated by our Info-HCVAE and Maxout-QG on 100 randomly selected paragraphs. Specifically, 20 human judges performed blind quality assessment of two sets of QA pairs that are presented in a random order, each of which contained two to five QA pairs. Each set of QA pairs is evalu
Method	Diversity	Consistency	Overall
Baseline	26%	34%	30%
Ours	47%	50%	52%
Tie	27%	16%	18%
Table 5: The results of human judgement in terms of diversity, consistency, and overall quality on the generated QA pairs.
Paragraph The scotland act 1998 which was passed by and given royal assent by queen Elizabeth ii on 19 november 1998, governs functions and role of the scottish parliament and delimits its legislative competence ...
GT what act sets forth the functions of the scottish parliament?
O-1 which act was passed in 1998?
O-2 which act governs role of the scottish parliament?
O-3 which act was passed by queen Elizabeth ii?
O-4 which act gave the scottish parliament the responsibility to determine its legislative policy?
ated in terms of the overall quality, diversity, and consistency between the generated QA pairs and the context. The results in Table 5 show that the QA pairs generated by our Info-HCVAE is evaluated to be more diverse and consistent, compared to ones generated by the baseline models.
One-to-Many QG To show that our Info-HCVAE can effectively tackle one-to-many mapping problem for question generation, we qualitatively analyze the generated questions for given a context and an answer from the SQuAD validation set. Specifically, we sample the question latent variables multiple times using the question prior network  $p_{\psi}(\mathbf{z}_{\mathbf{x}} \mid \mathbf{c})$ , and then feed them to question generation networks  $p_{\theta}(\mathbf{x} \mid \mathbf{z}_{\mathbf{x}}, \mathbf{y}, \mathbf{c})$  with the answer. The example in Table 6 shows that our Info-HCVAE generates diverse and semantically consistent questions given an answer. We provide more qualitative examples in Appendix D.
Latent Space Interpolation To examine if InfoHCVAE learns meaningful latent space of QA pairs, we qualitatively analyze the QA pairs generated by interpolating between two latent codes of it on SQuAD training set. We first encode  $\mathbf{z}_{\mathbf{x}}$  from two QA pairs using posterior networks of  $q_{\phi}(\mathbf{z}_{\mathbf{x}}|\mathbf{x},\mathbf{c})$  and then sample  $\mathbf{z}_{\mathbf{y}}$  from interpolated values of  $\mathbf{z}_{\mathbf{x}}$  using prior networks  $p_{\psi}(\mathbf{z}_{\mathbf{y}}|\mathbf{z}_{\mathbf{x}},\mathbf{c})$  to generate corresponding QA pairs. Table 7 shows that the semantic of the QA pairs generated smoothly transit from one latent to another with high diversity and consistency. We provide more qualitative examples
Table 6: Examples of one-to-many mapping of our Info-HCVAE. The answer is highlighted by pink. GT denotes the ground-truth question. O- denotes questions generated by Info-HCVAE.
Paragraph ... Atop the main building's gold dome is a golden statue of the virgin mary. ... Next to the main building is the basilica of the sacred heart. Immediately behind the basilica is the grotto, ... a marian place of prayer and reflection. ... At the end of the main drive ..., is a simple, modern stone statue of mary.
Oril	Q what is the grotto at notre dame?
A a marian place of prayer and reflection
Gen	Q where is the grotto at?
A a marian place of prayer and reflection
Q what place is behind the basilica of prayer?
A grotto
Q what is next to the main building at notre dame?
A the basilica of the sacred heart
Q what is at the end of the main drive?
A stone statue of mary
Ori2	Q what sits on top of the main building at notre dame?
A a golden statue of the virgin mary
Table 7: QA pairs generated by interpolating between two latent codes encoded by our posterior networks. Ori1 and Ori2 are from training set of SQuAD.
in Appendix D.
4.5 Semi-supervised QA
We now validate our model in a semi-supervised setting, where the model uses both the ground truth labels and the generated labels to solve the QA task, to see whether the generated QA pairs help improve the performance of a QA model in a conventional setting. Since such synthetic datasets consisting of generated QA pairs may inevitably contain some noise (Zhang and Bansal, 2019; Dong et al., 2019; Alberti et al., 2019), we further refine the QA pairs by using the heuristic suggested by Dong et al. (2019), to replace the generated answers whose F1 score to the prediction of the QA model trained on the human annotated data is lower than a set threshold. We select the threshold of 40.0 for the QA pair refinement model via cross-validation on the SQuAD dataset, and used it for the experiments. Please see Appendix C for more details.
SQuAD We first perform semi-supervised QA experiments on SQuAD using the synthetic QA pairs generated by our model. For the contexts, we use both the paragraphs in the original SQuAD (S) dataset, and the new paragraphs in the HarvestingQA dataset (H). Using Info-HCVAE, we generate 10 different QA pairs by sampling from the latent spaces (denoted as  $S \times 10$ ). For the baseline, we use Semantic-QG (Zhang and Bansal, 2019) with the beam search size of 10 to obtain the same number of QA pairs. We also generate new QA pairs
Data	EM	F1
SQuAD	80.25	88.23
Semantic-QG (baseline)
+S×10	81.20 (+0.95)	88.36 (+0.13)
+H×100%	81.03 (+0.78)	88.79 (+0.56)
+S×10 + H×100%	81.44 (+1.19)	88.72 (+0.49)
Info-HCVAE (ours)
+S×10	82.09 (+1.84)	89.11 (+0.88)
+H×10%	81.37 (+1.12)	88.85 (+0.62)
+H×20%	81.68 (+1.43)	89.06 (+0.93)
+H×30%	81.76 (+1.51)	89.12 (+0.89)
+H×50%	82.17 (+1.92)	89.38 (+1.15)
+H×100%	82.37 (+2.12)	89.63 (+1.40)
+S×10 + H×100%	82.19 (+1.94)	89.84 (+1.59)
using different portions of paragraphs provided in HarvestingQA (denoted as  $\mathrm{H} \times 10\% - \mathrm{H} \times 100\%$ ), by sampling one latent variable per context. Table 8 shows that our framework improves the accuracy of the BERT-base model by 2.12 (EM) and 1.59 (F1) points, significantly outperforming Semantic-QG. NQ and TriviaQA Our model is most useful when we do not have any labeled data for a target dataset. To show how well our QAG model performs in such a setting, we train the QA model using only the QA pairs generated by our model trained on SQuAD and test it on the target datasets (NQ and TriviaQA). We generate multiple QA pairs from each context of the target dataset, sampling from the latent space one to ten times (denoted by  $\mathrm{N} \times 1-10$  or  $\mathrm{T} \times 1-10$  in Table 9). Then, we fine-tune the QA model pretrained on the SQuAD dataset with the generated QA pairs from the two datasets. Table 9 shows that as we augment training data with larger number of synthetic QA pairs, the performance of the QA model significantly increases, significantly outperforming the QA model trained on SQuAD only. Yet, models trained with our QAG still largely underperform models trained with human labels, due to the distributional discrepancy between the source and the target dataset.
We proposed a novel probabilistic generative framework for generating diverse and consistent question-answer (QA) pairs from given texts. Specifically, our model learns the joint distribution of question and answer given context with a hierarchically conditional variational autoencoder, while enforcing consistency between generated QA pairs by maximizing their mutual information with a novel In-
Table 8: The results of semi-supervised QA experiments on SQuAD. All the results are the performances on our test set.
Data	EM	F1
Natural Questions
SQuAD	42.77	57.29
+N×1	46.70 (+3.94)	61.08 (+3.79)
+N×2	46.95 (+4.19)	61.34 (+4.05)
+N×3	47.73 (+4.96)	61.98 (+4.69)
+N×5	48.19 (+5.42)	62.21 (+4.92)
+N×10	48.44 (+5.67)	62.69 (+5.40)
NQ	61.65	73.91
TriviaQA
SQuAD	48.96	57.98
+T×1	49.65 (+0.69)	59.13 (+1.21)
+T×2	50.01 (+1.05)	59.08 (+1.10)
+T×3	49.71 (+0.75)	59.49 (+1.51)
+T×5	50.14 (+1.18)	59.21 (+1.23)
+T×10	49.65 (+0.69)	59.20 (+1.22)
Trivia	64.55	70.42
Table 9: The result of semi-supervised QA experiments on Natural Questions and TriviaQA dataset. All results are the performance on our test set.
foMax regularizer. To our knowledge, ours is the first successful probabilistic QAG model. We evaluated the QAG performance of our model by the accuracy of the BERT-base QA model trained using the generated questions on multiple datasets, on which it largely outperformed the state-of-the-art QAG baseline (+6.59-10.69 in EM), even with a smaller number of QA pairs. We further validated our model for semi-supervised QA, where it improved the performance of the BERT-base QA model on the SQuAD by 2.12 in EM, significantly outperforming the state-of-the-art model. As future work, we plan to extend our QAG model to a meta-learning framework, for generalization over diverse datasets.
This work was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-01410, Research Development of Question Generation for Deep Learning based Semantic Search Domain Extension, No.2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion, No.2019-0-00075, and Artificial Intelligence Graduate School Program (KAIST)).
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, R. Devon Hjelm, and Aaron C. Courville. 2018. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016.
Ying-Hong Chan and Yao-Chung Fan. 2019. A recurrent bert-based model for question generation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. 2018. Latent alignment and variational attention. In Advances in Neural Information Processing Systems, NIPS 2018.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019.
Bhuwan Dhingra, Danish Danish, and Dheeraj Rajagopal. 2018. Simple and effective semi-supervised question answering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NACCL-HLT, 2018.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, NeurIPS, 2019.
Jiachen Du, Wenjie Li, Yulan He, Ruifeng Xu, Lidong Bing, and Xuan Wang. 2018. Variational autoregressive decoder for neural response generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. EMNLP 2018.