SlowGuess/ACL_2020_OCR huggingface dataset:
Dong Bok Lee $^{1*}$  Seanie Lee $^{1,3*}$  Woo Tae Jeong $^{3}$  Donghwan Kim $^{3}$  Sung Ju Hwang $^{1,2}$
KAIST<sup>1</sup>, ATRICS<sup>2</sup>, 42Maru Inc.<sup>3</sup>, South Korea
{markhi, lsnfamily02, sjhwang82}@kaist.ac.kr
{wtjeong,scissors}@42maru.com
One of the most crucial challenges in question answering (QA) is the scarcity of labeled data, since it is costly to obtain question-answer (QA) pairs for a target text domain with human annotation. An alternative approach to tackle the problem is to use automatically generated QA pairs from either the problem context or from large amount of unstructured texts (e.g. Wikipedia). In this work, we propose a hierarchical conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts, while maximizing the mutual information between generated QA pairs to ensure their consistency. We validate our Information Maximizing Hierarchical Conditional Variational AutoEncoder (Info-HCVAE) on several benchmark datasets by evaluating the performance of the QA model (BERT-base) using only the generated QA pairs (QA-based evaluation) or by using both the generated and human-labeled pairs (semi-supervised learning) for training, against state-of-the-art baseline models. The results show that our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
Extractive Question Answering (QA) is one of the most fundamental and important tasks for natural language understanding. Thanks to the increased complexity of deep neural networks and use of knowledge transfer from the language models pretrained on large-scale corpora (Peters et al., 2018; Devlin et al., 2019; Dong et al., 2019), the state-of-the-art QA models have achieved human-level performance on several benchmark datasets (Rajpurkar et al., 2016, 2018). However, what is also
Paragraph (Input) Philadelphia has more murals than any other u.s. city, thanks in part to the 1984 creation of the department of recreation's mural arts program, ... The program has funded more than 2,800 murals
Q1 which city has more murals than any other city?
A1 philadelphia
Q2 why philadelphia has more murals?
A2 the 1984 creation of the department of recreation's mural arts program
Q3 when did the department of recreation's mural arts program start?
A3 1984
Q4 how many murals funded the graffiti arts program by the department of recreation?
A4 more than 2,800
Table 1: An example of QA pairs generated with our framework. The paragraph is an extract from Wikipedia provided by Du and Cardie (2018). For more examples, please see Appendix D.
crucial to the success of the recent data-driven models, is the availability of large-scale QA datasets. To deploy the state-of-the-art QA models to real-world applications, we need to construct high-quality datasets with large volumes of QA pairs to train them; however, this will be costly, requiring a massive amount of human efforts and time.
Question generation (QG), or Question-Answer pair generation (QAG), is a popular approach to overcome this data scarcity challenge. Some of the recent works resort to semi-supervised learning, by leveraging large amount of unlabeled text (e.g. Wikipedia) to generate synthetic QA pairs with the help of QG systems (Tang et al., 2017; Yang et al., 2017; Tang et al., 2018; Sachan and Xing, 2018). However, existing QG systems have overlooked an important point that generating QA pairs from a context consisting of unstructured texts, is essentially a one-to-many problem. Sequence-to-sequence models are known to generate generic sequences (Zhao et al., 2017a) without much variety, as they are trained with maximum likelihood estimation. This is highly suboptimal for QAG
<sup>1</sup>The generated QA pairs and the code can be found at https://github.com/seanie12/Info-HCVAE
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 208-224
since the contexts given to the model often contain richer information that could be exploited to generate multiple QA pairs.
To tackle the above issue, we propose a novel probabilistic deep generative model for QA pair generation. Specifically, our model is a hierarchical conditional variational autoencoder (HCVAE) with two separate latent spaces for question and answer conditioned on the context, where the answer latent space is additionally conditioned on the question latent space. During generation, this hierarchical conditional VAE first generates an answer given a context, and then generates a question given both the answer and the context, by sampling from both latent spaces. This probabilistic approach allows the model to generate diverse QA pairs focusing on different parts of a context at each time.
Another crucial challenge of the QG task is to ensure the consistency between a question and its corresponding answer, since they should be semantically dependent on each other such that the question is answerable from the given answer and the context. In this paper, we tackle this consistency issue by maximizing the mutual information (Belghazi et al., 2018; Hjelm et al., 2019; Yeh and Chen, 2019) between the generated QA pairs. We empirically validate that the proposed mutual information maximization significantly improves the QA-pair consistency. Combining both the hierarchical CVAE and the InfoMax regularizer together, we propose a novel probabilistic generative QAG model which we refer to as Information Maximizing Hierarchical Conditional Variational AutoEncoder (Info-HCVAE). Our Info-HCVAE generates diverse and consistent QA pairs even from a very short context (see Table 1).
But how should we quantitatively measure the quality of the generated QA pairs? Popular evaluation metrics (e.g. BLEU (Papineni et al., 2002), ROUGE (Lin and Hovy, 2002), METEOR (Banerjee and Lavie, 2005)) for text generation only tell how similar the generated QA pairs are to GroundTruth (GT) QA pairs, and are not directly correlated with their actual quality (Nema and Khapra, 2018; Zhang and Bansal, 2019). Therefore, we use the QA-based Evaluation (QAE) metric proposed by Zhang and Bansal (2019), which measures how well the generated QA pairs match the distribution of GT QA pairs. Yet, in a semi-supervised learning setting where we already have GT labels, we need novel QA pairs that are different from GT QA pairs
for the additional QA pairs to be truly effective. Thus, we propose a novel metric, Reverse QAE (R-QAE), which is low if the generated QA pairs are novel and diverse.
We experimentally validate our QAG model on SQuAD v1.1 (Rajpurkar et al., 2016), Natural Questions (Kwiatkowski et al., 2019), and TriviaQA (Joshi et al., 2017) datasets, with both QAE and R-QAE using BERT-base (Devlin et al., 2019) as the QA model. Our QAG model obtains high QAE and low R-QAE, largely outperforming state-of-the-art baselines using a significantly smaller number of contexts. Further experimental results for semi-supervised QA on the three datasets using the SQuAD as the labeled dataset show that our model achieves significant improvements over the state-of-the-art baseline (+2.12 on SQuAD, +5.67 on NQ, and +1.18 on Trivia QA in EM).
- We propose a novel hierarchical variational framework for generating diverse QA pairs from a single context, which is, to our knowledge, the first probabilistic generative model for question-answer pair generation (QAG).
- We propose an InfoMax regularizer which effectively enforces the consistency between the generated QA pairs, by maximizing their mutual information. This is a novel approach in resolving consistency between QA pairs for QAG.
- We evaluate our framework on several benchmark datasets by either training a new model entirely using generated QA pairs (QA-based evaluation), or use both ground-truth and generated QA pairs (semi-supervised QA). Our model achieves impressive performances on both tasks, largely outperforming existing QAG baselines.
Question and Question-Answer Pair Generation Early works on Question Generation (QG) mostly resort to rule-based approaches (Heilman and Smith, 2010; Lindberg et al., 2013; Labutov et al., 2015). However, recently, encoder-decoder based neural architectures (Du et al., 2017; Zhou et al., 2017) have gained popularity as they outperform rule-based methods. Some of them use paragraph-level information (Du and Cardie, 2018; Song et al., 2018; Liu et al., 2019; Zhao et al., 2018; Kim et al., 2019; Sun et al., 2018) as additional information. Reinforcement learning is a popular
approach to train the neural QG models, where the reward is defined as the evaluation metrics (Song et al., 2017; Kumar et al., 2018), or the QA accuracy/likelihood (Yuan et al., 2017; Hosking and Riedel, 2019; Zhang and Bansal, 2019). State-of-the-art QG models (Alberti et al., 2019; Dong et al., 2019; Chan and Fan, 2019) use pre-trained language models. Question-Answer Pair Generation (QAG) from contexts, which is our main target, is a relatively less explored topic tackled by only a few recent works (Du and Cardie, 2018; Alberti et al., 2019; Dong et al., 2019). To the best of our knowledge, we are the first to propose a probabilistic generative model for end-to-end QAG; Yao et al. (2018) use VAE for QG, but they do not tackle QAG. Moreover, we effectively resolve the QA-pair consistency issue by maximizing their mutual information with an InfoMax regularizer (Belghazi et al., 2018; Hjelm et al., 2019; Yeh and Chen, 2019), which is another contribution of our work.
Semi-supervised QA with QG With the help of QG models, it is possible to train the QA models in a semi-supervised learning manner to obtain improved performance. Tang et al. (2017) apply dual learning to jointly train QA and QG on unlabeled dataset. Yang et al. (2017) and Tang et al. (2018) train QG and QA in a GAN framework (Goodfellow et al., 2014). Sachan and Xing (2018) propose a curriculum learning to supervise the QG model to gradually generate difficult questions for the QA model. Dhingra et al. (2018) introduce a cloze-style QAG method to pretrain a QA model. Zhang and Bansal (2019) propose to filter out low-quality synthetic questions by the answer likelihood. While we focus on the answerable setting in this paper, few recent works tackle the unanswerable settings. Zhu et al. (2019) use neural networks to edit answerable questions into unanswerable ones, and perform semi-supervised QA. Alberti et al. (2019) and Dong et al. (2019) convert generated questions into unanswerable ones using heuristics, and filter or replace corresponding answers based on EM or F1.
Variational Autoencoders Variational autoencoders (VAEs) (Kingma and Welling, 2014) are probabilistic generative models used in a variety of natural language understanding tasks, including language modeling (Bowman et al., 2016), dialogue generation (Serban et al., 2017; Zhao et al., 2017b; Park et al., 2018; Du et al., 2018; Qiu et al., 2019), and machine translation (Zhang et al., 2016;
Su et al., 2018; Deng et al., 2018). In this work, we propose a novel hierarchical conditional VAE framework with an InfoMax regularization for generating a pair of samples with high consistency.
Our goal is to generate diverse and consistent QA pairs to tackle the data scarcity challenge in the extractive QA task. Formally, given a context  $\mathbf{c}$  which contains  $M$  tokens,  $\mathbf{c} = (c_{1},\dots ,c_{M})$ , we want to generate QA pairs  $(\mathbf{x},\mathbf{y})$  where  $\mathbf{x} = (x_{1},\dots ,x_{N})$  is the question containing  $N$  tokens and  $\mathbf{y} = (y_{1},\dots ,y_{L})$  is its corresponding answer containing  $L$  tokens. We aim to tackle the QAG task by learning the conditional joint distribution of the question and answer given the context,  $p(\mathbf{x},\mathbf{y}|\mathbf{c})$ , from which we can sample the QA pairs:
$$
(\mathbf {x}, \mathbf {y}) \sim p (\mathbf {x}, \mathbf {y} | \mathbf {c})
$$
We estimate  $p(\mathbf{x},\mathbf{y}|\mathbf{c})$  with a probabilistic deep generative model, which we describe next.
3.1 Hierarchical Conditional VAE
We propose to approximate the unknown conditional joint distribution  $p(\mathbf{x},\mathbf{y}|\mathbf{c})$ , with a variational autoencoder (VAE) framework (Kingma and Welling, 2014). However, instead of directly learning a common latent space for both question and answer, we model  $p(\mathbf{x},\mathbf{y}|\mathbf{c})$  in a hierarchical conditional VAE framework with a separate latent space for question and answer as follows:
$$
\begin{array}{l} p _ {\theta} (\mathbf {x}, \mathbf {y} | \mathbf {c}) \\ = \int_ {\mathbf {z} _ {\mathbf {x}}} \sum_ {\mathbf {z} _ {\mathbf {y}}} p _ {\theta} (\mathbf {x} | \mathbf {z} _ {\mathbf {x}}, \mathbf {y}, \mathbf {c}) p _ {\theta} (\mathbf {y} | \mathbf {z} _ {\mathbf {x}}, \mathbf {z} _ {\mathbf {y}}, \mathbf {c}) \cdot \\ p _ {\psi} (\mathbf {z} _ {\mathbf {y}} | \mathbf {z} _ {\mathbf {x}}, \mathbf {c}) p _ {\psi} (\mathbf {z} _ {\mathbf {x}} | \mathbf {c}) d \mathbf {z} _ {\mathbf {x}} \\ \end{array}
$$
where  $\mathbf{z}_{\mathbf{x}}$  and  $\mathbf{z}_{\mathbf{y}}$  are latent variables for question and answer respectively, and the  $p_{\psi}(\mathbf{z}_{\mathbf{x}}|\mathbf{c})$  and  $p_{\psi}(\mathbf{z}_{\mathbf{y}}|\mathbf{z}_{\mathbf{x}},\mathbf{c})$  are their conditional priors following an isotropic Gaussian distribution and a categorical distribution (Figure 1-(a)). We decompose the latent space of question and answer, since the answer is always a finite span of context  $\mathbf{c}$ , which can be modeled well by a categorical distribution, while a continuous latent space is a more appropriate choice for question since there could be unlimited valid questions from a single context. Moreover, we design the bi-directional dependency flow of joint distribution for QA. By leveraging hierarchical structure, we enforce the answer latent variables
Figure 1: The conceptual illustration of the proposed HCVAE model encoding and decoding question and its corresponding answer jointly. The dashed line refers to the generative process of HCVAE.
Figure 2: The directed graphical model for HCVAE. The gray and white nodes denote observed and latent variables.
to be dependent on the question latent variables in  $p_{\psi}(\mathbf{z}_{\mathbf{y}}|\mathbf{z}_{\mathbf{x}},\mathbf{c})$  and achieve the reverse dependency by sampling question  $\mathbf{x}\sim p_{\theta}(\mathbf{x}|\mathbf{z}_{\mathbf{x}},\mathbf{y},\mathbf{c})$ . We then use a variational posterior  $q_{\phi}(\cdot)$  to maximize the Evidence Lower Bound (ELBO) as follows (The complete derivation is provided in Appendix A):
$$
\begin{array}{l} \log p _ {\theta} (\mathbf {x}, \mathbf {y} | \mathbf {c}) \geq \mathbb {E} _ {\mathbf {z} _ {\mathbf {x}} \sim q _ {\phi} (\mathbf {z} _ {\mathbf {x}} | \mathbf {x}, \mathbf {c})} [ \log p _ {\theta} (\mathbf {x} | \mathbf {z} _ {\mathbf {x}}, \mathbf {y}, \mathbf {c}) ] \\ + \mathbb {E} _ {\mathbf {z} _ {\mathbf {y}} \sim q _ {\phi} (\mathbf {z} _ {\mathbf {y}} | \mathbf {z} _ {\mathbf {x}}, \mathbf {y}, \mathbf {c})} [ \log p _ {\theta} (\mathbf {y} | \mathbf {z} _ {\mathbf {y}}, \mathbf {c}) ] \\ - \mathrm {D} _ {\mathrm {K L}} \left[ q _ {\phi} \left(\mathbf {z} _ {\mathbf {y}} \mid \mathbf {z} _ {\mathbf {x}}, \mathbf {y}, \mathbf {c}\right) \right\lvert   \left| p _ {\psi} \left(\mathbf {z} _ {\mathbf {y}} \mid \mathbf {z} _ {\mathbf {x}}, \mathbf {c}\right) \right] \\ - \mathrm {D} _ {\mathrm {K L}} \left[ q _ {\phi} \left(\mathbf {z} _ {\mathbf {x}} | \mathbf {x}, \mathbf {c}\right) | | p _ {\psi} \left(\mathbf {z} _ {\mathbf {x}} | \mathbf {c}\right) \right] \\ =: \mathcal {L} _ {\mathrm {H C V A E}} \\ \end{array}
$$
where  $\theta, \phi,$  and  $\psi$  are the parameters of the generation, posterior, and prior network, respectively. We refer to this model as a Hierarchical Conditional Variational Autoencoder (HCVAE) framework.
Figure 2 shows the directed graphical model of our HCVAE. The generative process is as follows:
1. Sample question L.V.:  $\mathbf{z}_{\mathbf{x}} \sim p_{\psi}(\mathbf{z}_{\mathbf{x}} \mid \mathbf{c})$
2. Sample answer L.V.:  $\mathbf{z_y}\sim p_\psi (\mathbf{z_y}|\mathbf{z_x},\mathbf{c})$
3. Generate an answer:  $\mathbf{y} \sim p_{\theta}(\mathbf{y}|\mathbf{z}_{\mathbf{y}},\mathbf{c})$
4. Generate a question:  $\mathbf{x} \sim p_{\theta}(\mathbf{x} \mid \mathbf{z}_{\mathbf{x}}, \mathbf{y}, \mathbf{c})$
Embedding We use the pre-trained word embedding network from BERT (Devlin et al., 2019) for posterior and prior networks, whereas the whole BERT is used as a contextualized word embedding model for the generative networks. For the answer
encoding, we use a binary token type id of BERT. Specifically, we encode all context tokens as 0s, except for the tokens which are part of answer span (highlighted words of context in Figure 1-(a) or -(c)), which we encode as 1s. We then feed the sequence of the word token ids, token type ids, and position ids into the embedding layer to encode the answer-aware context. We fix all the embedding layers in HCVAE during training.
Prior Networks We use two different conditional prior networks  $p_{\psi}(\mathbf{z}_{\mathbf{x}}|\mathbf{c}), p_{\psi}(\mathbf{z}_{\mathbf{y}}|\mathbf{z}_{\mathbf{x}}, \mathbf{c})$  to model context-dependent priors (the dashed lines in Figure 1-(a)). To obtain the parameters of isotropic Gaussian  $\mathcal{N}(\boldsymbol{\mu}, \sigma^2 \mathbf{I})$  for  $p_{\psi}(\mathbf{z}_{\mathbf{x}}|\mathbf{c})$ , we use a bidirectional LSTM (Bi-LSTM) to encode the word embeddings of the context into the hidden representations, and then feed them into a Multi-Layer Perceptron (MLP). We model  $p_{\psi}(\mathbf{z}_{\mathbf{y}}|\mathbf{z}_{\mathbf{x}}, \mathbf{c})$  following a categorical distribution  $\operatorname{Cat}(\pi)$ , by computing the parameter  $\pi$  from  $\mathbf{z}_{\mathbf{x}}$  and the hidden representation of the context using another MLP.
Posterior Networks We use two conditional posterior networks  $q_{\phi}(\mathbf{z}_{\mathbf{x}}|\mathbf{x},\mathbf{c}),q_{\phi}(\mathbf{z}_{\mathbf{y}}|\mathbf{z}_{\mathbf{x}},\mathbf{y},\mathbf{c})$  to approximate true posterior distributions of latent variables for both question  $\mathbf{x}$  and answer  $\mathbf{y}$ . We use two Bi-LSTM encoders to output the hidden representations of question and context given their word embeddings. Then, we feed the two hidden representations into MLP to obtain the parameters of Gaussian distribution,  $\mu^{\prime}$  and  $\sigma^{\prime}$  (upper right corner in Figure 1-(a)). We use the reparameterization trick (Kingma and Welling, 2014) to train the model with backpropagation since the stochastic sampling process  $\mathbf{z}_{\mathbf{x}}\sim q_{\phi}(\mathbf{z}_{\mathbf{x}}|\mathbf{x},\mathbf{c})$  is nondifferentiable. We use another Bi-LSTM to encode the word embedding of answer-aware context into the hidden representation. Then, we feed the hidden representation and  $\mathbf{z}_{\mathbf{x}}$  into MLP to compute the parameters  $\pi^{\prime}$  of categorical distribution (lower right corner in Figure 1-(a)). We use the categorical reparameterization trick with gumbel-softmax
(Maddison et al., 2017; Jang et al., 2017) to enable backpropagation through sampled discrete latent variables.
Answer Generation Networks Since we consider extractive QA, we can factorize  $p_{\theta}(\mathbf{y}|\mathbf{z}_{\mathbf{y}},\mathbf{c})$  into  $p_{\theta}(y_s|\mathbf{z}_{\mathbf{y}},\mathbf{c})$  and  $p_{\theta}(y_e|\mathbf{z}_{\mathbf{y}},\mathbf{c})$ , where  $y_{s}$  and  $y_{e}$  are the start and the end position of an answer span (highlighted words in Figure 1-(b)), respectively. To obtain MLE estimators for both, we first encode the context  $\mathbf{c}$  into the contextualized word embedding of  $\mathbf{E}^{\mathrm{c}} = \{\mathbf{e}_1^{\mathrm{c}},\dots ,\mathbf{e}_M^{\mathrm{c}}\}$  with the pre-trained BERT. We compute the final hidden representation of context and the latent variable  $\mathbf{z}_{\mathbf{y}}$  with a heuristic matching layer (Mou et al., 2016) and a Bi-LSTM:
$$
\begin{array}{l} \mathbf {f} _ {i} = \left[ \mathbf {e} _ {i} ^ {\mathbf {c}}; \mathbf {z} _ {\mathbf {y}}; \left| \mathbf {e} _ {i} ^ {\mathbf {c}} - \mathbf {z} _ {\mathbf {y}} \right|; \mathbf {e} _ {i} ^ {\mathbf {c}} \odot \mathbf {z} _ {\mathbf {y}} \right] \\ \overrightarrow {\mathbf {h}} _ {i} = \overrightarrow {\operatorname {L S T M}} ([ \mathbf {f} _ {i}, \overrightarrow {\mathbf {h}} _ {i - 1} ]) \\ \overleftarrow {\mathbf {h}} _ {i} = \overleftarrow {\mathrm {L S T M}} ([ \mathbf {f} _ {i}, \overleftarrow {\mathbf {h}} _ {i + 1} ]) \\ \mathbf {H} = \left[ \begin{array}{c c} \overrightarrow {\mathbf {h}} _ {i}; & \overleftarrow {\mathbf {h}} _ {i} \end{array} \right] _ {i = 1} ^ {M} \\ \end{array}
$$
where  $\mathbf{z}_{\mathbf{y}}$  is linearly transformed, and  $\mathbf{H} \in \mathbb{R}^{d_{\mathbf{y}} \times M}$  is the final hidden representation. Then, we feed  $\mathbf{H}$  into two separate linear layers to predict  $y_{s}$  and  $y_{e}$ . Question Generation Networks We design the encoder-decoder architecture for our QG network by mainly adopting from our baselines (Zhao et al., 2018; Zhang and Bansal, 2019). For encoding, we use pre-trained BERT to encode the answer-specific context into the contextualized word embedding, and then use a two-layer Bi-LSTM to encode it into the hidden representation (in Figure 1-(c)). We apply a gated self-attention mechanism (Wang et al., 2017) to the hidden representation to better capture long-term dependencies within the context, to obtain a new hidden representation  $\hat{\mathbf{H}} \in \mathbb{R}^{d_{\mathbf{x}} \times M}$ .
The decoder is a two-layered LSTM which receives the latent variable  $\mathbf{z}_{\mathbf{x}}$  as an initial state. It uses an attention mechanism (Luong et al., 2015) to dynamically aggregate  $\hat{\mathbf{H}}$  at each decoding step into a context vector of  $\mathbf{s}_j$ , using the  $j$ -th decoder hidden representation  $\mathbf{d}_j \in \mathbb{R}^{d_{\mathbf{x}}}$  (in Figure 1-(c)). Then, we feed  $\mathbf{d}_j$  and  $\mathbf{s}_j$  into MLP with maxout activation (Goodfellow et al., 2013) to compute the final hidden representation  $\hat{\mathbf{d}}_j$  as follows:
$$
\begin{array}{l} \mathbf {d} _ {0} = \mathbf {z} _ {\mathbf {x}}, \mathbf {d} _ {j} = \mathrm {L S T M} ([ \mathbf {e} _ {j - 1} ^ {\mathbf {x}}, \mathbf {d} _ {j - 1} ]) \\ \mathbf {r} _ {j} = \hat {\mathbf {H}} ^ {T} \mathbf {W} ^ {\mathbf {a}} \mathbf {d} _ {j}, \mathbf {a} _ {j} = \operatorname {s o f t m a x} (\mathbf {r} _ {j}), \mathbf {s} _ {j} = \hat {\mathbf {H}} \mathbf {a} _ {j} \\ \hat {\mathbf {d}} _ {j} = \operatorname {M L P} \left(\left[ \begin{array}{l} \mathbf {d} _ {j}; \mathbf {s} _ {j} \end{array} \right]\right) \\ \end{array}
$$
where  $\mathbf{z}_{\mathbf{x}}$  is linearly transformed, and  $\mathbf{e}_j^{\mathbf{x}}$  is the  $j$ -th question word embedding. The probability vector over the vocabulary is computed as
$p(\mathbf{x}_j|\mathbf{x}_{< j},\mathbf{z}_{\mathbf{x}},\mathbf{y},\mathbf{c}) = \mathrm{softmax}(\mathbf{W}^{\mathrm{e}}\hat{\mathbf{d}}_j)$ . We initialize the weight matrix  $\mathbf{W}^{\mathrm{e}}$  as the pretrained word embedding matrix and fix it during training. Further, we use the copy mechanism (Zhao et al., 2018), so that the model can directly copy tokens from the context. We also greedily decode questions to ensure that all stochasticity comes from the sampling of the latent variables.
3.2 Consistent QA Pair Generation with Mutual Information Maximization