SlowGuess/ACL_2020_OCR huggingface dataset:
Daya Guo $^{1*}$ , Duyu Tang $^{2}$ , Nan Duan $^{2}$ , Jian Yin $^{1}$ , Daxin Jiang $^{3}$  and Ming Zhou $^{2}$
<sup>1</sup> The School of Data and Computer Science, Sun Yat-sen University.  
<sup>2</sup> Microsoft Research Asia, Beijing, China  
<sup>3</sup> Microsoft Search Technology Center Asia, Beijing, China  
{guody5@mail2, issjyin@mail}.sysu.edu.cn  
{dutang, NANDuan, djiang, mingzhou}@microsoft.com
Generating inferential texts about an event in different perspectives requires reasoning over different contexts that the event occurs. Existing works usually ignore the context that is not explicitly provided, resulting in a context-independent semantic representation that struggles to support the generation. To address this, we propose an approach that automatically finds evidence for an event from a large text corpus, and leverages the evidence to guide the generation of inferential texts. Our approach works in an encoder-decoder manner and is equipped with a Vector Quantised-Variational Autoencoder, where the encoder outputs representations from a distribution over discrete variables. Such discrete representations enable automatically selecting relevant evidence, which not only facilitates evidence-aware generation, but also provides a natural way to uncover rationales behind the generation. Our approach provides state-of-the-art performance on both Event2Mind and ATOMIC datasets. More importantly, we find that with discrete representations, our model selectively uses evidence to generate different inferential texts.
Inferential text generation aims to understand daily-life events and generate texts about their underlying causes, effects, and mental states of event participants, which is crucial for automated commonsense reasoning. Taking Figure 1 as an example, given an event "PersonX reads PersonY's diary", the cause of the participant "PersonX" is to "obtain Person Y's secrets" and the mental state of "PersonX" is "guilty". Standard approaches for inferential text generation (Rashkin et al., 2018; Sap et al., 2019; Bosselut et al., 2019; Du et al., 2019) typically only
Figure 1: An examples of inferential text generation on mental states of event participants. We show two kinds of reasonable inferences for the event under different background knowledge that is absent in the dataset.
take the event as the input, while ignoring the background knowledge that provides crucial evidence to generate reasonable inferences. For example, if the background knowledge of this example is "PersonY invites PersonX to read his diary", the outputs should be different.
In this paper, we present an evidence-aware generative model, which first retrieves relevant evidence from a large text corpus and then leverages retrieved evidence to guide the generation of inferential texts. Our model is built upon Transformer-based (Vaswani et al., 2017) encoder-decoder architecture, and is equipped with Vector Quantised-Variational Autoencoder to map an event to a discrete latent representation (van den Oord et al., 2017). These discrete representations embody the latent semantic distribution of inferences given the event, thus supporting selection of relevant evidence as background knowledge to guide the generation in different perspectives. Furthermore, our model has two attractive properties: (1) it avoids the problem of posterior collapse, caused by latent variables being ignored, in traditional variational autoencoder with continuous latent variables (van den Oord et al., 2017), and more importantly (2) it uncovers the rationale of a generation to some extent through tracing back the evidence that guides the generation and the selected discrete representation of the event.
* Work done while this author was an intern at Microsoft Research.
6118
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6118-6129 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We evaluate our approach on Event2Mind (Rashkin et al., 2018) and ATOMIC (Sap et al., 2019) datasets, both of which focus on reasoning about causes and effects of events and mental states of event participants. Experimental results show that our approach achieves state-of-the-art performances on both datasets. Further analysis shows that our approach can equip the generation with an explicit control over the semantics of latent variables and selected evidence to generate inferential texts in different perspective. The source codes are available at https://github.com/microsoft/EA-VQ-VAE.
2 Task Definition and Datasets
Figure 1 shows an example of the task, which aims to generate inferential texts about causes and effects of daily-life events and mental states of the events participants. Formally, given an event  $x = \{x_{1}, x_{2}, \ldots, x_{n}\}$  and an inference dimension  $r$  such as causes of the event, the goal is to generate multiple inferential texts  $Y = \{y^{(1)}, y^{(2)}, \ldots, y^{(m)}\}^{1}$ , where the background knowledge of the event is absent in the dataset.
We conduct experiments on Event2Mind $^{2}$  (Rashkin et al., 2018) and ATOMIC $^{3}$  (Sap et al., 2019) datasets. Both datasets contain about 25,000 unique events extracted from multiple data sources and provide multiple inferences under different inference dimensions by crowd-sourcing on Amazon Mechanical Turk. Event2Mind and ATOMIC contain 2.6 and 3.6 inferences on average per example, respectively. Event2Mind focuses on three inference dimensions related to mental states of participants (i.e. intents and reactions of the events participants), while ATOMIC has broader inference dimensions including mental states, probable pre- and post conditions of the event, and persona status. More details about the two datasets are provided in the Appendix A.
3 Overview of the Approach
We present our approach in this section, which first retrieves relevant evidence from a large text corpus, and then utilizes retrieved evidence as background knowledge to generate inferences.
Figure 2 gives an overview of our approach.
Figure 2: An overview of our approach.
First, our encoder takes an event as the input and outputs a semantic representation  $z$  from a distribution over discrete latent variables, which is based on Vector Quantised-Variational Autoencoder (VQVAE) (van den Oord et al., 2017). We then use the event as a query to retrieve top  $K$  evidence from a large text corpus as background knowledge. Lastly, the evidence-aware decoder takes the semantic representation and evidence as the input and generates the inference  $y$ , where the semantic representation selectively uses relevant evidence as background knowledge to guide the generation of inferences.
3.1 Vector Quantised-Variational Autoencoder
Figure 3 illustrates the model architecture of our approach. The model is based on encoder-decoder framework equipped with Vector Quantised-Variational Autoencoder (VQ-VAE) (van den Oord et al., 2017), where the VQ-VAE is learned to model the latent semantic distribution within inferences given an event. Latent variables  $z$  from the VQ-VAE will be used to calculate the relevant of retrieved evidence in the semantic space to guide the generation.
Compared with continuous VAEs, VQ-VAE does not suffer from "posterior collapse" issues that latent variables are often ignored with a powerful decoder (van den Oord et al., 2017). VQ-VAE mainly consists of three parts: a codebook for modeling the latent semantic distribution within inferences over discrete latent variables, a recognition network for modeling a posterior distribution  $q_{\phi}(z|x,y)$ , and a prior network for inferring a prior distribution  $p_{\theta}(z|x)$ .
<sup>1</sup>We use inference and inferential text interchangably
$^{2}$ https://uwnlp.github.io/Event2Mind/
<sup>3</sup>https://homes.cs.washington.edu/
~msap/ATOMIC/
6119
Figure 3: The model architecture of our approach.
Codebook A codebook aims to model the latent semantic discrete distribution within inferences, which is composed of  $k$  discrete latent variables (i.e.  $k$ -way categorical). We define the codebook as an embedding table  $T \in R^{k \times d}$ , where  $d$  is the dimension of latent variables. The semantic latent variable  $z$  is indexed from the posterior distribution  $q_{\phi}(z|x,y)$  in the training phase and the prior distribution  $p_{\theta}(z|x)$  in the inference phase over the codebook, respectively.
Posterior Distribution We follow van den Oord et al. (2017) to model a discrete posterior distribution  $q_{\phi}(z|x,y)$  over the codebook. First, we use Transformer (Vaswani et al., 2017) with two layers as our encoder, where the input sequence is the concatenation of an event  $x$  and its inference  $y$ . In order to obtain the representation of an example  $(x,y)$ , we add a special token in the last of the input sequence and take the hidden state  $h_{(x,y)}$  of the special token as the representation of the example. The posterior categorical probability distribution  $q_{\phi}(z|x,y)$  is defined as one-hot as follows.
$$
q _ {\phi} \left(z _ {k} \mid x, y\right) = \left\{ \begin{array}{l} 1 \text {i f} k = \underset {j} {\arg \min } \left\| h _ {(x, y)} - z _ {j} \right\| _ {2} \\ 0 \text {o t h e r w i s e} \end{array} \right. \tag {1}
$$
As we can see, the hidden state  $h_{(x,y)}$  of the example is mapped onto the nearest element  $z'$  of the codebook under the posterior distribution  $q_{\phi}(z|x,y)$ .
$$
z ^ {\prime} = z _ {k} \quad \text {w h e r e} k = \underset {j} {\arg \min } \left\| h _ {(x, y)} - z _ {j} \right\| _ {2} \tag {2}
$$
Prior Distribution In the inference phase, only the event  $x$  is given, which requires a prior distribution estimator to infer the prior distribution  $p_{\theta}(z|x)$ . Since the prior distribution is crucial for the inference phase, we use a powerful pre-trained
language model such as RoBERTa (Liu et al., 2019) to encode the event into a hidden state  $h$ . Since the prior distribution is categorical, we then use a  $k$ -way classifier following a softmax function to infer the prior distribution, where  $W_{k} \in R^{d \times k}$  is the model parameters.
$$
p _ {\theta} (z | x) = \operatorname {s o f t m a x} \left(h W _ {k}\right) \tag {3}
$$
The training detail of the VQ-VAE will be introduced in the Section 3.4.
3.2 Evidence Retrieval
In this section, we describe how to retrieve event-related evidence as background knowledge. Given an event, we expect that retrieved evidence can contain the event and provide its context as a clue to guide the generation.
To retrieve event-related evidence, we use the event as a query to search evidence from a large text corpus. Specifically, we first remove stop words in the given event and then concatenate the words as a query to search evidence from the corpus by Elastic Search engine<sup>4</sup>. The engine ranks the matching scores between the query and all sentences using BM25 and select top  $K$  sentences as evidence  $C = \{c_1, c_2, \dots, c_K\}$ . To provide detailed context about the event, we build our corpus upon BooksCorpus (Zhu et al., 2015) that consists of 11,038 story books, since stories usually give a detailed account of an event such as causes and effects of the event.
3.3 Evidence-Aware Decoder
In this section, we propose an evidence-aware decoder, which consists of two components, evidence selection and a generator, respectively. Evidence selection aims to calculate a context distribution
<sup>4</sup>https://www/elastic.co/
6120
$p_{s}(c|z)$  given a latent variable  $z$  to model the relevance of retrieved evidence, while the generator  $p_{m}(y|x,c)$  takes an event  $x$  and evidence  $c$  as the input to generate the inferential text  $y$ .
3.3.1 Evidence Selection
The relevance of retrieved evidence is different depending on the semantics of inference, which requires a context distribution to model the relevance. For examples, given an event "PersonX reads PersonY's diary" and its inference "PersonX feels guilty", the relevance of the evidence "PersonX stole PersonY's diary" should be higher than that of the evidence "PersonY invites PersonX to read his diary". However, inferences are unseen in the inference phase, thus we cannot use inferences to model the context distribution. Instead, we utilize semantic latent variables from the VQ-VAE that models the latent semantic distribution of inferences given an event to calculate the relevance of retrieved evidence.
Evidence selection aims to calculate a context distribution  $p_{s}(c|z)$  over retrieved evidence given a semantic latent variable  $z$  to model the relevance of retrieved evidence. Considering that term-based retrieval (i.e. BM25) may fail to retrieve relevant evidences and all retrieved evidence cannot support the generation, we add an empty evidence  $c_{\phi}$  into the set  $C$  of retrieved evidence as the placeholder. We first use Transformer with two layers to encode retrieved evidence into context vectors  $H_{C} = \{h_{c_{1}}, h_{c_{2}},.., h_{c_{K}}, h_{c_{\phi}}\}$  in the semantic space. Then, the context distribution  $p_{s}(c|z)$  over retrieved evidence given the semantic latent variable  $z$  is calculated as one-hot as follows.
$$
p _ {s} \left(c _ {k} | z\right) = \left\{ \begin{array}{l l} 1 & i f k = \underset {j} {\arg \min } \left| \left| h _ {c _ {j}} - z \right| \right| _ {2} \\ 0 & o t h e r w i s e \end{array} \right. \tag {4}
$$
As we can see, the latent variable  $z$  is mapped onto the nearest element  $c_{z}$  of the retrieved evidence under the context distribution  $p_{s}(c|z)$ .
$$
c _ {z} = c _ {k} \quad \text {w h e r e} k = \underset {j} {\arg \min } | | h _ {c _ {j}} - z | | _ {2} \tag {5}
$$
Another "soft" distribution such as using an attention mechanism to calculate the relevance of retrieved evidence can also model the context distribution, but we choose the one-hot distribution as our context distribution since it maps the latent variable  $z$  onto the nearest element of the retrieved evidence, the property of which can help effectively learn the model (described in the Section 3.4).
3.3.2 Generator
Recently, Transformer-based (Vaswani et al., 2017) language models like GPT-2 (Radford et al., 2019) have achieved strong performance in text generation, which is pre-trained from a large-scale text corpus and then fine-tuned on downstream tasks. In this work, we use the GPT-2  $p_m(y|x,c)$  as the backbone of our generator and further take retrieved evidence into account.
A general approach to utilize evidence to guide the generation is to calculate the context vector  $h_c = \sum_{i=1}^{K+1} p_s(c_i | z) h_{c_i}$  as the input of GPT-2 according to the relevance  $p_s(c | z)$  of retrieved evidence. However, this approach changes the architecture of GPT-2, invalidating the original weights of pre-trained GPT-2. Instead, we sample an evidence  $c$  from the context distribution  $p_s(c | z)$  and then concatenate the event and the selected evidence as the input.
To make the paper self-contained, we briefly describe the GPT-2, which takes an evidence and an event as the input and generates the inference  $y = \{y_{1},y_{2},\dots,y_{n}\}$ . This model applies N transformer layers over the input tokens to produce an output distribution over target tokens:
$$
h ^ {0} = [ c; x; y _ {<   t} ] W _ {e} + W _ {p}
$$
$$
h ^ {l} = \text {t r a n s f o r m e r} _ {l - 1} \left(h ^ {l - 1}\right) \tag {6}
$$
$$
p (y _ {t}) = s o f t m a x (h _ {l a s t} ^ {N - 1} W _ {e} ^ {T})
$$
where  $W_{e}$  is the token embedding matrix,  $W_{p}$  is the position embedding matrix, and  $h_{last}^{N - 1}$  is the hidden state of the last token on the top layer. Each transformer layer transformer  $l - 1$  contains an architecturally identical transformer block that applies a masked multi-headed self-attention operation followed by a feed forward layer over the input  $h^{l - 1}$  in the  $l$ -th layer.
$$
\hat {g} ^ {l} = M u l t i A t t n \left(h ^ {l - 1}\right)
$$
$$
g ^ {l} = L N \left(\hat {g} ^ {l} + h ^ {l - 1}\right) \tag {7}
$$
$$
\hat {h} ^ {l} = F F N (g ^ {l})
$$
$$
h ^ {l} = L N \left(\hat {h} ^ {l} + g ^ {l}\right)
$$
where MultiAttn is a masked multi-headed self-attention mechanism, which is similar to Vaswani et al. (2017),  $FFN$  is a two layers feed forward network, and  $LN$  represents a layer normalization operation (Ba et al., 2016).
6121
Our entire approach corresponds to the following generative process. Given an event  $x$ , we first sample a latent variable  $z$  from the VQ-VAE  $p_{\theta}(z|x)$ . We then select relevant evidence  $c$  according to the semantics of the latent variable from the context distribution  $p_s(c|z)$ . Finally, the generator  $p_m(y|x,c)$  takes the event  $x$  and the selected evidence  $c$  as the input and generate the inference  $y$ . Therefore, the probability distribution  $p(y|x)$  over inferences  $y$  given the event  $x$  is formulated as follow.
$$
p (y | x) = \sum_ {z \in T} \sum_ {c \in C} p _ {m} (y | x, c) p _ {s} (c | z) p _ {\theta} (z | x) \tag {8}
$$
A straightforward method for learning our model might be maximizing the marginal likelihood by joint learning, but it is computationally intractable. Instead, we first learn the VQ-VAE with the prior distribution  $p_{\theta}(z|x)$  in isolation, which can enable the codebook to capture the latent semantics within inferences. Then, we train the evidence-aware decoder under the posterior distribution  $q_{\phi}(z|x,y)$ .
Training VQ-VAE To enable the codebook to capture the latent semantics within inferences, we train the VQ-VAE by reconstructing the inferential text  $y$  using the latent variable  $z$ . We use the pre-trained language model GPT-2 (Radford et al., 2019) as our decoder to generate the inference  $p(y|x,z)$ , where the input is the sum of token embedding, position embedding and the latent variable  $z$ . To make reconstruction better conditioned on the latent variable, we replace each query in the multi-head self-attention mechanism with the sum of the latent variable and the query, as well for keys, values and hidden states on the top layer. We follow van den Oord et al. (2017) to learn the VQ-VAE by minimizing the loss function.
$$
\begin{array}{l} l o s s _ {r e c} = - l o g p (y | x, h _ {(x, y)} + s g [ z - h _ {(x, y)} ]) + \\ \left. \left| \left| s g \left[ h _ {(x, y)} \right] - z \right| \right| _ {2} ^ {2} + \beta \right| \left| \left| h _ {(x, y)} - s g [ z ] \right| \right| _ {2} ^ {2} \tag {9} \\ \end{array}
$$
where  $sg$  stands for the stop gradient operator that has zero partial derivatives during differentiation, and  $\beta$  is a hyperparameter which controls the speed to change the latent variable. We set the  $\beta$  as 0.25 in all experiments. The decoder optimizes the first loss term (reconstruction) only, the encoder optimizes the first and the last loss terms, and the codebook are updated by the middle loss term.
We obtain the posterior distribution  $q_{\phi}(z|x,y)$  after optimizing the encoder and the codebook. Af
terward, we learn the prior distribution estimator to infer the prior distribution  $p_{\theta}(z|x)$ . Since the posterior distribution is categorical, we can calculate approximate prior distributions as follow in the training dataset  $D$ , where  $N_{(x)}$  is the number of examples that includes the event  $x$ .
$$
p (z | x) = \sum_ {(x, y _ {i}) \in D} \frac {q _ {\phi} (z | x , y _ {i})}{N _ {(x)}} \tag {10}
$$
Therefore, we can fit the prior distributions by minimizing the KL divergence.
$$
l o s s _ {p r i o r} = K L (p (z | x) | | p _ {\theta} (z | x)) \tag {11}
$$
Training Evidence-Aware Decoder After training VQ-VAE, we jointly learn the context distribution  $p_{s}(c|z)$  and the generator  $p_{m}(y|x,c)$  by maximizing the following marginal likelihood under the posterior distribution  $q_{\phi}(z|x,y)$ .
$$
\operatorname {l o g p} (y | x) = E _ {z \sim q _ {\phi}} \left[ \sum_ {c \in C} \operatorname {l o g p} _ {m} (y | x, c) p _ {s} (c | z) \right] \tag {12}
$$
According to the Equation 2, the example  $(x,y)$  is mapped onto the nearest element  $z^{\prime}$  of the codebook under the posterior distribution  $q_{\phi}(z|x,y)$ . Meanwhile, according to the Equation 5, the latent variable  $z^{\prime}$  is mapped onto the nearest element  $c_{z^{\prime}}$  of retrieved evidence. Therefore, the objective in Equation 12 can be simplified as follow.
$$
\operatorname {l o g p} (y | x) = \operatorname {l o g p} _ {m} (y | x, c _ {z ^ {\prime}}) + \operatorname {l o g p} _ {s} \left(c _ {z ^ {\prime}} \mid z ^ {\prime}\right) \tag {13}
$$
Since the ground truth evidence for the example is unobserved, we cannot directly train the model by maximizing the marginal likelihood. To remedy this problem, we use reinforcement learning algorithm to optimize the objective.
$$
\begin{array}{l} R = \delta (p _ {m} (y | x, c _ {z ^ {\prime}}) - p _ {m} (y | x, c _ {r})) \\ \operatorname {l o g p} (y | x) = \operatorname {l o g p} _ {m} (y | x, c _ {z ^ {\prime}}) + R \operatorname {l o g p} _ {s} \left(c _ {z ^ {\prime}} \mid z ^ {\prime}\right) \tag {14} \\ \end{array}
$$
where  $R$  is the reward designed to guide the model training,  $\delta(x)$  is 1 if  $x$  is larger than 0 otherwise  $-1$ , and  $c_{r}$  is a randomly selected evidence where  $c_{r} \neq c_{z'}$ . The idea of designing the reward is that correct evidence should increase the probability of the gold inference compared with other evidence. Note that there is no real gradient defined for  $p_{s}(c|z)$ , instead, we approximate the gradient similar to the straight-through estimator (Bengio et al., 2013).
$$
\log p (y | x) = \log p _ {m} \left(y | x, c _ {z ^ {\prime}}\right) - R \| h _ {c _ {z ^ {\prime}}} - z ^ {\prime} \| _ {2} ^ {2} \tag {15}
$$
Methods	xIntent	xNeed	xAttr	xEffect	xReact	xWant	oEffect	oReact	oWant	Overall
Single Task
S2S	8.17	12.35	2.96	5.26	3.43	13.44	6.42	4.09	7.08	7.02
VRNMT	9.52	13.35	4.87	4.42	7.64	9.80	13.71	5.28	10.79	8.82
CWVAE	12.12	15.67	5.63	14.64	8.13	15.01	11.63	8.58	13.83	11.69
Multi Task
S2S*	24.53	23.85	5.06	9.44	5.38	24.68	7.93	5.60	21.30	14.20
COMET*	25.82	25.54	5.39	10.39	5.36	26.41	8.43	5.65	21.96	15.00
COMET	-	-	-	-	-	-	-	-	-	15.10
EA-VQ-VAE	26.89	25.95	5.72	10.96	5.68	25.94	8.78	6.10	22.48	15.40
Thus, we can optimize the evidence-aware decoder by maximizing the marginal likelihood in the Equation 15. Please see more details about the model hyperparameters in Appendix B.
4.1 Model Comparisons
Following Sap et al. (2019), we first use the average BLEU-2 score between each sequence in the top 10 predictions and the gold generations to evaluate the accuracy of generations. We report the result of existing methods on ATOMIC and Event2Mind datasets in the Table 1 and Table 2, respectively.
Table 1: BLEU score on nine inference dimensions of the ATOMIC test dataset with different approaches. For inference dimensions, "x" and "o" refers to PersonX and others, respectively (e.g. "xAttr": attribute of PersonX, "oEffect": effect on others). The tag (*) means re-implementation.
Methods	xIntent	xReact	oReact	Overall
Single Task
S2S	2.75	2.11	5.18	3.35
VRNMT	4.81	3.94	6.61	4.03
CWVAE	12.98	5.65	6.97	8.53
Multi Task
S2S*	19.18	4.81	4.29	9.43
COMET*	21.64	5.10	4.36	10.37
EA-VQ-VAE	23.39	5.74	4.81	11.31
These approaches are divided into two groups. The first group trains distinct models for each inference dimension separately, while the second group trains a model in a multi-task learning way for all inference dimensions. S2S is a RNN-based sequence-to-sequence model (Sutskever et al., 2014). VRNMT (Su et al., 2018) introduces a sequence of recurrent latent variables to model the semantic distribution of inferences. CWVAE propose a context-aware variational autoencoder (Du et al., 2019) to acquire context information, which
is first pre-trained on the auxiliary dataset and then fine-tuned for each inference dimension. COMET (Bosselut et al., 2019) concatenate the event with an inference dimension as the input and fine-tune the pre-trained GPT-2. Since COMET does not report the performance for each inference dimension, we re-implement the model for better comparison. Our approach is abbreviated as EA-VQ-VAE, short for Evidence-Aware Vector Quantised Variational AutoEncoder.