SlowGuess/NAACL_2021_OCR huggingface dataset:
Mehdi Rezaee
University of Maryland Baltimore County
Baltimore, MD 21250 USA
rezaeel@umbc.edu
Francis Ferraro
ferraro@umbc.edu
Within the context of event modeling and understanding, we propose a new method for neural sequence modeling that takes partially-observed sequences of discrete, external knowledge into account. We construct a sequential neural variational autoencoder, which uses Gumbel-Softmax reparametrization within a carefully defined encoder, to allow for successful backpropagation during training. The core idea is to allow semisupervised external discrete knowledge to guide, but not restrict, the variational latent parameters during training. Our experiments indicate that our approach not only outperforms multiple baselines and the state-of-the-art in narrative script induction, but also converges more quickly.
Event scripts are a classic way of summarizing events, participants, and other relevant information as a way of analyzing complex situations (Schank and Abelson, 1977). To learn these scripts we must be able to group similar-events together, learn common patterns/sequences of events, and learn to represent an event's arguments (Minsky, 1974). While continuous embeddings can be learned for events and their arguments (Ferraro et al., 2017; Weber et al., 2018a), the direct inclusion of more structured, discrete knowledge is helpful in learning event representations (Ferraro and Van Durme, 2016). Obtaining fully accurate structured knowledge can be difficult, so when the external knowledge is neither sufficiently reliable nor present, a natural question arises: how can our models use the knowledge that is present?
Generative probabilistic models provide a framework for doing so: external knowledge is a random variable, which can be observed or latent, and the data/observations are generated (explained) from it. Knowledge that is discrete, sequential, or both
Figure 1: An overview of event modeling, where the observed events (black text) are generated via a sequence of semi-observed random variables. In this case, the random variables are directed to take on the meaning of semantic frames that can be helpful to explain the events. Unobserved frames are in orange and observed frames are in blue. Some connections are more important than others, indicated by the weighted arrows.
such as for script learning—complicates the development of neural generative models.
In this paper, we provide a successful approach for incorporating partially-observed, discrete, sequential external knowledge in a neural generative model. We specifically examine event sequence modeling augmented by semantic frames. Frames (Minsky, 1974, i.a.) are a semantic representation designed to capture the common and general knowledge we have about events, situations, and things. They have been effective in providing crucial information for modeling and understanding the meaning of events (Peng and Roth, 2016; Ferraro et al., 2017; Padia et al., 2018; Zhang et al., 2020). Though we focus on semantic frames as our source of external knowledge, we argue this work is applicable to other similar types of knowledge.
We examine the problem of modeling observed event tuples as a partially observed sequence of semantic frames. Consider the following three events, preceded by their corresponding bracketed frames, from Fig. 1:
[EVENT] crash occurred at station.
[IMPACT] train collided with train.
[KILLING] killed passengers and injured.
We can see that even without knowing the [KILLING] frame, the [EVENT] and [IMPACT]
Association for Computational Linguistics: Human Language Technologies, pages 4701-4716
frames can help predict the word killed in the third event; the frames summarize the events and can be used as guidance for the latent variables to represent the data. On the other hand, words like crash, station and killed from the first and third events play a role in predicting [IMPACT] in the second event. Overall, to successfully represent events, beyond capturing the event to event connections, we propose to consider all the information from the frames to events and frames to frames.
In this work, we study the effect of tying discrete, sequential latent variables to partially-observable, noisy (imperfect) semantic frames. Like Weber et al. (2018b), our semi-supervised model is a bidirectional auto-encoder, with a structured collection of latent variables separating the encoder and decoder, and attention mechanisms on both the encoder and decoder. Rather than applying vector quantization, we adopt a Gumbel-Softmax (Jang et al., 2017) ancestral sampling method to easily switch between the observed frames and latent ones, where we inject the observed frame information on the Gumbel-Softmax parameters before sampling. Overall, our contributions are:
- We demonstrate how to learn a VAE that contains sequential, discrete, and partially-observed latent variables.
- We show that adding partially-observed, external, semantic frame knowledge to our structured, neural generative model leads to improvements over the current state-of-the-art on recent core event modeling tasks. Our approach leads to faster training convergence.
- We show that our semi-supervised model, though developed as a generative model, can effectively predict the labels that it may not observe. Additionally, we find that our model outperforms a discriminatively trained model with full supervision.
Our work builds on both event modeling and latent generative models. In this section, we outline relevant background and related work.
2.1 Latent Generative Modeling
Generative latent variable models learn a mapping from the low-dimensional hidden variables  $\mathbf{f}$  to the observed data points  $\mathbf{x}$ , where the hidden representation captures the high-level information to
explain the data. Mathematically, the joint probability  $p(\mathbf{x},\mathbf{f};\theta)$  factorizes as follows
$$
p (\mathbf {x}, \mathbf {f}; \theta) = p (\mathbf {f}) p (\mathbf {x} | \mathbf {f}; \theta), \tag {1}
$$
where  $\theta$  represents the model parameters. Since in practice maximizing the log-likelihood is intractable, we approximate the posterior by defining  $q(\mathbf{f}|\mathbf{x};\phi)$  and maximize the ELBO (Kingma and Welling, 2013) as a surrogate objective:
$$
\mathcal {L} _ {\theta , \phi} = \mathbb {E} _ {q (\mathbf {f} | \mathbf {x}; \phi)} \log \frac {p (\mathbf {x} , \mathbf {f} ; \theta)}{q (\mathbf {f} | \mathbf {x} ; \phi)}. \tag {2}
$$
In this paper, we are interested in studying a specific case; the input  $\mathbf{x}$  is a sequence of  $T$  tokens, we have  $M$  sequential discrete latent variables  $\mathbf{z} = \{z_m\}_{m=1}^M$ , where each  $z_m$  takes  $F$  discrete values. While there have been effective proposals for unsupervised optimization of  $\theta$  and  $\phi$ , we focus on learning partially observed sequences of these variables. That is, we assume that in the training phase some values are observed while others are latent. We incorporate this partially observed, external knowledge to the  $\phi$  parameters to guide the inference. The inferred latent variables later will be used to reconstruct to the tokens.
Kingma et al. (2014) generalized VAEs to the semi-supervised setup, but they assume that the dataset can be split into observed and unobserved samples and they have defined separate loss functions for each case; in our work, we allow portions of a sequence to be latent. Teng et al. (2020) characterized the semi-supervised VAEs via sparse latent variables; see Mousavi et al. (2019) for an in-depth study of additional sparse models.
Of the approaches that have been developed for handling discrete latent variables in a neural model (Vahdat et al., 2018a,b; Lorberbom et al., 2019, i.a.), we use the Gumbel-Softmax reparametrization (Jang et al., 2017; Maddison et al., 2016). This approximates a discrete draw with logits  $\pi$  as  $\mathrm{softmax}(\frac{\pi + g}{\tau})$ , where  $g$  is a vector of Gumbel(0,1) draws and  $\tau$  is an annealing temperature that allows targeted behavior; its ease, customizability, and efficacy are big advantages.
2.2 Event Modeling
Sequential event modeling, as in this paper, can be viewed as a type of script or schema induction (Schank and Abelson, 1977) via language modeling techniques. Mooney and DeJong (1985) provided an early analysis of explanatory schema
Figure 2: Our encoder (left) and decoder (right). The orange nodes mean that the frame is latent  $(I_m = 0)$ , while the blue nodes indicate observed frames  $(I_m$  is one-hot). In the encoder, the RNN hidden vectors are aligned with the frames to predict the next frame. The decoder utilizes the inferred frame information in the reconstruction.
generating system to process narratives, and Pichotta and Mooney (2016) applied an LSTM-based model to predict the event arguments. Modi (2016) proposed a neural network model to predict randomly missing events, while Rudinger et al. (2015) showed how neural language modeling can be used for sequential event prediction. Weber et al. (2018a) and Ding et al. (2019) used tensor-based decomposition methods for event representation. Weber et al. (2020) studied causality in event modeling via a latent neural language model.
Previous work has also examined how to incorporate or learn various forms of semantic representations while modeling events. Cheung et al. (2013) introduced an HMM-based model to explain event sequences via latent frames. Materna (2012), Chambers (2013) and Bamman et al. (2013) provided structured graphical models to learn event models over syntactic dependencies; Ferraro and Van Durme (2016) unified and generalized these approaches to capture varying levels of semantic forms and representation. Kallmeyer et al. (2018) proposed a Bayesian network based on a hierarchical dependency between syntactic dependencies and frames. Ribeiro et al. (2019) provided an analysis of clustering predicates and their arguments to infer semantic frames.
Variational autoencoders and attention networks (Kingma and Welling, 2013; Bahdanau et al., 2014), allowed Bisk et al. (2019) to use RNNs with attention to capture the abstract and concrete script representations. Weber et al. (2018b) came up with a recurrent autoencoder model (HAQAE), which used vector-quantization to learn hierarchical dependencies among discrete latent variables and an observed event sequence. Kiyomaru et al. (2019) suggested generating next events using a condi
tional VAE-based model.
In another thread of research, Chen et al. (2018) utilized labels in conjunction with latent variables, but unlike Weber et al. (2018b), their model's latent variables are conditionally independent and do not form a hierarchy. Sønderby et al. (2016) proposed a sequential latent structure with Gaussian latent variables. Lévin et al. (2019), similar to our model structure, provided an analysis of hierarchical relaxed categorical sampling but for the unsupervised settings.
Our focus in this paper is modeling sequential event structure. In this section, we describe our variational autoencoder model, and demonstrate how partially-observed external knowledge can be injected into the learning process. We provide an overview of our joint model in §3.1 and Fig. 2. Our model operates on sequences of events: it consists of an encoder (§3.2) that encodes the sequence of events as a new sequence of frames (higher-level, more abstract representations), and a decoder (§3.3) that learns how to reconstruct the original sequence of events from the representation provided by the encoder. During training (§3.4), the model can make use of partially-observed sequential knowledge to enrich the representations produced by the encoder. In §3.5 we summarize the novel aspects of our model.
3.1 Model Setup
We define each document as a sequence of  $M$  events. In keeping with previous work on event representation, each event is represented as a lexicalized 4-tuple: the core event predicate (verb), two main arguments (subject and object), and event
modifier (if applicable) (Pichotta and Mooney, 2016; Weber et al., 2018b). For simplicity, we can write each document as a sequence of  $T$  words  $\mathbf{w} = \{w_t\}_{t=1}^T$ , where  $T = 4M$  and each  $w_t$  is from a vocabulary of size  $V$ .<sup>1</sup> Fig. 1 gives an example of 3 events: during learning (but not testing) our model would have access to some, but not all, frames to lightly guide training (in this case, the first two frames but not the third).
While lexically rich, this 4-tuple representation is limited in the knowledge that can be directly encoded. Therefore, our model assumes that each document  $\mathbf{w}$  can be explained jointly with a collection of  $M$  random variables  $f_{m}$ :  $\boldsymbol {f} = \{f_m\}_{m = 1}^M$ . The joint probability for our model factorizes as
$$
p (\boldsymbol {w}, \boldsymbol {f}) = \prod_ {t = 1} ^ {T} p \left(w _ {t} \mid \boldsymbol {f}, w _ {<   t}\right) \prod_ {m = 1} ^ {M} p \left(f _ {m} \mid f _ {m - 1}\right). \tag {3}
$$
For event modeling, each  $f_{m}$  represents a semantic frame. We assume there are  $F$  unique frames and let  $f_{m}$  be a discrete variable indicating which frame, if any, was triggered by event  $m$ .<sup>2</sup>
In the general case,  $\mathbf{f}$  is completely unobserved, and so inference for this model requires marginalizing over  $\mathbf{f}$ : when  $F \gg 1$ , optimizing the likelihood is intractable. We follow amortized variational inference (Kingma and Welling, 2013) as an alternative approach and use an ancestral sampling technique to compute it. We define  $q(\boldsymbol{f}|\boldsymbol{w})$  as the variational distribution over  $\boldsymbol{f}$ , which can be thought of as stochastically encoding  $\boldsymbol{w}$  as  $\boldsymbol{f}$ .
Our method is semi-supervised, so we follow Kingma et al. (2014), Chen et al. (2018) and Ye et al. (2020) and optimize a weighted variant of the evidence lower bound (ELBO),
$$
\mathcal {L} = \overbrace {\mathbb {E} _ {q (\boldsymbol {f} | \boldsymbol {w})} \log p (\boldsymbol {w} | \boldsymbol {f})} ^ {\text {R e c o n s t r u c t i o n t e r m}} + \overbrace {\alpha_ {q} \mathbb {E} _ {q (\boldsymbol {f} | \boldsymbol {w})} \log \frac {p (\boldsymbol {f})}{q (\boldsymbol {f} | \boldsymbol {w})}} ^ {\text {K L t e r m}},
$$
Supervised classification term
$$
+ \overbrace {\alpha_ {c} \mathcal {L} _ {c} (q (\boldsymbol {f} | \boldsymbol {w}))}, \tag {4}
$$
where  $\mathcal{L}_c(q(\boldsymbol {f}|\boldsymbol {w}))$  is a classification objective that encourages  $q$  to predict the frames that actually were observed, and  $\alpha_{q}$  and  $\alpha_{c}$  are empirically-set to give different weight to the KL vs. classification terms. We define  $\mathcal{L}_c$  in §3.4. The reconstruc
tion term learns to generate the observed events  $\pmb{w}$  across all valid encodings  $\pmb{f}$ , while the KL term uses the prior  $p(\pmb{f})$  to regularize  $q$ .
Optimizing Eq. (4) is in general intractable, so we sample  $S$  chains of variables  $\pmb{f}^{(1)},\dots,\pmb{f}^{(S)}$  from  $q(\pmb {f}|\pmb {w})$  and approximate Eq. (4) as
$$
\begin{array}{l} \mathcal {L} \approx \frac {1}{S} \sum_ {s} \left[ \log p (\boldsymbol {w} | \boldsymbol {f} ^ {(s)}) + \alpha_ {q} \log \frac {p (\boldsymbol {f} ^ {(s)})}{q (\boldsymbol {f} ^ {(s)} | \boldsymbol {w})} + \right. \\ \left. \alpha_ {c} \mathcal {L} _ {c} \left(q \left(\boldsymbol {f} ^ {(s)} \mid \boldsymbol {w}\right)\right) \right]. \tag {5} \\ \end{array}
$$
As our model is designed to allow the injection of external knowledge  $\pmb{I}$ , we define the variational distribution as  $q(\pmb{f}|w;I)$ . In our experiments,  $I_{m}$  is a binary vector encoding which (if any) frame is observed for event  $m$ . For example in Fig. 1, we have  $I_{1} = 1$  and  $I_{3} = 0$ . We define
$$
q (\boldsymbol {f} | \boldsymbol {w}; \boldsymbol {I}) = \prod_ {m = 1} ^ {M} q \left(f _ {m} \mid f _ {m - 1}, I _ {m}, \boldsymbol {w}\right). \tag {6}
$$
We broadly refer to Eq. (6) as our encoder; we detail this in §3.2. In §3.3 we describe how we compute the reconstruction term, and in §3.4 we provide our semi-supervised training procedure.
The reconstruction term relies on the frame samples given by the encoder. As discussed above though, we must be able to draw chains of variables  $\pmb{f}$ , by iteratively sampling  $f_{m} \sim q(\cdot | f_{m-1}, \pmb{w}; \pmb{I})$ , in a way that allows the external knowledge  $\pmb{I}$  to guide, but not restrict,  $\pmb{f}$ . This is a deceptively difficult task, as the encoder must be able to take the external knowledge into account in a way that neither prevents nor harms back-propagation and learning. We solve this problem by learning to compute a good representation  $\gamma_{m}$  for each event, and sampling the current frame  $f_{m}$  from a Gumbel-Softmax distribution (Jang et al., 2017) parametrized by  $\gamma_{m}$ .
Alg. 1 gives a detailed description of our encoder. We first run our event sequence through a recurrent network (like an RNN or bi-LSTM); if  $\pmb{w}$  is  $T$  tokens long, this produces  $T$  hidden representations, each of size  $d_h$ . Let this collection be  $\mathbf{H} \in \mathbb{R}^{T \times d_h}$ . Our encoder proceeds iteratively over each of the  $M$  events as follows: given the previous sampled frame  $f_{m-1}$ , the encoder first computes a weighted embedding  $e_{f_{m-1}}$  of this previous frame (line 1).
Table 4 in the Appendix provides all the notations.
2Our model is theoretically adaptable to making use of multiple frames, though we assume each event triggers at most one frame.
3If a frame is observed, then  $I_{m}$  is a one-hot vector where the index of the observed frame is 1. Otherwise  $I_{m} = \overrightarrow{\mathbf{0}}$
Algorithm 1 Encoder: The following algorithm shows how we compute the next frame  $f_{m}$  given the previous frame  $f_{m - 1}$ . We compute and return a hidden frame representation  $\gamma_{m}$ , and  $f_{m}$  via a continuous Gumbel-Softmax reparametrization.
Input: previous frame  $f_{m - 1}$ $\triangleright f_{m - 1}\in \mathbb{R}^{F}$  current frame observation  $(I_m)$  encoder GRU hidden states  $\mathbf{H}\in \mathbb{R}^{T\times d_h}$
Parameters:  $W_{\mathrm{in}} \in \mathbb{R}^{d_h \times d_e}$ ,  $W_{\mathrm{out}} \in \mathbb{R}^{F \times d_h}$ , frames embeddings  $\pmb{E}^{F} \in \mathbb{R}^{F \times d_{e}}$
Output:  $f_{m},\gamma_{m}$
1:  $e_{f_{m - 1}} = f_{m - 1}^{\top}\pmb{E}^{F}$
2:  $\pmb{\alpha} \gets \operatorname{Softmax}(\mathbf{H}W_{\mathrm{in}}e_{f_{m-1}}^{\top}) \triangleright \mathrm{Attn}$ . Scores
3:  $\pmb{c}_m \gets \pmb{H}^\top \pmb{\alpha}$  ▷ Context Vector
4:  $\pmb{\gamma}_{m}^{\prime}\gets W_{\mathrm{out}}(\tanh (W_{\mathrm{in}}e_{f_{m - 1}}) + \tanh (\pmb {c}_m))$
5:  $\gamma_{m}\gets \gamma_{m} + \| \gamma_{m}\|$ $I_{m}$   ObservaTio
6:  $q(f_{m}|f_{m - 1})\gets \mathrm{GumbelSoftmax}(\pmb{\gamma}_{m})$
7:  $f_{m}\sim q(f_{m}|f_{m - 1})$ $\triangleright f_m\in \mathbb{R}^F$
Next, it calculates the similarity between  $e_{f_{m-1}}$  and RNN hidden representations and all recurrent hidden states  $\mathbf{H}$  (line 2). After deriving the attention scores, the weighted average of hidden states  $(c_m)$  summarizes the role of tokens in influencing the frame  $f_m$  for the  $m^{\text{th}}$  event (line 3). We then combine the previous frame embedding  $e_{f_{m-1}}$  and the current context vector  $c_m$  to obtain a representation  $\gamma_m'$  for the  $m^{\text{th}}$  event (line 4).
While  $\gamma_{m}^{\prime}$  may be an appropriate representation if no external knowledge is provided, our encoder needs to be able to inject any provided external knowledge  $I_{m}$ . Our model defines a chain of variables—some of which may be observed and some of which may be latent—so care must be taken to preserve the gradient flow within the network. We note that an initial strategy of solely using  $I_{m}$  instead of  $f_{m}$  (whenever  $I_{m}$  is provided) is not sufficient to ensure gradient flow. Instead, we incorporate the observed information given by  $I_{m}$  by adding this information to the output of the encoder logits before drawing  $f_{m}$  samples (line 5). This remedy motivates the encoder to softly increase the importance of the observed frames during the training. Finally, we draw  $f_{m}$  from the Gumbel-Softmax distribution (line 7).
For example, in Fig. 1, when the model knows that [IMPACT] is triggered, it increases the value of [IMPACT] in  $\gamma_{m}$  to encourage [IMPACT] to be sampled, but it does not prevent other frames from being sampled. On the other hand, when a frame is
not observed in training, such as for the third event ([KILLING]),  $\gamma_{m}$  is not adjusted.
Since each draw  $f_{m}$  from a Gumbel-Softmax is a simplex vector, given learnable frame embeddings  $\pmb{E}^{F}$ , we can obtain an aggregate frame representation  $e_{m}$  by calculating  $e_{m} = f_{m}^{\top}\pmb{E}^{F}$ . This can be thought of as roughly extracting row  $m$  from  $\pmb{E}^{F}$  for low entropy draws  $f_{m}$ , and using many frames in the representation for high entropy draws. Via the temperature hyperparameter, the Gumbel-Softmax allows us to control the entropy.
3.3 Decoder
Our decoder (Alg. 2) must be able to reconstruct the input event token sequence from the frame representations  $\pmb{f} = (f_{1},\dots ,f_{M})$  computed by the encoder. In contrast to the encoder, the decoder is relatively simple: we use an auto-regressive (left-to-right) GRU to produce hidden representations  $z_{t}$  for each token we need to reconstruct, but we enrich that representation via an attention mechanism over  $\pmb{f}$ . Specifically, we use both  $\pmb{f}$  and the same learned frame embeddings  $\mathbf{E}^{F}$  from the encoder to compute inferred, contextualized frame embeddings as  $\pmb{E}^{M} = \pmb{f}\pmb{\mathrm{E}}^{F}\in \mathbb{R}^{M\times d_{e}}$ . For each output token (time step  $t$ ), we align the decoder GRU hidden state  $h_t$  with the rows of  $\pmb{E}^{M}$  (line 1). After calculating the scores for each frame embedding, we obtain the output context vector  $c_{t}$  (line 2), which is used in conjunction with the hidden state of the decoder  $z_{t}$  to generate the  $w_{t}$  token (line 5). In Fig. 1, the collection of all the three frames and the tokens from the first event will be used to predict the Train token from the second event.
3.4 Training Process
We now analyze the different terms in Eq. (5). In our experiments, we have set the number of samples  $S$  to be 1. From Eq. (6), and using the sampled sequence of frames  $f_{1}, f_{2}, \ldots, f_{M}$  from our encoder, we approximate the reconstruction term as  $\mathcal{L}_w = \sum_t \log p(w_t | f_1, f_2, \ldots, f_M; z_t)$ , where  $z_t$  is the decoder GRU's hidden representation after having reconstructed the previous  $t - 1$  words in the event sequence.
Looking at the KL-term, we define the prior frame-to-frame distribution as  $p(f_m|f_{m-1}) = 1 / F$ , and let the variational distribution capture the dependency between the frames. A similar type of strategy has been exploited successfully by Chen et al. (2018) to make computation simpler. We see the computational benefits
Algorithm 2 Decoder: To (re)generate each token in the event sequence, we compute an attention  $\hat{\alpha}$  over the sequence of frame random variables  $f$  (from Alg. 1). This attention weights each frame's contribution to generating the current word.
Input:  $\mathbf{E}^{M} \in \mathbb{R}^{M \times d_{e}}$  (computed as  $f\mathbf{E}^{F}$ ) decoder's current hidden state  $z_{t} \in \mathbb{R}^{d_{h}}$  Parameters:  $\hat{W}_{\mathrm{in}} \in \mathbb{R}^{d_e \times d_h}, \hat{W}_{\mathrm{out}} \in \mathbb{R}^{V \times d_e}, \mathbf{E}^{F}$  Output:  $w_{t}$
1:  $\hat{\alpha} \gets \operatorname{Softmax}(\boldsymbol{E}^{M}\hat{W}_{\mathrm{in}}z_{t})$ $\triangleright$  Attn. Scores
2:  $c_{t}\gets E^{M^{\top}}\hat{\alpha}$
3:  $g \gets \hat{W}_{\mathrm{out}}(\tanh (\hat{W}_{\mathrm{in}}z_t) + \tanh (\boldsymbol{c}_t))$
4:  $p(w_{t}|\boldsymbol {f};z_{t})\propto \exp (g)$
5:  $w_{t}\sim p(w_{t}|f;z_{t})$