SlowGuess/NAACL_2021_OCR huggingface dataset:
of a uniform prior: splitting the KL term into  $\mathbb{E}_{q(\pmb {f}|\pmb {w})}\log p(\pmb {f}) - \mathbb{E}_{q(\pmb {f}|\pmb {w})}\log q(\pmb {f}|\pmb {w})$  allows us to neglect the first term. For the second term, we normalize the Gumbel-Softmax logits  $\gamma_{m}$ , i.e.,  $\gamma_{m} = \mathrm{Softmax}(\gamma_{m})$ , and compute
$$
\mathcal {L} _ {q} = - \mathbb {E} _ {q (\pmb {f} | \pmb {w})} \log q (\pmb {f} | \pmb {w}) \approx - \sum_ {m} \gamma_ {m} ^ {\top} \log \gamma_ {m}.
$$
$\mathcal{L}_q$  encourages the entropy of the variational distribution to be high which makes it hard for the encoder to distinguish the true frames from the wrong ones. We add a fixed and constant regularization coefficient  $\alpha_{q}$  to decrease the effect of this term (Bowman et al., 2015). We define the classification loss as  $\mathcal{L}_c = -\sum_{I_m > 0}I_m^\top \log \gamma_m$ , to encourage  $q$  to be good at predicting any frames that were actually observed. We weight  $\mathcal{L}_c$  by a fixed coefficient  $\alpha_{c}$ . Summing these losses together, we arrive at our objective function:
$$
\mathcal {L} = \mathcal {L} _ {w} + \alpha_ {q} \mathcal {L} _ {q} + \alpha_ {c} \mathcal {L} _ {c}. \tag {7}
$$
3.5 Relation to prior event modeling
A number of efforts have leveraged frame induction for event modeling (Cheung et al., 2013; Chambers, 2013; Ferraro and Van Durme, 2016; Kallmeyer et al., 2018; Ribeiro et al., 2019). These methods are restricted to explicit connections between events and their corresponding frames; they do not capture all the possible connections between the observed events and frames. Weber et al. (2018b) proposed a hierarchical unsupervised attention structure (HAQAE) that corrects for this. HAQAE uses vector quantization (Van Den Oord et al., 2017) to capture sequential event structure via tree-based latent variables.
Our model is related to HAQAE (Weber et al., 2018b), though with important differences. While HAQAE relies on unsupervised deterministic inference, we aim to incorporate the frame information in a softer, guided fashion. The core differences are: our model supports partially-observed frame sequences (i.e., semi-supervised learning); the linear-chain connection among the event variables in our model is simpler than the tree-based structure in HAQAE; while both works use attention mechanisms in the encoder and decoder, our attention mechanism is based on addition rather than concatenation; and our handling of latent discrete variables is based on the Gumbel-Softmax reparametrization, rather than vector quantization. We discuss these differences further in Appendix C.1.
We test the performance of our model on a portion of the Concretely Annotated Wikipedia dataset (Ferraro et al., 2014), which is a dump of English Wikipedia that has been annotated with the outputs of more than a dozen NLP analytics; we use this as it has readily-available FrameNet annotations provided via SemaFor (Das et al., 2014). Our training data has  $457\mathrm{k}$  documents, our validation set has  $16\mathrm{k}$  documents, and our test set has  $21\mathrm{k}$  documents. More than  $99\%$  of the frames are concentrated in the first 500 most common frames, so we set  $F = 500$ . Nearly  $15\%$  of the events did not have any frame, many of which were due to auxiliary/modal verb structures; as a result, we did not include them. For all the experiments, the vocabulary size  $(V)$  is set as  $40k$  and the number of events  $(M)$  is 5; this is to maintain comparability with HAQAE. For the documents that had more than 5 events, we extracted the first 5 events that had frames. For both the validation and test datasets, we have set  $I_{m} = 0$  for all the events; frames are only observed during training.
Documents are fed to the model as a sequence of events with verb, subj, object and modifier elements. The events are separated with a special separating  $\langle \mathrm{TUP} \rangle$  token and the missing elements are represented with a special NONE token. In order to facilitate semi-supervised training and examine the impact of frame knowledge, we introduce a user-set value  $\epsilon$ : in each document, for event  $m$ , the true value of the frame is preserved in  $I_{m}$  with probability  $\epsilon$ , while with probability  $1 - \epsilon$  we set  $I_{m} = 0$ . This  $\epsilon$  is set and fixed prior to each experiment. For
Model	ε	PPL
Valid	Test
RNNLM	-	61.34 ±2.05	61.80 ±4.81
RNNLM+ROLE	-	66.07 ±0.40	60.99 ±2.11
HAQAE	-	24.39 ±0.46	21.38 ±0.25
Ours	0.0	41.18 ±0.69	36.28 ±0.74
Ours	0.2	38.52 ±0.83	33.31 ±0.63
Ours	0.4	37.79 ±0.52	33.12 ±0.54
Ours	0.5	35.84 ±0.66	31.11 ±0.85
Ours	0.7	24.20 ±1.07	21.19 ±0.76
Ours	0.8	23.68 ±0.75	20.77 ±0.73
Ours	0.9	22.52 ±0.62	19.84 ±0.52
(a) Validation and test per-word perplexities (lower is better). We always outperform RNNLM and RNNLM+ROLE, and outperform HAQAE when automatically extracted frames are sufficiently available during training  $(\epsilon \in \{0.7, 0.8, 0.9\})$ .
Model	ε	Inv Narr Cloze
Wiki	NYT
Valid	Test	Valid	Test
RNNLM	-	20.33 ±0.56	21.37 ±0.98	18.11 ±0.41	17.86 ±0.80
RNNLM+ROLE	-	19.57 ±0.68	19.69 ±0.97	17.56 ±0.10	17.95 ±0.25
HAQAE	-	29.18 ±1.40	24.88 ±1.35	20.5 ±1.31	22.11 ±0.49
Ours	0.0	43.80 ±2.93	45.75 ±3.47	29.40 ±1.17	28.63 ±0.37
Ours	0.2	45.78 ±1.53	44.38 ±2.10	29.50 ±1.13	29.30 ±1.45
Ours	0.4	47.65 ±3.40	47.88 ±3.59	30.01 ±1.27	30.61 ±0.37
Ours	0.5	42.38 ±2.41	40.18 ±0.90	29.36 ±1.58	29.95 ±0.97
Ours	0.7	38.40 ±1.20	39.08 ±1.55	29.15 ±0.95	30.13 ±0.66
Ours	0.8	39.48 ±3.02	38.96 ±2.75	29.50 ±0.30	30.33 ±0.81
Ours	0.9	35.61 ±0.62	35.56 ±1.70	28.41 ±0.29	29.01 ±0.84
(b) Inverse Narrative Cloze scores (higher is better), averaged across 3 runs, with standard deviation reported. Some frame observation  $(\epsilon = 0.4)$  is most effective across Wikipedia and NYT, though we outperform our baselines, including the SOTA, at any level of frame observation. For the NYT dataset, we first trained the model on the Wikipedia dataset and then did the tests on the NYT valid and test inverse narrative cloze datasets.
Table 1: Validation and test results for per-word perplexity (Table 1a: lower is better) and inverse narrative cloze accuracy (Table 1b: higher is better). Recall that  $\epsilon$  is the (average) percent of frames observed during training though during evaluation no frames are observed.
all the experiments we set  $\alpha_{q}$  and  $\alpha_{c}$  as 0.1, found empirically on the validation data.
Setup We represent words by their pretrained Glove 300 embeddings and used gradient clipping at 5.0 to prevent exploding gradients. We use a two layer of bi-directional GRU for the encoder, and a two layer uni-directional GRU for the decoder (with a hidden dimension of 512 for both). See Appendix B for additional computational details.
Baselines In our experiments, we compare our proposed methods against the following methods:
- RNNLM: We report the performance of a sequence to sequence language model with the same structure used in our own model. A Bidirectional GRU cell with two layers, hidden dimension of 512, gradient clipping at 5 and Glove 300 embeddings to represent words.
- RNNLM+ROLE (Pichotta and Mooney, 2016): This model has the same structure as RNNLM, but the role for each token (verb, subject, object, modifier) as a learnable embedding vector is concatenated to the token embeddings and then it is fed to the model. The embedding dimension for roles is 300.
- HAQAE (Weber et al., 2018b) This work is the most similar to ours. For fairness, we seed HAQAE with the same dimension GRUs and pretrained embeddings.
4.1 Evaluations
To measure the effectiveness of our proposed model for event representation, we first report the perplexity and Inverse Narrative Cloze metrics.
Perplexity We summarize our per-word perplexity results in Table 1a, which compares our event chain model, with varying  $\epsilon$  values, to the three baselines. Recall that  $\epsilon$  refers to the (average) percent of frames observed during training. During evaluation no frames are observed; this ensures a fair comparison to our baselines.
As clearly seen, our model outperforms other baselines across both the validation and test datasets. We find that increasing the observation probability  $\epsilon$  consistently yields performance improvement. For any value of  $\epsilon$  we outperform RNNLM and RNNLM+ROLE. HAQAE outperforms our model for  $\epsilon \leq 0.5$ , while we outperform HAQAE for  $\epsilon \in \{0.7, 0.8, 0.9\}$ . This suggests that while the tree-based latent structure can be helpful when external, semantic knowledge is not sufficiently available during training, a simpler linear structure can be successfully guided by that knowledge when it is available. Finally, recall that the external frame annotations are automatically provided, without human curation: this suggests that our model does not require perfectly, curated annotations. These observations support the hypothesis that frame observations, in conjunction with latent variables, provide a benefit to event modeling.
In our case, perplexity provides an indication of the model's ability to predict the next event and arguments.
<sup>4</sup>https://github.com/mmrezaee/SSDVAE
Tokens	βenc(Frames Given Tokens)
kills	KILLING	DEATH	HUNTING_SUCCESS_OR_FAILURE	HIT_TARGET	ATTACK
paper	SUBMITTING_DOCUMENTS	SUMMARIZING	SIGN	DECIDING	EXPLAINING_the_FACTS
business	COMMERCE_PAY	COMMERCEBuy	EXPENSIVENESS	RENTING	REPORTING
Frames	βdec(Tokens Given Frames)
CAUSATION	raise	rendered	caused	induced	brought
PERSONAL_RELATIONSHIP	dated	dating	married	divorced	widowed
FINISH_COMPETITION	lost	compete	won	competed	competes
Table 2: Results for the outputs of the attention layer, the upper table shows the  $\beta_{\mathrm{enc}}$  and the bottom table shows the  $\beta_{\mathrm{dec}}$ , when  $\epsilon = 0.7$ . Each row shows the top 5 words for each clustering. See Table 9 in the Appendix for more examples.
Inverse Narrative Cloze This task has been proposed by Weber et al. (2018b) to evaluate the ability of models to classify the legitimate sequence of events over detractor sequences. For this task, we have created two Wiki-based datasets from our validation and test datasets, each with 2k samples. Each sample has 6 options in which the first events are the same and only one of the options represents the actual sequence of events. All the options have a fixed size of 6 events and the one that has the lowest perplexity is selected as the correct one. We also consider two NYT inverse narrative cloze datasets that are publicly available. All the models are trained on the Wiki dataset and then classifications are done on the NYT dataset (no NYT training data was publicly available).
Table 1b presents the results for this task. Our method tends to achieve a superior classification score over all the baselines, even for small  $\epsilon$ . Our model also yields performance improvements on the NYT validation and test datasets. We observe that the inverse narrative cloze scores for the NYT datasets is almost independent from the  $\epsilon$ . We suspect this due to the different domains between training (Wikipedia) and testing (newswire).
Note that while our model's perplexity improved monotonically as  $\epsilon$  increased, we do not see monotonic changes, with respect to  $\epsilon$ , for this task. By examining computed quantities from our model, we observed both that a high  $\epsilon$  resulted in very low entropy attention and that frames very often attended to the verb of the event—it learned this association despite never being explicitly directly to. While this is a benefit to localized next word prediction (i.e., perplexity), it is detrimental to inverse narrative cloze. On the other hand, lower  $\epsilon$  resulted in slightly higher attention entropy, suggesting that less peaky attention allows the model to capture more of the entire event sequence and improve global coherence.
4.2 Qualitative Analysis of Attention
To illustrate the effectiveness of our proposed attention mechanism, in Table 2 we show the most likely frames given tokens  $(\beta_{\mathrm{enc}})$ , and tokens given frames  $(\beta_{\mathrm{dec}})$ . We define  $\beta_{\mathrm{enc}} = W_{\mathrm{out}} \tanh(\boldsymbol{H}^{\top})$  and  $\beta_{\mathrm{dec}} = \hat{W}_{\mathrm{out}} \tanh(\boldsymbol{E}^{M^{\top}})$  where  $\beta_{\mathrm{enc}} \in \mathbb{R}^{F \times T}$  provides a contextual token-to-frame soft clustering matrix for each document and analogously  $\beta_{\mathrm{dec}} \in \mathbb{R}^{V \times M}$  provides a frame-to-word soft-clustering contextualized in part based on the inferred frames. We argue that these clusters are useful for analyzing and interpreting the model and its predictions. Our experiments demonstrate that the frames in the encoder (Table 2, top) mostly attend to the verbs and similarly the decoder utilizes expected and reasonable frames to predict the next verb. Note that we have not restricted the frames and tokens connection: the attention mechanism makes the ultimate decision for these connections.
We note that these clusters are a result of our attention mechanisms. Recall that in both the encoder and decoder algorithms, after computing the context vectors, we use the addition of two  $\tanh(\cdot)$  functions with the goal of separating the GRU hidden states and frame embeddings (line 3). This is a different computation from the bi-linear attention mechanism (Luong et al., 2015) that applies the  $\tanh(\cdot)$  function over concatenation. Our additive approach was inspired by the neural topic modeling method from Dieng et al. (2016), which similarly uses additive factors to learn an expressive and predictive neural component and the classic "topics" (distributions/clusters over words) that traditional topic models excel at finding. While theoretical guarantees are beyond our scope, qualitative analyses suggest that our additive attention lets the model learn reasonable soft clusters of tokens into frame-based "topics." See Table 6 in the Appendix for an empirical comparison and validation of our use of addition rather than concatenation in the attention mechanisms.
$^{6}$ https://git.io/Jkm46
Model	ε	Valid	Test
Acc	Prec	f1	Acc	Prec	f1
RNNLM	-	0.89	0.73	0.66	0.88	0.71	0.65
RNNLM + ROLE	-	0.89	0.75	0.69	0.88	0.74	0.68
Ours	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Ours	0.20	0.59	0.27	0.28	0.58	0.27	0.28
Ours	0.40	0.77	0.49	0.50	0.77	0.49	0.50
Ours	0.50	0.79	0.51	0.48	0.79	0.50	0.48
Ours	0.70	0.85	0.69	0.65	0.84	0.69	0.65
Ours	0.80	0.86	0.77	0.74	0.85	0.76	0.74
Ours	0.90	0.87	0.81	0.78	0.86	0.81	0.79
Table 3: Accuracy and macro precision and F1-score, averaged across three different runs. We present standard deviations in Table 5.
4.3 How Discriminative Is A Latent Node?
Though we develop a generative model, we want to make sure the latent nodes are capable of leveraging the frame information in the decoder. We examine this assessing the ability of one single latent node to classify the frame for an event. We repurpose the Wikipedia language modeling dataset into a new training data set with 1,643,656 samples, validation with 57,261 samples and test with 75903 samples. We used 500 frame labels. Each sample is a single event. We fixed the number of latent nodes to be one. We use RNNLM and RNNLM+ROLE as baselines, adding a linear classifier layer followed by the softmax function on top of the bidirectional GRU last hidden vectors and a dropout of 0.15 on the logits. We trained all the models with the aforementioned training dataset, and tuned the hyper parameters on the validation dataset.
We trained the RNNLM and RNNLM+ROLE baselines in a purely supervised way, whereas our model mixed supervised (discriminative) and unsupervised (generative) training. The baselines observed all of the frame labels in the training set; our model only observed frame values in training with probability  $\epsilon$ , which it predicted from  $\gamma_{m}$ . The parameters leading to the highest accuracy were chosen to evaluate the classification on the test dataset. The results for this task are summarized in Table 3. Our method is an attention based model which captures all the dependencies in each event to construct the latent representation, but the baselines are autoregressive models. Our encoder acts like a discriminative classifier to predict the frames, where they will later be used in the decoder to construct the events. We expect the model performance to be comparable to RNNLM and RNNLM+ROLE in terms of classification when  $\epsilon$  is high. Our model with larger  $\epsilon$  tends to achieve better performance in terms of macro precision and macro F1-score.
Figure 3: Validation NLL during training of our model with  $\epsilon \in \{0.2, 0.7, 0.9\}$  and HAQAE (the red curve). Epochs are displayed with a square root transform.
4.4 Training Speed
Our experiments show that our proposed approach converges faster than the existing HAQAE model. For fairness, we have used the same data-loader, batch size as 100, learning rate as  $10^{-3}$  and Adam optimizer. In Fig. 3, on average each iteration takes 0.2951 seconds for HAQAE and 0.2958 seconds for our model. From Fig. 3 we can see that for sufficiently high values of  $\epsilon$  our model is converging both better, in terms of negative log-likelihood (NLL), and faster—though for small  $\epsilon$ , our model still converges much faster. The reasons for this can be boiled down to utilizing Gumbel-Softmax rather than VQ-VAE, and also injection information in the form of frames jointly.
We showed how to learn a semi-supervised VAE with partially observed, sequential, discrete latent variables. We used Gumbel-Softmax and a modified attention to learn a highly effective event language model (low perplexity), predictor of how an initial event may progress (improved inverse narrative cloze), and a task-based classifier (outperforming fully supervised systems). We believe that future work could extend our method by incorporating other sources or types of knowledge (such as entity or "commonsense" knowledge), and by using other forms of a prior distribution, such as "plug-and-play" priors (Guo et al., 2019; Mohammadi et al., 2021; Laumont et al., 2021).
Acknowledgements and Funding Disclosure We would also like to thank the anonymous reviewers for their comments, questions, and sug
gestions. Some experiments were conducted on the UMBC HPCF, supported by the National Science Foundation under Grant No. CNS-1920079. We'd also like to thank the reviewers for their comments and suggestions. This material is based in part upon work supported by the National Science Foundation under Grant Nos. IIS-1940931 and IIS-2024878. This material is also based on research that is in part supported by the Air Force Research Laboratory (AFRL), DARPA, for the KAIROS program under agreement number FA8750-19-2-1003. The U.S.Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either express or implied, of the Air Force Research Laboratory (AFRL), DARPA, or the U.S. Government.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
David Bamman, Brendan O'Connor, and Noah A. Smith. 2013. Learning latent personas of film characters. In ACL.
Yonatan Bisk, Jan Buys, Karl Pichotta, and Yejin Choi. 2019. Benchmarking hierarchical script knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4077-4085.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349.
Nathanael Chambers. 2013. Event schema induction with a probabilistic entity-driven model. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807.
Mingda Chen, Qingming Tang, Karen Livescu, and Kevin Gimpel. 2018. Variational sequential labelers for semi-supervised learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 215-226.
Jackie Chi Kit Cheung, Hoifung Poon, and Lucy Vanderwende. 2013. Probabilistic frame induction. *NAACL HLT* 2013, pages 837-846.
Dipanjan Das, Desai Chen, Andre FT Martins, Nathan Schneider, and Noah A Smith. 2014. Frame-semantic parsing. Computational linguistics, 40(1):9-56.
Adji B Dieng, Chong Wang, Jianfeng Gao, and John Paisley. 2016. Topicrn: A recurrent neural network with long-range semantic dependency. arXiv preprint arXiv:1611.01702.
Xiao Ding, Kuo Liao, Ting Liu, Zhongyang Li, and Junwen Duan. 2019. Event representation learning enhanced with external commonsense knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4896-4905.
Francis Ferraro, Adam Poliak, Ryan Cotterell, and Benjamin Van Durme. 2017. Frame-based continuous lexical semantics through exponential family tensor factorization and semantic proto-roles. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM* 2017), pages 97–103, Vancouver, Canada. Association for Computational Linguistics.
Francis Ferraro, Max Thomas, Matthew R. Gormley, Travis Wolfe, Craig Harman, and Benjamin Van Durme. 2014. Concretely Annotated Corpora. In 4th Workshop on Automated Knowledge Base Construction (AKBC), Montreal, Canada.
Francis Ferraro and Benjamin Van Durme. 2016. A unified bayesian model of scripts, frames and language. In Thirtieth AAAI Conference on Artificial Intelligence.
Bichuan Guo, Yuxing Han, and Jiangtao Wen. 2019. Agem: Solving linear inverse problems via deep priors and sampling. Advances in Neural Information Processing Systems, 32:547-558.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In ICLR.
Laura Kallmeyer, Behrang QaseimiZadeh, and Jackie Chi Kit Cheung. 2018. Coarse lexical frame acquisition at the syntax-semantics interface using a latent-variable pcfg model. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 130-141.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pages 3581-3589.