SlowGuess/NAACL_2021_OCR huggingface dataset:
Hirokazu Kiyomaru, Kazumasa Omura, Yugo Murawaki, Daisuke Kawahara, and Sadao Kurohashi. 2019. Diversity-aware event prediction based on a conditional variational autoencoder with reconstruction. In Proceedings of the First Workshop on Common sense Inference in Natural Language Processing, pages 113-122.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. 2021. Bayesian imaging using plug & play priors: when langevin meets tweedie. arXiv preprint arXiv:2103.04715.
Valentin Lievin, Andrea Dittadi, Lars Maaløe, and Ole Winther. 2019. Towards hierarchical discrete variational autoencoders. In 2nd Symposium on Advances in Approximate Bayesian Inference (AABI).
Guy Lorberbom, Andreea Gane, Tommi Jaakkola, and Tamir Hazan. 2019. Direct optimization through argmax for discrete variational auto-encoder. In Advances in Neural Information Processing Systems, pages 6203-6214.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2016. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712.
Jiri Materna. 2012. LDA-Frames: an unsupervised approach to generating semantic frames. In Computational Linguistics and Intelligent Text Processing, pages 376-387. Springer.
Marvin Minsky. 1974. A framework for representing knowledge. MIT-AI Laboratory Memo 306.
Ashutosh Modi. 2016. Event embeddings for semantic script modeling. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 75-83.
Narges Mohammadi, Marvin M Doyley, and Mujdat Cetin. 2021. Ultrasound elasticity imaging using physics-based models and learning-based plug-and-play priors. arXiv preprint arXiv:2103.14096.
Raymond J Mooney and Gerald DeJong. 1985. Learning schemata for natural language processing. In *IJ-CAI*, pages 681–687.
Seyedahmad Mousavi, Mohammad Mehdi Rezaee Taghiabadi, and Ramin Ayanzadeh. 2019. A survey on compressive sensing: Classical results and recent advancements. arXiv preprint arXiv:1908.01014.
Ankur Padia, Francis Ferraro, and Tim Finin. 2018. Team UMBC-FEVER: Claim verification using semantic lexical resources. In Proceedings of the
First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 161-165, Brussels, Belgium. Workshop at EMNLP.
Haoruo Peng and Dan Roth. 2016. Two discourse driven language models for semantics. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 290-300, Berlin, Germany. Association for Computational Linguistics.
Karl Pichotta and Raymond J Mooney. 2016. Learning statistical scripts with lstm recurrent neural networks. In Thirtieth AAAI Conference on Artificial Intelligence.
Eugenio Ribeiro, Vânia Mendonça, Ricardo Ribeiro, David Martins de Matos, Alberto Sardinha, Ana Lucia Santos, and Luísca Coheur. 2019. L2f/inesc-id at semeval-2019 task 2: Unsupervised lexical semantic frame induction using contextualized word representations. In SemEval@NAACL-HLT.
Rachel Rudinger, Pushpendre Rastogi, Francis Ferraro, and Benjamin Van Durme. 2015. Script induction as language modeling. In EMNLP.
Roger C Schank and Robert P Abelson. 1977. Scripts, plans, goals, and understanding: an inquiry into human knowledge structures.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. 2016. Ladder variational autoencoders. In Advances in neural information processing systems, pages 3738-3746.
Michael Teng, Tuan Anh Le, Adam Scibior, and Frank Wood. 2020. Semi-supervised sequential generative models. In Conference on Uncertainty in Artificial Intelligence, pages 649-658. PMLR.
Arash Vahdat, Evgeny Andriyash, and William Macready. 2018a. Dvae#: Discrete variational autoencoders with relaxed boltzmann priors. In Advances in Neural Information Processing Systems, pages 1864-1874.
Arash Vahdat, William G Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny Andriyash. 2018b. Dvae++: Discrete variational autoencoders with overlapping transformations. arXiv preprint arXiv:1802.04920.
Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pages 6306-6315.
Noah Weber, Niranjan Balasubramanian, and Nathanael Chambers. 2018a. Event representations with tensor-based compositions. In Thirty-Second AAAI Conference on Artificial Intelligence.
Noah Weber, Rachel Rudinger, and Benjamin Van Durme. 2020. Causal inference of script knowledge. In Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing (EMNLP).
Noah Weber, Leena Shekhar, Niranjan Balasubramanian, and Nathanael Chambers. 2018b. Hierarchical quantized representations for script generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3783-3792.
Rong Ye, Wenxian Shi, Hao Zhou, Zhongyu Wei, and Lei Li. 2020. Variational template machine for data-to-text generation. arXiv preprint arXiv:2002.01127.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuaijiang Zhang, Xi Zhou, and Xiang Zhou. 2020. Semantics-aware BERT for language understanding. In AAAI.
Symbol	Description	Dimension
F	Frames vocabulary size	N
V	Tokens vocabulary size	N
M	Number of events, per sequence	N
T	Number of words, per sequence (4M)	N
de	Frame emb. dim.	N
dh	RNN hidden state dim.	N
EF	Learned frame emb.	RF×de
EM	Embeddings of sampled frames (EM = fEF)	RM×de
Im	Observed frame (external one-hot)	RF
fm	Sampled frame (simplex)	RF
H	RNN hidden states from the encoder	RT×dh
zt	RNN hidden state from the decoder	Rdh
Win	Learned frames-to-hidden-states weights (Encoder)	Rdh×de
Win	Learned hidden-states-to-frames weights (Decoder)	Rde×dh
Wout	Contextualized frame emb. (Encoder)	RF×dh
Wout	Learned frames-to-words weights (Decoder)	RV×de
α	Attention scores over hidden states (Encoder)	RT
ˆα	Attention scores over frame emb. (Decoder)	RM
cm	Context vector (Encoder)	Rdh
ct	Context vector (Decoder)	Rde
γm	Gumbel-Softmax params. (Encoder)	RF
wt	Tokens (onehot)	RV
Table 4: Notations used in this paper
B Computing Infrastructure
We used the Adam optimizer with initial learning rate  $10^{-3}$  and early stopping (lack of validation performance improvement for 10 iterations). We represent events by their pretrained Glove 300 embeddings and utilized gradient clipping at 5.0 to prevent exploding gradients. The Gumbel-Softmax temperature is fixed to  $\tau = 0.5$ . We have not used dropout or batch norm on any layer. We have used two layers of Bi-directional GRU cells with a hidden dimension of 512 for the encoder module and Unidirectional GRU with the same configuration for the decoder. Each model was trained using a single GPU (a Titan RTX RTX 2080 TI, or a Quadro 8000), though we note that neither our models nor the baselines required the full memory of any GPU (e.g., our models used roughly 6GB of GPU memory for a batch of 100 documents).
C Additional Insights into Novelty
We have previously mentioned how our work is most similar to HAQAE (Weber et al., 2018b). In this section, we provide a brief overview of HAQAE (Appendix C.1) and then highlight differences (Fig. 4), with examples (Appendix C.2).
C.1 Overview of HAQAE
HAQAE provides an unsupervised tree structure based on the vector quantization variational autoencoder (VQVAE) over  $M$  latent variables. Each
(a) VQVAE
(b) Gumbel-Softmax and Emb (ours)
Figure 4: (a) VQ-VAE in HAQAE works based on the minimum distance between the query vector and the embeddings. (b) our approach first draws a Gumbel-Softmax sample and then extracts the relevant row by doing vector to matrix multiplication.
latent variable  $z_{i}$  is defined in the embedding space denoted as  $e$ . The varaiational distribution to approximate  $\mathbf{z} = \{z_1, z_2, \dots, z_M\}_{i=1}^M$  given tokens  $x$  is defined as follows:
$$
q (\mathbf {z} | x) = q _ {0} \left(z _ {0} | x\right) \prod_ {i = 1} ^ {M - 1} q _ {i} \left(z _ {i} \mid \text {p a r e n t ＿ o f} \left(z _ {i}\right), x\right).
$$
The encoder calculates the attention over the input RNN hidden states  $\mathbf{h}_x$  and the parent of  $z_{i}$  to define  $q_{i}(z_{i} = k|z_{i - 1},x)$  ..
$$
\left\{ \begin{array}{l l} 1 & \mathrm {k = \operatorname* {a r g m i n} _ {j} \| g _ {i} (x , p a r e n t \_ o f (z _ {i})) - e _ {i j} \| _ {2}} \\ 0 & \text {e l s e w i s e ,} \end{array} \right.
$$
where  $g_{i}(x,\mathrm{parent\_of}(z_{i}))$  computes bilinear attention between  $\mathbf{h}_x$  and parent_of  $(z_{i})$ .
In this setting, the variational distribution is deterministic; right after deriving the latent query vector it will be compared with a lookup embedding table and the row with minimum distance is selected, see Fig. 4a. The decoder reconstructs the tokens as  $p(x_{i}|\mathbf{z})$  that calculates the attention over latent variables and the RNN hidden states. A reconstruction loss  $L_{j}^{R}$  and a "commit" loss (Weber et al., 2018b) loss  $L_{j}^{C}$  force  $g_{j}(x,\mathrm{parent\_of}(z_{i}))$  to be close to the embedding referred to by  $q_{i}(z_{i})$ . Both  $L_{j}^{R}$  and  $L_{j}^{C}$  terms rely on a deterministic mapping that is based on a nearest neighbor computation that makes it difficult to inject guiding information to the latent variable.
C.2 Frame Vector Norm
Like HAQAE, we use embeddings  $\pmb{E}^{F}$  instead of directly using the Gumbel-Softmax frame sam
Model	ε	Valid	Test
Acc	Prec	f1	Acc	Prec	f1
RNNLM	-	0.89 ±0.001	0.73 ±0.004	0.66 ±0.008	0.88 ±0.005	0.71 ±0.014	0.65 ±0.006
RNNLM + ROLE	-	0.89 ±0.004	0.75 ±0.022	0.69 ±0.026	0.88 ±0.005	0.74 ±0.017	0.68 ±0.020
Ours	0.00	0.00 ±0.001	0.00 ±0.001	0.00 ±0.000	0.00 ±0.001	0.00 ±0.000	0.00 ±0.000
Ours	0.20	0.59 ±0.020	0.27 ±0.005	0.28 ±0.090	0.58 ±0.010	0.27 ±0.050	0.28 ±0.010
Ours	0.40	0.77 ±0.012	0.49 ±0.130	0.50 ±0.010	0.77 ±0.010	0.49 ±0.060	0.50 ±0.010
Ours	0.50	0.79 ±0.010	0.51 ±0.080	0.48 ±0.080	0.79 ±0.009	0.50 ±0.080	0.48 ±0.050
Ours	0.70	0.85 ±0.007	0.69 ±0.050	0.65 ±0.040	0.84 ±0.013	0.69 ±0.050	0.65 ±0.020
Ours	0.80	0.86 ±0.003	0.77 ±0.013	0.74 ±0.006	0.85 ±0.005	0.76 ±0.008	0.74 ±0.010
Ours	0.90	0.87 ±0.002	0.81 ±0.007	0.78 ±0.006	0.86 ±0.011	0.81 ±0.020	0.79 ±0.010
Table 5: Accuracy and macro precision and F1-score, averaged across 3 runs, with standard deviation reported.
ε	Addition	Concatenation
Valid	Test	Valid	Test
0.0	41.18 ±0.69	36.28 ±0.74	48.86 ±0.13	42.76 ±0.08
0.2	38.52 ±0.83	33.31 ±0.63	48.73 ±0.18	42.76 ±0.06
0.4	37.79 ±0.52	33.12 ±0.54	49.34 ±0.32	43.18 ±0.21
0.5	35.84 ±0.66	31.11 ±0.85	49.83 ±0.24	43.90 ±0.32
0.7	24.20 ±1.07	21.19 ±0.76	52.41 ±0.38	45.97 ±0.48
0.8	23.68 ±0.75	20.77 ±0.73	55.13 ±0.31	48.27 ±0.21
0.9	22.52±0.62	19.84 ±0.52	58.63 ±0.25	51.39 ±0.34
Table 6: Validation and test per-word perplexities with bilinear-attention (lower is better).
plies. In our model definition, each frame  $f_{m} = [f_{m,1}, f_{m,2}, \ldots, f_{m,F}]$  sampled from the Gumbel-Softmax distribution is a simplex vector:
$$
0 \leq f _ {m, i} \leq 1, \quad \sum_ {i = 1} ^ {F} f _ {m, i} = 1. \tag {8}
$$
So  $\| f_m\| _p = \left(\sum_{i = 1}^F f_{m,i}^p\right)^{1 / p}\leq F^{1 / p}$ . After sampling a frame simplex vector, we multiply it to the frame embeddings matrix  $\pmb{E}^{F}$ . With an appropriate temperature for Gumbel-Softmax, the simplex would be approximately a one-hot vector and the multiplication maps the simplex vector to the embeddings space without any limitation on the norm.
In this section, we provide additional quantitative and qualitative results, to supplement what was presented in  $\S 4$ .
D.1 Standard Deviation for Frame Prediction (§4.3)
In Table 5, we see the average results of classification metrics with their corresponding standard deviations.
D.2 Importance of Using Addition rather than Concatenation in Attention
To demonstrate the effectiveness of our proposed attention mechanism, we compare the addition
against the concatenation (regular bilinear attention) method. We report the results on the Wikipedia dataset in Table 6. Experiments indicate that the regular bilinear attention structure with larger  $\epsilon$  obtains worse performance. These results confirm the claim that the proposed approach benefits from the addition structure.
D.3 Generated Sentences
Recall that the reconstruction loss is
$$
\mathcal {L} _ {w} \approx \frac {1}{S} \sum_ {s} \log p (\pmb {w} | f _ {1} ^ {(s)}, f _ {2} ^ {(s)}, \dots f _ {M} ^ {(s)}; z),
$$
where  $f_{m}^{(s)} \sim q(f_{m}|f_{m - 1}^{(s)},I_{m},\boldsymbol {w})$ . Based on these formulations, we provide some examples of generated scripts. Given the seed, the model first predicts the first frame  $f_{1}$ , then it predicts the next verb  $v_{1}$  and similarly samples the tokens one-by-one. During the event generations, if the sampled token is unknown, the decoder samples again. As we see in Table 7, the generated events and frames samples are consistent which shows the ability of model in event representation.
D.4 Inferred Frames
Using Alg. 1, we can see the frames sampled during the training and validation. In Table 8, we provide some examples of frame inferring for both training and validation examples. We observe that for training examples when  $\epsilon > 0.5$ , almost all the observed frames and inferred frames are the same. In other words, the model prediction is almost the same as the ground truth.
Interestingly, the model is more flexible in sampling the latent frames (orange ones). In Table 8a the model is estimating HAVE_ASSOCIATED instead of the POSSESSION frame. In Table 8b, instead of PROCESS_START we have ACTIV
Seed	Event 1	Event 2
elected taylor election at [CHANGE_OF_LEADERSHIP]	served she attorney as [ASSISTANCE]
graduated ford berkeley at [SCRUTINY]	earned he theology in [EARNINGS_AND_LOSSES]	sworn she house to [COMMITMENT]
released album 2008 in [RELEASING]	helped song lyrics none [ASSISTANCE]	documented he book in [RECORDING]
created county named and [INTENTIONALLY_CREATE]	totals district population of [AMOUNTING_TO]	made chart song with [CAUSATION]
published he December on [SUMMARIZING]	written book published and [TEXT Creation]	served district city none [ASSISTANCE]
played music show on [PERFORMERS_AND_ROLES]	starred film music none [PERFORMERS_AND_ROLES]	place book stories among [PLACING]
aired series canada in [EXPRESSING_PUBLICLY]	features series stories none [INCLUSION]	tend lyrics music as [LIKELIHOOD]
consists band members of [INCLUSION]	belong music party to [MEMBERSHIP]	abused characters film in [ABUSING]
ITY_START and finally in Table 8d we have TAKING_SIDES rather than SUPPORTING.
Some wrong predictions like CATASTROPHE instead of CAUSATION in Table 8c can be considered as the effect of other words like "pressure" in "resulted pressure revolution in". In some cases like Table 8b the predicted frame BEING_NAMED is a better choice than the ground truth APPOINTING. In the same vein FINISH_COMPETITION is a better option rather than GETTING in Table 8f.
D.5 Clustering
Here we provide more examples for  $\beta_{\mathrm{enc}}$  and  $\beta_{\mathrm{dec}}$  in Table 9. Our experiments show that by sorting the tokens in each frame, the first 20 words are mostly verbs. And among different token types, verbs are better classifiers for frames.
Table 7: Generated scripts and the inferred frames (in brackets) from the seed event in boldface  $\left( {\epsilon  = {0.7}}\right)$
Event	Ground Truth	Inferred Frame
1	estimated product prices in	ESTIMATING	ESTIMATING
2	rose which billions to	CHANGE_POSITION_ON_A Scale	CHANGE_POSITION_ON_A Scale
3	had maharashtra per as of	POSSSESSION	POSSSESSION
4	houses mumbai headquarters none	BUILDINGS	BUILDINGS
5	have % offices none	POSSSESSION	HAVE_ASSOCIATED
(a) Example 1 (training)
Event	Ground Truth	Inferred Frame
1	competed she olympics in	WIN_PRIZE	WIN_PRIZE
2	set she championships at	CAUSE_TO_START	CAUSE_TO_START
3	named she <unk> in	APPOINTING	BEING Named
4	started she had but	PROCESS_START	ACTIVITY_START
5	had she height because_of	POSSESSION	POSSESSION
(b) Example 2 (training)
Event	Ground Truth	Inferred Frame
1	arose that revolution after	COMING_TO_BE	COMING_TO_BE
2	supported who charter none	TAKING_SIDES	TAKING_SIDES
3	developed pressure classes from	PROGRESS	PROGRESS
4	remained monarchy run and	STATE_CONTINUE	STATE_CONTINUE
5	resulted pressure revolution in	CAUSATION	CATASTrophe
(c) Example 3 (training)
Event	Ground Truth	Inferred Frame
1	features it form in	INCLUSION	INCLUSION
2	allows format provides and	PERMITTING	DENY_PERMISSION
3	provides format forum none	SUPPLY	SUPPLY
4	rely readers staff upon	RELIANCE	RELIANCE
5	support citations assertions none	SUPPORTING	TAKING_SIDES
(d) Example 4 (validation)
Event	Ground Truth	Inferred Frame
1	attended he university none	ATTENDING	ATTENDING
2	moved he 1946 in	MOTION	TRAVEL
3	married he yan in	PERSONAL_RELATIONSHIP	FORMING_RELATIONSHIPS
4	worked they moved and	BEING_EMPLOYED	BEING_EMPLOYED
5	moved they beijing to	MOTION	MOTION
(e) Example 5 (validation)
Event	Ground Truth	Inferred Frame
1	had he achievements none	POSSSESSION	HAVE_ASSOCIATED
2	competed he finished and	FINISH_COMPETITION	REQUIRED_EVENT
3	finished he relay in	PROCESS_COMPLETED_STATE	ACTIVITY_DONE_STATE
4	set he three between	INTENTIONALLYCreate	INTENTIONALLY CREATE
5	won he championships at	GETTING	FINISH_COMPETITION
(f) Example 6 (validation)
Table 8: Sequences of 5 events and the inferred frames during training with partial frame observation, and validation without any observation. Blue frames are observed and orange frames are latent.
Verbs	βenc(Frames Given Verbs)
executed	EXECUTION	VERDICT	HIRING	RITE	BEING_OPERATIONAL
start	ACTIVITY_START	PROCESS_START	DURATION_RELATION	ACTIVITY_DONE_STATE	RELATIVE_TIME
fought	HOSTILE_ENCOUNTER	PARTICIPATION	SUCCEED_OR_FAILURE	QUITTING	DEATH
worked	BEING_EMPLOYED	WORKING_ON	HIRING	BEING_OPERATIONAL	EDUCATION_TOACHING
made	CAUSATION	MANUFACTURING	INTENTIONALLYCreate	CAUSE_CHANGES	CREATING
dominated	DOMINATE_SITUATION	CONTROL	BOUNDARY	SEPARATING	INVADING
created	INTENTIONALLY_CREATE	CREATING	CAUSE_CHANGES	COMING_TO_BE	CAUSATION
houses	BUILDINGS	LOCATIVE_RELATION	RESIDENCE	COMMERCEBuy	CAUSE_EXPANSION
see	PERCEPTION_EXPERIENCE	GRASP	EVIDENCE	SIMILARITY	EXPERIENCECUBJ
start	ACTIVITY_START	Process_start	Secrecy_status	Sign_agreement	Activity_finish
gave	GIVING	GETTING	BRANDING	SUPPLY	NEEDING
passed	TRAVERSING	UNDERGO_CHANGES	CONTROL	CHANGE_OF_LEADERSHIP	ARRIVING
sold	COMMERCESELL	COMMERCE-buy	MANUFACTURING	PREVENT_FROM_HAVING	RELEASENG
rose	MOTION_DIRECTIONAL	CHANGE_POSITON_ON_A Scale	GETTING_UP	BOUNDARY	EXPANSION
returned	ARRIVING	TRAVERSING	INTENTIONALLY_AFFECT	HAVE_ASSOCIATED	GETTING
enlisted	BECOMING_A Member	COLLABORATION	APPOINTING	COTHEME	COME_TOGETHER
met	COME_TOGETHER	MEET_WITH	MAKE_ACQUANTANCE	DISCUSSION	BOUNDARY
entered	ACTIVITY_START	ARRIVING	PROCESS_START	UNDERGO_CHANGES	MASS_MOTION
obtained	GETTING	COMMERCE BUY	GIVING	CONQUERING	EARNINGS_AND_LOSSES
chosen	CHOOSING	VOLUBILITY	LEADERSHIP	BE_SUBSET_OF	COMING_UP_WITH
Frames	βdec(Tokens Given FRAMES)
BECOMING	become	becomes	became	turned	goes
SHOOT_PROJECTIONLES	launched	fired	shoots	shoot	rocket
JUDGMENTCOMMUNICATION	criticized	charged	criticised	praised	accused
SHOOT_PROJECTIONLES	launched	fired	shoots	shoot	rocket
JUDGMENT COMMUNICATION	criticized	charged	criticised	praised	accused
BEING_NAMED	named	entitled	known	called	honour
ASSISTANCE	serve	serves	served	helps	assisted
CHANGE_OF_LEADERSHIP	elected	elects	councillors	installed	overthrow
REVEAL_secret	exposed	revealed	disclosed	reveal	confessed
TEXT creation	composed	authored	composes	drafted	written
EXPRESSIONPublicLY	aired	voiced	expresses	express	airs
ARREST	arrested	apprehended	booked	jailed	re-arrested
COLLABORATION	cooperate	collaborated	teamed	partnered	cooperated
WIN_PRIZE	competing	compete	competes	competed	mid-1950s
LEADERSHIP	governs	administered	administrers	presides	ruled
INCLUSION	contains	included	includes	include	excluded
SELF_MOTION	sailed	walks	ran	danced	walk
INGESTION	eaten	drank	drink	consumed	eat
TAKING_SIDES	supported	sided	supporting	endorsed	endorses
Table 9: Results for the outputs of the attention layer, the upper table shows the  $\beta_{\mathrm{enc}}$  and the bottom table shows the  $\beta_{\mathrm{dec}}$ , when  $\epsilon = 0.7$ . Each row shows the top 5 words for each clustering.