SlowGuess/NAACL_2021_OCR huggingface dataset:
Naushad UzZaman, Hector Llorens, Leon Derczynski, James Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 1-9.
Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. 2019. Fine-grained temporal relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2906-2919, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Thomas Wolf, L Debut, V Sanh, J Chaumont, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Jiaxuan You, Rex Ying, Xiang Ren, William L Hamilton, and Jure Leskovec. 2018. Graphrn: Generating realistic graphs with deep auto-regressive models. arXiv preprint arXiv:1802.08773.
A Learning Event Communities Using Community Detection
In this section, we provide the details on the community detection algorithm used by our method. We define the temporal event communities to be a division of the temporal graph  $\mathbf{G}(\mathbf{V},\mathbf{E})$  into sub-graphs  $\mathbf{G}_1(\mathbf{V}_1,\mathbf{E}_1),\mathbf{G}_2(\mathbf{V}_2,\mathbf{E}_2),\dots,\mathbf{G}_k(\mathbf{V}_k,\mathbf{E}_k)$  such that the events in a community (sub-graph)  $\mathbf{G}_{\mathrm{i}}$  are more co-referential to each other as opposed to the other events in the temporal graph. We use the undirected link between two events  $e_j,e_i$  as a proxy for them being co-referential, and learn temporal event communities utilizing the concept of modularity, first introduced by (Newman and Girvan, 2004).
Formally, let  $\mathbf{A}$  be the undirected adjacency matrix for a temporal graph  $\mathbf{G}(\mathbf{V},\mathbf{E})$  such that  $\mathbf{A}(e_i,e_j) = 1$  if  $e_i$  and  $e_j$  are connected by a temporal relation, and 0 otherwise. Further, let  $\delta (e_i,e_j) = 1$  if events  $e_i,e_j$  belong to the same temporal community, and 0 otherwise. For a given  $\delta$ , we denote the fraction of the edges that exist between events that belong to the same communities by  $f_{same} = \frac{\sum_{e_i,e_j\in\mathbf{E}}\mathbf{A}(e_i,e_j)\delta(e_i,e_j)}{2|\mathbf{E}|}$ . Where the  $2|\mathbf{E}|$  in the denominator is due to the fact that  $\mathbf{A}$  treats  $\mathbf{G}$  as an undirected graph. Let the popularity  $p$  of an event  $e_i$  be the number of events that are linked to it i.e.  $p(e_i) = \sum_{e_j\in \mathbf{E}}\mathbf{A}(e_i,e_j)$ . The probability of randomly picking an event  $e_i$  when sampled by popularity is  $\frac{p(e_i)}{\sum_{e_j\in\mathbf{E}}p(e_i)} = \frac{p(e_i)}{2|\mathbf{E}|}$ . Thus, if edges are created randomly by sampling nodes by popularity  $p$  of the nodes, the fraction of edges within the communities,  $f_{random}$ , is given by
$$
f _ {r a n d o m} = \frac {\sum_ {e _ {i} , e _ {j} \in \mathbf {E}} p (e _ {i}) p (e _ {j}) \delta (e _ {i} , e _ {j})}{2 | \mathbf {E} | * 2 | \mathbf {E} |}
$$
Finally, defining modularity,  $Q$ , to be  $f_{same} - f_{random}$ :
$$
Q = \frac {1}{2 | \mathbf {E} |} * \sum_ {e _ {i}, e _ {j} \in \mathbf {E}} \mathbf {A} (e _ {i}, e _ {j}) - \frac {p (e _ {i}) p (e _ {j}) \delta (e _ {i} , e _ {j})}{2 | \mathbf {E} |}
$$
We want to learn community assignments  $\delta$  that maximize  $Q$ . A high  $Q$  would promote  $f_{same} > f_{random}$  and thereby encourage highly inter-connected event communities. Calculating such  $\delta$  directly is not tractable, since the complexity of such an operation would be at least exponential
in the number of events (Newman, 2004). We use the fast implementation provided by (Clauset et al., 2004) for calculating event communities iteratively. The algorithm converges at  $Q$  0.3. We use a similar approximation at test time: given a document  $D$ , we first break it down into sub-documents using CAEVO and then feed each sub-document to our method.
B Using a smaller block size
We found that the performance drops when using a block size of 300 and batch size of 2. Table 10 presents the results.
BLEU	MTR	RG	DOT%
25.01	27.95	60.99	91.71
vP	vR	VF1	eP	eR	EF1
70.31	64.75	65.68	29.43	24.83	24.27
Table 10: Results for TG-Gen using a block size of 300 and a block size of 2.
C Masked Language Modeling Using Transformers
In this section, we expand on the design of the transformer blocks. For ease of reference, we reiterate our training methodology. We train a (separate) conditional language model to solve both the tasks. Specifically, given a training corpus of the form  $\{(x_i,y_i)\}$ , we aim to estimate the distribution  $p_{\theta}(\boldsymbol{y}_i \mid \boldsymbol{x}_i)$ . Given a training example  $(x_i,y_i)$  we set  $\boldsymbol{u}_i = \boldsymbol{x}_i \| \boldsymbol{y}_i^8$ .  $p_{\theta}(\boldsymbol{u}_i)$  can then be factorized as a sequence of auto-regressive conditional probabilities using the chain rule:  $p_{\theta}(\boldsymbol{u}_i) = \prod_{k=1}^{n} p(u_{i,k} | \boldsymbol{u}_{i, < k})$ , where  $u_{i,k}$  denotes the  $k^{th}$  token of the  $i^{th}$  sequence, and  $u_{i, < k}$  denotes the sequence of tokens  $\{u_1, u_2, \ldots, u_{k-1}\}$ . Language models are typically trained by minimizing a cross-entropy loss  $-\log p_{\theta}(\boldsymbol{u}_i)$  over each sequence  $\boldsymbol{u}_i$  in  $\mathbf{X}$ . However, the cross-entropy loss captures the joint distribution  $p_{\theta}(\boldsymbol{x}_i, \boldsymbol{y}_i)$ , and is not aligned with our goal of learning conditional distribution  $p_{\theta}(\boldsymbol{y}_i | \boldsymbol{x}_i)$ . To circumvent this, we train our model by masking the loss terms corresponding to the input  $\boldsymbol{x}_i$ , similar to Bosselut et al. (2019). Let  $\boldsymbol{m}_i$  be a mask vector for each sequence  $\boldsymbol{u}_i$ , set to 0 for positions corresponding to  $\boldsymbol{x}_i$ , and 1 otherwise
${}^{8}\parallel$  denotes concatenation
Figure 3: Event temporal graph and the extracted communities for a sample document. Each community is shown in different color. The singleton nodes (gray) are dropped. The nodes are only annotated with the verbs for brevity. The edge labels and directions are not used for community detection.
i.e.  $m_{i,j} = 1$  if  $j > |\pmb{x}_i|$ , else 0. We combine the mask vector with our factorization of  $p_\theta(\pmb{u}_i)$  to formulate a masked language modeling loss, which is minimized over the training corpus  $\mathbf{X}$  to estimate the optimal  $\theta$ :
$$
\mathcal {L} _ {\text {m a s k e d}} (\mathbf {X}) = - \sum_ {i = 1} ^ {| \mathbf {X} |} \sum_ {j = 1} ^ {| x _ {i} | + | y _ {i} |} m _ {i, j} * \log \left(p _ {\theta} \left(u _ {i, j} \mid \boldsymbol {u} _ {i, <   j}\right)\right)
$$
Note that the formulation of masked loss is opaque to the underlying architecture, and can be implemented with a simple change to the loss function. Intuitively, the model is optimized for only the output sequence  $y_{i}$ .
C.1 Adapting GPT-2 for Masked Language Modeling
In practice, we use GPT-2 (Radford et al., 2019) based on transformer architecture (Vaswani et al., 2017) for our implementation. An input sequence  $\pmb{u}_i$  of length  $n$  is first embedded to a continuous representation denoted by  $\pmb{u}_i^{(0)}\in \mathbb{R}^{nd}$ .  $\pmb{u}_i^{(0)}$  is then passed through a series of  $L$  transformer blocks to obtain the output sequence  $\pmb{u}_i^{(L)}\in \mathbb{R}^{nh}$ . Each transformer block (Vaswani et al., 2017) consists of two operations: an auto-regressive version of the multiheaded self-attention (Vaswani et al., 2017) operation (AutoRegMultiHead) followed by a feedforward operation (FFN). Each of these operations is surrounded by a residual connection (He et al., 2016) and followed by a layer normalization (Ba et al., 2016) operation. Denoting by  $\pmb{u}^{(l - 1)}$  the input to the  $l^{th}$  transformer block, the operations are
in a transformer block are defined as follows:
$$
\tilde {\boldsymbol {u}} _ {a t t n} ^ {l} = \operatorname {A u t o R e g M u l t i H e a d} \left(\boldsymbol {u} ^ {(l - 1)}\right)
$$
$$
\boldsymbol {u} _ {a t t} ^ {(l)} = \text {L a y e r N o r m} \left(\tilde {\boldsymbol {u}} _ {a t t} ^ {(l)} + \boldsymbol {u} ^ {(l - 1)}\right)
$$
$$
\tilde {\pmb {u}} _ {f f n} ^ {(l)} = F F N (\pmb {u} _ {a t t} ^ {(l)})
$$
$$
\pmb {u} ^ {(l)} = L a y e r N o r m (\tilde {\pmb {u}} _ {f f n} ^ {(l)} + \pmb {u} _ {a t t} ^ {(l)})
$$
Where AutoRegMultiHead is an auto-regressive version of the multiheaded self-attention (Vaswani et al., 2017) that restricts the attention to the sequence seen so far (in accordance with the chain rule), and  $FFN$  is a feed-forward network (MLP). After obtaining  $\pmb{u_i}^{(L)}$ , we set  $p_{\phi}(\pmb{u_i}) = \mathrm{softmax}(\pmb{u_i}^{(L)}*\mathbf{W}_e)$ , where  $\mathbf{W}_e\in \mathbb{R}^{h|V}(|V|$  is the size of the vocabulary). Finally, we calculate the masked loss as  $\mathcal{L}(\pmb{u_i}) = \pmb{m_i}^T\odot \log (p_{\phi}(\pmb{u_i}))$ , and the optimal  $\phi$  is obtained by minimizing  $\mathcal{L}_{\mathrm{masked}}(\mathbf{X}) = -\sum_{i = 1}^{|\mathbf{X}|}\mathcal{L}(\pmb{u_i})$ .
Tables 11, 12, and 13 list various statistics calculated from the source data.
Figures 4-9 show randomly picked examples from the test corpus. Each figure shows the text, the corresponding true graph, and the graph predicted by GPT-2.
They were parents and grandparents who thought they had fully grasped the perils facing their teenagers in the tough working-class streets north of Kennedy Airport in Queens. They said that they understood the power of peers, the lure of gangs and drugs, the impact of movies and music and television that with relentless repetition depicted a culture of casual violence. And, they said, they believed they had taken the necessary precautions. They gave their children beepers and cellular phones to keep better track of them. They lined up mentors, and set curfews, and encouraged faith.
Descriptor	#Articles
terrorism	40909
murders and attempted murders	25169
united states international relations	17761
united states armament and defense	16785
airlines and airplanes	16103
world trade center (nyc)	15145
demonstrations and riots	14477
hijacking	14472
politics and government	6270
bombs and explosives	5607
Table 11: Top Descriptors for the filtered Dataset. Note that each article is typically assigned more than one descriptor.
Event verb	Raw frequency	% Frequency
said	647685	9.60
say	57667	0.86
had	47320	0.70
killed	43369	0.64
told	42983	0.64
found	41733	0.62
made	40544	0.60
war	35257	0.52
get	30726	0.46
make	29407	0.44
Relation	Raw Frequency	% Frequency
BEFORE	2436201	54.51
AFTER	1772071	39.65
IS INCLUDED	131052	2.93
SIMULTANEOUS	112509	2.52
INCLUDES	17465	0.39
Table 13: Relation Frequency in our Corpus
Relation	Frequency
BEFORE	98715
AFTER	68582
IS INCLUDED	6179
SIMULTANEOUS	6209
INCLUDES	285
Table 14: Edges in Generated Graphs: Top
Table 12: Most frequent events extracted by CAEVO.
Iran , with its 65 million Shiite , its powerful army and its ancient civilization , is the de facto master of the Persian Gulf .Tehran is clearly pleased that Iraq ' s 15 million Shiite will more or less control their country eventually . In Lebanon , with one million Shiite , the well-armed Hezbollah militia has proved itself a most effective military-social-political group , which even forced both American and Israeli armed forces from the country . There are 400,000 Shiite in Bahrain and several million more in pockets from Pakistan to Saudi Arabia .
In the last month, Eastman Kodak, which has been shrinking for the last decade, announced plans to lay off an additional 6,000 to 9,000 workers. The Rochester police vowed to step up efforts after it reported a homicide rate that was the worst in years. Candidates competing for the region's top offices are holding regular news conferences to criticize one another.
He dreams of being a biochemist and revels in science fiction novels, insect collecting and dismantling broken electronic equipment. He acknowledges that his actions were potentially dangerous. 'People can't see the difference between showing off, testing the boundaries and killing people,' he said. 'I went into the school to see what I could get away with, not because I wanted to kill anyone.' 'Michael said the most difficult part of his punishment was the six weeks he spent at the Essex County Youth House in Newark. One cellmate, he said, beat him and tried to strangle him.
In a dispatch that Mr. Altgens wrote for the agency later that day, he said: There was a burst of noise, the second one I heard, and pieces of flesh appeared to fly from President Kennedy 's car. Blood covered the whole left side of his head. Mrs. Kennedy saw what had happened to her husband. She grabbed him, exclaiming, Oh, no! The Associated Press, in its book on the assassination, The Torch is Passed ... which was published soon afterward, republished Mr. Altgens 's photograph of the First Lady and the agent with a caption saying it shows Secret Service Agent Clint Hill leaping toward Mrs. Kennedy as she desperately moves for help in the first moment of horror.
An article yesterday about the release of an Egyptian chemist who had been held by Egyptian police as a suspect in the July 7 London transit bombings misstated the date he was arrested in Cairo. It was July 14, not June 14.