# DEEDP: Document-Level Event Extraction Model Incorporating Dependency Paths
## 4 Methodology
### Sentence-Level Encoder  
**Transformer Encoder**. Following most pre-training language models, such as BERT [29], MBERT [30], MLRIP [31], SpanBERT [32], and document-level event and event element joint extraction models such as Doc2 EDAG [8] and MMR [16], we applied Transformer [14] as our basic encoder, which utilizes a multi-head attention and masking strategy to capture the lexical and syntactic information for each token in the input sequence, and the corresponding contextual representation for each token can be generated.  
**Input Embeddings**. Modelling more features of the input tokens facilitates downstream tasks. Compared with Doc2 EDAG, MMR, and DEEB-RNN [33], we modeled a part-of-speech (POS) feature, entity type feature, and document topic feature to represent each token of the input sentence, rather than using only the token and position features. Formally, given a document \(\mathcal{D}=\left\{S_{i}\right\}_{i=1}^{N_{\text{s}}}\) containing \(N_{\text{s}}\) sentences, sentence \(S_{i}\) can be denoted as a sequence of tokens \([w_{i,1},w_{i,2},\cdots,w_{i,l}]\), where \(l\) is the length of the sentence, and \(w_{i,j}\in R^{d}\) denotes the representation for the \(j\)th token of the \(S_{i}\). The representations of the token sequence were fed into the Transformer Encoder, _TF_-1, and the final hidden state \(C_{i}\) of the \(S_{i}\) could be used for down-stream tasks, which can be calculated as follows:  
\[C_{i}=TF-1(w_{i,1},\cdots,w_{i,l}) \tag{1}\]  
where \(C_{i}=[c_{i,1},c_{i,2},\cdots,c_{i,l}]\), \(c_{i,j}\in R^{d}\) is the final hidden state for the \(j\)th token of \(S_{i}\) and \(C_{i}\in R^{l\times d}\) is the final hidden state of \(S_{i}\). Then \(C_{i}\) was fed into a CRF layer for sequence labelling. After that, we could obtain the candidate event arguments.  
Given a token, its input representation is the sum of the embeddings for token, segment, position, POS, entity type, and document topic. A visualization of the input embedding is illustrated in Figure 3.  
**Entity Recognition**. Entity recognition aims to extract all candidate elements for the events described in a given document and is always taken as a sequence tagging task. In this proposed work, we performed entity recognition at the sentence level and followed GIT and Doc2 EDAG, which added a CRF layer to the final Transformer Encoder to conduct sequence tagging for the input sentences. To be specific, given a sentence s, we applied a Transformer encoder (_TF_-1) to encode it and the final hidden states \(C_{i}=TF-1(s)\) were  
Figure 2: DEEDP model overview. For a given document, we first conduct NER with the sentence-level encoder. Then we feed the sentence-aware representations of entities and sentences of the input document to the document-level encoder to acquire their document-aware representations. To aggregate both representations for each entity and enhance their semantic, we designed a Transformer-M and stacked it with a Transformer. Finally, the fused representations of entities and document-aware representations of sentences are fed into the event record extraction module for event record construction.  
fed into the CRF layer to conduct entity recognition with the BIO (Begin, Inside, Other) schema. For training, we minimized the sequence tagging loss as follows:  
\[\mathcal{L}(ner)=-\sum_{s\in\mathcal{D}}\log(P(y_{s}|s)) \tag{2}\]  
where \(y_{s}\) is the golden-label tagging sequence for \(s\). With respect to inference, we applied the Veterbi decoding algorithm to obtain the golden label sequence.