# DEEDP: Document-Level Event Extraction Model Incorporating Dependency Paths
## 4 Methodology
### Sentence-Document Feature Fusion  
At this stage, we introduced a novel network to enhance document-aware representations for entities by integrating the sentence-document features. We stacked an advanced Transformer encoder, named _Transformer-M_, and Transformer encoder, _TF-3_, together to achieve this goal, where _Transformer-M_ helped to extract more sentence-level features and the _TF-3_ was applied to integrate different kind features. Figure 2 depicts the novel network architecture. **Transformer-M**. To explicitly model the syntactic features and aggregate sentence-aware and document-aware representations for each entity, we designed a novel encoder based on Transformer, named _Transformer-M_ (_TF-M_). Figure 4d depicts the network architecture of _TF-M_. We added a Syntactic Feature Attention mechanism, _SF-ATT_, to Transformer, which was used to capture the syntactic features of the input sentences and enhance the representations for each token contained in the sentences. In addition, we concatenated the embeddings of each token used in _TF-1_ and the document-level representation of \(S_{1}\) as the input of _TF-M_, which helped us to make full use of the information of both the document-level and local sentence to model the long dependency. Specifically, the input embedding of the _j_th token can be calculated as:  
\[w_{i,j}^{{}^{\prime}}=\big{[}h_{i}^{s};w_{i,j}\big{]} \tag{7}\]  
where \(w_{i,j}\in R^{d}\) is the \(j^{th}\) token input embedding used in _TF-1_; \(h_{i}^{s}\in R^{d}\) is the document-level representation for \(S_{i}\); and \(w_{i,j}^{{}^{\prime}}\in R^{2d}\) represents the input embedding vector of the \(j^{th}\) token. token for \(S_{i}\), which is fed into TF-M to obtain feature enhanced representation. \([\cdot\ ;\ \cdot]\) denotes the concatenation operation. As shown in Figure 4a, for the given document and sentences \(\mathcal{D}=\{S_{i}\}_{i=1}^{N_{i}}\), we first applied the Spacy tool to construct the dependency relation for each token \(w_{i,j}\in S_{i}\) and its parent. We were then able to obtain the dependency paths and the link matrix could be constructed with them. Subsequently, we were able to construct a syntactic feature mask matrix for the input sentence, which was used in _TF-M_. To be specific, in the _i_th layer of _TF-M_, the Syntactic Feature Attention and Multi-head Attention (_MH-ATT_) hidden states were taken together to enhance the representations for each token in the input sentence; the former mainly focuses on the syntactic relations of \(w_{i,j}\in S_{i}\) and its parent, and the latter is responsible for modelling the whole lexical and semantic features for all tokens in the sentence, which can be formalized as follows:  
\[z_{i,j,1}^{(k)}=MH-ATT\Big{(}h_{i,1}^{(k-1)},\cdots,h_{i,l}^{(k-1)}\Big{)} \tag{8}\]  
\[z_{i,j,2}^{(k)}=SF-ATT\Big{(}h_{i,1}^{(k-1)},\cdots,h_{i,l}^{(k-1)}\Big{)} \tag{9}\]  
\[z_{i,j}^{(k)}=z_{i,j,1}^{(k)}+z_{i,j,2}^{(k)} \tag{10}\]  
where \(z_{i,j,1}^{(k)}\in R^{d}\) and \(z_{i,j,2}^{(k)}\in R^{d}\) denote the hidden state calculated after the _k_th layer of MH-ATT and SF-ATT for the _j_th token of \(S_{i}\), respectively. The variables \(z_{i,j}^{(k)}\in R^{d}\) denote the hidden state calculated with the _k_th layer. Then \(z_{i,j,1}^{(k)}\) and \(z_{i,j,2}^{(k)}\). \(z_{i,j}^{(k)}\) were integrated with the hidden state calculated by the MH-ATT to enhance the representation for the token \(w_{i,j}\). The inputs were fed into _SF-ATT_ to capture their explicit syntactic features, and the calculation can be formalized as follows:  
\[z_{i}=\sum_{j=1}^{l}\alpha_{i,j}\cdot(w_{j}\cdot W_{v}) \tag{11}\]  
\[\alpha_{i,j}=\frac{exp\big{(}a_{i,j}\big{)}}{\sqrt{d}\cdot\sum_{k=1}^{l}exp \big{(}a_{i,j}\big{)}} \tag{12}\]  
Figure 4: The overall workflow of Syntactic Feature Mask matrix construction and the architecture of _Transformer-M_ encoder.