# Document-Level Event Role Filler Extraction Using Key-Value Memory Network
## 3 Methods
### Document-Level Representation  
Document-level encoders mainly use key-value memory networks. Memory networks were first proposed in 2014 by [PERSON] and [PERSON] et al. within the field of QA [12]. After which, [PERSON] et al. in 16 further proposed the key-value memory network model. This was later applied to the NER task by [PERSON] et al. [23], significantly improving the overall performance. In order to address the problem wherein BiLSTM cannot make good use of document-level contextual information, this paper also borrows an idea from [PERSON] et al. and uses a key-value memory network to memorize the document-level contextual representation.  
If there are m slots, then \((\text{K}_{1},\text{V}_{1}),\ldots,(\text{K}_{\text{m}},\text{V}_{\text{m}})\). In each vector pair, we take the word-vector output from the pre-trained language as the K and the hidden layer state \(h_{i}\) corresponding to each instance of the BiLSTM encoder output as the value. The updated k-v are used as k and v, respectively, in the attention mechanism, and then the document-level contextual representation is calculated according to the formula.  
Memory update: Since each word may appear multiple times in the same article, it will appear to occupy multiple slots. This requires a constant update of the key-value: the word embedding updates the corresponding k, and the hidden layer state \(h_{i}\) updates the v. For example, the i-th word \(X_{i}\) may occupy more than one slot. After the calculation, the value of the next slot will be different from the current value, and then this slot value will be rewritten in order for the slot value to be updated when the same word is encountered.  
Memory query: For a word i in a sentence, the model extracts the position of the word in all sentences and forms a subset of size T by reverse indexing: \(\big{(}k_{sub_{1}},v_{sub_{1}}\big{)},\ldots,\big{(}k_{sub_{T}},v_{sub_{T}} \big{)}\). Information about the location of each word in the memory network is recorded to the index set. The reverse index set records information about the position of each word in the memory network. Words appear in the article as many times as T.  
The key-value memory network calculates the weights of the document-level representations mainly through an attention mechanism. For each word, it uses \(k_{j}\in\big{[}k_{sub_{1}};\ldots;k_{sub_{T}}\big{]}\) as attention keys and \(v_{j}\in\big{[}v_{sub_{1}};\ldots;v_{sub_{T}}\big{]}\) as attention values. The embedding \(Wq_{i}\) of the query term is then used as the attention query \(q_{i}\). In this paper, we attempt the following three different compatibility functions to explore their impacts on document-level information \(u_{ij}=o\big{(}q_{i},k_{j}\big{)}\):  
(1) Dot-product attention  
\[\text{o}_{1}\big{(}q_{i},k_{j}\big{)}=q_{i}k_{j}^{T}. \tag{4}\](2) Scaled dot-product attention [24]:  
\[o_{2}\big{(}q_{i},k_{j}\big{)}=\frac{q_{i}k_{j}^{T}}{\sqrt{d_{w}}}. \tag{5}\]  
The \(d_{w}\) denotes the dimension of the word embedding and T denotes the number of occurrences of this word in the training set.  
(3) Cosine similarity  
\[o_{3}\big{(}q_{i},k_{j}\big{)}=\frac{q_{i}k_{j}^{T}}{\|q_{i}\|\|k_{j}\|}. \tag{6}\]  
The document-level representation calculation formula is:  
\[\begin{split}&\alpha_{\text{ij}}=\frac{\exp\big{(}u_{ij}\big{)}}{ \sum_{z=1}^{T}\exp\big{(}u_{ij}\big{)}}\\ & r_{i}=\sum_{j=1}^{T}\alpha_{ij}v_{j}\end{split} \tag{7}\]