# Document-Level Event Role Filler Extraction Using Key-Value Memory Network
## 4 Experiments
### Experimental Settings  
We compared all other models on the same task and dataset. (1) GLACTER [1] is composed of a sentence classifier and a set of per-event theoretical elements initially populated with templates, and its results are obtained from the probability product of normalized sentences and phrases. (2) TIER [2] proposes a multi-stage approach. The processing is divided into three stages: the classification of event documents, event sentence recognition, and noun phrase analysis. \begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline \hline
**Models** & \multicolumn{3}{c}{**Head Noun Match**} & \multicolumn{3}{c}{**Exact Match**} \\ \hline  & Prec. & Recall & F-1 & Prec. & Recall & F-1 \\ GLACTER & 47.80 & 57.20 & 52.08 & - & - & - & - \\ TIER & 50.80 & 61.40 & 55.60 & - & - & - & - \\ Cohesion Extract & 57.80 & 59.40 & 58.59 & - & - & - & - \\ Multi-Granularity Reader & 56.44 & 62.77 & 59.44 & 52.03 & 56.81 & 54.32 \\ KDR & 56.20 & **64.83** & **60.20** & 52.91 & 57.63 & **55.16** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Macro averaging results for a document-level event extraction task. \begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline \hline
**Models** & \multicolumn{3}{c}{**PerpInd**} & \multicolumn{3}{c}{**PerpOrg**} & \multicolumn{3}{c}{**Target**} & \multicolumn{3}{c}{**Victim**} & \multicolumn{3}{c}{**Weapon**} \\ \hline  & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\ GLACTER & 51 & 58 & 54 & 34 & 45 & 38 & 42 & 72 & 53 & 55 & 58 & 56 & 57 & 53 & 55 \\ TIER & 52 & 58 & 55 & 55 & 48 & 51 & 55 & 61 & 58 & 63 & 59 & 61 & 62 & 64 & 63 \\ Cohesion Extract & 54 & 57 & 56 & 54 & 49 & 51 & 55 & 68 & 61 & 63 & 59 & 61 & 62 & 64 & 63 \\ Multi-Granularity Reader & 53 & 52 & 52 & 51 & 68 & 58 & 60 & 64 & 62 & 49 & 62 & 55 & 68 & 67 & 68 \\ KDR & 54 & 51 & 52 & 56 & 65 & **60** & 58 & 61 & 59 & 47 & 53 & 50 & 69 & 66 & **68** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The results for each event role based on the head-noun matching metric. (3) Cohesion Extract [25] uses a bottom-up approach to first identify candidate-fillable theorems in a document and then refine the set of candidate theorems using a cohesive sentence classifier. (4) Multi-Granularity Reader [6] is a multi-granularity model architecture that explores the effects of different sentence lengths on the model results and demonstrates that the model results grow and then decline with sentence length. We also use the end-to-end neural sequence model key-value memory network as the model for the document-level part--with the word embedding in the sentence level as the key value k and the corresponding hidden layer state \(h_{i}\) as the value--and then calculate the document-level contextual information through the attention mechanism, which incorporates the sentence-level information to a greater extent. The last row of Table 1 shows a significant improvement in the overall results. We use two evaluation results with the macro average results in Table 1. For a detailed understanding of how the model extracts filling results for each event argument, the results for each event role are presented in Table 2. From these two tables, it can be seen that the multi-level end-to-end sequence model can achieve almost the same results as the pipeline model. In the results for the document-level model, it is clear that the effects of the key-value memory network are better than the results of splicing after direct multi-sentence input. We used two evaluation approaches, and the macroscopic average results are shown in Table 1. From both evaluation approaches, on the MUC-4 dataset and with the same end-to-end neural sequence model, the model with the key-value memory network was able to achieve nearly 61 percent effectiveness--significantly higher than the multi-granularity reader model for the same task.