# Comprehensive Document Summarization with Refined Self-Matching Mechanism
## 5 Results
### Discussion  
Throughout the ablation testing process, the self-matching mechanism contributed the most to the model, to further explore the influence of self-matching. In this part, we discuss different level document encoding strategies, that is, the impact of document information representation for summarization task. We have implemented the unidirectional LSTM with a simple linear projection based classifier, namely \"UniLSTM + Classifier\", and the bidirectional LSTM with classifier (as the basic model), namely \"BiLSTM + Classifier\". The bidirectional LSTM with self-matching mechanism, namely \"BiLSTM + Self Matching\", to further study how the richness of global information work and affect the document information encoding.  
As shown in Table 3, performance on the bidirectional LSTM based model were better than the model with single LSTM, since bidirectional LSTM can encode documents from both forward and backward, that can include more document information. On the contrary, unidirectional LSTM may lose numerous effective features due to memory problems. After the addition of the self-matching mechanism, the performance improved consistently on three indicators, which gives a clear direction to our future work, that is, to improve the global information richness of the document encoding.  
From previous Table 1, it can be seen that BERT significantly improved the overall performance, even surpassed the contribution of any single component. To study whether the strength of pre-trained knowledge would cover the effect of RSME, referring to Figure 1, we select GloVe [34] or BERT in the embedding layer, and remove the refined self-matching layer, and kept the rest unchanged, forming GloVe-basic and BERT-basic. The experimental results in Table 4 show that the proposed RSME has a promising improvement on the baseline of BERT and GloVe.We found that architectures with context-independent GloVe made little contribution to the current models, while models equipped with BERT are improved with a large margin, which explains that RSME is irreplaceable in comprehensive document information extraction.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Models** & **ROUGE-1** & **ROUGE-2** & **ROUGE-L** \\ \hline RSME & 41.5 & 18.8 & 37.7 \\ GloVe-basic & 40.4 & 18.1 & 36.7 \\ GloVe-RSME & 41.4 & 18.7 & 37.8 \\ BERT-basic & 42.0 & 19.3 & 38.5 \\ BERT-RSME & 42.4 & 19.8 & 38.9 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Analysis of pre-trained knowledge on experimental results.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Models** & **ROUGE-1** & **ROUGE-2** & **ROUGE-L** \\ \hline RSME & 41.5 & 18.8 & 37.7 \\ -Gaussian bias & 41.1 & 18.6 & 37.7 \\ -Self Matching & 40.4 & 18.2 & 37.0 \\ -Pointer Network & 40.2 & 18.0 & 36.5 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation test for RSME, “-” means removing the corresponding component based the previous system. In short, “-Pointer Network” represents a model contains only a document encoder and a simple classifier.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Models** & **ROUGE-1** & **ROUGE-2** & **ROUGE-L** \\ \hline UniLSTM + Classifier & 39.8 & 17.9 & 36.4 \\ BiLSTM + Classifier & 40.2 & 18.0 & 36.5 \\ BiLSTM + Self Matching & 40.9 & 18.3 & 37.6 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Analysis of the effect of encoding information richness on experimental results.