# A Topical Category-Aware Neural Text Summarizer
## 5 Experiments
### Experimental Results  
Figure 3 shows the ROUGE-1 change according to \(\varepsilon\) values in Equation (8). When \(\varepsilon\) is zero, CAM is not reflected at all into attention weights. Then, the proposed model becomes equivalent to the attn baseline model. When \(\varepsilon\) increases from 0 to 0.4, the ROUGE-1 score increases monotonically. On theother hand, when \(\varepsilon\) is larger than 0.4, the score decreases steeply. This implies that the effect of CAM is maximized at \(\varepsilon=0.4\). Thus, all experiments below are performed with \(\varepsilon=0.4\). Table 4 shows that the proposed model outperforms the two baselines. The seq-baseline achieves 39.6703 of ROUGE-1, 17.4243 of ROUGE-2, and 31.0626 of ROUGE-L and the attn-baseline achieves 40.3402, 23.1140, and 36.2059, respectively. The attn-baseline shows 0.6699, 5.6897, and 5.1433 higher performances than the seq-baseline. This is because the attn-baseline uses different context vectors for every position of word generation while the seq-baseline clips to an identical one. On the other hand, the proposed model achieves 43.2400 of ROUGE-1, 24.6724 of ROUGE-2, and 37.8523 of ROUGE-L, which is 2.8998, 1.5584, and 1.6464 higher than the attn-baseline. The main difference between the attn-baseline and the proposed model is whether or not CAM is reflected into the attention, and ROUGE measures the overlaps between the summary written by a human being and the summary generated by a machine. This implies that the reflection of topical information to the attention helps generate a summary that is closer to the human-made summary. Another thing to note is that the performance difference between the proposed model and the attn-baseline in ROUGE-1 is larger than those in ROUGE-2 and ROUGE-L. This is because CAM assigns its attention at a word level. Since CAM is originated from computer vision, it regards a word as a pixel in an image. That is, it focuses on the topic-related words independently. As a result, the surrounding words of a topic-related word are not paid sufficient attention to by CAM. There are several previous studies that adopted the NYT corpus for learning their own summarization model. Table 5 compares the proposed model with the previous models by [PERSON] et al. [6], and by [PERSON] et al. [33]. It shows the result obtained by training each model using the same data as the proposed model. [PERSON] et al. achieved 42.53 of ROUGE-1, 24.62 of ROUGE-2, and 36.18 of ROUGE-L and [PERSON] et al. achieved 43.01, 25.12, and 36.93, respectively. The proposed model shows the highest scores in all metrics except ROUGE-2. Thus, this table proves that the proposed model generates high-quality summaries. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**ROUGE**} \\ \cline{2-4}  & **1** & **2** & **L** \\ \hline Seq-Baseline & 39.6703 & 17.4243 & 31.0626 \\ Attn-Baseline & 40.3402 & 23.1140 & 36.2059 \\ Proposed Model & **43.2400** & **24.6724** & **37.8523** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: ROUGE score on the test set. Figure 3: The change of ROUGE-1 scores according to \(\varepsilon\) on the validation set. Figure 4 depicts the attention heatmaps between a source text and a generated summary, where Figure 4a is generated by the proposed model and 4b is generated by the attn-baseline. The X-axis of the heatmap represents the tokens of a source text and the Y-axis is the tokens of a summary. The closer the color of the heatmap is to yellow, the higher the attention weight is. Thus, the color of purple implies low attention weight. When comparing the token \"department\", the second token of the source text in both figures, its color in Figure 4a is nearer to yellow than that in Figure 4b, which implies that the importance of \"department\" increases by CAM when the first token is generated. Since the category of the source text is \"Politics and Government\", CAM enhances its attention to the token \"department\".