# Abstractive Summarizers Become Emotional on News Summarization
## 6 Emotions in Summarization Systems
### Emotions of Novel Words  
Abstractive summarizers are moderately good at generating summary-worthy novel words that are not present in the source. These novel words could convey a set of emotions. However, whether the emotions of the novel words are those expected in the reference summary is still unclear. We study it on the test sets of Cnn/Dailymail and XSum by computing the precision between the emotions of the novel words in a generated summary and all the emotions in the reference summary.  
Let \(\mathcal{E}_{s}\) be the set of emotions in a reference summary and \(\mathcal{E}_{s}\) the set of emotions of the novel words in a generated one, precision (P) for \(N\) samples is computed as shown in Equation (3); where the intersection refers to the emotions in common between those found in the novel words and the reference summary. We only consider those cases where there are novel words with emotions in the generated summary (\(|\mathcal{E}_{s}|>0\)) and the reference one has words with emotions (\(|\mathcal{E}_{s}|>0\)).  
\[\mathrm{P}=\frac{1}{N}\sum_{\forall(s,s)}\frac{|\mathcal{E}_{s}\cap\mathcal{E }_{s}|}{|\mathcal{E}_{s}|} \tag{3}\]  
We also compute the recall (R) to see how many of the emotions in the reference summary are covered by the emotions of the novel words in the generated summary. Recall is computed as shown in Equation (4).  
\[\mathrm{R}=\frac{1}{N}\sum_{\forall(s,s)}\frac{|\mathcal{E}_{s}\cap\mathcal{E }_{s}|}{|\mathcal{E}_{s}|} \tag{4}\]  
Table 4 shows precision and recall for each model and corpora, along with other data statistics used to compute them. Most of the novel words generated by the models have emotions that match those of the reference summaries since precision is higher than 75% in  
Figure 3: Emotional bias of each model for each emotion in Cnn/Dailymail and XSum. Correlations are statistically significant (\(p\)-value is 0 in all the cases).  
all cases. Pegasus is the system that shows the highest precision in both corpora. T5 has slightly higher precision than Bart in Cnn/Dailymail, but not in Xsum.  
The precision of all the models is higher in Cnn/Dailymail than in Xsum. The abstractive models generate more novel words in Xsum (4.9 novel words per summary) than in Cnn/DailyMail (0.9 novel words per summary). Then, generating more novel words will likely include more non-expected emotions. Table A3 of Appendix B shows an example from Xsum where the emotions of the novel words in a summary generated by Pegasus do not match exactly the emotions of the reference summary.  
Both in Cnn/Dailymail and XSUM, Pegasus generates novel words in more samples than Bart and T5 (lowest **Samples w/o novel words** ). However, for a larger number of samples than Bart and T5 in Cnn/Dailymail, the novel words generated by Pegasus do not convey emotions (highest **Samples \(|\mathcal{E}_{\delta}|=\mathbf{0}\)**). By contrast, Pegasus generates emotional novel words for a slightly larger number of samples than Bart and T5 in Xsum (lowest **Samples \(|\mathcal{E}_{\delta}|=\mathbf{0}\)**). We notice that the models generate more novel words in Xsum than in Cnn/Dailymail, but the number of samples where novel words do not convey emotions is similar in both corpora.  
Interestingly, the recall is between 34% and 51%, which suggests that the emotions of the novel words are enough to cover, approximately, at least a third part of the overall emotional content of the reference summaries. Bart has the highest recall in Cnn/Dailymail and Pegasus in Xsum. Although it is difficult to explain why, the number of emotions in the reference summaries (lower in Xsum than in Cnn/Dailymail) could play a big role.  
Considering the overall results of the two corpora, the difference in recall is significant. We consider that it is due to the difference in the introduction of new words in both cases. The fact that the Xsum corpus is much more abstractive in nature than the Cnn/DailyMail means that the former incorporates a greater number of novel words, and therefore, more emotional content.