## 3 The Proposed System
### Sentence Classification
#### 3.2.1 Target Sentence Labeling
In the case of the comparison between our system and the fine-tuned BERT, we obtained \(W=5.0\) (the minimum of the sum of ranks above and below zero) and \(pvalue=0.87\). Thus, we can conclude that there is no statistically significant difference between the two classifiers. In the case of the comparison between our system and the prompt-tuned BERT, we obtained \(W=4.5\) and \(pvalue=0.05\). Thus, we can conclude that there is a statistically significant difference between the two classifiers



The analysis of the figures shows that the results are dependent on the class and the news item. For example, our system and the fine-tuned BERT perform equally on the DS class in the news about [PERSON] coming out, while our system obtains better performances than the fine-tuned BERT on the same class in the case of the news about Italy's victory, even if the fine-tuned BERT has a higher F1 score in general. Figure 4: Confusion matrix of the prompt-tuned BERT on the news about [PERSON] coming out. Figure 5: Confusion matrix of our system on the news about Italy winning Euro 2020. Figure 3: Confusion matrix of the fine-tuned BERT on the news about [PERSON] coming out. Figures 8-10 show the average of the confusion matrices on all the tested news. Here, we can observe that the BERT classifiers perform slightly better than our system on the VDS class, while our system performs better on the DS class. Furthermore, the fine-tuned BERT outperforms our system and prompt-tuned BERT on the SS class. In general, the SS and DS classes present the greatest challenges for both BERT classifiers and our system. In the first case, we point out that several events in our datasets are slightly unbalanced with respect to the SS class. This is explained by the fact that two different articles on the same event often have a few similar sentences belonging to the SS class while the majority of the sentences belong to the DS and VDS classes. Conversely, the DS class is certainly the most challenging to predict. This difficulty may arise from the fact that the DS class is the least homogeneous of the three. Typically, it includes sentences with the same informative content as the most similar sentence in the reference article but expressed in a different linguistic style. We can conclude that the sentence classification method implemented in our system works quite well compared to the labeling performed by human users and used as ground truth. Furthermore, on average, our classifier achieves slightly lower performance compared to the BERT classifier trained with model tuning. However, it is important to note that BERT always requires fine-tuning with annotated data, whereas our system can be used without any tuning phase. Indeed, we can directly employ default thresholds, as demonstrated in our experiments. Figure 6: Confusion matrix of the fine-tuned BERT on the news about Italy winning Euro 2020. Figure 7: Confusion matrix of the prompt-tuned BERT on the news about Italy winning Euro 2020. ### Text Summarization Evaluation

In order to evaluate the quality of the generated summaries, we resorted again to crowdsourcing. We carried out an experiment to evaluate the quality of the information contained in the summaries generated from the target sentences extracted from our system. Specifically, for each of the news items in the dataset, we presented the raters with (i) the reference article; (ii) the target sentences extracted from the target articles, as described in Sections 3.2 and 3.3; and (iii) the summary generated from such sentences, as described in Section 3.4. For each summary, raters were asked to provide a rating for (i) the novelty, with respect to the reference article; (ii) factual correctness, with respect to the target sentences; and (iii) information completeness, with respect to the target sentences. Each summary

Figure 8: Average confusion matrix of our system. Figure 10: Average confusion matrix of the prompt-tuned BERT. Figure 9: Average confusion matrix of the fine-tuned BERT.