# BART-IT: An Efficient Sequence-to-Sequence Model for Italian Text Summarization
## 4 Experiments
### Experimental Results  
This section outlines the main experimental results achieved on the analyzed datasets used during the evaluation phase. To analyze both model efficiency and effectiveness, we report the result obtained on the test set and the number of parameters separately for each model. The model size can be used as a proxy for the complexity of the model and as a way to gauge the performance of models of different sizes. Tables 3-5 separately report the results of both extractive and abstractive summarization models on the three different datasets using ROUGE and BERTScore metrics. \begin{table}
\begin{tabular}{c c c c c c} \hline
**Model** & **\# Parameters** & **R1** & **R2** & **RL** & **BERTScore** \\ \hline LEAD-2 & \(\times\) & 31.88 & 14.11 & 21.68 & 70.84 \\ TextRank & \(\times\) & 26.39 & 9.06 & 16.81 & 68.63 \\ LexRank & \(\times\) & 29.85 & 11.69 & 19.58 & 69.9 \\ \hline IT5-base & 220 M & 33.99 & 15.59 & 24.91 & 70.3 \\ BART-IT & 140 M & 35.42 & 15.88 & 25.12 & 73.24 \\ \hline mT5 & 390 M & 34.13 & 15.76 & 24.84 & 72.77 \\ mBART & 610 M & 36.52 & 17.52 & 26.14 & 73.4 \\ \hline \end{tabular}
\end{table} 
Table 3: Summarization evaluation results on the FanPage dataset. The # parameters column is empty for unsupervised baseline models since they do not include any trainable parameters. News summarization datasets  
FanPage and IlPost [30] are two datasets containing Italian news articles and their corresponding summaries. Both datasets can be classified as _medium-sized_ datasets since they contain less than 100,000 of documents in the training, validation, and test sets. BART-IT significantly outperforms all extractive baseline models and IT5 with comparable model sizes on both datasets. Tables 3 and 4 show the comparison between BART-IT and the other models on the two datasets. Regarding ROUGE-1 and ROUGE-2 scores, BART-IT outperforms IT5 by a margin of 1.43 and 0.29 points (i.e., 4.2% and 1.8% relative improvement) on FanPage and 4.43 and 3.91 points (i.e., 13.4% and 25.17% relative improvement) on IIPost. In terms of BERTScore, BART-IT outperforms IT5 by a margin of 2.93 (4.2%) and 4.3 (6.11%) points on FanPage and IlPost, respectively. When comparing BART-IT with multilingual models, we observe that BART-IT can perform better than mT5, but it is outperformed by mBART on both datasets at the cost of significantly more computationally intensive model training. More specifically, both mBART and mT5 models are trained on a multilingual corpus and contain significantly more parameters than BART-IT (e.g., 610 M parameters for mBART and 390 M parameters for mT5 compared to 140 M parameters for BART-IT), which can be a reason for the performance gap between the models. WITS [31] is a large dataset of Wikipedia articles and their corresponding summaries. The dataset contains roughly 700,000 documents and has been used in previous works to evaluate the performance of abstractive summarization models [31]. Despite the fact that having a large number of documents can be beneficial for training accurate models, the Wikipedia articles are rather diversified in length and topic, making the task of summarization significantly more challenging. Analyzing the results on the WITS dataset reported in Table 5, we observe that BART-IT performs significantly better than all the other models, including the multilingual ones. Even though the number of parameters of BART-IT is significantly fewer than both mBART and mT5, the model is able to outperform them in terms of ROUGE metrics. Considering the BERTScore metric, mT5 performs slightly better than BART-IT. The results indicate that multilingual models are able to learn the mapping