## 6 Results
### Summarization Performance of the Models for CATALAN  
The performance of the models was evaluated using the ROUGE metrics [32] and BERTScore metric [33]. For each metric, we calculated the average F1 score and its 95% confidence interval by using bootstrapping. Results are shown in Table 5.  
The average F1 scores are shown in a normal font size and their confidence intervals in a smaller font size, placed at the right-side of the score. The best average score for each metric within a test partition is remarked in bold style. The confidence intervals are shown in blue color if their range intersects with the confidence interval of the best score value of the metric within the same test partition; in other case, the confidence intervals are presented in black color.  
The Table 5 shows, regarding the TESTi partition, that the NASca model performs similarly compared to the multilingual mBART model. mBART presents significantly better BERTScore result than NASca while there are overlappings in the confidence intervals in the ROUGE measures. The mT5 model has obtained a significant lower performance than the other two models, despite the fact that mT5 contains the Catalan language in its pretraining phase unlike the mBART model. We hypothesize that the pretraining dataset could influence the results. It could be that the data considered for Catalan to pretrain mT5 differs so much from our domain. Also, the proportion of languages similar to Catalan in the pretraining corpus could be related to this effect.  
In the case of the TESTni partition, there is a significant overall reduction of the performance in most of the metrics of the three models in comparison to the TESTi partition. Generally speaking, the NASca model has significantly better performance in almost all ROUGE metrics compared to the multilingual models, although there is an  
\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Partition** & **Model** & **ROUGE-1** & **ROUGE-2** & **ROUGE-1** & **ROUGE-Ls** & **BERTScore** \\ \hline \multirow{3}{*}{TESTi} & NASCA & **28.84**(28.62, 29.61) & 11.68(0.151, 11.68) & 22.78(0.241, 29.46) & 23.30(23.13, 23.24) & 71.85(0.78, 71.52) \\  & mBART & 28.59(28.28, 28.77) & **11.89**(0.17.12, 12.04) & **23.00**(28.21, 21.66) & **23.39**(23.22, 23.55) & **72.03**(0.76, 72.20) \\  & mT5 & 27.01(28.54, 27.18) & 10.70(0.54, 10.85) & 21.81(0.26, 25.97) & 22.12(29.56, 22.29) & 71.55(0.78, 71.52) \\ \hline \multirow{3}{*}{TESTni} & NASCA & **28.19**(27.56, 28.42) & 11.20(0.599, 11.41) & **21.45**(21.20, 21.66) & **22.44**(21.22, 22.67) & 70.14(0.58, 70.22) \\  & mBART & 27.46(27.24, 27.69) & 11.04(0.881, 11.29) & 21.13(0.28, 21.37) & 22.01(0.27, 22.24) & 70.33(0.35, 70.42) \\ \cline{1-1}  & mT5 & 27.00(28.57, 27.23) & **11.28**(0.14, 15.25) & 21.27(0.21, 25.51) & 22.01(0.28, 22.23) & **70.56**(0.47, 70.65) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average F1 scores and confidence intervals of models in summarization task in Catalan.  
overlapping between the confidence interval of NASCa and that of mT5 in ROUGE-2. According to BERTScore, the mT5 model obtains significant differences in comparison to the scores of the NASCa and mBART models.  
Taking into account the higher scores and the generalization capabilities, the results of the monolingual model are significant better than the multilingual ones. In one side, mBART has similar performance than NASCa model in the TESTI partition, however, the performance reduction in the second test partition indicates that the model generalizes worse than the other two models. On the other side, the mT5 model generalizes better than mBART, since the drop of the perfomance between the TESTI and the TESTNI is lower in mT5 than mBART, however, mT5 presents significantly lower performance than that of the NASCa model.