## 6 Results and Discussion
### Experiment I: Stereotype vs. Nonstereotype  
In this section, we present the results concerning the identification of stereotype about immigrants. In Table 3 we can appreciate that all the models obtained an _accuracy_ above 0.73. The highest result was obtained by BETO with 0.861 of _accuracy_ and a standard deviation of 0.016. This performance was significantly better than the one of SpanBERTa and M-BERT for \(p<0.05\) using the Mann-Withney U Test. We use this test, also known as Wilcoxon-Mann-Whitney test (a nonparametric alternative to the Student's \(t\)-test), considering that we randomly removed instances from the most populated class each time. Therefore, we assume independence among the results of the models. We see that LR, SVM, and RF obtained a higher accuracy compared to SpanBERTa and XLM-RoBERTa. We believe that this could be associated with the size of the dataset since more data could have had an impact on deep learning models achieving better results [92].  
Besides the fact that stereotypes involve more than the presence of specific words, word n-grams with the highest Pointwise Mutual Information (PMI) (PMI makes it possible to see the most relevant features (i.e., words, n-grams of words) for each topic and is computed as \(PMI(L,w)=\log\frac{p(Lw)}{p(L)p(w)}\). Where \(p(L,w)\) is the probability of a feature to appear in a text labeled as \(L\), \(p(L)\) is the probability of a label (we assume the label distribution to be uniform), and \(p(w)\) is the probability of \(w\).) to each label allow us to see that nonstereotypical texts talk more about _ayudas_ (help) to refugees and Africa (the country some immigrants come from), and it is mostly mentioned in the speeches we are working with), _acuerdos_ (agreement) between countries, etc. While in stereotypical texts we find more commonly bigrams such as _immigration illegal_ (illegal immigration), _immigrantes irregulares_ (irregular immigrants), and _regularizacion masiva_ (massive regularization) among others that indirectly reflect problems associated to immigration (see Table 4).  
Interestingly, it is not evident from observing the relevant n-grams why some of them are more related to the stereotypes about immigrants and others are not. This confirms that for the study of stereotypes about immigrants we have to go beyond the representative keywords that could define the social group. In other words, we should not rely only on intuitive words to base the measuring of bias in the case of stereotypes. In this sense, automatic approaches can detect other patterns that escape human detection.  
We also confirm that the detection of stereotypes in this work, concerning immigrants, is not about characterising two opposite social groups but the immigrant group only. The nonstereotypical texts are neutral in this sense referring only to the topic of immigration without stereotyping at all.  
In Table 5, we present the confusion matrix of BETO when obtaining the results shown in the Table 3. The model was similarly effective at predicting stereotypes and nonstereotypes, with a bit more confusion with the label Nonstereotype. Table 6 shows some examples where BETO misclassified the texts and their predictions.  
\begin{table}
\begin{tabular}{c c c c} \hline
**BETO** & **SpanBERTa** & **M-BERT** & **XLM-RoBERTa** \\ \hline
0.861 \(\pm\) 0.016 & 0.766 * \(\pm\) 0.021 \({}_{p=0.00018}\) & 0.829 * \(\pm\) 0.022 \({}_{p=0.00736}\) & 0.780 \(\pm\) 0.105 \({}_{p=0.06057}\) \\ \hline
**LR** & **SVM** & **NB** & **RF** \\ \hline
0.82 & 0.81 & 0.73 & 0.81 \\ \hline \end{tabular}
\end{table}
Table 3: Accuracy achieved by each model in Experiment I on Stereotype vs. Nonstereotype. We indicate the \(p\)-value of the Mann–Whitney U test regarding the alternative hypothesis that the results of BETO are the highest compared to the other transformers. With * we indicate when accuracy is significantly lower than the result of BETO. The hypothesis is accepted with \(p<0.05\) except for XLM-RoBERTa model.