## 4 Tailoring Pre-Trained Transformer Models for Detection in Spanish
### Fine-Tuning: Adjustment of Model Hyperparameters and Evaluation Metrics
Unlike automated techniques, manual tuning facilitates iterative adjustments of parameters informed by practical observations, allowing for a more targeted and effective investigation of the hyperparameter landscape. This method minimized the likelihood of missing crucial settings and granted greater control over the optimization process, ultimately resulting in outcomes that aligned more closely with our specific requirements. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyperparameters** & **BERT Models** & **RoBERTa Models** \\ \hline \multirow{4}{*}{} & **BERT-Base** & _RoBERTa-Base_ \\  & **BERT-Multilingual** & _RoBERTa-Multilingual_ \\  & **BETO** & _RoBERTa-BERTIN_ \\ \hline Learning rate & \(5\times 10^{-5}\) & \(10^{-5}\) \\ \hline Max sequence length & 512 & 128 \\ \hline Batch size & 10 & 16 \\ \hline Epochs & 10 & 10 \\ \hline Drop out rate & 0.1 & 0.15 \\ \hline L2 regularization & 0.005 & 0.001 \\ \hline Seed & 10 & 16 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Hyperparameter values that have yielded the best performance in the 3 BERT models and 4 RoBERTa models fine-tuned in our approach. * Lastly, during the fine-tuning process, it is essential to measure various metrics to evaluate the performance of each model trained on our Spanish political fake news dataset. In Table 4, we describe the metrics adopted in our approach, obtained using the _Sklearn_ library [79], where _True Positives_ (TP) represent instances that are correctly predicted as positive (real news) by the model, while _False Positives_ (FP) are instances that the model incorrectly predicts as genuine news when they are, in fact, false. Similarly, _False Negatives_ (FN) occur when the model incorrectly predicts instances as negative (fake news) that are truly positive (real news), and _True Negatives_ (TN) are instances that are correctly predicted as deceptive news by the model. Although AUC-ROC (Area Under the Curve) is a widely used metric in binary classification, high values may not be representative in our scenario, where we work with synthetic and non-synthetic data. In this context, elevated AUC-ROC values would suggest strong performance in distinguishing real from fake news, but this could be unrealistic due to the possibility that the synthetic nature of the fake news might make these samples easier for the model to detect, potentially leading to results that do not generalize well to real-world settings. While AUC-ROC is often useful for balanced datasets, it can overlook specific types of errors, giving an overly optimistic impression of the model's effectiveness. For these reasons, we prioritized metrics that provide a more detailed evaluation of error management. Specifically, by focusing on precision and recall, we evaluate the model's ability to balance the trade-off between false positives (misclassifying real news as fake) and false negatives (failing to identify fake news), ensuring robust error handling. The F1 score further strengthens this assessment by offering a balanced metric that combines both precision and recall, while MCC offers a broader view of the model's performance by accounting for all types of outcomes. Before focusing on the results and experimental discussion, we emphasize the fine-tuning of the evaluated models, detailing the process where key concepts discussed in this section, such as hyperparameter tuning and cross-validation, are integrated. The fine-tuning and testing codes are available on GitHub at [[https://github.com/gssi-uvigo/debatrue](https://github.com/gssi-uvigo/debatrue)]([https://github.com/gssi-uvigo/debatrue](https://github.com/gssi-uvigo/debatrue)) (accessed on 1 October 2024). * Initially, the pre-trained models are loaded using the relevant Hugging Face classes (e.g., RobertaForSequenceClassification for the RoBERTa-BNE model), and the number of labels is set to two for our fake news detection task. The entire model is fine-tuned without freezing any layers, allowing all layers to be optimized during training to fully adjust to the binary classification task.