## 4 Tailoring Pre-Trained Transformer Models for Detection in Spanish
### Fine-Tuning: Adjustment of Model Hyperparameters and Evaluation Metrics  
In order to fine-tune the previously mentioned BERT and RoBERTa models on our domain-specific dataset containing Spanish political news, we made use of the Transformer library [78], and employed cross-validation to address the potential issue of overfitting in the models we intended to fine-tune. * In particular, cross-validation allows us to assess the performance of the learned models and ensure their robustness in generalizing to unseen data. It works by partitioning the dataset into multiple subsets or folds, where the model is evaluated on one subset after being trained on the other ones in each iteration, ultimately yielding an average performance metric. This process aids in detecting overfitting as it highlights any significant performance discrepancies across the folds. If the model exhibits varying performance on different subsets, it signals that it may be overfitting to the training data, as it struggles to generalize to new, unseen data. In our approach, we have set aside 20% of the data for testing the final model (_test data_), and on the remaining 80%, we have implemented cross-validation with 5 folds (\(K=5\)). In each fold or iteration, the model is trained on 80% of the available data (_training data_) and its performance is evaluated on the remaining 20% (_evaluation data_). * Regarding the fine-tuning, this process also encompasses the refinement of hyperparameter values, whose appropriate configuration is critical to build effective models. Both BERT and its RoBERTa models share common hyperparameters due to their Transformer-based architecture and pre-training process: (i) the _learning rate_ controls how much the model weights are updated after each training step; (ii) the _maximum sequence length_ determines the maximum size of the input that the model can handle; (iii) the _batch size_ defines the number of training examples processed in each iteration, affecting memory usage and training speed; (iv) the _number of epochs_ indicates how many times the model goes through the entire training dataset; and (v) _regularization techniques_, such as L2 regularization or weight decay, that help prevent overfitting. To optimize the hyperparameters, we carried out a series of experiments, adjusting their values based on the unique characteristics of our domain-specific dataset and the specific requirements of our binary text classification task. In doing so, we managed to strike a balance between achieving high performance on the training data and ensuring effective generalization to unseen data for each model. We carefully tuned the hyperparameters of the Transformer models evaluated through an extensive series of experiments, exploring different combinations until identifying the settings that offered the best performance. This iterative process allowed us to determine the optimal values that maximize the models' accuracy. The chosen values for each type of model are presented in Table 3, where we also specify the seed used for reproducibility by consistently initializing random processes across model runs. Although tools such as GridSearchCV and RandomizedSearchCV, along with more sophisticated options like Optuna and Hyperopt, provide valuable techniques for hyperparameter tuning, they have notable drawbacks in our particular scope. GridSearchCV systematically examines a specified grid of parameters, but it can become resource intensive as the number of parameters increases. In contrast, RandomizedSearchCV enhances efficiency by randomly selecting parameter values but may overlook significant combinations. In light of these challenges, we chose to pursue a manual tuning approach, which offers the flexibility to concentrate on essential parameters based on our expertise in the field, thus avoiding unnecessary computational demands.