# Cross-Lingual Cross-Domain Transfer Learning for
## 4 Experiments
### Performance on Auxiliary Tasks--Stage 1  
The overall performance of the models in the auxiliary tasks is described, which serves as the basis for the CLMs used throughout this work. This initial validation was performed only in English, using _bert-base-uncased_ as a baseline to evaluate performance. Recall that this validation aims to demonstrate competitive performance on the auxiliary tasks. The construction of CLMs uses _bert-base-multilingual-uncased_ ([[https://huggingface.co/bert-base-multilingual-uncased](https://huggingface.co/bert-base-multilingual-uncased)]([https://huggingface.co/bert-base-multilingual-uncased](https://huggingface.co/bert-base-multilingual-uncased)), accessed on 22 June 2025) and the hyperparameters described below. **Stance classification**. As in BUT-FIT's implementation [87], a weighted loss function is used to address dataset imbalance. In practice, we experimented with variations such as gradient clipping, bootstrapping resampling, and learning rates. The weighted loss provided the best performance. The best results were obtained using dropout at 0.2, three epochs, the learning rate at 1 \(\times\) 10\({}^{-5}\), and the size of the training and validation batch at 2. Table 3 summarizes the results obtained, by seed, during training. The seeds were obtained by using a random number generator. The best model achieved 0.605 Macro-F1. This approach is competitive with the results reported by [PERSON] et al. [87], although they report around 0.531 Macro-F1 when using both Reddit and Twitter data. In contrast, we achieve a score of around 0.6 using Twitter data. **Bot detection**. After an initial exploration of the parameter space, the best results were obtained using dropout at 0.1, 5 epochs, a learning rate at \(1\times 10^{-5}\), and training and validation batch size at 4. We explored the impact of the sampling strategies as well as the maximum number of tweets to be sampled. Hence, we considered the following options:  
* _ALEN_, number of tweets per author: 5, 15, 25, 50, 75, and 100. * Sampling strategies: tail, head, head+tail. For the English language, we set the variable _MAXLEN_ to 512 because it is the limit of tokens processable by mBERT. By combining these options, we achieved 18 settings in which 3 random seeds were evaluated after training for 5 epochs. In each setting, _ALEN_ tweets, without replacement, were sampled from the available messages of an author, and then for each tweet sampled in this way, \(N=\lfloor MAXLEN/ALEN\rfloor\) characters were extracted. Table 4 (third column) shows the results of these experiments. Taking into account the results presented in Table 4, we chose the configuration with the best result during the construction of the CLMs. This configuration has Macro-F1 0.969 with parameters ALEN = 25 and sampling strategy = _tail_. In the context of bot detection in Spanish (see Table 4, fourth column), the same experimental setting was used for model selection, as previously established. Observations indicate that, in general, the results are slightly lower than those achieved when detecting bots in English. The best-performing model, which was selected for transfer learning, achieved a Macro-F1 score of 0.933 with ALEN = 50, and employed a sampling strategy