## 7 Model Selection and Prototyping
### Experimental Setting  
As a first experiment, we trained our models with a fixed set of hyperparameters over 50 epochs. Given that these models reach a plateau on the 0-F1 metric after 8-10 epochs, we decided to keep the hyperparameters involved in the fine-tuning operation fixed, such as the optimizer learning rate equal to 2 \(\times\) 10\({}^{-5}\), its weight decay equal to 0.01, and the batch size equal to 16. We made these choices because a small modification of these hyperparameters did not significantly impact the results for a fine-tuning of only 10 epochs. Each model was evaluated on the test set on the F1-score metric, calculated as the harmonic mean of precision and recall; given the high imbalance in the class labels, we focused on maximising the F1 score for the zero class (0-F1). First, we decided to test different types of pre-trained BERT models that are freely available on the HuggingFace repository ([[https://huggingface.co/](https://huggingface.co/)]([https://huggingface.co/](https://huggingface.co/)) (accessed on 8 August 2023)); we tested four configurations of an Italian BERT model ([[https://huggingface.co/dbmdz/bert-base-italian-uncased](https://huggingface.co/dbmdz/bert-base-italian-uncased)]([https://huggingface.co/dbmdz/bert-base-italian-uncased](https://huggingface.co/dbmdz/bert-base-italian-uncased)); [[https://huggingface.co/dbmdz/bert-base-italian-xcl-uncased](https://huggingface.co/dbmdz/bert-base-italian-xcl-uncased)]([https://huggingface.co/dbmdz/bert-base-italian-xcl-uncased](https://huggingface.co/dbmdz/bert-base-italian-xcl-uncased)), [[https://huggingface.co/dbmdz/bert-base-italian-xcl-cased](https://huggingface.co/dbmdz/bert-base-italian-xcl-cased)]([https://huggingface.co/dbmdz/bert-base-italian-xcl-cased](https://huggingface.co/dbmdz/bert-base-italian-xcl-cased)) (accessed on 8 August 2023)): cased or uncased, base or XXL. The cased version, unlike the uncased one, recognizes capital letters and normal letters. Moreover, the base model version is trained with the Italian Wikipedia dump (approximately 13 GB of text), whereas the XXL version is trained with the Italian part of the OSCAR corpus (in total approximately 81 GB of training data). Furthermore, we tested the multilingual cased and uncased versions of BERT trained by the authors of the original paper ([[https://huggingface.co/bert-base-multilingual-uncased](https://huggingface.co/bert-base-multilingual-uncased)]([https://huggingface.co/bert-base-multilingual-uncased](https://huggingface.co/bert-base-multilingual-uncased)); [[https://huggingface.co/bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased)]([https://huggingface.co/bert-base-multilingual-cased](https://huggingface.co/bert-base-multilingual-cased)) (accessed on 8 August 2023)); we then tried two different models, a multilingual RoBERTa ([[https://huggingface.co/xlm-roberta-base](https://huggingface.co/xlm-roberta-base)]([https://huggingface.co/xlm-roberta-base](https://huggingface.co/xlm-roberta-base)) (accessed on \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Type** & **\# Instances** & **\# 0 Class** & **\# 1 Class** & **\% 0 Class** \\ \hline Answers & 412 & 282 & 130 & 31.55\% \\ \hline Concepts & 1065 & 717 & 348 & 32.68\% \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Test dataset composition. In the first row, the data are aggregated by answers and in the second by concepts. 11 August 2023)) [41] and an Italian Electra ([[https://huggingface.co/dbmdz/electra-base-italian-xcl-cased-discriminator](https://huggingface.co/dbmdz/electra-base-italian-xcl-cased-discriminator)]([https://huggingface.co/dbmdz/electra-base-italian-xcl-cased-discriminator](https://huggingface.co/dbmdz/electra-base-italian-xcl-cased-discriminator)) (accessed on 11 August 2023)) [42].