## 7 Model Selection and Prototyping
### Best Models  
As for the results of our grid search, we report the configuration and performance of the five best models in Table 3. In order to compare the results, we aggregate the concepts-based models by the user answers, multiplying each binary prediction (concept present, concept absent) by its relative weight, and summing them. If the sum is greater than the 80 threshold, we set the prediction value for the answer to 1, otherwise to 0.  
As we can see in Table 3, the results are very similar. Three out of five of the best models leverage an Italian version of BERT; even if the manipulation of the embedding matrix with the max and mean operations is on average better than the other values, the best model takes advantage of the pooler output and the others prefer the minimum manipulation. Finally, even if on average the balancing of the training dataset worsens performance, four out of five best models are trained with the smaller and balanced dataset.  
As we can see, the best CONC model reaches a 0.72 of 0-F1, whereas the best REF model stops at 0.69. The best QREF and QCONC models perform even less well; the first has a 0-F1 equal to 0.70 and the second equal to 0.68. In Table 2 in Appendix A, we report the complete sets of hyperparameters for the best CONC, QCONC, REF, and QREF models.  
Besides the fact that the best model is concept-based, there are other reasons why the company choses to implement a concept-based model over the one based on the reference response. With concepts, human teachers can easily and quickly assess automatic corrections. From their point of view, it is easier to decide if a set of small key concepts is contained in the user's answers with respect to comparing two long and complex answers. The second main reason is a direct consequence of the classification task; with the REF and QREF models, the artificial intelligence system can tell human evaluators if an answer by a user has a score greater than or less than 80, but not the real score. With concept-based models, human correctors can assign a weight to the concepts and the user's answer score can be more accurate. For example, if a question has 4 concepts and each of them has a weight equal to 25, the automatic corrector can produce a score equal to 0, 25, 50, 75, or 100, depending on how many concepts the model recognises as present in the user's response.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Model** & **BERT** & **Classifier Input** & **Max. Length** & **Balanced** & **Dropout** & **Freeze** & **0-F1** & **1-F1** \\ \hline CONC & IT-XXL-UNC & Pooler & 512 & True & 0.1 & False & **0.72** & **0.87** \\ \hline CONC & W-MLM & Min Emb. & 128 & True & 0.1 & False & 0.69 & **0.87** \\ \hline CONC & ITA-UNC & Min Emb. & 512 & False & 0.4 & False & 0.69 & 0.86 \\ \hline CONC & MULTI-UNC & Pooler & 512 & True & 0.2 & False & 0.69 & 0.86 \\ \hline REF & MULTI-UNC & Min Emb. & 512 & True & 0.2 & False & 0.69 & 0.85 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Best five model configurations that maximize the 0-F1 score on the test set. In this table, the base BERT model, the classifier input, and the hyperparameters values are reported. The best values of 0-F1 and 1-F1 are highlighted in bold.