# Automated Assessment of Inferences Using Pre-Trained Language Models
## 2 Materials and Methods
### Transfer Learning and Evaluation Using k-Fold Cross-Validation  
The inference classifiers were trained by fine-tuning the three pre-trained language models as follows. Each pair of input sentences and corresponding responses was concatenated with the separator symbol ([SEP]) between them. The combined text was then provided as the input. For each base model, the last layer (classification head) was replaced with a dense layer with the output size set to one, and its weights were initialized with random values. As a result, the model will produce a single number for each input, and this output is used as the logit of the target class. The loss is defined by the binary cross entropy between the output of the model and true class label as follows:  
\[\begin{array}{l}\text{Loss}=-(\text{y}\log(\sigma(\text{z}))+(1-\text{y}) \text{log}(\sigma(\text{z}))),\\ \sigma(\text{z})=\frac{1}{1+e^{-z}},\end{array}\]  
where y is the true class label (0 or 1), z is the model output, and \(\sigma\) is the sigmoid function that transforms the input logit into a probability. This loss is minimized using the Adam optimizer [35].  
Hyperparameters for the training process were optimized as follows: As suggested by the authors of the pre-training models [26], the batch size, learning rate, warm-up ratio, and weight decay were varied independently, and the highest F1 score for each model was reported. First, the batch size was set to either 4, 8, 16, and 32. Larger batch sizes provide a more accurate estimate of the gradient, resulting in more stable training. However, they require more memory and processing power. Second, the learning rate was set to either \(10^{-5}\), \(2\times 10^{-5}\), \(3\times 10^{-5}\), and \(5\times 10^{-5}\). Too low a learning rate can lead to a long convergence time or to being fixed at a local minimum, while too high a learning rate can cause the training to be unstable and diverge. Third, the warm-up ratio was set to either 0, 0.1, 0.2, and 0.6. During the warm-up iterations, the learning rate gradually increased from zero to the target learning rate. This technique helps to stabilize the fine-tuning in the early iterations. Fourth, the weight decay was set to either 0, 0.01, 0.02, 0.04, and 0.08. The weight decay regularizes the model and prevents overfitting by penalizing large weights. The four hyperparameters varied independently, resulting in a total of 320 configurations.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **BERT-Base** & **RoBERTa-Base** & **RoBERTa-Large** \\ \hline Number of layers & 12 & 12 & 24 \\ \hline Number of parameters & 110 million & 110 million & 335 million \\ \hline Masking & static & dynamic & dynamic \\ \hline Next sentence prediction & included & removed & removed \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of pre-trained models’ architecture and training objectives.  
For each base model, five-fold cross-validation was performed for each training configuration in order to rigorously assess the classification accuracy and generalization ability of the model. Specifically, the dataset was randomly divided into five distinct subsets, so that the proportions of the inference subtypes were equal. Four of these subsets were used for training and the remaining one was used for validation in order to calculate the F1 score with corresponding precision and recall scores. This cycle was repeated five times, with each subset serving as a validation set once. As a result, five F1 scores were collected for each training configuration. The configuration corresponding to the highest average F1 score was found for each base model. The five F1 scores of the best model were compared for models that were fine-tuned from different base models, using the paired \(t\)-test.