## 5 Evaluation
### Performance on the Rule-Based Dataset  
We draw upon the skewed rule-based dataset for assessing model robustness. This dataset only contains annotations that are (i) obvious to human experts and (ii) can be derived from well-established heuristics (Section 5.1.2).  
These experiments, therefore, aim at assessing the classifiers' robustness against grave mistakes, i.e., cases where automatic classifications clearly violate human intuition. They have been designed to indicate the impact of mitigation strategies such as increasing model complexity (i.e., large language models) and enhancing model diversity (ensemble method) on the model's robustness against these types of mistakes.  
Table 14 summarizes the performance of all four methods on the offshorable indicator. Both the Transformer and the ensemble model provide very good results for offshorability, while the ChatGPT heuristic scored even below the SVM baseline.  
Table 15 compares the systems' performance on the automatable indicator. The ensemble model provides the best results for the offshorable indicator and the DistilBERT classifier excels for the automatable category. Both models are very similar in terms of performance and clearly outperform the SVM baseline and the GPT heuristic.  
These results provide the following insights:  
* All models that have been fine-tuned on the random selection dataset have benefited tremendously from training on task-specific data. Consequently, even the fine-tuned SVM classifier outperforms the GPT heuristics.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **f1** & **Precision** & **Recall** & **Accuracy** \\ \hline SVM baseline & 0.58 & 0.68 & 0.64 & 0.64 \\ distilbert-base-multilingual-cased & **0.75** & **0.75** & **0.74** & **0.74** \\ ChatGPT combined with Heuristic & 0.55 & 0.56 & 0.55 & 0.55 \\ Ensemble model & 0.73 & 0.74 & 0.73 & 0.73 \\ \hline \hline \end{tabular}
\end{table}
Table 13: The“automatable” indicator’s classification performance on the randomly selected dataset. The best results are indicated in bold.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **f1** & **Precision** & **Recall** & **Accuracy** \\ \hline SVM baseline & 0.85 & 0.91 & 0.90 & 0.90 \\ distilbert-base-multilingual-cased & 0.93 & 0.94 & 0.93 & 0.93 \\ ChatGPT combined with Heuristic & 0.76 & 0.79 & 0.80 & 0.80 \\ Ensemble model & **0.94** & **0.95** & **0.94** & **0.94** \\ \hline \hline \end{tabular}
\end{table}
Table 14: Performance for the “offshorable” indicator on the skewed rule-based dataset. The best results are indicated in bold.  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model** & **f1** & **Precision** & **Recall** & **Accuracy** \\ \hline SVM baseline & 0.60 & 0.64 & 0.68 & 0.68 \\ distilbert-base-multilingual-cased & **0.76** & **0.77** & **0.77** & **0.77** \\ ChatGPT combined with Heuristic & 0.69 & 0.69 & 0.68 & 0.68 \\ Ensemble model & 0.74 & **0.77** & 0.73 & 0.73 \\ \hline \hline \end{tabular}
\end{table}
Table 12: The “offshorable” indicator’s classification performance on the randomly selected dataset. The best results are indicated in bold.  
Both DistilBERT, the ensemble, and GPT heuristics leverage background knowledge from the base models, which improves model performance compared to the SVM baseline but is insufficient to compensate for fine-tuning on domain data (compare GPT's performance). GPT heuristics, therefore, struggle with providing accurate classifications based on the limited context available through prompt engineering.
* Additional ensemble complexity does not translate into better performance.