# Large Language Model-Powered Automated Assessment: A Systematic Review
## 3 Results
### Details of LLMPAA Research
#### 3.2.3 LLM's and Evaluation Metrics  
In this systematic review, the full texts of the 49 selected studies were carefully analyzed to extract the metrics reported for evaluating LLMPAA systems. Rather than conducting new statistical tests, the study focused on identifying and categorizing the validation metrics used in each study. These metrics were descriptively compiled and classified according to their function--whether they assessed model accuracy, agreement with human raters, or inter-rater consistency. This approach enabled a comparative understanding of how performance and reliability are reported across different research contexts and also revealed areas where statistical validation practices could be strengthened. Based on the analysis, the LLMs utilized and the corresponding evaluation metrics are summarized in Table 5. The metrics employed across the 49 studies have been categorized into three groups:* Metrics for determining model performance (accuracy, F1-score, precision, and recall). * Metrics based on measuring the agreement of LLMs with human (Cohen's kappa--CK, linear weighted kappa--LWK, quadratic weighted kappa--QWK, correlation coefficient--r). * Metrics based on measuring the reliability and consistency between different evaluators (human or LLM) (intraclass correlation coefficient--ICC). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**References** & **LLMs** & **Evaluation Metrics** & **Results** \\ \hline
[13] & BERT, ELECTRA, Multilingual BERT, RoBERTa & F1: 0.72 & BERT model performed the highest F1 score. \\ \hline
[14] & BERT-XLNET, CNN-BiLSTM, BERT-BiLSTM, R2 BERT, ATT-CNN-LSTM & CK: 0.91 & BERT-BiLSTM achieved better agreement compared to other models. \\ \hline
[15] & 30-manually extracted features + 300-word2 vec + 768-BERT, 300-word2 vec + 768-BERT, 768-BERT, 30-manually extracted features + 768-BERT, 300-word2 vec & ACC: 0.75, QWK: 0.77 & The best agreement was achieved with the 30-manually extracted features + 300-word2 vec + 768-BERT model. \\ \hline
[16] & all-MiniLM-L6-v2, paraphrase-MiniLM-L6-v2, paraphrase-albert-small-v2, & ACC: 82.56, _Abstract_: & Different models showed accuracy up to 82.56\% in binary classification and multi-qa-MiniLM-L6-cos-v1, & 56.11, _Abstract_: 0.76, & up to 56.11\% in multi-class classification. \\ \hline
[17] & GPT-4 & r: 0.382; ICC: 0.447 &
\begin{tabular}{c} GPT-4 showed weak correlation with human and low consistency. \\ \end{tabular} \\ \hline
[18] & ABC-BERT-FTM, BERT & QWK: 0.9804 &
\begin{tabular}{c} The ABC-BERT-FTM model achieved the highest agreement. \\ \end{tabular} \\ \hline
[19] & BigBird, BERT, DualBERT-Trans-CNN, & & \\  & Considering-Content-XLNet, & QWK: 0.791 &
\begin{tabular}{c} The highest performance was achieved with the Trans-BERT-MS-ML-R model. \\ \end{tabular} \\ \hline
[20] & ACTA, BERT & QWK: 0.99 &
\begin{tabular}{c} The ACTA model achieved the highest agreement with human. \\ \end{tabular} \\ \hline
[21] & GPT-4 & r: 0.85 &
\begin{tabular}{c} GPT-4 showed successful results with high correlation and low error rate. \\ \end{tabular} \\ \hline
[6] & BERT & QWK: 0.78 &
\begin{tabular}{c} BERT achieved high agreement after fine-tuning. \\ \end{tabular} \\ \hline
[22] & LaBSE, mBERT & ACC: 0.88, CK: 0.83, &
\begin{tabular}{c} LaBSE performed better in all \\ languages, showing superior \\ performance in QWK and accuracy. \\ \end{tabular} \\ \hline
[23] & Gemini 1.0 Pro, GPT-4 & ACC: 0.68 (binary), 0.59 (multi) &
\begin{tabular}{c} Gemini performed better in binary \\ classification, while GPT-4 showed \\ better performance in multi-class \\ classification. \\ \end{tabular} \\ \hline
[8] & GPT-3.5 Turbo (Few-Shot, Zero-Shot), GPT-4 & LWK: 0.94 (2-class), & GPT-4 (Few-Shot) achieved the highest agreement. \\ \hline
[24] & Qwen-max-0428 & r: 0.491 &
\begin{tabular}{c} Qwen-max-0428 achieved low \\ agreement with humans. \\ \end{tabular} \\ \hline
[1] & BERT (fine-tuned), LSTM & r: 0.761 &
\begin{tabular}{c} BERT + LSTM showed the highest \\ correlation. \\ \end{tabular} \\ \hline \hline \end{tabular}
\end{table}