# Large Language Model-Powered Automated Assessment: A Systematic Review
## 3 Results
### Details of LLMPAA Research
#### 3.2.3 LLM's and Evaluation Metrics
In studies investigating the agreement of LLMs with human (N = 40), it has been found that the metrics QWK (n = 18), CK (n = 5), LWK (n = 1), and r (n = 16) were the most frequently used. One study [20], which focused on assessing ASAG performance in medical education, reported a notably high QWK value of 0.99. Analysis of the studies employing the QWK metric (n = 15) revealed that 40% achieved excellent agreement (QWK = 0.81-1.00), 33% demon \begin{table}
\begin{tabular}{c c c c} \hline \hline
**References** & **LLMs** & **Evaluation Metrics** & **Results** \\ \hline
[48] & Claude 2, GPT-3.5, GPT-4 & QWK: 0.567 & The highest agreement was achieved with GPT-4. \\ \hline
[49] & GPT-4 & CK: 0.88 & GPT-4 achieved the best agreement. \\ \hline
[2] & BLOOM, GPT-3, YouChat & P: 0.488, R: 0.912 & GPT-3 showed high recall and low precision. \\ \hline
[50] & GPT-3.5 (Base), GPT-3.5 (fine-tuned: FTP1, FTP2, FTPB), GPT-4 (Base) & QWK: 0.78 & Fine-tuned models FT-PI achieved the highest QWK value. \\ \hline
[51] & all-distilroberta-v1, all-MiniLM-L6-v2, paraphrase-albert-small-v2, paraphrase-MiniLM-L6-v2, multi-qa-distilbert-cos-v1, multi-qa-apment-base-dot-v1, stsb-distilbert-base, stsb-roberta-large & \\ \hline
[52] & BERT & CK: 0.75 & BERT achieved agreement with humans. \\ \hline
[53] & BERT, Bi-LSTM, CNN-T BERT, fine-tuned BERT, Xsum-T BERT, YAKE-T BERT & QWK: 0.789 & Xsum-T BERT and YAKE-T BERT achieved the highest agreement. \\ \hline
[54] & BERT, Bi-LSTM, Capsule Network, fine-tuned BERT & r: 0.897 & BERT showed high correlation with the extended dataset. \\ \hline
[55] & BERT-MTL-fine-tune, BERT-MTL-fine-tune (CELA dataset), BERT-fine-tune & QWK: 0.83 & BERT-MTL-fine-tune achieved the highest agreement. \\ \hline
[56] & GPT-4 & r: 0.67–0.82 & GPT-4 showed medium correlation with humans. \\ \hline
[57] & Bard, GPT-3.5 (default, fine-tuned) & ICC: 0.972 & Fine-tuned GPT-3.5 performed with the highest consistency. \\ \hline \hline \end{tabular} This table summarizes the LLMs used in the reviewed studies and the evaluation metrics applied. “Model” refers to the specific LLM or its variant (e.g., GPT-4, BERT), and indicates whether it was pre-trained, fine-tuned, or used in a zero-shot setting. “Evaluation Metrics” are grouped into three categories: (1) performance (accuracy, F1-score, precision, recall), (2) agreement with human raters (QWK, CK, LWK, r), and (3) inter-rater reliability (ICC). Studies often used multiple metrics to strengthen validity. Fine-tuned models were typically trained on labeled datasets aligned with scoring rubrics. In some cases, prompt engineering and temperature adjustments were used to improve scoring consistency and control output variability.  
\end{table} 
Table 5: _Cont._strated good agreement (QWK = 0.61-0.80), and 20% reported moderate agreement (QWK = 0.41-0.60). In studies that only used the CK metric (n = 4), it was found that in half of the cases, the model showed excellent agreement, while in the other half, the model demonstrated good agreement. Among the studies that exclusively examined the correlation coefficient (n = 12), 91% reported strong agreement (r = 0.70-1.00), while one of the studies [24] reported low agreement (r = 0.30-0.50). A review of the literature identified three studies that assessed the consistency and reliability between different evaluators (human- or LLM-based). These studies employed the intraclass correlation coefficient (ICC, n = 6) to evaluate the alignment between the model-generated scores and those provided by various raters. In two of these studies, the model demonstrated a good level of consistency (ICC = 0.75-0.90), whereas in one study, a moderate level of consistency was observed (ICC = 0.50-0.75). Additionally, there are studies in which both ICC and correlation coefficient (r) metrics were used concurrently.