# Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
## 4 Experimental Setup
### Prompt Performance Evaluation Metrics  
To assess the effectiveness of a prompt, we evaluate how well the LLM's relevance predictions align with human-labeled judgments. A prompt is considered effective if the LLM produces outputs that closely match human assessments. To measure this agreement quantitatively, we use [PERSON]'s kappa (\(\kappa\)), which is a statistical measure of inter-rater reliability that accounts for agreement occurring by chance. This metric is particularly appropriate for our task, as it evaluates the degree to which LLM-generated labels replicate human judgments in a principled and chance-corrected manner. [PERSON]'s \(\kappa\) has also been adopted in recent studies evaluating the reliability of LLM-based relevance judgments [25].  
Given that the LLM's predicted relevance is denoted as \(\mathsf{rel}_{\mathsf{pred}}\) and the human-labeled ground-truth relevance as \(\mathsf{rel}_{\mathsf{truth}}\), [PERSON]'s kappa is computed as follows:  
\[\kappa=\frac{P_{o}-P_{e}}{1-P_{e}} \tag{1}\]  
where \(P_{o}\) represents the observed agreement between the LLM's predictions and human labels, while \(P_{e}\) corresponds to the expected agreement that would occur by chance. To compute \(P_{o}\), we measure the proportion of cases where the LLM's prediction matches the human annotation. In a binary classification setting, this can be expressed as  
\[P_{o}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}(\mathsf{rel}_{\mathsf{truth}i}= \mathsf{rel}_{\mathsf{pred}i}) \tag{2}\]  
where \(\mathbf{1}(\cdot)\) is an indicator function that returns 1 if the LLM's prediction matches the human label and 0 otherwise, and \(N\) represents the total number of samples. The expected agreement \(P_{e}\) accounts for the probability that the LLM and human annotators agree  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Usage** & **TREC DL Year** & \begin{tabular}{c} **Number of** \\ **Queries** \\ \end{tabular} & \
\begin{tabular}{c} **Number of Data** \\ \end{tabular} &
\begin{tabular}{c} **Number of Sampled** \\ **Data** \\ \end{tabular} \\ \hline Training & 2019 & 43 & 9260 & 500 \\ Test & 2020 & 54 & 11,386 & 500 \\ Validation & 2021 & 53 & 10,828 & 500 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Overview of the TREC DL passage datasets utilized in the study. The 2019 dataset is used for training, the 2020 dataset for testing, and the 2021 dataset for validation. Due to the resource-intensive nature of evaluating the entire dataset, three separate test samples of 500 data points each are created for evaluation purposes.  
by chance. This is computed based on the marginal distributions of the human and LLM predictions:  
\[P_{e}=\sum_{j\in\{0,1\}}\left(\frac{\sum_{i=1}^{N}\mathbf{1}(\mathsf{rel}_{ \mathsf{truth}i}=j)}{N}\times\frac{\sum_{i=1}^{N}\mathbf{1}(\mathsf{rel}_{ \mathsf{pred}i}=j)}{N}\right) \tag{3}\]  
By utilizing [PERSON]'s kappa (\(\kappa\)), we evaluate how well a given prompt aligns the LLM's responses with human judgments, where a higher \(\kappa\) value indicates stronger agreement and consequently a more effective prompt.