# Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
## 3 Methodology
### APO-CF: Automatic Prompt Optimization via Confusion Matrix Feedback  
Relevance evaluation typically relies on simple prompts such as \"Is the passage relevant for the query?\", as illustrated in Figure 1. The key challenge in this task lies in interpreting the term \"relevant\", which is inherently subjective [29]. A broader interpretation increases false positives (FPs), while a narrower interpretation results in more false negatives (FNs). Since the accuracy of relevance evaluation depends on how this term is defined, learning an appropriate interpretation from training data is crucial for building reliable IR systems. Our approach follows the reflection-based prompt optimization framework, similar to APO [10], but enhances it by explicitly utilizing the confusion matrix. As illustrated in Figure 2, we use training data annotated with human relevance judgments to compare the LLM-predicted outputs against the ground truth, generating a confusion matrix that categorizes predictions into true positives (TPs), true negatives (TNs), false positives (FPs), and false negatives (FNs). To guide the optimization of prompts, we design a structured prompt template, shown in Figure 3. This template presents the confusion matrix alongside the initial instruction and provides specific guidelines for refinement based on the observed classification errors. The instruction refinement strategy is determined according to the following four cases:  
* **False positives (FPs) are high:** The current instruction is too lenient, allowing irrelevant passages to be classified as relevant (\"Yes\"). The instruction is modified to adopt a stricter interpretation of \"answering\", requiring passages to explicitly and directly address the query without ambiguity. Figure 2: Workflow of confusion matrix-based prompt optimization. The training data and initial prompt are used to predict relevance, and a confusion matrix is computed by comparing predictions to human-labeled ground truth. Based on the confusion matrix, the prompt is optimized to improve classification performance. * **False negatives (FNs) are high:** The current instruction is too restrictive, preventing relevant passages from being correctly classified. In this case, the instruction is revised to allow for more flexibility, such as accepting partial answers, inferred connections, or indirectly useful information as valid responses. * **Both false positives (FPs) and false negatives (FNs) are high:** The instruction is likely unclear or inconsistent, causing unstable classification behavior. To address this, the instruction is refined by explicitly defining the threshold for what constitutes a valid answer and improving the overall clarity and specificity. * **Neither false positives nor false negatives are significantly high:** The instruction is already well calibrated. However, slight optimizations, such as polishing phrasing or improving example demonstrations, may further enhance precision and recall balance without drastically altering the instruction's intent. This structured refinement ensures that prompts are updated in a systematic, data-driven manner based on actual model behavior observed through the confusion matrix. By leveraging the full distribution of prediction outcomes--including both correct and incorrect predictions--our method enables efficient and targeted prompt optimization. A key advantage of our approach is that it achieves significant performance improvements with only a single update step, in contrast to iterative methods requiring multiple LLM calls. This is possible because the confusion matrix provides an explicit, structured guide for understanding and correcting model weaknesses. By learning the interpretation  
Figure 3: A structured prompt template for refining instructions based on the confusion matrix. The template provides the current instruction, classification results in the form of a confusion matrix, and refinement guidelines to adjust the instruction based on observed FP and FN rates.