# Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
###### Abstract  
Evaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resulting in significant computational overhead and limited practical applicability. In this paper, we propose a novel prompt optimization method that leverages LLM-based confusion matrix feedback to efficiently optimize prompts for the relevance evaluation task. Unlike previous approaches, our method systematically analyzes LLM predictions--both correct and incorrect--using a confusion matrix, enabling prompt refinement through a single-step update. Our experiments in realistic IR scenarios demonstrate that our method achieves competitive or superior performance compared to existing methods while drastically reducing computational costs, highlighting its potential as a practical and scalable solution.  
m +
Footnote †: journal: Applied sciences  
1  
e1e  
While effective in exploring diverse prompt formulations, this approach incurs substantial computational costs due to the large number of LLM calls required at each optimization step.  
Reflection-based methods, including APO [10], refine prompts iteratively by analyzing previous errors and performance trajectories [12; 13; 14; 15]. Although these methods enhance the consistency of prompt refinements and simulate human-like learning behaviors, they typically demand multiple rounds of error analysis and prompt updates, leading to high computational overhead. Furthermore, reflection-based techniques tend to focus primarily on correcting incorrect predictions, often neglecting insights from correctly predicted examples. This bias can result in suboptimal updates and overlooked opportunities for broader performance improvements.  
Despite the progress made by these automated methods, two significant challenges remain unresolved: the high computational cost associated with prompt optimization, and the potential bias introduced by focusing solely on erroneous cases. These limitations hinder the practicality and scalability of LLM-based relevance evaluation systems, especially when applied to large-scale retrieval tasks in real-world environments.  
Motivated by these challenges, we aim to develop a more efficient and balanced approach to prompt optimization for relevance evaluation. Specifically, we propose a novel confusion matrix-based method that systematically analyzes both correct and incorrect model predictions. By leveraging the full distribution of prediction outcomes, our method refines prompts through a single, targeted update, thereby reducing computational costs while maintaining or improving evaluation quality. For this purpose, we introduce APO-CF (Automatic Prompt Optimization via Confusion Feedback), a confusion matrix-guided prompt optimization framework designed for query-passage relevance evaluation. Our method minimizes the number of LLM calls required for optimization while enhancing robustness by considering the complete prediction landscape. To validate the effectiveness of APO-CF, we conduct experiments under realistic IR settings, measuring the alignment between human relevance judgments and ranking results obtained with optimized prompts. Our results demonstrate that APO-CF achieves competitive or superior performance compared to existing prompt optimization techniques, while significantly reducing computational resources.  
Through this contribution, we offer a practical and scalable solution for deploying LLM-based assessors in real-world IR systems, addressing key obstacles in computational efficiency and balanced prompt refinement.