# Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
## 3 Methodology
### Applying Existing Prompt Optimization Methods to Relevance Evaluation  
To establish strong and representative baselines for our relevance evaluation task, we adopt three widely cited prompt optimization methods: APE, OPRO, and APO. These methods were originally developed for general prompt refinement tasks, but we adapt them to the specific context of query-passage relevance evaluation. They were selected to cover the two primary categories of automated prompt optimization: APE exemplifies resampling-based strategies, while APO and OPRO represent reflection-based approaches. This selection ensures a comprehensive and balanced comparison across different optimiza  
Figure 1: An example prompt for relevance evaluation. It consists of an instruction, few-shot examples, and an input. The initial prompt can be either manually designed by humans or automatically generated by an LLM. tion paradigms. While these methods have shown effectiveness in various NLP settings, they face two notable limitations when applied to relevance evaluation. First, they require a large number of LLM calls, making them computationally expensive. Many approaches iteratively modify prompts by generating multiple candidates and selecting the best-performing ones. Since each candidate requires evaluation with multiple sampled data points, the total number of LLM calls increases significantly, making real-world deployment impractical. Second, reflection-based methods primarily focus on correcting wrong predictions without fully leveraging the entire distribution of training data. While identifying and correcting mistakes is important, this narrow focus can introduce biases, leading to degradation in the overall quality of prompt performance. To quantitatively compare the computational efficiency of these methods, we follow the evaluation framework proposed by [PERSON] et al. [28], which provides a standardized methodology for estimating the number of LLM calls required during prompt optimization. Based on this framework, we calculate the minimum number of LLM calls for each method, assuming that each prompt evaluation involves 100 sampled examples to maintain consistency with prior studies. Table 1 summarizes these results. Iterative-APE [26] follows a resampling-based strategy, where the LLM generates numerous similar prompt variations and selects the most effective candidate. Unlike true optimization methods, Iterative-APE does not refine a given prompt but instead searches for an effective one from a large candidate pool. This search process involves generating 100 new prompts at each step, with 50 prompts selected for the next iteration. Given that 10 iterations are performed, this results in at least 100,000 LLM calls, making it inefficient. The reliance on exploratory resampling without systematic improvements makes this approach costly and impractical for large-scale applications. To adapt this method to our relevance evaluation task, we modified the base prompt used for paraphrasing so that it explicitly fits the binary decision setting of determining the relevance between a query and a passage. OPRO [27] adopts an implicit reflection-based strategy, where the LLM refines prompts by considering past optimization trajectories rather than explicitly analyzing errors. Unlike APO, OPRO does not provide structured error analysis but instead updates prompts based on previous performance patterns. However, this method still relies on extensive iterations (20+ steps), expanding the prompt set by a factor of 8 per iteration, leading to a total of 16,000 LLM calls. Since updates are made based only on past performance rather than the full dataset, optimization results can be unpredictable and inefficient. In our implementation, the prompt generation process was guided by relevance-evaluation-specific instructions, ensuring that the synthesized prompts align with the binary decision objective. APO [10] employs an explicit reflection-based approach, where the LLM systematically analyzes reasons for prediction errors and modifies prompts accordingly.