# Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
## 7 Conclusions  
In this work, we introduced APO-CF, a data-driven prompt optimization method that leverages confusion matrix analysis to refine prompts effectively. Unlike traditional LLM-based reflection methods, which rely on self-generated feedback that may not accurately diagnose errors, APO-CF updates prompts based on systematic patterns in the training data, leading to more reliable and generalizable improvements.  
Our experimental results demonstrate that APO-CF outperforms existing automated prompt optimization methods, including APO, OPRO, and APE, across multiple LLMs. While APO tends to overfit to individual error cases and OPRO struggles to generalize beyond past prompt iterations, APE benefits from a broader search space but lacks targeted refinement. APO-CF, in contrast, effectively captures overall data characteristics, aiming to achieve balanced and meaningful updates that enhance model performance.  
One of the key advantages of APO-CF is its efficiency in real-world applications. Unlike iterative methods such as APO, which require multiple LLM calls for refinement, APO-CF updates the prompt in a single step, significantly reducing computational cost and API usage. This makes it practically viable for large-scale applications, where excessive LLM calls can be prohibitively expensive. Despite using fewer LLM calls, APO-CF achievescomparable or superior results, demonstrating that a well-structured, data-driven update can be more effective than repeated refinements.  
While our results confirm the potential of APO-CF in optimizing prompts for query-passage relevance evaluation, this study also has certain limitations. In particular, the method has been tested on a limited set of LLMs; namely, GPT-4o, GPT-4o-mini, and DeepSeek-Chat. Given the architectural and behavioral variability across LLMs, the performance and stability of APO-CF may differ when applied to other models. Therefore, the generalizability of our approach beyond the evaluated models remains an open question.  
Future work should investigate how APO-CF performs across a broader spectrum of LLM architectures and tasks. Additionally, model-specific tuning strategies and adaptive feedback mechanisms can be explored to further improve prompt robustness and ensure adaptability across diverse application settings.  
This research received no external funding.  
Not applicable.  
The raw data supporting the conclusions of this article will be made available by the author on request. The data are not publicly available because they were randomly sampled from a publicly available dataset, which is cited and linked in the main text of the article.  
This work was supported by Hankuk University of Foreign Studies Research Fund of 2025.  
The author is employed by Hankuk University of Foreign Studies in South Korea. The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as potential conflicts of interest.  
The following abbreviations are used in this manuscript:  
\begin{tabular}{l l} LLM & Large language model \\ IR & Information retrieval \\ NLP & Natural language processing \\ APO & Automatic Prompt Optimization \\ APO-CF & Automatic Prompt Optimization via Confusion Matrix Feedback \\ APE & Automated Prompt Engineering \\ OPRO & Optimization by Prompting \\ \end{tabular}