# Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
## 4 Experimental Setup
### Dataset and LLMs  
Dataset. For our experiments, we utilize the test sets from the MS MARCO TREC DL passage datasets spanning three years ([[https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019](https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019)]([https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019](https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019)) (accessed on 15 February 2025); [[https://microsoft.github.io/msmarco/TREC-Deep-Learning-2020](https://microsoft.github.io/msmarco/TREC-Deep-Learning-2020)]([https://microsoft.github.io/msmarco/TREC-Deep-Learning-2020](https://microsoft.github.io/msmarco/TREC-Deep-Learning-2020)) (accessed on 15 February 2025)); [[https://microsoft.github.io/msmarco/TREC-Deep-Learning-2021](https://microsoft.github.io/msmarco/TREC-Deep-Learning-2021)]([https://microsoft.github.io/msmarco/TREC-Deep-Learning-2021](https://microsoft.github.io/msmarco/TREC-Deep-Learning-2021)) (accessed on 15 February 2025)). These datasets contain query-passage pairs, where relevance is assessed by human annotators. To demonstrate the robustness and generalizability of our findings, we select these three specific years because they have been widely used in previous work for relevance evaluation [25]. Evaluating across multiple years helps mitigate dataset-specific biases and ensures that the effectiveness of our method is not limited to a single test condition. As summarized in Table 2, we sample 500 data points from each year's dataset for evaluation, considering the computational cost of processing the full dataset. The 2019 dataset is used for training, while the 2021 dataset serves as the validation set, which is utilized when selecting the top-\(k\) prompts after each expansion step. Since APO-CF performs a single update, it does not require validation data. For the test phase, we use the 2020 dataset to evaluate the final performance of optimized prompts. To ensure robustness and mitigate variance, we create three independent test samples of 500 data points each from the 2020 dataset. The final performance is reported as the average across these three test samples. While using full test sets would be ideal, it is computationally intensive. Therefore, we empirically validate that using three random samples of 500 points each yields results that closely match the full dataset results. This comparison is provided in Appendix A. The TREC DL 2019 dataset consists of 9260 query-passage pairs, with the relevance between each pair assessed by human evaluators. Similarly, the TREC DL 2020 and 2021 test sets follow the same format, comprising 11,386 and 10,828 query-passage pairs with relevance judgments, respectively. Relevance in these pairs is rated on a 4-point scale: \"Perfectly relevant\", \"Highly relevant\", \"Related\", and \"Irrelevant\". For binary classification tasks, we simplify this 4-point relevance scale to a binary \"Yes\" or \"No\" judgment. Specifically, the categories of \"Perfectly relevant\" and \"Highly relevant\" are consolidated into a \"Yes\" category, while \"Related\" and \"Irrelevant\" are classified as \"No\". LIMs. For our experiments, we utilize GPT-4o, GPT-4o-mini, and DeepSeek-Chat as the large language models (LLMs). To access GPT-4o and GPT-4o-mini we use the OpenAI API ([[https://openai.com/api/](https://openai.com/api/)]([https://openai.com/api/](https://openai.com/api/)) (accessed on 15 February 2025)), while for DeepSeek-Chat we rely on the DeepSeek API ([[https://api-docs.deepseek.com/](https://api-docs.deepseek.com/)]([https://api-docs.deepseek.com/](https://api-docs.deepseek.com/)) (accessed on 15 February 2025)). These APIs provide seamless access to the models' capabilities while ensuring reproducibility in experimental settings. To minimize output variability, we set the temperature parameter to 0 across all API calls, ensuring deterministic outputs that simplify analysis and comparison across models. We selected these models to represent a range of capacities and architectural backgrounds.