Pencils Down! Automatic Rubric-based Evaluation of Retrieve/Generate Systems

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-24 17:52:39

Abstract: Current IR evaluation paradigms are challenged by large language models (LLMs)  and retrieval-augmented generation (RAG) methods. Furthermore, evaluation either resorts to expensive human judgments or lead to an over-reliance on LLMs.

To remedy this situation, we introduce the RUBRIC metric, which puts information retrieval systems to the proverbial test. This metric leverages a bank of query-related test questions to quantify relevant information content that is contained in the systems' responses.
The process involves (1) decomposing the query into detailed questions, and (2) checking each for answerability using passages in the system response. Using three TREC benchmarks, we demonstrate that our LLM-based RUBRIC approach works successfully. Unlike previous LLM-based evaluation measures, our paradigm lends itself for incorporating a human-in-the-loop without the danger of over-reliance on AI or resorting to expensive manual passage-level judgments. Moreover, our evaluation is repeatable and extensible and can be scored with existing evaluation tools.

Main paper: 
 introduction: 
 The advent of large language models (LLMs) has led to a plethora of information retrieval systems that combine traditional retrieval with neural ranking and natural language generation-but it is unclear how to reliably evaluate such systems. In this paper, we propose the RUBRIC evaluation paradigm which measures to which extent the systems' responses contain information content that is relevant, concise, and complete. The evaluation paradigm should take advantage of the abilities of LLMs while ensuring that human judges have the final say in determining relevance. Moreover, we develop an evaluation paradigm that is repeatable and reusable, while avoiding the need to employ human judges for tedious tasks.Traditionally, information retrieval systems are evaluated with manual assessments. This involves human judges determining the relevance of system-generated responses to specific queries. While this method is valued for its depth of insight, manually evaluating the output of retrieval systems becomes impractical as the volume of information increases. Unfortunately, restricting the scope of evaluation, will make it hard to identify subtle quality differences between systems, potentially hindering the development of more sophisticated IR approaches.To address the drawbacks of manual evaluation, there has been a shift towards automated methods. A popular evaluation approach is to directly ask LLMs whether a passage is relevant for a query. Empirically this has been shown to work well [10, 18, 26 , -inter alia], but skepticism remains about whether LLMs can be trusted to replicate the nuanced understanding of humans in the judgment process, especially when a deep contextual understanding of complex user needs is required. Without reliable human oversight, there is no way of knowing when this problem arises. Faggioli et al. [10] discuss many issues that arise when humans are completely removed from the evaluation process.A significant challenge in current evaluation approaches is the lack of effective collaboration between human judges and AI [9] . Faggioli et al. [10] elaborates a wide-range of theoretical concerns, centered on questions of trustworthiness and reliability of LLMs now and in the future. Wang et al. [28] empirically demonstrate that LLMs exhibit unfair positional bias towards candidates displayed for evaluation. Liu et al. [17] demonstrate that evaluator-LLMs give a higher score to systems that are based on the same LLM. Some have suggested to ask human judges to verify an LLM's decision. However, Fok and Weld [11] have shown that human judges might over-rely on AI-generated rationales, negatively affecting their objectivity. In an opinion article, Faggioli et al. [9] suggest to better integrate humans into the definition of relevance. In this work, we develop such an approach.Our approach. In light of these limitations, we propose RUBRICa novel approach towards evaluating information retrieval systems. Our framework integrates LLMs and human judges, establishing a division of labor that plays to the strengths of both parties. We focus on breaking down the concept of "relevance" into a grading rubric of multiple concise questions that must be addressed in a system's response in order to be considered relevant. This yields a more structured and unbiased evaluation process. Our method is fast and efficient due to leveraging LLMs to scan all retrieved passages for answers to these test questions. At the same time, our RUBRIC paradigm puts human judges in charge of defining relevance through multiple concise test questions, thus maintaining the depth of human insight while minimizing over-reliance on AI.Defining relevance via grading rubrics. We believe that the task of breaking a larger information need into the set of concise questions is more natural for humans to accomplish than to directly judge the relevance of text. The process is akin to designing a grading rubric for essay grading. Educators routinely break down complex assignments into specific criteria or questions, allowing for a more objective and detailed assessment of student work.Similarly, in the context of IR system evaluation, decomposing each information need into distinct, answerable questions transforms the abstract concept of "relevance" into tangible criteria that can be systematically assessed. This process naturally aligns with human cognitive strengths, such as critical thinking and identifying semantic errors, enabling judges to focus on defining what constitutes relevant information through a less subjective lens.Obviously every search query needs to be associated with its own grading rubric. But once these grading rubrics are in place, our RUBRIC framework leverages the capabilities of LLMs to conduct a systematic, replicable, and efficient evaluation of the responses retrieved and synthesized by information retrieval systems. LLMs will scan through vast amounts of retrieved material, identifying and assessing the presence of answers to the predefined questions on the rubric. This process not only significantly reduces the time and resources required compared to manual evaluation but also ensures a consistent and objective application of the evaluation criteria across different systems and queries.Furthermore, the use of LLMs in this capacity supports a dynamic and scalable evaluation process, that can be replicated whenever new information retrieval systems are to be tested.To ensure that the grading rubric is complete and is indicative of relevance, our paradigm encourages human judges to inspect some system's responses along with automatic grades assigned by our evaluation paradigm. As the LLM scans system responses during grading, it can also extract free-form answers (examples in Figure 1 ) that may inform humans how to further refine the grading rubric, creating a feedback loop that continuously enhances the evaluation framework via a dialog between human judges and the LLM.Contributions. We develop an evaluation approach that, (1) is amenable to integrating humans and LLMs so that it plays to each of their strengths, (2) never requires manual passage-level relevance judgments, (3) benefits from latest advances in large language models, (4) yields reusable test collections that can evaluate (future) systems, especially those that employ language generation, (5) allows to expand the test collection post-hoc to reveal differences between systems.Experimentally we demonstrate that our rubric-based evaluation approach agrees with traditional evaluation paradigms, as quantified by rank correlation of system leaderboards from three TREC test collections. 
related work 2.1 llm-based relevance assessment: 
 While our approach does not attempt to imitate the passage-level relevance-judgment process, several recent methods studied this approach. The idea of direct grading prompts is to ask an LLM whether a passage answers the query. We include several of these direct grading prompts as baselines in our empirical evaluation. Sun et al. [25] uses this direct grading prompt to rerank passages. Faggioli et al. [10] produce automatic relevance labels for data from the TREC Deep Learning track. In 1SLs, MacAvaney and Soldaini [18] focus on evaluating passages with a DuoPrompt, that instructs an LLM to indicate which of two passages is more relevant for a query. Thomas et al. [26] empirically compare the ability of crowd workers and LLMs to perform document-level relevance judgments. They find that especially the label quality of crowd-workers is inferior to fully automatic LLM-based relevance labels. Thomas et al. are using a very detailed prompt (Figure 1 in [26] ), clarifying the role and query description and asking the LLM to comment on the query intent and trustworthiness. We study an abridged version of this prompt in the empirical evaluation.As discussed in the introduction, several voices raised critiques about using LLM's for relevance labels even with human supervision [10, 11, 17] . We provide an alternative to better integrate human judges into this process.There is a long history in the IR community to evaluate the relevance of documents by breaking down the information need into a set of "nuggets" (also called query intents, facts, or SCUs) that can each be evaluated independently [16] . The common definition of a nugget is "the smallest portion of text that constitutes relevant information in and of itself" [21] .With the advent of LLMs, nugget-style evaluation is being revamped, most recently in the TREC Crisis Facts track [19] : Judges are asked identify atomic "facts" (similar to nuggets). System responses are analysed for mentions of these facts, either via a boolean OR or with an embedding-based method.Our proposal is related in that we break down the information need into a set of rubric elements that represent relevance. In Section 5.1 we discuss a variation of our approach that use rubrics of nuggets instead of questions.The idea of basing an evaluation on a bank of test questions has been widely discussed in literature on summarization [3] . Eyal et al. [8] suggest an system evaluation score that is based on the number of questions that a question answering system can correctly answer using the system response-a principle that our approach follows.Many approaches use a Cloze-style idea to generate questions from a given gold summary or source text, generating multiplechoice questions [12] , questions with exact-match answer verification [6] , or entities-centric questions [8, 27] .In information retrieval evaluation there is no source text or gold summary to generate questions from. Sander and Dietz [23] avoid this problem in the EXAM Answerability Metric by using human-designed multiple-choice test questions that would indicate relevance for the search queries of TREC CAR Y3. They use a question answering system to automatically check whether system responses can answer their test questions.Where EXAM uses a pre-neural question answering system that is limited to multiple-choice questions, our RUBRIC approach builds on the advent of modern LLMs to permit open-ended questions. Additionally, we offer automated support for creating test questions.Many approaches integrate passages, questions, and LLMs in some form. This includes improving question answering via retrievalaugmentation [14, 22] . Improve fact verification, by breaking each claim down into several questions [30] . Exploiting the self-verification ability of LLMs to improve the reasoning [29] . Evaluating the quality of LLMs with multiple tests [2, 15] . Arabzadeh et al. [1] develop an approach to improve LLM embeddings, by generating "liar" questions that cannot be answered with the given context. 
approach overview: 
 Our approach is based on the idea of developing a grading rubric for each query: a set of questions that a good system response should be able to answer. Our proposed RUBRIC metric quantifies the coverage and quality of relevant information content provided in system responses. These responses could be retrieved from a corpus, generated from scratch, or generated with retrieval-augmentation. The systems are graded based on query-specific grading rubrics of test questions, tracking which rubric elements are addressed and how relevant, complete, and accurate the provided answer is. The more test questions can be addressed well, the higher the RUBRIC evaluation score of the system.We remark that the grading rubrics are intended to be secret: systems under evaluation should not have access to the grading rubric when responding to the search query.Inputs. Our RUBRIC evaluation system assumes the following inputs:(1) A set of queries, optionally with query subtopics.(2) A set of system responses, which can come in the form of a passage ranking or a list of generated passages. (3) If available, a grading rubric for each query to be refined.Outputs. As part of the evaluation, our approach will operate in three phases (depicted in Figure 1 ), creating the following outputs: 1. Designing grading rubrics: A process of creating a rubric of test questions for each query, each question representing one important piece of information that should be addressed in the system's response. Our framework supports grading rubrics that expect unstructured answers as well as those verifiableQuestion Generation: TREC DL Prompt Question Generation: TREC CAR Y3 Prompt Break the query '{query_title}' into concise questions that must be answered. Generate 10 concise insightful questions that reveal whether information relevant for '{query_title}' was provided, showcasing a deep understanding of the subject matter. Avoid basic or introductory-level inquiries. Keep the questions short.Explore the connection between '{query_title}' with a specific focus on the subtopic '{query_subtopic}'. Generate insightful questions that delve into advanced aspects of '{query_subtopic}', showcasing a deep understanding of the subject matter. Avoid basic or introductory-level inquiries.Grading: Self-rating Prompt Can the question be answered based on the available context? choose one: -5: The answer is highly relevant, complete, and accurate. -4: The answer is mostly relevant and complete but may have minor gaps or inaccuracies. -3: The answer is partially relevant and complete, with noticeable gaps or inaccuracies.-2: The answer has limited relevance and completeness, with significant gaps or inaccuracies.-1: The answer is minimally relevant or complete, with substantial shortcomings.-0: The answer is not relevant or complete at all. Question: {question} Context: {context} with gold standard answer keys. While human judges should design the grading rubric, if desired, the rubric creation can be seeded with automatically generated grading rubrics. This is discussed in Sections 4.1. 2. Graded system responses: All passages in system responses are automatically graded via an LLM: Each passage is scanned for information content that addresses each rubric element, assessing the quality of the provided information on a scale from 0 (worst) to 5 (best), as elaborated in Section 4.2.an evaluation score that is based on how well rubric elements are addressed in the system's response. Our RUBRIC metric derives a relevance label for each passage, based on the best addressed rubric element and computes each system's evaluation score with trec_eval based on these relevance labels as described in Section 4.3.Human-in-the-loop. We envision that human judges focus their efforts on designing grading rubrics (Phase 1), while LLMs are automatically grading system responses (Phase 2). Next, human judges should inspect some the grading results to improve the grading rubric by reformulating, adding, or deleting questions and adjusting the LLM's prompt/few-shot exemplars/fine-tuning setup to provide more accurate grades. Once this process is complete, system evaluation scores are computed based on the graded responses, in a fashion similar to trec_eval.While our workbench software is designed to incorporating a human-in-the-loop, 2 in this article we focus on the feasibility of the automatic part of this evaluation paradigm and demonstrating the verification process in Section 6.7. We leave the human subject study to future work.Reusability of test collections. As new systems are developed, these can be graded and evaluated with the developed grading rubrics. By dividing the evaluation process into rubric generation and automatic grading, our approach avoids the problem of unjudged passages (also called "holes" [18] ) in test collections. RUBRIC grading pipeline can be applied to update the "qrels" file whenever new passages are retrieved or generated by new systems.This process allows our evaluation paradigm to be easily incorporated into shared tasks of evaluation venues like TREC, NTCIR, or CLEF, as the only the "qrels" file needs to be distributed to participants to develop systems with the RUBRIC metric.Extensibility of test collections. Traditionally, test collections are created by pooling system responses, and then frozen once completed. However, as increasingly better systems are developed, these may obtain new information content that should have been incorporated in the grading rubric, but were previously not known by human judges. The RUBRIC evaluation paradigm readily supports modifying the grading rubric in light of new information, to be deployed as an updated version of the test collection.Below in Section 4, we describe the best performing implementation of this RUBRIC framework, before detailing alternative implementations and baselines in Section 5 which are included in the experimental evaluation. 
generated rubrics for self-ratings 4.1 phase 1: generated grading rubrics: 
 While human judges should focus on the rubric creation task, our system can provide an initial seed rubric for each query via a generative LLM.In our experiment, we use GPT 3.5 to obtain initial grading rubrics (to be refined by human judges). The prompt is designed to elicit a set of concise, insightful test questions 𝑟 based on the query, tailored to specific goals of the IR task and domain. For TREC DL we ask to break the question query into concise sub-questions. In application to TREC CAR Y3, we ask for questions that explore the connection between the broad query with a specific focus on each Did rock n roll have regional variations or was its impact worldwide? 𝑟 10 Were there any significant changes in the music industry that paved the way for rock n roll? Query title: The Integumentary System Query subtopic: Structure of the Skin r1 How does the epidermis, dermis, and hypodermis work together to provide protection, sensation, and regulation for the body? r2 Can the integumentary system be compromised by diseases and conditions, and if so, how does this impact the health of the skin? r3 How does the skin act as a barrier against pathogens and other foreign substances? Passage: b95bf325b7fdacac183b1daf7c118be407f52a3a The skin is the largest organ in the human body. Skin is made up of three layers, the epidermis, dermis and the fat layer, also called the hypodermis. The epidermis is the outer layer of skin that keeps vital fluids in and harmful bacteria out of the body. The dermis is the inner layer of skin that contains blood vessels, nerves, hair follicles, oil, and sweat glands. Severe damage to large areas of skin exposes the human organism to dehydration and infections that can result in death. TREC judgment: 3 (MUST be mentioned) subtopic. The complete prompts used in the experimental evaluation are listed in Table 1 .From these prompts we obtain test questions asking for unstructured free-form answers, such as given in Figure 2 .In this phase, we use an LLM to identify all relevant material in passages of system's responses. We consider each each question 𝑟 on the rubric and track results per passage 𝑝 as grade(𝑟, 𝑝), a numerical grade.To initialize the grading phase, we pre-process system responses to obtain a set of paragraph-sized plain text passages, each up to 400 tokens in length (associated with a unique passage_id).Grading by self-rating. Each grade(𝑟, 𝑝) quantifies how well rubric question 𝑟 ∈ 𝑅 𝑞 is addressed in passage 𝑝 ∈ 𝑃 on a scale from 0 (worst) to 5 (best), as depicted in Figure 1 . We lean on the ability of modern LLMs to match language patterns and ask the LLM to self-rate the answerability of question 𝑟 using each passage 𝑝 as context using the grading prompt given in Table 2 .Critical elements of the prompt is to ask "how relevant, complete, and accurate" the answer is, and to emphasize that we seek answers with the available context. The exact phrasing of the prompt was suggested by GPT 4 for use with the FLAN-T5-large model.We observe that LLMs are very reliable when matching concise information content according to the grading rubric, a phenomenon Weng et al. [29] calls self-verification behavior. We find that in the vast majority of cases, modern LLMs, such as FLAN-T5-large, indeed respond with a numerical code between 0 and 5. In the remaining (rare) cases, we assign a rating of 1 by default or 0 when expressions of unanswerability are encountered. 3To support human judges to oversee this process, we complement the numerical self-rating grade with an extracted textual answer (depicted in small font in Figure 1 ). In the manual verification (cf. Section 6.7), we find that numerical grades mostly line up with extracted answers.In contrast to Sander et al. [23] , who evaluate answerability with multiple-choice questions, our process avoids many technical difficulties in matching gold answers in the light of different ways to phrase a correct answer. As demonstrated in Figure 1 , many different answers are correct and extracting such answers is helpful for manual verification. Furthermore, if desired, the extracted answer can also be used to complement the self-rating process with verification of the LLMs's answer against the gold answer key.Based on the grades of each passage/question combination, we can derive a RUBRIC evaluation score for each system, which is averaged across all queries in the test set.For each query, we associate each passage 𝑝 with a relevance label according to the best grade achieved on any of the rubric elements 𝑟 . relevance-label(𝑝) = maxEQUATIONWe expose these RUBRIC-based relevance labels as a trec_eval compatible relevance file (aka "qrels" file). This permits to implement our novel Rubric Score evaluation metric with an established evaluation tool-chain such as trec_eval, building on traditional evaluation metrics. By configuring trec_eval to use the multirelevance grading threshold 4 𝜏, we only count passages as relevant that obtain at least a minimum grade of 𝜏 on any of the rubric elements. Moreover, our provided software can be configured to emit a relevance label based on the best grade achieved by at least 𝑚 rubric elements instead of just the best.Empirically we find that the RUBRIC evaluation metric obtains a high correlation with official leaderboards of all three test sets. 
rubric variations and baselines: 
 In this section we elaborate an alternative variation as well as some state-of-the-art baselines we explore in the empirical evaluation in Section 6.Instead of basing the grading rubric on questions, we can build on work of nugget-based evaluation [16, 21, 24] and create grading rubrics 𝑅 𝑞 comprised of nuggets or key facts.The only difference lies in the changing prompts. For generating nugget-based grading rubrics, the prompts listed in Table 1 need to be adjusted to ask for "key facts" instead of questions. For grading in Phase 2, the prompts in Table 2 need to ask "to which extent a key fact is covered" in the given passage. (We list the complete prompts in the online appendix.)In the experimental evaluation (Section 6.2), we find that the nugget-based approach works less well. We believe the main reason is that LLMs are trained on a wide range of question answering benchmarks, but only very few test collections with nuggets.The EXAM method [23] uses a pre-neural question answering system from AI2, which was designed to answer multiple-choice questions with given context for questions in the style of the ARC and TQA datasets. The answer is verified as correct when the question answering system responds with the correct choice, resulting in binary grades per passage and question. The downside is that this question answering system was difficult to set up, 5 In contrast, our RUBRIC approach uses a modern LLM-based question answering system that is easy to integrate.A very competitive approach is to ask an LLM whether a passage 𝑝 is relevant for a given query 𝑞-without intermediaries such as test questions or nuggets [10, 20, 25, 26] . These methods have been empirically shown to be very capable, hence we include these in the experimental evaluation as an upper-bound reference. However, as described above [11, 17] , it is not possible to incorporate the humans into this direct grading paradigm without (1) the danger of judges' over-reliance on AI during verification [11] or (2) the need for manual passage-level judgments. 
experimental evaluation (of the evaluation metric) 6.1 experimental setup: 
 Evaluation methodology. We study our approach on three TREC datasets by providing an alternative evaluation of systems submitted to the respective TREC tracks. We demonstrate that our method reproduces the official leaderboard results. A higher rank correlation in terms of Kendall's tau and Spearman's rank correlation coefficient implies better evaluation paradigm.Additionally, we compare our automatically predicted passage relevance labels to manually produced official TREC judgments, in terms of count statistics and of Cohen's kappa inter-annotator agreement which corrects for chance agreement.Datasets. We use the following test collections: TREC DL 2019 [5] : Using 43 queries in the question-form from the Deep Learning track, harvested from search logs. The system's task is to retrieve passages from a web collection that answer the query. The official track received 35 systems, metrics are NDCG@10, MAP, and MRR. TREC DL 2020 [4] : Similar setup as the previous Deep Learning track, but with 54 additional queries and 59 submitted systems. TREC CAR Y3 [7] : Comprising 131 queries and 721 query subtopics from the TREC Complex Answer Retrieval track. These were harvested from titles and section headings from school text books provided in the TQA dataset [13] . The system's task is to retrieve Wikipedia passages to synthesize a per-query response that covers all query subtopics. Official track metrics are MAP, NDCG@20, and R-precision; of 22 systems were submitted to this track, several have identical rankings. We use 16 distinguishable systems used by Sander et al.We compare several variations of our RUBRIC paradigm as well as a range of established baselines. RUBRIC: Represents our proposed implementation described in Section 4 using generated grading rubrics (Phase 1), grading with self-ratings (Phase 2), to derive RUBRIC relevance labels. We obtain ten questions for each query-subtopic in TREC CAR Y3, and each query in TREC DL. Nugget RUBRIC: Same as RUBRIC, but using a rubric of nuggets instead of questions.While any LLM can be used for grading in our evaluation paradigm, in this work we focus on affordable LLMs and use GPT 3.5 6 for rubric generation and the recent FLAN-T5-large model with the text2text-generation pipeline from Hugging Face. 7 This allows to complete RUBRIC grade annotations for TREC DL 2019 within 1 hour on an NVIDIA A40 GPU. We grade all passages in official judgments and the top 20 of all submitted system runs. This results in 85,329 passages in TREC CAR Y3, 9,260 passages in TREC DL 2019, and 11,386 passages for TREC DL 2020.To demonstrate the quality of our approach we compare to several established baselines. EXAM [23] : Using a pre-neural question answering system on the multiple choice question from the TQA dataset, as described in Section 4.3. Results are taken from the original the paper (available for TREC CAR Y3 only).Additionally we compare to the following direct grading prompts (cf. Section 5.3) using the same LLM as above (FLAN-T5-large).Thomas [26] : "Instruction: You are a search quality rater evaluating the relevance of passages. Given a query and a passages, you must provide a score on an integer scale of 0 to 2 with the Leaderboard Correlation. We compare different evaluation paradigms by how well their leaderboards correlate with the official leaderboard of respective datasets. Relevance labels are derived from RUBRIC grades and the direct grading prompts, to be used as "qrels" files. Each system is scored using trec_eval with these relevance labels using official evaluation metrics recommended for each dataset. The leaderboard ranks all systems based on their evaluation score. The correlations between such leaderboards and the official leaderboard of the TREC track is measured with Spearman's rank correlation and Kendall's tau correlation. Both correlation measures range from -1 (worst) to 1 (best) with 0 referring to uncorrelated leaderboards. We use a scikit-learn implementation of both rank correlation measures, where tied systems are assigned the average of their ranks.Significance testing. Significance tests do not apply to these rank correlation measures. Instead we measure standard error bars for each system's evaluation score, then estimate their impact on the rank correlation results, which is about ± 0.05. Best methods marked in bold-italics, methods within that range are considered equally good (marked in bold).Table 3 presents how well system rankings on the leaderboard under each evaluation metric correlated with the official leaderboard of TREC DL. We evaluate the generated relevance labels using official track metrics normalized discounted cummulative gain (NDCG@10), 8 average precision (MAP), and reciprocal rank (MRR). Since both Spearman's and Kendall's rank correlation results paint the same picture, we omit some cases here, but provide full results in the online appendix. Across all evaluation results in both 2019 and 2020 test sets, we find that our proposed RUBRIC method is consistently among the best performing metrics (e.g., 6/8 wins for self-ratings of 𝜏 = 5). For illustration, we provide an excerpt of the RUBRIC leaderboard for TREC DL 2020 in Table 4 .We find that using question-based rubrics obtains slightly better results than nugget-based rubrics (cf. Nugget RUBRIC). We suspect that this is due to the abundance of question answering datasets used to train LLMs, while only few datasets with nuggets are available.Many methods obtain Spearman rank correlations above 0.9 (a metric ranging from -1 to +1), indicating that empirically all these LLM methods are strong contenders for IR evaluation paradigms. However, it is more natural to integrate a human judge in the RUBRIC paradigm.We demonstrate that our RUBRIC approach can be used to evaluate systems that use natural language generation. We use GPT-4 and 3.5 to develop six systems that generate system responses in response to all TREC DL 2020 queries using the following prompts. GPT*-wiki: "Generate a 1000-word long Wikipedia article on {query_title}" GPT*-web: "Generate a web page for {query_title}'' GPT*-question: "{query_title}?" These systems were not submitted as systems to the TREC track, and hence, were not manually assessed. The evaluation results are integrated in Table 4 , marked with "*". We demonstrate that MRRbased Rubric Score is able to compare these methods on methods on the official leaderboard. The ranks are shifted, because the official leaderboard does not rank our six methods.We observe that our GPT-question methods are placed on top of the leaderboard, while the GPT-web methods are placed below rank 52, GPT-wiki place around rank 40. When using a recall-based measure such as MAP, we find that GPT-wiki based method place best, as these responses cover a wide-range of facts (results omitted). We conclude that despite our GPT methods not participating in the TREC judgment pool, we can observe their relative value. Furthermore, we find that only few submitted systems swap ranks between these two leaderboards.We analyze the grade/judgment agreement between manual TREC DL 2020 judgments and predicted relevance labels in Table 5 . Cohen's 𝜅 inter-annotator agreement referring to boxed cells, confirming a good per-passage correlation. According to track guidelines [5] , judgment level 1 indicates a non-relevant passage.Tallying each relevance label against judgments, we demonstrate that relevant judgments (2 and 3), correlate the highest with a grade of 4, and judgment level 0 correlate the highest with a grade of 0. We also observe this in the manual verification (Section 6.7).When collapsing RUBRIC grades to a binary scale (4 and 5 relating to relevant judgments 2 and 3) we confirm good correlation with a Cohen's 𝜅 of 0.25. We compare to the direct grading prompt of Sun et al. [25] , which is obtains best NDCG@10 on TREC DL 2020 (Table 3 ). Their prompt yield slightly lower Cohen's 𝜅 of 0.23 according to our reproduction. In comparison between these methods our method misses 272 relevant passages, but avoids to assign a relevant label to 1115 non-relevant passages. Hence, we confirm that RUBRIC is a competitive method. We expect to see further improvements when humans are integrating into this paradigm.As displayed in Table 3 , our proposed RUBRIC method continues to provide strong results on the TREC CAR Y3 dataset, obtaining near-perfect correlation with the official leaderboard and a good grade/judge inter-annotator agreement of 0.38 (Table 6 ).In contrast, LLM-based direct grading prompts Sun, FagB, HELM, and Thomas, which were head-to-head on TREC DL, are now dropping to a Spearman's rank correlation below 0.7. We speculate that the broad topical queries of the CAR collection, benefit from breaking the information need into different questions that can be verified individually.Furthermore, we outperform Sander's EXAM method [23] by lieu of a modern LLM-based grading method that can handle openended questions. In their paper, Sander remarks that the rank correlation between MAP and RPrec metrics based on the official judgments obtains about 0.94 of Spearmans' and 0.86 in Kendall's tau rank correlation. We point out relevance labels produced that our RUBRIC approach reach this level as well. 3 depicts how three of the generated questions are addressed by a passage that was manually judged as highly relevant by TREC assessors.For TREC DL 2020 query 940547, we analyzed 20 high ranked passages with all 10 test questions and manually verified the resulting 200 automatically assigned grades. The average rubric grade is 2.48, which coincides with the average grade on the range from 0 (worst) to 5 (best). We find that most of the time either grade 0 or 4 is awarded-grade 5 only 24 times.The grade distribution differs per test question, with passages being awarded generally a higher grade on the test questions of widely mentioned facts such as music styles (3.9) and pioneers (2.7). As expected, some rubric questions that address less prevalent facts obtain a lower grade across all passages. This is the case for the rubric question on social factors (average grade 1.8) and impact of technological developments (0.8).Focusing on relevant rubric grades (4 and 5), in about 75 cases the extracted answer was indeed a correct interpretation of the passage and the rubric question (versus 41 incorrect). Questions about pioneers, influences, events, and recordings are nearly perfectly answered, whenever the answer was contained in the passage.Most of the extraction mistakes are due to misinterpretations of the question, for example for the question about whether rock'n'roll evolved from existing music styles, for 8 of 20 passages the extracted answer said "rock n roll"-a non-answer. For the question on whether its impacts were worldwide, for 16 passages the extracted answer is "worldwide" without the passage elaborating this fact. We suspect in the latter case, the LLM is answering this question from memory instead of the provided context. In the case of several mistakes we found that the passage indeed discussed the question (deserving a high grade), but the extracted answer was incorrect.To identify spurious test questions we count how often a negatively judged passages obtains a positive grade for a question. In this example, spurious questions are whether rock'n'roll evolved from existing styles (116 passages) and about the exact start of rock'n'roll (102 passages).Furthermore, we can analyze passages with relevant judgments that are not associated with a positive grade. This would imply that additional questions should be added to the grading rubric. In the example above, no such passages exist.However for query 1108651 "what the best way to get clothes white", we find a few relevant passages about bleach, that technically did not answer the rubric question "How does soaking clothes in bleach affect their whiteness?". This question would be better reformulated to "Will bleach turn clothes white?" Detailed data of this verification is available in the online appendix (URL in abstract). 
conclusion: 
 With RUBRIC we are proposing an alternative evaluation approach that integrates human judges, but not to create or verify manual passage-level relevance judgments. Instead, a grading rubric is created as part of the topic development, envisioning that each question addresses one important piece of information content. As a result, whenever such questions are answerable with responses from a retrieval / generation system, we conclude that the system provides relevant information.Using three TREC data sets, we demonstrate that (1) our proposed approach can reproduce official TREC leaderboards nearly perfectly (Spearman's rank of 0.97); and (2) it is a strong contender in comparison to other recent LLM-based relevance label predictors [10, 15, 25, 26] . In contrast, RUBRIC offers a clear path towards integrating a human-in-the-loop, by supporting the refinement of the grading rubrics, which is how relevance is defined by judges.We believe that further research will improve the question bank generation, as well as study the positive effects of this approach on the quality, cost, and satisfaction of human judges. In this work we demonstrate that this approach is a worthwhile option to consider.We hope that by providing an easy means to use the RUBRIC evaluation metric via trec_eval, we offer an evaluation system that can be easily adopted by future IR evaluation tracks, offering organizers an avenue to reduce assessment costs and to obtain reusable test collections for generative information systems. 
 
 Appendix: URL to online appendix in abstract."unanswerable", "no", "no answer", "not enough information", "unknown", "it is not possible to tell", "it does not say", or "no relevant information".Conference'17, July 2017, Washington, DC, USA Anon.We were unable to install the question answering system used in the original EXAM method.gpt-3.5-turbo-instructhttps://huggingface.co/google/flan-t5-largeWhile for NDCG, trec_eval uses multi-relevance grades, the grading threshold 𝜏 is ignored, yielding same evaluation score. 


Decision: The authors propose an LLM-based method for the automatic evaluation of search results in form of generated texts but also for ranked results. The idea is related to the previous EXAM metric in that it basically also asks how many questions related to a given information need can be answered from a search result (generated text or ranked documents).

Automatic evaluation attracted quite some interest recently, so that the paper is timely. The presented idea in general also is interesting. For my metareview, I disregard the 'weak reject' review as it somewhat indicates that the reviewer misinterpreted the paper's contribution ("the core idea of this paper is to [...] enhance the results based on these subanswers [...] normal routine for RAG"). 

Overall, I suggest to accept the submission but would hope that some important points will be clarified / included in a final version (cf. the individual comments in the reviews). Most importantly, the current experiments focus solely on the correlation of evaluation results using the newly proposed scheme to evaluation results based on previous schemes. Given that there is some level of correlation, the next natural question is how much effort is needed to run the evaluation with RUBRIC compared to other ideas. Not having anything on that end at all in the paper really is a pity. The paper could easily be a clear 'accept' candidate with effort-oriented results. In case of "competetiveness" to for instance traditional TREC assessor effort the applicability would be indicated, in case of more effort than for traditional schemes, the RUBRIC idea could be an interesting starting point to improve the required human effort in future work. I really hope that at least some results on the effort / efficiency angle will be included in case of acceptance as potential readers of the paper will really want to learn about the required effort.

Reviews: review: main_review: main_review: To my understanding, the core idea of this paper is to convert a query into multiple subqueries and then enhance the results based on these subanswers. This idea is a normal routine for RAG which is not new. After reading this paper carefully, I am still not very clear what the unique contribution of this paper is. Lack of novelty is my concern for this paper. In addition, this paper is more like an experimental report than a research paper for proposing a new method. 

The writing of this paper is good. However, some typos can be found in the paper. For example, on page 5, "We consider each each question 𝑟 on the rubric" ==> "We consider each question 𝑟 on the rubric".
main_review: main_review: The authors posit that RUBRIC collections are more reusable because the grading rubrics can be updated when new relevant documents are found.  This would clearly bias evaluation results towards the system producing those documents.

In figures 2 and 3, "generated grading rubrics" are shown, but the text seems to imply that after the rubrics are generated, they are reviewed and refined manually by a person.  Are the rubrics shown the generated ones, or the edited ones?

Section 4.2 states that certain elements of the prompt text are "critical", but does not explain why or what this means.  Since (as they write) the prompt was suggested by an LLM, how do they know it is critical?

Footnote 5 says that they were unable to install the model for EXAM, but EXAM baselines were still produced.  I'm not sure how that can be... if they couldn't use the model, how were the results produced?  Perhaps they used another model that they could install?

A comparison is made to Thomas et al's prompts, which are significantly longer and were developed and refined on their own version of GPT-4.  Might that argue that the prompts are not ideal for the models used by the authors of this paper?

I would think that just as we worry about assessor agreement, we would also worry about model agreement, not just agreement of the model to the assessor, but agreement among models.  Why were these particular models (GPT-3.5, FLAN-T5) chosen for this paper?
main_review: main_review: •  Readability and Understandability: The paper is well-written to a large extent, with clear problem statement, methods, and results. However, the understandability can be improved with algorithms or diagrams describing the method.

•  Relevance of the problem: The problem addressed by the paper is highly relevant. As information retrieval systems, particularly those augmented by large language models, become increasingly complex, traditional evaluation methods struggle to keep pace. The paper addresses a critical need in the IR community for more scalable, efficient, and effective evaluation methods that reduce reliance on extensive manual labor while still incorporating human judgment.

•  Originality: The RUBRIC metric introduces an original approach to evaluating information retrieval systems by combining automated processes with human oversight. The method of decomposing queries into a set of detailed questions and using these to assess the relevance and completeness of system responses is innovative. This approach not only leverages the capabilities of large language models but also maintains human involvement in the evaluation process, striking a balance between automation and human insight.

•  Interest of the Results: The results are interesting and demonstrate the potential of the RUBRIC metric to effectively evaluate information retrieval systems. The paper shows that the RUBRIC approach correlates well with traditional evaluation paradigms using TREC benchmarks, suggesting that it can serve as a viable alternative to more labor-intensive manual methods. The ability to extend and reuse the test collections also adds significant value to the approach.

•  Interest to ICTIR Community: The paper is likely to be of high interest to the ICTIR community. The challenge of efficiently evaluating complex IR systems is a pressing issue, and the community is actively seeking solutions that can handle the nuances introduced by advanced AI technologies. The integration of LLMs with human judgment in a structured evaluation framework addresses a topical area of research and practice in information retrieval. The method's scalability, repeatability, and potential to reduce the burden of manual evaluation while maintaining quality are particularly compelling aspects that would attract attention from researchers and practitioners alike.
 
• Impact: RUBRIC offers a scalable, repeatable, and less subjective evaluation method for complex IR systems. Its integration of human insights and LLMs addresses a crucial need in the era of advanced AI, potentially setting a new standard for evaluation and influencing future research.
main_review: main_review: - It is unclear why the authors limited the number of subquestions to 10, as not all queries may require the same number of subquestions. 

- Why did the authors decide to to conduct experiments in a zero-shot setting. The results in Table 1 and for the prompts could potentially differ with in-context learning examples.

- The paper heavily references the Trec_eval tool in section 3.3, which might limit its applicability. There are other IR evaluation tools that would be used for implementation. I think these details should have moved to implementation details and the approach overview should not be limited to a tool.

- The authors mention the "best performing implementation" of the RUBRIC framework in section 3 without clear justification for this designation.

- The use of different LLMs throughout the experiments (gpt to flan-T5 )  is inconsistent, raising methodological concerns.