ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

Authors: Yaswanth Narsupalli, Abhranil Chandra, Sreevatsa Muppirala, Manish Gupta, Pawan Goyal
Primary area: foundation or frontier models, including LLMs
Keywords: Multi-agent framework, reasoning, evaluation, multimodal, hierarchy, LLMs, VLMs

Abstract:
Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves.  We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more test-time compute efficient solution. ReFeR-Lite is $\sim12-14\times$ more test-time compute efficient than previous works while being comparably accurate to ReFeR-Turbo.

Decision: reject
Reviewer scores: 5, 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729805633489

thread_type:
official_review

<thread_object>
creator: reviewer_u6dr

<content>
summary:
The authors propose an LLM-based automatic evaluation framework that draws inspiration from the academic peer review process: independent LLMs score and comment a generation, and then these scores, comments, and the original generation are fed to an "area chair" LLM that produces a final score. They evaluate this method by assessing correlation with human annotations on existing labeled datasets. They compare to existing LLM-automatic evaluation methods.

strengths:
- Automatic evaluation is an impactful problem, and I think the design space remains wide open. The grounding of evaluation in human annotations is great.

- The inclusion of an analysis section is great; I have some questions about Sections 5.1 and 5.3 (see Weaknesses / Questions), but Section 5.2 was very clear.

- I appreciate that the appendices are filled out, and that the authors included their code at submission time.

weaknesses:
My main concern with this method is that in evaluations, the methods are not test-time compute matched. One explanation for ReFeR's stronger performance in Table 2 is that the system simply spends more inference compute than the other methods; this could mean that the peer review structure is not actually integral to performance. Since the paper's contribution is in the peer review structure, I think this is an important question to clarify.

- I appreciate that the authors compute a Cost column in Table 2, but this is only the AC cost. The authors justify this column as the monetary cost of querying an OpenAI API. However, in practice, this is not the only cost we have in evaluation: test-time compute (number of FLOPs processed) would be a more valid metric. (After all, in reality the authors are paying Together AI for inference of the open weight models as well.) I suspect that if we looked at test-time compute, ReFeR becomes significantly more heavyweight than other methods: after all, we need to inference 4 large language models!

- To make the performance - inference compute tradeoff more apparent, could the authors make a scatterplot with the corrected inference compute metric as the x-axis? An even stronger visualization would modulate n (or some other test-time compute parameter) for each method and plot how evaluation performance changes with increased test-time compute; we can then compare ReFeR to G-Eval, for example, by checking whether the ReFeR curve consistently sits above the G-Eval one.

    - I'd suggest drawing this up in a setting where the AC model is also an open-weight model; this way, we know all parameter counts.

I also would like to ask the authors to provide results for a few additional baselines: (1) just the AC, and (2) a simple average of the peer agent scores. Could the authors add GPT-4o-mini / GPT-4o (the AC model) and a simple average of scores as baseline rows in the "Peer Agents" section of Tables 2 and 3? I appreciate that Section 5.3 offers some insight into these questions, but I would like to directly see the human correlation scores of these baselines.

questions:
I would love to see the plots asked for in the Weaknesses box, and I will increase my score if appropriate. Additionally:

- I had some questions about how the G-Eval and Analyze-Rate baselines were configured; it would be good to move this information into the main text.
    - Is the backbone model matched to the AC model used in ReFeR? (GPT-4o-mini for NLG/Reasoning, GPT-4o for multimodal)?
    - Was n=20 for these?

- Similarly, was the single-agent CoT model matched to the AC model? Which agents were used in the multi-agent baseline?

- Figure 2 was a bit challenging to parse: which model is the AC model for the "Llama + Gemma + Nemo" point, and which models represent the "3 Peers + Mixtral" point?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730245190130

thread_type:
official_review

<thread_object>
creator: reviewer_so4x

<content>
summary:
The paper introduces ReFeR, a hierarchical framework for evaluating generative models using a multi-agent system inspired by peer review. It leverages large language and vision-language models to assess outputs, providing more accurate, explainable evaluations than existing benchmarks. ReFeR is also good at reasoning tasks and offers two variants: ReFeR-Turbo (higher performance) and ReFeR-Lite (cost-effective). The framework demonstrates strong results across text, image, and reasoning evaluations, surpassing prior methods in both accuracy and efficiency.

strengths:
- This work proposes an interesting framework for generative task evaluation inspired by the hierarchical peer review process with peer review agents from different LLMs and area chair evaluation. This provides a more systematic evaluation framework. It also gives more explainability for evaluation.
- ReFeR is flexible - it demonstrates effectiveness on various generative tasks in different modalities including text, image, and multimodal tasks. 
- There are analysis and ablation studies of the framework, which provides more insights on how to select models, how many models to use, etc.
- The paper is generally well written and easy to understand.

weaknesses:
- Even with the ReFeR-Lite variant, the framework still demands quite significant computational resources, making it still expensive as an evaluation tool.
- Another concern is that this framework can be considered as a hierarchical version of LLM-as-a-Judge with more carefully designed prompting, which may limit the novelty of this work.

questions:
Maybe I missed something, but how are the costs in Table 2-4 calculated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534631438

thread_type:
official_review

<thread_object>
creator: reviewer_m8bc

<content>
summary:
The paper introduces ReFeR, a hierarchical framework for evaluating AI-generated outputs that lack predefined correct answers, inspired by academic peer review processes. The framework consists of two main components: a Peer Review Body where multiple language models act as peer reviewers providing independent evaluations, and an Area Chair module where a more capable language model synthesizes these reviews into a final assessment.

The authors present two variants of the framework: ReFeR-Turbo, which generates multiple responses for higher accuracy but at greater computational cost, and ReFeR-Lite, which uses single responses for better efficiency. The framework also introduces enhanced evaluation guidelines and an auto-prompt generation system to improve the quality of assessments, making it a comprehensive solution for evaluating open-ended AI outputs.

strengths:
The originality is in adapting academic peer review processes to AI evaluation, offering a different approach to assessing LLM outputs. Quality is evident through comprehensive testing across reasoning tasks and multimodal outputs, with the framework showing consistent performance across different types of evaluation challenges. The paper is nicely structured, presenting both theoretical foundations and practical implementations through its two variants (ReFeR-Turbo and ReFeR-Lite). The framework's ability to handle both text and multimodal outputs while providing reasoned evaluations makes it particularly valuable for many real-world applications.

weaknesses:
The paper's main weakness is its unclear primary contribution, as it's difficult to distinguish how the ReFeR framework meaningfully differs from other multi-LLM evaluation systems beyond borrowing concepts from academic peer review (MoA). Additionally, the paper's organization is scattered across too many topics, with important elements like the instruction tuning dataset (shown in Figure 1) being buried in the appendix rather than properly discussed in the main text.

questions:
Isn't the concept of using multiple LLMs to achieve better results in this context just derived from Mixture of Agents? Seems like an extension of the work; yet no reference.
How is the coverage of the instruction tuning dataset, does it scale?
Is ReFeR's main purpose to be a  framework to create instruction tuning sets? Or an evaluation system?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702990520

thread_type:
official_review

<thread_object>
creator: reviewer_kwmk

<content>
summary:
This paper proposes a multi-agent framework, namely ReFeR, composed of a two-level hierarchical structure, where the two levels simulate the roles of reviewers and area chairs in the peer review process. The method has been compared comprehensively with mainstream approaches in terms of NLG evaluation tasks, multimodal evaluation tasks and for reasoning tasks.

strengths:
1. The paper is well written and the method is easy to reproduce;
2. Extending the evaluation tasks to reasoning tasks is a good generalization;
3. Some key issues, such as how to select models, how many models to use, are addressed though detailed ablation study.

weaknesses:
1. The distinctions between this method and similar methods, such as multi-agent debate or multi-agent peer review, are not very clear. For example, debating or summarizing are merely different forms of prompts. In fact, for the multi-agent peer review method, if a reviewer can receive information from all other reviewers before refining their own review, they would essentially be playing a role very similar to that of an area chair.
2. The two-level hierarchical structure that summarizes through an area chair is not fundamentally different from ensemble methods like majority voting; it is merely implemented via prompts. Additionally, the difference between Turbo and Lite versions is simply one of 1 versus 20 rounds of integration.
3. The improvement in NLG evaluation is not particularly significant, and on reasoning tasks, the improvement is only more noticeable on AQuA, while it is not significant on other datasets. Moreover, the differences in performance may also be due to different model selections, such as the models used as peers not being entirely consistent with those used in other methods, leading to unfair comparisons.
4. The two-level hierarchical structure has not been theoretically proven to be necessarily better than debate methods; it seems more that different methods have different strengths, weaknesses, and application scenarios.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708834112

thread_type:
official_review

<thread_object>
creator: reviewer_2cby

<content>
summary:
The paper proposes a hierarchical multi-LLM framework using a peer review structure for text evaluation and reasoning. The framework employs smaller models as peer reviewers and a larger model as an area chair to synthesize evaluations. The work demonstrates empirical improvements over existing approaches while providing rationales for other choices in the system including models and prompts.

strengths:
- The framework shows improvement in reasoning of multi-agent systems under the framework proposed.
- The evaluations/experiments presented are sound and well ablated.
- The error analysis section is useful and provides insights on where the framework could be improved further.

weaknesses:
- The paper highlights on lines 36-47 that there are not many works on using multiple-LLMs for evaluation, but there do exist some methods using multiple LLMs, for example: https://openreview.net/forum?id=FQepisCUWu and https://arxiv.org/html/2401.16788v1. The authors highlight very early and strongly the novelty of the work of using multiple-LLMs for evaluation but existence of the above works do bring in questions. The paper would benefit much more from better literature review, and inclusion of these other methods in the results section for a fair and faithful comparison.
- The work claims of developing a novel prompting schema by introducing "guidelines"(that can be auto-generated) and lists it as a main contribution. However, I think creating detailed prompts for specialized tasks is a well known prompt engineering practice, thus I do not see it as a substantial contribution.
- Several key claims require stronger support, especially on explainability and robustness, see questions below.

questions:
1. The paper claims that on line 51-52 the framework promotes explainability and robustness.
 - Explainabiltiy: It is not clear how this happens? Why is trying to use another LLM to explain another LLM's output sound? What makes the judge LLM trustworthy especially given the hallucination problem in LLMs. It would be useful to see where the peer and AC explanations differ, and quantitative metrics to support the claim of explainability.
- Robustness: It is not clear how the framework promotes robustness? Can perturbations in prompts not affect agentic systems? Is the claim based on the fact that a perturbation for some LLM will not affect another, if so do we have a measure of what part of perturbation space agentic systems are more robust to?  Are certain combinations of LLMs more robust than the others?
2. Similarly to the previous point, the conclusion says the framework is robust, but I see no experiments to validate this claim. This could be supported to some extent by showing the standard deviations in final ratings of each peer/AC, and the complete framework. However, in order to call the framework robust, there have to be evaluations of this system on common adversarial attacks to LLMs.
3. The work says on line 99-100 that the area chair should be a larger or more capable LM, but it is not clear why? Is using smaller peers justified because of efficiency? Given the ablation where a weaker area chair(section 5.2) also works, why use a larger one?
4.  I assume temperature was set to 1 to ensure variation in the responses when n>1, but is there a reason this performs better than setting temp=0 and using n=1?
5. Could there be analysis on the rating distributions of the framework when $n>1$?
6. I am unsure how $C_{final}$ is produced when n>1, because $S_i$ can be averaged but not sure how aggregations work for $C_i$.
7. If possible, it would be useful to see ablations on $n$, this could be done by choosing subsets from results of n=20 runs if the results are stored.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733309854946

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal Summary and Final Remarks

comment:
We sincerely thank all reviewers for their thoughtful and detailed feedback, which has greatly helped in refining our work and clarifying its contributions. Below, we summarise the key points addressed during the rebuttal phase, additional experiments conducted, and the importance of this research for the broader community.

## **Key Contributions**
Our paper introduces **ReFeR**, a **hierarchical framework inspired by the peer review process**, which utilizes **multiple AI agents** (LLMs/VLMs) for the evaluation of generative outputs. We:
- **Demonstrated its superiority** over existing methods like **G-Eval** and **Analyze-Rate** through enhanced reasoning and evaluation accuracy.
- **Validated ReFeR's applicability** across diverse modalities (**text, images, multimodal tasks**) and reasoning datasets.
- Introduced two variants:
  - **ReFeR-Turbo**, for higher performance.
  - **ReFeR-Lite**, which balances accuracy and **test-time compute efficiency**.

## **Enhancements During Rebuttal**
1. **Expanded Baselines and Comparisons**:
   - Added **Simple Averaged Peer**, **ChatEval**, and **LLM-as-a-Judge** as baselines and highlighted clear performance improvements.
   - Conducted experiments on setups where the **area chair uses diverse peer configurations** (heterogeneous models with temperature-scaled homogeneous setups).
   - Performed experiments to show that the **framework's structure** is the reason for maximum performance rather than simply sampling more responses.

2. **Visualization and Analysis**:
   - Included **performance vs. test-time compute trade-off analysis** as suggested by reviewers.
   - Demonstrated that **ReFeR consistently outperforms baselines** across varying levels of inference compute.
   - Conducted experiments with **best peer as AC**.
   - Explored the **effect of temperature** on the framework using the same setup with different temperature values.
   - Examined **sampling responses** and their performances for varied **n** (i.e., **n=1, 5, 10, 20**).
   - Conducted an **analysis on the rating distributions** of the framework when **n > 1**.

3. **Clarifications**:
   - Refined discussions on **robustness**, **explainability**, and the **architectural novelty** of our hierarchical approach.
   - Provided detailed **error analyses**, ablations on **prompt structures**, and **peer-AC model combinations**.
   - Explained in detail why a **few works are not valid baselines** and how the **performance gain in correlation** is significant.

4. **Evaluation of Hallucination**:
   - Acknowledging that different approaches have strengths and weaknesses, we conducted experiments on **reasoning response quality** using the **HHEM-2.1 model** to assess hallucination on **GSM8k**. ReFeR outperformed baseline methods in producing higher-quality responses.

5. **Additional Experiments**:
   - Conducted studies using **open-weight models** (e.g., Qwen-2.5-72B in evaluation and Qwen-2.5-7B in reasoning) for **reproducibility and transparency**.
   - Showed the framework's ability to **scale across varying configurations** (number of peers, compute limits).

## **Why This Research Matters**
The evaluation of generative AI outputs is a critical and challenging task. ReFeR represents a step forward in human-like, systematic, scalable evaluation methods, providing enhanced reasoning capabilities and a framework adaptable to various domains. Its hierarchical, modular structure offers insights for improving multi-agent collaboration systems beyond the context of LLM evaluation. We also hypothesise that this architecture of ReFeR can be applied to various completely orthogonal setting as we observe that this structure works in real world and hence we believe that this idea has to reach more people and attract newer research.

## **Gratitude to the Reviewers**
We deeply appreciate the reviewers' efforts, constructive feedback, and engagement. While the rebuttal phase lacked engagement from some reviewers, we have strived to address all raised concerns comprehensively. Your contributions have significantly strengthened this work and its presentation.

We look forward to seeing ReFeR inspire future research and practical applications. Thank you for your time, expertise, and consideration.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received ratings of 6, 6, 5, 5, 5, and was recommended for rejection by majority of reviewers.

The paper introduces ReFeR, a hierarchical multi-agent framework inspired by the academic peer review process. It utilizes multiple LLMs or VLMs as peer evaluators and a more capable model as an "area chair" to synthesize evaluations and produce final scores and reasoning. ReFeR is tested across text evaluation tasks (TopicalChat, SummEval), multimodal evaluation (image captioning and generation), and reasoning tasks (AQuA, BBH, CSQA, GSM8K). The authors claim significant empirical improvements over prior methods like G-Eval and Analyze-Rate in terms of accuracy, cost-efficiency, and reasoning ability.

Strengths
- The authors test ReFeR across multiple domains (text, multimodal, reasoning) and provide thorough comparisons to baselines, including recent methods like G-Eval and Analyze-Rate.
- The hierarchical structure is an interesting and intuitive extension of multi-agent systems, simulating real-world peer review processes.
- The authors conduct extensive experiments, including ablations on the number of peer models, area chair selection, and prompting strategies, offering insights into framework behavior.

Area for improvements:
- The performance improvements over existing baselines are modest (ranging from 0.01–0.04 in correlation metrics). While ReFeR-Turbo shows the best results, the gains may not justify the significant computational overhead.
- Limited novelty: The hierarchical structure, while well-implemented, is an engineering extension of existing multi-agent frameworks (e.g., multi-agent peer review, debate frameworks). The novelty contribution is limited as prior works have explored similar structures with single models.
- The proposed prompting schema, while effective, is specific to the ReFeR framework and cannot be generalized easily. The novelty of "evaluation guidelines" is also questionable, as detailed prompts are a common practice in LLM research.
- Evaluation costs: Despite the introduction of ReFeR-Lite, the framework still requires significant computational resources, which limits its practical applicability for large-scale tasks.

While the paper is well-executed, its contributions are incremental, and the computational cost remains a limiting factor. For future iterations, the authors could strengthen the theoretical underpinnings, reduce overhead, and demonstrate broader applicability to justify its acceptance.