Benchmarking LLMs' Judgments with No Gold Standard

Authors: Shengwei Xu, Yuxuan Lu, Grant Schoenebeck, Yuqing Kong
Primary area: datasets and benchmarks
Keywords: Benchmarking, Peer Review, Mutual Information, Data Contamination, Large Language Models

Abstract:
We introduce the GEM (Generative Estimator for Mutual Information), an evaluation metric for assessing language generation by large language models (LLMs), particularly in generating informative judgments, without the need for a gold standard reference. GEM broadens the scenarios where we can benchmark LLM generation performance-from traditional ones, like machine translation and summarization, where gold standard references are readily available, to subjective tasks without clear gold standards, such as academic peer review.

GEM uses a generative model to estimate mutual information between candidate and reference responses, without requiring the reference to be a gold standard. In experiments on two human-annotated datasets, GEM demonstrates competitive correlations with human scores compared to the state-of-the-art GPT-4o Examiner, and outperforms all other baselines. Additionally, GEM is more robust against strategic manipulation, such as rephrasing or elongation, which can artificially inflate scores under a GPT-4o Examiner. 

We also present GRE-bench (Generating Review Evaluation Benchmark) which evaluates LLMs based on how well they can generate high-quality peer reviews for academic research papers.  Because GRE-bench is based upon GEM, it inherits its robustness properties.  Additionally, GRE-bench circumvents data contamination problems (or data leakage) by using the continuous influx of new open-access research papers and peer reviews each year. We show GRE-bench results of various popular LLMs on their peer review capabilities using the ICLR2023 dataset.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729958489398

thread_type:
official_review

<thread_object>
creator: reviewer_jk45

<content>
summary:
This work is motivated by the importance of finding a benchmarking method that is not reliant on gold-standard references. The proposed method(s) also attempt to circumvent known issues about using generative models as evaluators, where they are often sensitive to degradation strategies and score-increasing manipulations. The method is empirically compared against a wide range of widely known evaluation metrics to showcase its effectiveness.

The authors also introduce a new benchmark for review generation which is built upon their method that inherits all its benefits. The benchmark is also claimed to combat data contamination by being updated with new review data as it comes out.

strengths:
The paper presents a valuable step towards doing away with the reliance on gold-standard references in evaluation. It's original and creative in the way it makes use of mutual information in the context of generative models.

The method is described rigorously and thoroughly. The flow and structure of the paper makes sense and conveys their ideas well.

The authors also seem to be highly aware and open about the limitations of the method. The paper actively critiques itself and presents answers and discussions on these critiques.

weaknesses:
While the paper is generally highly verbose discussion-wise, I find the experiments and results parts of the paper to be a bit lacking in discussion and analysis. My concerns on this matter are made more specific in the "Questions" part of this review.

Some minor grammar/spelling mistakes: lines 96-97 "..the score can relies solely...", line 455 "Table" is misspelled.

questions:
1. I'm not sure I understand the setup of the first result, am I right in understanding that what you do here is generate k candidates per review, and then get the average GEM score for each candidate by averaging over all other generated candidates when used as references? And then you make the classification by choosing the candidate that resulted in the highest GEM score? How is the classification made? If I'm correct in understanding this, then I don't see how the following sentence from the conclusion applies: "...while our method does not require gold-standard references, it requires reference that includes original human-generated information." Since the way I understand it now, it doesn't really need any human-generated references to generate these scores.

2. Why do you think LMExaminer performs so much better in Sentence Deletion and Deletion & Completion? I feel like that's quite a significant result that hasn't been discussed.

3. "Figure 5 shows that, within each LLM family, such as the Llama-3 or Claude series, models with larger parameter sizes tend to perform better. Moreover, newer versions tend to outperform their predecessors with similar model sizes, as seen in comparisons between Llama-3.1 and Llama-3. These observations further validate the effectiveness of our GEM and GEM-S metrics in accurately evaluating LLMs."
Can GRE-Bench only work with the GEM metrics? Do you think you would observe something similar if you used different metrics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540094393

thread_type:
official_review

<thread_object>
creator: reviewer_bcjd

<content>
summary:
The paper mainly discusses benchmarking LLM's judgments in the lack of gold standard reference answer, but with a pool of vanilla reference answers by estimating the mutual information between a candidate and reference. The authors also show that this technique is also robust against perturbations and manipulations. Lastly, they also present a benchmark/test set to evaluate how good LLMs can generate high-quality reviews for academic paper.

strengths:
- The paper is well-written, and the problem statements is well-motivated. The authors provide cogent and coherent arguments throughout the paper. I personally think this is a very valid issue for LLM-based evaluators as not always we have a single human-verified gold-standard reference, but a set of good-enough responses to aggregate and use as a ground-truth. The paper also has a good mathematical rigor to prove its hypothesis. 
- The benchmark created in the dataset is a plus point as it can help future works. 
- I also appreciate the finding that GEM is robust to perturbations as lately there has been a good discussion around LLM having blind spots in evaluation and missing out on subtle disturbances to the input.

weaknesses:
- I do not see any blaring errors in the paper, however, I am a suspicious about the experimental setup as such. Bottom line, the problem statement discusses a **very challenging** problem: _LLMs generating "judicious" reviews for very long and complex scientific papers_. I am not sure if most of the existing LLMs are good enough for such a humongous task and if they do provide good results which can be analyses with some guarantee, as the LLMs are required to have a decent knowledge of the domain as well as a good understanding of the hypothesis in the paper, which even humans struggle to understand in some cases. In a way, this feels like just applying an LLM to a task to see how it fares. But the authors to talk about previous works that have discussed using LLMs as reviewers, and I see that most of them have shown negative results in such cases.
- Related to the above point, I have several questions for the authors, which I hope will be answered to give me some more clarity on the nitty-gritty details of the work.

questions:
- Missing citations: 
    - [Finding Blind Spots in Evaluator LLMs with Interpretable Checklists](https://arxiv.org/abs/2406.13439)
    - [Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://aclanthology.org/2024.findings-eacl.71/)
- In Appendix D3, I see that the whole paper is given in text format, but most of the scientific literature also contains images that are quite important and referenced multiple times. What happens in that case? Similarly, how are tables being managed? Also, are these papers converted from PDF to text format, if so, are they being verified for OCR errors? The papers presented are usually 10 pages in ICLR and 8 pages in *CL, so are you sure the paper is not beyond the token limit of the LLM? Lastly, which LLM was used for the prompt in D3?
- For ICLR dataset, it is mentioned that one reviewer is the candidate and the remaining 2 are reference. Aren't only 2 references too less? I feel in this case the sample bias will be quite high. I urge the authors to clear this for me. 
- For the rephrase manipulation, were any sanity checks conducted to make sure that the semantics did not change? Since we are operating at very long-context lengths (document-level), this could be an issue for smaller models like llama-3.1-8b. GPT-4o is quite large and powerful as compared to llama-3-8b and has much better instruction following capabilities, so was this disparity maintained purposefully? If not, I feel llama-3.1-405B or any of the other larger models should be used in place of it which are good at instruction-following.
- For the GRE-Bench, where the LLM is generating reviews, were all 3 human reviews taken as references?
- Line 472 is very redundant and but obvious. Smaller models do not have good instruction following abilities. 
- The word Table has been misspelled as Tabel at a couple of places. Please fix it. 
- In Figure-4 (and Table-6), I am quite dubious about the parameter counts being mentioned there, as most of closed-source models do not reveal anything, especially for GPT models. Also, the color-scheme in Fig-4 is not comfortable to the eye to make clear demarcations across the bars. Please fix that as well. 
- In Prompt D2, the model is provided with one review, which resembles a 1-shot example in a way. In case of generative tasks, there have been cases when the model tries to follow the verbatim or structure of the provided few shots, and the final result lacks creativity. Was this checked in any way? 
- What were the generation parameters in each prompting case?
- How were the outputs checked for hallucinations as the generations are quite long?
- In F1, was the same prompt format as in D2 used while finetuning the model?
- There should be some comparison of llama-3.1-8b-Instruct and its finetuned version (as mentioned in F1) to see the gains due to finetuning. Also, finetuning an already finetuned model is slightly tricky as the distribution shift may cause the model to collapse. The general trend is to use a base model directly for this case.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730634625317

thread_type:
official_review

<thread_object>
creator: reviewer_cxhl

<content>
summary:
This paper presents GEM (Generative Estimator for Mutual Information), a new evaluation metric for assessing the quality of text responses generated by LLMs in tasks that lack a clear gold standard reference.  GEM aims to provide an accurate and manipulation-resistant metric by estimating the mutual information between candidate responses and a set of reference responses, which are not required to be perfect. The main idea is to measure the semantic informativeness of candidate responses by filtering out aspects like language style and syntax, focusing on how much information the candidate's response reveals about the references.

strengths:
The paper is clearly written and rigorously formulated. The research topic is very timely and important, given the increasing use of LLMs and the challenges of evaluating their performance in subjective tasks.

weaknesses:
I have some questions and concerns:

1. Impact of Suboptimal Human References: I am concerned about GEM's reliability when human references are of lower quality than LLM outputs. Could the LLM used for estimating probability distributions learn to favor responses that resemble flawed human references, even if they are less informative, potentially leading to inaccurate scores? 

2. Performance stability according to the choice of evaluation-LM and preprocessing LLM: How sensitive is GEM to the choice of LLMs used for evaluation (Llama-3.1) and preprocessing (GPT-4)? Would using different LLMs, especially open-source ones, impact the results? LLMs are released and updated frequently, and I wonder how we can deal with such frequent updates to make the GEM score consistent and reliable over time.

questions:
This is a kind of optional question out of my curiosity. 

It seems that GEM is a relative scoring/evaluation framework. If so, what would be the pros and cons of GEM compared to other relative evaluation frameworks, such as the ELO rating system commonly used in generation task leaderboards, like those in Chatbot arenas?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730897600552

thread_type:
official_review

<thread_object>
creator: reviewer_8skk

<content>
summary:
The paper introduces GEM, a novel metric for evaluating the performance of large language models in generating informative judgments using given references but not necessary gold standards, and presents GRE-bench, a benchmark for assessing the quality of peer reviews generated by LLMs.

GEM is designed to assess the informativeness of judgments in scenarios where gold standard references are unavailable, expanding the scope of benchmarking to subjective tasks like peer review.

strengths:
Using mutual information for evaluation subjective generative response is novel as my understanding.

The paper give theoretic guarantees of the proposed method.

Experiments show the method works well on the task of evaluating reviews for academic papers or student proposals.

weaknesses:
Only one type of evaluation tasks has been assessed: paper review evaluation.  It would be more convicing if the paper can conduct experiments of more subjective evaluation tasks, such as summarization, dialog, etc.  

The paper uses Llama 3.1 8B as the evaluation LM.  I would like to see how different LMs effect the performance of the evaluation.

Figure 4 is not mentioned in the paper.

questions:
Using LLM to estimate the conditional probility seems problematic:  While P(x,y)=P(x)*P(y|x)=P(y)*P(x|y), P_LLM(x)*P_LLM(x|y) != P_LLM(y)*P_LLM(x|y).  I wonder if this will cause problems in the estimation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732080607743

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all the reviewers for their thoughtful comments and appreciate that reviewers explicitly state our paper is “novel”, “well-motivated”, “well-written” and “rigorous”. 

We want to highlight that, to the best of our knowledge, our paper is the first paper exploring evaluating LLMs with tasks that have no gold-standard references through an information theoretical lens.

Additionally, we thank reviewer bCjd for “appreciating the finding that GEM is robust to perturbations”. The manipulation resistance of our GEM metrics implies its effectiveness surpassing the GPT4 examiner in evaluating judgment qualities. We appreciate the suggestions by reviewer bCjd on the literature studying blind spots of LLMs in evaluation, which further support our findings and results.

We also thank reviewer bCjd for acknowledging our contribution to building the GRE-bench dataset where we use 18 LLMs to generate reviews for 300 papers in ICLR2023. Indeed, we also provide a methodology for utilizing the rich and valuable data from ICLR reviews to help benchmark LLMs while circumventing data contamination. We’d like to give credit to all the reviews and the conference for making our methodology possible.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732162904432

thread_type:
official_comment

<thread_object>
creator: area_chair_dsx5

<content>
title:
Reminder: Please respond and update the score if necessary

comment:
Dear Reviewers,

Kindly ensure that you respond proactively to the authors' replies so we can foster a productive discussion. If necessary, please update your score accordingly. We greatly appreciate the time and effort you’ve dedicated to the review process, and your contributions are key to making this process run smoothly.

Thank you,

AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732776317903

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision Notes

comment:
Dear Reviewers,

Thank you again for your thorough and constructive feedback on our paper!

We are gratified that the reviewers found our paper to be novel, well-motivated, well-written, and rigorous. This positive assessment affirms the significance of the problem we are addressing and the strength of our proposed methodology. At the same time, your insightful questions and critiques have enabled us to further strengthen our paper.

In this revision, we have carefully addressed each of the points raised by the reviewers. Key updates include:

1. Additional experiments validating the robustness of our GEM metrics across various preprocessing models (Appendix A.3) and evaluation models (Appendix A.2), to address questions from reviewers 8Skk and CxHL. Most of our experiment results remain consistent across various preprocessing models and evaluation models.

2. New experiments in Appendix A.4 applying the GEM metric to evaluate Yelp reviews, demonstrating the broad applicability of our approach beyond peer review and peer grading contexts, as requested by reviewer 8Skk.

3. Additional experiments examining the symmetric property of LLM estimated distribution in Appendix A.5, as suggested by reviewer 8Skk.

4. A discussion in Sections 3.2 and 5 of how our GEM-S metric enables hierarchical evaluation across different cognitive levels. By varying the synopsis granularity, GEM-S can differentiate weaker models based on basic abilities like summarization, and stronger models on complex skills like critical analysis. We leverage the nature that peer review is a challenging task that encompasses multiple levels of abilities, as reviewer bCjd suggests, to create a meaningful evaluation framework. 

5. Expanded results analysis in Section 4.2, including an explanation for LMExaminer's strong performance on "Sentence Deletion" degradations, as requested by reviewer jK45.

6. Clarifications on the experimental setup, such as API parameters and PDF parsing, added to Appendix C, as requested by reviewer bCjd.

7. Improved presentation of Figure 4, with clearer notation that parameter counts for some closed-source models are estimates, based on reviewer bCjd's feedback.

8. Additional citations added to the Introduction, as suggested by reviewer bCjd.

The full set of major changes are highlighted in **blue** (if the whole section is new, we only highlight the title for better reading experience.) throughout the revised manuscript. We hope these modifications fully address the reviewers' concerns and further strengthen the contributions of our work.

Once again, we are deeply grateful to the reviewers for their constructive critique. We also want to express our appreciation to ICLR for generously providing the review data that made this work possible. We believe our proposed GEM metrics and GRE-bench provide methodologies to make effective use of this invaluable resource to enable more **accurate**, **manipulation-resistant**, and **data-contamination-robust** evaluation of language models on open-ended generation tasks even without gold-standard reference.

We look forward to further discussion and to the opportunity to continue refining our work with your guidance.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces GEM, a metric for evaluating large language models (LLMs) based on the informativeness of their generated responses without needing gold-standard references. GEM estimates mutual information between candidate and reference responses, making it ideal for subjective tasks like peer reviews. The accompanying benchmark, GRE-bench, assesses peer review quality generated by LLMs using GEM, offering a robust, manipulation-resistant evaluation. GRE-bench mitigates generative model evaluation issues and updates its data to prevent contamination. GEM's effectiveness is validated against existing metrics.

On a positive note, the reviewers commend the paper for its innovative use of mutual information to evaluate subjective generative responses. The approach provides theoretical guarantees and effectively demonstrates its capability in tasks such as academic paper review evaluation. The paper is clearly written and rigorously structured, effectively addressing the pressing challenge of assessing LLM performance in subjective tasks where gold-standard references are absent. Its mathematical rigor, coupled with the introduction of a new benchmark dataset, significantly contributes to ongoing and future research efforts. Additionally, GEM stands out for its robustness to input perturbations. The authors are also praised for their transparency in acknowledging the method's limitations, offering insightful self-critical analysis and discussions throughout the paper.

The reviewers express several critiques and concerns regarding the paper's scope and methodology. They recommend broadening the evaluation to encompass additional tasks such as summarization and dialogue, which would enhance the study's credibility. The reliance on a single language model, Llama 3.1 8B, raises questions about the potential impact of using diverse models on the results. Moreover, there are concerns about GEM's reliability when faced with suboptimal human references and its sensitivity to the choice of both evaluation and preprocessing LLMs. The capability of current LLMs to produce insightful reviews for complex scientific papers is questioned, suggesting that the study may lean more towards an exploratory approach. Although the paper offers detailed discussions, the sections on experiments and results are perceived as lacking in-depth analysis, prompting the reviewer to seek further clarification in future revisions.

The authors have made significant updates to the paper, effectively addressing the previously raised issues. 

The paper makes a significant contribution by introducing a method to estimate mutual information between candidate and reference responses without necessitating the reference to be a gold standard. I think this approach holds the potential for substantial impact. I am leaning to recommend acceptance of the paper.