MetaMetrics: Calibrating Metrics for Generation Tasks Using Human Preferences

Authors: Genta Indra Winata, David Anugraha, Lucky Susanto, Garry Kuwanto, Derry Tanti Wijaya
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: metrics, human preferences, calibrating, generation

Abstract:
Understanding the quality of a performance evaluation metric is crucial for ensuring that model outputs align with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics, a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. MetaMetrics aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes MetaMetrics a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730575835941

thread_type:
official_review

<thread_object>
creator: reviewer_nkfe

<content>
summary:
This paper introduces a novel evaluation metric, MetaMetric, designed to better align with human preferences. The proposed metric is a weighted combination of existing metrics, where the weights are learned in a supervised manner. The authors demonstrate the effectiveness of MetaMetric through experiments on various natural language generation tasks, e.g., summarization, translation, question answering and image captioning. Additionally, they show that this metric can be useful for reward model scoring. They also provide some analysis about interpretability and efficiency of their metric.

strengths:
1. The experiments are comprehensive, including various types of natural language generation tasks.
2. The topic of developing a human-aligned evaluation metric for NLP generation tasks is a long-discussed but unsolved challenge. This paper explores a possibility to push forward in this direction.
3. The proposed metric is easily extendable to other tasks where preliminary metrics have already existed. This extensibility makes the metric potentially applicable to a broader range of applications.

weaknesses:
I have several concerns regarding the experiments, particularly the scope of the problem and the evaluation settings, and how they support the authors' claims. Additionally, the presentation of the paper requires refinement.
1. **Limited Contribution:** The paper's contribution appears limited. It essentially addresses a traditional supervised learning problem with input of small dimensions and output with single normalized value. The authors train a regression model on supervised-learning datasets, using conventional metrics as inputs and human preference values as outputs. The experimental results involve testing this regression model on the test set of these datasets and comparing different metrics using correlation of the rankings of metrics with ground truth human preference values. It is unsurprising that MetaMetric outperforms other metrics in the in-domain generalization setting, which is the primary focus of this paper. The authors should also clearly present their experimental settings, such as how they create the training, validation and testing sets. For example, the setting for Question Answering tasks is not clear. It would be interesting to add more OOD and real scenarios to test this metric. See following comments for detailed feedback.
2. **Lack of Out-of-domain (OOD) Generalization Evaluation:** The authors should place more emphasis on the OOD setting, as human preference data is not always available. I found only one one OOD setting presented in Figure 3. Based on in-domain performance, the GP model works the best. However, the GP model performs similarly to CLIP-S and METEOR in the OOD setting. (The statement about the superiority of MetaMetric-Cap over any other individual metric, in lines 426 and 427, is misleading.) This suggests that the use cases for the trained metric are limited, and its performance may be worse than using a single conventional metric. It would al os be interesting to evaluate metaMetric trained on one task but tested on another, which is a more challenging OOD setting than the one in Figure 3.
3. **Lack of Evaluation in Real Settings:** The evaluations are limited by the training datasets with human-annotated preference values. It is unclear whether this metric is useful for real applications, as suggested by the authors in lines 41 and 42. For example, the authors could follow a similar evaluation procedure to DPO and PPO papers, using the proposed metric to evaluate various LLMs - before and after RLHF-tuned - and compare their win rates based on MetaMetric.
4. **Reference-based and Reference-free Settings:** In line 83, the authors claim to design the metric for two settings: reference-based and reference-free. If I understand correctly, this depends on the choice of metrics rather than the model design. If the chosen metrics require references, then MetaMetric can only be reference-based. In the experiments, the authors only distinguish these two settings for Machine Translation, not for other tasks. It would be better that the authors add both settings for all the tasks.
5. **Presentation Issues:** The paper's presentation is suboptimal. Major issues include:
    1. The caption of a table should be placed above it.
    2. Figure 2 is not cited anywhere in the paper.
    3. The authors cite a table in the Appendix as crucial support for their claims (see line 357 for Table 9). The authors should put the information in the main text.
    4. Other minor issues are listed in the Questions section.

questions:
1. **Questions related to Introduction Section:**
    1. The first paragraph is a bit confusing. The authors discuss the shortcomings of BLEU and BERTScore in the main text, but Figure 1 presents BLEU and CLIP-S. Additionally, what criteria are used in Figure 1 to determine if a metric is good or not? For the figure on the right hand side, why are BLEU 0.2 and MeMetric 0.4 considered good, but CLIP-S 0.6 is considered bad? What is the threshold?
    2. The authors state, "models that optimize for human preference ... have demonstrated superior performance." What metrics have these models used to demonstrate human alignment? The authors avoid discussing this in their paper, instead comparing their metric with conventional metrics not originally designed for human preference alignment. For example, Ouyang et al. [1] used the Perspective AI score to measure toxicity.
    3. In line 42, the authors mention "evaluation metrics that accurately reflect human subjective judgements." Since human judgements are **subjective**, is it possible that a metric can **accurately** measure these varied judgements across different individuals?
    4. In line 53, the statement relating the necessity of human aligned metrics to the complexity of tasks seems implausible. Regardless of task complexity, we always seek metrics that align with human evaluations.
2. **Universality of MetaMetric:** For conventional metrics (e.g., BLEU, ROUGE, METEOR), they are used across various natural language generation tasks. Is it feasible to have a single MetaMetric applicable to all natural language generation tasks?
3. **Choice of Training Methods:** Why did the authors choose Bayesian Optimization and Boosting to train MetaMetrics? Is it possible to use linear regression models with regularizations?
4. **Efficiency of MetaMetric:**. According to Figure 4, MetaMetric requires about 8 metrics for computation. Can the authors explain why computing approximately 8 metrics is still considered efficient?
5. **Interpretability:** Can the authors explain why some metrics align more closely with human evaluations while others do not? I'm curious whether there is any intuition behind this.
6. **Compute Efficiency for RM:** In lines 433 and 472, Table 4 does not show compute efficiency. The authors should include the model size for MetaMetrics-RM.

[1] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.

**Minor Issues:**
1. In line 215, there is a duplicate "is"
2. Presentation suggestion: Section 2 could be merged with Section 7.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730751100291

thread_type:
official_review

<thread_object>
creator: reviewer_hgah

<content>
summary:
This paper introduces Meta-Metrics, a calibrated metric for evaluating generation tasks like summarization, QA, Machine Translation and Image Captioning,etc. It raises the issue of metrics excelling at one dimension and falling at others while evaluating on multiple dimensions (like coherence, fluency, relevance,etc in summeval), hence proposing a calibrated metric based on these several metrics, aiming to achieve a higher alignment with human preferences in the evaluation scores. The authors propose using methods like Bayesian Optimization, Boosting to effectively choose/filter the various metrics and give a combined metric to have higher correlation with human scores.

strengths:
1. The paper presents innovative mathematical approaches of using BO modeled as Gaussian Process, and XG Boosting on existing metrics to create a new metric.
2. The paper has tested their method extensively on several datasets across multiple domains.

weaknesses:
1. Although the metametrics have shown they have higher alignment with human preferences, the optimization methods (BO and Boosting) are to be considered black boxes and offer very less explainability.  Understanding how these methods find the combination of metrics and interactions between different metrics within the model is not possible. Hence explanation on why they correlate more with human scores is not there.

questions:
1. It is very good that the authors have shown their method on multiple datasets, but some more experiments/analysis on the proposed methods would strengthen the paper more. One such thing can be, the authors used Kendall-Tau as the objective calibration function. How were the results when spearman correlation is used as a calibration function?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730788603076

thread_type:
official_review

<thread_object>
creator: reviewer_y9ei

<content>
summary:
This paper introduces meta-metrics, which optimizes the combination of existing metrics to enhance their alignment with human preference. The paper investigated mainly two method GP and XGBoost to try to learn a best weight vector for a set of base metrics to combine. For different datasets, they select different base metrics and train a new weight vector for combinations. Results show that meta-metrics can outperform existing baseline metrics in summarization, translation, question answering, image captioning and reward modeling scenarios.

strengths:
1. This paper proposes a new paradigm to ensemble existing metrics for evaluation of different tasks, learning a weight vector through GP or XGBoost and get the weighted combination of base metrics. This is a novel approach and gets good performance.
2. The paper is well-motivated and clearly written, with nice figures and tables illustrating the concepts.
3. Extensive experiments across summarization, machine translation, QA, reward modeling, which covered a wide areas. The authors also provide analysis of the weights distribution across the base metrics. Providing many insights.

weaknesses:
1. The performance gain is relatively small compared to base metrics, especially for machine translation and reward modeling tasks, it's unclear whether these marginal performance gains are worth the training cost.
2. The proposed approach requires training a new combination weight vector once the base metrics have changed, thus limiting the usability of the approach.

questions:
1. Please provide the training details. From my understanding, when training a combined metametric for reward modeling, you need to run inference on all of the training data using each of the base metrics first, and then train your weight vector to optimize for the human preference. How much GPU hours have you spent on the inference, and the training? How many GPUs did you use for the experiments?
2. Could you provide a list of base metrics used for each meta-metric variant in one place? I only found the base metrics for reward modeling  in line 474 and it's unclear what's the base metrics used for other benchmarks. if they are also baselines, it's better to mark them in the results table.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731224153789

thread_type:
official_review

<thread_object>
creator: reviewer_h1lp

<content>
summary:
This paper introduces MetaMetrics, a framework designed to calibrate evaluation metrics to better align with human preferences. By leveraging existing metrics that capture different aspects of human preferences, MetaMetrics employs learning-based methods to optimize these metrics for various generative tasks. To demonstrate the effectiveness of MetaMetrics in aligning with human preferences, the authors compute correlations between the calibrated metrics and quantified human ratings. Experiments are conducted across diverse tasks, including Text Summarization, Machine Translation, Question Answering, Image Captioning, and Reward Model Scoring.

strengths:
- The paper addresses a critical problem in the field of text generation: the lack of automatic metrics that accurately reflect human preferences. Current approaches often rely heavily on human annotations to evaluate performance. By proposing a method to calibrate existing metrics to align with human preferences, this work aims to reduce the reliance on labor-intensive human evaluation processes.

- The methodology is simple yet effective. The proposed approach captures the multifaceted nature of human preferences while incurring minimal additional cost, enabling efficient calibration of existing metrics.

- Extensive experiments validate the proposed method. The authors conduct experiments across a variety of tasks, including Text Summarization, Machine Translation, Question Answering, Image Captioning, and Reward Model Scoring. This demonstrates the task-agnostic advantages of the method and its broad applicability.

weaknesses:
- Empirical and ad hoc nature of MetaMetrics: The calibration process for MetaMetrics relies on empirical and ad hoc weighting of existing metrics for each dataset and human preference. These weights are highly dependent on the specific dataset and task, which raises concerns about their generalizability to a broader range of datasets or tasks.

- Lack of ensembled baselines: While the paper compares MetaMetrics against individual metrics, it does not include ensembled baselines, such as averaged or weighted-average scores of individual metrics. Including these as baselines would provide a fairer comparison and strengthen the evaluation.

- Absence of human studies in experiments: Although MetaMetrics aims to align with human preferences in evaluating model performance, the study does not involve an actual model being evaluated and analyzed through human studies. To comprehensively validate MetaMetrics, experiments involving direct human evaluation of model outputs are necessary.

- Unclear human preference in certain tasks: In some evaluation tasks, the connection between the ratings and human preferences is not adequately explained. For instance, it is unclear how ratings like sys Pearson, seg Pearson, and acc-t in Machine Translation tasks, or NQ, TQ, NarrativeQA, and SemEval in Question Answering tasks, align with human preferences. More detailed explanations of these ratings and their relationship to human preferences are needed, especially for readers unfamiliar with these tasks.

- High computational cost for reward model ensembling: The use of multiple metrics in MetaMetrics can significantly increase evaluation costs, particularly in reward model scoring. For example, in online RLHF training, simultaneously running multiple reward models (e.g., 7 models in Table 4) could make the process computationally infeasible.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731323405756

thread_type:
official_review

<thread_object>
creator: reviewer_4vev

<content>
summary:
This paper proposed a method to align the automatic evaluation metrics with human preference evaluation, called MetaMetrics. Specifically, it's optimized to learn a set of weights that can combine automatic metrics to mimic human preference. Two methods are discussed and experimented for the optimization. The experiments cover both text and multimodal evaluation, demonstrating a satisfactory performance of the proposed method.

strengths:
1. The research problem this paper studied is very interesting and useful. Automatic evaluation metrics can cover a wide range of benchmarks. However, they may not align with human evaluation. Given human evaluation is time-consuming and expensive, and also current model is required to be more generalizable to various tasks, aligning the automatic metrics with human evaluation preference becomes an urgent and important research topic.

2. This paper illustrates the motivation, problem definition, and methods to optimize the proposed MetaMetrics in a clear and easy-to-follow way.

3. The experiments are conducted in a wide range of downstream tasks, from text to multimodal, as well as training a reward model.

weaknesses:
1. This paper assumes that human preference can be approximated by a weighted combination of metric scores. But it seems to be too simple, and might not always hold. For example, human's evaluation of helpfulness might require both the model to perform well in reasoning and knowledge-retrieving, which we assume are separately measured by two benchmarks. Then a weighted summation of these two benchmarks might not be suitable, instead, a multiplication might be more proper.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732092982373

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all Reviewers and AC

comment:
We sincerely thank all the reviewers for their insightful and constructive feedback and for dedicating their time to reviewing our work. We appreciate the reviewers for raising their scores and recognizing the significant contribution of our paper in advancing research on new evaluation metrics. We have diligently updated our paper and addressed all comments from each reviewer, and we are eager to respond to any further inquiries or feedback you may have. We ran new experiments and reported in the revision to ensure all concerns are addressed properly.

We are particularly grateful for the acknowledgment of several key aspects of our work:

1. **Alignment with Human Preferences**: Our paper introduces MetaMetrics, which **addresses the critical challenge of aligning automatic evaluation metrics with human preferences in generation tasks**. Given the limitations of current metrics in accurately reflecting human judgments, achieving this alignment is crucial for reducing the reliance on costly and time-consuming human evaluations and improving the generalizability of models across diverse tasks.

2. **Calibrated Weighted Ensemble**: MetaMetrics calibrates existing metrics using Bayesian Optimization and Boosting to create a weighted ensemble. Through comprehensive experiments spanning five different tasks and two modalities (text and image), such as Text Summarization, Machine Translation, Question Answering, Image Captioning, and Reward Model Scoring, we demonstrate its effectiveness.

3. **Versatile and Task-Agnostic Solution**: Our proposed approach, which ensembles existing metrics, **provides a versatile and task-agnostic solution.** We conduct training in low-resource conditions, multi-modality, and evaluate across multiple contexts, including out-of-domain settings.

As per reviewers' requests, we ran new experiments and added the following results:

1. **Enhanced Robustness Section for Out-of-Domain (OOD) Settings (Added New Experiments)**: As requested by a reviewer, we have **expanded the robustness section** in Section F.7, including additional experiments. **Results from cross-lingual and cross-dataset settings are summarized to provide a more comprehensive understanding.**

2. **Ablation Studies (Added New Experiments)**: As requested by a reviewer, we have included **an ablation studies section** in Section F.6. These studies explore different calibration functions (such as Spearman correlation) and GP with various weighting functions (**multiplicative** and **a combination of linear and multiplicative**).

3. **Added more metrics to calibrate (Added New Experiments)**: As requested by a reviewer, we have **added more metrics** to evaluate MetaMetrics for SummEval dataset.

4. **Expanded Baselines (Added New Experiments)**: As requested by a reviewer, we have **added more baselines** for uniform weighting and weighted averaging in Tables 1, 2, 3, and 4.

5. **Added calibration/inference time and cost tables for Summeval (Added New Experiments)**. As requested by a reviewer.

6. **Added spearman evaluation for Summeval (Added New Experiments)**. As requested by a reviewer.

We have added several additional tables to further demonstrate the benefits of MetaMetrics and to address the concerns raised by other reviewers. For your convenience, please note the updated table numbers from our previous discussion:

- Table 12 for MetaMetrics-CAP is now Table 13.
- Tables 16 and 17 for MetaMetrics-MT cross-lingual in WMT-23 and WMT-24 have been renumbered to Table 17 and Table 18, respectively.
- Table 18 for MetaMetrics-CAP Cross-Dataset is now Table 19.

In response to the reviewer H1Lp's request, made just one day before the revision period ended, we will conduct an additional experiment on RLHF for our reward model for camera-ready. This experiment aims to demonstrate that reward models developed using MetaMetrics excel not only in the RewardBench benchmark but also in other LLM benchmarks. While we believe this experiment is not strictly necessary to substantiate the core strengths of our claim, since we have already provided ample evidence across various tasks in the paper, it serves to further illustrate the versatility of MetaMetrics in additional applications. Given the timing of the request and our swift agreement with the reviewer, the results of this experiment will be included in the Appendix of the camera-ready version as a supplementary addition.

**We have answered all concerns from all reviewers** and we hope this work will be useful for future research.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a metametrics that is more aligned with human preference on evaluating the generation tasks. The idea is quite simple, which can be summarized based on the rebuttal comment: 
1. Identify the downstream task, such as image captioning.
2. Train MetaMetrics using datasets that include human ratings. Evaluate the performance on an in-domain held-out set, selecting the MetaMetrics based on the performance of this same in-domain held-out set (not the test set). The test set is not used at this stage.
3. Evaluate the selected MetaMetrics on an unseen test dataset split that includes human ratings. This is a standard machine learning evaluation. We use Kendall or Pearson correlation to assess language model behavior relative to human preferences. The test dataset split can include both in-domain unseen test data and out-of-domain unseen test data (e.g., cross-lingual and cross-dataset scenarios). For the in-domain setting, use 70% of the dataset as the test split. For the out-of-domain setting, we use the whole dataset as test set.

The proposed approach demonstrates the effectiveness and usefulness in evlauation and shows good human alignment. 

Strength:
1. The paper studies an important task (evaluation for generation task)
2. The approach is simple yet effective
3. The paper is easy to follow

Weakness:
1. The author might need to follow the reviewer suggestions to provide more ablation study such as evaluations of the LLMs before and after alignment.

Overall, I think this is a good paper and recommend accept.