MetaMetrics: Calibrating Metrics for Generation Tasks Using Human Preferences

## Abstract

Abstract Understanding the quality of a performance evaluation metric is crucial for ensuring that model outputs align with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics , a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. MetaMetrics aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes MetaMetrics a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts.

## 1 Introduction

Evaluating machine-generated sentences has long been one of the main challenges in natural language processing (NLP). Callison-Burch et al. () provide multiple examples where a high BLEU score does not necessarily indicate true sentence similarity, and conversely, highly similar sentence pairs can receive low BLEU scores. BERTScore (Zhang et al., ) is designed to capture semantic similarities, also falls short in capturing the all the nuances to have a comprehensive evaluation. Figure 1 illustrates two such cases where traditional metrics either overestimate or underestimate the alignment of generated sentences. Such cases highlight a critical limitation for robust evaluation, as it may fail to account for the multifaceted aspects of language quality.

In light of recent advancements such as Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al., ), ensuring that generated outputs align with human preferences has become increasingly critical. Models that optimize for human preference, rather than solely relying on traditional metrics, have demonstrated superior performance in producing content that aligns with human preference (Rafailov et al., ; Winata et al., ). This shift highlights the need for evaluation metrics that accurately reflect human subjective judgments across multiple dimensions. Such metrics are essential for guiding models to generate more human-aligned outputs by comparing their results against human judgments. Metrics that exhibit a high correlation with human ratings are considered more reliable and effective for evaluating model performance.

This is particularly important for NLP tasks, where the subtleties of human language and context significantly influence quality assessments. For example, in machine translation (Freitag et al., ; Juraska et al., ), the accuracy and fluency of the translated text are critical factors considered by humans. Similarly, in text summarization (Fabbri et al., ), the coherence, relevance, and conciseness of the summary are key aspects that need evaluation. Beyond NLP tasks, vision-language (VL) tasks like image captioning also benefit from reliable evaluation metrics (Hessel et al., ). In these tasks, the generated captions must accurately describe the image content while maintaining grammatical correctness and contextual relevance. The complexity of these tasks highlights the need for metrics that can effectively capture the quality of the generated content in a manner consistent with human judgment. Each metric may be good for a specific human preference aspect, and it is necessary to identify which metric is suitable for each aspect to ensure comprehensive evaluation.

##### Many Metrics Are Not Tuned on Human Preferences.

Standard metrics that are commonly used to evaluate downstream language tasks, such as BLEU (Papineni et al., ) and BERTScore (Zhang et al., ), often fail to align with human preferences. In many generation tasks, it is crucial to assess the quality of outputs based on human judgment, as these evaluations comprise multiple dimensions of quality. Therefore, we need a comprehensive framework that allows us to identify, refine, and utilize effective metrics for evaluating generated content in a way that aligns with human judges.

Motivated by this insight, we introduce MetaMetrics, a meta-metric designed to better align with human preferences by calibrating multiple metrics using human assessment scores. Our method is both efficient and fast, eliminating the need for extensive training. Our contributions are three-fold:

- •
We introduce MetaMetrics, a supervisedly calibrated, flexible, and modality- and language-agnostic metric that aligns closely with human preferences on five different tasks. Our metric is designed to operate in two distinct settings: reference-based and reference-free, providing versatility across a wide range of tasks and modalities.
- •
We demonstrate that MetaMetrics is adaptable to a wide range of requirements, including performance optimization and metric efficiency through a comprehensive benchmarking of diverse metrics across a range of language and vision tasks Section 4. We also show that MetaMetrics can be used effectively as a reward model.(^1^11MetaMetrics ranks 4th out of 144 on the overall leaderboard, utilizing the fewest parameters, and secures 2nd place on the detailed leaderboard at [RewardBench](https://huggingface.co/spaces/allenai/reward-bench). The leaderboard reflects data as of October 6th, 2024.) This adaptability ensures that MetaMetrics can be calibrated to suit various human preference aspects.
- •
We demonstrate that MetaMetrics outperforms existing metrics by leveraging complementary metrics to enhance overall performance. We conduct a detailed analysis to understand how MetaMetrics attributes weight or attention to specific metrics, thereby measuring the contribution of each component. We release the code and models to facilitate reproducibility and empower researchers and practitioners to utilize and extend our metric for various applications.(^2^22We release MetaMetrics as an open-source library at [https://github.com/meta-metrics/metametrics](https://github.com/meta-metrics/metametrics).)

Figure: Figure 1: Examples of image captioning on THumB 1.0 dataset comparing MetaMetrics against BLEU and CLIP-S scores. MetaMetrics scores of predicted captions are closer to human ratings compared to BLEU and CLIP-S scores in the left and right images, respectively.
Refer to caption: extracted/5907857/images/example.png

## 2 Aren’t Existing Metrics Good Enough?

The challenge of developing metrics tailored to specific tasks is not a new issue. For a long time, researchers have struggled to identify appropriate metrics that align with human preferences. In this section, we outline the rationale for the need for a new evaluation metric that is both suitable for our tasks and aligned with human preferences.

##### Commonly Used Metrics are Not Robust and Unrepresentative.

Metrics such as Perplexity (PPL) (Jelinek et al., ; Bengio et al., ) and BLEU (Papineni et al., ) are widely used to measure the quality of generated text in various generation tasks. However, they are not always the most suitable metrics, particularly when accounting for variations in writing styles and minor character differences due to diacritics (Freitag et al., ). Critics have noted that PPL is an English-centric metric that performs well for English but is less effective for other languages, such as Japanese (Kuribayashi et al., ). THumB (Kasai et al., ) further revealed that widely used metrics fail to adequately capture the semantic quality and diversity of high-quality human annotations, leading to misleading evaluations of model performance, particularly for captions.

##### Limited Capability of an Individual Metric.

Single metrics like BERTScore (Zhang et al., ), ROUGE (Lin, ), and METEOR (Banerjee & Lavie, ) are beneficial and capable of measuring the quality of generated content. However, they have significant limitations. For instance, in summarization tasks, BERTScore (Recall) excels in assessing consistency but falls short in evaluating coherence. Conversely, BERTScore (F1) performs well in measuring relevance but not consistency (Fabbri et al., ). This variability means that a single metric may perform well in one aspect but poorly in another, making it challenging to select the appropriate metric without extensive benchmarking. This ambiguity complicates the process of choosing and utilizing the most suitable metric for a specific use case.

##### One Metric with Too Many Variants and Implementations.

Metrics like BERTScore allow the use of various BERT models, but the sheer number of options can make it difficult to identify the best one without a systematic approach. This can lead to an exhaustive search process. BERT models trained on English often underperform on non-English languages, and vice versa. Non-English languages in Latin script may also yield poorer results compared to those in their native scripts. Compounding these issues, many metrics are heavily parameterized, yet their settings are frequently undocumented, resulting in score variations across implementations (Post, ; Grusky, ).

To tackle these challenges, we propose a new paradigm for creating customizable metrics that are robust across various tasks and closely aligned with human preferences. Our approach will enable systematic evaluation and automatic selection of the most suitable combination of metrics that is aligned with human judgments on target tasks, ensuring optimal performance and relevance.

## 3 MetaMetrics

In this section, we outline the notations and definitions used in our proposed method, and detail the conditions required for optimizing the method based on human preferences. Additionally, we discuss the key factors influencing the optimization process.

Figure: Figure 2: MetaMetrics Framework for reference-free ($\theta_{\textsc{MM}}^{\text{noref}}$, left) and reference-based setting ($\theta_{\textsc{MM}}^{\text{ref}}$, right). MetaMetrics integrates multiple metrics $\theta_{i}$ and their scores $\hat{y}_{i}$, learns a function $\Phi$ to combine them into a score $\hat{y}_{\textsc{MM}}$ that aligns best with the human judgment score.
Refer to caption: extracted/5907857/images/framework.png

### 3.1 Preliminaries

##### Definition.

We define $\theta_{i}$ as a metric function that maps a sample input $x$ to a score $\hat{y}_{i}$, where $i\in\{1,\ldots,N\}$ denotes different metrics. Note that each $\theta_{i}$ depends on the type of task it is applied to. For reference-based metric, the data is evaluated in the context of $x=(x_{\text{hyp}},x_{\text{ref}})$, where $x_{\text{hyp}}$ and $x_{\text{ref}}$ correspond to the hypothesis text and the reference text, respectively. For reference-free metric, only $x_{\text{hyp}}$ will be used. In VL tasks, the input is extended to include an image, which would correspond to $x=(x_{\text{text}},x_{\text{image}})$ where $x_{\text{text}}$ and $x_{\text{image}}$ correspond to the caption and image, respectively.

Given a set of $N$ evaluation metrics $\{\hat{y}_{1},\ldots,\hat{y}_{N}\}$, we define $\Phi$ to compute a scalar meta-metric score of $\hat{y}_{\textsc{MM}}$. The idea of utilizing multiple metrics is to combine scores from multiple metrics regardless of the metric types. Overall, we define $\theta_{\textsc{MM}}$ as a meta-metric function where $\hat{y}_{\textsc{MM}}$ is computed as follows:

$$ $\displaystyle\hat{y}_{i}=\theta_{i}(x),\quad\hat{y}_{\textsc{MM}}=\theta_{ \textsc{MM}}(x)=\Phi(\hat{y}_{1},\cdots,\hat{y}_{N}).$ (1) $$

The MetaMetrics $\theta_{\textsc{MM}}$ is used to calculate the objective value $\rho(\hat{y}_{\textsc{MM}},z)$, where $\rho$ is measures alignment with $z\in\mathbb{R}$, the human preference score. In practice, $z$ can represent various scores annotated by human judges, such as coherence and fluency.

### 3.2 Human Preference Optimization

##### Notations and Objective.

Recall that we aim to calibrate $\theta_{\textsc{MM}}$, that maximizes an objective calibration function $\rho(\hat{y}_{\text{MM}},z)$, where $z$ denotes human assessment scores–encompassing any score annotated by human evaluators. $\rho$ is a function that measures the alignment between $z$ and MetaMetrics scores, $\hat{y}_{\text{MM}}$. MetaMetrics is designed to combine scores of multiple metrics $\theta_{1}(x),\theta_{2}(x),\ldots,\theta_{N}(x)$, learning the weights $w_{i}$ to assign to each $\hat{y}_{i}=\theta_{i}(x)$ to maximize $\rho(\hat{y}_{\text{MM}},z)$. Each metric has its score $\hat{y}_{i}$ ranges within a specific minimum and maximum value. Some metrics, particularly neural-based ones, can fall outside this defined range. To standardize these metrics, we need to normalize them to a common scale that ranges from 0 to 1. In this scale, 0 represents poor translation performance, while 1 represents perfect translation performance. We apply pre-processing to $\hat{y}_{i}$ before we learn to combine these scores in MetaMetrics training. The pre-processing is defined in Section B.2.

The advantage of MetaMetrics lies in its flexibility and adaptability to different tasks or domains. Certain metrics on certain tasks may exhibit strong correlations with human judgments, and by constructing a composite metric that learns to integrate these reliable metrics with human judgments, we can enhance the overall correlation with human evaluations.

### 3.3 Optimization Methods

In this work, we focus on two optimization methodologies to train MetaMetrics: Bayesian Optimization (BO) and Boosting. BO offers the advantage of interpretability, allowing us to clearly identify which metrics contribute most significantly to the final outcome. Conversely, the Boosting method excels in enhancing alignment and accounting for the compositionality of different metrics, even when dealing with more complex functions. Although we can measure the contribution of each metric, the clarity and distinctness of these contributions are more pronounced with BO compared to Boosting.

#### 3.3.1 Bayesian Optimization (BO)

BO constructs a posterior distribution of functions, typically modeled as a Gaussian Process (GP), to represent the function being optimized. As observations accumulate, the posterior distribution becomes more precise, allowing the algorithm to identify which regions of the parameter space to explore and which to ignore. BO is particularly effective for predicting unknown functions based on observed data, making it a robust method for optimizing complex and costly-to-evaluate functions such as the correlation between evaluation metric scores and human assessment scores.

Formally, we want to solve for the optimum weights $\mathbf{w}^{*}=\arg\max_{\mathbf{w}\in\mathbf{W}}\rho(\hat{y}_{\textsc{MM}}(
\mathbf{w}),z)$, where $\rho(\hat{y}_{\textsc{MM}}(\mathbf{w}),z)$ is the alignment between MetaMetrics score $\hat{y}_{\textsc{MM}}(\mathbf{w})$ with the human preference score $z$, $\mathbf{w}=[w_{1},w_{2},\ldots,w_{N}]$, $\hat{y}_{\textsc{MM}}(\mathbf{w})=w_{1}\hat{y}_{1}+w_{2}\hat{y}_{2}+\ldots+w_{
N}\hat{y}_{N}$, and $\mathbf{W}=[\mathbf{w}_{1},\mathbf{w}_{2},\dots,\mathbf{w}_{K}]$ is the set of weight vectors.

We assume a GP prior: that the alignment between the weighted combination of metric scores and the human preference score follows a GP $\rho(\hat{y}_{\textsc{MM}}(\mathbf{w}),z)\sim\mathcal{GP}(\mu(\mathbf{w}),k(
\mathbf{w},\mathbf{w^{\prime}}))$, where $\mu(\mathbf{w})$ is is the prior mean of the alignment and $k(\mathbf{w},\mathbf{w^{\prime}})$ is the kernel function that measures the similarity between two sets of weights $\mathbf{w}$ and $\mathbf{w^{\prime}}$. After observing a set of weight vectors $\mathbf{W}=[\mathbf{w}_{1},\mathbf{w}_{2},\dots,\mathbf{w}_{k}]$ and their corresponding alignments $\mathbf{\rho}=[\rho_{1},\rho_{2},\dots,\rho_{k}]$, the GP posterior can be updated to predict the alignment for a new weight vector $\mathbf{w}$. To guide the search for the optimal weights $\mathbf{w}$, BO maximizes a function and selects the next weight vector $\mathbf{w}_{k+1}$ to evaluate using the function. BO’s function balances exploration and exploitation using the posterior mean and variance, iterating until convergence or budget exhaustion. While challenging, maximizing the function is computationally efficient with standard tools, making BO suitable for resource-intensive sampling scenarios. GP employs a multivariate Gaussian distribution, offering a sparse representation of metrics that enhances interpretability by identifying key contributors to the objective.

#### 3.3.2 Boosting Method

The boosting method we investigate is Extreme Gradient Boosting (XGBoost) (Chen & Guestrin, ). Gradient-boosted trees have long been recognized as a robust technique, supported by extensive literature demonstrating their effectiveness (Friedman, ). To enhance the efficiency of our metric, we implement iterative pruning to eliminate less important metrics from the input, resulting in a more compact and faster metric.

**Table 1: Kendall correlation results with human ratings for summarization task. Coh., Cons., Rel., and Faith. corresponds to coherence, consistency, relevance, and faithfulness respectively. We only present the best-performing metric variant from each metric category, more detailed results for all metrics and their variants can be found in Table 9 in the Appendix. Bold and underlined values indicate the best and second best performance, respectively. Our MetaMetrics-SUM correlates the best with human ratings across different datasets and aspects of evaluation.**
| Metric | SummEval | Benchmark LLM | Combined |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Coh. | Cons. | Fluency | Rel. | Faith. | Coh. | Cons. | Coh. | Cons. | Avg. |
| BLEU | 0.110 | 0.126 | 0.113 | 0.170 | 0.162 | 0.246 | 0.252 | 0.029 | 0.082 | 0.143 |
| chrF | 0.143 | 0.094 | 0.071 | 0.198 | 0.272 | 0.163 | 0.342 | 0.068 | 0.218 | 0.174 |
| METEOR | 0.077 | 0.102 | 0.072 | 0.162 | 0.251 | 0.182 | 0.331 | 0.033 | 0.164 | 0.153 |
| ROUGE-WE1 | 0.115 | 0.088 | 0.081 | 0.169 | 0.223 | 0.218 | 0.292 | 0.198 | 0.272 | 0.184 |
| BARTScore (Mean) | 0.086 | 0.074 | 0.040 | 0.143 | 0.080 | 0.216 | 0.166 | 0.210 | 0.173 | 0.132 |
| BLEURT (Max) | 0.185 | 0.070 | 0.114 | 0.189 | 0.046 | 0.264 | 0.143 | 0.229 | 0.143 | 0.154 |
| BERTScore (f1) | 0.105 | 0.100 | 0.120 | 0.181 | 0.171 | 0.280 | 0.283 | 0.306 | 0.304 | 0.206 |
| MetaMetrics-Sum |  |  |  |  |  |  |  |  |  |  |
| XGBoost | 0.190 | 0.151 | 0.131 | 0.271 | 0.406 | 0.411 | 0.397 | 0.403 | 0.367 | 0.303 |
| XGBoost (Iterative Top 5) | 0.141 | 0.115 | 0.160 | 0.208 | 0.377 | 0.378 | 0.383 | 0.369 | 0.351 | 0.276 |
| GP | 0.170 | 0.142 | 0.115 | 0.233 | 0.356 | 0.308 | 0.383 | 0.315 | 0.333 | 0.262 |

##### Iterative-based Pruning.

When training an XGBoost regressor model, we can initially use a large set of metrics. However, this approach is less efficient than the BO model at inference time. To address the scalability and efficiency issues, we propose an iterative pruning method (Algorithm 1). We conduct $k$ iterations of XGBoost training, starting from the full set of metrics and removing the metric with the least feature importance at each iteration. We return the best XGBoost model among $k$ iterations based on the cross-validation value of $\rho(\hat{y}_{\text{MM}},z)$ measured during training.

## 4 Experiments Setup

In this work, we explore two optimization methodologies: BO and Boosting. BO provides interpretability, clearly highlighting the metrics that significantly impact the final outcome. In contrast, Boosting enhances alignment and addresses the compositionality of metrics, even for complex functions. While we can measure each metric’s contribution through feature importance in Boosting, BO is more interpretable than Boosting. We also compare calibrating with all the metrics and calibrating only using top-5 correlated metrics from the tuning set.

##### Abstractive Text Summarization.

We use the SummEval (Fabbri et al., ) and BenchmarkLLM (Zhang et al., ) datasets for text summarization evaluation. SummEval, based on CNN/DailyMail (Hermann et al., ), is rated by human annotators on coherence, consistency, fluency, and relevance using a 1 to 5 Likert scale. BenchmarkLLM includes both CNN/DailyMail and XSUM (Narayan et al., ) articles, with summaries assessed for faithfulness (binary scale), coherence, and consistency (1 to 5 Likert scale). To evaluate the generalization capabilities of MetaMetrics, we combine coherence and consistency scores from both datasets, ensuring no overlap in articles, allowing us to assess MetaMetrics’s performance across different human annotators. We use the Kendall $\tau$ correlation function as our objective calibration function $\rho$. We refer to our metric as MetaMetrics-SUM (Table 1).

##### Machine Translation.

For this task, we train both reference-free and reference-based versions of the metric. We train our metric using the 3-year training data from WMT shared tasks datasets from 2020 to 2022 that are annotated using MQM annotation scores. We evaluate on MQM dataset from WMT23 shared task. We use kendall $\tau$ correlation function as our objective calibration function $\rho$. We refer to our metric as MetaMetrics-MT (Table 2).

**Table 2: Kendall correlation results with human ratings on WMT23 (MQM). ^†The results are collected from Freitag et al. (). More detailed results for all metrics and their variants can be found in Table 11 in the Appendix. Bold and underlined values indicate the best and second best performance, respectively. Our MetaMetrics-MT correlates with human judgments better than or comparably to the best WMT evaluation metrics across different languages and evaluation settings.**
| Metric | overall | en-de | he-en | zh-en |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | sys/seg | sys | seg | seg | sys | seg | seg | sys | seg | seg |
|  | avg-corr | pearson | pearson | acc-t | pearson | pearson | acc-t | pearson | pearson | acc-t |
| Reference-based Metric |  |  |  |  |  |  |  |  |  |  |
| chrF^† | 0.694 | 0.866 | 0.232 | 0.519 | 0.776 | 0.221 | 0.460 | 0.809 | 0.063 | 0.485 |
| BLEU^† | 0.696 | 0.917 | 0.192 | 0.520 | 0.769 | 0.220 | 0.442 | 0.734 | 0.119 | 0.472 |
| BERTScore^† | 0.742 | 0.891 | 0.325 | 0.528 | 0.895 | 0.335 | 0.515 | 0.810 | 0.236 | 0.499 |
| Yisi-1^† | 0.754 | 0.925 | 0.366 | 0.542 | 0.917 | 0.395 | 0.529 | 0.823 | 0.290 | 0.504 |
| MetricX-23-XXL^† | 0.808 | 0.977 | 0.585 | 0.603 | 0.910 | 0.548 | 0.577 | 0.873 | 0.625 | 0.531 |
| XCOMET-Ensemble^† | 0.825 | 0.980 | 0.695 | 0.604 | 0.950 | 0.556 | 0.586 | 0.927 | 0.650 | 0.543 |
| COMET | 0.779 | 0.990 | 0.432 | 0.575 | 0.940 | 0.401 | 0.531 | 0.898 | 0.396 | 0.514 |
| MetaMetrics-MT |  |  |  |  |  |  |  |  |  |  |
| GP | 0.819 | 0.970 | 0.638 | 0.610 | 0.947 | 0.546 | 0.590 | 0.900 | 0.646 | 0.539 |
| XGBoost | 0.825 | 0.992 | 0.680 | 0.616 | 0.957 | 0.557 | 0.574 | 0.929 | 0.637 | 0.546 |
| Reference-free Metric |  |  |  |  |  |  |  |  |  |  |
| GEMBA-MQM^† | 0.802 | 0.993 | 0.502 | 0.572 | 0.939 | 0.401 | 0.564 | 0.991 | 0.449 | 0.522 |
| XCOMET-QE-Ensemble^† | 0.808 | 0.974 | 0.679 | 0.588 | 0.909 | 0.498 | 0.554 | 0.892 | 0.647 | 0.533 |
| MetricX-23-QE-XXL | 0.797 | 0.934 | 0.547 | 0.607 | 0.813 | 0.459 | 0.575 | 0.877 | 0.652 | 0.531 |
| CometKiwi-QE-XL | 0.786 | 0.976 | 0.447 | 0.571 | 0.900 | 0.384 | 0.533 | 0.974 | 0.430 | 0.522 |
| MetaMetrics-MT-QE |  |  |  |  |  |  |  |  |  |  |
| GP | 0.801 | 0.934 | 0.556 | 0.609 | 0.815 | 0.474 | 0.578 | 0.900 | 0.660 | 0.537 |
| XGBoost | 0.805 | 0.967 | 0.583 | 0.604 | 0.881 | 0.509 | 0.568 | 0.869 | 0.642 | 0.526 |

##### Question Answering.

For this task, we evaluate MetaMetrics performance across multiple QA subtasks, including Open Domain QA (Open-QA) (Rajpurkar, ; Berant et al., ; Petroni et al., ), Reading Comprehension QA (RCQA) (Bajaj et al., ; Fisch et al., ; Yang et al., ), and Reasoning QA. Open-QA retrieves answers from large, unstructured datasets like Wikipedia, where answers are objective. RCQA requires the model to read a passage to derive answers directly from it, while Reasoning QA emphasizes logical inference, where answers cannot be directly extracted from the passage.

Due to the nature of our paper, we require QA datasets with human evaluations. The data we use are collected from two sources: Evaluating Question Answering Evaluation (EQAE) (Chen et al., ) and EVOUNA (Wang et al., ) and include Open-QA datasets: Natural Questions (NQ) (Kwiatkowski et al., ) and TriviaQA (TQ) (Joshi et al., ), RCQA dataset: NarrativeQA (Kočiskỳ et al., ), and
Reasoning QA dataset: SemEval 2018 Task 11 (SemEval) (Ostermann et al., ). We refer to our metric as MetaMetrics-QA (Table 3).

**Table 3: Kendall correlation results of MetaMetrics-QA. More detailed results for all metrics and their variants can be found in Table 10 in the Appendix. Bold and underlined values indicate the best and second best performance, respectively. In most cases, our MetaMetrics-QA outperforms the other metrics in correlation with human judgments.**
| Setup | EVOUNA | EQAE | Combined |  |  |
| --- | --- | --- | --- | --- | --- |
|  | NQ | TQ | NarrativeQA | SemEval |  |
| ArmoRM Metrics |  |  |  |  |  |
| ultrafeedback-honesty | 0.202 | 0.281 | 0.297 | 0.335 | 0.296 |
| helpsteer-helpfulness | 0.204 | 0.293 | 0.332 | 0.427 | 0.302 |
| helpsteer-correctness | 0.206 | 0.300 | 0.336 | 0.403 | 0.305 |
| argilla-judge-lm | 0.243 | 0.301 | 0.253 | 0.337 | 0.332 |
| n-gram-based Metrics |  |  |  |  |  |
| BLEU1 | 0.461 | 0.353 | 0.362 | 0.260 | 0.424 |
| ROUGE-L | 0.495 | 0.399 | 0.584 | 0.518 | 0.454 |
| METEOR | 0.517 | 0.435 | 0.570 | 0.461 | 0.494 |
| MetaMetrics-QA |  |  |  |  |  |
| GP | 0.518 | 0.448 | 0.586 | 0.464 | 0.512 |
| XGBoost | 0.543 | 0.471 | 0.601 | 0.486 | 0.536 |
| XGBoost (Iterative Top 5) | 0.540 | 0.488 | 0.626 | 0.503 | 0.528 |

##### Image Captioning.

We use Flickr8k-Expert (Hodosh et al., ) and THumB 1.0 (Kasai et al., ) human evaluation datasets. Flickr8k-Expert consists 5,882 image-caption pairs, each having 3 human annotators with a scale of 1–4 where 1 is worst and 4 is best. The THumB 1.0 dataset is a rubric-based human evaluation framework applied to a subset of the MSCOCO dataset (Lin et al., ). The final score is of a scale of 1–5 where 1 is worst and 5 is best. We refer to our metric as MetaMetrics-Cap (Figure 3).

##### Reward Model Scoring.

We use MetaMetrics as reward model to score text generation from LLM. We use RewardBench (Lambert et al., ) as the test benchmark. We use Skywork Reward Data Collection(^3^33Skyword Reward Data Collection can be accessed at [https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1)) for training our MetaMetrics, including HelpSteer2 (Wang et al., ), OffsetBias (Park et al., ), WildGuard (Han et al., ), and Magpie (Xu et al., ). In addition, we add Preference Test Data(^4^44Preference Test Data from AI2 can be accessed at [https://huggingface.co/datasets/allenai/preference-test-sets](https://huggingface.co/datasets/allenai/preference-test-sets)) to increase the size of the human preference dataset. We refer to our metric as MetaMetrics-RM and report the accuracy of predicting that the chosen sample is ranked higher than the rejected sample (Table 4).

## 5 Results

In this section, we present the results of our MetaMetrics across five diverse tasks, covering both NLP and VL downstream applications.

##### Abstractive Text Summarization.

For this task, we refer to our metric as MetaMetrics-SUM. Table 9 shows the results of abstractive text summarization task. Overall, our proposed model with XGBoost outperforms all other baselines, including all ensemble models and overall best automatic metric (BERTScore (F1)). An interesting observation we find is our iterative model with fewer features achieves slightly better average correlation score ($\Delta$ 0.004) compared to the model with full metrics. The MetaMetrics-SUM with GP also performs considerably well, also outperforms all other baselines.

Figure: Figure 3: Kendall correlation results of MetaMetrics-Cap. Cross-Dataset means that MetaMetrics is tuned on THumB 1.0 and tested on Flickr8k, and vice versa. Our MetaMetrics-Cap outperforms other metrics across different datasets and shows robustness even when trained and tested on different datasets (Cross-Dataset setting). More detailed results for all metrics and their variants can be found in Table 12 in the Appendix.
Refer to caption: extracted/5907857/images/captionplot.png

**Table 4: Accuracy results in percentages for Reward-Model-As-A-Metric. More detailed results for all metrics and their variants can be found in Table 13 in the Appendix. Values that are bolded indicate the best performance, while those that are underlined represent the second-best performance. Our MetaMetrics-RM achieves the best or comparable results in 2 out of the 4 categories in RewardBench.**
| Metric | Score | Chat | Chat Hard | Safety | Reasoning |
| --- | --- | --- | --- | --- | --- |
| Llama-3.1-Nemotron-70B-Reward | 94.1 | 97.5 | 85.7 | 95.1 | 98.1 |
| Skywork-Reward-Gemma-2-27B | 93.8 | 95.8 | 91.4 | 91.9 | 96.1 |
| TextEval-Llama3.1-70B | 93.5 | 94.1 | 90.1 | 93.2 | 96.4 |
| Skywork-Critic-Llama-3.1-70B | 93.3 | 96.6 | 87.9 | 93.1 | 95.5 |
| SFR-LLaMa-3.1-70B-Judge-r | 92.7 | 96.9 | 84.8 | 91.6 | 97.6 |
| URM-LLaMa-3.1-8B | 92.9 | 95.5 | 88.2 | 91.1 | 97.0 |
| Skywork-Reward-Llama-3.1-8B | 92.5 | 95.8 | 87.3 | 90.8 | 96.2 |
| GRM-Llama3-8B | 91.5 | 95.5 | 86.2 | 90.8 | 93.6 |
| ArmoRM-Llama3-8B-v0.1 | 90.4 | 96.9 | 76.8 | 90.5 | 97.3 |
| URM-LLaMa-3-8B | 89.9 | 96.9 | 78.7 | 88.2 | 95.7 |
| MetaMetrics-RM (GP) | 93.4 | 98.3 | 86.4 | 90.8 | 98.2 |
| MetaMetrics-RM-XGBoost |  |  |  |  |  |
| Iterative Best | 92.9 | 95.8 | 88.4 | 91.8 | 95.8 |

##### Machine Translation.

In this task, we demonstrate that MetaMetrics-MT in the reference-based setting outperforms all baseline models, including ensemble methods (Table 2). Notably, the top competitor, XCOMET-Ensemble, achieves impressive results on the WMT23 evaluation leaderboard, particularly excelling in the two-segment Pearson correlation metric for en-de and zh-en. However, MetaMetrics consistently surpasses XCOMET-Ensemble across all other tasks, leading to a higher average correlation overall, achieving the new state-of-the-art performance on WMT23 evaluation.

In the reference-free setting, MetaMetrics-MT-QE performs comparably to XCOMET-QE-Ensemble. While XCOMET-Ensemble shows slightly better average correlation, our proposed model demonstrates superior performance on the acc-t metric. This is particularly significant, as the acc-t metric is finely tuned to align with the Kendall correlation $\tau$ objective function, indicating a stronger correlation with human evaluations.

##### Question Answering.

Table 3 presents the results of MetaMetrics-QA, which consistently outperforms both standard and existing metrics across all evaluation levels. While ROUGE performs slightly better than our method in the SemEval dataset, our proposed model demonstrates a significant advantage over others in different datasets, achieving improvements of up to 0.044 points. Notably, our iterative-based approach enhances the performance of the vanilla MetaMetrics-QA when using XGB, all while utilizing fewer metrics. Additionally, for the GP evaluation, selecting all metrics or just the top five yields comparable combined scores, indicating the robustness of our metric selection strategy.

##### Image Captioning.

Figure 3 presents the results of MetaMetrics-Cap and cross-dataset tuning. MetaMetrics-Cap consistently performs better compared to any individual metric, even when the tuning is performed on a dataset from a different distribution. Although tuning on the same distribution yields the best results, we observe that MetaMetrics-Cap still outperforms any individual metric, demonstrating its robustness across varying data distributions.

##### Metric Model Scoring.

Table 4 highlights the performance of MetaMetrics-RM, which achieves state-of-the-art results(^5^55We use the RewardBench leaderboard from October 6th, 2024, as the basis for comparison.) for GP and competes effectively with the top seven metrics, including the advanced 70B Nemotron Reward model. These results demonstrate that our proposed metric is both efficient and effective, showcasing its robustness across diverse categories of tasks, datasets, and evaluation settings. Overall, MetaMetrics-RM represents a significant advancement in reward metric evaluation, offering high performance and broad applicability for researchers and practitioners alike.

## 6 Analysis and Discussion

In this section, we analyze the advantages of the methods in terms of interpretability, efficiency, and robustness. Additionally, we discuss how MetaMetrics enhances the capability of model evaluation.

### 6.1 Interpretability

Figure: Figure 4: GP weights (Left) and XGBoost Features Importance (Center) with Intra-Metric Correlation (Right) for MetaMetrics-QA.
Refer to caption: extracted/5907857/images/h_gbvxgb_full_final_3.png

Interpretability is a key aspect in MetaMetrics, as the optimization process inherently reveals the impact of each metric on the final score. The chosen methods, such as feature importance in Boosting and weights analysis in Bayesian Optimization, provide a straightforward way to quantify each metric’s contribution to human preference alignment. As shown in Figure 4, this interpretability is highlighted in the QA task, showcasing the importance of each metric in human preference alignment. For example, since ROUGE and ROUGE-L are two highly correlated metrics, Bayesian Optimization often drops one of them during the tuning process.

### 6.2 Efficiency

##### Improved Performance with Fewer Parameters.

Table 4
highlights the compute efficiency and superior performance of MetaMetrics-RM. MetaMetrics-RM, utilizing GP, combines GRM-Gemma-2B, Internlm2-7B-Reward, Skywork-Reward-Llama-3.1-8B, Internlm2-1.8B-Reward, and GRM-Llama3-8B totaling approximately 27B parameters. Despite this relatively small compute memory footprint, it outperforms models more than 2$\times$ its size. This demonstrates that MetaMetrics-RM offers state-of-the-art accuracy using significantly more cost-effective models, making it an ideal choice for resource-constrained applications while delivering even better performance.

##### High Parallelization.

MetaMetrics is designed to be embarrassingly parallel, enabling each metric to be executed independently without requiring inter-model communication, which results in exceptional efficiency. Unlike large models with 27 billion parameters that often face bottlenecks, our approach supports the development of highly effective models without such time constraints. Furthermore, MetaMetrics excels at selecting a sparse yet impactful set of metrics from a vast pool of potential evaluation metrics. As shown in Figure 4, both GP and XGBoost can identify a small subset of key metrics that strongly correlate with human preferences, thereby reducing the need to evaluate a wide range of metrics. This selective approach identifies essential metrics for specific tasks while minimizing computational overhead without compromising performance. In essence, MetaMetrics is ideally suited for low-resource settings, providing superior alignment with human preferences across multiple tasks.

### 6.3 Robustness

We evaluate the robustness of MetaMetrics through cross-dataset experiments in image captioning, tuning on one dataset and testing on another. Despite variations in domain and content, our results consistently demonstrate the effectiveness of MetaMetrics. Figure 3 shows that MetaMetrics-Cap tuned on THumB 1.0 and tested on Flickr8k and vice-versa out performs all individual metrics.
This robustness indicates that MetaMetrics is not overfitting to specific dataset characteristics, but rather learning to combine metrics in a way that aligns with general human judgments of caption quality. This generalization capability is crucial for real-world applications, where the evaluation metric may need to perform well on diverse and potentially out-of-domain data.

## 7 Related Work

Performance evaluation metrics for natural language generation tasks can be categorized into several categories based on how the metric compares generated texts with reference texts.

##### Surface-level Metrics.

These metrics compare the generated text to reference text at the word level, focusing on n-gram overlap or direct lexical matching. Common metrics include BLEU that measures precision of n-grams between the generated- and reference-texts (Papineni et al., ), ROUGE that measures recall (Lin, ), METEOR (Banerjee & Lavie, ) that goes beyond exact word overlap and considers stemming, synonyms, and paraphrasing, and chrF (Popović, ) that calculates similarity based on character n-grams rather than word n-grams using F-score.

##### Embedding-based Metrics.

These metrics rely on word or sentence embeddings to measure semantic similarity between generated and reference text, capturing deeper meaning rather than surface overlap. Examples include MoverScore (Zhao et al., ), which considers semantic similarity at the word level and accounts for word movement (alignment) between texts, and BERTScore (Zhang et al., ), a neural-based metric that compares semantic similarity and understands contextual relationships using BERT embeddings. COMET (Rei et al., ) and BLEURT (Sellam et al., ) are neural-based metrics that use contextualized embeddings from models like BERT as inputs to train models to generate prediction estimates of human judgments. For vision-language tasks, ClipScore (Hessel et al., ) uses embeddings to compare vision and language modalities.

##### Ensemble-based Metrics.

Ensemble-based metrics combine multiple individual metrics to enhance evaluation robustness through averaging (Adiwardana et al., ) or heuristics (Phy et al., ). The Absolute-Rating Multi-Objective Reward Model (ArmoRM) (Wang et al., ) is a SotA reward model designed for RLHF (Ouyang et al., ). ArmoRM uses absolute ratings across 19 objectives trained to predict human preference scores.

## 8 Conclusion

Understanding the quality of a performance evaluation metric is crucial for ensuring alignment with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics, a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our method demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. Our proposed metric aligns closely with human preferences and it is also highly extendable and easily integrable into any application. This makes it a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts.

## Limitations and Future Works

In our experiment, we utilize metrics that are commonly employed to evaluate each downstream task. However, due to limitations in computational resources, we exclude LLMs exceeding 10.7B parameters to ensure that our metrics can be executed on commercial GPU resources with a maximum memory capacity of 40GB. Additionally, we restrict our exploration of metrics to avoid exhaustively incorporating LLM-based metrics, given our capacity and resource constraints.

Looking ahead, there are significant opportunities for further exploration. We can investigate metrics across additional languages and in larger multilingual generation tasks, where we identify MetaMetrics for multilingual applications (Winata et al., ). Furthermore, exploring metrics in the context of speech modalities presents another promising avenue for research.

## Appendix A Metrics

In this section, we list all metrics we use in our experiments.

### A.1 Abstractive Text Summarization

We use the following metrics:

n-Gram-based Metrics: BLEU-4 (Papineni et al., ), chrF (Popović, ), METEOR (Banerjee & Lavie, ), ROUGE-1, ROUGE-3 , ROUGE-4, ROUGE-L (Lin, ), ROUGE-WE1 (Ng & Abrecht, ).

Embedding-based Metrics: BARTScore (Yuan et al., ), BLEURT (Sellam et al., ), BERTScore (Zhang et al., ).

LLM-based Metrics: SummaQA (Scialom et al., ).

### A.2 Machine Translation

We use the following metrics:

n-Gram-based Metrics: chrF (Popović, ), BLEU (Papineni et al., );

Embedding-based Metrics: BERTScore (Zhang et al., ), Yisi-1 (Lo, );

LLM-based Metrics: MetricX-23 (L, XL, XXL) (Juraska et al., ), COMET, CometKiwi (Rei et al., ), XCOMET (Guerreiro et al., ), GEMBA-MQM (Kocmi & Federmann, ).

For machine translation, we optimize the Kendall correlation during our tuning.

### A.3 Question Answering

We use the following metrics:

n-Gram-based Metrics: BLEU-1, BLEU-4 BLEU (Papineni et al., ), ROUGE, ROUGE-L (Lin, ), METEOR (Banerjee & Lavie, );

Embedding-based Metrics: BERTScore (Zhang et al., );

LLM-based Metrics: ArmoRM-Llama3-8B-v0.1 (ArmoRM) (Wang et al., ).

### A.4 Image Captioning

We use the following metrics:

n-Gram based metrics BLEU (Papineni et al., ), ROUGE (Lin, ), METEOR (Banerjee & Lavie, ), SPICE (Anderson et al., ), CIDEr (Vedantam et al., );

Embedding-based Metrics TIGEr (Jiang et al., ) and CLIPScore (Both reference-based and reference-free) (Hessel et al., ).

### A.5 Reward Model Scoring

The reward model used in the experiment is listed in Table 5.

**Table 5: Reward Models.**
| Reward Model / Metric | Hugging Face Link |
| --- | --- |
| nvidia/Llama-3.1-Nemotron-70B-Reward | [https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward](https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward) |
| SF-Foundation/TextEval-Llama3.1-70B | [https://huggingface.co/SF-Foundation/TextEval-Llama3.1-70B](https://huggingface.co/SF-Foundation/TextEval-Llama3.1-70B) |
| Skywork/Skywork-Critic-Llama-3.1-70B | [https://huggingface.co/Skywork/Skywork-Critic-Llama-3.1-70B](https://huggingface.co/Skywork/Skywork-Critic-Llama-3.1-70B) |
| Salesforce/SFR-LLaMa-3.1-70B-Judge-r | [https://huggingface.co/Salesforce/SFR-LLaMa-3.1-70B-Judge-r](https://huggingface.co/Salesforce/SFR-LLaMa-3.1-70B-Judge-r) |
| internlm/internlm2-1_8b-reward (Cai et al., ) | [https://huggingface.co/internlm/internlm2-1_8b-reward](https://huggingface.co/internlm/internlm2-1_8b-reward) |
| internlm/internlm2-7b-reward (Cai et al., ) | [https://huggingface.co/internlm/internlm2-7b-reward](https://huggingface.co/internlm/internlm2-7b-reward) |
| LxzGordon/URM-LLaMa-3-8B | [https://huggingface.co/LxzGordon/URM-LLaMa-3-8B](https://huggingface.co/LxzGordon/URM-LLaMa-3-8B) |
| LxzGordon/URM-LLaMa-3.1-8B | [https://huggingface.co/LxzGordon/URM-LLaMa-3.1-8B](https://huggingface.co/LxzGordon/URM-LLaMa-3.1-8B) |
| Ray2333/GRM-Gemma-2B-rewardmodel-ft | [https://huggingface.co/Ray2333/GRM-Gemma-2B-rewardmodel-ft](https://huggingface.co/Ray2333/GRM-Gemma-2B-rewardmodel-ft) |
| Ray2333/GRM-Llama3-8B-rewardmodel-ft | [https://huggingface.co/Ray2333/GRM-Llama3-8B-rewardmodel-ft](https://huggingface.co/Ray2333/GRM-Llama3-8B-rewardmodel-ft) |
| sfairXC/FsfairX-LLaMA3-RM-v0.1 | [https://huggingface.co/sfairXC/FsfairX-LLaMA3-RM-v0.1](https://huggingface.co/sfairXC/FsfairX-LLaMA3-RM-v0.1) |
| Skywork/Skywork-Reward-Llama-3.1-8B | [https://huggingface.co/Skywork/Skywork-Reward-Llama-3.1-8B](https://huggingface.co/Skywork/Skywork-Reward-Llama-3.1-8B) |
| weqweasdas/RM-Mistral-7B | [https://huggingface.co/weqweasdas/RM-Mistral-7B](https://huggingface.co/weqweasdas/RM-Mistral-7B) |

## Appendix B Datasets

In this section, we present the data statistics and outline the preprocessing procedures used in our work.

### B.1 Data Statistics

Table 6 presents the dataset split sizes for tuning and testing.

**Table 6: Dataset statistics used in the experiments.**
| Dataset Name | Type | Tuning Size | Testing Size |
| --- | --- | --- | --- |
| Machine Translation |  |  |  |
| WMT23 (MQM) (Freitag et al., ) | Translation | - | 136,260 |
| WMT20-22 (MQM) (Freitag et al., ) | Translation | 150,343 | - |
| Text Summarization |  |  |  |
| SummEval (Fabbri et al., ) | Summarization | 510 | 1,190 |
| BenchmarkLLM (Zhang et al., ) | Summarization | 1,035 | 2,415 |
| Combined | Summarization | 1,545 | 3,605 |
| Question Answering |  |  |  |
| NQ (Kwiatkowski et al., ) | Open-QA | 4,528 | 10,567 |
| TQ (Joshi et al., ) | Open-QA | 2,906 | 6,783 |
| NarrativeQA (Kočiskỳ et al., ) | RCQA | 150 | 350 |
| SemEval (Ostermann et al., ) | Reasoning QA | 90 | 210 |
| Combined | Multidomain-QA | 7,675 | 17,909 |
| Image Captioning |  |  |  |
| Flickr8k-Expert (Hodosh et al., ) | Captioning | 1,746 | 4,076 |
| THumB 1.0 (Kasai et al., ) | Captioning | 600 | 1,400 |
| Reward Model Scoring |  |  |  |
| Skywork-Reward-Preference-80K-v0.1 | Math, Code, Chat | 81,973 | - |
| allenai/preference-test-sets | Math, Code, Chat | 43,175 | - |
| RewardBench (Lambert et al., ) | Chat, Safety, Reasoning | - | 8,108 |

### B.2 Data Preprocessing

##### Pre-processing.

We apply pre-processing before we use the scores in the training. The pre-processing can be defined as follows:

- 1.
Clipping: Let the valid range for $\hat{y}_{i}$ be defined by $[\hat{y}_{i}^{\text{min}},\hat{y}_{i}^{\text{max}}]$. The clipped metric $\hat{y}_{i}^{\prime}$ can be defined as:
$\displaystyle\hat{y}_{i}^{\prime}=\begin{cases}\hat{y}_{i}^{\text{min}}&\text{
if }\hat{y}_{i}<\hat{y}_{i}^{\text{min}},\\
\hat{y}_{i}&\text{if }\hat{y}_{i}^{\text{min}}\leq\hat{y}_{i}\leq\hat{y}_{i}^{
\text{max}},\\
\hat{y}_{i}^{\text{max}}&\text{if }\hat{y}_{i}>\hat{y}_{i}^{\text{max}}.\end{cases}$
(2)
- 2.
Normalization: After clipping, the score is normalized to a common scale of $[0,1]$:
$\displaystyle\hat{y}_{i}=\frac{\hat{y}_{i}^{\prime}-\hat{y}_{i}^{\text{min}}}{
\hat{y}_{i}^{\text{max}}-\hat{y}_{i}^{\text{min}}}.$
(3)
- 3.
Inversion (if applicable): If the metric is such that higher scores indicate worse performance, we invert the normalized score:
$\displaystyle\hat{y}_{i}=1-\hat{y}_{i}.$
(4)

## Appendix C Optimization

In this section, we explore the optimization details of MetaMetrics.

### C.1 Method Details

#### C.1.1 Matern Kernel

We train BO using GP with Matern kernel, a generalization of the RBF, distinguished by an additional parameter that controls the smoothness of the resulting function (Williams & Rasmussen, ). Conversely, as the parameter approaches infinity, the Matern kernel converges to the RBF kernel. The kernel is described as below:

$$ $\displaystyle k(\mathbf{w},\mathbf{w^{\prime}})=\frac{1}{\Gamma(\nu)2^{\nu-1}} \Bigg{(}\frac{\sqrt{2\nu}}{l}d(\mathbf{w},\mathbf{w^{\prime}})\Bigg{)}^{\nu}K_ {\nu}\Bigg{(}\frac{\sqrt{2\nu}}{l}d(\mathbf{w},\mathbf{w^{\prime}})\Bigg{)},$ (5) $$

where $d(\cdot,\cdot)$ is the Euclidean distance, $K_{\nu}(\cdot)$ is a modified Bessel function and $\Gamma(\cdot)$ is the gamma function.

#### C.1.2 Iterative-based Pruning

Below is the implementation of Iterative-based Pruning performed during training.

Figure: Algorithm 1 Iterative-based Pruning with XGBoost

### C.2 Hyper-parameters

#### C.2.1 Bayesian Optimization

Table 7 describes the hyper-parameter settings that we use for our experiments. For the Bayesian optimization, we run GP with a Matern kernel Williams & Rasmussen (), a generalization of the RBF kernel, using $\nu=2.5$.

**Table 7: Hyper-parameters used for Gaussian Process on each task.**
| Metric | Hyper-parameter | Value |
| --- | --- | --- |
| MetaMetrics-SUM | init$_$points | 5 |
|  | n$_$iter | 100 |
| MetaMetrics-MT | init$_$points | 5 |
|  | n$_$iter | 100 |
| MetaMetrics-QA | init$_$points | 5 |
|  | n$_$iter | 40 |
| MetaMetrics-Cap | init$_$points | 5 |
|  | n$_$iter | 100 |
| MetaMetrics-RM | init$_$points | 5 |
|  | n$_$iter | 100 |

#### C.2.2 Boosting Method

For XGBoost training, we use a different objective depending on the task. We perform parameter searching with hyper-parameter values in Table 8.

**Table 8: Initial hyper-parameter values used for parameter searching during XGBoost.**
| Metric | Hyper-parameter | Value |
| --- | --- | --- |
| MetaMetrics-SUM | n$_$estimators$_$low | 100 |
|  | n$_$estimators$_$high | 1000 |
|  | n$_$estimators$_$step | 100 |
|  | n$_$estimators$_$prior | Uniform |
|  | objective | reg$:$absoluteerror |
| MetaMetrics-MT | n$_$estimators$_$low | 100 |
|  | n$_$estimators$_$high | 1000 |
|  | n$_$estimators$_$step | 100 |
|  | n$_$estimators$_$prior | Uniform |
|  | objective | reg$:$absoluteerror |
| MetaMetrics-QA | n$_$estimators$_$low | 100 |
|  | n$_$estimators$_$high | 400 |
|  | n$_$estimators$_$step | 25 |
|  | n$_$estimators$_$prior | Uniform |
|  | objective | reg$:$squaredlogerror |
| MetaMetrics-Cap | n$_$estimators$_$low | 100 |
|  | n$_$estimators$_$high | 1000 |
|  | n$_$estimators$_$step | 100 |
|  | n$_$estimators$_$prior | Uniform |
|  | objective | reg$:$squaredlogerror |
| MetaMetrics-RM | n$_$estimators$_$low | 100 |
|  | n$_$estimators$_$high | 1000 |
|  | n$_$estimators$_$step | 100 |
|  | n$_$estimators$_$prior | Uniform |
|  | objective | rank:pairwise |

**Table 9: Test result for summarization task. Coh., Cons., Rel., and Faith. corresponds to coherence, consistency, relevance, and faithfulness respectively. Bold and underlined values indicate the best and second best performance, respectively. The Combined column reflects results when all datasets are merged.**
| Metric | SummEval | Benchmark LLM | Combined |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Coh. | Cons. | Fluency | Rel. | Faith. | Coh. | Cons. | Coh. | Cons. | Avg. |
| BLEU | 0.110 | 0.126 | 0.113 | 0.170 | 0.162 | 0.246 | 0.252 | 0.029 | 0.082 | 0.143 |
| chrF | 0.143 | 0.094 | 0.071 | 0.198 | 0.272 | 0.163 | 0.342 | 0.068 | 0.218 | 0.174 |
| METEOR | 0.077 | 0.102 | 0.072 | 0.162 | 0.251 | 0.182 | 0.331 | 0.033 | 0.164 | 0.153 |
| ROUGE-WE1 | 0.115 | 0.088 | 0.081 | 0.169 | 0.223 | 0.218 | 0.292 | 0.198 | 0.272 | 0.184 |
| ROUGE-1 | 0.112 | 0.103 | 0.061 | 0.189 | 0.195 | 0.252 | 0.290 | 0.198 | 0.248 | 0.183 |
| ROUGE-3 | 0.099 | 0.115 | 0.048 | 0.154 | 0.138 | 0.142 | 0.225 | 0.112 | 0.191 | 0.136 |
| ROUGE-4 | 0.096 | 0.122 | 0.067 | 0.143 | 0.138 | 0.085 | 0.200 | 0.043 | 0.148 | 0.116 |
| ROUGE-l | 0.110 | 0.087 | 0.069 | 0.157 | 0.137 | 0.270 | 0.239 | 0.217 | 0.210 | 0.166 |
| BARTScore (Max) | 0.084 | 0.065 | 0.048 | 0.135 | 0.080 | 0.216 | 0.166 | 0.054 | 0.057 | 0.101 |
| BARTScore (Mean) | 0.086 | 0.074 | 0.040 | 0.143 | 0.080 | 0.216 | 0.166 | 0.210 | 0.173 | 0.132 |
| BLEURT (Max) | 0.185 | 0.070 | 0.114 | 0.189 | 0.046 | 0.264 | 0.143 | 0.229 | 0.143 | 0.154 |
| BLEURT (Mean) | 0.099 | 0.011 | 0.029 | 0.130 | 0.046 | 0.264 | 0.143 | 0.265 | 0.173 | 0.129 |
| BERTScore (f1) | 0.105 | 0.100 | 0.120 | 0.181 | 0.171 | 0.280 | 0.283 | 0.306 | 0.304 | 0.206 |
| BERTScore (Recall) | 0.104 | 0.111 | 0.093 | 0.176 | 0.228 | 0.223 | 0.333 | 0.218 | 0.300 | 0.198 |
| SummaQA (F1) | 0.051 | 0.120 | 0.106 | 0.114 | 0.318 | 0.038 | 0.293 | -0.014 | 0.180 | 0.134 |
| SummaQA (Prob) | 0.070 | 0.100 | 0.070 | 0.138 | 0.330 | -0.003 | 0.303 | -0.028 | 0.206 | 0.132 |
| MetaMetrics-Sum |  |  |  |  |  |  |  |  |  |  |
| GP (Full) | 0.170 | 0.142 | 0.115 | 0.233 | 0.356 | 0.308 | 0.383 | 0.315 | 0.333 | 0.262 |
| GP (Top 5) | 0.149 | 0.137 | 0.117 | 0.211 | 0.342 | 0.307 | 0.370 | 0.296 | 0.316 | 0.249 |
| GP (Oracle Top 5) | 0.169 | 0.138 | 0.121 | 0.230 | 0.356 | .308 | 0.384 | 0.296 | 0.316 | 0.258 |
| XGBoost | 0.190 | 0.151 | 0.131 | 0.271 | 0.406 | 0.411 | 0.397 | 0.403 | 0.367 | 0.303 |
| XGBoost (Iterative Best) | 0.185 | 0.165 | 0.181 | 0.255 | 0.406 | 0.404 | 0.394 | 0.402 | 0.369 | 0.307 |
| XGBoost (Iterative Top 5) | 0.141 | 0.115 | 0.160 | 0.208 | 0.377 | 0.378 | 0.383 | 0.369 | 0.351 | 0.276 |
| XGBoost (Top 5) | 0.110 | 0.096 | 0.120 | 0.186 | 0.314 | 0.353 | 0.369 | 0.343 | 0.327 | 0.246 |
| XGBoost (Oracle Top 5) | 0.135 | 0.090 | 0.134 | 0.219 | 0.319 | 0.354 | 0.378 | 0.343 | 0.325 | 0.255 |

**Table 10: Full performance of MetaMetrics-QA compared to the 8 best-performing standalone metrics and other baseline setups. Bold and underlined values indicate the best and second best performance, respectively. The Combined column reflects results when all datasets are merged.**
| Setup | EVOUNA | EQAE | Combined |  |  |
| --- | --- | --- | --- | --- | --- |
|  | NQ | TQ | NarrativeQA | SemEval |  |
| ArmoRM Metrics |  |  |  |  |  |
| ultrafeedback-honesty | 0.202 | 0.281 | 0.297 | 0.335 | 0.296 |
| helpsteer-helpfulness | 0.204 | 0.293 | 0.332 | 0.427 | 0.302 |
| helpsteer-correctness | 0.206 | 0.300 | 0.336 | 0.403 | 0.305 |
| argilla-judge-lm | 0.243 | 0.301 | 0.253 | 0.337 | 0.332 |
| n-gram-based Metrics |  |  |  |  |  |
| BLEU1 | 0.461 | 0.353 | 0.362 | 0.260 | 0.424 |
| ROUGE | 0.494 | 0.399 | 0.586 | 0.521 | 0.454 |
| ROUGE-L | 0.495 | 0.399 | 0.584 | 0.518 | 0.454 |
| METEOR | 0.517 | 0.435 | 0.570 | 0.461 | 0.494 |
| MetaMetrics-QA |  |  |  |  |  |
| GP (Full) | 0.518 | 0.448 | 0.586 | 0.464 | 0.512 |
| GP (Top 5) | 0.506 | 0.450 | 0.570 | 0.452 | 0.512 |
| GP (Oracle) | 0.523 | 0.446 | 0.570 | 0.474 | 0.512 |
| XGBoost (Full) | 0.543 | 0.471 | 0.601 | 0.486 | 0.536 |
| XGBoost (Top 5) | 0.535 | 0.491 | 0.624 | 0.513 | 0.534 |
| XGBoost (Oracle) | 0.543 | 0.479 | 0.619 | 0.520 | 0.534 |
| XGBoost (Iterative Best) | 0.545 | 0.488 | 0.626 | 0.510 | 0.538 |
| XGBoost (Iterative Top 5) | 0.540 | 0.488 | 0.626 | 0.503 | 0.528 |

## Appendix D Detailed Results

In this section, we provide more detailed results that could not be included in the main paper due to space constraints.

### D.1 RewardBench Leaderboard

The leaderboard ranking screenshot is shown in Figure 5.

Figure: Figure 5: RewardBench Leaderboard Screenshot (Updated on October 6th, 2024).
Refer to caption: extracted/5907857/images/reward_bench.png

Figure: Figure 6: RewardBench Detailed Leaderboard Screenshot (Updated on October 6th, 2024).
Refer to caption: extracted/5907857/images/reward_bench_detail.png

**Table 11: Correlation with human ratings on WMT23 (MQM). ^†The results are collected from Freitag et al. (). Bold and underlined values indicate the best and second best performance, respectively.**
| Metric | overall | en-de | he-en | zh-en |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | sys/seg | sys | seg | seg | sys | seg | seg | sys | seg | seg |
|  | avg-corr | pearson | pearson | acc-t | pearson | pearson | acc-t | pearson | pearson | acc-t |
| Reference-based Metric |  |  |  |  |  |  |  |  |  |  |
| chrF^† | 0.694 | 0.866 | 0.232 | 0.519 | 0.776 | 0.221 | 0.460 | 0.809 | 0.063 | 0.485 |
| BLEU^† | 0.696 | 0.917 | 0.192 | 0.520 | 0.769 | 0.220 | 0.442 | 0.734 | 0.119 | 0.472 |
| BERTScore^† | 0.742 | 0.891 | 0.325 | 0.528 | 0.895 | 0.335 | 0.515 | 0.810 | 0.236 | 0.499 |
| Yisi-1^† | 0.754 | 0.925 | 0.366 | 0.542 | 0.917 | 0.395 | 0.529 | 0.823 | 0.290 | 0.504 |
| MetricX-23-XXL^† | 0.808 | 0.977 | 0.585 | 0.603 | 0.910 | 0.548 | 0.577 | 0.873 | 0.625 | 0.531 |
| XCOMET-Ensemble^† | 0.825 | 0.980 | 0.695 | 0.604 | 0.950 | 0.556 | 0.586 | 0.927 | 0.650 | 0.543 |
| COMET | 0.779 | 0.990 | 0.432 | 0.575 | 0.940 | 0.401 | 0.531 | 0.898 | 0.396 | 0.514 |
| XCOMET-XL | 0.817 | 0.970 | 0.670 | 0.596 | 0.949 | 0.530 | 0.576 | 0.928 | 0.607 | 0.530 |
| XCOMET-XXL | 0.817 | 0.983 | 0.660 | 0.602 | 0.952 | 0.465 | 0.560 | 0.965 | 0.597 | 0.546 |
| MetaMetrics-MT |  |  |  |  |  |  |  |  |  |  |
| GP | 0.819 | 0.970 | 0.638 | 0.610 | 0.947 | 0.546 | 0.590 | 0.900 | 0.646 | 0.539 |
| XGBoost | 0.825 | 0.992 | 0.680 | 0.616 | 0.957 | 0.557 | 0.574 | 0.929 | 0.637 | 0.546 |
| Reference-free Metric |  |  |  |  |  |  |  |  |  |  |
| mbr-metricx-qe^† | 0.788 | 0.976 | 0.571 | 0.584 | 0.915 | 0.411 | 0.553 | 0.936 | 0.489 | 0.537 |
| MetricX-23-QE^† | 0.800 | 0.969 | 0.626 | 0.596 | 0.858 | 0.520 | 0.564 | 0.859 | 0.647 | 0.527 |
| GEMBA-MQM^† | 0.802 | 0.993 | 0.502 | 0.572 | 0.939 | 0.401 | 0.564 | 0.991 | 0.449 | 0.522 |
| XCOMET-QE-Ensemble^† | 0.808 | 0.974 | 0.679 | 0.588 | 0.909 | 0.498 | 0.554 | 0.892 | 0.647 | 0.533 |
| CometKiwi-QE | 0.781 | 0.946 | 0.475 | 0.569 | 0.859 | 0.387 | 0.544 | 0.963 | 0.442 | 0.525 |
| MetricX-23-QE-L | 0.763 | 0.868 | 0.501 | 0.577 | 0.616 | 0.419 | 0.536 | 0.835 | 0.622 | 0.508 |
| MetricX-23-QE-XXL | 0.797 | 0.934 | 0.547 | 0.607 | 0.813 | 0.459 | 0.575 | 0.877 | 0.652 | 0.531 |
| CometKiwi-QE-XL | 0.786 | 0.976 | 0.447 | 0.571 | 0.900 | 0.384 | 0.533 | 0.974 | 0.430 | 0.522 |
| MetaMetrics-MT-QE |  |  |  |  |  |  |  |  |  |  |
| GP | 0.801 | 0.934 | 0.556 | 0.609 | 0.815 | 0.474 | 0.578 | 0.900 | 0.660 | 0.537 |
| XGBoost | 0.805 | 0.967 | 0.583 | 0.604 | 0.881 | 0.509 | 0.568 | 0.869 | 0.642 | 0.526 |

### D.2 Abstractive Text Summarization

Table 9 provides a detailed breakdown of the correlations between various metrics and human preference measurements for abstractive text summarization.

### D.3 Machine Translation

Table 11 describes the detailed breakdown of the correlations between metrics with human annotators in WMT23.

### D.4 Question Answering

Table 10 provides a detailed breakdown of the correlations between various metrics and human ratings for the Question Answering task.

**Table 12: Kendall correlation results of MetaMetrics-Cap is tuned on THumB 1.0 and tested on Flickr8k, and vice versa. Scores are reported for multiple metrics, with the average performance (avg.) across datasets. Bold and underlined values indicate the best and second best performance, respectively.**
| Metric | Flickr8k | THumB 1.0 | avg. |
| --- | --- | --- | --- |
| BLEU-1 | 0.377 | 0.264 | 0.321 |
| BLEU-4 | 0.361 | 0.210 | 0.286 |
| ROUGE | 0.375 | 0.237 | 0.306 |
| METEOR | 0.481 | 0.281 | 0.381 |
| CIDEr | 0.498 | 0.263 | 0.381 |
| SPICE-F | 0.591 | 0.200 | 0.396 |
| SPICE-PR | 0.603 | 0.170 | 0.387 |
| SPICE-RE | 0.579 | 0.221 | 0.400 |
| TIGEr | 0.542 | 0.153 | 0.348 |
| CLIP-S | 0.514 | 0.292 | 0.403 |
| RefCLIP-S | 0.576 | 0.210 | 0.393 |
| MetaMetrics-Cap |  |  |  |
| GP (Full) | 0.664 | 0.329 | 0.496 |
| GP (Top 5) | 0.655 | 0.277 | 0.466 |
| XGBoost (Full) | 0.670 | 0.277 | 0.474 |
| XGBoost (Top 5) | 0.664 | 0.283 | 0.474 |
| XGBoost (Iterative-Best) | 0.662 | 0.285 | 0.474 |
| XGBoost (Iterative-Top5) | 0.659 | 0.276 | 0.468 |
| MetaMetrics-Cap (Cross-Dataset) |  |  |  |
| GP (Full) | 0.638 | 0.298 | 0.468 |
| GP (Top 5) | 0.560 | 0.246 | 0.403 |
| XGBoost (Full) | 0.595 | 0.323 | 0.449 |
| XGBoost (Top 5) | 0.575 | 0.311 | 0.443 |
| XGBoost (Iterative-Best) | 0.635 | 0.321 | 0.478 |
| XGBoost (Iterative-Top 5) | 0.634 | 0.301 | 0.467 |

### D.5 Image Captioning

Table 12 provides a detailed breakdown of the correlations between various metrics and human annotations for Image Captioning.

### D.6 Reward Model Scoring

Table 13 describes the detailed breakdown of accuracy between metrics with human preference.

**Table 13: Accuracy results for Reward-Model-As-A-Metric. Bold and underlined values indicate the best and second best performance, respectively.**
| Metric | Score | Chat | Chat Hard | Safety | Reasoning |
| --- | --- | --- | --- | --- | --- |
| Llama-3.1-Nemotron-70B-Reward | 94.1 | 97.5 | 85.7 | 95.1 | 98.1 |
| Skywork-Reward-Gemma-2-27B | 93.8 | 95.8 | 91.4 | 91.9 | 96.1 |
| TextEval-Llama3.1-70B | 93.5 | 94.1 | 90.1 | 93.2 | 96.4 |
| Skywork-Critic-Llama-3.1-70B | 93.3 | 96.6 | 87.9 | 93.1 | 95.5 |
| URM-LLaMa-3.1-8B | 92.9 | 95.5 | 88.2 | 91.1 | 97.0 |
| SFR-LLaMa-3.1-70B-Judge-r | 92.7 | 96.9 | 84.8 | 91.6 | 97.6 |
| Skywork-Reward-Llama-3.1-8B | 92.5 | 95.8 | 87.3 | 90.8 | 96.2 |
| GRM-Llama3-8B | 91.5 | 95.5 | 86.2 | 90.8 | 93.6 |
| Internlm2-20B-Reward | 90.2 | 98.9 | 76.5 | 89.5 | 95.8 |
| ArmoRM-Llama3-8B-v0.1 | 90.4 | 96.9 | 76.8 | 90.5 | 97.3 |
| URM-LLaMa-3-8B | 89.9 | 96.9 | 78.7 | 88.2 | 95.7 |
| Internlm2-7B-Reward | 87.6 | 99.2 | 69.5 | 87.2 | 94.5 |
| MetaMetrics-RM |  |  |  |  |  |
| GP | 93.4 | 98.3 | 86.4 | 90.8 | 98.2 |
| GP (Top 3) | 93.0 | 95.5 | 87.7 | 91.6 | 97.0 |
| XGBoost | 92.8 | 96.1 | 88.4 | 90.9 | 95.8 |
| XGBoost (Iterative Best) | 92.9 | 95.8 | 88.4 | 91.8 | 95.8 |
| XGBoost (Top 3) | 93.0 | 86.0 | 90.3 | 93.0 | 95.5 |

## Appendix E Weights and Feature Importance

### E.1 Question Answering

There are 8 setups for the question answering task. The resulting weights or feature importance for each setup are reported in Figure 7.

Figure: Figure 7: Weights and Feature Importance for MetaMetrics-QA.
Refer to caption: extracted/5907857/images/h_qa_all_3.png

### E.2 Machine Translation

There are 2 setups for MT, reference-based and reference-free. Both setups’ resulting weights or feature importance are reported in 8

Figure: (a) Weights and Feature Importance for each reference-based metric used for WMT23 task.
Refer to caption: extracted/5907857/images/mt-ref-based.png

### E.3 Image Captioning

We utilize two datasets for Image Captioning, each with two setups: a) cross-dataset and b) in-dataset. The resulting weights and feature importance are presented in Figure 9.

Figure: Figure 9: Weights and Feature Importance for MetaMetrics-Cap.
Refer to caption: extracted/5907857/images/h_cap_full.png

### E.4 Reward Modeling

The weights and feature importance from the resulting optimization are reported in Figure 9.

Figure: Figure 10: Weights and Feature Importance for MetaMetrics-RM.
Refer to caption: extracted/5907857/images/h_rm_full.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. The research problem this paper studied is very interesting and useful. Automatic evaluation metrics can cover a wide range of benchmarks. However, they may not align with human evaluation. Given human evaluation is time-consuming and expensive, and also current model is required to be more generalizable to various tasks, aligning the automatic metrics with human evaluation preference becomes an urgent and important research topic.

2. This paper illustrates the motivation, problem definition, and methods to optimize the proposed MetaMetrics in a clear and easy-to-follow way.

3. The experiments are conducted in a wide range of downstream tasks, from text to multimodal, as well as training a reward model.
summary: This paper proposed a method to align the automatic evaluation metrics with human preference evaluation, called MetaMetrics. Specifically, it's optimized to learn a set of weights that can combine automatic metrics to mimic human preference. Two methods are discussed and experimented for the optimization. The experiments cover both text and multimodal evaluation, demonstrating a satisfactory performance of the proposed method.
weaknesses: 1. This paper assumes that human preference can be approximated by a weighted combination of metric scores. But it seems to be too simple, and might not always hold. For example, human's evaluation of helpfulness might require both the model to perform well in reasoning and knowledge-retrieving, which we assume are separately measured by two benchmarks. Then a weighted summation of these two benchmarks might not be suitable, instead, a multiplication might be more proper.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - The paper addresses a critical problem in the field of text generation: the lack of automatic metrics that accurately reflect human preferences. Current approaches often rely heavily on human annotations to evaluate performance. By proposing a method to calibrate existing metrics to align with human preferences, this work aims to reduce the reliance on labor-intensive human evaluation processes.

- The methodology is simple yet effective. The proposed approach captures the multifaceted nature of human preferences while incurring minimal additional cost, enabling efficient calibration of existing metrics.

- Extensive experiments validate the proposed method. The authors conduct experiments across a variety of tasks, including Text Summarization, Machine Translation, Question Answering, Image Captioning, and Reward Model Scoring. This demonstrates the task-agnostic advantages of the method and its broad applicability.
summary: This paper introduces MetaMetrics, a framework designed to calibrate evaluation metrics to better align with human preferences. By leveraging existing metrics that capture different aspects of human preferences, MetaMetrics employs learning-based methods to optimize these metrics for various generative tasks. To demonstrate the effectiveness of MetaMetrics in aligning with human preferences, the authors compute correlations between the calibrated metrics and quantified human ratings. Experiments are conducted across diverse tasks, including Text Summarization, Machine Translation, Question Answering, Image Captioning, and Reward Model Scoring.
weaknesses: - Empirical and ad hoc nature of MetaMetrics: The calibration process for MetaMetrics relies on empirical and ad hoc weighting of existing metrics for each dataset and human preference. These weights are highly dependent on the specific dataset and task, which raises concerns about their generalizability to a broader range of datasets or tasks.

- Lack of ensembled baselines: While the paper compares MetaMetrics against individual metrics, it does not include ensembled baselines, such as averaged or weighted-average scores of individual metrics. Including these as baselines would provide a fairer comparison and strengthen the evaluation.

- Absence of human studies in experiments: Although MetaMetrics aims to align with human preferences in evaluating model performance, the study does not involve an actual model being evaluated and analyzed through human studies. To comprehensively validate MetaMetrics, experiments involving direct human evaluation of model outputs are necessary.

- Unclear human preference in certain tasks: In some evaluation tasks, the connection between the ratings and human preferences is not adequately explained. For instance, it is unclear how ratings like sys Pearson, seg Pearson, and acc-t in Machine Translation tasks, or NQ, TQ, NarrativeQA, and SemEval in Question Answering tasks, align with human preferences. More detailed explanations of these ratings and their relationship to human preferences are needed, especially for readers unfamiliar with these tasks.

- High computational cost for reward model ensembling: The use of multiple metrics in MetaMetrics can significantly increase evaluation costs, particularly in reward model scoring. For example, in online RLHF training, simultaneously running multiple reward models (e.g., 7 models in Table 4) could make the process computationally infeasible.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Please provide the training details. From my understanding, when training a combined metametric for reward modeling, you need to run inference on all of the training data using each of the base metrics first, and then train your weight vector to optimize for the human preference. How much GPU hours have you spent on the inference, and the training? How many GPUs did you use for the experiments?
2. Could you provide a list of base metrics used for each meta-metric variant in one place? I only found the base metrics for reward modeling  in line 474 and it's unclear what's the base metrics used for other benchmarks. if they are also baselines, it's better to mark them in the results table.
soundness: 2
strengths: 1. This paper proposes a new paradigm to ensemble existing metrics for evaluation of different tasks, learning a weight vector through GP or XGBoost and get the weighted combination of base metrics. This is a novel approach and gets good performance.
2. The paper is well-motivated and clearly written, with nice figures and tables illustrating the concepts.
3. Extensive experiments across summarization, machine translation, QA, reward modeling, which covered a wide areas. The authors also provide analysis of the weights distribution across the base metrics. Providing many insights.
summary: This paper introduces meta-metrics, which optimizes the combination of existing metrics to enhance their alignment with human preference. The paper investigated mainly two method GP and XGBoost to try to learn a best weight vector for a set of base metrics to combine. For different datasets, they select different base metrics and train a new weight vector for combinations. Results show that meta-metrics can outperform existing baseline metrics in summarization, translation, question answering, image captioning and reward modeling scenarios.
weaknesses: 1. The performance gain is relatively small compared to base metrics, especially for machine translation and reward modeling tasks, it's unclear whether these marginal performance gains are worth the training cost.
2. The proposed approach requires training a new combination weight vector once the base metrics have changed, thus limiting the usability of the approach.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. It is very good that the authors have shown their method on multiple datasets, but some more experiments/analysis on the proposed methods would strengthen the paper more. One such thing can be, the authors used Kendall-Tau as the objective calibration function. How were the results when spearman correlation is used as a calibration function?
soundness: 3
strengths: 1. The paper presents innovative mathematical approaches of using BO modeled as Gaussian Process, and XG Boosting on existing metrics to create a new metric.
2. The paper has tested their method extensively on several datasets across multiple domains.
summary: This paper introduces Meta-Metrics, a calibrated metric for evaluating generation tasks like summarization, QA, Machine Translation and Image Captioning,etc. It raises the issue of metrics excelling at one dimension and falling at others while evaluating on multiple dimensions (like coherence, fluency, relevance,etc in summeval), hence proposing a calibrated metric based on these several metrics, aiming to achieve a higher alignment with human preferences in the evaluation scores. The authors propose using methods like Bayesian Optimization, Boosting to effectively choose/filter the various metrics and give a combined metric to have higher correlation with human scores.
weaknesses: 1. Although the metametrics have shown they have higher alignment with human preferences, the optimization methods (BO and Boosting) are to be considered black boxes and offer very less explainability.  Understanding how these methods find the combination of metrics and interactions between different metrics within the model is not possible. Hence explanation on why they correlate more with human scores is not there.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. **Questions related to Introduction Section:**
    1. The first paragraph is a bit confusing. The authors discuss the shortcomings of BLEU and BERTScore in the main text, but Figure 1 presents BLEU and CLIP-S. Additionally, what criteria are used in Figure 1 to determine if a metric is good or not? For the figure on the right hand side, why are BLEU 0.2 and MeMetric 0.4 considered good, but CLIP-S 0.6 is considered bad? What is the threshold?
    2. The authors state, "models that optimize for human preference ... have demonstrated superior performance." What metrics have these models used to demonstrate human alignment? The authors avoid discussing this in their paper, instead comparing their metric with conventional metrics not originally designed for human preference alignment. For example, Ouyang et al. [1] used the Perspective AI score to measure toxicity.
    3. In line 42, the authors mention "evaluation metrics that accurately reflect human subjective judgements." Since human judgements are **subjective**, is it possible that a metric can **accurately** measure these varied judgements across different individuals?
    4. In line 53, the statement relating the necessity of human aligned metrics to the complexity of tasks seems implausible. Regardless of task complexity, we always seek metrics that align with human evaluations.
2. **Universality of MetaMetric:** For conventional metrics (e.g., BLEU, ROUGE, METEOR), they are used across various natural language generation tasks. Is it feasible to have a single MetaMetric applicable to all natural language generation tasks?
3. **Choice of Training Methods:** Why did the authors choose Bayesian Optimization and Boosting to train MetaMetrics? Is it possible to use linear regression models with regularizations?
4. **Efficiency of MetaMetric:**. According to Figure 4, MetaMetric requires about 8 metrics for computation. Can the authors explain why computing approximately 8 metrics is still considered efficient?
5. **Interpretability:** Can the authors explain why some metrics align more closely with human evaluations while others do not? I'm curious whether there is any intuition behind this.
6. **Compute Efficiency for RM:** In lines 433 and 472, Table 4 does not show compute efficiency. The authors should include the model size for MetaMetrics-RM.

[1] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.

**Minor Issues:**
1. In line 215, there is a duplicate "is"
2. Presentation suggestion: Section 2 could be merged with Section 7.
soundness: 2
strengths: 1. The experiments are comprehensive, including various types of natural language generation tasks.
2. The topic of developing a human-aligned evaluation metric for NLP generation tasks is a long-discussed but unsolved challenge. This paper explores a possibility to push forward in this direction.
3. The proposed metric is easily extendable to other tasks where preliminary metrics have already existed. This extensibility makes the metric potentially applicable to a broader range of applications.
summary: This paper introduces a novel evaluation metric, MetaMetric, designed to better align with human preferences. The proposed metric is a weighted combination of existing metrics, where the weights are learned in a supervised manner. The authors demonstrate the effectiveness of MetaMetric through experiments on various natural language generation tasks, e.g., summarization, translation, question answering and image captioning. Additionally, they show that this metric can be useful for reward model scoring. They also provide some analysis about interpretability and efficiency of their metric.
weaknesses: I have several concerns regarding the experiments, particularly the scope of the problem and the evaluation settings, and how they support the authors' claims. Additionally, the presentation of the paper requires refinement.
1. **Limited Contribution:** The paper's contribution appears limited. It essentially addresses a traditional supervised learning problem with input of small dimensions and output with single normalized value. The authors train a regression model on supervised-learning datasets, using conventional metrics as inputs and human preference values as outputs. The experimental results involve testing this regression model on the test set of these datasets and comparing different metrics using correlation of the rankings of metrics with ground truth human preference values. It is unsurprising that MetaMetric outperforms other metrics in the in-domain generalization setting, which is the primary focus of this paper. The authors should also clearly present their experimental settings, such as how they create the training, validation and testing sets. For example, the setting for Question Answering tasks is not clear. It would be interesting to add more OOD and real scenarios to test this metric. See following comments for detailed feedback.
2. **Lack of Out-of-domain (OOD) Generalization Evaluation:** The authors should place more emphasis on the OOD setting, as human preference data is not always available. I found only one one OOD setting presented in Figure 3. Based on in-domain performance, the GP model works the best. However, the GP model performs similarly to CLIP-S and METEOR in the OOD setting. (The statement about the superiority of MetaMetric-Cap over any other individual metric, in lines 426 and 427, is misleading.) This suggests that the use cases for the trained metric are limited, and its performance may be worse than using a single conventional metric. It would al os be interesting to evaluate metaMetric trained on one task but tested on another, which is a more challenging OOD setting than the one in Figure 3.
3. **Lack of Evaluation in Real Settings:** The evaluations are limited by the training datasets with human-annotated preference values. It is unclear whether this metric is useful for real applications, as suggested by the authors in lines 41 and 42. For example, the authors could follow a similar evaluation procedure to DPO and PPO papers, using the proposed metric to evaluate various LLMs - before and after RLHF-tuned - and compare their win rates based on MetaMetric.
4. **Reference-based and Reference-free Settings:** In line 83, the authors claim to design the metric for two settings: reference-based and reference-free. If I understand correctly, this depends on the choice of metrics rather than the model design. If the chosen metrics require references, then MetaMetric can only be reference-based. In the experiments, the authors only distinguish these two settings for Machine Translation, not for other tasks. It would be better that the authors add both settings for all the tasks.
5. **Presentation Issues:** The paper's presentation is suboptimal. Major issues include:
    1. The caption of a table should be placed above it.
    2. Figure 2 is not cited anywhere in the paper.
    3. The authors cite a table in the Appendix as crucial support for their claims (see line 357 for Table 9). The authors should put the information in the main text.
    4. Other minor issues are listed in the Questions section.

Meta-review: This paper proposed a metametrics that is more aligned with human preference on evaluating the generation tasks. The idea is quite simple, which can be summarized based on the rebuttal comment: 
1. Identify the downstream task, such as image captioning.
2. Train MetaMetrics using datasets that include human ratings. Evaluate the performance on an in-domain held-out set, selecting the MetaMetrics based on the performance of this same in-domain held-out set (not the test set). The test set is not used at this stage.
3. Evaluate the selected MetaMetrics on an unseen test dataset split that includes human ratings. This is a standard machine learning evaluation. We use Kendall or Pearson correlation to assess language model behavior relative to human preferences. The test dataset split can include both in-domain unseen test data and out-of-domain unseen test data (e.g., cross-lingual and cross-dataset scenarios). For the in-domain setting, use 70% of the dataset as the test split. For the out-of-domain setting, we use the whole dataset as test set.

The proposed approach demonstrates the effectiveness and usefulness in evlauation and shows good human alignment. 

Strength:
1. The paper studies an important task (evaluation for generation task)
2. The approach is simple yet effective
3. The paper is easy to follow

Weakness:
1. The author might need to follow the reviewer suggestions to provide more ablation study such as evaluations of the LLMs before and after alignment.

Overall, I think this is a good paper and recommend accept.

Decision: Accept (Poster)