MetaMetrics: Calibrating Metrics for Generation Tasks Using Human Preferences
Authors: Genta Indra Winata
David Anugraha
Lucky Susanto
Garry Kuwanto
Derry Tanti Wijaya
Venue: ICLR 2025 Poster (2025)
Keywords: metrics
human preferences
calibrating
generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: MetaMetrics is a calibrated, language- and modality-agnostic meta-metric designed to evaluate generation tasks based on human preferences.

Abstract: Understanding the quality of a performance evaluation metric is crucial for ensuring that model outputs align with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce MetaMetrics, a calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. MetaMetrics optimizes the combination of existing metrics to enhance their alignment with human preferences. Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi-domain scenarios. MetaMetrics aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes MetaMetrics a powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts.

Published as a conference paper at ICLR 2025
METAMETRICS: CALIBRATING METRICS FOR GENERATION TASKS USING HUMAN PREFERENCES
Genta Indra Winata li+ David Anugraha Lucky Susanto Garry Kuwanto Derry Tanti Wijaya *Capital One UUiiversity of Toronto *Monash University Indonesia
*Boston University
genta. wwwnana@capiitaaa... edu,
com, anugraha@cs toronto. edu, lucky. wusantu w fgkuwanto, wi jaya /aba. edu
ABSTRACT
Understanding the quality of performance evaluation metric is crucial for en- suring that model outputs align with human preferences. However, remains unclear how well each metric captures the diverse aspects of these preferences, metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metries to specific aspects of human preference, catering to the unique characteristics of each aspect. We introduce METAMETRIE
calibrated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. METAMETRICS optimizes the combination of existing metries to enhance their alignment with human pref- erences, Our metric demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multi- lingual and multi-domaii scenarios. METAMETRICS aligns closely with human preferences and is highly extendable and easily integrable into any application. This makes METAMETRICE eration tasks, ensuring that metrics are more representative of human judgment across diverse contexts.
powerful tool for improving the evaluation of gen-
INTRODUCTION
Evaluating machine-generatte sentences has long been one of the main challenges in natural lan- guage processing (NLP). CCalison-Burch et al. (2006) provide multiple examples where high BLEU score does not necessarily indicate true sentence similarity. and conversely, highly similar sentence pairs can receive low BLEU scores. BERTScore (Zhang et al., 2019) is designed to cap- ture semantic similarities, also falls short in capturing the all the nuances have comprehensive evaluation. Figure illustrates two such cases where traditional metrics either overestimate or under- estimate the alignment of generated sentences. Such cases highlight a critical limitation for robust evaluation, as it may fail to account for the multifaceted aspects of language quality.
In light of recent advancements such as Reinforcement Learning with Human Feedback (RLHF) (Ouyang et al. 2022). ensuring that generated outputs align with human preferences has become increasingly critical. Models that optimize for human preference, rather than solely rely- ing on traditional metrics, have demonstrated superior performance in producing content that aligns with human preference (Rafailov al., 2024| Winata et al. 2024b). This shift highlights the need for evaluation metrics that accurately reflect human subjective judgments across multiple dimensions. Such metrics are essential for guiding models to generate more numan--ligned outputs by compar- ing their results against human judgments. Metrics that exhibit high correlation with human ratings are considered more reliable and effective for evaluating model performance.
This is particularly important for NLP tasks, where the subtleties of human language and context significantly influence quality assessments. For example, in machine translation (Freitag et al.) [003] Juraska et al. 2023). the accuracy and fluency of the translated text critical factors considered by humans. Similarly, text summarization (Fabbri et al. 2021), l), the coherence, relevance, and con-
"The work was done outside Capital One. Equul contribution.

---

Published as a conference paper at ICLR 2025
BLEU 0.1 X CLIP-S 8.8
BLBU 0.2
CLIP-S 0.6 X
METAMETRICS
METAMETRICS 0.4
References
References
? little boy riding skis next to an adult riding skis
constroction sorne with
cranes against the blue aky
Prediction
Prediction
Human Rating 5.0
Human Rating 2.0
X young child standing slia
X city street filled with lots of traffie
In the snow.
Figure I: Examples of image captioning on THumB 1.0 dataset comparing METAMETRICS against BLEU and CLIP-S scores. METAMETRICS scores predicted captions are closer to human ratings compared to BLEU and CLIP-S scores in the left and right images, respectively.
ciseness of the summary are key aspects that need evaluation. Beyond NLP tasks, vision-language (VL) tasks like image captioning also benefit from reliable evaluation metrics (Hessel et al., [02TT) In these tasks, the generated captions must accurately describe the image content while maintaining grammatical correctness and contextual relevance. The complexity of these tasks highlights the need for metrics that can effectively capture the quality of the generated content in manner consistent with human judgment. Each metric may be good for specific human preference aspect, and it is necessary to identify which metric suitable for each aspect to ensure comprehensive evaluation.
Many Metries Are Not Tuned on Human Preferences. Standard metrics that are com- monly used to evaluate downstream language tasks, such as BLEU (Papineni et al. 2002) and BERTScore (Zhang et al. 2019), often fail to align with human preferences. In many generation tasks, it is crucial to assess the quality of outputs based on human judgment, these evaluations comprise multiple dimensions of quality. Therefore, we need a comprehensive framework that al- lows to identify, refine, and utilize effective metries for evaluating generated content in a way that aligns with human judges.
Motivated by this insight, we introduce METAMETRICS, meta-metric designed better align with human preferences by calibrating multiple metrics using human assessment scores. Our method is both efficient and fast, eliminating the need for extensive training. Our contributions are three-fold:
a supervisedly calibrated, flexible, and modality- and language-agnostic metric that aligns closely with human preferences on five different tasks. Our metric is designed to operate in two distinct settings: reference-based and reference-free, providing versatility across a wide range of tasks and modalities.
We introduce
We demonstrate that METAMETRICS is adaptable to a wide range of requirements, includ- ing performance optimization and metric efficiency through a comprehensive benchmark- ing of diverse metrics across range of language and vision tasks $4 We also show that METAMETRICS can be used effectively as a reward model This adaptability ensures that METRI be calibrated to suit various human preference aspects.
We demonstrate that METAMETRICS outperforms existing metrics by leveraging comple- mentary metrics enhance overall performance. We conduct detailed analysis un- derstand how METAMETRICS attributes weight or attention to specific metrics, thereby measuring the contribution of each component. We will release the code and models to facilitate reproducibility and empower researchers and practitioners to utilize and extend our metric for various applications!
2 AREN'T EXISTING METRICS GOOD ENOUGH?
The challenge of developing metrics tailored to specific tasks is not new issue. For long time, researchers have struggled to identify appropriate metrics that align with human preferences. this
'META ranks 5th of 155 the overall leaderboard, utilizing the fewest parameters, and secures 3rd place detailed leaderhoard at RewarddBench It reflccts data as November 16th, 2024.
We release META METRICS open-source library httpt: : //gi thu.. com/meta--e metaammtrrs cs/ metamet ricis
2

---

Published as a conference paper at ICLR 2025
section, we outline the rationale for the need for new evaluation metric that is both suitable for our tasks and aligned with human preferences.
Commonly Used Metrics are Not Robust and Unrepresentative Metrics such as Perplexity (PPL) (Jelinek et al. 1977 Bengio et al., 2000) and BLEU (Papineni et al. 2002) are widely used to measure the quality of generated text in various generation tasks. However, they are not al- ways the most suitable metrics, particularly when accounting for variations in writing styles and minor character differences due to diacritics (Freitag et al. 2022). Critics have noted that PPL is an English-centric metric that performs well for English but is less effective for other languages, such as Japanese (Kuribayashi et al., (00T). THumB (Kasai et al.) 2022) further revealed that widely used metries fail to adequately capture the semantic quality and diversity of high-quality human annotations, leading to misleading evaluations of model performance, particularly for captions.
Limited Capability of Individual Metric. Single metrics like BERTScore (Zhang et al., 2019), ROUGE (Lin, 2004). and METEOR (Banerjee & Lavie, 2005) are beneficial and capable of measuring the quality of generated content. However, they have significant limitations. For in- stance, in summarization tasks, BERTScore (Recall) excels in assessing consistency but falls short evaluating coherence. Conversely, BERTScore (f1) performs well in measuring relevance consistency (Fabbri et 202T). This variability means that single metric may perform well in one aspect but poorly in another, making challenging to select the appropriate metric without extensive benchmarking. This ambiguity complicates the process of choosing and utilizing the most suitable metric for a specific use case.
not
One Metric with Too Many Variants and Implementations. Metrics like BERTScore allow the use of various BERT models, but the sheer number of options can make difficult to identify the best without systematic approach. This can lead to exhaustive search process. BERT models trained on English often underperform on non-English languages, and vice versa. Non-English languages in Latin script may also yield poorer results compared those in their native scripts. Compounding these issues, many metrics are heavily parameterized, yet their settings are frequently undocumented, resulting in score variations across implementations (Post) 2018 Grusky 2023).
To tackle these challenges, we propose a new paradigm for creating customizable metrics that are robust across various tasks and closely aligned with human preferences. Our approach will enable systematic evaluation and automatic selection of the most suitable combination of metrics that is aligned with human judgments on target tasks, ensuring optimal performance and relevance.
3 METAMETRICS
In this section, we outline the notations and definitions used in our proposed method, and detail the conditions required for optimizing the method based on human preferences. Additionally, we discuss the key factors influencing the optimization process.
3.1 PRELIMINARIES
Definition. We define 0: a metric function that maps sample input to a score D.. where i E (1....,N) denotes different metrics. Each 6; depends on the type of task it is applied to. For reference-batt based metric, the data is evaluated in the context of x (Jhypp Tyat). where Thyp and wee correspond to the hypothesis text and the reference text, respectively. For reference-free free metric, only Thyp will be used. In VL tasks, the input is extended include an image, which would correspond to (Zeets Timage) where Ziext and Timage correspond to the caption and image, respectively.
Given a set of N evaluation metrics (ūI...., ØN J. we define & to compute a scalar meta-metric score of ŪMM- The idea of utilizing multiple metrics is to combine scores from multiple metrics regardless of the metric types. Overall, we define OMM as a meta-metric function where Ймм is computed follows:
0. (z)), ÍMM (M)(()) (5n, (N).
(I)
The METAMETRICS fM is used to calculate the objective value P(ÙMM: 2). where is measures alignment with z E R. the human preference score. In practice, & can represent various scores annotated by human judges, such coherence and fluency.
3

---

Published as a conference paper at ICLR 2025
3.2 HUMAN PREFERENCE OPTIMIZATION
Notations and Objective. Recall that we aim to calibrate OMM. that maximizes an objective calibration function P(INM; 2), where z denotes human assessment scores-encompassing any score annotated by human evaluators. is a function that measures alignment between and METAMETRICS scores, ÿMM- METAMETRICS is designed to combine scores of multiple met- 8N learning the weights wy assign to each y; (x(x:) to maximize
rics 01 (2r), p(ÚMM. 2). Each metric has its score Ys ranges within a specific minimum and maximum value. Some metrics, particularly neural-based ones, can fall outside this defined range. To standardize these metrics, we need to normalize them to a common scale that ranges from O to 1. In this scale, O represents poor translation performance, while represents perfect translation performance. We pre-process Us before combining these scores during METAMETRICS training, as detailed in Ap- pendix [.2] The advantage of METAMETRICS lies in its flexibility and adaptability to different tasks and domains. Certain metrics on certain tasks may exhibit strong correlations with human judgments, and by constructing a composite metric that learns to integrate these reliable metrics with human judgments, can enhance the overall correlation with human evaluations.
3.3 OPTIMIZATION METHODS
In this work, we focus on two optimization methodologies to train METAMETRICS: Bayesian Op- timization (BO) and Boosting- BO offers the advantage of interpretability, allowing us to clearly identify which metrics contribute most significantly to the final outcome. Conversely, Boosting ex- cels in enhancing alignment and accounting for the compositionality of different metrics, even when dealing with more complex functions. Although we can measure the contribution of each metric, the clarity and distinctness of these contributions are more pronounced with BO compared to Boosting.
3.3. BAYESIAN OPTIMIZATION (BO)
BO constructs posterior distribution of functions, typically modeled as a Gaussian Process (GP), to represent the function being optimized. As observations accumulate, the posterior distribution becomes more precise, allowing the algorithm to identify which regions of the parameter space to explore and which to ignore. BO is particularly effective for predicting unknown functions based on observed data, making it robust method for optimizing complex and -ostly-to-evaluate functions such as the correlation between evaluation metric scores and human assessment scores.
Formally, we want solve for the optimum weights w' arg maxwe W (()aa(()):.. 2), where P( (w, 2) is the aligmment between METAMETRICS score ÿMM (w) with the human preference score In this paper, we apply linear weighting for our metrics. We define (w) = ΣiEN Wiği. As part of our ablation study, we explore alternative weighting functions, such multiplicative scoring, which calculates the score by applying weighted sum the prod- uct of two metric scores. However, we find that linear weighting outperforms the multiplicative approach. Further details can be found in the Appendix C and F.6
[w], W2, ... wN],
We assume a GP prior: that the alignment between the weighted combination of metric scores and the human preference score follows a GP P(MMM(W) (w). ~ GP(((G)),)))))) ka k(w, w')). where μ(w) is is
the prior mean of the alignment and & k(w, w') is the kernel function that measures the similarity be- tween two sets of weights w and w'. After observing a set of weight vectors W = =[W1, W2,..., Wi] and their corresponding alignments p = [MI, [MI, PM,.,..,., Pkk]. diet the alignment for a new weight vector w. To guide the search for the optimal weights w, BO maximizes a function and selects the next weight vector W to evaluate using the function. BO's function balances exploration and exploitation using the posterior mean and variance, iterating until convergence budget exhaustion. While challenging, maximizing the function is computationally efficient with standard tools, making BO suitable for resource- resource-intensive intensive sampling scenarios. GP em- ploys a multivariate Gaussian distribution, offering sparse representation of metrics that enhances interpretability by identifying key contributors to the objective.
the GP posterior can be updated to pre-
3.3.2 BOOSTING METHOD
The boosting method we investigate is Extreme Gradient Boosting (XGBoost) (Chen & Guestrin 2016). Gradient-boosted trees have long been recognized as a robust technique, supported by exten- sive literature demonstrating their effectiveness (Friedman 2001). To enhance the efficiency of our
4

---

Published as a conference paper at ICLR 2025
metric, we implement iterative pruning to eliminate less important metrics from the input, resulting in a more compact and faster metric.
-tesed Pruning. When training XGBoost regressor model, can initially use large set of metrics. However, this approach is less efficient than the BO model at inference time. To address the scalability and efficiency issues, we propose an iterative pruning method (Algorithm M We conduct k iterations of XGBoost training, starting from the full set of metrics and removing the metric with the least feature importance at each iteration. We return the best XGBoost model among k iterations based on the cross-validation value of p(iMM, 2) measured during training.
4 EXPERIMENTS SETUP
In this work, we explore two optimization methodologies: BQ and Boosting. BO provides inter- pretability, highlighting metries that significantly impact the final outcome. In contrast, Boosting enhances alignment and addresses the compositionality of metrics, even for complex functions. While we can measure each metric's contribution through feature importance in Boosting, BO is more interpretable than Boosting. We also compare calibrating with all the metrics and calibrating only using top-5 correlated metrics from the tuning set. For each evaluation task in abstractive text summarization, question answering. and image captioning, we use 30%-70% train-test split as no predefined split was available the datasets used for these tasks. Alternatively, we follow existing standardized benchmarks for machine translation and reward model scoring.
A Abstractive Text Summarization. For this task, text summarization evaluation. SummEval, based on CNN/DailyMail (Hermann et al. 2015). is rated by human annotators on coherence, consistency, fluency. and relevance using to 5 Likert scale, and we use the average human annotation score. For additional benchmarking, we also evalu- ate on BenchmarkLLM (BLLM) (Zhang et al. (024) dataset, which includes both CNyi and XSUM (Narayan et al. 2018) articles, with summaries assessed for faithfulness (binary scale), coherence, and consistency Appendix Table We use the Kendall correlation function as our objective calibration function p. We refer to our metric as METAMETRICS-SUM (Table
the SummEval (Fabbri et al. 2021 for
to Likert scale). For this dataset, the results are provided in the
Machine Translation. For this task, we train both reference-free and reference-based versions of the metric. We train our metric using the 3-year training data from WMT shared tasks datasets from 2020 to 2022 that are annotated using MQM annotation scores. We evaluate on MQM dataset from WMT23 and WMT24 shared task (Freitag et al.) a [023] 2024). We use kendall correlation function as our objective calibration function p. We refer to our metric as METAMETRICS-MT (Table 2)
Question Answering. For this task, we evaluate METAMETRICS performance across multiple QA
subtasks, including Open Domain QA (Open-QA) (Rajpurkar 2016; Berant et al. 2013 Petroni
ct al. 2019), Reading Comprehension QA (RCQA) (Bajaj et al. 2016 Fisch et al. 2019 Yang
et al. 2018). and Reasoning QA. Open-QA retrieves answers from large, unstructured datasets like
Wikipedia. RCQA requires the model to read a passage to derive answers directly from it, while Reasoning QA emphasizes logical inference, where answers cannot be directly extracted from the
passage. Due to the nature of our paper, we require QA datasets with human evaluations. The data we use are collected from two sources: Evaluating Question Answering Evaluation (EQAE)
Chen et al., 2019) and EVOUNA (Wang et al.] 2024a) and include Open- QA datasets: Natural Questions (NQ) (Kwiatkowski et al., 2019) and TriviaQA (TQ) (Joshi et al.] 2017), RCQA dataset: NarrativeQA (Kočiský et al. 2018), and Reasoning QA dataset: SemEEva 2018 Task 11 (SemEval)
(Ostermann et al. 2018). We refer to our metric as METAMETRICS-QA. (Table 3
Image Captioning. For this task, we use Flickr&k-Expert (Hodosh et al. 2013) and THumB 1.0 (Kasai et al. 2022) human evaluation datasets. FlickrSk-Expert consists 5,882 image-caption pairs, each having 3 human annotators with a scale of 1-4 where is worst and 4 is best. The THumB 1.0 dataset is a rubric- based human evaluation framework applied to a subset of the MSCOCO dataset (Lin et al. (014)). The final score is of a scale of 1-5 where is worst and 5 is best. We refer to our metric as METAMETRICS, CAA (Figure 2).
Reward Model Scoring, We use METAMETRICS as reward model to score text generation from L.M.M We use RewardBench (Lambert al. 2024) the test benchmark. We use cleaned Sky-
5

---

Published as a conference paper at ICLR 2025
Table I: Kendall and spearman correlation results with human ratings for summarization task on SummEval. For comprehensive results, please refer to Appendix Table Coh., Cons., Fluency, and Rel. corresponds to coherence, consistency, and fluency, relevance, respectively. Bold and underlined values indicate the best and second best performance, respectively.
Metric
Kendall Fluency
Spearman Fluency
Coh.
Cons.
Rel.
Avg.
Coh.
Cons.
Rel.
Avg.
BLEU chrF
0.110
0.126
0.113
0.170
0.130
0.157
0.160
0.145
0.239
0.175
0.143
0.094
0.071
0.198
0.127
0.205
0.119
0.091
0.278
0.173
METEOR
0.077
0.102
0.072
0.162
0.103
0.108
0.130
0.093
0.229
0.140
ROUGE-WEI WEl
0.115
0.088
0.081
0.169
0.113
0.164
0.112
0.105
0.237
0.115
BLEURT (Max)
0.185
0.070
0.114
0.189
0.140
0.262
0.088
0.062
0.194
0.152
BERT Score ((1)
0.105
0.100
0.120
0.181
0.127
0.150
0.128
0.155
0.256
0.172
METRICS
BARTScore (Mean)
0.086
0.074
0.040
0.143
0,086
0.123
0.094
0.052
0.202
0.118
UniEval G-Eval (GPT4)
0.413
0.353
0.359
0.324
0.362
0.577
0.439
0.458
0.446
0.480
0.429
0.413
0.409
0.437
0.422
0.565
0.510
0.470
0.581
0.531
ENSEMBLE BASELINES
Uniform
0.141
0.133
0.117
0.213
0.151
0.201
0.170
0.150
0.298
0.205
Weighted Avg
0.150
0.141
0.122
0.220
0.159
0.215
0,179
0.158
0.309
0.215
METAMETRICS
0.140
0.179
0.354
0.236
XGBoost
0.192
0.186
0.186
0.276
0.210
0.274
0.236
0.239
0.386
0.284
METRICS
GP
0.454
0.419
0.409
0..49
0.433
0.609
0.519
0.470
0.601
0.550
(Top
0.461
0.428
0.437
XGBoost
0.476
0.367
0.404
0.447
0.424
0.642
0.460
0.512
0.600
0.553
XGBoost (Top 2)
0.476
0.436
0.430
0.445
0.447
0.636
0.508
0.511
0,594
0.562
Table 2: Kendall correlation results with human ratings on WMT23 (MQM). Thhe results are col- lected from Freitag et al. (2023). More detailed results for all metrics and their variants can be found in Table T2 in the Appendix. Bold and underlined values indicate the best and second best per- formance, respectively. Our META METRICS-MT correlates with human judgments better than or comparably to the best WMT evaluation metrics across different languages and evaluation settings.
Metric
overall sys/eg ayg-
en-de seg
be-en seg pearson
zh-en seg
sys
seg
sys
seg
Sys
seg
REFERENCE METRIC
chrFl
0.694
0.866
0.232
0.519
0.776
0.221
0.460
0.809
0.063
0.485
BLEU
0.696
0.472
BERTScor!
0.742
0.891
0.325
0.528
0.895
0.335
0.515
0.810
0.236
0.499
Yisi-1
0.754
0.542
0.529
0.823
0.504
MctricX-2-XXXL
0.808
0.977
0.585
0.603
0.910
0.548
0.577
0.873
0.625
0,531
CCOMMET-Rasemblel
0.825
0.927
0.650
0.543
COMET
0.779
0.396
ENSEMBLE BASELINES
Uniffom
0.824
0,989
0.688
0.610
0.954
0.546
0.588
0.935
0.641
0.543
Weighted Ang
0.827
METAMETRICS-ME
GP
0.819
0.970
0.638
0.610
0.947
0.546
0.590
0.900
0.646
0.539
XGBoost
0.546
REFERENCE- FREE METRIC
GEMBA- MOM
0.802
0.993
0.502
0.572
0.939
0.401
0.564
0
0.449
0.522
XCOMET-QE Ensemble!
0,808
0.974
0.679
0.588
0.909
0.198
0.554
0.892
0.617
0.533
Metric 23 QE QE-XXL
0.797
0.934
0.547
0.607
0.813
0.459
0.575
0.877
0.652
0.531
CometKiwi QE-XL
0.786
0.976
0.447
0.571
0.900
0.384
0.533
0.974
0.430
0.522
ENSEMBLE BASELINES
Uniform
0.801
0.935
0.552
0.600
0.816
0.484
0.566
0.948
0.637
0.540
Weighted Avg
0,802
0.934
0.555
0.603
0.811
0.487
0.568
0.943
0.647
0.540
METAMETRICS-MT-QQ
0.801
0.934
0.556
0.609
0.815
0.474
0.578
0.900
0.660
0.537
XGBoost
0.805
0.967
0.583
0.604
0
0.5009
0.568
0.869
0.642
0.526
work Reward Data Collection for training our METAMETRICS, including HelpSteer2 (Wang et al. 2024ch, OffsetBias (Park al. (024). WildGuard (Han et al. 2024), and Magpie (Xu et al., 2024). In addition, we add Preference Test Date increase the size of the human preference dataset. We
*Cleaned Skywork Reward Data Collection can be accessed https::/huugiingfaccc.
datasets/nater sambert/skyookk-peeeere -00. leaned
PPefeernnee Test Data from A12 can be accessed at httpps:/huugg: http:///wwgggaaac.caaaaaaaaaaaaa
allena/ppeferreee

---

Published as a conference paper at ICLR 2025
refer to metric METAMETRICS- RM and report the accuracy of predicting that the chosen sample is ranked higher than the rejected sample (Table
RESULTS
In this section, we present the results of our METAMETRICS across five diverse tasks, covering both NLP and VL downstream applications.
Table 3: Kendall correlation results of METAMETRICS-QA. More detailed results for all metrics and their variants can be found in Table I in the Appendix. Bold and underlined values indicate the best and second best performance, respectively.
Setup
EVOUNA
EQAE
Combined
NQ
TQ
NarrativeQA
SemEval
ARMORM METRICS
ultrafeedback honesty
0.202
0.281
0.297
0.335
0.296
helpfuleesl
0.204
0.293
0.332
0.427
0.302
helpsteer-correctesss
0.206
0.300
0.336
0.403
0.305
ungilla-judge-mm
0.243
0.301
0.253
0.337
0.332
N-GRAM- BASED METRICS
BLEUI
0,461
0.353
0.362
0.260
0.424
ROUGE-L METEOR
0.495
0.399
0.584
0.518
0.454
0,517
0.435
0.570
0.461
0.494
ENSEMBLE BASELINES
Uniform
0.425
0.405
0.564
0.487
0.449
Weighted Avg.
0.497
0.423
0.582
0.467
0.484
METAMETRICS-QU
(Linear)
0.518
0.448
0.586
0.464
0.512
XGBoost
0.543
0.471
0.601
0.486
0,536
XGBoost (Iterative Top 5)
0.540
0.488
0.626
0.503
0.528
Abstractive Text Summarization. For this task, we refer to our metric as METAMETRICS- Table shows the results of abstractive text summarization task. Overall, our proposed model outperforms all other baselines, including all ensemble models and overall best automatic metric (A 0.023). The METAMETRICS- SUM with GP also performs considerably well, also outperforms all other basclines. Additionally, we present the results of BenchmarkLLM in Appendix Table [0
Machine Translation. In this task, we demonstrate that METAMETRICS-MT in the reference- based setting outperforms baseline models, including ensemble methods (Table ( Notably, the top competitor, CCOMETTEEnsemlee achieves impressive results on the WMT23 evaluation leaderboard, particularly excelling the two-segment Pearson correlation metric for en-de and zh-en. However, METAMETRICS consistently surpasses CCOMET-Ensemble across all other tasks, leading to higher average correlation overall, achieving the new state-of-the-ar perfor- mance WMT23 evaluation. In the feference-free setting, METAMETRICS-MTT comparably to XCOMET-QE CCOMET-QQE-Ensemble. While SCOMET-Ensemin shows slightly better average correlation, our proposed model demonstrates superior performance on the acc-t metric. This is particularly significant, as the acc-t metric is finely tuned to align with the Kendall correlation objective function, indicating stronger correlation with human evaluations.
performs
Question Answering. T the results of METAMET QA,, which consistently out- performs existing metrics across all evaluation levels. While ROUGE performs slightly better than our method the SemEval dataset, our proposed model demonstrates a significant advantage over others in different datasets, achieving improvements of up to 0.044 points. Notably, our iterative- based approach enhances the performance of the vanilla META METRICS-OR) when using XGB, all while utilizing fewer metrics. Also, for the GP evaluation, choosing either all metrics or just the top five results in similar combined scores, showing that our metric selection strategy is robust.
Image Captioning. Figure 2 ing. METAMETRICS-CAP consistently performs better compared to any individual metric,
presents the results of METAMETRICS- CAP and cross-dataset tun-

---

Published as a conference paper at ICLR 2025
Flickräk
Badie Rot
Mebalatcc
0.4
Cress Iitaset
0.9
THumb 1.0
0.3
0.1
0.0
Bmd
NETEON
und
Gan Dka
Ben Bant
tue lan
Oen
Oen FAbd
Figure 2: Kendall correlation results of METAMETRICS-CAN Cross-Dataset means that METAMETRICS is tuned on THumB 1.0 and tested on Flickrök, and vice versa. Our METAMETRICS--CE outperforms other metrics across different datasets and shows robustness even when trained and tested on different datasets (Cross-Datasee setting). More detailed results for all metrics and their variants can be found in Table 13 in the Appendix.
when the tuning is performed on a dataset from a different distribution. Although tuning on the same distribution yields the best results, we observe that METAMETRICS-CAN still outperforms any individual metric, demonstrating its robustness across varying data distributions.
Reward Model Scoring. Table 4 highlights the performance of METAMETTICS-RRM which achieves state-of-the-art results for GP and competes effectively with the top seven metrics, in- cluding the advanced 70B Nemotron Reward model. These results demonstrate that our proposed metric is both efficient and effective, showcasing its robustness across diverse categories of tasks, datasets, and evaluation settings. Overall, METAMETRICS RM represents significant advance- ment in reward metric evaluation, offering high performance and broad applicability for researchers and practitioners alike.
Table 4: Accuracy results in percentages for Reward-Modee-As-AAMMee More detailed results for all metrics and their variants can be found in Table 14 in the Appendix. Bold and underlined values indicate the best and second best performance, respectively. Our METAMETRICS- -RM achieves the best or comparable results in 2 out of the categories in RewardBench.
Metric
Score
Chat
Chat Hard
Safety
Reasoning
Llama-3. 1.nma-3.1-Nemmie J..---eeooron----- 70B-770BBa---- Reward
94.1
97.5
85.7
95.1
98.1
Skywork- Reward- Reward-Gemma--2-
93.8
95.8
91.4
91.9
96.1
TextEval-Llamai) 1-70BB
93.5
94.1
90.1
93.2
96.4
Skywark-Critie-llama 3.1-
93.3
96.6
87.9
93.1
95.5
1FR-LLLaMaa331--0B-nnng-g
92.7
96.9
84.8
91.6
97.6
URM-LLaMa-3311-3
92.9
95.5
88.2
91.1
97.0
Skywort-Rewardt-laam-....
92.5
95.8
87.3
90.8
96.2
GRM-LJama3-88
91.5
95.5
86.2
90.8
93.6
ArmoRM-Llana3-8BBv0.
90.4
96.9
76.8
90.5
97.3
URM-LLaMM--388
89.9
96.9
78.7
88.2
95.7
ENSEMBLE BASELINES
Uniform
92.5
98.3
83.1
90.8
97.6
Weighted Avg.
92.5
98.3
83.1
90.7
97.6
METAMETRICS-RM
GP
93.5
98.9
86.2
90.7
98.2
XGBoost (Iterative Best)
92.9
95.8
89.7
92.2
94.0
ANALYSIS AND DISCUSSION
In this section, we analyze the advantages of the methods in of interpretability, efficiency, and robustness. We discuss how METAMETRICS enhances the capability of model evaluation.
8

---

Published as a conference paper at ICLR 2025
6.1 INTERPRETABILITY
[DM] bull
10H1-008
[hul
Full Correlation Healmap Every Netric used in Question Answering
Baman sccrs
350
600
000
complnn
021
330
529 8:0
010
1:
500
8.08
031
betione
Moul
6.09
21
Figure 3: GP weights (Left) and XGBoost Features Importance (Center) with Intra-Metric Corre- lation (Right) for METAMETRICS-OL
Interpretability is a key aspect in METAMETRICS, as the optimization process inherently reveals the impact of each metric the final score. The chosen methods, such as feature importance in Boosting and weights analysis Bayesian Optimization, provide a straightforward way to quantify each metric's contribution to human preference alignment. As shown Figure this interpretabil- ity is highlighted in the QA task, showcasing the importance of each metric in human preference alignment. For example, since ROUGE and ROUGE-L are two highly correlated metries, Bayesian Optimization often drops one of them during the tuning process.
6.2 EFFECTIVENESS AND EFFICIENCY
Improved Performance with Fewer Parameters. Table highlights the compute efficiency and superior performance of METAMETRICC-RN, METAMETRICS-RM utilizing GP. Despite this rel- atively small compute memory footprint, it outperforms models with 70B parameters. This demon- strates that METAMETRICS-RM, offers state-of-the-art accuracy using significantly more cost- effective models, making it an ideal choice for resource-constrained applications while delivering even better performance.
High Parallelization. METAMETRICS is designed to be embarrassingly parallel, enabling each metric to be executed independently without requiring inter-model communication, which results in exceptional efficiency. Unlike large models with 27B parameters that often face bottlenecks, our approach supports the development of highly effective models without such time constraints. Furthermore, METAMETRICS excels at selecting a sparse yet impactful set of metrics from vast pool of potential evaluation metries. As shown in Figure 3 both GP and XGBoost can identify a small subset of key metrics that strongly correlate with human preferences, thereby reducing the need to evaluate a wide range of metrics. This selective approach identifies essential metrics for specific tasks while minimizing computational overhead without compromising performance. In essence, METAMETRICS ideally suited for low-resource settings, providing superior alignment with human preferences across multiple tasks.
6.3 ROBUSTNESS
We present our robustness evaluation in both cross-lingual and cross-dataset scenarios. Compre- hensive experimental results are detailed Subsection F7 For the cross-lingual evaluation, we assess method WMT23 and WMT24 using unseen language pairs. Specifically, we evaluate METAMETRICS on he-en (Hebrew Subbrew-English) en-es (English-Spanish) and ja-zh (Japanese- Chinese). Overall, approach outperforms all existing baseline metrics. We also evaluate the robustness METAMETRICS through cross-datased experiments image captioning, tuning on dataset and testing another. Despite variations in domain and content, results consis-

---

Published as a conference paper at ICLR 2025
tently demonstrate the effectiveness of METAMETRICS. Figure shows that METAMETRICS- tuned on THumB 1.0 and tested on Flickråk and vice-versa out performs all individual metrics. This robustness indicates that METAMETRICS is not overfitting specific dataset characteristics, but rather learning to combine metrics in a way that aligns with general human judgments of caption quality. This generalization capability is crucial for real-world applications, where the evaluation metric may need to perform well on diverse and potentially out-of-domain data.
RELATED WORK
Performance evaluation metries for natural language generation tasks can be categorized into several categories based on how the metric compares generated texts with reference texts.
Surface-level Metrics. These metrics compare the generated text to reference text at the word level, focusing on n-gram overlap direct lexical matching. Common metrics include BLEU that measures precision of n-grams between the generated- and reference-texts (Papineni et al. 2002).
ROUGE that measures recall (Lin 2004), METEOR (Banerjee & Lavie 2005) that goes beyond exact word overlap and considers stemming, synonyms, and paraphrasing, and chrF (Popović 2015) that calculates similarity based on character n-grams rather than word n-grams using F-score.
Embedding-basee Metries. These metries rely on word or sentence embeddings to measure se- mantic similarity between generated and reference text, capturing deeper meaning rather than sur- face overlap. Examples include MoverScore (Zhao et al. 2019), which considers semantic similar- ity at the word level and accounts for word movement (alignment) between texts, and BERTScore (Zhang et al. 2019), a neural-based metric that compares semantic similarity and understands con- textual relationships using BERT embeddings. COMET (Rei al. 2022) and BLEURT (Sellam) iet al. 2020) are neural-based metrics that use contextualized embeddings from models like BERT as inputs to train models to generate prediction estimates of human judgments. For vision-language tasks, ClipScore (Hessel et al. 2021 uses embeddings to compare vision and language modalities.
Ensemble-based Metrics. hance evaluation robustness through averaging (Adiwardana et al., 2020) or heuristics (Phy et al.
Ensemble-based metrics combine multiple individual metrics to en-
2020). The Absolute- Ratlung Multi- culti-Obbeective Reward Model (ArmoRM) (Wang et al. 2024b) is
SotA reward model designed for RLHF (Ouyang et al. 2022).
LLM-based Metrics. LLM- metrics leverage generative language models to assess the qual-
ity of an input by generating scores. Examples include BARTScore (Yuan et al.] [22T), G-Eval (Liu) et al. 2023) utilizing the GPT-4 model (Achiam et al. (023)) Unieval (Zhong et al. 2022) utilizing
the TS model (Raffel et al. 2020), Metric (Juraska et al.) 2023] 2024), COMET, CometKiwi (Rei
al. 2022). XCOMET (Guerreiro et al. 2023), and TEMBA-MQN (Koemi & Federmann 2023)
8 CONCLUSION
Understanding the quality of a performance evaluation metric is crucial for ensuring alignment with human preferences. However, it remains unclear how well each metric captures the diverse aspects of these preferences, as metrics often excel in one particular area but not across all dimensions. To address this, it is essential to systematically calibrate metrics specific aspects of human prefer- ence, catering to the unique characteristics of each aspect. We introduce METAMETRICS, a cali- brated meta-metric designed to evaluate generation tasks across different modalities in a supervised manner. METAMETRICS optimizes the combination of existing metrics to enhance their alignment with human preferences. Our method demonstrates flexibility and effectiveness in both language and vision downstream tasks, showing significant benefits across various multilingual and multi- domain scenarios. Our proposed metric aligns closely with human preferences and is also highly extendable and easily integrable into any application. This makes it powerful tool for improving the evaluation of generation tasks, ensuring that metrics are more representative of human judgment across diverse contexts.
ACKNOWLEDDMENTSS
We would like to express our sincere gratitude to Mingyang Zhou for his insightful suggestions and discussions this project.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Al- ham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Monojit Choudhury. Towards measuring and modeling" culture" in llms: A survey. arXiv preprint arXiv:2403.1 15412, 2024.
Daniel Adiwardana, Minh- Shah-Thang Luong, David So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. Towards human-like open- domain chatbot. arXiv preprint arXiv:2001.099)
2020.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382-398. Springer, 2016.
David Anugraha, Genta Indra Winata, Chenyue Li, Patrick Amadeus Irawan, and En-Shiun Annic Lee. Proxylm: Predicting language model performance on multilingual tasks via proxy models. arXiv preprint wrXiv:2406.09334 2024.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint urXiv:161 I. 1002688, 2016.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72 2005.
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural information processing systems, 13, 2000.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answe annwer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533-1544, 2013.
Mickael Binois and Nathan Wycoff. A survey on high-dimensional gaussian process modeling with application bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization. 2(2):1-26 2022.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231..15127 2023.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024.
Chris Kallison-Burchh Miles Osborne, and Philipp Koehn. Re-evaluating the role of bleu in machine translation research. In 11th conference of the european chapter of the association for computa- fional linguistics, pp. 249-256, 2006.
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Evaluating question answering evaluation. In Proceedings of the 2nd workshop on machine reading for question answering, pp- 119-124, 2019.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Alexander R Fabbri, Wojciech Kryšciriski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. Summeval: Re-evaluating summarization evaluation. Transactions of the Asso- ciation for Computational Linguistics, 9:391 4099,-409, 2021.

---

Published as a conference paper at ICLR 2025
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop Machine Reading for Question Answering, pp. 1-13, 2019.
Peter Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv: 2018.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André FT Martins. Results of wmt22 metrics shared task: Stop using becu-neural metries better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 46-68, 2022.
Markus Freitag, Nitika Mathur, Chi-kiu Lo. Eleftherios Avramidis, Ricardo Rei. Brian Thompson, Tom Koemi, Frédéric Blain, Daniel Deutsch, Craig Stewart, et al. Results of wmm23 metrics shared task: Metries might be guilty but references not innocent. In Proceedings of the Eighth Conference on Machine Translation, pp- 578-628, 2023.
Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frédéric Blain, Tom Kocmi, Jiayi Wang, et al. Are Ilms breaking mt metries? results of the wmt24 metrics shared task. In Proceedings of the Ninth Conference Machine Translation, pp. 47-81, 2024.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Max Grusky. Rogue scores. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1914-1934, 2023.
Nuno M Guerreiro, Ricardo Rei, Daan Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins, xcomet: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint 20Xiv:2310.1044 2023.
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of lmm.. arXiv preprint arXiv:2406.18499 2024.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 7514-7528, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 68511 2020.
Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853-899, 2013.
Fred Jelinek, Robert Mercer, Lalit R Bahl, and James K Baker. Perplexity-a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, and Jianfeng Gao. Tiger: Text-to-image grounding for image caption evaluation. arXiv preprint arXiv:1909.0200 02050. 2019.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
12

---

Published as a conference paper at ICLR 2025
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 6282 6293, 2020.
Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. Metricx-23: The google submission to the wmt 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pp. 756-767, 2023.
Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. Metricx-24 24: The google submission to the wmt 2024 metrics shared task. arXiv preprini urXiv:2410.03983 2024.
Jungo Kasai, Keisuke Sakaguchi, Lavinia Dunagan, Jacob Morrison, Ronan Le Bras, Yejin Choi, and Noah A Smith. Transparent human evaluation for image captioning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies. pp. 3464-3478, 2022.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Tom Koemi and Christian Federmann. Gemba-mqm: Detecting translation quality error spans with gpt-4, arxiv preprint arXiv:2310.13988 2023.
Tom Koemi, Christian Federmann, Roman Grundkiewicz. Marcin Junczys-oowmunt Hitokazu Matsushita, and Arul Menezes. To ship or not to ship: An extensive evaluation of auto- matic metries for machine translation. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Ra- jen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag. Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Koemi, Andre Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sixth Conference Machine Translation, pp- 478-494, Online, November 2021. Association for Computational Linguistics. URL https: wmn 1.57
7 //aclanthology. 2wt.tthoooooo.ogg2222222 otg//021.1222...
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. Lower perplexity is not always human-like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5203-5217, 2021.
Garry Kuwanto, Chaitanya Agarwal, Genta Indra Winata, and Derry Tanti Wijaya. Linguistics the- ory meets Ilm: Code-switched text generation via equivalence constrained large language models. arXiv preprint arXiv:2410.22666, 2024.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Nathan Lambert, Valentina Pyatkin. Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Chun-Liang Li, Kirthevasan Kandasamy, Barnabás Póczos, and Jeff Schneider. High dimensional bayesian optimization via restricted projection pursuit models. In Artificial Intelligence and Statistics, pp.
PMLR, 2016.
Chin-Yes New Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014. URL https:/ /api. semant hor.scchoooaoaaoooo.oooo.oo (oorpusD::41133)
13

---

Published as a conference paper at ICLR 2025
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human aligmment. arXiv preprint arXiv:2303. 2023.
Chi-kiu Lo. Extended study using pretrained language models and yisi-1 for machine translation evaluation. In Proceedings of the Fifth Conference on Machine Translation, pp. 895-902, 2020.
Mohit Malu, Gautam Dasarathy, and Andreas Spanias. Bayesian optimization in nigh-dimensional spaces: A brief survey. In 2021 12th International Conference on Information, Intelligence, Systems & Applications (IISA), pp. 1-8. IEEE, 2021.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't 't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, 2018.
Amin Nayebi, Alexander Munteanu, and Matthias Poloczek. A framework for bayesian optimiza- tion in embedded subspaces. In International Conference on Machine Learning, pp. 4752-4761. PMLR, 2019.
Jun-Ping Ng and Viktoria Abrecht. Better summarization evaluation with word embeddings for rouge. arXiv preprint arXiv:1508.06034 2015.
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. Semeval- 2018 task 11: Machine comprehension using commonsense knowledge. Proceedings of the 12th International Workshop on semantic evaluation, pp. 747-757, 2018.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators. arXiv preprint arXiv:2407.0m 06551, 2024.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCLP),, pp. 2463-2473, 2019.
Vitou Phy, Yang Zhao, and Akiko Aizawa. Deconstruct reconstruct configurable evaluation metric for open-domain dialogue systems. Proceedings of the 28th International Conference Computational Linguistics, pp. 4178, 2020.
Maja Popović. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the tenth workshop statistical machine translation, pp. 392-395, 2015.
Matt Post. A call for clarity in reporting bleu scores. Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint
arXiv: 00..16666.55555 2016.
14

---

Published as a conference paper at ICLR 2025
Ricardo Rei, José GC De Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André FT Martins. Comet-22: Unbabel-ist 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference Machine Translation (WMT), pp. 578-585, 2022.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. Answers unite! unsupervised metrics for reinforced summarization models. arXiv preprint 2019.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, 2020.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. Evaluating open-qa evaluation. Advances Neural Information Processing Systems, 36, 2024a.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and cexture-of-expertt arXiv preprint arXiv:2406. 12845, 2024b.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top por-performing reward models. arXiv preprint arXiv:2406. 08673, 2024c.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. The decades progress on code-switching research in nip: A systematic survey on trends and challenges. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1936-2977 2023.
Genta Indra Winata, Zhaojiang Lin, and Pascale Fung. Learning multilingual emteeddnes for code-switching named entity recognition. In Proceedings of the 4th Workshop Representation Learning for NLP (Rep144i.P-2019, pp. 181-186, 2019a.
Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. Code-switched language models using neural based synthetic data from parallel sentences. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL., pp. 271-280, 2019b.
Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, et al. Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual question answering global cuisines. arXiv preprint Sttt::2410.127000 2024a.
Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D Yao, Shi-Xiong Zhang, and Sambit Sahu. Preference tuning with human feedback on language, speech, and vision tasks: A survey, arXiv preprint arXiv:2409) 564, 2024b.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned 1lms with nothing. arXiv preprint arXiv:2406. 08464, 2024.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2380 2018.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text gener- ation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
15

---

Published as a conference paper at ICLR 2025
Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. Pangea: A fully open multilingual multimodal Ilm for 39 languages. arXiv preprint arXiv:2410.16153, 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 1904. 09675, 2019.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Compulational Linguistics, 12:39-57, 2024.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. Moverscore:
Text generation evaluating with contextualized embeddings and earth mover distance. In Proceed- Empirical Methods in Natural Language Processing and the 9th
ings of the 2019 Conference International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Association for Computational Linguisties, 2019.
Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat: Learn- ing to reason about spatial sounds with large language models. arXiv preprint arXiv:2402.01591, 2024.
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. Towards unified cilti-dimeesional evaluator for text generation. arXiv preprint arXiv:2210.07199, 2022.
A LIMITATIONS AND FUTURE WORK
In our experiment, we utilize metrics that are commonly employed evaluate each downstream task. However, due to limitations in computational resources, we exclude LLMs exceeding 10.7B param- eters to ensure that our metrics can be executed on commercial GPU resources with maximum memory capacity of 48GB. Additionally, we restrict our exploration of metrics to avoid exhaus- tively incorporating LLM-based metrics, given our capacity and resource constraints. Furthermore, leave the exploration of RLHF with PPO to future studies.
Looking ahead, there are significant opportunities for further exploration. We can explore into met- rics across various promising research avenues, such as additional languages, expanded vision and speech modalities (Ho et al. 2020 Blattmann et al.) 2023) Zheng et al.) 2024), and larger multi- lingual generation tasks. In particular, we identify METAMETRICS
valuable tool for multilin- gual (Winata et al. 2019a Joshi et al. [2200] AAnugraaa et al.) 2024), multi-cultural (Adilazuarda al. 2024| Winata al. [2024a] Yue |l.] and code-switching (Winata al. 2019b 2023
Kuwanto et al. 2024) applications.
B METRICS
In this section, we list all metrics we use in our experiments.
B. ABSTRACTIVE TEXT SUMMARIZATION
We use the following metrics:
n-Gram-based Metrics: BLEU-4 Papineni et al. 2002), chrF (Popović 2015).
METEOR (Baner-
jee & Lavie, 2005), ROUGE-I, ROUGE-3 ROUGE-4 4,4, ROUGE-L (Lin 2004).
ROUGE- WET (g
& Abrecht 2015).
Embedding-based Metrics: BARTScore Yuan et al. 2021), BLEURT (Sllam et al. 2020
BERTScore (Zhang et al. 2019).
-Lasedded Metrics: SummaQA (Scialom et al. 2019), UniEval (Zhong et al. 2022), G-Eval
Liu
al. 2023)
16

---

Published as a conference paper at ICLR 2025
B.2 MACHINE TRANSLATION
We use the following metrics:
n-Gram-based Metries: chrF (Popović 2015). BLEU Pappineni et al. 2002);
Embedding- baased Metries: BERTScore Zhang et 2019). Yisi-1 Lo 2020);
LLM- --s-- Metrics: MetricX-23 (L., XL, XXL) (Juraska et al.
2023). COMET, CometKiwi RRei
al. 2022). XCOMET (Guerreiro et al. 2023), GÉMBA-MQM (Koemi & Federmann) 2023
For machine translation, we optimize the Kendall correlation during our tuning.
B.3 QUESTION ANSWERING
use the following metrics:
n-Gram-based Metrics: BLEU-1, BLEU-4 BLEU (Papineni et al. 2002). ROUGE, ROUGE-L.
(Lin 2004), METEOR (Banerjee Lavie
2005):
Embedding- based Metrics: BERTScore (Zhang et al. 2019):
LLM-based Metrics: AnmoRM-Llama3-3B- -B (ArmoRM) (Wang et al.) 2024b).
B.4 IMAGE CAPTIONING
We use the following metrics:
n-Gram based metrics BLEU (Papineni et al. 2002). ROUGE (Lin 2004), METEOR (Banerjee &
Lavie 2005), SPICE (Anderson al. 2016). CIDEr Vedantam al. (0155)
Embedding- Metries TIGEr (Jing et al. 2019) and CLIPScore (Both reference-based and reference-free) (Hessel et al. 2021).
B.5
REWARD MODEL SCORING
The reward model used in the experiment is listed in Table
Table 5: Reward Models.
Reward Model Metric
Hugging Face Link
Saadia/LLlmma3333-mmoorroooon 700-oetrrroo----ea-aeeaaa Reward
bttya://ugglop Lgugglagfac. co/ncidda//// aalaa--333 1-Benot T00--rrooo-TT--orr Roward
3F-Foummttiio/TeeeixxxsLLaaaaa. 1-77B
estpas /houpt nagaaee oo/ Tounder (oaat 1.2.mm 7on
httpa://hugpinnge etyeeeyee Skywort
Balesfoce/SSRRLLLaMaa337778 Judgee-
Btt5a: Alssa://huqiagaane
mncmin//nnemmmm22118---wwwwd (Cai
2024
tttps://hugg wgtare.
intneeestt reserrd
mncminm/ntemmnn22---ewwaad (Cai et al.
2024
https:/huuggl www.ce/ggggggg...e..oooo. htt nnineerste 7b- revard
LxGGorbo/UUNMLLLMM---888
Sttga: htwaaa:://uueccaaaeeecoooo htwp://aeeeeoeooeeeeoaooooooee /Ulle Liaata
AxrGordo/IURMMLLLMaa333335
ttpp::/huuglgggfece
sttpa:/hugggaggeeee.. http://hwuplagaaao...///a23333
sttpa://uuqiinggaee 2aw.aaa3ee...aaa3333333 Lians cawardrodel-1
StinxCCFFftur--LLMMAAR-RR-A
bttg:://huggi.an wttp://awwwcccoaaaeooeeoaeo 11--3----------------- -M
Skywork/SkyworkkBr
Rewwaly Reward 3ow.ro----...aaaaaaa-aa...333 1.mma-31--8B
oo.aacccooooo.....oo.. wt/.oy/eri Skywork Www.ooo---owwo----- Bowoxd-1lana- 3.118
weqweasdaa/RMMMMissall17
/hough
IMM sostrul-no
C METRIC DESIGN
C.1 FRAMEWORK
Figure illlstrrtes the framework for both reference-free and reference-based settings.
17

---

Published as a conference paper at ICLR 2025
MM
y
02
0
3/
0
V/3
moonef
noref A
Xiee
0g
3
D
rof M
O
y
reference-free metric
reference-basee based metric
Figure 4: METAMETRICS Framework for reference-free (MM left) and reference- setting (OMM) right). METRICS integrates multiple metrics 03 and their scores dis learns a function to combine them into score 7MM that aligns best with the human judgment score.
C.2 ALTERNATIVE WEIGHTING SCORE
Multiplicative GP Recall that in the wenehh-wiggttnn scheme for GP, given a set of N evaluation metrics (1 92. ................................ _N] with their associated weights of {wi, W2,.... WN), GP linear given by ÚMM-LINEAR (W) = =
To explore more weighting schemes, we introduce "Multiplicative" GP, which accounts for inter- actions between metrics through pairwise products. For N metrics, there are c = (2) N- unique pairwise combinations. Let Wpair be the weight vector for these combinations, defined as [Wi, W2,..., We) and let paar be the vector pairwise products, where ypai = [0.0, S
2
W pair
i
Nj. The Multiplicative GP output is then computed as MMMMmmulil W (Wpir)) pair Spair-
D DATASETS
In this section, present the data statistics and outline the preprocessing procedures used in our work.
D.1 DATA STATISTICS
Table 6 presents the dataset split sizes for tuning and testing.
D.2 DATA PREPROCESSING
Pre-processing, We apply pre-processing before we use the scores in the training. The pre- processing can be defined follows:
1. Clipping: Let the valid range for ÿ be defined by [oimin ". The clipped metrie i can be defined as:
Omin
0: < immnn
0/
Úi Gmax
g. < ÿr S gmaxx
(2)
if 0 gmax
2. Normalization: After clipping, the score is normalized common scale of [0, 1]:
V/ f
0:
mmux Gumin
3. Inversion (if applicable): If the metrie is such that higher scores indicate worse perfor- mance, we invert the normalized score:
0: 1 Di-
18

---

Published as a conference paper at ICLR 2025
Table 6: Dataset statistics used in the experiments.
Dataset Name
Type
Tuning Size
Testing Size
Machine Translation
WMT23 (MQM) (Freitag al. 2023 WMT20-22 (MQM) (Freitag al. 2022)
Translation Translation
136,260
150,343
Text Summarization
SummEval (Fabbri al.. 2021
Summarization
510
1,190
BenchmarkLLM (Zhang et al.
2024
Summarization
1,035
2,415
Combined
Summarization
1,545
3,605
Question Answering
NQ (kwitkooski et al. 2019 TQ
Open-QA
4,528
10,567
(Josh al. 2017
Open-QA
2,906
6,783
NarrativeQA (Kočısky et al.
2018
RCQA
150
350
SemEval OStermann el al. Combined
2018
Reasoning QA
90
210
Multidomain QA
7,675
17,909
Image Captioning
Flickr8k-Exper (Hodosh et al. THumB 1.0 Kasai et al. 2022
2013
Captioning Captioning
1,746
4,076
600
1,400
Reward Model Scoring
Skywork-Reward-T Reward-Preferenec SeefffrencceOOK--ve
Math, Code, Chat
81,973
allenai/pefferenceeestssse RewardBench (Lamberl al. 2024
Math, Code, Chat Chat, Safety, Reasoning
43,175
8,108
E
OPTIMIZATION
In this section, we explore the optimization details of METAMETRICS.
E.1 METHOD DETAILS
E.1.1 MATERN KERNEL
We train BO using GP with Matern kernel, generalization of the RBF, distinguished by an ad- ditional parameter that controls the smoothness of the resulting function (Williams & Rasmussen, 2006). Conversely, as the parameter approaches infinity, the Matern kernel converges to the RBF kernel. The kernel is described as below:
V2u d(w. (ww.w)) Ku
(22 d(w, (.V/)ddd.w))
k(w, w') T(D)2r-1
(5)
where .(.,.) is the Euclidean distance, (.):) is a modified Bessel function and I(-) is the gamma function.
1.1
ITERATIVE- BASED BASSED PRUNING
Below the implementation of Iterative-hased Pruning performed during training.
E.2
E.2.1 BAYESIAN OPTIMIZATION
Table 7 describes the hyper-parameter parameter settings that for experiments. For the Bayesian optimization, we run GP with a Matern kernel Williams & Rasmussen (2006). a generalization of the RBF kernel, using 2.5.
19

---

Published as a conference paper at ICLR 2025
Algorithm Iterative-based Pruning with XGBoost
1:
procedure ITERATTIEXXCBOOSTCXXX
2: F p+1
Initial set of features
3:
Performance history
4:
Fieast +1
Feature pruning history
5:
for i 4- 1- do
6:
Train (
XGB on X, with CV I f- Importance PXGB)
7:
Compute feature importance from @XCB D Store performance score Identify least important feature
8:
Pli] + p (X=),
9:
Sieast [i] f argmin(Za)
10:
Fiest fleast F F {ficas)
Remove least important feature
12:
end for
4- argmax(P)
Find the best index iteration D Best features from highest performance
14:
Foest f- -FUU Ficast 1i*
Train final fxQR on X Fiee return fagh
17:
end procedure
Table 7: Hyper-paramettrs used for Gaussian Process each task.
Metric
Hyper-parameter
Value
METAMETRICS-SUM
init_points
iter
100
METAMETRICS-MI
init_points
i
100
METAMETRICS-QA
init-points
n_iter
40
METAMETRICS-CAP
init_points
iter
100
METAMETRICS-RM
init_points
nite
100
E.2.2
BOOSTING METHOD
For XGBoost training, different objective depending the task. We perform parameter searching with hyper-parameter values in Table g
F DETAILED RESULTS
In this section, we provide more detailed results that could be included in the main paper due to space constraints.
F.1 ABSTRACTIVE TEXT SUMMARIZATION
Table 9 provides a detailed breakdown of the correlations between various metrics and human pref- erence measurements for abstractive text summarization SummEval LLM. Table I0 provides a detailed breakdown of the correlations between various metrics and human preference measure- ments for abstractive text summarization Benchmark L.M..
20

---

Published as a conference paper at ICLR 2025
Table 8: Initial hyper- paraeer values used for parameter searching during XGBoost.
Metric
Value
METAMETRICS-SUM
n_estimators_ke
100
estimators high
1000
n_estimators_ster
100
n. n.esttmators, ppror objective
Uniform
reg:squarederror
METAMETRICS-MI
n_estimators_ko
100
n estimators high
1000
estimators step
100
n_estimators-pricc objective
Uniform
reg:absoluteerror
METAMETRICS-QA
n.estimators. low
100
estimators high
400
n estimators step n.estimators.prion objective
25
Uniform
reg:squaredlogerro
METAMETRICS-CAP
estimators low
100
n_estimators_higg hhigh
1000
n_estimators.ss
100
neesstmmatrrsppit objective
Uniform
regesquaredlogerror
METAMETRICS-PE RM
n estimators low
100
n_estimators_hin
1000
n_estimators_ster
100
n estimators prior objective
Uniform
rank:pairwise
F.2 MACHINE: TRANSLATIONN
Table 12 describes the detailed breakdown of the correlations between metrics with human annota- tors WMT23.
F.3 QUESTION ANSWERING
Table II provides detailed breakdown of the correlations between various metries and human ratings for the Question Answering task.
F.4 IMAGE CAPTIONING
Table 13
provides detailed breakdown of the correlations between various metrics and human annotations for Image Captioning.
F.5
REWARD MODEL SCORING
Table 14 describes the detailed breakdown of accuracy between metries with human preference on reward model scoring.
F.6 ABLATION STUDY
presents the results of an ablation study on the application of GP using three different weighting methods: linear, multiplicative, and combined (linear and multiplicative) Summariza- tion. Machine Translation, and Reward Model tasks as described in C The results indicate that the linear weighting method generally outperforms the other approaches average. This is likely due
Table I5
21

---

Published as a conference paper at ICLR 2025
Table 9: Kendall and spearman correlation detailed results with human ratings for summarization task on SummEval. Coh., Cons., Fluency, and Rel. corresponds to coherence, consistency, and fluency, relevance, respectively. Bold and underlined values indicate the best and second best per- formance, respectively.
Metric
Kendall Fluency
Spearman Fluency
Coh.
Cons.
Rel.
Avg.
Coh.
Cons.
Rel.
Avg.
BLEU
0.110
0.126
0.113
0.170
0.130
0.157
0.160
0.145
0.239
0.175
chrF
0.143
0.094
0.071
0.198
0.127
0.205
0.1
0.091
0.278
0.173
METEOR
0.077
0.102
0.072
0.162
0.103
0.108
0.130
0.093
0.229
0.140
ROUGE-1
0.112
0.103
0.061
0.189
0.116
0.159
0.130
0.079
0.264
0.158
ROUGE-3
0.099
0.115
0.048
0.154
0.104
0.142
0.147
0.063
0.219
0.143
ROUGE-4
0.096
0.122
0.067
0.143
0.107
0.137
0.155
0.087
0.203
0.146
ROUGE-I
0.110
0.087
0.069
0.157
0.106
0.156
0.111
0.089
0.219
0.144
ROUGE-WID WEI
0.115
0.088
0.081
0.169
0.113
0.164
0.112
0.105
0.237
0.115
BLEURT (Max)
0.185
0.070
0.114
0.189
0.140
0,262
0.088
0.062
0.194
0.152
BLEURT (Mean)
0.099
0.011
0.029
0.130
0.067
0,123
0.094
0.052
0.202
0.118
BERTScore (fI)
0.100
0.120
0.181
0.150
0.128
0.256
0.172
BERT'Score (Recall)
0.104
0.111
0.093
0.176
0.121
0.148
0.141
0.120
0.250
0.165
SummaQA (fI)
0.051
0.120
0.106
0.114
0.098
0.073
0.152
0.136
0.165
0.132
SummaQA (Prob)
0.070
0.100
0.070
0.138
0.095
0.098
0.128
0.092
0.196
0.129
LLM-BASED METRICS
BARTScore (Mean)
0.086
0.074
0.040
0.143
0.086
0.123
0.094
0.052
0.202
0.118
UniEval G-Eval (GPT4)
0.413
0.353
0.324
0.577
0.439
0.446
0.480
0.429
0.413
0.409
0.437
0.422
0.565
0.510
0.470
0.581
0.531
ENSEMBLE BASELINES
Uniform
0.141
0.133
0.117
0.213
0.151
0.201
0.170
0.150
0.298
0,205
Weighted Avg
0.150
0.141
0.122
0.220
0.159
0.215
0.179
0.158
0.309
0.215
METAMETRICS-SUM
GP
0,172
0.140
0.130
0.252
0.174
0.244
0.179
0.167
0.354
0.236
XGBoost
0.192
0.186
0.186
0.276
0.210
0.274
0.236
0.239
0.386
0.284
W/ LLM-BASED METRICS
GP
0.454
0.419
0.409
0,449
0.433
0.609
0.519
0.470
0.601
0,550
GP (Top 2)
0.461
0.428
0.409
0.449
0.437
0.628
0.528
0.470
0.601
0.557
XGBoost
0.476
0.367
0.460
0.512
0.600
0.553
XGBoost (Top
0.476
0,436
0,430
0.445
0.447
0.636
0.508
0.511
0.594
0.562
to GP's preference for operating on lower-dimensional spaces (Li et al. 2016 Frazier 2018 Nayebi
et al. 2019; Malu et al. 2021] Binois & Wycoff (2022). These findings reiterate the benefits of using fewer metrics for evaluation, which simplifies the optimization process.
Table 16 compares Kendall and Spearman correlations with human ratings for the summarization task using XGBoost. The results show minimal performance differences between the objective functions, which is expected given the similarity of evaluation measurement between Spearman and Kendall correlations.
F.7 ROBUSTNESS
We demonstrate the robustness of The cross-lingual setting assesses our method's is capability with unseen languages, while the cross- dataset setting evaluates its performance in the face distribution shifts or when the dataset origi- nates from different domain.
in two scenarios: cross-lingual and cross-dataset.
F.7.1 CROSS-LINGUAL SETTING
We evaluate our method on WMT23 and WMT24 for unseen language pairs. To calibrate METAMETRICS, we employ three specific language pairs: zh-en (Chinese-Englishh en en-de Einglis-Germann and en-ru (English-Russe Russian). For WMT23, we test our calibrated metric on unseen language pairs, such he-en (Hebrew-Englishh with results presented in Table 17 For WMT24, we extend the evaluation to include en-es Singlish-Spanisee and ja-zh (Japanese- Chinese), with results detailed in Table T8
22

---

Published as a conference paper at ICLR 2025
Table 10: Kendall correlation results with human ratings for summarization task on Benchmark
LLM. Faith., Coh., and Cons. correspond to faithfulness, coherence, and consistency, respectively. Bold and underlined values indicate the best and second best performance, respectively.
Metric
Benchmark LLM
Faith.
Coh.
Cons.
BLEU
0.162
0.246
0.252
chrF
0.272
0.163
0.342
METEOR
0,251
0.182
0.331
ROUGE-1
0.195
0.252
0.290
ROUGE-3
0.138
0.142
0.225
ROUGE-4
0.138
0.085
0,200
ROUGE-I
0.137
0.270
0.239
ROUGE-WEI
0.223
0.218
0.292
BARTScore (Mean)
0,080
0.216
0.166
BLEURT (Max)
0.046
0.264
0.143
BLEURT (Mean)
0.046
0,264
0.143
BERTScore (fl)
0.171
0.280
0.283
BERTScore (Recall)
0.228
0.223
0.333
SummaQA (fl)
0.318
0.038
0.293
SummaQA (Prob)
0,330
-0.003
0.303
ENSEMBLE BASELINES
Uniform
0.226
0.234
0.328
Weighted Avg
0.256
0.264
0.337
METAMETRICS
GP
0.356
0.308
0.383
XGBoost
0.406
0.411
0.397
XGBoost (Iterative Top
0.377
0.378
0.383
F.7.2 CROSS-DATASET SETTING
In addition to the cross-lingual setting, we also explore a cross-dataset scenario specifically for image captioning, as shown Table 19 Here, we present the results when METAMETRICS-CC. is tuned on THuMB 1.0 and tested Flickr8k, and vice versa. Our findings reveal that METAMETRICS: CAP, whether using GP or XGBoost, outperforms individual metrics, demonstrating its robustness in the presence of data shifts.
F.8 TIME BENCHMARK
We benchmark the time elapsed to training and performing inference. All experiments run on the same machine with AMD EPYC 9354 32-Core Processor and NVIDIA RTX 6000 Ada GPU with 48GB memory. We show the detailed runtime for train and test split, and calibration on SummEval dataset in Table 20 Specifically, for G-Eval (GPT4), it costs around US$82 to run top p of frequency penalty of O,
SummEval benchmark with max token of 5, temperature
a presence penalty of 0, and completion choices of 20.
split with METAMETRICS SUM, is important to To compare the proper time evaluation CEEMEEERRISSUMEEE evaluation process. Specifically, for account for all the components of individual metrics, we can consider the inference time on the test split. On the other hand, for we need to consider the total time, consisting of inference time both the
META METRICS-SUUS train and test splits, and then the calibration time using the train set. For example, G-Eval (GPT4) takes 7,173.50 seconds to evaluate the test split, while XGBoost (all metrics including LLM- based metrics) takes 3,976.47 + 9,292.83 + 647.08 13,916.38 seconds evaluate on the test split.
Certain metric groups, such as BLEURT, SummaQA, and BERTScore, have multiple variations, all of which can be computed in single evaluation pass. Therefore, the time measurement for METAMETRICS reflects the total duration of the pass, rather than the summation of time of all individual rows.
23

---

Published as a conference paper at ICLR 2025
Table 11: Full performance of METAMETRICS- compared the 8 best-performing standalone metrics and other baseline setups. Bold and underlined values indicate the best and second best performance, respectively. The Combined column reflects results when all datasets are merged.
Setup
EVOUNA
EQAE
Combined
NQ
TQ
NarrativeQA
SemEval
ARMORM METRICS
ultaaeedbackkhhomee
0.202
0.281
0.297
0.335
0.296
hellfueesss
0.204
0.293
0.332
0,,427
0.302
helpsteer-coorrecnnes.
0.206
0.300
0.336
0.403
0,305
argilla-judge-Im
0.243
0.301
0.253
0.337
0.332
N-GRAM-BASED METRICS
BLEUI
0.461
0.353
0.362
0,2660
0.424
ROUGE
0,494
0.399
0.586
0.521
0,454
ROUGE-L
0.495
0.399
0.584
0.518
0.454
METEOR
0.517
0.435
0.570
0,461
0.494
ENSEMBLE BASELINES
Uniform
0.425
0.405
0.564
0,487
0.449
Weighted Avg.
0.497
0.423
0.582
0.467
0.484
METAMETRICS-QA
GP (All)
0.518
0.448
0.586
0),464
0.512
GP (Top 5)
0.506
0.450
0.570
0,4552
0.512
XGBoost (All)
0,543
0.471
0.601
0,486
0,536
XGBoost (Top 5)
0.535
0.491
0.624
0.513
0.534
XGBoost (Iterative Best)
0,545
0.488
0.626
0.510
0.538
XGBoost (Iterative Top 5)
0.540
0.488
0.626
0.503
0.528
An important observation is that the bottleneck for METAMETRICS-SUM lies in the inference time for both the train and test splits, which dominates the total runtime compared to the calibration time. This particularly evident when METAMETRICS-SUM includes metrics like G-Eval. The calibration time is relatively minor, with the longest calibration being for GP (all metrics with LLM- based metrics), which takes 647.08 seconds. However, if the metric outputs are already available, the inference step can be skipped.
Currently, the reported time for processing the train and test splits for METAMETRICS sumes sequential evaluation of metrics. However, this sequential approach can be optimized by running metrics in parallel, as their evaluations are completely independent each other. other words, these computations are embarrassingly parallelizable. With this optimization, the total run- time for METAMETRICS- SUU would effectively be the time taken by the slowest metric, plus the calibration time. For example, for GP (all metrics except LLM-based metrics), the longest metric that takes to be evaluated SummaQA, which takes 790. seconds. Therefore, the total running time when running METAMETRICS-NUS in completely parallel manner will be 790.1 402.51 = 1, 1,192.62 seconds.
as-
G WEIGHTS AND FEATURE IMPORTANCE
G. QUESTION ANSWERING
There are 8 setups for the question-answering task. The resulting weights or feature importance for each setup are reported in Figure
GP-ALL (NQ), The selected metrics are helpsteer-verbositg ArmoRM-overall, meteor, and rougel.
24

---

Published as a conference paper at ICLR 2025
Table 12: Correlation with human ratings on WMT23 (MQM). 'The results are collected from Fre-
itag al. (2023). Bold and underlined values indicate the best and second best performance, spectively.
Metric
overall sys/seg avg-comi
en-de 50g peurson
he-en scg pearson
zh-en 50g
sys pearsun
sog 902-1
sys peatson
scg a0C-1
sys peatson
seg
pearson
BCC-1
REFERENCE BAREEE BASED METRIC
chrifl
0.694
0.866
0.776
BLEUT
0.696
0.917
0.192
0.520
0.769
0.220
0.442
0.734
0.119
0.472
BERTScore!
0.742
0.891
0.325
0.528
0.895
0.335
0,515
0.810
0.236
0.499
Yisi-1"
0.754
0.925
0.366
0,542
0.917
0.395
0.529
0.823
0.290
0.504
MetricX-23-XXII
0.808
0.977
0.585
0.603
0.910
0.548
0.577
0.873
0.625
0.531
XCOMET Ensemble'
0.825
0.980
0.695
0.604
0.950
0.556
0.586
0.927
0.650
0.543
0.779
0.990
0.575
0.940
XCOMET-XL
0.817
0.970
0.670
0.596
0.949
0.530
0,576
0.928
0.607
0,530
XCOMET-XXL
0.817
0.983
0.660
0.602
0.952
0.465
0,560
0.965
0.597
0.546
ENSEMBLE BASELINES
Uniform
0.824
0.989
0.688
0,610
0.954
0.546
0.588
0.935
0.641
0.543
Weighted
0.827
0.989
0.690
0.613
0.955
0.545
0.587
0.937
0.642
0.545
METAMETRICS-MT
GP
0.819
0.970
0.638
0.610
0.947
0.546
0.590
0.900
0.646
0.539
XGBoost
0.825
0.992
0.680
0.616
0.957
0.557
0.574
0.929
0.637
0.546
REFERENCE:
mbr-metricx-get
0.788
0.976
0.571
0,584
0.915
0.553
0.936
0.489
0.537
MetricX-23-QFF -F
0,800
0.969
0.626
0.596
0.858
0.520
0.564
0.859
0.647
0.527
GEMBA-MQM!
0.802
0.993
0.502
0.572
0.939
0.401
0.564
0.991
0.449
0.522
0.808
ComerKiwi-or QF
0.781
0.946
0.475
0.569
0.859
0.387
0.544
0.963
0.442
0,525
23
0.763
MetricX-23--4 23--2-3-QXX--------
0.797
0.934
0.547
0.607
0.813
0.459
0.575
0.877
0.652
0,531
CometKiwi--EEXLI
0,786
0.976
0.447
0.571
0.900
0.384
0,533
0.974
0.430
ENSEMBLE BASELINES
Uniform
0.801
0.935
0.552
0.000
0.816
0.484
0.566
0.948
Weighted Avg
0.802
0.934
0.555
0.603
0.811
0.487
0,568
0.943
0.647
0,540
METAMETRICS MT-QE
GP
0.801
0.934
0.556
0.609
0.815
0.474
0.578
0.900
0.660
0.537
XGBoost
0.805
0.967
0.583
0.604
0.881
0.509
0.568
0.869
0.642
0.526
GP-ALL (QQ). The selected metrics are helpsteer-coherence, helpsteeer-copleextty helpsteer-verbasity, wwwrffbiinst.followwiii ArmoRM-overall, meteor, rouge, and rougel.
code-inst-followinn
GP-ALL (NarrativeQA). The selected metrics code-style, ultrafb-truthfulnes -vvoall bertscore-fl, bertscore-p, bertscore-r, bleu4, meteor, and rouge.
beaeertails-is.afee uttrafb-innt..olooo.in
code-readbility.
GP-ALL (SemEval). The selected metrics are code-complexity, code-explanation, code-inst-followinng code-readability, code-style, hellpsteer-ccceeencaa, hel pessn prometheus-score, ultrafb-overall, bewtscore-r,.rteee...rrr rouggell
and
GP-ALL (Combined).
The selected metrics are code-style, -elpstte-----rrreetteees helpsteer-werboost ArmoRM-overall bl meteor, and rouge.
GP-Top (NQ). The selected metrics are bertscore-r, bleul, meteor, rouge and rougel.
GP-Top 5 (TQ). The selected metrics are helpsteer--oorrechhess, bleul, meteor, rouge, and rougel.
GP-Top 5 (NarrativeQA). The selected metrics are bertscore-fl, bertscore-r, meteor, rouge, and rougel.
25

---

Published as a conference paper at ICLR 2025
Table 13: Kendall correlation results of METAMETRICS-CAM is tuned on THumB 1.0 and tested on Flickr8k, and vice versa. Scores are reported for multiple metrics, with the average performance (avg.) across datasets. Bold and underlined values indicate the best and second best performance, respectively.
Metric
Flickrök
THumB 1.0
avg.
BLEU-1
0.377
0.264
0.321
BLEU-4
0.361
0.210
0.286
ROUGE
0.375
0.237
0.306
METEOR
0.481
0.281
0.381
CIDEr
0.498
0.263
0.381
SPICE-F
0.591
0.200
0.396
SPICE-PR
0.603
0.170
0.387
SPICE-RE
0.579
0.221
0,400
TIGEr
0,542
0.153
0.348
CLIP-S
0.514
0.292
0,403
RefCLIP-S
0.576
0.210
0.393
METAMETRICS-CAP
GP (All)
0.664
0.329
0.496
GP (Top 5)
0.655
0.277
0.466
XGBoost (All)
0.670
0,277
0.474
XGBoost (Top 5)
0.664
0.283
0.474
XGBoost (neerative-Bes))
0.662
0.285
0.474
XGBoost (Iterative-Top5)
0.659
0,276
0.468
METAMETRICCSCAP (Cross-Dalaset)
GP (All)
0.638
0.298
0.468
GP (Top 5)
0.560
0.246
0.403
XGBoost (All)
0.595
0.323
0.449
XGBoost (Top
0.575
0.311
0.443
XGBoost (Iterative-Bes Best)
0,635
0.321
0.478
XGBoost (Tterative-Top
0.634
0.301
0.467
GP-Top 5 (SemEval). rouge, and rougel.
The selected metrics are helppsteerr-ccheenne, bleu4, meteor,
GP-Top (Combined). rouge, and rougel.
The selected metrics are argilla-judge.111 bleul, meteor,
G.2 MACHINE TRANSLATION
There are settings for MT: reference-based and reference-free. Both setups' resulting weights or feature importance are reported in Figure 6 The weights selected for GP are as follows:
GP-ALL (Reference-based), The selected metrics are MetricX-XXL. COMET, and XCOMET-XI.
GP-ALL (Reference-freee. The selected metrics are MetricX-OE-XX
MetricX-Op -Or--e COMETKiwi-QR and SAMETTiwwiXL--O
G.3 IMAGE CAPTIONING
We utilize datasets for Image Captioning, each with two setups: a) cross-dataset and b) in- dataset. The resulting weights and feature importance are presented Figure
The weights selected in GP follows:
26

---

Published as a conference paper at ICLR 2025
Table 14: Accuracy results for Rewadd-Model-As-A-Meee Bold and underlined values indicate the best and second best performance, respectively.
Metric
Score
Chat
Chat Hard
Safety
Reasoning
(aama-3.-Neeottren Reward
94.1
97.5
85.7
95.1
98.1
wwwork-RRwwrd-Gemmmeee
93.8
95.8
91.4
91.9
96.1
DoxlEval-Llama33--7000
93.5
94.1
90.1
93.2
96.4
Skywork-Criic-LLamee3--000
93.3
96.6
87.9
93.1
95.5
URM-LLaMM--31-888
92.9
95.5
88.2
91.1
97.0
SFR-LLaMMa-331-00---uuue
92.7
96.9
84.8
91.6
97.6
3kywort-ReewrddJJaaa33..111
92.5
95.8
87.3
90.8
96.2
GRM-LJama3-8B
91.5
95.5
86.2
90.8
93.6
Internlm2-200--ewward
90.2
98.9
76.5
89.5
95.8
ArmoRM-LLana--BB--00
90.4
96.9
76.8
90.5
97.3
URM-LLaMa-3-88
89.9
96.9
78.7
88.2
95.7
Internlm2-78-Reward
87.6
99.2
69.5
87.2
94.5
ENSEMBLE BASELINES
Uniform
92.5
98.3
83.1
90.8
97.6
Weighted Avg.
92.5
98.3
83.1
90.7
97.6
METAMETRICS-RM
GP
93.5
98.9
86.2
90.7
98.2
GP (Top 3)
92.8
98.3
84.4
90.7
97.9
XGBoost
92.9
95.8
89.7
92.2
94.0
XGBoost (Iterative Best)
92.7
95.5
89.0
91.6
94.7
XGBoost (Iterative Top 3)
92.3
95.5
87.7
91.2
94.8
XGBoost (Top 3)
92.2
96.6
86.2
91.2
94.7
Table 15: Ablation study on GP using linear, multiplicative, and combined (linear and multiplicative) weightings on Machine Translation (MT), and Reward Model tasks.
Method
MT
MT (QE)
Reward Model
GP (Linear)
0.819
0.801
93.5
GP (Multiplicative)
0.822
0.798
91.2
GP (Combined)
0.818
0.803
92.1
Table 16: Kendall correlation results with human ratings for summarization task with Spearman and Kendall the evaluation functions. Coh., Cons., Rel., and Faith. corresponds to coherence, consistency, relevance, and faithfulness respectively.
Metric
Eval
Benchmark LLM Coh.
Combined Coh.
Coh.
Cons.
Fluency
Rel.
Faith.
Cons.
Cons.
Avg.
XGBoost (Kendall)
0.475
0.376
0.407
0.439
0.406
0.411
0.397
0.403
0.367
0.409 0.409
XGBoost (Spearman)
0.476
0.376
0.403
0.448
0.406
0.411
0.397
0.402
0.369
GP-ALL (Flickr8k). The selected metrics cider, clipscore, meteor, ref-clipscore, spice_f, spice_pr, spice_re, and tiger.
GP-ALL (THumB 1.0). The selected metrics are bleul, clipscore, meteor, ref-clipscore rouge, spice_re, and tiger.
GP Top 5 (Flickrök), The selected metrics are zef-clipscore, spice f, spice pr, spice re, and tiger.
27

---

Published as a conference paper at ICLR 2025
Table 17: Results on Cross-lingual settings on WMT23 for Hebrew-English (he-en). Bold and underline values indicate the best and second best performance, respectively.
Metric
he-en
sys pearson
seg
seg
pearson
acc-L
chrFt
0.776
0.221
0.460
BLEUT
0.769
0.220
0.442
BERTScore!
0.895
0.335
0,515
Yisi-1
0.917
0.395
0.529
MetricX-23-XXLL
0.910
0.548
0.577
CCOMET-Ensemble
0.950
0.556
0.586
COMET
0.940
0,401
0.531
XCOMET-XL
0.949
0,530
0,576
XCOMET-XXL
0.952
0,465
0.560
METAMETRICS-MT
GP
0.947
0.546
0.590
XGBoost
0.957
0.557
0.574
Table 18: Average Results of acc-t and Soft-pairwise accuracy (SPA) on Cross-lingual settings on WMT24 (Freitag al. 2024) for English-Spaniss (en-es) and Japanese- Chinese (ja-zh). We report the number following the metric calculated by WMT24 organizers for language breakdown perfor- mance since the test set labels are unseen to the authors. Bold and underline values indicate the best and second best performance, respectively.
Metric
en-es
ja-zh
avg
sentinel-ref-mqm
0.631
0.490
0.561
BLEU
0.596
0.588
0.592
spBLEU
0.602
0.590
0.596
chrF
0.615
0.616
0.616
0.630
0.602
0.616
BERTScore
0.594
0.651
0.623
MEE4
0.635
0.625
0.630
damonmonli
0.688
0.633
0.661
YiSi-1
0.657
0.666
0.662
PrismRefSmall
0.666
0.667
0.667
PrisiRefMedium
0.734
0.545
0.640
BLCOM_I
0.698
0.676
0.687
BLEURT-20
0.688
0.687
COMET-22
0.744
0.636
0.690
XCOмET
0.740
0.700
0.720
MetrieX-24 (Hybrid)
0.742
0.718
0.730
METAMETRICS-MT (GP)
0.745
0.717
0.731
GP Top 5 (THumB 1.0). rouge.
The selected metrics are bleui, cider, cl clipscore, meteor, and
G.4 TEXT SUMMARIZATION
The weights and feature importance from the resulting optimization reported in Figure g Figure 9 and Figure for SummEval, BLLM, and the merged dataset respectively. The weights selected GP follows:
G.4.1 w/o LLM-BASED METRICS
GP-ALL (SummEval: Coherence). The selected metrics are BLEURT (Max), chrF, Rouge-1, Rouge-4, and Summa QA (Prob).
28

---

Published as a conference paper at ICLR 2025
Table 19: Kendall correlation Cross-Dataset results of METAMETRICS- when tuned on THumB 1.0 and tested on Flickr8k, and vice versa. The columns correspond to the testing dataset. Scores are reported for multiple metries, with the average performance (avg.) across Cross-Dataset results. Bold and underlined values indicate the best and second best performance, respectively.
Metric
Flickr8k
THumB 1.0
avg.
BLEU-1
0.377
0.264
0.321
BLEU-4
0.361
0.210
0.286
ROUGE
0.375
0.237
0.306
METEOR
0.481
0.281
0.381
CIDEr
0.498
0.263
0.381
SPICE-F
0,591
0.200
0.396
SPICE-PR
0.603
0.170
0.387
SPICE-RE
0.579
0.221
0,400
TIGEr
0.542
0.153
0.348
CLIP-S
0.514
0.292
0.403
RefCLIP-S
0.576
0.210
0.393
METAMETRICS-CAN
GP (All)
0.638
0.298
0.468
GP (Top 5)
0.560
0.246
0.403
XGBoost (All)
0.595
0.323
0.449
XGBoost (Top 5)
0.575
0.311
0,443
XGBoost (Iterative-Bend Best)
0.635
0.321
0.478
XGBoost (Iterative-Top 5)
0.634
0.301
0.467
GP-ALL (vum
Consistency). The selected metrics BERTScore (Recall), BLEURT (Max), chrF, SummaQA (f1), , and SunmaQA (Prob).
GP-ALL (SummEval: Fluency). The selected metrics BERTScore (Recall), BLEURT (Max), chrF, SummaQA (f1), and SummaQA (Prob).
GP-ALL (SummEval: Relevance). The selected metrics BERTScore (Recall), BLEURT (Mean), chrF, Rouge-WEL, WE1, Rouge-1, Rouge-4, and SummaQA (Prob).
G.4.2 w/ LLM-BASED METRICS
GP-ALL (SummEval: Coherence). The selected metrics are BARTScore (Mean), BLEURT (Max), BLEURT (Mean), Rouge-3, Rouge-4, SummaQA (Prob), G-Eval, and UniEval.
GP-ALL (Slm
Consistency). The selected metries are BLEURT (Max), Summa (Prob), G-Eva1, and UniEval.
GP-ALL (SummEval: Fluency).
The selected metric is G-Eval.
Relevance). The selected metrics are BLEURT (Mean), chrF, Rouge-3, 3, Rouge-4, SummaQA (Prob), G-Eval, and Un Eval.
GP-ALL (SummEval:
GP-Top 2 (SummEval: Coherence). The selected metrics are G-Eval and UniEval.
GP-Top 2 (SummEyal:
Consistency).
The selected metrics G-Eval and Uni Eval.
GP-Top 2 (SummEval: Fluency).
The selected metrics are G-Eval and UniEval.
GP-Top 2 (SummEyal: Relevance).
The selected metrics are G- Eval and Eval.
29

---

Published as a conference paper at ICLR 2025
Table 20: Runtime for inference time on train and test split and the calibration time for METAMETRICS. Rows with exact same run time is due to the groups of metrics (e.g., BLEURT (Max) and BLEURT (Mean)) being calculated in a single pass. tG-Eval (GPT4) costs around US$82 to run SummEval benchmark with max token of 5, temperature of 2, top of 1, frequency penalty of presence penalty of O, and completion choices of 20. The running time reported for METAMETRICS-SUNT for train and test split is in sequential manner. In other words, running each metric in an embarrassingly parallel way can reduce the overall inference time. To compare the proper time evaluation test split with METAMETRICS SUM, it is important to account for all the components of METAMETRICH METARCS-TRISSEE SARRIISSSUMMM SUM's s evaluation process. Specifically, for individual metrics, consider the inference time the test split. On the other hand, for METAMETRICS- we need to consider the inference time on both the train and test splits, and then the calibration time using the train set. For example, G-Eval (GPT4) takes 7,173.50 seconds evaluate the test split, while XGBoost (all metrics including LLM-based metrics) takes 3,976.47 + 9,292.83 + 647.08 13,916.38 seconds evaluate on the test split.
Metric
Inference Time (in sec)
Calibration Time (in sec)
Train Split
Test Split
BLEU
0.72
1.73
N/A
chrF
3.37
8.67
N/A
METEOR
5.74
13.57
N/A
ROUGE-1
57.19
162.77
N/A
ROUGE-3
68.24
151.90
N/A
ROUGE-4
70.43
149.65
N/A
ROUGE-1
66.53
155.15
N/A
ROUGE-WEI
56.35
100.19
N/A
BLEURT (Max)
139.87
336.22
N/A
BLEURT (Mean)
139.87
336.22
N/A
BERTScore (fl)
1.74
3,99
N/A
BERTScore (Recall)
1.74
3.99
N/A
SummaQA (fl)
322.07
790.11
N/A
SummaQA (Prob)
322.07
790.11
N/A
LLM-BASED METRICS
BARTScore (Max)
22.39
50.83
N/A
BARTScore (Mean)
22.39
50.83
N/A
UniEval
80.37
194.55
N/A
G-Eval (GPT4)
3,081.46
7
N/A
ENSEMBLE BASELINES
Uniform
3,976.47
9,292.83
N/A
Weighted Avg
3,976.47
9,292.83
N/A
METAMETRICC-SUM
GP
792,25
1,873.95
402.51
XGBoost
792.25
1,873.95
276.52
w/ LLM-BASED BBASED METRICS
GP
3,976.47
9,292.83
647.08
GP (Top 2)
3,161.83*
7,368.051
187.73
XGBoost
3,976.47
9,292.831
337.88
XGBoost (Top 2)
3,161.837
7,368.05t
223.69
GP-ALL (BLLM: Faithfulness). The selected metrics are chrF, SummaQA (f1), and SummaQA (Prob).
GP-ALL (BLLM: Coherence). The selected metrics are BARTScore, BERIScore (f1),
BLEURT, Rouge-1, and Rouge-I.
30

---

Published as a conference paper at ICLR 2025
IOAU
(04)- GPP 1ltp
XG8 1ip
(04)
XGB (ivi
EGB
800
044
000 Dme
code compleing
400
844
081
899
041
0x1
839
6.24
400 800 0.
Metrics
600 0m:
030
500 041 001
6.08
0.80
8.30
0250
6.99
6:00
0485
0.1
0:00
033
8.19
aggit overll puality
080 000
0000
K00
BA00
code syc
800
611
6:00
Metrics
6:00
6.00
Amallt
4000
betweer
BIN
6:00
retevt
DEX Can
0.0
BES
Figure 5: Weights and Feature Importance for METAMETRICS-QU
1.0
comettanige
0.13
0.02
0.21
0.01
0.8
mebric qe mebrick large comebteni allai-a--op
0.06
0.06
0.27
0.02
Metrics
XOB
Figure 6: Weights and Feature Importance for METAMETRICS-MT.
GP-ALL (BLLM: Relevance). The selected metrics are BERTŠcore (Recall), chrF, METEOR, Rouge-WE1, SummaQA (f1), and SummagA (Prob).
31

---

Published as a conference paper at ICLR 2025
Captoniig
Captoring
Curkeeiing Tep
DMC
CO9
X31
ENI
IXC
E37
EXI
Metrics
PMMC
EX1
C11
2U5
SUC
6.1
831
120
2
212
114
6.0
G24
Figure Weights and Feature Importance for METAMETRICS-CAE
GP-Top 5 (BLLM: Faithfulness). (f1) and Summa@a (Prob).
The selected metrics are METEOR, ROUGE-WEE Summa
GP-Top 5 (BLLM: Coherence). The selected metrics are BERTScore (f1), BLEURT, Rouge-1, and Rouge-I.
GP-Top (BLLM: Relevance). The selected metries are BERTScore (Recall), chrF, METEOR, Rouge-WE1, and SummaQA (Prob).
GP-ALL (Combined: Coherence). The selected metrics are BARTScore (Mean), BERTScore (f1), and BLEURT (Mean).
GP-ALL (Combined:
Relevance). The selected metrics are BERTScore (f1), BERTScore (Recall), BLEURT (Mean), SummaQA (f1) and SummaQA
(Prob),
GP-Top 5 (Combined: (Mean).
Coherence). The selected metrics are BERTScore (f1) and BLEURT
GP-Top 5 (Combined:
The selected metrics are BERTScore (f1), BERTScore (Recall), Rouge-WEL, and Rouge-1.
Relevance).
G.5 REWARD MODELING
The weights and feature importance from the resulting optimization are reported in Figure II The weights selected in GP follows:
GP-ALL. The selected
metrics
are
GRM-Llama3 80---8B--- rewweeedddddee URM-LLaMa-3.1 3.1-8B.al1, 1nternlm2--D-rewardd..
Rkwword Llama-3.am....aaaa 3.1--B........111 and
GP-Top 3. The selected metrics are
all,
seyyorddRewwrrd-ulmmae 1- all, and Intern.m22-7b-eeree
H CORRELATION MEASUREMENT DETAILS
This section describes the details of correlation measurements used in experiments.
Kendall's Tau Kendall's Tau measures rank correlation by assessing the agreement between two rankings based the proportion of concordant and discordant pairs. It is commonly used to eval- uate how closely an evaluation metric's 's rankings align with human rankings in tasks like Machine Translation.
32

---

Published as a conference paper at ICLR 2025
Nerbsor, peat
5.00
8:30
8.0
Bly
800
690
5:00
8:0
bauut JiA4
8.31
180
0.3
6:30
Metrics
1:00
033
6.9
800
006
880
500
5:00
046
8.00
8:00
mage
8:00
6:4
2.06
9.00
818
8.0
0:38
9.0
8.00
000
880
8:00
5:00
0.0
9:00
8:00
8:00
612
8:08
8:00
Sum
XGB (with LLM) ALL
Sum XOB tne LLM) ALL
Sumn XBB TRpp
berboor
8.4
6:4
5.7
6:0
bertscor PROR
804
8.00
654
6:0
0.6
0.6
0:55
6:77
6.44
6:5
0.01
6:39
8.04
0:35
0.33
6:27
Metrics
gred
912
0.2
0:1
5:35
0:33
6.71
6.11
1.32
8.04
0.07
6:55
8:44
0.64
0:05
6.44
0.04
0.55
5:55
0:55
0.0
unival
0:33
0.77
Figure 8: Weights and Feature Importance for METAMETRICS-SUMM. the SummEval dataset.
H.1 MACHINE TRANSLATION TASK CORRELATION
System-Level Pearson Correlation (sys pearson) This metric evaluates the correlation between metric and human scores aggregated at the system level. It is calculated the Pearson correlation coefficient between the averaged scores each system.
Segment-Level Pearson Correlation (seg pearson) At the segment level, correlation is computed by flattening the system x segment score matrices into vectors and comparing the metric-gent generated vector with the human-evaluatio vector. This provides a more granular assessment of alignment.
System-level pairwise ranking accuracy (seg acc-t) System-level pairwise ranking accuracy, as proposed by Kocmi et al. (2021) measures the agreement between metric and human judgments in ranking translation systems. It is computed as the percentage of system pairs for which the metric's ranking matches numan-providee rankings. Rankings are based on pooled data across all language pairs, making this metric robust for system comparisons.
33

---

Published as a conference paper at ICLR 2025
Sansoor
G9B
880
205
60
541
0.
Mestics
186
334
304
6S0
3.4
6:8
334
1:0
8.4
33
3:1
014
3.11
3.5
Metrics
129
L18
835
3.4
145
649
334
Dugel
DM
617
531
617
334
330
121
538
232
2:0
Figure 9; Weights and Feature Importance for METAMETRICS-SOME on the BLLM dataset.
34

---

Published as a conference paper at ICLR 2025
Contened 1g
Combined
SMTSSO pM
545
880
609
649
8.4
543
Moxt
0.1
531
Metrics
8.00
644
644
Metric
OagenaL
mapt
mgg
Figure 10: Weights and Feature Importance for METAMEETRCSSSUME on the merged dataset.
35

---

Published as a conference paper at ICLR 2025
Reward Modeling
1.0
FsfariX-LLMMA3-RMM011...
0.00
0.06
0.08
GRM-Gem Denmma--Brreaardmddde-f all
0.16
0.16
0.15
0.21
0.8
GeM-Laama--8Breaaaddddeee-.....
0.20
0.32
0.09
0.26
0.10
RM-Mistral-7B. all
0.00
0.06
0.07
0.6
Metrics
Skywork-Rewward-Laees1--88 all
1.00
0.93
0.25
0.63
0.28
0.46
URM-LLaMa-3-88. all
0.00
0.04
0.4
URM-LLaMB-3.1-88.22 all
0.14
0.22
0.24
0.33
0.2
internim211 (hermim2-1_B-reewarrd_.. all
0.03
internlm2-7b-rewarrd.. all
0.47
0.48
0.08
0.11
0.09
GP-ALL ALL
XGB ALL
XGB Iter Best
GP Top
XGB Top
XGB Iter Top
Figure 11: Weights and Feature Importance for METAMETRICS-RM
36


Reviews:
[{"review_id": "FVqp6IifWe", "reviewer": "Reviewer_4VeV", "summary": "This paper proposed a method to align the automatic evaluation metrics with human preference evaluation, called MetaMetrics. Specifically, it's optimized to learn a set of weights that can combine automatic metrics to mimic human preference. Two methods are discussed and experimented for the optimization. The experiments cover both text and multimodal evaluation, demonstrating a satisfactory performance of the proposed method.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research problem this paper studied is very interesting and useful. Automatic evaluation metrics can cover a wide range of benchmarks. However, they may not align with human evaluation. Given human evaluation is time-consuming and expensive, and also current model is required to be more generalizable to various tasks, aligning the automatic metrics with human evaluation preference becomes an urgent and important research topic.\n\n2. This paper illustrates the motivation, problem definition, and methods to optimize the proposed MetaMetrics in a clear and easy-to-follow way.\n\n3. The experiments are conducted in a wide range of downstream tasks, from text to multimodal, as well as training a reward model.", "weaknesses": "1. This paper assumes that human preference can be approximated by a weighted combination of metric scores. But it seems to be too simple, and might not always hold. For example, human's evaluation of helpfulness might require both the model to perform well in reasoning and knowledge-retrieving, which we assume are separately measured by two benchmarks. Then a weighted summation of these two benchmarks might not be suitable, instead, a multiplication might be more proper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZNiaMJcxBx", "reviewer": "Reviewer_H1Lp", "summary": "This paper introduces MetaMetrics, a framework designed to calibrate evaluation metrics to better align with human preferences. By leveraging existing metrics that capture different aspects of human preferences, MetaMetrics employs learning-based methods to optimize these metrics for various generative tasks. To demonstrate the effectiveness of MetaMetrics in aligning with human preferences, the authors compute correlations between the calibrated metrics and quantified human ratings. Experiments are conducted across diverse tasks, including Text Summarization, Machine Translation, Question Answering, Image Captioning, and Reward Model Scoring.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a critical problem in the field of text generation: the lack of automatic metrics that accurately reflect human preferences. Current approaches often rely heavily on human annotations to evaluate performance. By proposing a method to calibrate existing metrics to align with human preferences, this work aims to reduce the reliance on labor-intensive human evaluation processes.\n\n- The methodology is simple yet effective. The proposed approach captures the multifaceted nature of human preferences while incurring minimal additional cost, enabling efficient calibration of existing metrics.\n\n- Extensive experiments validate the proposed method. The authors conduct experiments across a variety of tasks, including Text Summarization, Machine Translation, Question Answering, Image Captioning, and Reward Model Scoring. This demonstrates the task-agnostic advantages of the method and its broad applicability.", "weaknesses": "- Empirical and ad hoc nature of MetaMetrics: The calibration process for MetaMetrics relies on empirical and ad hoc weighting of existing metrics for each dataset and human preference. These weights are highly dependent on the specific dataset and task, which raises concerns about their generalizability to a broader range of datasets or tasks.\n\n- Lack of ensembled baselines: While the paper compares MetaMetrics against individual metrics, it does not include ensembled baselines, such as averaged or weighted-average scores of individual metrics. Including these as baselines would provide a fairer comparison and strengthen the evaluation.\n\n- Absence of human studies in experiments: Although MetaMetrics aims to align with human preferences in evaluating model performance, the study does not involve an actual model being evaluated and analyzed through human studies. To comprehensively validate MetaMetrics, experiments involving direct human evaluation of model outputs are necessary.\n\n- Unclear human preference in certain tasks: In some evaluation tasks, the connection between the ratings and human preferences is not adequately explained. For instance, it is unclear how ratings like sys Pearson, seg Pearson, and acc-t in Machine Translation tasks, or NQ, TQ, NarrativeQA, and SemEval in Question Answering tasks, align with human preferences. More detailed explanations of these ratings and their relationship to human preferences are needed, especially for readers unfamiliar with these tasks.\n\n- High computational cost for reward model ensembling: The use of multiple metrics in MetaMetrics can significantly increase evaluation costs, particularly in reward model scoring. For example, in online RLHF training, simultaneously running multiple reward models (e.g., 7 models in Table 4) could make the process computationally infeasible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rWjFZggZoq", "reviewer": "Reviewer_y9ei", "summary": "This paper introduces meta-metrics, which optimizes the combination of existing metrics to enhance their alignment with human preference. The paper investigated mainly two method GP and XGBoost to try to learn a best weight vector for a set of base metrics to combine. For different datasets, they select different base metrics and train a new weight vector for combinations. Results show that meta-metrics can outperform existing baseline metrics in summarization, translation, question answering, image captioning and reward modeling scenarios.", "questions": "1. Please provide the training details. From my understanding, when training a combined metametric for reward modeling, you need to run inference on all of the training data using each of the base metrics first, and then train your weight vector to optimize for the human preference. How much GPU hours have you spent on the inference, and the training? How many GPUs did you use for the experiments?\n2. Could you provide a list of base metrics used for each meta-metric variant in one place? I only found the base metrics for reward modeling  in line 474 and it's unclear what's the base metrics used for other benchmarks. if they are also baselines, it's better to mark them in the results table.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a new paradigm to ensemble existing metrics for evaluation of different tasks, learning a weight vector through GP or XGBoost and get the weighted combination of base metrics. This is a novel approach and gets good performance.\n2. The paper is well-motivated and clearly written, with nice figures and tables illustrating the concepts.\n3. Extensive experiments across summarization, machine translation, QA, reward modeling, which covered a wide areas. The authors also provide analysis of the weights distribution across the base metrics. Providing many insights.", "weaknesses": "1. The performance gain is relatively small compared to base metrics, especially for machine translation and reward modeling tasks, it's unclear whether these marginal performance gains are worth the training cost.\n2. The proposed approach requires training a new combination weight vector once the base metrics have changed, thus limiting the usability of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EaCaAirwcm", "reviewer": "Reviewer_HGah", "summary": "This paper introduces Meta-Metrics, a calibrated metric for evaluating generation tasks like summarization, QA, Machine Translation and Image Captioning,etc. It raises the issue of metrics excelling at one dimension and falling at others while evaluating on multiple dimensions (like coherence, fluency, relevance,etc in summeval), hence proposing a calibrated metric based on these several metrics, aiming to achieve a higher alignment with human preferences in the evaluation scores. The authors propose using methods like Bayesian Optimization, Boosting to effectively choose/filter the various metrics and give a combined metric to have higher correlation with human scores.", "questions": "1. It is very good that the authors have shown their method on multiple datasets, but some more experiments/analysis on the proposed methods would strengthen the paper more. One such thing can be, the authors used Kendall-Tau as the objective calibration function. How were the results when spearman correlation is used as a calibration function?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents innovative mathematical approaches of using BO modeled as Gaussian Process, and XG Boosting on existing metrics to create a new metric.\n2. The paper has tested their method extensively on several datasets across multiple domains.", "weaknesses": "1. Although the metametrics have shown they have higher alignment with human preferences, the optimization methods (BO and Boosting) are to be considered black boxes and offer very less explainability.  Understanding how these methods find the combination of metrics and interactions between different metrics within the model is not possible. Hence explanation on why they correlate more with human scores is not there.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xPXcmzYmgZ", "reviewer": "Reviewer_nKfe", "summary": "This paper introduces a novel evaluation metric, MetaMetric, designed to better align with human preferences. The proposed metric is a weighted combination of existing metrics, where the weights are learned in a supervised manner. The authors demonstrate the effectiveness of MetaMetric through experiments on various natural language generation tasks, e.g., summarization, translation, question answering and image captioning. Additionally, they show that this metric can be useful for reward model scoring. They also provide some analysis about interpretability and efficiency of their metric.", "questions": "1. **Questions related to Introduction Section:**\n    1. The first paragraph is a bit confusing. The authors discuss the shortcomings of BLEU and BERTScore in the main text, but Figure 1 presents BLEU and CLIP-S. Additionally, what criteria are used in Figure 1 to determine if a metric is good or not? For the figure on the right hand side, why are BLEU 0.2 and MeMetric 0.4 considered good, but CLIP-S 0.6 is considered bad? What is the threshold?\n    2. The authors state, \"models that optimize for human preference ... have demonstrated superior performance.\" What metrics have these models used to demonstrate human alignment? The authors avoid discussing this in their paper, instead comparing their metric with conventional metrics not originally designed for human preference alignment. For example, Ouyang et al. [1] used the Perspective AI score to measure toxicity.\n    3. In line 42, the authors mention \"evaluation metrics that accurately reflect human subjective judgements.\" Since human judgements are **subjective**, is it possible that a metric can **accurately** measure these varied judgements across different individuals?\n    4. In line 53, the statement relating the necessity of human aligned metrics to the complexity of tasks seems implausible. Regardless of task complexity, we always seek metrics that align with human evaluations.\n2. **Universality of MetaMetric:** For conventional metrics (e.g., BLEU, ROUGE, METEOR), they are used across various natural language generation tasks. Is it feasible to have a single MetaMetric applicable to all natural language generation tasks?\n3. **Choice of Training Methods:** Why did the authors choose Bayesian Optimization and Boosting to train MetaMetrics? Is it possible to use linear regression models with regularizations?\n4. **Efficiency of MetaMetric:**. According to Figure 4, MetaMetric requires about 8 metrics for computation. Can the authors explain why computing approximately 8 metrics is still considered efficient?\n5. **Interpretability:** Can the authors explain why some metrics align more closely with human evaluations while others do not? I'm curious whether there is any intuition behind this.\n6. **Compute Efficiency for RM:** In lines 433 and 472, Table 4 does not show compute efficiency. The authors should include the model size for MetaMetrics-RM.\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in neural information processing systems 35 (2022): 27730-27744.\n\n**Minor Issues:**\n1. In line 215, there is a duplicate \"is\"\n2. Presentation suggestion: Section 2 could be merged with Section 7.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experiments are comprehensive, including various types of natural language generation tasks.\n2. The topic of developing a human-aligned evaluation metric for NLP generation tasks is a long-discussed but unsolved challenge. This paper explores a possibility to push forward in this direction.\n3. The proposed metric is easily extendable to other tasks where preliminary metrics have already existed. This extensibility makes the metric potentially applicable to a broader range of applications.", "weaknesses": "I have several concerns regarding the experiments, particularly the scope of the problem and the evaluation settings, and how they support the authors' claims. Additionally, the presentation of the paper requires refinement.\n1. **Limited Contribution:** The paper's contribution appears limited. It essentially addresses a traditional supervised learning problem with input of small dimensions and output with single normalized value. The authors train a regression model on supervised-learning datasets, using conventional metrics as inputs and human preference values as outputs. The experimental results involve testing this regression model on the test set of these datasets and comparing different metrics using correlation of the rankings of metrics with ground truth human preference values. It is unsurprising that MetaMetric outperforms other metrics in the in-domain generalization setting, which is the primary focus of this paper. The authors should also clearly present their experimental settings, such as how they create the training, validation and testing sets. For example, the setting for Question Answering tasks is not clear. It would be interesting to add more OOD and real scenarios to test this metric. See following comments for detailed feedback.\n2. **Lack of Out-of-domain (OOD) Generalization Evaluation:** The authors should place more emphasis on the OOD setting, as human preference data is not always available. I found only one one OOD setting presented in Figure 3. Based on in-domain performance, the GP model works the best. However, the GP model performs similarly to CLIP-S and METEOR in the OOD setting. (The statement about the superiority of MetaMetric-Cap over any other individual metric, in lines 426 and 427, is misleading.) This suggests that the use cases for the trained metric are limited, and its performance may be worse than using a single conventional metric. It would al os be interesting to evaluate metaMetric trained on one task but tested on another, which is a more challenging OOD setting than the one in Figure 3.\n3. **Lack of Evaluation in Real Settings:** The evaluations are limited by the training datasets with human-annotated preference values. It is unclear whether this metric is useful for real applications, as suggested by the authors in lines 41 and 42. For example, the authors could follow a similar evaluation procedure to DPO and PPO papers, using the proposed metric to evaluate various LLMs - before and after RLHF-tuned - and compare their win rates based on MetaMetric.\n4. **Reference-based and Reference-free Settings:** In line 83, the authors claim to design the metric for two settings: reference-based and reference-free. If I understand correctly, this depends on the choice of metrics rather than the model design. If the chosen metrics require references, then MetaMetric can only be reference-based. In the experiments, the authors only distinguish these two settings for Machine Translation, not for other tasks. It would be better that the authors add both settings for all the tasks.\n5. **Presentation Issues:** The paper's presentation is suboptimal. Major issues include:\n    1. The caption of a table should be placed above it.\n    2. Figure 2 is not cited anywhere in the paper.\n    3. The authors cite a table in the Appendix as crucial support for their claims (see line 357 for Table 9). The authors should put the information in the main text.\n    4. Other minor issues are listed in the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)