Faithful Model Evaluation for Model-Based Metrics
Authors: Qian Hu
Palash Goyal
Rahul Gupta
Venue: EMNLP 2023 Main (2023)
Keywords: Evaluation
Model-Based Metrics

Abstract: Statistical significance testing is used in natural language processing (NLP) to determine whether the results of a study or experiment are likely to be due to chance or if they reflect a genuine relationship. A key step in significance testing is the estimation of confidence interval which is a function of sample variance. Sample variance calculation is straightforward when evaluating against ground truth. However, in many cases, a metric model is often used for evaluation. For example, to compare toxicity of two large language models, a toxicity classifier is used for evaluation. Existing works usually do not consider the variance change due to metric model errors, which can lead to wrong conclusions. In this work, we establish the mathematical foundation of significance testing for model-based metrics. With experiments on public benchmark datasets and a production system, we show that considering metric model errors to calculate sample variances for model-based metrics changes the conclusions in certain experiments.

Faithful Model Evaluation for Model-Based Metries
Palash Goyal", Qian Hu', Rahul Gupta
(palashg, huqia, gupra) @ @mmazon.com
Amazon Alexa AI
Abstract
based metrics evaluate the outputs of NLP model using another machine learning model such as using toxicity classifier to evaluate the toxicity of the generated texts by text generation model, while deterministic metrics evaluate the outputs of NLP model using annotated ground truth In significance testing, computing the confidence in- terval plays pivotal role in reaching precise clusions. The computation of this interval relies on the sample variance, which differs depending on whether deterministic metrics or model-based met- rics are used. In the case of deterministic metrics, the sample variance corresponds to the variance of the collected samples. However, for model-based metrics, where results are predicted by machine learning models, the sample variance is influenced by the model's prediction errors. Existing works using model-based metrics for model evaluation do not consider prediction errors for significance test- ing, risking inaccurate conclusions. this work, we establish the mathematical foundation of signif- icance testing for model-based metrics.
Statistical significance testing is used in nat- ural language processing (NLP) to determine whether the results a study or experiment are likely to be due to chance or if they reflect genuine relationship. A key step in significance testing is the estimation of confidence interval which is function of sample variance. Sample variance calculation straightforward when evaluating against ground truth. However, in many cases, a metric model is often used for evaluation. For example, compare toxicity of two large language models, toxicity classi- fier is used for evaluation. Existing works usu- ally do not consider the variance change due metric model errors, which can lead to wrong conclusions. In this work, we establish mathematical foundation of significance test- ing for model-basee metrics. With experiments on public benchmark datasets and produc- tion system, we show that considering metric model errors to calculate sample variances for modell-aaeed metries changes the conclusions in certain experiments.
We conduct several experiments model-based metrics including hate speech detec- tion (Hartvigsen al., 2022) and user perceived de- fects detection. The experimental results show that considering prediction errors in significance test- ing changes the conclusions in certain experiments. Thus, we propose that the research community uti- lizes our framework for performing statistical test- ing with model-based metrics. In the following sections, we derive the mathematical equations for significance testing for model-based metrics and conduct experiments on several public benchmark datasets and production system. Finally, we dis- cuss related works and draw final conclusion.
using
1 Introduction
In the field of natural language processing (NLP), continuous progress hinges upon the development of novel techniques that outperform existing ones. However, accurately assessing the effectiveness these new techniques requires a comprehensive evaluation framework. Model evaluation serves as the foundation for assessing the performance and impact of NLP advancements. Significance testing is a crucial tool in the evaluation process, enabling to derive accurate conclusions. It allows to determine whether the obtained evaluation results hold significance or are merely coincidental.
As the cost of human annotation for evaluating models using deterministic metrics substantial, there is a growing trend towards utilizing model- based metrics for evaluation purposes. Model- "These authors contributed equally to this work.
2 Model Evaluation with Model-Based Metries
In this section, we provide background on signifi- cance testing and then derive how we can modify

---

it incorporate model's prediction errors when using model-based metrics.
symmetric confidence interval does contain 0 (Smithson, 2003).
2.1 Background Significance Testing
(Sa (Ja-2qV
2 / (d), 2q (drP(d)) (8)
Significance testing statistical analysis used to estimate the relationship between two statistical variables. When evaluating two models, we want to know if the performance of the two models is significantly different. In this work, we assume the model evaluation is binary classification task such whether the classified domain is correct not in domain classification task, or the generated text is toxic or non-toxt ton-toxic in toxicity classification task. Given two models C and T, their outputs are evaluated using a deterministic metric. The evaluation results are as following,
where 29 is the critical value and a is the confi- dence level, Sa = JT Sc. (arP(d) Var C)) VCr Var (for the case C and T dependent, the formula is derived in Ap- pendix A.3). For 95% confidence level, 2g 1.96.
2.2 Significance Testing with Model-based Metries
For model-based metrics, the performance of mod- els C and T are evaluated by a metric model M, which is usually a statistical model with prediction errors. Thus, the sample variances calculated by (1) Equation 5 and 6 are the variances of the observed evaluation values instead of the true evaluation val- ues. In this section, we derive the sample variance considering the prediction errors.
Jo. SC.Nc:
and
JI,1 JI. JT.
(2)
where fc., fr. (0, 13 are the evaluation results for outputs generated by models C' and T respec- tively; Nc and Ny the number of samples used evaluate models C and T. Their performance is estimated as the mean of the results,
Note that the following equations apply to both models C and T. Assume we have N indepen- dent and identically distributed (IID) samples of evaluations, let N be the random variable denot- ing the number observed positive samples. As we assume a binary classification task, each obser- (3) vation observes Bernoulli distribution. Therefore, the random variable No observes binomial distri- bution with success probability pd (which can be estimated by using Equation 3 or 4). Thus, have
Ne Dioeca fca fc Ne
and
Nr E fra
JT =
(4)
Nr
No Bin(N.p
(9)
The variances of the mean of evaluation results for C' and T using deterministic metric are
We aim to estimate the variance of distribution for the real positive samples, N ~ Bin(N, RR). Towards this goal, we derive the probability p
1 Σ
Var (C) No(No - 1) i=1
(faa fc)², (5)
P(R 1) following
T
p P 1/0 1)P(O
Var (T))
(N Σ
(JT.- fr) (6)
+ +
- 1)0)=1000)P((0 = 0) (10)
1=1
-0000 + 100 or
respectively, where D represents deterministic met- ric.
where n10 and 100' are precision and false omis- sion rate, respectively, which be estimated from the metric model M's performance on its testing data. The variance of binomial distribu- tion is Np(1 - p). therefore, the variance of N Var (N is
We want to know if their performance is signifi- cantly different, which is formally stated as null hypothesis Hp and an alternate hypothesis Ha:
Ho : Je = Fr. Ha fc # fr
(7)
According to the central limit theorem, two sample means are statistically different if the following
N nO +p 110 100' p. 11- nO - MO
(11)

---

where M represents model- metric. The variance of the model performance is
et al., 2021) manually curated dataset for bias measurement in open-ended language generation, which consists of 23,679 English text generation prompts for bias benchmarking in five domains in- cluding profession, gender, race, religion, and polit- ical idcology. RealToxicityPrompts (Gehman et al., 2020) has 100K naturally occurring, sentence-level prompts extracted from a large corpus of English web text.
N
Var (NT)
pd (1 p*)
Var
N
(12)
Since the population mean is unknown and vari- ance is estimated with sampled mean p the above estimator a biased estimation. The corrected un- biased estimation using Bessel's correction (So, 2008) to account for the decreased degree of free- dom is
3.1.1 Result Analysis
Table shows the experimental results. the ta- blc, we show average toxicity score, average treat- ment effect (ATE), variance and confidence interval. ATE is calculated as the difference between aver- age toxicity score of the two models, specifically, it is the average toxicity score of GPT-Neo subtracted by the average toxicity score of GPT2 (we consider GPT2 as the baseline).
N+ N
p²
(1
pd)
Var
M
(13)
Therefore, the 95% confidence interval for model- based metrics is
(fa - 1.96V Var Mar)(d), (d), 1.9966 Var (d)),
(14)
where =a= fc. Var =
From the table, we can see that there is sig- nificant increase in variance when consider the metric model errors. For example, on BOLD dataset, the variance of GPT2 changes from 1.92e-7 to 7.50e-6 (39x increase in variance). Disregard- ing the metric model errors, the confidence interval is (-0.00325, -0.00114 leading to the conclusion that we can reject the null hypothesis and reach- ing the conclusion that GPT-Neo produces output with significant lower toxicity than GPT2. How- ever, when we consider the metric model errors, the confidence interval is (-0.00978, 0.00538), which shows insignificant difference and we cannot reject null hypothesis. In this case, considering metric model errors changes the final conclusion. On Re- alToxicityPromppe dataset, we also see big differ- ence in variance change, but the conclusion is not changed.
Var (C) + Var Note that if metric model is perfect, the variance and confidence interval becomes the same as equations 5, 6 and 8. For proof, Appendix A.1. Note that the formula can be easily extended to multi-class case (see Ap- pendix A.2).
3 Experiments
We perform several experiments on public bench- mark datasets and a production system to validate the proposed framework. In this section, we first introduce the experimental details on public bench- mark datasets and then describe the experiments on the production system. Finally, we report the experimental results and analysis.
3.1 Experiments on Public Datasets
We select toxicity detection in natural language generation the base task. The goal of this task is to detect if the generated text is toxic using toxicity classifier. We adopt a state-of-the-art toxic- ity classifier, RoxiRnaaTTxiien (Hartvigsen et al., 2022) to detect toxicity in the generated text. We estimate precision and false omission rate (FOR) of this classifier on the manually annotated test set from ToxiGen. The estimated precision and FOR are 0.8897 and 0.22769, respectively.
3.2 Experiments in Production System
Besides conducting experiments on public models and benchmark datasets, we also perform experi- ments on live traffic a production system of a lead voice agent. The task is natural language un- derstanding such as domain classification, intent classification, etc. We compare the performance of two NLU models estimated by machine learning model based customer utterances and system responses (Gupta et al., 2021). The precision and FOR of the metric model are estimated on manually annotated datasets. The dataset used for experiment is de-identified.
We compare two generation models GPT2 (Radford al., 2019) and GPT-Nco (Black al., 2021). To generate the text, we utilize prompts from BOLD (Dhamala et al., 2021) and Roaico tyPrompts (Gehman et al., 2020). BOLD (Dhamala

---

Table 1: Experiment Results on Public Benchmark Datasets.
Dataset
Meangerz
MN
Var aP12
CP
Var aP12
GPINeo
BOLD 0,00456
0.00236
-0,00219
1.92--7
9,97e-8
(-0003255, -0,01144)
7.50c-6
7.46-66
1-0.00978, 0.00538)
Real RealToxxicity Prompes
0.09124
0.09157
0.00333
8.34p-7
8.37e-7
(-0.00220, 0.00286)
2.06247e-6
2.0G54056-6
1-0.00655 0.0043))
In the table. Meanarr means the average toxicity score of GPT2 model. Similarly, Meangamenn the average taxicity scure of GPT Neo model.
Var means the variance of of GPT2 model using deterinistic metric. Var"apm means variance the GPT2 model using model hased metric. C1P mcans confidence interval using deterministic metric, and C1 miaas conlidence interval using model-based metrie.
Table 2: Experiment Results in Production System.
Experiment
Varoc
Var²
CIP
Var
Var"
CPM
Expl
7.29e-5
1.0le-4
(-0.05291,
10.00116)
1.82c-4
2.04e-4
(-0.06556, 0.01148)
Exp2
1.03e-5
1.00e-5
(0.00007, 0.01777)
1,07e-5
1.05e-5
(-0.00010, 0.01794)
Exp3
4.93e-9
3.92c-9
(0.00009, 0.00047)
1.64-88
1.63--8
(-0.00007, 0.00063)
Exp4
2.14e-8
2.11e-8
(0.00006, 0.00086)
6.77e-8
6.64e-8
(-0.00026, 1.00118)
Exp5
8.84e-9
8.84e-9
(0.000004, 0.00053)
1.19e-8
1.19e-8
(-0.00004, 0,00006)
The notation is similar in Table 1.
3.2.1 Result Analysis
the right conclusion when conducting significance testing for model-based metrics.
Table 2 shows the experimental results in produc- tion system. The experiments are conducted on dif- ferent NLU domains and devices. From the results, we can see that considering metric model errors has a big impact on variance estimation and also changes the final conclusion of the experiments.
Conclusion
Significance testing is an important tool for us draw accurate conclusions for evaluating NLP mod- els. Existing evaluation works using model-based metrics do not consider model variance for signifi- cance testing, which can lead to wrong conclusions. In this work, we lay the mathematical foundation of significance testing for model-based metrics. We conduct experiments on public benchmarks and production system. The significance testing results on these experiments show that model based errors need to be considered and incorporated for accurate evaluation.
4 Related Work
In their paper, Dror et al. (Dror al., 2018) estab- lished fundamental concepts of significance testing in NLP research and proposed a simple practical protocol for statistical significance test selection in NLP setups. Bang-Kirkpatrick et al. (Berg- Kirkpatrick et al., 2012) investigate the relation between significance level and the magnitude of the gain. They also studied how the standard i.i.d. notion of significance hold up when there is domain shift. With the increasing of evaluation datasets, is becoming common evaluate NLP models on multiple datasets. Such multiple comparison poses challenges for significance testing in NLP. Dror et al. (Dror et al., 2017) proposes replicability analy- sis framework for the analysis of multiple compar- isons between NLP algorithms. In their book, Dror et al. (Simpson, 2021) discuss the opportunities and challenges of using significance testing in NLP. The book covers multiple topics including choos- ing the appropriate significance test for NLP task, dealing with the uniqueness of significance testing for non-convex deep neural networks, etc.
In this work, focus primarily on comput- ing confidence interval with model-based metrics which use binary classification. In the future, we plan to extend our work to more general types of model-based metrics. Further, we assumed that the samples are independent and identically distributed. In practice, we often have a score associated with the metric model which can be used to relax this assumption. We leave this as future work.
Limitations
This work mainly focuses on significance testing of binary categorical metrics and two sample t- test. We do not explore other types of metrics and statistical tests. We leave them to future work.
The current works on significance testing in NLP do not cover the area of model-based metrics. In this work, we propose to consider model prediction errors for more faithful evaluation and reaching
References
Taylor Berg-Kirkpatrick. David Burkett, and Dan Klein. 2012. empirical investigation of statistical signifi- cance in nip. In Proceedings of the 2012 Joint Con-

---

ference Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995-1005.
A
Appendix
A.1
Sample Variance When the Metric Model is Perfect
Sid Black, Leo Gao, Phil Wang. Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow. you use this software, please cite using these metadata.
Equation 13 is the variance when considering met- ric model prediction errors. When the metric model does not make errors, the precision and false omission rate is and 0, respectively. Therefore. R and paIO into Equation 10, we get p = =pO Therefore.
Substituting the values
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishma, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM conference on fairness, accountability and Iransparency, pages 862-872.
N N
p0 (1-p9) p0) N
Var
(15)
Next, we prove that Equation 15 equals Equation
Rotem Dror, Gili Baumer, Marina Bogomolov, and Roi Reichart. 2017. Replicability analysis for natural language processing; Testing significance with mul- tiple datasets. Transactions the Association for Computational Linguistics, 5:471-486.
5. For Equation 5, we denote the number of 1s as
Ni. the total number of samples as Nc, and p Ne fca No following
Equation 5 can be rewritten as No
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume Long papers), pages 1383-1392.
Var (CC
NG
1
Σ
(So. - - Jc}²
No(No 1) i=1
E1c,=1
p0)² E1c
Samuel Gehman, Suchin Gururangan, Maarten Sap. Yejin Choi, and Noah A Smith. 2020. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. arXiv preprint 14X1v:2009.1166.
Nc 100-1) 1)
Ni 2 (No- N1)
(16)
Nc 1)
N1 2NI + Nolp0p2
Saurabh Gupta, Xing Fan, Derek Liu, Benjamin Yao, Yuan Ling, Kun Zhou, Tuan-Hung Pham, and Edward Guo. 2021. Robertaiq: An efficient framework for automatic interaction quality estimation of dialogue systems. In KDD 202/ Workshop on Data- -fffiiffc Machine Learning.
No
p NC 2Nc(1))2 + Nc(p
No -11
p (1-1-))
No N
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap. Dipankar Ray, and Ece Kamar, 2022. Toxigen: A large-scale machine-generater dataset for implicit and adversarial hate speech detection. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.
Therefore, Equation 15 equals Equation 5 and 6. So Equation 13 is generalized version of Equation 5 and 6, when considering metric model prediction errors.
Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark. Miles Brundage. and Ilya Sutskever. 2019. Better language models and their implications. OpenAl Blog https://openai. com/blog/bettter htwp:boogbbetter-annuug---e language-modele 1(2).
A.2 Application to Multi-Class Use Case
To apply the proposed approach to multi-class use case, we need to derive d for multi-class classifi- cation tasks, which is as following
Edwin Simpson. 2021. Statistical significance testing for natural language processing.
p
Σ
P(R 110 o;) (17)
Michael Smithson. 2003. Confidence intervals. 140. Sage. Confidence Statements and Interval Estimates.
0:C (o).
Stephen 2008. Why the sample variance biased estimator? Griffith University, Tech. Rep., 9.
where o; is the i-th class observation value. The variance of C and can derived using n".

---

A.3 Variance When C and T are Dependent this section, we derive the variance of the difference for model-based metrics when C' and T are dependent. When C' and T are depen- dent, Var (d) - Var (T - C) - Var (C) + Var + 2Cov (C,T), where Cov (C, is the covariance between C' and T. In the following, we derive Cov (C,T).
Section 2.2) and P =z - y) can be esti- mated empirically. Using Bessel's correction, we get
Cov (C,T)
1
Σ
P(fe 11f8 x)
N -
(C(0
P P = 1159 - v) P(Je = H, 14 = 4)
Cov CO,
Cov M (fc, (fc, fT)
R 1 (PÊPT)
N (Elfe: ffl -
(20)
N Σ zy 2E(0.1):EE0013 PU 10 y) /
(P 11,6 1)-
PCPT),
(18)
where J8 and f are the real values of a sample from C and T, respectively.
The first term can be converted as following:
P(J = 11,0=11 =1)
Σ
= 1, -1, 118
(E(0))))E((0113
Σ
P(fe =
= y)
(0.1) (0.1)
P(f x, 10 y)
Assuming conditional independence of fl and f given f6 I, fo y, we have the following.
Σ
P 1 = x. 10 = y)*
(0.1
P( = 1118 =x =)) u
P(Je = x, 12 = v)
Σ
=E(0.1),.E(0.1)
PUJ 1118 x2)P( = 1 -)
PUB = x, 14 =))
(19)
where P(J = 1158 = x). P( 118 are model performance metrics and can be esti- mated on a held-out dataset (similar to eq. 10 in


Reviews:
[{"review_id": "sqMDHV9V15", "reviewer": "Reviewer_ZfFZ", "summary": "", "questions": "I don't have further questions besides asking the authors to respond to the previous section.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for the review and providing the suggestions that can improve the paper quality. Given the feedback, there are two concerns and a minor question. We address them below.\n\n1. Only tested two datasets and a few models. It would also benefit greatly from having model-based metrics with different levels of variance.\n\n    **Only tested two datasets and a few models**\n    > Our goal in the paper is twofold: 1) raise awareness in the community that using metric based models can lead to inaccurate conclusions, 2) establishing a mathematical foundation of significance testing when using metric based models. Our experiments on the two data sets covering two different tasks (toxicity detection and domain/intent classification) and two different systems (public benchmark and a production system) show that this issue is present and that using the proposed modifications in the paper we can correct these issues. Although performing experiments on more datasets would show further applicability, respectfully, we believe experiments on the selected datasets demonstrate the key point of the paper. \n\n    **Different levels of variance**\n    > The experimental results show that the variance of the model-based metrics spans a wide range. For example, in the production system, the variances range from as low as 1.19e-8 to as high as 2.04e-4.\n\n2. Only for binary classification. I concede that the authors acknowledged this point in the limitations section, but it's a bit odd given that section 2.1 is set up for a more general scenario.\n\n    > The proposed approach is generally applicable to multi-class classification if the classes are independent of each other. In the paper, for simplicity and brevity, we show the derivation for a binary classification use case. We derived the formula for multi-class use case as the following and we’ll add it to the camera-ready.\n    \n    > ---\n    >To apply the proposed approach to multi-class use case, we need to derive $p^R$ for multi-class classification tasks, which is as following\n    > \\begin{equation}\n    > \\begin{split}\n    > p^R = P(R=1) = \\sum_{o_i \\in \\{ 0, 1, ..., N_O \\} } P(R=1|O=o_i) P(O=o_i)\n    > \\end{split}\n    > \\end{equation}\n    > where $o_i$ is the $i$-th class observation value. The variance of C and T can be derived using $p^R$.\n\n3. In Table 2, the superscripts seem to be inconsistent with previous notations: should've been D and M?\n\n    > Thank you for pointing this typo out. We’ll fix this for the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Very important topic given the current trend of using model-based metrics in more and more places.\n2. Very clearly written and easy to understand. The experiments are solid and to the point.", "reasons_to_reject": "1. Only tested two datasets and a few models. It would also benefit greatly from having model-based metrics with different levels of variance.\n2. Only for binary classification. I concede that the authors acknowledged this point in the limitations section, but it's a bit odd given that section 2.1 is set up for a more general scenario.", "questions_to_authors": "I don't have further questions besides asking the authors to respond to the previous section.", "paper_topic_and_main_contributions": "This paper introduces a significance test for model-based metrics. It lays out the mathematical foundation of this test and applies it to both public and production datasets. In both cases, the conclusions might be changed when the variance from model prediction is taken into consideration.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Table 2, the superscripts seem to be inconsistent with previous notations: should've been D and M?"}, "extra_text": {}}, {"review_id": "R3vg1Ow8cp", "reviewer": "Reviewer_sWNL", "summary": "", "questions": "(1) Is the proposed method suitable for precision, recall and F-score?\n\n\n(2) The paper assumed that the evaluation results of $C$ and $D$ are i.i.d. Is the assumption too strict?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for the review and providing the suggestions that can improve the paper quality. Given the feedback, there is a technical concern on an equation in the paper. We address it below.\n\n1. I consider it is unreasonable to decompose the variance Var^M(d) into the form of  Var^M(C) + VarM(T) because the evaluation results outputted by models  and  are frequently correlated.\n\n    > In the paper, we make the assumption that C and T are independent. This assumption applies to several use cases such as online A/B testing. However, to address reviewer’s concern, we also relax the assumption and derive the covariance formula [here](https://www.dropbox.com/scl/fi/9d200jjqkff7y1sf8dvg4/FaithfulEvaluation_Supplementary.pdf?dl=0&rlkey=a8atyf12rt17n5zntkhfrhlnu). We will include this as an appendix in the camera-ready version. Note that we make a weaker conditional independence assumption that $f_C^R$ is independent of $f_T^R$ given $f_C^O$ and $f_T^O$. This assumption is weaker since we are not assuming that $C$ and $T$ are independent but if we know the observed values of both, the real values can be independently derived.\n\n2. Moreover, I don't think the proposed method is suitable for the commonly used metrics, such as precision, recall, F-score.\n\n    > Similar to accuracy, to calculate the variance for precision and recall, we can use the derived $p^R$. Therefore F-1 score, as we can derive F-1 score from precision and recall.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is interesting. The mathematical expressions are straightforward and effective. The experimental results demonstrated the reasonability of the variance estimation proposed in the paper.", "reasons_to_reject": "I consider it is unreasonable to decompose the variance $Var^M(d)$ into the form of $Var^M(C)+Var^M(T)$ because the evaluation results outputted by models $C$ and $T$ are frequently correlated. Moreover, I don't think the proposed method is suitable for the commonly used metrics, such as precision, recall, F-score.", "questions_to_authors": "(1) Is the proposed method suitable for precision, recall and F-score?\n\n\n(2) The paper assumed that the evaluation results of $C$ and $D$ are i.i.d. Is the assumption too strict?", "paper_topic_and_main_contributions": "The paper provided a novel estimation method with regard to the variance of the evaluation results in model-based metrics and proposed a novel significance testing method based on the proposed variance estimator coupled with the two sample z test for comparing two NLP models according to model-based metrics.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None", "Justification_for_Ethical_Concerns": "None"}, "extra_text": {}}, {"review_id": "aNa7YTag2D", "reviewer": "Reviewer_eZXe", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for the review and providing the suggestions that can improve the paper quality. Given the feedback, there is one concern. We address it below.\n\n1. The proposed approach is parametrized by the evaluation model precision and false omission rate. However, those parameters are estimated with in-domain data, i.e., data from the same distribution as the data used for training the evaluation model. This may imply that the model performance on out-of-domain examples is not well estimated and therefore affect the variance estimation.\n\n    > We mentioned in the paper that the evaluation model precision and false omission rate are estimated on the evaluation dataset, not on the training data. Out-of-domain estimation is not considered in the paper. We can consider it as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper addresses a prevalent issue related to recent evaluations: evaluations based on another LMs may not be as accurate as using a gold standard.\n* The proposed solutions seem a good drop-in replacement for the actual significant test.  \n", "reasons_to_reject": "* The proposed approach is parametrized by the evaluation model precision and false omission rate. However, those parameters are estimated with in-domain data, i.e., data from the same distribution as the data used for training the evaluation model. This may imply that the model performance on out-of-domain examples is not well estimated and therefore affect the variance estimation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Statistical significance tests are used to determine whether the results of a study or experiment are likely to be due to chance or if they reflect a genuine relationship. However, in the recent years many experiments are evaluated using machine learning models. Those models introduce errors in the evaluations, and therefore, affect the confidence intervals that could derive to wrong conclusions. This paper addresses this problem and proposes a solution to take into consideration the errors produced by the evaluation model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a significance test for model-based metrics. This topic is crucial in NLP given that many experiments are evaluated using machine learning models which can errors in the evaluations, and  affect the confidence intervals that could lead to wrong conclusions. 
The paper is clear and well written. While the experiments are conducted only on few tasks and binary classification tasks, a generalization of the formula is proposed that could applied to multi-class classification tasks.
Decision: Accept-Main