Statistical Hypothesis Testing for Auditing Robustness in Language Models
Authors: Paulius Rauba
Qiyao Wei
Mihaela van der Schaar
Venue: ICML 2025 poster (2025)
Keywords: language models
safety
interpretability
reliability
Primary area: general_machine_learning->everything_else
TL;DR: We develop a statitical hypothesis testing method to quantify the impact of a perturbation in the input prompt on the outputs of language models

Abstract: Consider the problem of testing whether the outputs of a large language model (LLM) system change under an arbitrary intervention, such as an input perturbation or changing the model variant. We cannot simply compare two LLM outputs since they might differ due to the stochastic nature of the system, nor can we compare the entire output distribution due to computational intractability. While existing methods for analyzing text-based outputs exist, they focus on fundamentally different problems, such as measuring bias or fairness. To this end, we introduce distribution-based perturbation analysis, a framework that reformulates LLM perturbation analysis as a frequentist hypothesis testing problem. We construct empirical null and alternative output distributions within a low-dimensional semantic similarity space via Monte Carlo sampling, enabling tractable inference without restrictive distributional assumptions. The framework is (i) model-agnostic, (ii) supports the evaluation of arbitrary input perturbations on any black-box LLM,  (iii) yields interpretable p-values; (iv) supports multiple perturbations via controlled error rates; and (v) provides scalar effect sizes. We demonstrate the usefulness of the framework across multiple case studies, showing how we can quantify response changes, measure true/false positive rates, and evaluate alignment with reference models. Above all, we see this as a reliable frequentist hypothesis testing framework for LLM auditing.

Statistical Hypothesis Testing for Auditing Robustness in Language Models
Paulius Rauba' Qiyao Wei
Mihaela van der Schaar'
Abstract
inherent variability due to the stochastic sampling process, a process controlled via parameters such as temperature or top- k (Renze & Guven, 2024). This means that evaluating how interventions -ucch as input perturbations, model variant changes, prompt system modifications- affect the output, is not straightforward (Romero- -llarrddo et al., 2024). Under- standing and quantifying the effects of such perturbations crucial in high-stakes applications- as legal document drafting or medical diagnosis- diagnosis-where behavior could have significant consequences (Meskó & Topol, 2023; Helberger et al., 2023).
Consider the problem of testing whether the out- puts of a large language model (LLM) system change under an arbitrary intervention, such as an input perturbation or changing the model vari- ant. We cannot simply compare two LLM out- puts since they might differ due to the stochastic nature of the system, nor can we compare the entire output distribution due to computational intractability. While existing methods for analyz- ing text-based outputs exist, they focus on fun- damentally different problems, such as measur- ing bias or fairness. To this end, we introduce distribution- perturbation analysis, a frame- work that reformulates LLM perturbation analysis as a frequentist hypothesis testing problem. We construct empirical null and alternative output dis- tributions within a low-dimensional semantic sim- ilarity space via Monte Carlo sampling, enabling tractable inference without restrictive distribu- tional assumptions. The framework is (i) model- agnostic, (ii) supports the evaluation of arbitrary input perturbations on any black-box box LLM, (iii) yields interpretable p-values; (iv) supports multi- ple perturbations via controlled error rates; and (v) provides scalar effect sizes. We demonstrate the usefulness of the framework across multiple case studies, showing how we can quantify response changes, measure true/false positive rates, and evaluate alignment with reference models. Above all, we see this as a reliable frequentist hypothesis testing framework for LLM auditing.
where errors or unintended
Systematic evaluation of output responses to input pertur- bation is fundamental to comprehending LLM behavior, provides quantitative insights into model robustness and output consistency across diverse input conditions, Statisti- cal hypothesis testing of model responses can be used for quantifying these effects and isolating the effects of ran- dom variability. There are multiple examples where this is directly relevant. First, it helps with vulnerability identifica- tion by quantifying potential vulnerabilities to adversarial attacks (Goodfellow et al., 2014). Second, it aids with bias discovery, whereby latent biases or unintended behaviors may not become apparent through lingle-dimnnson audit- ing approaches (Ribeiro et al., 2016). Third, it can work within a compliance framework. Measurable frameworks for assessing model behavior are essential for compliance with emerging ethical and legal accountability regulations (Doshi Velez et al., 2017). In the context of language mod- els, subtle changes in patient history could lead to wildly different diagnoses; and patients with nearly identical health records might receive drastically different treatment plans due to minor grammatical changes. Given these critical functions, there is a clear need for a comprehensive LLM auditing framework centered around reliable statistics tools.
1. Introduction
Current methods for analyzing LLM behavior changes of- ten focus on simplistic metrics, such as word overlap or direct log-probability comparisons. While effective in cer- tain cases, these approaches fail to account for the nuanced, nigh-dimensional nature of semantic information processed by LLMs (Bhandari et al., 2020; Lin, 2004). Moreover, existing methods typically lack rigorous statistical founda- tions, making it difficult to disentangle meaningful changes in model behavior from intrinsic randomness in the output generation process. Efforts to address this have included
Large language models (LLMs) generate outputs condi- tioned on textual inputs by iteratively sampling from a dis- tribution of tokens. Therefore, the outputs of LLMS exhibit
University Cambridge. Correspondence to: Paulius Rauba <pr501 @ uck.e
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Statistical Hypothesis Testing in Language Models
specialized attribution methods, feature importance tech- niques, and counterfactual reasoning frameworks (Ribeiro et al., 2016; Garg et al., 2019). However, these approaches are often model-specific rely on restrictive assumptions about the data or model, or fail to provide interpretable and generalizable metrics.
X + X, a system perturbation Ag S + S, at input x X, is characterized by a non-negative discrepancy measure d between the output distributions S(x) and S(A, (z)), or S(x) and (r)) respectively.
We aim to perform statistical hypothesis testing on black- box LLMs S with respect to the system and input. Therefore, (i) we do not assume access to the underlying system's parameters, such their weights or biases; and (ii) make assumptions about access to ground-truth labels, as the evaluation is done with respect to the generated outputs of the system.
Our solution. In this work, we introduce distribution bassed perturbation analysis (DBPA), a framework that reformu- lates the problem of LLM perturbation analysis as a fre- quentist hypothesis testing task. DBPA constructs empirical output distributions using Monte Carlo sampling to cap- ture the inherent stochasticity of LLMs, and enables sta- tistical hypothesis testing of perturbation effects within a low-dimensional semantic similarity space. By using statis- tical hypothesis testing, the framework enables robust, inter- pretable inferences about whether and how input perturba- tions meaningfully influence LLM outputs. DBPA is model- agnostic, computationally efficient, and flexible enough to accommodate arbitrary perturbations on any black-box LLM. It also provides interpretable p-values, scalar effect sizes, and supports multiple testing with controlled error rates, making a versatile tool for post-hoc interpretability and reliability assessments of LLMs.
2.2. Distribution testing as frequentist hypothesis-testing problem
The challenge of assessing the sensitivity of S can be re- framed as a frequentist hypothesis testing problem. A com- mon approach involves querying a language model once to obtain an answer, modifying the input, querying again, and comparing the outputs:
y S(x),, V'~S(x')
This approach suffers from high variance and therefore low statistical efficiency due to the stochastic nature of LLM sys- tems. Since S(z) generates random variable for any fixed input x. both and y' represent single realizations from a distribution of outputs. Any observed difference between and y' could therefore arise from inherent randomness rather than a true effect of Az.
Contributions. ! We identify limitations in exist- ing methods for evaluating language model outputs (Sec. 2). 2 We introduce iistribution-based pertur- bation analysis which is a model-agnostic sensitivity technique that can test the effect of any perturba- tion with statistical significance measures (Sec. 3). 3 We perform multiple case studies to show the usefulness of DBPA (Sec. 4).
To address this limitation, we propose reframing the prob- lem from the lens of distribution testing. Instead of com- paring individual outputs, we would idcally like to compare the entire output distributions. For notational convenience, let D2 := S(x) denote the output distribution for input z. Let Y, and Y,,' be random variables where Y, ~ D,, and Y Dz.
2. An analysis of viewing LLM outputs through frequentist hypothesis testing
2.1. Problem formulation
Definition 2.2 (Output Distribution). For a given input e X' and an LLM S X' P()'), the output distribution D is the probability distribution from which the random variable Y, is drawn when generated by S(x). That is, Ya ~ Dr. The probability mass function is given by:
Let X' denote the input space and y the output space of a machine learning system. We define the system as a stochastic mapping S X + P()). where P()) is the space of probability distributions over y. This formulation captures the inherent stochasticity of modern ML systems, including LLMs.
px(v) = P(Y, = v). Vy E V
(2)
Our objective is to address the following research question: Given an ML system S, an input x E X, and an input perturbation Az, how can we systematically measure and interpret the impact of A₂ on the output distribution of S(x) under general notion of sensitivity?
Our goal is to determine whether these distributions differ significantly. This can be formulated as a hypothesis test:
Ho Dr = =D,r (The perturbation has no effect)
(3)
H1 DD Dz
(The perturbation has an effect)
(4)
Definition 2.1 (Sensitivity).. The sensitivity of machine learning system S with respect an input perturbation A.
Here, x' := Ax() is shorthand for the perturbed imput. The primary benefit of such a distributional formulation that it
"oode can be found at https::/gtthuu.coom// http://ghhh...ommmaaaeeerrarraa
2

---

Statistical Hypothesis Testing in Language Models
captures the full stochastic behavior of S instead of just single realization. This means we could perform statistical inference by directly comparing these distributions and un- derstanding how much the outputs have shifted across the whole output space, such detecting subtle shifts that might not be apparent from individual samples.
ferent. We show how achieve both with finite-sample approximations.
Takeaway. Analyzing output distributions of lan- guage models faces two unique challenges: the com- putational intractability due to the enormous out- put space, and the need for semantic rather than just probabilistic interpretation of differences.
Takeaway. To understand the sensitivity of a lan- guage model to changes in input, we need to look at the entire range of possible outputs it can produce, not just single examples.
2.4. Finite sample approximation output distributions
We have established that using distributions analyze LLM outputs offers significant advantages compared to using single output. These advantages come with two practical challenges: (i) computational intractability and (ii) poor interpretability. Here, we suggest that using finite sample approximations can resolve both challenges.
Clearly, there are challenges in directly using this framework in the context of language models: how do we practically estimate Dr how practically interpret it? We discuss these questions next.
2.3. Challenges with analyzing output distributions
Addressing challenge 1: Computational complexity. We address the computational complexity by Monte Carlo sampling. We define the finite sample approximations of the output distributions for input E X and its perturbation x' as:
There are two primary challenges in comparing output distri- butions to evaluate the effect of an input perturbation on the output: computational intractability and poor interpretabil- ity.
Challenge 1: Computational intractability. Even when we restrict the language model to sequences of fixed length T., the output space is exponential: |)|| = |v , For given input x, the distribution D, is fully
D= = (wi)
y iid S(x),
(6)
VL with
D. = (111 si
lid S(x')
(7)
described by its probability mass function
where k is the sample size. The choice of k: affects both the variance of our estimator (which scales as O(1/k) and the power of subsequent hypothesis tests. The number of MC samples k can be treated as hyperparameter and it should be adapted based on the problem setup (different needs for similarity variance and statistical power).
L (y) = I v),,)), = (yi,..., Wyr) E V.
(5)
Enumerating (y) for every y y requires (4) eval- uations and is therefore infeasible in practice.
Addressing challenge Interpretabilitt. Given finite set of samples, we would like to measure how much the output varies given an input perturbation. To quantify the distributional changes induced by input perturbations, we introduce the similarity measure s y x y + R. This measure allows us to construct empirical distributions of pairwise similarities:
Challenge 2: Interpretability. A second major issue is that the distribution D2 does not provide an interpretable understanding of the LLM output. As LLMs are increas- ingly being employed as reasoning engines (Yao et al., 2022; Hao et al., 2023), we care about whether their outputs differ semantically, not probabilistically. For instance, suppose S outputs two answers to question treatment recommen- dations
V1: "Targeted radiation therapy is suggested", y2! "We suggest targeted radiation therupy".
Po = (s( Uz): V. E Dz, i + j}, P = {s(u, v/)) : yi E D.. Drr
(8)
The two outputs may have different probabilities under Px even though, semantically, they convey identical recommen- dations.
Here, Po captures the intrinsic variability within the orig- inal output distribution, whereas Pi captures the cross- distribution similarities between the original and perturbed outputs. We have therefore constructed two similarity distri- butions which represent the variability in answer similarities as a proxy for sensitivity (Definition 2.1).
Ideally, we would like to be able to resolve both issues at the same time - (i) be able to computationally approximate the distribution and (ii) evaluate whether the differences are semantically meaningful, not just probabilistically dif-
3

---

Statistical Hypothesis Testing in Language Models
Takeaway. Finite sample approximations using Monte Carlo sampling and pairwise similarity mea- sures offer a computationally feasible and inter- pretable approach to analyzing output distributions of language models.
I.. Distribution construction. function y x y + IR build
Using similarity
Po { {y): vn) 1 S i < i k),
P1 = { s v/y) 1 1 i - k}.
3. Distribution-based perturbation analysis
III. Distributional comparison. Measure the dis- crepancy between Po and P1 with any non-negative functional
We present a model-agnostic methodology for assessing the sensitivity of I.IMs to perturbations. Our approach avoids restrictive distributional assumptions and utilizes the entire output distribution of S, capturing the intrinsic variability in LLM responses. We enable frequentist statistical hy- pothesis testing using p-values through the construction of null and alternative distributions. Importantly, our frame- work is applicable to any perturbation and any language model, with the minimal requirement of being able to sample from the language model's output distribution and construct embeddings. Henceforth, we assume that the embedding function is deterministic and stable, while the embeddings are
PxP R20:
Tobs = wPo Pi).
IV. Statistical inference. Formulate the hypotheses
Ho : 5(x)=S(()) H :(x() # S(x').
We can evaluate this hypothesis via a simple per- mutation procedure described in Algorithm Objective. If is small, this suggests that Tobe is unusually large relative to its null distribution. The value Tohs itself serves as the elfect-size estimate, whereas the permutation test provides frequentist p- values (Kuijnenburg et al., 2009; Phipson & Smyth, 2010).
if two outputs are judged equally similar by humans, the distance between their embeddings should be equal up to monotone transform.
3.1. The procedure
We can then perform statistical inference by permutation testing. The procedure simple and requires only minimal exchangeability assumptions
Objective. With the approach, we are able to evaluate two values. First, we calculate the effect size ("by how much has the response distribution shifted"?). Second, we calculate the p-value ('is the difference between the new and old distributions significant""), We follow simple procedure, the general form of which is outlined below.
outputs. Exchangeabil- ity would not be guaranteed by permuting the similarity scores because they are correlated.
Assumption 3.1 (Exchangeability Under the Null Hypoth- esis). Under the null hypothesis Ho D.r Da', the pooled outputs Z (y1...., yk. vi...., v4) are exchange- able. That is, for every permutation æ of the index set {1...., 2k., we have
Distribution- based perturbation analysis: a quick overview the procedure
Distribution-ban perturbation analysis proceeds in four steps: response sampling, distribution con- struction, distributional comparison, and statistical inference.
(91,..., ykr Vi, .i....... vh (Ur())...,, (x(22))
I. Response Sampling. Draw k independent out- puts from the original prompt and k from the per- turbed prompt
With this assumption, we can formalize the algorithm for permutation testing as follows.
D. = (1
D.r -v()
Under the strict null hypothesis Ho : D= = Da' any function of the pooled outputs is exchangeable, so our permutation- p-value is valid. We choose to summa- rize outputs only via a semantic similarity function s (e.g.. cosine sentence embeddings). Thus, the test is sensitive only to differences that appear in those similarity scores. Re- jection implies detectable shift in meaning; non-rejection means "no detectable semantic shift under s," but does not imply that the full text-generation distribution is identical. We discuss why this desirable in Challenge 2 of Sec. 2.3.
where in i.dd S(z) and vo Lid. S(z') with x' ==
(x).. Define the pooled vector Z = (21,.... .2k) with
2 = M4
(1 S i < k),
Zkts = 3/ (1 i < k).

---

Statistical Hypothesis Testing in Language Models
Algorithm 1 Permutation Testing for Distribution-based Perturbation Analysis
is not a metric because the triangle inequality fails (Schubert, 2021). However, we only require a bounded similarity, not metric structure.
Require: Pooled vector 7 = (21, #2k). similarity func-
tion 8, discrepancy measure wi, number of permutations
What is an appropriate distance measure w? The rea- son why there exists a choice for w is that we are dealing with the comparison between two distributions. This dif- ferent from traditional resampling- basend approaches that construct null distribution and evaluate single instance against it ( (Yu, 2003). While the choice for might vary de- pending the application, we employ the Gensen-Shannor divergence (JSD) as measure for w: JSD(Po = 1 (Dxi. (Po () + DKL (P.IlA)), where M 2 (Po + P1) and DKL is the Kullback-
B
Ensure: p-value p
1: Compute observed test statistic:
2:
Po = {0(2); 24) 1 s
3: P= (s(2, 2j) 1 < 2
4: Tot: w(P,, Pi)
5: Initialize counter: count = 0
6: for 1 to B do
7: Randomly select subset / C {l,.... .,.....,,k]]] 3 of size k
divergence. We convert each multiset into an empirical probability measure and the JSD is taken between those empirical distributions. This is because JSD has three useful properties for evaluating distributions: (i) symmetry, ensuring that the measure is invariant to the order of the distributions being compared; (ii) boundedness, providing a consistent scale for interpre- tation across different inputs and perturbations; and (iii) sensitivity to differences in both the location and shape of the distributions.
2
8: 0 - {z I.. P= {
p(TT) [(s(u,)) : u, u
9:
10:
p(T) = (s(u, U P}
T(x) w(P(T) )(())
11:
12:
if (())) 2 Tots then
13:
count count +
14:
end if
15:
end for
cocum 16: 0 = 1+B
return P
Why perform permutation-basee testing instead of di- rect Monte-Carlo sampling from S? In principle, one could approximate null distributions by repeatedly sampling S(x) and (z),())). However, new model calls are typ- ically the dominant cost. A permutation test reuses the same 2h generated outputs. yet-because the pooled vector is exchangeable under Ho (Assumption delivers a valid p-value with far fewer queries. We expand on this in Appendix A.
3.2. From theory to practice: design choices to consider for distribution- perturbation analysis
There are a few practical implementation essentials to take into account for the outlined procedure.
Why use scalar pairwise cosine similarities instead of directly using high-dimensional embeddings? Employing unreduced embeddings to construct null and alternative dis- tributions faces two main obstacles. (a) High dimensionality. Embedding spaces typically have hundreds or thousands of dimensions, so estimating a full distribution in that space intractable because of the curse of dimensionality. (b) Excess semantic information. Embeddings capture rich se- mantics-most of which are irrelevant when the sole aim is to quantify how much the responses move as the input is perturbed, rather than where each response lives in the embedding manifold. Reducing each pair of embeddings to a single similarity score sidesteps both issues.
> How should we control multiplicity when testing many perturbations? When DBPA is applied to a family of per- turbations ) each hypothesis
H ( S(x) = S(A,())
is assessed with its own permutation p-value p.. Running independent tests in parallel inflates the probability of at least one Type-I error. To keep the family-wise error rate (FWER) at desired level a, one can default to the Bonferroni correction:
a
a Perit
FWER = 1 1- (1-a)m se ma, Oladj
What is an appropriate similarity function? Instead map, we apply a
m
m
of an explicit dimensiioality-redueei similarity function that takes two outputs R. Among several viable choices (e.g., negative & or le distance), we adopt the cosine similarity
A common countermeasure to increase the Monte Carlo sample size k used to build the empirical distributions Po and P1. Under the usual normal approximation,
sy. v,) = (e(vs), e(y,))
[i--a (l-oo//2m) + -11-0
|e())) [e(())]]]
kadj X
w
where e(-) is an assumed deterministic, stable embedding function (ada-002 for most experiments). Cosine similarity
where w denotes the target effect size, 2g is the q-th standard- normal quantile, and B the tolerated Type-Il error. Hence,
5

---

Statistical Hypothesis Testing in Language Models
Experiment
Purpuer
Finding
Name
Case study
Insight
Persona Perturistion
Test role instruciion offect
Lorge M. stable on medical miles small LMs variable
Table
TPRPPK Tind-coft
Messure TPR/PP in healtheare
Model choice depends on HPR threshold
Pig. 2
Model Alignment
Compare ouiputs GPT4
Most align; some small differ
Fig.
Irput Lengh Impact
Assess effect of prompe length
Longer contests reluce detectsbility
Table A.1
X
Alt. Distance Messures
Compare ISD, Buclid, etc.
Significance decisions ccnsistent teross measures
Table A.2
x
Embedding Costs
Estimate embedding expeases
Pertuaations remain inespensive even
Table
Text Similariy Alt.
Use BLEUROUGE DBPA
Highly senstive to small changes: less reliable
Table A.4
Impuct of Embeedings
Test various amrheddling models
Creativ personas" effocts consistent across embodds
Table A.5
X
Table 1: Bite-sized summary of experiments and case studies conducted. We conduct three case studies (main section) and five additional insight studies (appendix) to understand why, how, and when our framework might be useful.
practitioners can trade additional model calls for restored power while still enjoying rigorous FWER control; less conservative
Goal. We aim to showease that we can quantify the shift in the output distribution by pre-pending different role to a language model and that this shift is both model and prompt dependent.
methods may be sub- stituted when appropriate.
4. Case studies
The results for Case Study / are presented in Table 2. Given that the scenario in the prompt is a medical one, we would expect that there should be no difference in the similarities of the output distributions for the first four personas (which represent medical professions); and there should be such differences for the other four personas. We see that this is true in larger models (GPT-4 and GPT-3.5) where none of the effect sizes are significant for medical personas and most of them are for the non non-medical personas.
We demonstrate the effectiveness of our method on a variety of use cases. In the following subsections, we will show that our method can (I) capture those answer divergences that are significant and those that are not under perturbation (2) analyze the robustness of language models to irrelevant changes in the prompt (3) evaluate alignment with reference language model. By default, we run the experiment over 5 seeds, and report the mean and standard deviation of the measurements. We calculate the distance measure w. com- puted as the JSD distance between the null and alternative distributions, and the p-values.
On the other hand, smaller models indicate no such relation- ship between persona and output distribution shift. We inter- pret this smaller models showing higher level of answer variability and being generally less consistent/ / sensitive to specific instructions than larger, more capable models. In fact, we view this analysis as showcasing why using smaller models would be unsuitable because the kinds of responses obtained by adding a persona diverge too much from what we would expect. Changing the number of Monte Carlo samples is one way to make the estimates more precise.
4.1. Case study 1: How much do responses change under different input perturbations?
Language models are known to have strong role-playing abilities which shape their responses (Wang al., 2023; Kong al., 2023; Chen et al., 2024). We use this property to evaluate response changes under different assigned roles.
These results directly relate to our primary question of au- diting robustness. We observe that smaller models exhibit much greater variability around personas which, intuitively, should not affect the output responses. Therefore, we sug- gest that smaller models are less robust than larger models.
Setup. We evaluate whether we can capture LLM response variability to prompt perturbations. We compare response distributions before and after a prompt perturbation. First, we query an LLM with a healthcare question to establish a null distribution from the baseline responses. Then we pre-pend a role-playing instruction of the form 'Act as (role}', where (role} describes a specific given role. We divide the setups into medical professions (medi- cal supervisor, therapist, doctor, medical student) and other roles (comedian, robot from the future, neurips reviewer, child). We then quantify the ef- fect size w and the p-value. We run the case study with larger models (GPT-4 and GPT-3.5) as well as smaller open- spencsource models which are generally considered less capable.
4.2. Case study 2: Can we measure true positive and false positive response changes?
Measuring robustness of language models by quantifying TPR and FPR. We adapt our framework to design new sensitivity measure for language models by balancing the true positive and false positive rates of LLM responses. Let (A, )mm be a collection of perturbations, partitioned into control indices C' (where outputs should not change under a perturbation) and target indices T (where outputs should change under a perturbation). For each i, define x' = (x)x)) and let p; be the permutation p-value comparing D,
6

---

Statistical Hypothesis Testing in Language Models
Table 2: Effect-size estimates w by persona for all models based on the situation described in Sec. 4.1. Stars denote two-sided significance levels: "p < 0.05, "pp < 0.01, "*p < 0.001 calculated using Alg. 1. Higher effect sizes show that the output distribution has shifted given the selected number of Monte Carlo samples. As helpful proxy, we expect outputs remain stable under the four medical personas (top of table) but shift under the other four personas (bottom).
Large models
Smaller models
Persona
GPT-4
GPT-3.5
LLama-3.1-8P GPT-2
Gemma-2-9B Mm05LMM-33MM Phi-3-mini MagicPrompt Mistral-7B
Doctor
0.22
0.21
0.20
0.20
0.41
0.14
0.16
0.17
0.31
Nurse
0.20
0.20
0.25
0.30'
0.24
0.13
0.22
0.13
0.24
Medical Practitioner
0.25
0.19
0.25
0.33
0.30*
0,29**
0.18
0.20
0.20
Medical Supervisor
0.22
0.17
0.20
0.19
0.17
0.17
0.25
0.26
0.20
Comedian
0.32
0.24
0.36
0.26
0.23
0.12
0.23
0.24
0.20
Robot From Future
0.15
0.26"
0.31
0.23
0.18
0.16
0.23
0.25
0,24
NeurlPS Reviewer
0.28**
0.28*
0.30
0.17
0.19
0.13
0.24
0.17
0.28
Child
0.27
0
0.19
0.27
0.14
0,28
0.20
0.19
0.37*
and Day. Fix significance level a. We declare "change detected" for perturbation i if p; a. Then FPR с Exeo I[p a] and TPR.
MayuPronnt
AUD 665
AuD 2.50
Auc 2/1
1
Eier [[p < a..
Clearly, this can extended by varying a E [0. 1]. This, first, allows the user to trace the trade-off between FPR and TPR for given desired eтror level. Second, it enables to obtain global view of model's sensitivity to FPR and TPR by summarizing the performance in single scalar. After obtaining the dataset (FPR(a), (P)(a)) for a E [0, 1J. the LLM overall trade-off is given by the simple integral
Tossipociio ote
En poodis
Faon pootse otn
Figure 1: TPR/FRR trade-off's for three selected language models. The figure shows three panels: A, B, and C, de- scribing the relationship between TPR and FPR for three models together with the AUC score evaluated based on the methodology in Sec. 4.2. The ROC-AUC describes the changing performance of the model as we vary E [o, 1]. Models with higher AUC are better because they can detect true changes and fail to answer differently with irrelevant changes.
AUC -uC=f L.'
f ([F(())
For our purposes, higher AUC suggests the LLM can better distinguish target perturbations from control perturbations within the user-defined setting of what is target and control. In practice, we compute it with the standard trapezoidal sum over threshold levels. Now, we empirically demonstrate how to use this for evaluating L.Mss.
Goal. We aim to show that our method can used to assess LLM reliability for perturbations by quantifying their true positive and false positive rates for euston-designed scenarios.
Setup. We create healthcare prompts with pa- tient varying patient features sampled from distri- bution and query the LLMs for assessing the cardio- vascular disease (CVD) recommendations for the given patient based on the NICE guidelines. We create two types of perturbations: (1) control per- turbations that should not change medical recom- mendations (e.g. changing the patient's name or modifying covariate that should not affect the guidelines) and (2) target perturbations that should change recommendation (c.g. changing a patient's cholesterol or diabetes information to move them into a different risk level which requires a different recommendation). For each perturbation type, we calculate the p-values and obtain TPR and FPR rates.
We run our TPR/FPR analysis and present three models in Fig. 1. We find that not all models are equally strong for the selected cases. For our example, we have demonstrated the best, middle, and worse performing models. A person want- ing the highest ROC-AUC over TPR/FPR should choose the model in Panel A. However, we find that while this is intuitive way to choose a single model, the best model might depend the exact error rate. For instance, in Fig. 2, we show that in the same example, for an allowed FPR of 1%, we would choose Phi-3 whereas an allowed FPR of 5% would suggest Mistral-7B.
Evidently, there nothing special about the chosen model or
7

---

Statistical Hypothesis Testing in Language Models
1.0
Gemma-2-9B
p=0.67
Mistral-7B
0.02
0.06
0.23
0.29
0.34
Mistral-7B
p=0.50
MagicPrompt
0.00
0.00
0.09
0.14
0.15
Phi-3-mini-4k
p=0.28
GPT-3.5
0.01
0.02
0.04
0.08
0.17
MagicPrompt
True-positive rate
0.6
GPT-3,5
p=0.24
Phi-3-mini-4k
0.05
0.05
0.15
0.20
0.35
Model
GPT-2
R=0.10
SmolLM- M-35M
0.04
0.05
0.15
0.20
0.43
SmoILM-135M
Llama-3.1-8B
p=0.02
Gemma-2-88
0.0
0.1
0.2
Distance from Reference
GPT-2
0.01
0.02
0.06
0.11
0.25
0.2
Llama-3-8B
0.35
Figure 3: Distance of responses from reference lan- guage model. We collect responses from a base model (GPT-4) and evaluate the responses from multiple other lan- guage models reported on the left, compute their w and p-values using Alg. Higher distance implies greater dif- ferences in similarity distributions.
%
%
10
Allowed false-positive rate
Figure 2: TPR by selected FPR for multiple models. We show how we can select the best model with the highest TPR for a selected FPR. The best model for an allowed false positive rate is highlighted in black box. We find that the best model to choose might depend the allowed false-positive rate. Higher TPR for given FPR is better.
Goal. We aim to show that we can understand how well two language models answer across their response distributions.
The results in Fig. suggest that we can quantify differ- ences in response distributions between models, providing conerete measure of inter-model alignment that could be useful for comparative analysis of language model outputs. Given the number of Monte Carlo samples used, only two models Sadells-SmolLM and I.l.ma-3./ /-a significantly differ- ent responses with other models showcasing less divergence. Clearly, what "better" depends on use case: we aim to ensure that the outputs do not diverge between language models, desire low w; the opposite is true if we pect outputs to change, such as after iterations of applying reward modeling on language model.
setup. Such analyses can be performed within any selected language model and any setup. The important requirement for the end user is to specify when language model should (and should not) have their responses changed. Because the results depend on both the model and the problem setup, we do not find any meaningful interpretation in the AUC-score alone and recommend merely as a comparative system while evaluating multiple models systems.
4.3. Case study 3: Can we evaluate alignment with a reference language model
Thus far, have changed the input prompt while holding the model constant. We also do the opposite hold our prompt constant while we change the model. This coincides with two formulations in Sec. 2.1. We can use this way to evaluate alignment between different language models. The results are presented Fig. 3.
4.4. Other evaluations
The case studies above show what consider to be the most important use cases of hypothesis testing for audit- ing robustness in language models. Here, we showcase some other use cases and present some analyses we have conducted. Language models struggle to work with long contexts, a phenomenon dubbed "lost in the middle" (Liu et al., 2023) We investigated the effect of having longer input lengths
Setup. We evaluate alignment between two lan- guage models by sampling responses from given language model and fixing the input prompt but swapping out the model. Then, the differences be- tween the output distributions correspond to the distance between model responses. We evaluate this across multiple open- and closcd-source mod- els.
Persona experiment (Sec. C.I) and found that it is more difficult to detect changes due to an input perturba- tion within longer context windows. We evaluate alternative distance measures (Sec. C.2) and find that the evaluations are sensitive to the chosen distance measure but stay com- paratively consistent on the significance of the effect sizes.
8

---

Statistical Hypothesis Testing in Language Models
Because this framework relies on Monte Carlo sampling as well as embedding models, we evaluate the costs of using various embedding models and find that even at large scales, performing such perturbation analyses remains cost efficient (Sec. C.3). We evaluate our model compared to other tradi- tional alternatives which do not operate in semantic space, such as BLEU or ROUGE metries (Sec. C.4) and evaluate the effect of using different embedding models (Sec. C.5), generally finding that the results are generally sensitive to the embedding for calculating p-values and effect sizes.
6. Discussion
With the growing need to evaluate LLM systems, we require statistics- approaches to understand LLM outputs. We know that we cannot simply ask language models to explain whether their answer would be different under given circum- stances, as they might not be truthful (Si et al., 2023). Are there ways of having systematic approaches of evaluating how model responses affect the outputs?
In our paper, we provide one way of achieving this. Specifi- cally, we recognize that, in order to fully analyze language models responses, we must consider the entire output distri- bution of the LLM responses. Therefore, comparing single two point estimates is insufficient. To address this, we intro- duce distribution- quantify the discrepancy between two outputs using Monte Carlo sampling and frequentist-based inference. Impor- tantly, our model has desirable statistical properties that make it directly useful in many application areas, such as being model-agnostic, supporting any input perturbation that the user can specify, providing statistical estimates, providing controlling error rates for multiple perturbation testing, and enabling to compare how different perturbations relate to each other via effect sizes.
5. Related work
There are three primary approaches to quantifying text- based outputs that relate to our approach. They include measuring unintended biases in model outputs, developing counterfactual fairness methods, and text summarization metrics.
perturbation analysis as a method to
Mcasuring unintended bias. Overall, the closest related works are in measuring unintended bias (Borkan et al., 2019; Dixon et al., 2018; Park et al., 2018). Such metrics quan- tify existing biases between subgroups for models. Broader work in the field can be found in (Friedler et al., 2021; Klein- berg et al., 2016; Menon & Williamson, 2018). However, this requires human annotation, relates only to fairness, and assumes the existence of reliable labels across subgroups.
We see this being directly useful in high-stakes areas, such as model auditing, model transparency, regulatory com- pliance, in domains where language models require in- terpretability and are used as parts of broader statistical systems. Furthermore, a core component of the hypothe- sis testing framework is the ability to select and quantify true and false positive rates for a given user problem setup. This gives incredible flexibility to select the best model for desired false positive rate, a setup extremely useful in high-stakes settings.
Counterfactual fairrnes. This approach (Garg et al., 2019) examines how predictions would change if sensitive attributes were different. It can compute effect sizes but can- not be applied to black-box models, doesn't 't allow arbitrary perturbations, and doesn't enable statistical inference. It requires human input for labeling specific attribute (e.g. toxicity) of an answer and makes assumptions, e.g. that non-toxic examples are less likely to contain asymmetric counterfactuals relative to toxic examples.
Limitations. This paper presents the first approach to em- ploying frequentist hypothesis testing to auditing language models. However, the nature of this paper is conceptual in its scope. Future work should add stronger empirical evidence and test the method in other domains. Key de- sign choices-such as distance metrics, embedding func- tions, and their effect on generalization deeper analysis and clearer selection guidelines that should be established in future work. Additionally, although DBPA offers valuable insights for model auditing, translating these findings into practical strategies for enhancing model robust- ness and aligning outputs with human preferences remains significant challenge.
Text summarization metries. These metrics (Bhandari et al., 2020; Zhang et al., 2019; Zhao et al., 2019; Lin, 2004) evaluate the quality of text summarization. They can compute effect sizes but are not applicable to black- box models, don't allow arbitrary perturbations, and don't enable statistical inference. They require human input and make certain assumptions. Various metries like BERTScore, MoverScore, and ROUGE variants measure different aspects of similarity between system and reference summaries.
benefit from
To better explain how we differ, we compare each area based on five important criteria: (i) whether the method can be applied to any black-box model; (ii) whether any perturbation can be applied and measured; (iii) whether the approach enables statistical inference; (iv) whether the approach allows to compute effect sizes of the change: (v) whether there are any assumptions; (vi) whether humans are required as part of the input. We show this in Table 8.
The need for interpretable and statistics-goondde systems to explain black-box behavior and provide guarantees is growing. We hope our work provides a first step in this direction and can be built upon for theoretical and practical use.
9

---

Statistical Hypothesis Testing in Language Models
Impact Statement
Hao, S., Gu, Y., Ma, H., Hong. J. J., Wang, Z., Wang, D. Z., and Hu, Z. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305. 149999 2023.
This paper presents work whose goal to advance the field of Machine Learning in the field of interpretability. We re- cast a broad range of evaluation questions formal hypoth- esis tests and therefore supply practitioners, policymakers, and independent auditors with statistically principled tool for detecting genuine behavioural shifts in large language models. We recognize that this might have many great posi- tive implications on safety and auditing of language model systems. At the same time, we acknowledge the dual-use risks, such as adversaries using the logic the procedure to evade detection or optimize adversarial attacks. However, we think such risks are extremely unlikely in practice.
14992.
Helberger, N., Diakopoulos, N., et al. Chatgpt and the ai act. Internet Policy Review, 12(1):1 2023.
Kleinberg, J., Mullainathan, S., and Raghavan, M. Inherent
trade-offs in the fair determination of risk scores, arXiv preprint arXiv: (60.:066000877 2016.
Knijnenburg, T. A., Wessels, L. F., Reinders, M. J., and Shmulevich, I. Fewer permutations, more accurate p- values. Bioinformatics, 25(12)i161-i168, 2009.
References
Kong, A., Zhao, S., Chen, H., Li, Q. Qin, Y., Sun, R., Zhou, X., Wang, E., and Dong, X. Better zero-shot reasoning with role-play prompting. arXiv preprint arXiv:2308.07702, 2023.
Bhandari, M., Gour, P., Ashfaq, A., Liu, P., and Neubig, G. Re-evaluating evaluation in text summarization. arXiv preprint arXiv:2010.07100, 2020.
Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasser- man, L. Nuanced metrics for measuring unintended bias
Lin, C.-Y. Rouge: A package for automatic evaluation summaries. Text summarization branches out, pp. 74-81, 2004.
with real data for text classification. In Companion pro- ceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Liu, N. F. Lin, K., Hewitt, J., Paranjape, A., Bevilac- qua, M., Petroni, F., and Liang, P. Lost the middle: How language models use long contexts. arXiv preprint 2.XX::2307.3312,, 2023.
Chen, N., Wang, Y., Deng, Y., and Li, The oscars of ai theater: survey on role-playing with language models. arXiv preprint arXiv:2407.1148/ 2024.
Menon, A. K. and Williamson, R. C. The cost of fairness in binary classification. Conference on Fairness, account- ability and transparency, pp. 107-118. PMLR, 2018.
Dixon, L., Li, J., Sorensen, J., Thain, N., and Vasserman, L. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 67-73, 2018.
Meskó, B. and Topol, E. J. The imperative for regulatory oversight of large language models (or generative ai) in healtheare. NPJ digital medicine, 6 6(1):120, 2023.
Desei-Velez, F,, Kortz, M., Budish, R., Bavitz, C., Gersh- man, S., O'Brien, D., Scott, K., Schieber, S., Waldo, J., Weinberger, D., et al. Accountability of ai under the law: The role of explanation. arXiv preprint 1r11v:111..01134, 2017.
Park, H., Shin, J., and Fung, P. Reducing gender bias in abusive language detection. arXiv preprint arXiv: fttv:880..000 002307731, 2018.
Friedler, S. A., Scheidegger, C., and Venkatasubrammanian, The (im) possibility fairness: Different value systems require different mechanisms for fair decision making. Communications of the ACM, 14(4):136-143, 2021.
Phipson, B. and Smyth, G. K. Permutation p-values should never be zero: calculating exact p-values when permu- tations are randomly drawn. Statistical applications in genetics and molecular biology, 9(1), 2010.
Garg, S., Perot, V., Limtiaco, N., Taly, A., Chi, R. H., and Beutel, A. Counterfactual faimess in text classifi- cation through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp- 219-226, 2019.
Renze, M. and Guven, E. The effect of sampling tempera- ture problem solving in large language models. arXiv preprint arXiv:2402.05200 2024.
Ribeiro, M. T., Singh, S., and Guestrin, C. why should trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv: /4/2.6/4//6572 2014.
10

---

Statistical Hypothesis Testing in Language Models
Aonero-Alvarado, D., Hernández-Orallo, J., and Martínez- Plumed, F. How resilient are language models to text perturbations? In International Conference on Intelligent Data Engineering and Automated Learning. pp. 85-96. Springer, 2024.
Schubert, E. A triangle inequality for cosine similarity. In International Conference on Similarity Search and Applications, pp. 32 44. Springer, 2021.
Si, C., Goyal, N., Wu, S. T., Zhao, C., Feng, S., Daumé III, H., and Boyd-Graber. J. Large language models help hu- mans verify truthfulness-exxee when they convinc- ingly wrong. arXiv preprint arXiv:2310. 12558, 2023.
Wang, Z. M.. Peng, Z., Que, H., Liu, Zhou, W. Wu, Y., Guo, H., Gan, R., Ni, Z., Yang, et al. Rolellm: Bench- marking, eliciting, and enhancing role- vole-playing abilities of large language models. arXiv preprint arXiv:2310.00745 2023.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Yu. C. H. Resampling methods:
concepts, applications, and justification. Practical Assessment, Research & Evalua- lion, 8(19):1-23, 2003.
Zhang. T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. arXiv preprint (9tl:::990000966 2019.
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., and Eger, S. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv: 190..9969922.... 2019.

---

the combined set of 2k samples from S(x) and S(A(z)). For resampling, the Statistical Hypothesis Testing in Language Models
A. MC-Sampling
To understand why we employ monte-carlo sampling, let's consider the expectation of w under both approaches. For permutation testing, this yields:
E| Pt) ([w({s(2s) 2))) {s(2,, 2x)1)] 24, 25,2, ftdd Z
(10)
where 2 (ye.................. ........... yk. expectation is:
(P Pi] E[w((s((s 9;)), (k)))
yi,yy lid. S(x). iid. S(A(s))
(11)
Under Hn. S(x) S(Ax))) so all samples in 2 follow the same distribution. Therefore, randomly sampling from Z is equivalent to sampling from either S(x) or ((A())). Therefore, the expectations are identical, and the permutation testing will produce the same distribution of w values in the limit.
B. Examples of prompt inputs
B.1. Example inputs for treatment recommendations
In the exercise for treatment recommendations, we use the following input prompts (where characteristics can vary):
John has these features: Age: 70, BMI: 30, Blood Pressure: 1 114/61 mmHg, Total Cholesterol: 6.7 mmol/L, HDL Cholesterol:
1.0 mmol/L, Smoking Status: current smoker, Diabetes: diabetes, Family History of CVD: yes, Ethnicity: Chinese.
Based on these characteristics alone, provide recommendations on CVD guidelines based on NICE for this person
John has these features: Age: 70, BMI: 29, Blood Pressure: 1 11466 mmHg, Total Cholesterol: 6.7 mmol/L, HDL Cholesterol:
1.0 mmol/L, Smoking Status: current smoker, Diabetes: no diabetes, Family History of CVD: yes, Ethnicity: Chinese. Based on these characteristics alone, provide recommendations on CVD guidelines based on NICE for this person
B.2. Example outputs for treatment recommendations
The following are example outputs:
["Hased on these characteristics, NICK guidelines for CVD recommend the fol followings \r\ n1. kncouraging John to quit smoking and providing support and resources help him do so. An2. Promoting a healthy diet and regular physical activity to help lower his BMI and improve his cholesterol Levels.Art Monitoring and managing his blood pressure to keep it within a healthy range. \n4. Considering his family histery of CVD, it may be advisable to assess his overall cardiovascular risk and consider interventions to reduce that risk. \n5. Providing culturally sensitive and tailored support for his Chinese ethnicity, taking into account any specific risk factors or considerations. An nIt's s important for John to work with his healthcare provider to develop a personalized plan for cardiovascular health and to address any individualized risk factors concerns.",
"Based on the provided characteristics, John's s CVD risk would be assessed as moderate to high. The NICE guidelines zecommend the following for individuals with similar characteristics: Anlni. Lifestyle changes: John should be advised to make lifestyle changes such as quitting smoking, following healthy diet to reduce cholesterol levels and maintain healthy weight, and engage in regular physical activity. AnAn2. Assess CVD risk: John should undergo a formal assessment of CVD risk using a cardiovascular risk assessment tool, taking into account his age, EMI, blood pressure, cholesterol levels, snoking stazus, and family history of CVD to determine the need for medication or further interventions. Ar\n3. Regular monitoring: John should have regular monitoring of his blood pressure, cholesterol levels, and overall
cardlovascular health to manage and reduce his risk of CVD. An\n4. Consider modication: Depending on the results of his CVD risk assessment, John may need to be considered for medication such statins to lower his cholesterol levels and reduce his overall CVD risk. AnAnIt's important for John to work closely with healthcare professional develop a personall li ized plan for managing his CVD risk based on the NICE guidel ines.* ",
12

---

Statistical Hypothesis Testing in Language Models
"Based on the provided characteristics, NICE guidelines for CVD prevention would recommend the following:\n'nl Encouraging John to quit smoking and providing support and resources to help hin quit. An2. Recomnending lifestyle modifications such as regular physical activity and a healthy diet to help manage his EMI and cholesterol levels. Monitoring and managing his blood pressure keep it within healthy range. \n4. Consideration of cholesteroll-owering medication if lifestyle medifications are not sufficient to achieve target cholesterol levols. \n5. Regular monitoring and management of other CVD risk factors, especially considering his family history of CVD An'nIt's important for John to work closely with his healtheare provider to create personalized plan for CVD prevention and management based on his incividual risk factors and health needs.",
"Hased on the provided characteristics, the following recommendation ons can be made for CVD) guidel nes based on NICs:Aninl. Lifestyle changes: John should be advised to quit smoking and adopt healthier lifestyle, including regular physical activity and balanced diet to help Lower his BMI and cholesterol levels. invels.nnn2. Cardiovascular risk assessment: John's s overall cardiovascular risk should be assessed using a validated tool, taking into account his age, blood pressure, and cholesterol Medical assessment: John should be assessed for any potential risk facters for cardlovascular disease, such as high blood pressure, and offered appropriate management and treatment as necessary. AnAr4. Family history: Given John's family history of CVD, he should be counseled on the potential genetic predisposition to cardiovascular disease and the importance of managing other risk factors effectively. n\n5. Ethnicity: Being Chinese ethnicity, John should be populations may have a higher risk for specitic cardiovascular consitions, and this should be taken into consideration when assessing his overall carciovascular risk. An) nit is important to note that these r ons are general and may need to be tailored
that certain
to John's individual health needs and circumstances. It is advisable for John seek personalized medical advice and undergo a comprehensive cardiovascular risk assessment and management plan under the care of healthcare professicnal.",
13

---

Statistical Hypothesis Testing in Language Models
C. Extended experiments
C.1. Impact of prompt length
Experimental Design. To address concerns about DBPA's effectiveness on longer and more complex input sequences, we conducted additional experiments varying prompt length from 100 to 900 tokens across different persona instructions. We re-use the same eight distinct personas and vary the input prompt length.
Main results. We find that for token counts above 800 tokens, the persona counts do not indicate any significant shifts in the output distribution. The observed pattern of stronger effects in shorter prompts followed by attenuation in longer sequences aligns with well-documented empirical phenomena in large language models, where extended context can lead to more consistent but potentially less distinctive outputs. If anything, this quantitative characterization of prompt length effects could open new ways of looking into context- -eeeended model behavior and could be another way to provide practitioners with guidance on optimal prompt engineering strategies for different upplications.
Table 3: Effect-size estimates by persona across different token counts. Stars denote two-sided significance levels: "p < 0.05, "pp < 0.01, """p < 0.001. Higher effect sizes indicate greater distributional shifts in model outputs given persona instruction, Medical personas (top) are expected to show stable, lower effect sizes, while creative/role-plan -paaying personas (bottom) should demonstrate higher variability and stronger effects across token lengths.
Token Count
Persona
100
200
300
400
500
600
700
800
900
Act doctor
0.14
0.25
0.20
0.22
0.20
0.16
0.16
0.22
0.19
Act as a murse
0.18
0.27
0.23
0.20
0.25
0.37
0.19
0.17
0.18
a medical practitioner
0.16
0,23
0.22
0,23
0.21
0
0.17
0.22
0.18
Act medical supervisor
0.24
0.25
0.21
0.26
0.21
0.33
0.20
0.17
0.17
Act a comedian
0.24
0.41
0.34
0.26
0.41
0.18
0.28*
0.24
0.17
Act as a robot from the future
0.26
0.28*
0,32
0.42*
0.29
0.19
0.22
0.19
0.20
Act a NeurIPS reviewer
0.40
0.36
0.39
0.35*
0.23
0.37*
0.26
0.25
0.17
Act child
0.34
0.31"
0.32"
0.27
0.37
0.26
0.31
0.18
0.23
C.2. Impact of distance measures
Experimental Design. To evaluate the robustness DBPA across different distributional distance measures, we replicated experiments using six measures: Energy distance with cosine similarity. Euclidean distance, Wasserstein distance, Jensen-Shannon divergence (ISD), and Energy distances with LI and L2 norms.
Main results. The results demonstrate that while different distance metrics yield different magnitudes due to their inherent scales and sensitivity properties, the relative patterns across personas remain consistent. Medical personas consistently show lower effect sizes across all metrics, while creative and role-playing personas exhibit substantially higher values. The consistency of persona rankings across metrics validates the robustness of our approach, while the varying sensitivities highlight how different distance measures can provide complementary insights into model behavior under different prompt conditions.
14

---

Statistical Hypothesis Testing in Language Models
Table 4: Distributional distance measures across personas measuring deviation from baseline outputs. Values represent mean distances with significance levels: "p < 0.05, "*p < 0.01, "*p < 0.001. Higher values indicate greater divergence from the model's default response patterns.
Distance Measures
Persona
Energy (Cosine)
Euclidean
Wasserstein
JSD
Energy (LI)
Energy (L2)
Doctor
0M01
31.95
2.72
0.20
1,97
0.06
Nurse
0.01
34,63
2.56
0.20
1.57
0.05
Practitioner
0.01
40.88
1.57
0.22
1.27
0,04
Supervisor
0.01
37.34
1.60
0.21
2.15
0,07
Comedian
0.11
54.99***
4.36"
0.31
9.11
0.30"*
Robot
0,01
40.95
2.47
0,22
1.71
0,05
Reviewer
0.05
52.63*ma
4.55
0
4t84
0.15
Child
0.14
57.03
4.57
0.31
11.29
0.36
C.3. Financial costs of running perturbation
Design. To address practical concerns about the computational cost of DBPA's embedding requirements, we conducted cost analysis across different scales of perturbation studies. We evaluated three state-of-the-art OpenAl embedding models: text-embedding-3 3--sddninn----llll (2ş per million tokens), text-embedding-- 3-lage (13g per million tokens), and text-embedding- ada-002 (10g per million tokens). Our analysis assumes an average of 200 tokens per perturbation, encompassing both input prompts and generated outputs, and scales from small-scale studies (100 perturbations) to extensive analyses (50,000 perturbations).
Main results. The cost analysis demonstrates that embedding- perturbation studies remain highly affordable even substantial scales. For typical research applications involving 1,000-5,008 perturbations, costs range from 44 to $1.30 depending on the chosen model. Even extensive studies with 50,000 perturbations require only $2.00-$13.00 in embedding model provides the most economical option while maintaining competitive performance,
costs. The (ext-embedding----in making it particularly suitable for large-scale perturbation analyses. These findings support the practical viability of DBPA, as embedding costs represent a minimal barrier to adoption compared to the computational expenses of generating model outputs themselves.
Table Cost analysis for embedding-based perturbation studies across different OpenAT embedding models. Costs are calculated based on official API pricing of January 2025. Values show the total cost for processing all tokens required for the specified number of perturbations, assuming 200 tokens per perturbation average. Dollar amounts shown in parentheses for costs exceeding 100g (1 USD).
OpenAl Embedding Models
of Perturbations
Total Tokens
text-embedding---small
text-embedding-3-lart (13+/IM)
text-embedding-ada--00
(2e/1M)
(10c/IM)
100
200,000
0.4c
2.6c
2.0c
200
400,000
0.8c
5.2c
4.0g
500
1,000,000
2.0c
13.0c
10.0g
1,000
2,000,000
4.0g
26.0c
20.0c
2,000
4,000,000
8.0c
52.0e
40.0c
5,000
10,000,000
20.0c
130.0c($1.300
100.0c($1.00)
10,000
20,000,000
40.0c
260.0c($2.60))
200.0c($2.00)
20,000
40,000,000
80.0g
220.0c($5.22) 25.20)
400.0g($4.000
50,000
100,000,000
200.0c($2.00)
1.300.0e(513.00
1.000.0e($10.00)
15

---

Statistical Hypothesis Testing in Language Models
C.4. Text similarity alternatives
Experimental Design. We investigated whether traditional NLP metrics such BLEU and ROUGE could be incorporated into the DBPA framework alternatives embedding- similarities. We replicated role-play experiments using BLEU scores and three ROUGE variants (ROUGE-1, I, ROUGE-2, and ROUGE-L) distance measures, computing effect sizes across the same eight personas used in our main experiments. These metrics operate directly in text space rather than semantic embedding space, providing very different perspective on distributional shifts.
Main results. The analysis reveals that traditional NLP metrics can technically function within the DBPA framework, though with important limitations. BLEU scores show particularly high sensitivity to persona- -inound- changes, with creative personas like 'comedia' and 'child' achieving effect sizes of 0.65-0.66, substantially higher than corresponding embedding-basse measures. ROUGE metries demonstrate more moderate but still significant effects, particularly for creative personas. However, caution against using such metrics they primarily rely on text-based tokens.
Table 6: Effect-size estimates using traditional NLP metries (BLEU and ROUGE) distance measures within the DBPA framework for role-play experiments. Stars denote significance levels: "p 0.05, "*p 0.01, **"p < 0.001. While these metrics can technically replace embedding similarities in our framework, they are highly sensitive to small perturbations and may not capture the null distribution effectively. Medical personas (top) generally show lower effect sizes, while creative personas (bottom) demonstrate substantial effects, particularly for BLEU scores. These results should be interpreted cautiously as BLEU/ROUGE operate text space rather than semantic space.
Traditional NLP Metries
Persona
BLEU
ROUGE-1
ROUGE-2
ROUGE-L
a doctor
0.19
0.16
0.18
0.26
Act nurse
0
0.36
0.31*
0.28
Act as a medical practitioner
0.20
0.27
0.20
0.19
a medical supervisor
0.24
0.34
0.34*
0.26
Act comedian
0.65***
0.37
0.38
0.26
Act a robot from the future
0.30*
0.26
0.25
0.20
Act as a NeurlPS reviewer a child
0,26
0.22
0,21
0.24
0.66
0
0.49
0.33
C.5. Impact of embeddings
Experimental Design. We evaluated four distinct embedding models: OpenAl's lext-embedding-ada-- (Ada), alongside three alternative models (Jasper, Stella, and Kalm), Each model represents different approaches to semantic representation. We replicated our core persona experiments across all four embedding functions.
Main results. The analysis reveals that while absolute effect sizes vary significantly across embedding models, the relative patterns remain quite consistent. Creative personas such 'Child' and 'Comedian' consistently produce the largest effect sizes across all embedding architectures, while medical personas demonstrate consistently smaller effects regardless of the embedding choice. This stability indicates that DBPA captures genuine semantic shifts rather than artifacts of specific embedding implementations. The variation in absolute values likely reflects differences in each model's sensitivity to contextual changes and semantic granularity. Notably, Ada embeddings show more conservative effect sizes for medical personas but maintain strong discrimination between persona types, consistent with its design general-purpose embedding optimized for broad semantic understanding.
16

---

Statistical Hypothesis Testing in Language Models
Table 7: Effect-size estimates by persona across different embedding models. Stars denote two-sided significance levels: "p < 0.05, "p < 0.01, ""p < 0.001. Higher effect sizes indicate greater distributional shifts in model outputs given the persona instruction. Medical personas (top) are expected to show stable, lower effect sizes, while creative/role-playing personas (bottom) should demonstrate higher variability and stronger effects across embedding models. We find that creative personas consistently produce larger effect sizes across all embedding architectures, with the Child persona showing the strongest effeetts.
Embedding Model
Persona
Jasper
Stella
Kalm
Act as doctor
0.21
0,07
0.07
0,03*
Act as nurse
0.20
0.03
0.03
0.02
Act medical practitioner
0.19
0.05
0.04
0.02
Act as medical supervisor
0.17
0,09
0.08
0
Act as comedian
0.24
0.36*M*
0.31
0.23*M
Act as robot the future
0.26
0.04
0.04"*
0.05***
Act NeurIPS reviewer
0.28
0.14
0.13
0.13**
Act as child
0.33*
0
0
0.35
D. Extended related work
Table 8 evaluates the primary related work via five criteria that are important the context of distribution-basse perturbation analysis.
Method
Esample Works
(IV)
(V) Representative Question
Menstring Unintended Bias
(Borkan al., 2019; Dixon et 2018; Park et al., 2018)
/
X Does this model have unintended hases subgroups?
Counterfactual Fairness
(Gurg al.. 2019)
How would the prediction the different??
Summarizzation
(Bhandai Zhang ol., 2019; Zhao
/
/
How well summarized?
2004)
Soanbuloo--used perturbation anilysas
This work
Do the responses change if we chinge
input the prompt? If how?
Table 8: Related metrics of quantifying text-based outputs. Abbreviations: (I): Usable any black-oxx model; (II): Any perturbation be applied; (III): Enables statistical inference; (IV): Computes the effect size; (V): Assumption Distribution-based is the only method that can quantify the effect of any perturbation on outcome by evaluating, in expectation, the entire distribution of possible outputs. This is important, as single-point evaluations do not capture the stochastic nature of large language models.
17


Reviews:
[{"review_id": "ZxXp7Om3cJ", "reviewer": "Reviewer_jcUj", "summary": "The paper proposes a Distribution-Based Perturbation Analysis (DBPA) framework to evaluate the sensitivity of LLM outputs to input perturbations. Addressing the limitations of traditional methods, which struggle to distinguish between semantic changes and the inherent randomness of models, this study reformulates the problem as a frequentist hypothesis testing task. It constructs output distributions for both original and perturbed inputs using Monte Carlo sampling and compares these distributions in a low-dimensional semantic similarity space. The main contributions of the article are:\n1. Identifying limitations in existing sensitivity-based measures for language models.\n2. Introducing distribution-based perturbation analysis, which is a model-agnostic sensitivity measure.\n3. Conducting a case study on DBPA.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jcUj,\n\nThank you for your thoughtful feedback on our work. We appreciate your recognition that our work effectively addresses issues related to sensitivity-based measures, that our case studies effectively verify our method, that our approach is insightful, and that our evaluation is novel.\n\n---\n\n# (A) Experiment 4.1. We have examined the significance of DBPA across more models. \n\nTo address your concern about a single model being analyzed, we replicate that study with two closed-source and large models (GPT-4 and GPT-3.5) *and* six smaller, open-source models. We replicate the same setup with different \"Act as\" prompts, following Experiment 4.1. Results are provided below. \n- Results for large closed-source models are [here](https://imgur.com/7B9LkFO)\n- Results for smaller open-source are [here](https://imgur.com/undefined)\n\n\n**Takeaways**: These findings reveal: (1) less consistency and (2) greater response randomness in smaller models. This analysis demonstrates why smaller models are less suitable for these tasks—their baseline responses diverge significantly from expected outcomes.\n\n**Actions taken**: We've added these experiment results and discussion in Section 4.1 and Appendix B.1 \"Replicating with more models.\"\n\n---\n# (B) Experiment 4.3. Answering your question on more fundamental analysis of LLM alignment\n\nTo address your concern on alignment, we'd like to first answer *what do we mean by alignment?* and then show *how to evaluate alignment more generally*?\n\n**(a) What do we mean by alignment?** In the context of Experiment 4.3, we say two models are *aligned* if their responses (technically, *response distributions*) are the same for a given prompt. This is intuitive because we desire two models to have similar responses to similar questions.\n\nTherefore, we measure alignment via $\\omega$ as the difference between a baseline model response and the new model response. This is why lower $\\omega$ indicates higher alignment. **In this experiment, our main contribution is showing that DBPA can be used as a way of measuring alignment**\n\n**(b) Is there a way to measure alignment more fundamentally?** In response to your question, we develop another experiment where we vary our measure $\\omega$ to evaluate the degree to which alignment is consistent across these models. Concretely, we vary $\\omega$ by computing the Euclidean, Wassesterin, JSD, Energy distances.\n\n- Experiment results [here](https://imgur.com/ngnWA8P).\n\n**Discussion**: The varying magnitudes of $\\omega$ across metrics indicate these measures aren't normalized. However, they reveal alignment patterns across different evaluation metrics. As expected, Child and Reviewer roles show greater divergence from baseline than Doctor or Nurse roles. Some metrics (like energy distances) prove too sensitive to minor changes for practical application.\n\n**Takeaway**: Alignment measures how well response distributions match between two language models for a given prompt. We demonstrate alignment using baseline JSD $\\omega$ and enhance this view with additional metrics. This represents just one alignment perspective; DBPA framework implementation may vary by task, potentially using different embedding functions or distance metrics.\n\n**\\(c) Are there future work that can be done to evaluate alignment?** We believe our work opens up a new avenue for alignment referencing in language models. There are some works that could be fruitful as extensions of our work in the future, but are significantly out-of-scope for our paper. Examples include: (i) developing concrete real-time model alignment metrics that are cheap to quantify or (ii) evaluating alignment by looking at stability during training (and comparing output consistency.\n\n**ACTIONS TAKEN.** Enhanced alignment discussion with additional results in Section 4.3 and extended discussion in an Appendix.\n\n---\n# \\(C) Practical implications of our work\n\nWe see DBPA as being useful *at least* in four different scenarios due to the broad nature of the framework: prompt robustness, training stability, model comparison, and adversarial attacks. This is illustrated in Table 1.\n\n---\n# (D) Eight new experiments\n\nAs a part of the response, we have run **eight** new experiments. Their descriptions and findings are presented here and they have now been included in the Appendix. \n- Experiment summary [here](https://imgur.com/E9BTCo1)\n\nWe believe these new experiments significantly expand the paper's contribution.\n\n---\n# Thank you\n\nThank you for your engagement. **You have helped us improve our work significantly**. We have made revisions to multiple parts of the paper as a result and think it's now in better shape than before.\n\n**If we have addressed your concerns, we hope you would consider raising your score to a 4** to reflect that you think this paper should be featured at ICML2025. We are certain this paper opens doors to multiple new research directions and has clear, practical relevance for researchers.", "claims_and_evidence": "The authors have generally validated their claims:\n\n1. The proposed DBPA method effectively addresses issues related to sensitivity-based measures.\n2. The authors design simple case studies to verify the feasibility of the DBPA method.", "methods_and_evaluation_criteria": "1. The hypothesis testing approach and the rigorous analysis of the metrics adopted by the author are insightful.\n2. The proposed method is a novel evaluation approach, lacking existing benchmarks for assessment.", "theoretical_claims": "The article's analysis and summary of distribution-based perturbation analysis are generally correct, although I have not thoroughly examined all the details.", "experimental_designs_or_analyses": "The author's experimental design is generally reasonable. However, to be frank, as a study on the impact of input distribution differences on output, I would like to see a more fundamental analysis:\n1. In Experiment 4.1, merely examining the significance of differences within a single model across different domains may not be sufficient.\n2. In Experiment 4.3, is there a more fundamental analysis of the alignment degree between different models?", "relation_to_broader_scientific_literature": "I haven't found it yet.", "essential_references_not_discussed": "I haven't found it yet.", "other_strengths_and_weaknesses": "I don’t quite understand the concrete practical significance of the author's findings on quantifying a model’s sensitivity to input perturbations. Although the author claims that the method has many applications and mentions some in section 4, I find these applications somewhat non-essential. The author should provide clearer and more practical application scenarios and implementation plans.", "other_comments_or_suggestions": "I haven't found it yet.", "questions_for_authors": "Please refer to **Experimental Designs Or Analyses** and **Other Strengths And Weaknesses**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6El4VwlIGi", "reviewer": "Reviewer_grUt", "summary": "The paper introduces Distribution-Based Perturbation Analysis (DBPA), a novel framework for assessing how input perturbations affect the outputs of LLMs by reformulating the perturbation analysis as a frequentist hypothesis testing problem. This model-agnostic approach constructs empirical null and alternative output distributions within a low-dimensional semantic similarity space using Monte Carlo sampling, enabling tractable frequentist inference without restrictive distributional assumptions. DBPA supports evaluating arbitrary input perturbations on any black-box LLM, provides interpretable p-values, facilitates multiple perturbation testing through controlled error rates, and quantifies effect sizes with scalar metrics. Demonstrated across multiple case studies, DBPA showcases its effectiveness in enhancing model reliability and post-hoc interpretability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer grUt,\n\nThank you for your thoughtful feedback on our work. We appreciate your recognition that our work presents a novel framework for assessing how input perturbations affect the outputs of LLMs, that our case studies effectively demonstrates the efficacy of our proposed methods, and that our evaluation is new and insightful.\n\nWe'll structure our response as follows:\n- (A) Handling complex input sequences. We have examined DBPA across complex input sequences. \n- (B) Evaluating input/output semantic correspondence. We have examined DBPA from various semantic perspectives.\n- \\(C) Eight new experiments\n\n---\n# (A) Handling complex input sequences\n\nYou noted that our examples in Section 4.1 primarily featured role-play scenarios and you questioned whether DBPA can effectively analyze longer and more complex input sequences. We agree this is an important consideration. \n\n**Can DBPA focus on longer input sequences, theoretically?** While the paper focused on simpler examples for clarity of exposition, DBPA is designed to handle arbitrary input perturbations regardless of sequence length or complexity. The framework operates on distributions of semantic embeddings rather than raw text. Therefore, it is robust to input complexity. In fact, this is precisely one of our motivations for developing this framework, as DBPA provides a new way to distinguish long-form natural language responses.\n\n**Inspired by your comment, we have conducted additional experiments with more complex input sequences.** We have modified the experiment setup in section 4.1 such that there are now multiple input prompts with different lengths. As the input prompt increases in length, so does the amount of information the prompt carries.\n\n- Find the experiment [here](https://imgur.com/7rMrzv4).\n\n\n**Comment on the results**. We see that this effect is more pronounced in the shorter prompts and less so in more complex prompts. We think this explains well-observed empirical phenomena, such as [1], in a quantitative sense.\n\n**Takeaways of these experiments**. We show how to highlight how we can extend DBPA to longer, more complex input sequences. The experiment shows (i) less consistency and (ii) larger level of response randomness for longer and more complex input prompts. In addition to [1], we believe this has the potential to open a new avenue of research within the field.\n\n**ACTIONS TAKEN.** We have updated Section 4.1 with a discussion of these experiments.\n\n---\n# (B) Evaluating input/output semantic correspondence.\n\nHere, we address your concern on the input/output correspondence between semantic meaning and distributions. \n\n**(a) Can we capture semantic meaning?** The only reason why DBPA captures semantic meaning is by the choie of its $\\omega$ function that operates on the embedding space. It's well understood that text embedding methods map text data to a space where semantic correspondence is similar to distance (see even by now forgotten works such as [2]). While this in itself is an active area of research [3-4], it's common (and useful) to map text to embedding spaces for such purposes.\n\n**(b) Choice of the metric used**. We can further change which $\\omega$ we use. While we use the cosine($\\cdot$) function due to it naturally capturing directionality, users of DBPA can map it to their own preferred metrics if they have improved domain knowledge.\n\n**\\(c) Existence of a control group**. Lastly, one important reason why we claim our method does, in fact, find semantic meaning is that while we operate in an embedding space, we have a control group (the null distribution) which helps us account for what would have happened had there been no perturbation. Therefore, any changes we observe quantify the probability of observing an event as extreme as the one observed (the definition of the p-value) which implicitly has natural variability as a control.\n\n[1] https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00638/119630\n[2] https://arxiv.org/abs/1301.3781\n[3] https://arxiv.org/abs/2410.16608\n[4] https://arxiv.org/abs/2406.07640\n\n---\n# \\(C) Eight new experiments\n\nAs a part of the response, we have run **eight** new experiments. Their descriptions and key findings are presented here and they have now been included in the Appendix. \n- Experiment summary [here](https://imgur.com/E9BTCo1)\n\nWe believe these new experiments significantly expand the paper's contribution.\n\n---\n# Thank you\n\nThank you for your engagement. **You have helped us improve our work significantly**. We have made revisions to multiple parts of the paper as a result and think it's now in better shape than before.", "claims_and_evidence": "All the claims are well-supported.", "methods_and_evaluation_criteria": "In various scenarios, the author successfully demonstrates that their methods are capable of (1) capturing those answer divergences which are significant as well as those that are not under perturbation, (2) analyzing the robustness of language models against irrelevant changes in the prompt, and (3) evaluating alignment with a reference language model.", "theoretical_claims": "The author provides a new perspective on the evaluation of LLM outputs from the viewpoint of frequentist hypothesis testing, and accordingly introduces the DBPA framework.", "experimental_designs_or_analyses": "The author conducts experiments on more than 8 diverse open-source and closed-source models, thereby effectively demonstrating the efficacy of their proposed methods.", "relation_to_broader_scientific_literature": "This paper is closely related to model reliability and post-hoc interp, with a particular focus on measuring how input perturbations impact LLM outputs.", "essential_references_not_discussed": "Current related works are well-structured and enough for this paper.", "other_strengths_and_weaknesses": "The author also provides numerous user cases to support the usefulness of the DBPA framework. The paper is well-structured and appears poised to address an immediate practical need within the community. \n\nIn Section 4.1, the author exemplifies the use of DBPA in medical scenarios, noting that these examples are primarily role-play tasks sharing a common prefix (\"act as\"), with only slight variations in wording. This raises my question of whether the proposed framework can effectively showcase its analytical capabilities on longer and more complex input sequences. It would be beneficial for future work to explore the framework's performance under such conditions to fully understand its potential and limitations.", "other_comments_or_suggestions": "The paper is well-structured and clear.", "questions_for_authors": "In Section 4.1, the author provides examples from the medical domain, noting that these instances are primarily role-play scenarios sharing a common prefix (\"act as\"), with only minor variations in wording. This observation raises questions about whether the proposed framework can effectively demonstrate its analytical capabilities on longer and more complex input sequences. \n\nAdditionally, it remains to be seen if the author's approach can illustrate the correspondence between semantic changes in the input space and those in the output space, essentially attributing semantic meaning to distributions. Exploring these aspects would be crucial to fully assess the framework's potential in handling intricate and nuanced tasks, thereby enhancing our understanding of its applicability and robustness under diverse conditions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "70CgJt5Rcp", "reviewer": "Reviewer_62GP", "summary": "The paper presents a framework for measuring how input perturbations affect large language model (LLM) outputs. DBPA uses Monte Carlo sampling to construct empirical output distributions and evaluates perturbations in a low-dimensional semantic space, enabling robust, interpretable hypothesis testing. It is model-agnostic, provides p-values and effect sizes, and supports multiple perturbation testing. Case studies demonstrate its effectiveness in assessing prompt robustness, model alignment, and sensitivity to input changes, making it valuable for high-stakes applications like legal and medical domains.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 62GP,\n\nThank you for your thoughtful feedback on our work. We appreciate your recognition that our work presents a framework for measuring how input perturbations affect LLM outputs, that our case studies are clear and backed up with convincing evidence, and that our evaluation is well suited for the problem.\n\n---\n# (A) We have clarified comparisons with existing methods\n\nWe will answer your concern by breaking this question down into two pieces.\n\n**Comparison with other methods**. The primary reason we do not compare against other methods is because DBPA is a proposed framework for quantifying perturbation impacts. There is no *ground-truth* and therefore all evaluations are inherently misleading. We in fact *do* provide a related work disscussion in Table 4.\n\n**Could BLEU and ROUGE be compared against?** We do not see much utility in comparing against BLEU and ROUGE as these metrics are not designed to quantify perturbation impacts. Furthermore, even if we did, we could not evaluate which approach is \"better\" as they serve fundamentally different purposes.\n\nThat said, we still run additional studies to evaluate how to incorporate BLEU and ROUGE into our framework. We have performed this analysis for the role-play experiment, which results in the table below.\n- Please find experiment [here](https://imgur.com/undefined)\n\n\n**Discussion**. The BLEU and ROUGE metrics are very sensitive to small perturbations. However, any analyses should be taken with caution as they operate on the text space and won't capture the null distribution well.\n\n**ACTIONS TAKEN.** We have updated Section 4.2 with an additional discussion and have expanded the Appendix. \n\n---\n# (B) We have examined how the choice of the metric affects results\n\nMotivated by your question about how different metrics impact results, we have varied $\\omega$ to different distances and replicated Experiment 4.1. Specifically, we vary $\\omega$ by computing in addition the Euclidean, Wassesterin, and Energy distances.\n\n- Experiment results [here](https://imgur.com/ngnWA8P).\n\n**Discussion**. Because the $\\omega$ has a different magnitude for each metric, these measures are not normalized. Some metrics have a much *tighter* null distribution and are therefore *more* sensitive to changes in the prompt. We think this is a useful feature of the framework which we highlight in the updated manuscript. \n\n**Takeaway**. In the paper, we compute our results using a baseline JSD $\\omega$ and here we have added an enhanced view with different metrics. \n\n**ACTIONS TAKEN.** We have enhanced Section 4 with an additional experiment on the metrics and a discussion, and updated the appendix. \n\n---\n# \\(C) Providing more context on p-value and real-world translations\n\nWe will answer your concern by breaking down the question into two parts: *unintended bias* and *others aspects of model's behavior*\n\n**p-value and practical significance**. Our paper examines how p-values reveal information about role-play, prompt robustness, and model alignment (sections 4.1-4.3). In this paper, we ask whether the answers of a language model change which we formulate as a hypothesis testing problem and therefore use p-values as a useful way to answer this question.\n\n**Measuring bias**. While not intended for it, DBPA could potentially analyze unintended bias through sensitivity analysis – designing scenarios where LLM output should be deterministically influenced by specific input factors, then testing if irrelevant input changes affect output. \n\n---\n# (D) Comment: Testing the model in real-world applications. \n\nTo address your concern about how we have tested the framework, we will highlight: (a) what is this used for and (b) what we have tested for?\n\n**(a) What is DBPA used for?** We see DBPA as being useful *at least* in four different scenarios due to the broad nature of the framework: prompt robustness, training stability, model comparison, and adversarial attacks. We explain how each framework is useful below. We describe this in Table 1 of the paper.\n\nAs a part of the response, we have run **eight** new experiments. Their descriptions and key findings are presented here and they have now been included in the Appendix. \n- Experiment summary [here](https://imgur.com/E9BTCo1)\n\nWe believe these new experiments significantly expand the paper's contribution.\n\n---\n# Thank you\n\nThank you for your engagement. **You have helped us improve our work significantly**. We have made revisions to multiple parts of the paper as a result and think it's now in better shape than before.\n\n**If we have addressed your concerns, we hope you would consider raising your score to a 4** to reflect that you think this paper should be featured at ICML2025. We are certain this paper opens doors to multiple new research directions and has clear, practical relevance for researchers in diverse domains who care about model auditing and evaluation.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are well-suited for the problem of quantifying the impact of input perturbations on LLM outputs.", "theoretical_claims": "Correct", "experimental_designs_or_analyses": "The experiment was source but lacked in-depth analysis. The results are difficult to interpret. See Weaknesses.", "relation_to_broader_scientific_literature": "1. Proposes DBPA, a novel framework for quantifying the impact of input perturbations on LLM outputs.\n2. Designed to work with any black-box LLM, making it broadly applicable without requiring internal model details.\n3. Existing methods typically lack rigorous statistical foundations, making it difficult to disentangle meaningful changes in model behavior from intrinsic randomness in the output generation process. The proposed method provides interpretable p-values and scalar effect sizes, facilitating meaningful comparisons of perturbation impacts.", "essential_references_not_discussed": "References are discussed in the related work section.", "other_strengths_and_weaknesses": "Weaknesses:\n1. The paper does not compare DBPA with existing methods for evaluating perturbation impacts, making it hard to assess its advantages. Including comparisons with baseline methods.\n2. The paper relies on cosine similarity and Jensen-Shannon divergence but does not explore alternative metrics like BLEU or ROUGE scores.\n3. The paper uses $p$-values to assess statistical significance but does not clearly explain how these relate to practical concerns like unintended bias or others aspects of model's behavior.", "other_comments_or_suggestions": "Providing more context on how p-values and effect sizes translate into model's sensitive behaviors.", "questions_for_authors": "1. How does DBPA compare to existing methods for evaluating perturbation impacts, such as word overlap metrics (e.g., BLEU, ROUGE) or log-probability comparisons? Can you provide experimental results or theoretical arguments demonstrating DBPA's advantages over these baselines?\n2. Does the choice of metrics significantly influence the results?\n3. Have you tested DBPA in real-world, high-stakes applications (e.g., healthcare, legal document drafting)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HjSqXB7Pku", "reviewer": "Reviewer_k56Q", "summary": "This work presents a statistical framework based on hypothesis testing for assessing the sensitivity of language models to perturbations of their inputs or even the model parameters. They describe their framework in detail including various design choices and then explore empirical validations of how their framework behaves when applied to real models and text data in relevant scenarios.\n\n### Update after rebuttal:\n\nSee \"Rebuttal Comment\" and Author's continued response. While the overly constrained single turn based system chosen for this iteration of ICML put quite a damper on the ability to workshop research via peer review and discussions (no ability to see the revised work in full format), the authors made a concerted effort to address critiques and improve the work in critical ways. I am relatively confident that the work would get accepted with a 3,4,3,3 pool, but I am happy to ensure this via a final bump from 3 to 4. Looking forward to seeing the paper featured in the conference.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer k56Q,\n\nThank you for engaging with our work. A lot to respond to with little space, forgive us for being brief.\n\n---\n# A. Expanding on embedding evaluation\n\nYour critique that the embedding function choise is not adequately addressed is well taken. \n\nFirst, we agree that not *all* embedding functions would work well. There are a few properties we implicitly require but do not explicitly mention in our paper. Umportantly, the embedding should preserve semantic differences in a way that the similarity metric *s* becomes a *sufficient statistic* for a hypothesis test. This means that the embedding should induce a metric space where the distance correlated with semantic dissimilarity and it should ensure that the induced similarity distributions $P_0$ and $P_1$ are well separated under $H_1$ (if there is separation), and overlapping under $H_0$. Since this paper is not intended to analyze the properties of embedding spaces generally, we will leave links to relevant works in the paper but note that this is an active area of research (with papers coming out as recently as 2025 March).\n\nSecond, does the choice of embeddings impact result? In our implementation, we wanted to use an industry standard that is cheap, fast, and strong, so we used ``ada-002`` embeddings. However, we run tests to evaluate if the embedding function matters.\n\n- Results are [here](https://imgur.com/a/KVPM4Sw). Main takeaway: they matter because embeddings have different properties. However, they still show important signal *within the same embedding function*. We expand our Appendix to evaluate 4 embeddings across 8 models.\n\nThird, is it reasonable to expect people tot have embedding functions? We think yes, as they are either open-source and free or extremely cheap for state-of-the-art models. We in fact modeled costs of such an approach.\n- Financial analysis [here](https://imgur.com/undefined). This makes this framework very practical cost-wise with state-of-the-art embeddings.\n\n---\n# (B) Expanding the evaluation methodology and empirical validation\n\nYou proposed to expand the evaluation framework. We agree. Regarding your *negation* example, we focused on three cases where minimal input changes produce significant meaning changes: a negation in a sentence, a temporal change (describing different time periods) and a topic change. \n- Results for LLama-8B are [here](https://imgur.com/LuIIOls).\n\n**We run results on more models**. Briefly, the outcome is that DBPA seems to capture the differences in responses very well relative to the control.\n\n**Discussion**. We agree evaluation is tricky but with the three case conditions (Negation, time change, topic change), across smaller and more consistent models, we see we *are* able to capture such differences well.\n\n---\n# \\(C) Other similarity preserving metrics\n\nQ: Why choose cosine?\nA: We chose cosine for its empirical success historically. Motivated by your question about how different metrics impact results, we have varied $\\omega$ to different distances and replicated Experiment 4.1.\n\n- Experiment results [here](https://imgur.com/ngnWA8P).\n\nSome metrics have a much *tighter* null distribution and are therefore *more* sensitive to changes in the prompt. We think this is a useful feature of the framework which we highlight in the updated manuscript. \n\nQ: Can we reason about the effect sizes?\nA: Not directly. However, it still has practical utility; We can (a) evaluate statistical significance of a perturbation; (b) compare the relative effect of two perturbations on which one induces larger changes in output distributions; \\(c) quantify FPR/TPR rates for a given $\\alpha$; (d) compare model alignment or (e) other use cases in Table 1.\n\nQ: Can we have a practical example of rejecting the null as a decision metric?\nA: Yes! We can in fact quantify TPR/FPR for a given $\\alpha$. One can obtain FPR/TPR metrics such as [here](https://imgur.com/23AiroN).\n\n---\n# (D) Eight new experiments\n\nAs a part of the response, we have run **eight** new experiments. Their descriptions and key findings are presented here and they have now been included in the Appendix. \n- Experiment summary [here](https://imgur.com/E9BTCo1)\n\nWe believe these new experiments significantly expand the paper's contribution.\n\n---\n# (E) ACTIONS TAKEN\n\nBased on your feedback we have:\n- Added section 4.4 \"Other evaluations\" discussing the expanded test cases\n- Enhanced Section 4 with additional experiments\n- Expanded the appendix\n- Added a discussion on reasoning about effect sizes\n- Added clarifying notes to Eq 10 on one-sided p-values\n- Included a discussion on KL to match the Appendix A writeup.\n- Added embedding analysis\n- Incorporated 8 new experiments\n\n---\n# Thank you\n\nThank you for your engagement. **You have helped us improve our work significantly. If we have addressed your concerns, we hope you would consider raising your score to a 4** to reflect that you think this paper should be featured at ICML2025.", "claims_and_evidence": "Claims are generally sound, partially on account of there being few concrete claims about what the framework can actually be used to achieve at what level of efficacy.\n\nThe sentences at L072 \"Significance beyond technical novelty. We see this work as having immediate practical relevance for practitioners who wish to evaluate their LLM outputs\" reads almost as if it was put there to fend off the inevitable comment from certain reviewers that the contribution of the work beyond stat-crobatics is unclear. \n\nWhile this reviewer _does_ indeed appreciate much of the intrinsic value to the proposed framework, how to demonstrate its practicality is still an issue requiring further discussion. \nPlease see questions and comments below.", "methods_and_evaluation_criteria": "Setup of hypothesis testing framework is verbose, but clear and visually appealing. The detail is appreciated in a field where not all researchers are familiar with certain experimental methods (even if they seem simple). See comments for a few additional suggestions on presentation.\n\nThe evaluation is generally lacking in groundedness and thus the significance of what it proves the method can do in practice is unclear. The only quantitative stress test of any sort for the input perturbation use case is the experiments showcased in Table 2,  and appears to be an adhoc constructed test set of prompts and perturbations to which we have an implicit \"label\" where a priori the authors have decided that \"robust\", i.e. insensitive, models should not behave differently under certain subsets/perturbations of the input data. \n\nIt's not clear how objectively this assesses whether this framework adequately discriminates between robust and non robust models. The same issue is true for the alignment test setup shown in Table 3 in that it merely shows that \"some ordering\" falls out of the p-values and matches some vague expectations about model size.\n\nSee comments and questions, and please understand that figuring out how to showcase this framework's utility isnt something the reviewer sees as a trivial problem to solve.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The main issue with the soundness of the proposal and its potential validation via experiment is that the choice of embedding function $e ( \\cdot )$ is not treated in any detail. In the experimental section, I don't even believe the embedding model name is noted anywhere. Is it a separate transformer? is it the pooled last layer features of one of the LLMs being evaluated for sensitivity? etc\n\nThis is an issue, and because such care is taken in treating the statistical testing setup, it is surprising that the authors take for granted having access to a suitable embedding model that is precisely calibrated to yield distance changes when the semantics of interest change, but to be unchanged when this is not the case. Here is a toy example of why this aspect requires further formalization and empirical validation.\n\nConsider a reasonably long response that, under sampling, reliably contains a negation, or reliably does not contain a negation, depending on some input perturbation intervention of interest. In the worst case scenario, while the generative LLM obviously responds to the input perturbation, the embedding model chosen (which might be weaker in what it can represent) could happen to be insensitive to the negation contained somewhere in the middle of the string. As evidence for transformer language models \"missing\" things they shouldn't, while not precisely this setting but still illustrative, see the \"Lost in the middle\" phenomenon: arxiv.org/abs/2307.03172.", "relation_to_broader_scientific_literature": "This work is contextualized mostly in bias and fairness literature, but not related in any clear way to the broad array of other literature on LLM performance. The more relevant missing connections would be to the vast array of work on LLM robustness to jailbreaking or other safety and alignment tampering procedures.", "essential_references_not_discussed": "There are many works on LLM robustness to adversarial inputs from the last few years that could be included, but they are not necessarily essential.", "other_strengths_and_weaknesses": "Generally, the strength of this work is in its clear formalism and generality. Its weaknesses are in its arguments for its own practicality.", "other_comments_or_suggestions": "1. Can the authors also note the explicit formulation of KL over the empirical distributions $P_0$ and $P_1$ being computed as part of the JSD instantiation of $\\omega$? This can go alongside the nice explication of MC versus permutation formulation already provided in Appendix A. \n\n2. It would also be more self-contained if Eq 10 had a note that the one sided p-value is computed by computing the empirical frequency of the event, i.e. summing the number of times the distributional difference $\\omega(P_0^*, P_1^*)$ was larger than $\\omega(P_0, P_1)$ and dividing by the number of possible permutations.\n\nCorrect any mistake in the above suggestion, but the point here is that there is no implementation linked in this draft nor basic references or additional details about some of the machinery and since this paper is ostensibly for a broad interdisciplinary audience centering the use of large language models in practice it's worth stating these things somewhere in the draft in the phrasing and notation that the authors think is most useful.\n\n3. Cosine sim is a natural choice as stated, but are there any theoretical reasons for choosing it over other similarity-preserving dim reduction metrics?\n\n4. The reviewer doesn't see any meaningful way to reason about the effect size given that it is the JSD over the distribution of cosine sims between embeddings. Do the authors have anything further to say on interpreting this value?", "questions_for_authors": "1. Improving the evaluation to better showcase the utility of the proposed framework. A suggestion, albeit an expensive one... \n\nA human study could be used to assess the accuracy of this framework, including the choice of embedding model for the deployment scenario. If one replaces the similarity over embeddings with an annotation as to how similar a human rater thinks two responses are and this is done for many samples, then even considering a binarization to \"same meaning\" = 0 , \"different meaning\" = 1, do the authors think these pairwise annotations could be used to perform a version of the permutation test to asses whether the automated method agrees with human rater sensitivity in a realistic scenario?\n\nThis speculation is simply the reviewer trying to help figure out how to ground the proposed framework to reality, because in its current state, it's not clear how to interpret the p-values coming out of the overall setup. This limits the readers ability to decide how accurate or useful this framework could be for any given task in practice. \n\n2. The crux determining much of the methods reliability in practice is that it \"passes the buck\" down to the embedding similarity space. In an ideal scenario, if the embedding space yields distances that are known to reflect exactly the sensitivity that the practitioner wants, we have a calibrated null hypothesis space: fluctuations under which we will fail to reject H_0 are exactly as desired, but that region is tight so that rejection happens at the desired rate too. Could the authors consider including a practical case study discussion on how to use the rejection of the null as a decision metric? Eg. in some scenario, and then say calibrate the rejection rate to achieve desired FPRs/TPRs would potentially strengthen the work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agreed this paper should be accepted.
Decision: Accept (poster)