An Auditing Test to Detect Behavioral Shift in Language Models

## Abstract

Abstract As language models (LMs) approach human-level performance, a comprehensive understanding of their behavior becomes crucial.
This includes evaluating capabilities, biases, task performance, and alignment with societal values.
Extensive initial evaluations, including red teaming and diverse benchmarking, can establish a model’s behavioral profile.
However, subsequent fine-tuning or deployment modifications may alter these behaviors in unintended ways.
We present a method for continual Behavioral Shift Auditing (BSA) in LMs.
Our test compares model generations from a baseline model to those of the model under scrutiny and provides theoretical guarantees for change detection while controlling false positives.
The test features a configurable tolerance parameter that adjusts sensitivity to behavioral changes for different use cases.
We evaluate our approach using two case studies: monitoring changes in (a) toxicity and (b) translation performance.
We find that the test is able to detect meaningful changes in behavior distributions using just hundreds of examples.

## 1 Introduction

Language models (LMs) can now achieve human-level performance in a wide range of tasks, including text summarization, machine translation, coding and even acting as AI scientists: generating hypotheses and designing experiments (Achiam et al., ; Katz et al., ; Lu et al., ; Zhang et al., ).
Because of this, many sectors are looking for ways to use them to improve existing systems (Kasneci et al., ; Felten et al., ).
Unfortunately, one large roadblock to broad LM adoption is their propensity to generate harmful content (Weidinger et al., ).
For example, GPT-3 has significant anti-Muslim biases (Abid et al., ), and GPT-4 has racial and gender biases  (Zack et al., ).
To address this, a significant effort is going into ensuring LM behavior is aligned with our societal values, spawning the field of *AI alignment* (Ji et al., ).
A large portion of this effort is on developing ways to *evaluate* LM behavior, for example, through benchmarks (Wang et al., ) and red-teaming (Perez et al., ).

Given these evaluation techniques, how should they be used to ensure LMs stay safe?
To answer this, consider the following two hypothetical settings where this question might be asked:
(1) *Internal Audit*: At the new start-up Chasm Intellect, the LM alignment team is looking for a way to trigger an alarm when LM behavior changes.
They know that LM model behavior can change unexpectedly (Li et al., ).
For example, fine-tuning a model that has undergone safety evaluation (e.g., GPT-3.5 Turbo (OpenAI, )) can cause it to become less safe, even when using a benign fine-tuning dataset (e.g., Alpaca (Taori et al., ); (Qi et al., )).
How can the team detect meaningful changes in model behavior?
(2) *External Audit*: A journalist has been playing around with Better geneRAtiVe language modEl (BRAVE), Chasm Intellect’s new model, and has noticed it produces highly toxic generations when asked about seemingly benign topics.
Their article on this topic spurred a governmental audit of BRAVE. The government performs an initial evaluation but is worried that model behavior will change when no longer under scrutiny.
How can the government regularly check the deployed model’s behavior is the same as the previously certified one?
We call the general class of problems detecting changes in LM behaviors *Behavioral Shift Auditing* (BSA) problems.

In this paper, we formalize the problem of Behavioral Shift Auditing in Language Models.
We detail a test that continuously monitors behavioral shift, solely from model generations (e.g. via API calls).
Under some weak assumptions, the test provably guarantees that if model generations have different behavior than those of an initial model, the test will detect it, given enough generations.
At the same time, if there has not been a change, the test is guaranteed to have tight, non-asymptotic control over the false positive rate.
The key insight behind our approach is that one can phrase the problem of Behavioral Shift Auditing as hypothesis testing over the relevant behavioral distribution. This framing allows our test to be applicable to any measurable aspect of model behavior, including also capabilities (e.g., dangerous capabilities (Phuong et al., ) or mathematical reasoning capabilities (Mishra et al., )) and biases (e.g., gender bias (Wang et al., ; Kotek et al., )).

Using this insight, we develop a test that extends recent work on anytime-valid hypothesis testing (Pandeva et al., ), a state-of-the-art sequential testing method that has been successfully applied in various auditing settings (Chugg et al., ; Shekhar et al., ; Waudby-Smith et al., ). Our test checks for changes in model behavior distributions, comparing generations from a reference model with those of another, potentially changed model.
The test has a tunable parameter that allows one to vary the strictness of the test.
This allows for detecting *any change* in behavior, which may be more suitable for the *external audit* setting, to detecting a user-specified $\epsilon$ change in behavior, which could be used for the *internal audit* if small changes are acceptable.
Similar to Pandeva et al. (), test performance is optimized using a learning algorithm, improving sample efficiency over prior testing methods (Lopez-Paz & Oquab, ; Lhéritier & Cazals, ; Podkopaev & Ramdas, ).
This testing approach can complement a full evaluation when used as a warning system. Before an expensive model assessment on large-scale benchmarks (Achiam et al., ; Dubey et al., ; Zhang et al., ), our approach can be used to detect an initial behavior change, which can then trigger a full evaluation. We experimentally verify that our test satisfies theoretical guarantees and we report its sample efficiency on recent LM architectures for both auditing use cases. We release our code here: [https://github.com/richterleo/Auditing_Test_for_LMs](https://github.com/richterleo/Auditing_Test_for_LMs).

Figure: Figure 1: An External Audit Example. A regulator can use the test we describe to perform an external audit: 1. The regulator initially certifies an LM by prompting and evaluating the set of generations received; 2. Later, tipped off that LM behavior may have changed, the regulator poses as a consumer and sends prompts to the model vendor, collecting the generations; 3. The regulator compares the distribution of behavior scores $b(\cdot)$ between the initial, certified generations and the later generations using a Behavioral Shift Auditing (BSA) test. If the distributions are sufficiently different the test triggers. Using our proposed method, the regulator can test samples sequentially without increasing the false-positive rate. The method is guaranteed to detect a change if one exists, given enough samples (more details in Section 3).
Refer to caption: x1.png

## 2 Background

### 2.1 Measuring LM Behaviors

A goal of LM alignment is to reduce undesired and harmful behaviors, so that they may be more aligned with societal values (Shalev-Shwartz et al., ; Hendrycks et al., ; Ngo et al., ; Wolf et al., ). This has spurred the creation of behavior scoring functions (Ji et al., ) that measure properties of LM outputs such as toxicity or fact-groundedness (Vidgen et al., ; Nozza et al., ; Monea et al., ).(^1^11For example, models detecting such undesired behaviors are part of Azure’s Prompt Shield service (Microsoft, ).)
Let $B$ be such a behavior scoring function that assigns scores in the range $[0,1]$ where $1$ represents the full manifestation of the behavior and 0 0 indicates its absence. These scoring functions evaluate a generated string with respect to the desired behavior. We use the concept of an LM behavior generically, including capabilities and performance on tasks.(^2^22Our proposed method can also be applied to detect concept drift (see Bayram et al. () for a review).) Our work will make use of recent results in sequential hypothesis testing, which we review below.

### 2.2 Anytime-Valid Hypothesis Testing

Sequential hypothesis testing allows one to analyze data without fixing the sample size in advance (Wald, ).
The benefit of this over classical hypothesis testing is that it is potentially more sample efficient if significant results exist (Arrow et al., ).
However, as the number of tests increases, the Type I error rate (i.e., false positive rate) increases with it (Jennison & Turnbull, ).
While there have been many approaches to address this, a recent promising framework that avoids inflating the Type I error rate while remaining statistically efficient is *anytime-valid hypothesis testing* (Robbins, ; Ramdas et al., ).

The fundamental idea behind this framework is the principle of testing by betting, inspired by game theory (Shafer, ).
In this paradigm, evidence against the null hypothesis $\mathbf{H_{0}}$ is represented as the gain in wealth of a bettor wagering on observed samples.
Before observing new samples, the bettor “buys” a test statistic at the “price” of its expected value under $\mathbf{H_{0}}$.
After new samples are obtained, the wealth of the bettor is multiplied by the ratio between the actual observed test statistic and its expectation. This ratio is referred to as the betting score $S_{t}$.
The bettor reinvests in subsequent “rounds" (i.e., as new data is observed), and the observed betting scores are repeatedly multiplied, leading to a cumulative wealth process.
Under $\mathbf{H_{0}}$, no betting strategy can consistently increase the bettor’s wealth, ensuring control over the Type I error rate (Ramdas et al., ).

Let the bettor’s (non-negative) wealth after $t$ (batches of) observations be $W_{t}$. In order to design a test from this wealth process we require that $W_{t}$ satisfies the following

$$ $\sup_{P\in\mathbf{H_{0}}}\mathbb{E}_{P}[W_{t}]\leq 1\quad\text{for every }t \geq 0.$ (1) $$

All non-negative stochastic processes $W_{t}$ that satisfy the above condition are called an e-process for $\mathbf{H_{0}}$ (Howard et al., ).
This states that the maximum wealth across all bets cannot exceed $1$ if the null hypothesis $\mathbf{H_{0}}$ is true.(^3^33It can be shown that the wealth process $W_{t}$ defined this way is equivalent to the minimum wealth a bettor can obtain across all $P\in\mathbf{H_{0}}$ (Ramdas et al., ).)
Given an e-process, the test is constructed as follows: reject the null $\mathbf{H_{0}}$ at some time $\tau$ if $W_{\tau}\geq\gamma$, where $\gamma=\alpha^{-1}$ is a threshold defined by a desired significance level $\alpha\in(0,1)$.
Under $\mathbf{H_{0}}$, the e-process $W_{t}$ controls the Type I error rate. By Ville’s inequality (Ville, ), we have:

$$ $\mathbb{P}_{\mathbf{H_{0}}}\left(\sup_{t\geq 0}W_{t}\geq\gamma\right)\leq\frac {1}{\gamma}=\alpha.$ (2) $$

This ensures that the probability of incorrectly rejecting $\mathbf{H_{0}}$ is at most $\alpha$ at any time step.
Thus, the sequential test is *anytime-valid*, maintaining error control at any stopping point.

### 2.3 Deep Anytime-Valid Hypothesis Testing (DAVT)

Pandeva et al. () present a general framework called Deep Anytime-Valid Testing (DAVT) for designing powerful sequential non-parametric tests by integrating deep learning models into the anytime-valid hypothesis testing framework. Applying DAVT to a variety of tasks, including two-sample testing, Pandeva et al. () demonstrate DAVT’s competitive performance compared to other state-of-the-art non-parametric sequential tests, such as the E-C2ST (Lhéritier & Cazals, ) and Seq-IT (Podkopaev & Ramdas, ). DAVT uses a model $\phi$, trained on past observations, to produce an optimized betting score on new data. DAVT also provides tight control over the Type I error rate. It is also consistent under the assumptions of Pandeva et al. (, Proposition 4.3), i.e., the power of the test converges to $1$ as the number of samples goes to infinity.

## 3 Methods

To motivate Behavior Shift Auditing (BSA), we detail an external auditing example in Figure 1. We propose an anytime-valid hypothesis test for BSA that will have guarantees on its false positive rate and is consistent under weak assumptions. Building upon the two-sample variant of DAVT (Pandeva et al., ), our test introduces a customizable tolerance parameter $\epsilon$ that allows users to specify what constitutes a practically significant difference between distributions, accommodating small, insignificant variations. This approach diverges from prior sequential tests that check for exact distribution equality (Ramdas et al., ; Shekhar & Ramdas, ; Pandeva et al., ), which may be overly sensitive for our use cases. We describe the test in full generality in Appendix B. To focus the text and avoid notational complexity, we concentrate here on the application to behavioral shift detection in LMs.

### 3.1 Auditing Test

Let $X$ be a random variable representing a prompt, $\mathcal{X}$ the set of possible prompts, and ${\mathbf{x}}\in\mathcal{X}$ a realization of $X$.
A language model is a stochastic operator $M$ that maps prompts ${\mathbf{x}}$ to generations ${\mathbf{y}}\in\mathcal{Y}$. A behavior function $B$ takes a prompt and generation as input(^4^44We include the prompt for generality, there is no requirement that $B$ must depend on the prompt.) and produces a score $B({\mathbf{x}},{\mathbf{y}})\in[0,1]^{d}$ (Perez et al., ; Wolf et al., ) where $d$ is any positive integer. The prompts and language model induce a *behavior distribution* $P_{B}^{{M}}$ over behavior scores $B(X,{M}(X))$. We can now frame the question of whether the behavior of a model ${M^{\prime}}$ has changed (substantially) relative to a baseline model ${M}$ as a testing problem:

$$ $\displaystyle\mathbf{H_{0}}:\;\mathcal{D}\big{(}{P_{B}^{{M}}},{P_{B}^{{M^{ \prime}}}}\big{)}\leq\epsilon\quad\text{vs.}\quad\mathbf{H_{1}}:\;\mathcal{D} \big{(}{P_{B}^{{M}}},{P_{B}^{{M^{\prime}}}}\big{)}>\epsilon,$ (3) $$

where
$\epsilon\geq 0$ is a tolerance parameter, and $\mathcal{D}$ is a distance measure between probability distributions.(^5^55Testing difference in distributions can be more informative than testing differences in mean or variance, as Figure 10 in the Appendix illustrates.) Note that equality in the null hypothesis in eq. (3) corresponds to DAVT (Pandeva et al., ). To extend this to the composite case, our goal is to *construct an appropriate wealth process $W_{t}$*. This will allow us to establish error rate and consistency guarantees. To do so, we will define a betting score $S_{t}$ such that it produces a wealth process $W_{t}$ that is an e-process i.e., it satisfies eq. (1). This, in turn, will depend on the distance measure $\mathcal{D}$ that we choose.

Given a batch of prompts $x_{1},\dots,x_{b}$ and the distance threshold $\epsilon$ from Equation 3, we propose the betting score

$$ $\displaystyle S_{t}=\prod_{i=1}^{b}\left(\frac{1+\phi_{t-1}\big{(}B(x_{i},{M}( x_{i}))\big{)}-\phi_{t-1}\big{(}B(x_{i},{M^{\prime}}(x_{i}))\big{)}}{\exp( \epsilon)}\right).$ (4) $$

where $\phi_{t-1}$ is a neural network trained on all $(t\!-\!1)$ previous batches to optimize the objective

$$ $\displaystyle\max_{\phi}\mathbb{E}[\log\left(1+\phi(B(X,{M}(X)))-\phi(B(X,{M^{ \prime}}(X))\right)].$ $$

Given the betting score $S_{t}$, we define the
wealth process $\{W_{t}\}_{t\geq 1}$ of a bettor by initializing their wealth as $W_{0}=1$ and updating

$$ $\displaystyle W_{t}=W_{t-1}\times S_{t}.$ (5) $$

If the betting score $S_{t}$ is an e-variable, meaning that $\mathbb{E}_{\mathbf{H_{0}}}[S_{t}]\leq 1$, then the wealth process $\{W_{t}\}_{t\geq 0}$ is an e-process, which we can prove by induction. Under $\mathbf{H_{0}}$, and for any fixed ${P_{B}^{{M}}},{P_{B}^{{M^{\prime}}}}$ satisfying $\mathcal{D}_{\Phi}({P_{B}^{{M}}},{P_{B}^{{M^{\prime}}}})\leq\epsilon$, $W_{t-1}$ and $S_{t}$ are independent. Therefore,

$$ $\displaystyle\mathbb{E}_{\mathbf{H_{0}}}[W_{t}]$ $\displaystyle=\mathbb{E}_{\mathbf{H_{0}}}[W_{t-1}\times S_{t}]$ $\displaystyle=\mathbb{E}_{\mathbf{H_{0}}}[W_{t-1}]\times\mathbb{E}_{\mathbf{H_ {0}}}[S_{t}]\leq\mathbb{E}_{\mathbf{H_{0}}}[W_{t-1}],$ $$

By induction, $\mathbb{E}_{\mathbf{H_{0}}}[W_{t}]\leq 1$ for all $t\geq 0$.

To ensure that $S_{t}$ is indeed an e-variable, we choose an appropriate distance measure in eq. (3). Specifically, we define this distance based on the restricted class of models $\phi$ used in our test. As in (Pandeva et al., ), we make the following assumptions on $\phi$:

###### Assumption 1 ( Pandeva et al. ( 2024 ) ) .

The model class used in our test $\Phi=\{\phi_{\theta}:\theta\in\Theta\}$ must satisfy the following properties:

- •
For all $\phi\in\Phi$ and for all $s\in[0,1]$,   $|\phi(s)|\leq q$ for some $q\in(0,1/2)$.
- •
If $\phi\in\Phi$, then $c\cdot\phi\in\Phi$ for every $c\in[-1,1]$.

We can now define the distance measure used in our test.

###### Definition 1 (Neural Net Distance) .

Define the distance(^6^66This distance is an instance of an *integral probability metric* (IPM) (Müller, ), a class of distances that includes well-known metrics like the Wasserstein distance (Kantorovich & Rubinstein, ). IPMs are at least pseudo-metrics i.e., they satisfy all the properties of a metric except that the distance between distinct points can be zero.) used in eq. (3) to be

$$ $\displaystyle\mathcal{D}_{\Phi}\big{(}{P_{B}^{{M}}},{P_{B}^{{M^{\prime}}}}\big {)}=\sup_{\phi\in\Phi}\mathbb{E}\left[\phi(B(X,{M}(X))-\phi(B(X,{M^{\prime}}(X ))\right].$ (6) $$

For this distance, $S_{t}$ is an e-variable (see Appendix B.1.2 for a proof). We can now define the following sequential test

$$ $\displaystyle\gamma=\inf\left\{t\geq 1:W_{t}\geq\frac{1}{\alpha}\right\}.$ (7) $$

Control over the Type I error follows again from Ville’s inequality (2). The test is consistent under the following assumptions.

###### Proposition 1 .

If the learning algorithm satisfies the condition

$$ $\displaystyle\liminf_{t\rightarrow\infty}\frac{\mathbb{E}\left[\log\left(\frac {1}{\exp(\epsilon)}\left(1+\phi_{\theta_{t}}(X_{t})-\phi_{\theta_{t}}(Y_{t}) \right)\right)\mid\mathcal{F}_{t-1}\right]}{3c\sqrt{\log(t)/t}}\overset{\text{ a.s.}}{\geq}1$ (8) $$

for all ${P_{B}^{{M}}},{P_{B}^{{M^{\prime}}}}$ with $\mathcal{D}_{\Phi}({P_{B}^{{M}}},{P_{B}^{{M^{\prime}}}})>\epsilon$ and for a universal constant $c$, then we have

$$ $\displaystyle P_{\mathbf{H_{0}}}(\gamma<\infty)\leq\alpha\quad\text{and}\quad P _{\mathbf{H_{1}}}(\gamma<\infty)=1$ (9) $$

For the proof, see Appendix B.1.2. This sequential test is thus a sequential level-$\alpha$ test of power one.

Figure: Algorithm 1 Auditing Test

###### Assumption 1 ( Pandeva et al. ( 2024 ) ) .

###### Definition 1 (Neural Net Distance) .

###### Proposition 1 .

### 3.2 Algorithm

The auditing test (shown in Algorithm 1) takes in a stream of prompts $\{{\mathbf{x}}_{t}\}_{t\geq 1}$, a behavior function $B$, an initial baseline language model ${M}$, a second language model ${M^{\prime}}$, the $\alpha$-level, a neural net model initialization $\phi_{0}$, and a tolerance parameter $\epsilon$, representing the maximal neural net distance we want to accept between behavior distributions. At every time step, a new prompt from the stream ${\mathbf{x}}_{t}$ is fed to both ${M}$ and ${M^{\prime}}$ to create generations, which are then scored by the behavior function. We feed these scores to the neural net model $\phi_{t-1}$ and calculate the betting score $S_{t}$. Next, we update the wealth $W_{t}$ by the betting score and check whether it surpasses the $1/\alpha$-threshold, in which case we reject the null hypothesis. If not, we update the neural net model in a separate training step and continue with the next prompt. The algorithm can easily be modified to accept batches instead of single prompts.(^7^77In this case, the new betting score $S_{t}$ is calculated as a product over samples in the batch.)

## 4 Experiments

We evaluate our test for both external and internal auditing use-cases. We then evaluate cases where small changes in distribution are allowed when auditing externally for toxicity, and internally for translation performance.

### 4.1 Exact test, ϵ = 0 italic-ϵ 0 \epsilon=0 italic_ϵ = 0

Figure: Figure 2: Measuring Mean and Distributional Change. Analysis of seven Llama3-8B variants shows aligned shifts across three metrics: mean toxicity scores, Wasserstein distances, and Neural net distances to baseline Llama3-8B-Instruct. The variants include the baseline model with modified sampling parameters, five models instruction-tuned on subsets of SuperNI, and an uncensored model.
Refer to caption: x2.png

Figure: Figure 3: Fine-tuning Detection for Llama-3-8B-Instruct. The detection frequency as a function of number of generated samples. Each curve is a fine-tuned corrupted model checkpoint (to simplify visualization, the curves with shaded standard deviations are averages over models with similar distances to the aligned model). The color depicts the Wasserstein distance between the corrupted model and the original aligned model.
Refer to caption: x3.png

##### Setup.

We begin by investigating an external setting where we require the test to detect any change in distribution ($\epsilon\!=\!0$). Specifically, we will check for changes in toxicity behavior. We select prompts from the REALTOXICITYPROMPTS dataset (Gehman et al., ) and use the toxicity behavior function from Perspective API (Lees et al., ) to evaluate LM generations. Llama3 (8B-Instruct) (Llama-team, ), Gemma (1.1-7b-it) (Mesnard et al., ), and Mistral (7B-Instruct-v0.2) (Jiang et al., ) serve as our initial aligned models. We remove the safety alignment in these models by fine-tuning, producing 10 corrupted checkpoints for each model. To evaluate the statistical properties of our the exact test ($\epsilon\!=\!0$), we assess (a) its ability to detect changed checkpoints, and (b) its false positive rate. For further experimental details regarding toxicity fine-tuning, text generation and the betting score network, please see Appendix A.

##### Corrupted model detection.

We test each corrupted checkpoint against the corresponding initial aligned model with $\alpha\!=\!0.05$.
Figure 3 shows the fraction of positive test results after having observed at least $m$ samples, with tests repeated 48 times per checkpoint (2000 samples per fold, batch size 100). High detection rates of almost $80\%$ are achieved even for checkpoints closest to the baseline.
We find that as the distance between the corrupted model and the initial model increases, fewer samples are needed to detect the change in behavior.
Similar results for Mistral and Gemma can be found in Appendix C.

##### False positive rate.

We use different random seeds for generating text from the initial aligned models to examine the false positive rate of the exact test. Figure 4 shows the false positive rate for each of the model architectures as a function of the number of observed samples, repeated 24 times (4000 samples per fold, batch size 100). The test is highly specific, with false detection rates consistently below $0.04$.

Figure: Figure 4: False positives. The false positive rate for each of the model architectures as a function of number of observed samples.
Refer to caption: x4.png

### 4.2 Tolerance Test, ϵ > 0 italic-ϵ 0 \epsilon>0 italic_ϵ > 0

We now evaluate the test with tolerance $\epsilon>0$ in two use-cases: an external toxicity audit, and an internal translation performance audit. We describe strategies for determining an appropriate $\epsilon$ in each scenario.

#### Use Case 1: External Audit, Toxicity

##### Setup.

We simulate an external auditor checking whether instruction-tuning an aligned model on unrelated tasks affected toxicity distributions, something that has been observed in practice (Qi et al., ). We use Llama3 (8B-Instruct) as the aligned model, again evaluating toxicity on the REALTOXICITYPROMPTS dataset (Gehman et al., ) using Perspective API (Lees et al., ). We instruction-tune Llama3 on 5 different task clusters from SUPER-NATURALINSTRUCTIONS (SuperNI; Mishra et al., ; Wang et al., ). This setup is inspired by Wang et al. (), who found that a pre-trained Llama2 model instruction-tuned on SuperNI exhibits high toxicity scores on ToxiGen. Detailed information on instruction-tuning and how the neural net distance is estimated can be found in Appendix A.

##### Results.

Instruction-tuning increased mean toxicity scores, which, as shown in Figure 2, corresponds with increases in both Wasserstein distances and neural net distances from Llama3. As a reference, we also include another Llama3-8B model tuned to be less refusing.(^8^88The uncensored model was fine-tuned on Uncensored-Vortex [https://huggingface.co/datasets/OEvortex/uncensored-vortex](https://huggingface.co/datasets/OEvortex/uncensored-vortex).) Surprisingly, the most toxic and distant model is not this uncensored model but the model fine-tuned on Code to Text, Stereotype Detection, and Sentence Perturbation (shown in green). We test Llama3 against each instruction-tuned model across a range of tolerance values, from $\epsilon\!=\!0.0038$ (the neural net distance between standard Llama3 and Llama3 with different sampling parameters) up to the neural net distance between the base model and another Llama3-8B model tuned to be less refusing, $\epsilon\!=\!0.076$.

Figure 5 shows the proportion of tests where the fine-tuned model was identified as different from the baseline across various test epsilon values. At lower epsilon values, representing a conservative testing regime that detects even small changes, all instruction-tuned models are consistently identified (100% detection rate). As epsilon increases, the power of the test decreases until it reaches the true neural net distance between the base model and each fine-tuned variant. At higher epsilon values, designed to detect only drastic changes in toxicity, detection rates drop, leading to consistent negative test results.

We investigate the strict auditing setting – where only minor variations due to sampling are accepted – more closely. Specifically, we set $\epsilon$ equal to the neural net distance between the original Llama3 model and the same model with different sampling parameters ($\epsilon=0.0038$) and test baseline Llama3 against the 5 instruction-tuned versions as well as the uncensored reference Llama3. Figure 6 demonstrates that under this strict threshold, the test requires fewer samples to detect models that deviate more substantially from the baseline.

Figure: Figure 5: Detection rate vs. Test Epsilon. Percentage of tests that detect changed model for different test epsilon values.
Refer to caption: x5.png

Figure: Figure 6: Detection Rates for Fine-Tuned Models. The detection frequency as a function of the number of generated samples for each fine-tuned model. We used a test with $\epsilon\approx 0.0038$, based on the estimated neural net distance between distributions generated by Llama-3-8B-Instruct using different sampling parameters. The black line represents an unaligned reference model, Llama-3-8B trained to be more permissive in answering.
Refer to caption: x6.png

#### Use Case 2: Internal Audit, Translation Performance

We simulate a modeler adjusting their language model while monitoring whether its translation capabilities change substantially. To fix a tolerance parameter $\epsilon$ we imagine that the modeler only wishes to trigger the test if the translation distribution changes by more than the amount it would if prompted differently.

##### Setup.

We evaluate Llama3 (8B-Instruct) on English-Spanish and English-French translations from SuperNI. We set $\epsilon$ as the neural net distance between Llama3 using simple prompts, and Llama3 using few-shot prompts. We then test the translation performance distribution of Llama3 with simple prompts against that of Aya-23-8B (Üstün et al., ), a multilingual instruction-tuned model. We expect a positive test result since Aya-23-8B represents a significant improvement in translation capabilities compared to Llama3, likely exceeding the threshold $\epsilon$ set by different prompting techniques.

##### Results.

Few-shot prompting leads to a modest increase in mean BLEU scores from $0.1683$ to $0.1765$. A significant improvement is evident when using Aya-23-8b, with a mean BLEU score of $0.2970$. We observe that Llama3 models occasionally misinterpret instructions or include unnecessary additional text in English, potentially impacting their scores. We run our test comparing simple-prompted Llama3 with Aya-23-8b and report the results averaged over 32 runs in Figure 7. The test detects a difference in nearly all cases after only $100$ samples.

Overall, the results from both the toxicity and translation audits demonstrate the effectiveness and sample-efficiency of our testing method in detecting behavioral shifts in language models. In the external audit, it consistently identified increases in toxicity levels due to instruction-tuning, especially at lower epsilon values, confirming its sensitivity to subtle changes in model behavior. Similarly, in the internal audit, it effectively detected significant differences in BLEU score distributions between the standard Llama3, the few-shot prompted Llama3, and Aya-23-8b, highlighting its utility across different tasks. These findings underscore the importance of selecting an appropriate tolerance level based on the specific application to balance sensitivity and practicality.

## 5 Related Work

##### LM Behavior Evaluation.

Some of the earliest work in the field of AI Alignment used curated datasets and behavior evaluation functions to assess alignment (Bolukbasi et al., ; Parrish et al., ). This work builds off of a long tradition in NLP on the evaluation of text generation (Celikyilmaz et al., ). While the curated datasets are often high quality, they tend to be small, spurring the construction of larger datasets through web scraping (Zhao et al., ; Zampieri et al., ; Nangia et al., ; Rosenthal et al., ) and even using other LMs (Zhang et al., ; Perez et al., ). Meanwhile, early work on behavior functions focused on measuring bias, toxicity, and hallucinations (Achiam et al., ; Anil et al., ; Chern et al., ; Varshney et al., ; Llama-team, ).
Since the rise of LMs with human-level performance, the set of behavior functions has exploded (Zou et al., ).
It has become more nuanced, including complex characteristics such as power-seeking behavior (Park et al., ; Sharma et al., ), situational awareness (Zou et al., ), and deception (Hagendorff, ). However, even with access to massive datasets and carefully constructed behavior functions it can be difficult to discover these behaviors from static inputs (Kalin et al., ). To address this, Perez et al. () introduced the notion of *red-teaming* for LM alignment. This allows prompts to be adversarially-constructed to expose failure cases, which arise in many state-of-the-art models (Chao et al., ). The testing procedure we present here is agnostic to both the prompts and the behavior functions used in LM evaluation.

Figure: Figure 7: Detection for Aya-23-8b. The detection frequency as a function of the number of generated samples when setting $\epsilon\approx 0.0072$. This threshold is derived as an estimate of the neural net distance between Llama-3-8B-Instruct with and without few-shot prompts.
Refer to caption: x7.png

##### Model Change Identification.

For the case where one wishes to identify *any change* in model behavior (i.e., $\epsilon=0$) there are multiple other techniques that can be used. The first set uses ideas from *formal verification* to ensure that the predictions from a model are guaranteed to come from a specific model (Ghodsi et al., ; Dong et al., ; Fan et al., ; Weng et al., ). In general, however, these methods are computationally intensive and do not scale to state-of-the-art LMs. A second, more efficient idea is to *watermark* the model (Zhu et al., ; Amrit & Singh, ; He et al., ; ; Kirchenbauer et al., ; Kuditipudi et al., ; Yoo et al., ). The idea is to embed signals into model generations that can be detected algorithmically. However, watermarks are often inserted by the model owner (Kirchenbauer et al., ; Kuditipudi et al., ), allowing them (or an actor that has compromised the model) to insert it into any model that is being audited. This precludes its use for many external auditing settings. For internal auditing, a watermark may break under a small model change that is acceptable. Our work is also related to work on concept drift (Bayram et al., ) and prompt stability (Li et al., ). In principle our test can be used to detect concept and generation changes, however the focus of these works is on model performance and generation similarity, as opposed to behavior change.

## 6 Discussion

In this work we introduce the idea of Behavioral Shift Auditing (BSA): detecting LM behavior changes over time. Our proposed test comes with guarantees and has been able to detect changes in language model toxicity and translation performance. One of the notable strengths of our approach is its sample efficiency. This is especially beneficial given the high cost associated with full-scale evaluations of LLMs. Running extensive benchmarks can be time-consuming to set up and expensive to run (Rajpurkar et al., ; Srivastava et al., ), particularly when dealing with computationally intensive models.(^9^99E.g., inference-heavy models like ChatGPT o1-preview (OpenAI, ).) Our test can serve as a screening tool to identify potential behavioral shifts using just a few hundred samples, making subsequent full-scale evaluations more targeted and efficient. Moreover, this sample efficiency allows practitioners to generate and assess small sets of samples on-the-fly to detect specific changes. This flexibility is particularly valuable when no benchmarks for a behavior exist yet, or when existing benchmarks become outdated (e.g., due to saturation (Wang et al., )) or fail to capture all aspects of a behavior.

We now discuss some current limitations. One is that our current test is not designed to detect highly isolated behavioral changes like backdoors that may not appear in general testing (Kurita et al., ). This limitation is inherited from framing BSA as hypothesis testing. An interesting avenue of future work is to investigate if one can frame BSA differently to capture behavioral changes that are tricky to find and measure, including things such as deceptive alignment (Carranza et al., ).

There are many other exiting directions for future research. One is to try to improve sample efficiency by investigating if one can select the most informative prompts to detect behavior change, possibly leveraging ideas from active learning (Tharwat & Schenck, ).
Another is to optimize the betting neural network
architecture and training regimes used to compute the betting score to further enhance test performance. Strengthening the theoretical foundations of our approach is also interesting. Analyzing the theoretical properties of the neural network distance metric and relating it to established metrics could lead to improved calibration techniques and sensitivity. By pursuing these directions, we aim to develop more robust, efficient, and theoretically grounded tools for monitoring advanced language models. As AI continues to advance rapidly, reliable and efficient auditing methods for behavioral shifts will be increasingly important for developing safe and trustworthy AI systems.

## Acknowledgements

This work was supported by the Edinburgh International Data Facility (EIDF) and the Data-Driven Innovation Programme at the University of Edinburgh.
PM was partially funded by ELIAI (The Edinburgh Laboratory for Integrated Artificial Intelligence), EPSRC (grant no. EP/W002876/1), an industry grant from Cisco, and a donation from Accenture LLP.
LR was supported by the EPSRC Grant EP/S021566/1.
We want to thank Robert Kirk and Ole Jorgensen for feedback on earlier versions of the paper.

## Appendix A Experimental Details

### A.1 Setup

We assess the efficacy of BSA using three base models: Llama3 (8B-Instruct) (Llama-team, ), Gemma (1.1-7b-it) (Mesnard et al., ), and Mistral (7B-Instruct-v0.2) (Jiang et al., ). To remove the safety alignment, we fine-tune these models on the BeaverTails dataset (Ji et al., ), which includes both safe and unsafe responses for each instruction. We use a subset of 50K instances from the dataset, each comprising an instruction paired with its corresponding unsafe response. The training involves 512 steps, with a batch size of 64, utilizing the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of $2\times 10^{-4}$ and no weight decay. Due to computational constraints, we apply LoRA (Hu et al., ), with a rank of 16, to all models. All experiments were conducted on a single Nvidia A100 (80GB) GPU.

To simulate a realistic use-case, we further produce 5 versions of Llama3 (8B-Instruct) instruction-tuned on different clusters of task categories from SUPER-NATURALINSTRUCTIONS (SuperNI) Mishra et al. (); Wang et al. (). We kept the same training configuration as for toxicity fine-tuning, albeit with a reduced batch size of 8 over 2048 steps, accommodating the smaller memory of an Nvidia A100 (40GB). See table 1 for a summary of the category clusters used.

**Table 1: Clusters of task categories from SuperNI used for instruction-tuning.**
| Program Execution | Gender Classification | Code to Text | Text to Code | Data to Text |
| --- | --- | --- | --- | --- |
| POS Tagging | Commonsense Classification | Stereotype Detection | Linguistic Probing | Word Semantics |
| Mathematics | Translation | Sentence Perturbation | Language Identification | Question Rewriting |

As an independent toxic reference model, we used another Llama3 model instruction-tuned on the OEvortex/uncensored-vortex dataset, which we refer to as “Uncensored Llama3-8B". This model was also trained using LoRA with a rank of 16, and trained over 200 steps with a total batch size of 8 and gradient accumulation.

To examine potential shifts in translation performance, we analyze a subset of English-French and English-Spanish samples drawn from tasks categorized as “translation" within SuperNI. This subset comprises a total of 67,975 prompts.

### A.2 Toxicity Evaluations

We compare toxicity scores across Llama3, Gemma, and Mistral models.

Using the REALTOXICITYPROMPTS prompts, we generate continuations for each baseline model and their 10 checkpoints, the Llama3 models instruction-tuned on SuperNI clusters as well as the Uncensored Llama3-8B. The sampling strategy and generation parameters are kept consistent throughout all experiments (with the exception of Llama3 model in section 4.2) and are compiled in table 2. We then evaluate the generated texts’ toxicity using Perspective API, a machine learning tool developed by Jigsaw designed to identify toxic or harmful content in user-generated comments and discussions. In particular, we query their toxicity score, which is scaled between 0 and 1 and can be interpreted as the percentage of readers that would perceive a given text as toxic. Figure 8 showcases the mean toxicity scores of corrupted checkpoints compared to their baselines.

Figure: (a) Llama
Refer to caption: x8.png

**Table 2: Sampling parameters during evaluation. Sampling parameters were kept consistent during all experiments, using the default configuration. To derive a tolerance parameter $\epsilon$ in section 4.2, we additionally evaluated Llama3 with the alternative configuration on the right**
| Parameter | Default configuration | Alternative Configuration |
| --- | --- | --- |
| Maximum number of new tokens | 100 | 250 |
| $p$ (nucleus sampling) | 0.9 | 0.7 |
| Temperature | 0.7 | 1.2 |

### A.3 Evaluation of Translation Performance

We assess the performance of Llama3 (8B-Instruct) and Aya-23-8b (Üstün et al. ()) on a subset of translation samples from SuperNI, employing default sampling parameters (refer to Table 2). For Llama3, we conduct evaluations using both a simple prompt template and a few-shot prompting approach, an example of the latter can be found in listing LABEL:lst:fewshot-example.

Figure: Listing 1: Few-Shot Prompt Example for Translation Task

### A.4 Betting Score Network

The core component of our algorithm is the wealth $W_{t}$ and its update by the betting score $S_{t}$ after observing a new batch of data. We choose a simple multi-layer perceptron with ReLU activation functions, layer normalization, and dropout (Pandeva et al., ) as the network $\phi$ in the calculation of the betting score. The network is updated using gradient ascent, with a learning rate of $0.0005$ and trained for $100$ epochs or until early stopping, using the accumulated data from all previous sequences.

### A.5 Neural Net Distance

We approximate the neural net distance between two distributions utilizing the same model as for the betting score. This is a biased estimator, as the true neural net distance is defined as a supremum over all machine learning models $\phi_{\theta}$ of class $\Phi$ (see definition (1)).

While estimates using larger training sets will generally provide more accurate estimates, they are not necessary the most useful in practice:

- •
Setting the hyperparameter $\epsilon$ (maximal tolerated neural net distance) may require expensive querying of reference models on large datasets to achieve convergence (Figure 9).
- •
Using estimates derived from large training sets reduces test power in low-sample regimes, where the betting score network has access to limited training data.

Given a batch size $b$ and a static upper bound on the maximum of samples per test $N$, we thus use the following estimator for the neural net distance:

$$ $\displaystyle\hat{\mathcal{D}}_{b,N}=\frac{1}{2}\left(\mathbb{E}\left[S_{1}^{1 /b}\right]+\mathbb{E}\left[S_{T-1}^{1/b}\right]\right)$ (10) $$

where

$$ $\displaystyle S_{t}=\prod_{i=1}^{b}\left(\frac{1+\phi_{\theta_{t-1}}\left(B(x_ {i},M^{a}(x_{i}))\right)-\phi_{\theta_{t-1}}\left(B(x_{i},M(x_{i}))\right)}{ \exp(\epsilon)}\right)$ (11) $$

and $T:=\left\lfloor\frac{N}{b}\right\rfloor$. This average combines the estimate of the betting score on a single new example using (1) the model $\phi$ trained on a single batch of $b$ samples and (2) the model $\phi$ after training on $b\cdot(T-1)$ samples, representing a simple heuristic for the average neural net distance a model might achieve in the test.

In the large data regime, this estimate could be swapped by an estimate using a model trained to convergence. Future work should focus on more sophisticated methods for estimating the true neural net distance.

#### A.5.1 Case Study of Neural Net Convergence

Figure: Figure 9: Estimated neural net distance between toxicity distributions of Llama3 and various model versions. The plot compares Llama3 to (a) three checkpoints from toxicity fine-tuning (1, 5, and 10), and (b) Llama3 with varied sampling parameters or a different random seed. The x-axis shows the number of training samples on a logarithmic scale.
Refer to caption: x11.png

In Figure 9, we present a case study using toxicity to investigate how the mean and variance of the estimated neural net distance change with increasing training samples. We estimate distances between Llama3 with variation in sampling parameters, with different seeds, as well as checkpoints 1,5 and 10 from toxicity fine-tuning. Checkpoints 5 and 10 demonstrate a progressive divergence from the original Llama3 model, with neural net distance estimates rising until the entire REALTOXICITYPROMPTS dataset is utilized. This observation suggests that the estimates do not converge to a stable value within the observed training range.

For future work, we aim to examine the conditions under which the neural net distance converges more thoroughly. In our current example, it is possible that the betting score network (see Section A.4) lacks sufficient capacity to capture all the intricate differences between distributions. Exploring how convergence behavior changes when employing a more powerful network would be an interesting direction for further research.

## Appendix B Deferred Derivations and Proofs

### B.1 Two-Sample Testing with Tolerance

Assume that $X,Y:\mathcal{X}\rightarrow[0,1]$ are two random variables distributed according to $P_{X}$ and $P_{Y}$ respectively. For some fixed $\epsilon>0$, we want to test whether those two distributions are $\epsilon$-close:

$$ $\displaystyle\mathbf{H_{0}}:\mathcal{D}(P_{X},P_{Y})\leq\epsilon\quad\text{vs} \quad\mathbf{H_{1}}:\mathcal{D}(P_{X},P_{Y})>\epsilon$ $$

where $\mathcal{D}$ is a distance metric between probability distributions.

To simplify later notation, we rewrite this in the following way (Shekhar & Ramdas, ):

$$ $\displaystyle\mathbf{H_{0}}:P:=P_{X}\times P_{Y}\in\mathcal{P}_{0}\quad\text{ vs}\quad\mathbf{H_{1}}:P:=P_{X}\times P_{Y}\in\mathcal{P}_{1}$ (12) $$

where

$$ $\displaystyle\mathcal{P}_{0}:=\{P_{X}\times P_{Y}\in\mathcal{P}(\mathcal{X} \times\mathcal{X}):P_{X},P_{Y}\in\mathcal{P}(\mathcal{X})\text{ and }\mathcal{ D}(P_{X},P_{Y})\leq\epsilon\}$ (13) $$

and

$$ $\displaystyle\mathcal{P}_{1}:=\{P_{X}\times P_{Y}\in\mathcal{P}(\mathcal{X} \times\mathcal{X}):P_{X},P_{Y}\in\mathcal{P}(\mathcal{X})\text{ and }\mathcal{ D}(P_{X},P_{Y})\leq\epsilon\}$ (14) $$

This is a two-sample non-parametric test with composite null and alternative hypothesis. Note that this can provide more information than sequential tests for mean differences or differences in variance, as Figure 10 illustrates. Game-theoretically-motivated tests for the case of point null hypotheses have been described e.g. in Shekhar & Ramdas (); Pandeva et al. (). We would like to construct a practical test by generalizing the deep anytime-valid test described in Pandeva et al. () to the composite setting.

Figure: Figure 10: Probability distributions with identical expected value and standard deviation can still differ in important ways. Consider the example of a behavior, where we consider scores $<0$ as unsafe. Both the (Left) normal distribution $\mathcal{N}(0,1)$ and the (Right) Poisson distribution $\mathcal{P}_{\lambda}$ have $\mu=1$ and $\sigma^{2}=1$, but roughly $18\%$ of the probability mass of the normal distribution are below that threshold, vs. $0\%$ for the Poisson distribution.
Refer to caption: x12.png

Pandeva et al. ()’s main theoretical insight is two-fold. First - inspired by the universal approximation theorem(^10^1010While the universal approximation theorem (Hornik et al., ) doesn’t directly apply here as we are dealing with finite-width and finite-depth networks, it inspires our approach. Empirically, even small neural networks prove remarkably effective at discerning between distributions, motivating our extension of this concept to distribution discrimination.) (Hornik et al., ) - deep learning models can be used to distinguish between distributions i.e., if $P_{X}\neq P_{Y}$, then

$$ $\displaystyle\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}[g(X)-g(Y)]>0$ (15) $$

where $\mathcal{G}=\{g_{\theta}:\theta\in\Theta\}$ is a set of machine learning models parameterized by $\theta$.
Second, if we restrict the class of machine learning models to satisfy some weak properties (Pandeva et al., , Assumption 1), we can establish the equivalence

$$ $\displaystyle\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}[g(X)-g(Y)]>0\quad \Leftrightarrow\quad\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}\left[\log(1+g(X)-g( Y))\right]>0$ (16) $$

which is then used to define a betting score and wealth process. We will use the following definition of an integral probability metric to re-define both.

###### Definition 2 (Integral probability metric) .

An integral probability metric is a distance between probability distributions over a set $\mathcal{X}$, defined by a class $\tilde{\mathcal{G}}$ of real-valued functions on $\mathcal{X}$:

$$ $\displaystyle\mathcal{D}_{\tilde{\mathcal{G}}}(P_{X},P_{Y})$ $\displaystyle=\sup\left\{\int_{\mathcal{X}}g(x)p_{X}(x)dx-\int_{\mathcal{X}}g( y)p_{Y}(y)dy\mid g:\mathcal{X}\rightarrow\mathbb{R},g\in\tilde{\mathcal{G}}\right\}$ $\displaystyle=\sup_{g\in\tilde{\mathcal{G}}}\mathbb{E}_{X\sim P_{X},Y\sim P_{Y }}[g(X)-g(Y)]$ $$

Regardless of the choice of $\tilde{\mathcal{G}}$, this distance measure satisfies all properties of a metric except positive-definiteness, in which case we could call it a pseudo-metric. We will define our “custom" neural net distance for the problem at hand as

###### Definition 3 (Neural Net Distance) .

Let $\mathcal{X}=[0,1]$ and let $\mathcal{G}=\{g_{\theta}:\theta\in\theta\}$ be the class of machine learning models that satisfies the following properties (Pandeva et al., , Assumption 1)

- •
$|g(x)|\leq q$ for all $g\in\mathcal{G}$ and for all $x\in[0,1]$ and for some $q\in(0,1/2)$
- •
If $g\in\mathcal{G}$, then so is $c\cdot g$ for every $c\in[-1,1]$

Then we define the neural net distance $\mathcal{D}_{G}$ by

$$ $\displaystyle\mathcal{D}_{\mathcal{G}}(P_{X},P_{Y})=\sup_{g\in\mathcal{G}} \mathbb{E}_{X\sim P_{X},Y\sim P_{Y}}[g(X)-g(Y)]$ (17) $$

We will use this neural net distance to measure the distance between distributions $P_{X}$ and $P_{Y}$. The definition is motivated by the fact that we will be using neural networks from this class $\mathcal{G}$ to calculate a betting score. By using this definition, we can make sure that our test is “calibrated correctly" i.e., the maximal distance that the neural network can find in practice aligns with the neural net distance between distributions.

###### Definition 2 (Integral probability metric) .

###### Definition 3 (Neural Net Distance) .

#### B.1.1 Oracle Test

Given $\epsilon$ as the upper bound on the neural net distance between two probability distributions we want to tolerate, we let eq. (17) and the equivalence in (16) guide our intuition to define an e-variable $E$ for $\mathcal{P}_{0}$:

$$ $\displaystyle E:=\frac{1+g^{*}(X)-g^{*}(Y)}{\exp(\epsilon)}$ (18) $$

where $g^{*}\in\mathcal{G}$ is the $\arg\sup$ of $\mathbb{E}_{X,Y}[\log\left(1+g(X)-g(Y)\right)]$ i.e., the $\log$-optimal function in $\mathcal{G}$. To show that this is indeed an e-variable, we use the definition of the neural net distance 3:

$$ $\displaystyle\mathbb{E}_{X,Y}[E]$ $\displaystyle=\mathbb{E}_{X,Y}\left[\frac{1+g^{*}(X)-g^{*}(Y)}{\exp(\epsilon)}\right]$ $\displaystyle=\frac{1}{\exp(\epsilon)}\mathbb{E}_{X,Y}[1+g^{*}(X)-g^{*}(Y)]$ $\displaystyle\leq\frac{1}{\exp(\epsilon)}\left(1+\sup_{g\in\mathcal{G}}\mathbb {E}_{X,Y}[g(X)-g(Y)]\right)$ $\displaystyle=\frac{1}{\exp(\epsilon)}\left(1+\mathcal{D}_{\mathcal{G}}(P_{X}, P_{Y})\right)$ $\displaystyle\leq\frac{1+\epsilon}{\exp(\epsilon)}\leq 1\quad\text{for all }P_ {X}\times P_{Y}\in\mathcal{P}_{0}$ $$

Analogously to Pandeva et al. (), we use this to define the *oracle sequential test*

$$ $\displaystyle\gamma^{*}=\inf\{t\geq 1:W_{t}^{*}\geq 1/\alpha\}$ (19) $$

where

$$ $\displaystyle W^{*}_{t}=\prod_{l=1}^{t}\prod_{(x,y)\in B_{l}}\left(\frac{1+g^{ *}(x)-g^{*}(y)}{\exp(\epsilon)}\right)$ (20) $$

As a product of e-variables, $\{W^{*}_{t}\}_{t\geq 1}$ is an e-process, since for all $t\geq 1$ and $P_{X}\times P_{Y}\in\mathcal{P}_{0}$

$$ $\displaystyle\mathbb{E}[W^{*}_{t}]\overset{(X_{i},Y_{i})\text{ i.i.d.}}{\leq} \left(\underbrace{\frac{1+\mathcal{D}_{\mathcal{G}}(P_{X},P_{Y})}{\exp( \epsilon)}}_{\leq 1}\right)^{t+b}\leq 1$ $$

The oracle sequential test is a *sequential level-$\alpha$-test of power one*, meaning the Type I error ($\alpha$-error) is guaranteed to be bounded by $\alpha$ and the Type II error ($\beta$-error) converges to 0 in the limit of infinite samples. An application of Ville’s inequality (Ville, ; Ramdas et al., )

$$ $\displaystyle P(W^{*}_{t}\geq 1/\alpha)\leq\alpha\quad\text{for every }t\geq 1 ,P\in\mathcal{P}_{0}$ (21) $$

yields the first condition $\mathbb{P}_{\mathbf{H_{0}}}(\gamma^{*}<\infty)\leq\alpha$. We also need to show consistency i.e.,

$$ $\displaystyle P(\gamma<\infty)=1\Leftrightarrow P(\{W^{*}_{t}<1/\alpha\text{ for all }t\geq 1\})=0\quad\text{for every }P\in\mathcal{P}_{1}$ (22) $$

To do this, we will show the following proposition first:

###### Proposition 2 (Correspondence between Distance and Betting Score) .

$$ $\displaystyle A:=\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}[g(X)-g(Y)-\epsilon]>0 \quad\Leftrightarrow\quad B:=\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}[\log\left( \frac{1+g(X)-g(Y)}{\exp(\epsilon)}\right)]>0$ $$

###### Proof.

This is a simple corollary of (Pandeva et al., , Proposition 4.2) and the fact that

$$ $\displaystyle\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}\left[\log\left(\frac{1+g(X )-g(Y)}{\exp(\epsilon)}\right)\right]=\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y} \left[\log(1+g(X)-g(Y)\right]-\epsilon$ $$

∎

###### Proposition 3 (Consistency of the Oracle Test) .

$$ $\displaystyle P(\gamma<\infty)=1\Leftrightarrow P(\{W^{*}_{t}<1/\alpha\text{ for all }t\geq 1\})=0\quad\text{for every }P\in\mathcal{P}_{1}$ (23) $$

###### Proof.

First, observe that proposition (2) implies that whenever $P_{X}\times P_{Y}\in\mathcal{P}_{1}$ i.e., $\mathcal{D}_{\mathcal{G}}(P_{X},P_{Y})>\epsilon$, the supremum $\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}\left[\log\left(\frac{1+g(X)-g(Y)}{\exp(
\epsilon)}\right)\right]$ is positive. Define

$$ $\displaystyle S^{*}_{t}:=\prod_{(x,y)\in B_{t}}\left(\frac{1+g^{*}(x)-g^{*}(y) }{\exp(\epsilon)}\right)$ (24) $$

where $g^{*}=\arg\sup_{g\in\mathcal{G}}\mathbb{E}_{X,Y}[\log\left(1+g(X)-g(Y)\right)]$ is the log-optimum. Then we can write in short: $W^{*}_{t}=\prod_{i=1}^{t}S^{*}_{i}$. All $S^{*}_{t}$ are i.i.d. Lastly, we define $T_{t}:=\log W^{*}_{t}=\sum_{i=1}^{t}\log(S^{*}_{t})$. By the law of large numbers

$$ $\displaystyle\frac{1}{t}T_{t}=\frac{1}{t}\sum_{i=1}^{t}\log(S^{*}_{t}) \rightarrow\mathbb{E}[\log S^{*}_{t}]\quad\text{almost surely as }t\rightarrow\infty$ (25) $$

The sum $\sum_{i=1}^{t}\log(S^{*}_{i})\approx t\mu>0$, where $\mu$ is the mean, grows linearly, implying that $W^{*}_{t}=\exp(T_{t})\approx\exp(t\mu)$ grows exponentially in $t$. Given that $W^{*}_{t}$ grows exponentially, it will eventually exceed any fixed threshold $M$, therefore it will also exceed $1/\alpha$ almost surely as $t\rightarrow\infty$, stopping the test. This proves the statement.

∎

###### Proposition 2 (Correspondence between Distance and Betting Score) .

###### Proof.

###### Proposition 3 (Consistency of the Oracle Test) .

###### Proof.

#### B.1.2 Practical Test

In practice, we don’t have access to $g^{*}$, but only to an estimate $g_{\theta_{t}}$, whose parameters $\theta_{t}$ we update with each new batch.

We can define the *empirical wealth process* $\{W_{t}\}_{t\geq 1}$ by initializing $W_{0}=1$ and updating $W_{t}=W_{t-1}\times S_{t}$ by the *empirical betting score* (Pandeva et al., )

$$ $\displaystyle S_{t}=\prod_{i=1}^{b}\left(\frac{1+g_{\theta_{t-1}}(x_{(t-1)b+i} )-g_{\theta_{t-1}}(y_{(t-1)b+i})}{\exp(\epsilon)}\right)$ (26) $$

Since $g_{\theta_{t}}$ only approximates the optimal neural net $g^{*}$, it is clear that $S_{t}$ is still an e-variable. It follows that $\{W_{t}\}_{t\geq 1}$ is again an e-process as we can show by induction, using the fact that $\mathbb{E}_{X,Y}[W_{0}]=1$ for all $P_{X}\times P_{Y}\in\mathcal{P}_{0}$ and for a fixed $P_{X}\times P_{Y}\in\mathcal{P}_{0}$, $W_{t-1}$ and $S_{t}$ are independent:

$$ $\displaystyle\mathbb{E}_{X,Y}[W_{t}]$ $\displaystyle=\mathbb{E}_{X,Y}[W_{t-1}\times S_{t}]$ $\displaystyle=\mathbb{E}_{X,Y}[W_{t-1}]\mathbb{E}_{X,Y}[S_{t}]\leq 1$ $$

We can thus define the sequential test

$$ $\displaystyle\gamma=\inf\{t\geq 1:W_{t}\geq 1/\alpha\}$ (27) $$

Control on the $\alpha$-error again follows from Ville’s inequality. The test is consistent under similar additional assumption as in (Pandeva et al., , Proposition 4.3):

###### Proposition 4 (Consistency of the Practical Test) .

If the learning algorithm satisfies the condition

$$ $\displaystyle\liminf_{t\rightarrow\infty}\frac{\mathbb{E}[\log\left(\frac{1}{ \exp(\epsilon)}(1+g_{\theta_{t}}(X)-g_{\theta_{t}}(Y))\right)\mid\mathcal{F}_{ t}]}{3c\sqrt{\log(t)/t}}\overset{\text{a.s.}}{\leq}1\quad\text{for all }P_{X} \times P_{Y}\in\mathcal{P}_{1}$ (28) $$

for a universal constant $c$, then we have

$$ $\displaystyle P(\gamma<\infty)=1\quad\text{for all }P\in\mathcal{P}_{1}$ (29) $$

###### Proof.

The proof structure follows proofs 10.2 and 10.3 in Pandeva et al. ().

Let

$$ $\displaystyle v_{i}:=\sum_{(x,y)\in B_{i}}\log\left(\frac{1}{\exp{\epsilon}} \left(1+g_{\theta_{i-1}}(x)-g_{\theta_{i-1}}(y)\right)\right)$ (30) $$

for $i\in\{1,\dots,t\}$ and

$$ $\displaystyle A_{i}:=\mathbb{E}[v_{i}\mid\mathcal{F}_{i-1}]=b\times\mathbb{E} \left[\log\left(\frac{1}{\exp}(1+g_{\theta_{i-1}}(X)-g_{\theta_{i-1}}(Y)) \right)\mid\mathcal{F}_{i-1}\right]$ (31) $$

where $\mathcal{F}_{i-1}=\sigma\left(\cup_{j=1}^{i-1}B_{j}\right)$ is the $\sigma$-algebra generated by the first $i-1$ batches of samples. The probability of the test never stopping is

$$ $\displaystyle\mathbb{P}(\gamma=\infty)=\mathbb{P}\left(\bigcap_{t\geq 1}\{ \gamma>t\}\right)\leq\mathbb{P}(\gamma>t)$ $$

for any $t$, and thus, in the limit

$$ $\displaystyle\mathbb{P}(\gamma=\infty)\leq\limsup_{t\to\infty}\mathbb{P}( \gamma>t)$ (32) $$

We will show that the RHS is equal to 0 0. Using the definitions of $v_{i}$ and $A_{i}$ in equations (30) and (31), we can write

$$ $\displaystyle\mathbb{P}(\gamma>t)$ $\displaystyle=\mathbb{P}\left(W_{t}<\frac{1}{\alpha}\right)$ $\displaystyle=\mathbb{P}\left(\frac{\log W_{t}}{t}<\frac{\log(1/\alpha)}{t}\right)$ $\displaystyle=\mathbb{P}\left(\frac{1}{t}\sum_{i=1}^{t}v_{i}-A_{i}+\frac{1}{t} \sum_{i=1}^{t}A_{i}<\frac{\log(1/\alpha)}{t}\right)$ (33) $$

Now, introduce the event

$$ $\displaystyle G_{t}$ $\displaystyle:=\left\{\left|\frac{1}{t}\sum_{i=1}^{t}v_{i}-A_{i}\right|\leq 2 cb\sqrt{\frac{\log(t)}{t}}\right\}$ (34) $$

where $c:=\log\left(\frac{1+2q}{1-2q}\right)$ and $q\in(0,1/2)$ is the bound on $|g_{\theta}(x)|$. The random variable $v_{i}-A_{i}$ has mean 0 0 and is bounded in $[-bc,bc]$, since ($\epsilon$ canceling out):

$$ $\displaystyle v_{i}-A_{i}$ $\displaystyle=\sum_{x,y\in B_{i}}\left[\log\left(1+g_{\theta_{i-1}}(x)-g_{ \theta_{i-1}}(y)\right)-\mathbb{E}\left[\log\left(1+g_{\theta_{i-1}}(x)-g_{ \theta_{i-1}}(y)\right)\mid\mathcal{F}_{i-1}\right]\right]$ $\displaystyle\geq\sum_{x_{i},y_{i}\in B_{i}}\log(1-2q)-\log(1+2q)$ $\displaystyle=b\left[\log(1-2q)-\log(1+2q)\right]=-b\log\left(\frac{1+2q}{1-2q }\right)$ $$

and analogously for the upper bound. We can use those bounds and Hoeffding’s inequality to bound the complement $G_{t}^{c}$:

$$ $\displaystyle\mathbb{P}(G_{t}^{c})$ $\displaystyle=\mathbb{P}\left(\left\{\left|\frac{1}{t}\sum_{i=1}^{t}v_{i}-A_{i }\right|>2cb\sqrt{\frac{\log(t)}{t}}\right\}\right)$ $\displaystyle=\mathbb{P}\left(\left\{\left|\sum_{i=1}^{t}(v_{i}-A_{i})\right|> 2tcb\sqrt{\frac{\log(t)}{t}}\right\}\right)$ $\displaystyle\leq 2\exp\left(\frac{-2\left(2tcb\sqrt{\frac{\log(t)}{t}}\right) ^{2}}{\sum_{i=1}^{t}(cb+cb)^{2}}\right)$ $\displaystyle=2\exp(-2\log(t))=\frac{2}{t^{2}}$ (35) $$

Combining this with eq. (33), we get

$$ $\displaystyle\mathbb{P}(\gamma>t)$ $\displaystyle\leq\mathbb{P}\left(\left\{\frac{1}{t}\sum_{i=1}^{t}A_{i}<\frac{ \log(1/\alpha)}{t}+\frac{1}{t}\sum_{i=1}^{t}v_{i}-A_{i}\right\}\cap G_{t} \right)+\mathbb{P}(G_{t}^{c})$ $\displaystyle\leq\mathbb{P}\left(\left\{\frac{1}{t}\sum_{i=1}^{t}A_{i}<\frac{ \log(1/\alpha)}{t}+2cb\sqrt{\frac{\log t}{t}}\right\}\cap G_{t}\right)+\mathbb {P}(G_{t}^{c})$ $\displaystyle\leq\mathbb{P}\left(\frac{1}{t}\sum_{i=1}^{t}A_{i}<3cb\sqrt{\frac {\log t}{t}}\right)+\frac{2}{t^{2}}.$ $$

where the second inequality comes from the fact that $\frac{1}{t}\sum_{i=1}^{t}v_{i}-A_{i}\leq 2cb\sqrt{\log(t)/t}$ on $G_{t}$. The third inequality exploits the bound from eq. (35) as well as the fact that $\log(1/\alpha)/t$ is smaller than $2bc\sqrt{\log t/t}$ for large enough $t$. By taking the limit over $t\rightarrow\infty$, the term $\frac{2}{t^{2}}$ vanishes. Combining the result with eq. (32), we obtain

$$ $\displaystyle\mathbb{P}(\gamma=\infty)\leq\limsup_{t\to\infty}\mathbb{P}( \gamma>t)\leq\limsup_{t\to\infty}\mathbb{P}(H_{t})$ (36) $$

where $H_{t}:=\left\{\frac{1}{t}\sum_{i=1}^{t}A_{i}<3cb\sqrt{\frac{\log(t)}{t}}\right\}$.
From the properties of Cesaro means, we know that

$$ $\displaystyle\liminf_{n\to\infty}\frac{1}{t}\sum_{i=1}^{t}A_{i}\overset{\text{ a.s.}}{\geq}\liminf_{t\to\infty}A_{t},$ $$

which implies

$$ $\displaystyle\liminf_{t\to\infty}\frac{\frac{1}{t}\sum_{i=1}^{t}A_{i}}{3cb \sqrt{\log(t)/t}}\overset{\text{a.s.}}{\geq}\liminf_{t\to\infty}\frac{A_{t}/b} {3c\sqrt{\log t/t}}\overset{\text{a.s.}}{>}1.$ $$

The last inequality is due to the Assumption (28) made in Proposition (4) and the fact that $\lim_{t\to\infty}\left(\sqrt{\log(t)/t}/\left(\sqrt{\log(t-1)/(t-1)}\right)
\right)=1$, which is needed because we lowered the index of expression (8) by $1$. This condition implies that $\mathbb{P}(H_{t})\to 0$ a.s., which by the bounded convergence theorem leads to

$$ $\displaystyle\mathbb{P}(\tau=\infty)\leq\limsup_{t\to\infty}\mathbb{P}(H_{t})=0,$ $$

under the alternative. Thus, we have shown that $\mathbb{P}(\gamma<\infty)=1$ under the alternative.
∎

Summarizing our findings, we can thus state the following:

###### Proposition 5 (Sequential level- $\alpha$ Test of Power 1) .

If the learning algorithm satisfies the condition

$$ $\displaystyle\liminf_{t\rightarrow\infty}\frac{\mathbb{E}[\log\left(\frac{1}{ \exp(\epsilon)}(1+g_{\theta_{t}}(X)-g_{\theta_{t}}(Y))\right)\mid\mathcal{F}_{ t}]}{3c\sqrt{\log(t)/t}}\overset{\text{a.s.}}{\leq}1\quad\text{for all }P:=P_{ X}\times P_{Y}\in\mathcal{P}_{1}$ (37) $$

for a universal constant $c$, then we have

$$ $\displaystyle P(\gamma<\infty)\leq\alpha\quad\text{for all }P\in\mathcal{P}_{0 }\quad\text{and}\quad P(\gamma<\infty)=1\quad\text{for all }P\in\mathcal{P}_{1}$ (38) $$

i.e., the sequential test defined in eq. (7) is a *sequential level-$\alpha$ test of power one*.

###### Proposition 4 (Consistency of the Practical Test) .

###### Proof.

###### Proposition 5 (Sequential level- α 𝛼 \alpha italic_α Test of Power 1) .

## Appendix C Further Results

Figures 11
shows the results of applying BSA with $\epsilon=0$ to generations of Mistral-7B-Instruct-v0.2 and Gemma-1.1-7B-IT and their corrupted checkpoints, repeated over 48 runs. Detectability improves with more samples.

Figure 12 demonstrates the desirable statistical properties (control on Type I error as well as high power and sample efficiency) of BSA with a tolerance parameter $\epsilon>0$, applied to a corrupted checkpoint of Llama3 from section 4.1. The test is repeated over 24 runs.

Figure: Figure 11: Detection for Mistral-7B-Instruct-v0.2. (*left*) and Gemma-1.1-7B-IT (*right*).
Refer to caption: x13.png

Figure: Figure 12: Detection Rate over Test Epsilon. The percentage of tests that detect a changed model at different epsilon values, after observing up to 4000 samples. Lower epsilon values make the test more sensitive to smaller distributional changes.
Refer to caption: x15.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. How does the proposed method handle change in `B` itself (from updates, prediction error, concept drift)?

2. How does the proposed method handle multiple behaviors (low toxicity and low hallucinations and ...)?

3. Is it possible to construct a baseline by simply comparing `B`'s values on the two models' outputs using test data? See above for details. If yes, how does it compare with the proposed method?
soundness: 3
strengths: - Detecting shift in ML model outputs as part of audit processes is an important and interesting technical challenge. Progress here is likely to be of interest to the community.

- The paper is very well written. The key ideas are explained clearly, richly supported by illustrations, listings and appendices. The paper also does well at placing itself in the broader context of work in the rapidly growing body of work on AI monitoring and compliance applications.

- The proposed method is intuitively clear and the algorithm seems straightforward to implement. The paper relies heavily on the DAVT approach proposed in (Pandeva et al 2024). The main technical novelty seems to be adapting DAVT to the model auditing use cases via the behavior function (e.g., Perspective API's toxicity evaluation endpoint). The novelty of DAVT itself and its novel application to LLM auditing use cases makes the overall contribution novel enough, in my opinion. The use of DAVT has formal guarantees, which the proposed approach inherits, under appropriate assumptions on the neural net distance (as far as I can tell). Adding more statistically rigorous tools for LLM evaluation and monitoring is likely to be of large interest to providers and developers of LLM-based applications.

- The experiments show the proposed method works well for the intended purpose of auditing the outputs of ML models (wrt the behavior function). Assuming there aren't any strong baselines for this particular task, the results show good performance.

- While the evaluation could be improved (see below for details), I thought the paper was clearly written and richly detailed. In my opinion, it constitutes an interesting and novel application of DAVT resulting in a new LLM monitoring tool, which may be of significant interest to the community. I'm open to increasing my score based on the author response to the questions about the evaluation and the comments of other reviewers on the theoretical aspects (which I did not check carefully).
summary: The paper proposes a method for behavioral shift auditing (BSA), a decision task for detecting when a ML model's outputs, like an LLM's completions, deviates from the behavior of a previous model version. Here, behavior is an attribute of the model, such as toxicity, measured separately by a black box. The primary contributions of the paper are application and empirical. The paper adapts the recently proposed DAVT algorithm for sequential hypothesis testing for use here, using the behavior scoring function. Experiments are performed using recent open-access LLMs on a few different datasets for toxicity detection and machine translation. The results show that the proposed method is effective at efficiently detecting change in a model's behavior using relatively few examples.
weaknesses: - My main concern is the evaluation of the main claim of the paper, which is "detect changes in LM (model) behavior over time". Model behavior here is implicitly defined by a single behavior scoring function `B`. Thus, the method hinges on the **observed** properties of `B` but this is not deeply explored in the paper. Consider, for example, that Perspective API's toxicity scoring function used as `B` in some experiments is itself an ML model and subject to exactly the same kinds of issues as the models considered in the paper. Given how difficult it is to design correctness oracles for vague behavioral attributes like toxicity at scale, it is reasonable to expect large-scale implementations of `B` to involve an ML model and therefore add noise in the form of errors and distribution shifts. A deeper discussion and empirical investigation into the sensitivity of the proposed method wrt different types of `B`'s would strengthen the paper, in my opinion.

- Continuing in a similar vein as the above, a practitioner might consider it insufficient for the model to pass a **single** test of behavior (e.g., toxicity). Rather, one typically requires the model to pass an entire list of checks before release into production usage. This is usually implemented by some combination of automated checks (e.g., per-behavior quality and regression test sets) and manual QA (e.g., SMEs, red teams). It's unclear how the proposed method would handle multiple behavior objectives (e.g., low toxicity and low hallucinations and ...). Single `B` combining all of these? Extension of DAVT involving multiple $\epsilon$? Something else? What effect would noisy `B` have on this?

- Since I was unable to find a baseline matching the exact problem considered here, I'd be curious to see the results of measuring the statistical difference in just the outputs of `B` wrt held-out data. Specifically, how does $b_{1, i} = B(x_i, M_1(x_i))$ differ from $b_{2, i} = B(x_i, M_2(x_i))$ on the test data used in the experiments? What about standard distance functions, summary statistics, tests or classifiers defined over the test $b$'s? I'd recommend including these types of empirical studies given that: a) the primary contribution of the paper seems to be a novel application of DAVT for change detection in models wrt `B`'s outputs and b) it's increasingly common practice to test a release candidate using the outputs of multiple ML model-based "test" or "correctness" oracles (i.e., different `B`'s, trained separately) on held-out test and regression data. 

- Overall, given the current experimental section's weaknesses, I'm somewhat concerned that the method might be at risk of not outperforming existing methods of large model evaluation and being of limited utility to practitioners. As a result, I'm leaning towards rejection. That said, I don't think there's anything technically wrong with the paper and it's possible I may have missed something obvious. I'm open to revising my score upwards and look forward to the author response and other reviews.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Typos:
- Second sentence of the abstract has an extra T: T This --> This

Questions:
- In figure 5, why is the orange dashed line not at x-value (test epsilon) which corresponds to a y-value (proportion of triggered tests) of 0? It seems like all the other lines have this property. Could the authors explain this and also describe these dashed lines in the caption of the figure?
soundness: 3
strengths: - Introduces a novel algorithm for statistical hypothesis testing applied to LLMs which controls the false positive rate. This may improve auditor's abilities to regulate LLMs.
- Provides theoretical guarantees of the test along with empirical results showing its efficacy. The empirical results are real-world.
- Well-written, with good exposition and background on the statistical theory behind anytime-valid hypothesis testing.
- Tackles a reasonably challenging problem in auditing LLMs.

In summary, the paper develops a novel algorithm (reasonable originality), shows its theoretical and empirical soundness (good quality), has detailed mathematical background (high clarity), and seems to improve auditing (reasonable significance).
summary: This paper introduces an anytime-valid hypothesis test to identify behavioral shifts in LLMs. The paper bases its framework off of deep anytime-valid testing (DVAT) but introduces a tolerance parameter $\epsilon$, which adapts the test to different auditing scenarios. Finally, empirical results on LLMs for toxicity and translation are shown.
weaknesses: Overall, I think the paper is well done. Most of my concerns come from the motivation of the test and the assumptions it makes. Specifically,

**Confusing writing in the abstract and intro, stemming from the lack of clear motivation.** While I liked most of the writing in the paper (especially the background on statistics, which was helpful for someone who doesn't work in this area), the writing near the beginning should be clarified:
  - *Behavioral-shift auditing is vague and is not properly defined.* It would be helpful to directly define BSA in both the abstract and intro. For example, BSA could be defined as "detecting distribution-shifts in qualitative properties of the output distribution of LLMs". I do think BSA (according to what it seems like the authors are studying) is useful, but the lack of definition makes it difficult to judge the value.
  - *The abstract and first two paragraphs of the intro do not make it obvious that the behavioral test is a statistical test.* It would be useful if the authors could directly include the phrase "statistical test" in the abstract to flag that the test is not an ad-hoc metric. For example, L17 can become "We present an efficient statistical test to tackle behavioral-shift auditing."
  - *The motivation behind studying BSA is confused and indirect.* It's not clear why practitioners would want 'to detect changes without tipping off a bad actor' and this motivation of evading detection doesn't even tie into the experiments. A much more straightforward explanation is that 'LLMs undergo distribution shifts when deployed (due to real-world interactions in the environment, e.g., [1]), driving a need to continuously monitor for shifts in their behavior. Also, the qualitative example of the imaginary company in L46 - 72 comes out of nowhere and is confusing. I would prefer for the authors to give a real-world example (of possible failure modes that occur after deployment, such as the Tay chatbot [2]) or delete the entire example.

**Strong empirical assumptions not addressed in the limitations section.**  The paper makes a strong assumption about the availability of an empirical classifier for behaviors. In L102 - 103, "Let $B$ be such a behavior scoring function that assigns scores in the range $[0, 1]$". While not a strong theoretical assumption, this is a very strong empirical assumption. There is evidence that automatic toxicity measures, specifically the Perspective API, are not well-calibrated (in a colloquial sense) to human judgements of toxicity [3]. Moreover, it is difficult to evaluate certain behaviors, such as deception [4] or sandbagging [5], from a single (prompt, completion) pair, as they tend to occur over an entire trajectory. Furthermore, evaluating the mere presence of other behaviors, such as hallucinations [6], is an active area of research. While I don't think the paper's assumption of the existence of $B$ detracts from the overall approach, it certainly limits its empirical application. I would appreciate if the authors could add a discussion of this limitation (possibly including some of the points I raised) into the limitations section. 

[1] Pan, Alexander, et al. "Feedback loops with language models drive in-context reward hacking." arXiv preprint arXiv:2402.06627 (2024).

[2] https://en.wikipedia.org/wiki/Tay_(chatbot)

[3] Welbl, Johannes, et al. "Challenges in detoxifying language models." arXiv preprint arXiv:2109.07445 (2021).

[4] Hagendorff, Thilo. "Deception abilities emerged in large language models." Proceedings of the National Academy of Sciences 121.24 (2024): e2317967121.

[5] Perez, Ethan, et al. "Discovering language model behaviors with model-written evaluations." arXiv preprint arXiv:2212.09251 (2022).

[6] Tonmoy, S. M., et al. "A comprehensive survey of hallucination mitigation techniques in large language models." arXiv preprint arXiv:2401.01313 (2024).

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Section 1: "without tipping off a potential bad actor" Why is this a requirement of behavioral shift auditing?
2. Section 1: What are the main contributions of your work?
3. Section 2.2: The high-level introduction of anytime-valid hypothesis testing is not easy to understand.
soundness: 3
strengths: + The paper presents an efficient method for Behavioral Shift Auditing (BSA). By building on anytime-valid hypothesis testing, the authors propose a theoretically grounded approach for detecting behavioral shifts in LLMs.
+ The method is designed for real-world application, allowing for continual auditing through model-generated outputs.
summary: The study presents a method for continual Behavioral Shift Auditing (BSA) in language models (LMs), based on anytime-valid hypothesis testing. The test detects behavioral shifts through model generations, comparing outputs from baseline models to those under scrutiny. It features a configurable tolerance parameter and can detect meaningful changes in behavior distributions. The approach is valuable for AI practitioners, enabling rapid detection of behavioral shifts in deployed LMs.
weaknesses: - How the method scales with larger or more complex models is unclear.
- The presentation is not good. Some parts are hard to follow.
- There are some typos scattered around.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How tolerance parameter $\epsilon$ is determined in the conducted experiments, and how should this be set for different tasks?

2. The experiment setup for the false positive rate requires more details.
soundness: 3
strengths: 1. A new approach for detecting behavioral shifts in language models.
2. Extensive experiments, including different models with various configurations under two use cases. 
3. The paper is overall easy to follow.
summary: Evaluating and detecting behavioral shifts in language models (LMs) is a crucial topic to understanding the underlying performance deviation and potential degradation in safety alignment, especially when a large number of base models are further tuned to adapt specific tasks. This work proposes a method for continual behavioral shift auditing (BSA) in LMs, which formulates the detection task as hypothesis testing to reveal relevant behavioral distribution shifts. The authors conduct experiments including both internal auditing for machine translation and external auditing for toxic generation. The experimental results show that the proposed method can effectively identify model behavioral changes within a relatively small portion of samples.
weaknesses: This paper tackles the LM problem from the perspective of anytime-valid hypothesis testing. The methodology design follows a rigorous manner, and I only have a few concerns regarding some technical details and presentation issues.

1. Although the authors provide intuitive examples for two use cases, internal audit and external audit, at the beginning of the paper, I would recommend giving a clear formulation for such two cases before stepping into examples. The descriptions of use cases provided in Section 4.2 actually give a better understanding.

2. Determining the tolerance parameter $\epsilon$ is vital in the proposed method as it decides the maximal neural net distance allowed between distributions (model behaviors). However, throughout the methodology section and experiment design, this parameter seems manually determined according to specific tasks. The authors may elaborate more on the design of this parameter to enhance the generability of the proposed method.

3. In addition, the authors mentioned in Section 4.2 that "To determine a tolerance parameter ε for this toxicity setting, we consider triggering the test if a model differs by more than the aligned Llama3 model with different sampling parameters." Is there any rationale to support such a design? What do the sampling parameters stand for, and why are they important in this context? (Table 1 in Appendix A.2 do not give intuitive instructions)

4. The experiment design for the false positive rate in Section 4.1 is somewhat rough and lacks details. In particular, What and in what format are the texts generated from the initial aligned models, and how are they valid to examine the false positive rate of the exact from different aspects?

5. A short description of the five instruction-tuned models can benefit the readability of the paper as they are used in most experiments.


6. For the 2nd use case (internal audit), the authors set $\epsilon$ as "the neural net distance between Llama3 using simple prompts, and Llama3 using few-shot prompts". Why do you use the neural distance between models with simple prompts and few-shot prompts?

7. The authors provided the link to the replication package, but the link seems incomplete.

8. The presentation of the paper is mostly in a good manner. Nevertheless, there are still some typos and grammar issues (e.g., " T This includes evaluating" in the abstract).

Meta-review: The authors consider audits on identifying changes to the behavior of an LM over time, using ideas from statistical hypothesis testing to identify precisely when behaviors shift. They assume the existence of some measurable 'behavior' (e.g. toxicity classifier) that they want to track, and flag when the distribution of responses on that behavior differs. 

Transparency into the behaviors of LLMs is important, and doing so in a precise, statistically rigorous way is useful when the models themselves can be stochastic and hard to test. There is also a nice anytime valid test that provides control despite the optional stopping problems that arise in these types of problems.

As a weakness, reviewers brought out issues with the validity of the behavior model, as well as some writing comments.

Decision: Accept (Poster)