Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test
Authors: Xiaoyuan Zhu
Yaowen Ye
Tianyi Alex Qiu
Hanlin Zhu
Sijun Tan
Ajraf Mannan
Jonathan Michala
Raluca Ada Popa
Willie Neiswanger
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
API auditing
untargeted fingerprinting
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: As API access becomes a primary interface to large language models (LLMs), users often interact with black-box systems that offer little transparency into the deployed model. To reduce costs or maliciously alter model behaviors, API providers may discreetly serve quantized or fine-tuned variants, which can degrade performance and compromise safety. Detecting such substitutions is difficult, as users lack access to model weights and, in most cases, even output logits. To tackle this problem, we propose a rank-based uniformity test (RUT) that can verify the behavioral equality of a black-box LLM to a locally deployed authentic model. Our method is accurate, query-efficient, and avoids detectable query patterns, making it robust to adversarial providers that reroute or mix responses upon the detection of testing attempts. We evaluate the approach across diverse query domains and threat scenarios, including quantization, harmful fine-tuning, jailbreak prompts, full model substitution, showing that it consistently achieves superior detection power over prior methods under constrained query budgets.

Published as a conference paper at ICLR 2026
AUDITING BLACK-BOX LLM APIS WITH A RANK- BASED UNIFORMITY TEST
Xiaoyuan Zhu'
Yaowen Ye
Tianyi Qiu Hanlin Zhu Sijun Tan
Ajraf Mannan' Jonathan Michala' Raluca Ada Popa² Willie Neiswanger'
Uniiersiity of Southern California University of California, Berkeley PPeking University
xzhu9839@usc. edu, elwinoberkeley. edu, qiutianyi. htt.yii.tt@@mmall.... com (hanlinzhu, si Oberkeeey. edu, {amannan, mi c raluca@eecs. berkeley. edu, neiswang@usc. edu i Co-second authors. $ Co-third authors.
ABSTRACT
As API access becomes a primary interface to large language models (LLMs), users often interact with black-box systems that offer little transparency into the deployed model. To reduce costs or maliciously alter model behaviors, API providers may discreetly serve quantized or fine-tuned variants, which can de- grade performance and compromise safety. Detecting such substitutions diffi- cult, users lack access to model weights and, in most cases, even output logits. To tackle this problem, we propose Rank-based Uniformity Test (RUT) that can verify the behavioral equality of black-box LLM locally deployed authentic model. Our method is accurate, query-efficient, and avoids detectable query pat- terns, making it robust adversarial providers that reroute or mix responses upon the detection of testing attempts. We evaluate the approach across diverse query domains and threat scenarios, including quantization, harmful fine-tuning, jail- break prompts, and full model substitution, showing that it consistently achieves superior detection power over prior methods under constrained query budgets. We release our code at https:////t thub. www.ccoc/ccccc http//yyhh12123p 3/UU..
INTRODUCTION
APIs have become central access point for large language models (LLMs) in consumer ap- plications, enterprise tools, and research work- flows (Anysphere Inc., 2025; Yun et al., 2025; ResearchFlow, 2025). However, while users query black- box APIs, they have little to visibility into the underlying model implemen- tation. Combined with the high cost of serv- ing large models and the latency pressure to re- duce -onetto-fisttt (TTFT), API providers are incentivized to deploy smaller or quan- costs.
1.5-
RUT(Ours)
Stalistical Power (P)
MMD (Gao 2025)
0.8-
0.4-
0.2-
0.0
0.2 0.4 0.6
6.8
Proportion of API Responses Incorrect Model
tized variants of the original model Such modifications, while opaque to end users, can degrade model performance and introduce safety risks (Egashira et al., 2024). In more concerning cases, providers may incorporate harmful fine-tuning, jailbreak-enablee prompts, or even misconfigured system compo- nents without realizing it (mirpo, 2025).
Figure 1: Statistical power of different meth- ods in detecting substitution of the Gemma-2- 9b-it with its 4-bit quantized variant, as the pro- portion of API responses from the quantized model increases. Our method significantly out- performs MMD (Gao al., 2025) and the Kol- mogorov-Smirnoy (KS) baseline.
system
These risks highlight the need for LLM API auditing the task of checking whether a deployed model is claimed. Yet this is particularly challenging in the black-box setting: users typically lack access to model weights and receive only limited metadata (c.g., top-5 token probabbbbillitss) This necessitates detection methods that rely solely on observed outputs. However, even such output-

---

Published as a conference paper at ICLR 2026
level methods face potential evasion: if the detection relies invoking LLM API with specially constructed query distributions, a dishonest API provider could detect the special pattern and reroute those queries to the original model they claim to serve. Worse still, even without knowing the detection strategy, an API provider could mix multiple models, making the response distribution harder to distinguish.
In this work, we focus on auditing API providers that serve open-weight models, and formulate the LLM API auditing model equality test following Gao et al. (2025): given query access to target API and a certified reference model of the expected configurations, the goal is to determine if the two produce statistically indistinguishable outputs on shared prompts. The auditor assumed to have full access to faithful reference implementation of the claimed model, such as its decoding parameters and logit outputs.
We propose that model equality test for auditing LLM APIs must satisfy three key criteria: ac- curacy, query efficiency, and robustness to adversarial attacks. Accuracy reflects how reliably a test can be used in practice. Query efficiency is critical for reducing operational overhead, which incentivizes more audits to ensure API models' integrity. Robustness is equally essential for real- world deployment, where audits must both evade detection by adversarial API providers and remain effective under targeted attacks.
While several methods have been proposed for model equality testing, they each fall short in one or more of these criteria (Table 1). Existing methods include Maximum Mean Discrepancy (MMD) (Gao et al., 2025), trained text classifiers (Sun et al., 2025), identity prompting (Huang al., 2025), and benchmark performance comparison (Chen et al., 2023a). However, Sun et al. (2025) require prohibitively many API queries; Huang et al. (2025) fail capture model variations such as size, version, or quantization (Cai et al., 2025); and Gao et al. (2025) and Chen et al. (2023a) rely on special query distributions that can be ad adversarially detected and bypassed via prompt caching (Gu et al., 2025).
Driven by these limitations, we pro- pose a Rank-based Uniformity Test (RUT)-an an asymmetric two-sample hypothesis test that addresses all three criteria simultaneously. In RUT, we sample one response from the target API and multiple responses from the reference model for each prompt, then compute the rank percentile of the API output within the reference distribution. If the target and reference models are identical, the percentiles should follow a uniform distribution. We detect deviations using the Cramér-von Mises test (Cramér, 1928). Our method requires only single API call per prompt, operates effectively on real-world, user-like queries, and avoids detectable patterns that adversarial providers might exploit.
Table Comparison of LLM auditing methods by accu- racy (Acc.), query-efficiecy (Q-Eff.), and robustness to adversarial providers (Rob.).
Method
Acc.
Q-Eff.
Rob.
RUT (Ours)
/
MMD (Gao et al., 2025)
x
Classificr (Sun et al., 2025)
X
/
ddentity-rrompiinn (Huang et al., 2025)
X
Benchmark (Chen et al., 2023a)
X
We evaluate RUT across a range of adversarial scenarios in which the API provider secretly substi- tutes the claimed model with an alternative. In Section 5.2, we study the case where the substitute is a quantized version of the original model. In Section 5.3, we test models augmented with a hidden jailbreaking system prompt. In Section 5.4, we examine models finetuned on instruction-followii data. In Section 5.5, we consider substitution with a completely different model. Finally, in Section 5.6, we test the methods on additional query distributions of math and coding.
Under a fixed API query budget, we find RUT outperforms both MMD and a Kolmogorov-Smiret test (KS) baseline across all settings. It consistently achieves higher statistical power and shows greater robustness to probabilistic substitution attacks (Figure I). Morcover, when applied to five real-world API-deployed models (Section 5.7), method yields detection results closely aligned with other methods and is more robust over string- based metrics on minor decoding mismatches.
2

---

Published as a conference paper at ICLR 2026
To summarize, the main contributions of our work include:
A novel test for auditing LLM APIs. We propose RUT, an asymmetric two-sample-test that needs only one API call per prompt and operates effectively natural queries, achieving query efficiency and by-design robustness to adversarial providers.
2. Empirical validation across diverse threat models. We validate RUT under diverse settings, including quantization, jailbreaking, SFT, and full model replacement.
3. Cross-validated audit of live commercial endpoints. We benchmark RUT side-by-side with established tests (MMD and KS) on three major public LLM APIs and demonstrate its practi- cality real-world black-box settings.
2 RELATED WORK
LLM fingerprinting. Fingerprinting approaches focus on identifying L.LMs by analyzing their out- puts. Active fingerprinting involves injecting backdoor-lic like behavior (Xu et al., 2024) into an LLM via finetuning, embedding watermarks (Kirchenbauer et al., 2023; Ren et al., 2023) into model' S text generation process, or intentionally crafting prompts to elicit unique outputs from different LLMs (Pasquini et al., 2024). Passive fingerprinting. inherent patterns in LLM-generated text (Su et al., 2023; Fu et al., 2025; Alhazbi et al., 2025). This builds on the observation that LLMs expose rich classifiers to identify a model (Sun et al., 2025). While related, fingerprinting aims to authenticate the origin of the model and prevent publisher overclaim. Consequently, fingerprints are designed to remain stable under fine-tuning or deployment changes. This is opposite to our auditing objec- tive. Prior work (Cai al., 2025) also shows that fingerprinting methods fail to detect quantized substitutions.
the other hand, focuses analyzing the
disiraaiies"-dddiiiuuttiin quirks that allow
Auditing LLM APIs. A growing body of work investigates whether black-box APIs faithfully serve the advertised model. The most straightforward audit is to evaluate models' benchmark per- formance (art, 2025; Eyuboglu al., 2024; Chen et al., 2023b), but raw performance alone cannot expose covert substitutions or partial routing. Gao et al. (2025) formalizes the problem as Model Equality Testing and shows that a kcrDel-MMD test can already flag public endpoints that deviate from their open-weight checkpoints. Concurrently to work, Cai et al. (2025) investigate the model substitution setting and show that API providers can evade detection through strategies such model quantization, randomized substitution, and benchmark evasion. Building on these insights, we propose a method that is more robust to such attacks and extend the threat model to include a broader range of realistic scenarios, such as jailbroken maliciously finetuned models.
PROBLEM FORMULATION
In this section, we formalize the LLM API auditing problem as a black-box model equality test between a target API and fully accessible reference model.
Models. We denote an LLM as a conditional distribution <(y|=; ip) over text output y E y given an input prompt x X and decoding parameters ip (e.g., temperature, top-p). In all experiments, we control the decoding parameters to be the same across models. Two such models are involved in the test. The reference model Ret is fully accessible locally; the verifier can draw an unlimited number of samples Yeef ~ Teer( - 22; sp) for any z and obtain relevant logits. The target model Tig is exposed only through a remote API that returns exactly one completion lhat ~ Tigs( x; s) per call and charges for every invocation. The total API budget is N prompts.
Hypotheses. The model equality test is cast as composite two-sample hypothesis test. The null hypothesis
Ho: (ege( x; = .tत[( 1x Vx E X
states that the deployed model is behaviorally identical to the declared one.
H:: 3 E X' such that Tig(- x; x; # )er(( (x)pp) sp)
captures any observable deviation, whether arising from quantization, fine-tuning, prompt injection, or full model replacement.
3

---

Published as a conference paper at ICLR 2026
Adversarial model provider. A dishonest model provider may attempt evade detection through probabilistic substitution. We model this behaviour by
Tige( x; p)) (1 - (())) )e x; (o) ((x) Nanl | 4p),
where Tatt is an arbitrary alternative model (e.g., quantized or adversarially finetuned version of ret and : X' + j0, 1] is an unknown routing function that may depend on the prompt and even on past interaction history (e.g., routing based on prompt caching (Gu et al., 2025)). When g(x) is small but positive on non-negligible fraction of typical prompts, the mixture in equation 3 significantly complicates discrimination; nevertheless, sound detection procedure must retain statistical power under such adversarial conditions.
Detector and objectives. detector a decision rule yp DN - {0, 1 whose output means "reject Ho". Here, D denotes the data collected with N API queries and many local model queries. For a user-specified significance level a E (0, 1), the detector must satisfy the type-I error constraint Рги, [x(DN) = 1] I a. Subject to this guarantee, the verifier seeks to maximize the statistical power
B aat) = P)(p(DN) = =1].
H1
uniformly over a broad class of substitution policies (q. Tatr;; to do so with as few costly API queries N possible; and to employ prompts indistinguishable from ordinary user traffic, thereby prevent- ing the provider from detecting and rerouting "suspicious-looking" inputs.
4
METHOD
We introduce our rank- based uniformity test in Section 4.1 and empirically identify the most dis- criminative score function to maximize the detection power in Section 4.2.
4.1
RANK-BASER UNIFORMITY TEST (RUT)
To determine whether the target model Tig is identical to the reference model Taet we propose rank-based uniformity test. This method assesses how typical the target model's responses are under the reference model's generation distribution.
Response score function. To perform a statistical test, we first define a ccalar-valued score function f: y X' -> IR that maps model response and prompt real number. This function assigns score to each output given the prompt, i.e.,
s = fly. x). where I E X, y E SER.
ideal score function f should induce an injective mapping y r> (") .)) for any fixed prompt E X. Under this assumption, each distinct response corresponds to unique score value, ensuring that the score distribution fully characterizes the model's outputs.
Uniformity as test signal. For each prompt E X' we sample response lhat Teg a;; sp) and compute its scalar score Sig: (Mig, .). To assess how typical this response under the reference model. we evaluate its rank in the reference model's score distribution.
We define the cumulative distribution function (CDF) the reference model's scores as:
Frna (s (x) = Py-mee(((ss))) (f), < s].
Since [y, æ) takes values in a discrete set, Fry is a step function. To ensure the rank statistic continuously distributed under the null hypothesis, apply a randomized quantile residual (Dumn & Smyth, 1996) to extend the probability integral transform (David & Johnson, 1948) to discrete distributions. Specifically, define the rank statistic
Tigt ( (Sigg) +U -U-PP((M (f(,,)) stg) U ~~ Uniform[O, 1].
(4)
where (s = F (f(y, x) < Stge) is the left-limit of the CDF at Sig, and (Pf = Stg:) is
the probability mass at Sig-- Under the null hypothesis Teg = Tell this rank statistic rig. E [0, 1] is uniformly distributed.
4

---

Published as a conference paper at ICLR 2026
Conversely, suppose that Fig Uniform(0. 1] under the randomized quantile residual construction. Since the CDF (- z) is stepwise and -on-eecre.,,ng, a uniformly distributed Yug implies that the score Sigi follows the same discrete distribution as ste- By injectivity of f, this further implies that Vigt ~ Ter(' x;; s). and hence Figt Tref-
Thus, with injective score function f. testing the uniformity of fig as defined in equation 4 offers a valid signal for distinguishing Vigt from Tee--
Empirical approximation of F In practice, it is intractable to build the true CDF F (- r))
Instead, we approximate it using an empirical CDF from m reference samples for each prompt.
Given target response Yi Tigs xy;; 0) and reference responses Vij Feel( m;; 0) for
1,... m, we compute the scalar scores
$; 31), Sij f(br
We then define the randomized rank statistics r; E [0. as
m Σ j=1
m 1[ [ty} U, > 1(s; = )
m
where U; Uniform[0, 1] is independent random variable to break ties uniformly, and ensure ri an unbiased estimator of Vigs given the prompt Tz.
Discriminative score function via empirical selection. While ideal injective score function would guarantce sensitivity to any difference between Tigt and Tiet constructing such function for which we can calculate the CDF is generally infeasible in practice.
To ensure that our test remains practically effective, we instead require the score function to be sufficiently discriminative, in the sense that it induces distinct score distributions whenever Tea * Tg:- Formally, for fixed prompt x E X, let
Stee ar) with y ~ . a2; and
xv) with y Tgg( [;)
We say that f sufficiently discriminative if the distributions of Smm and Sty differ whenever Jet F Tg, i.e.,
Tref(- x5 (p) * (ege( x)
Ps # Ps
Under this condition, differences in response distributions are reflected in the score distributions, causing the ranks to deviate from uniformity.
Thus, we aim to find the most discriminative score function among several promising candidates through empirical experiments. Section 4.2, we compare five candidate score functions log- likelihood, token rank, log-rank, entropy, and the log-likelihood log-rank ratio (Su al., 2023) and find that log-rank the most discriminative in practice for separating responses by mief and Tigts and therefore adopt it in our uniformity test.
Full test procedure. We now present the full RUT procedure.
Let (x1, (II,..., 3n} c X be set of prompts. For each prompt 24, sample response from the target model,
y/ Tigal 0);
and m responses from the reference model,
Vi ) 2z -,....,,
We compute the log-rank scores
(1), 8ij = = (Wbi,
and the corresponding randomized rank statistics {r:)2_.-
We apply the Cramér-von Mises (CvM) test (Cramér, 1928) to assess the deviations between {re he and Uniform (0, 1). The test evaluates the null hypothesis
Ho rt Uniform(0. for all i.
5

---

Published as a conference paper at ICLR 2026
The CvM test statistic is defined as
2i
2
w²
+
T(i)
12n
2n
where r(1) S T(2) S. ... S T(n) are the ordered rank statistics.
To compute the p-value, we compare the observed statistic wat. to the distribution of the CvM statistic Whlll computed under the null hypothesis. The p-value given by
p-value PHo [wmill 2 wats]
We reject Ho and conclude that Tit and Tet are different if p-value < 0.05.
4.2 SCORE FUNCTION SELECTION
The RUT requires scalar score function f(y. 2). To identify function that best captures dis- tributional differences between models, we consider five candidate functions. Log-likelihood: log Teeryy zr). Token rank: the average rank of response tokens in y, where a token's rank its position in the vocabulary ordered by the meer's next-token probabilities. Log-rank: the average
of the logarithm of the token rank. Entropy: predictive entropy for y under [et())). Log-likelihood log-rank ratio (LRR): the ratio between log- likelihood and log-rank. (Su et al., 2023).
To identify the most discriminative score func- tion, we conduct a Monte Carlo evaluation con- sisting of 500 independent trials. In each trial, randomly select 10 prompts from the Wild- Chat (Zhao et al., 2024) dataset and sample 50 completions per prompt from both wee and Figh using fixed temperature of 0.5 and a maxi- mum length of 30 tokens. For each candidate score function, we compute the average AU- ROC (Bradley, 1997) across the 10 prompts for each trial, yielding distribution of 500 AU- ROC scores per function. The full algorithm to calculate per score function average AUROC included in Appendix A.1. Across different model comparisons, we find that log-rank con- sistently yields the most separable AUROC dis- tribution from 0.5, indicating the strongest dis- criminative power. Figure 2 shows an example comparing Gemma-2-9b-it with its 4-bit quantized variant. Based the results, select log-rank as the scoring function for RUT. Complete AUROC results are provided in Appendix A.2. We also present formal analysis of RUT's statistical power with the log-rank score function under adver- sarial perturbations in Appendix C.
loglikelhood
rank
logrank
entropy
60
20
0.50
0.51
0.52
0.53
0.54
0.55
0.5.
0.57
AUROC
Figure 2: Distribution of AUROC scores for five candidate score functions across 500 trials comparing Gemma-2-9b-ir and its 4-bit quantized variant. Log-rank achieves the most separable dis- tribution from the random level 0.5, indicating perior power in distinguishing different models.
5 EXPERIMENTS
In this section, we evaluate RUT across diverse model substitution scenarios, including quantiza- tion (Section 5.2), jailbrekks (Section 5.3), SFT (Section 5.4), full model replacement (Section 5.5), additional query domains (Section 5.6), and real-world API providers (Section 5.7). Detection per- formance is compared against MMD and a KS baseline using statistical power AUC as the primary metric. We also include case study on detecting decoding parameter mismatch in Appendix B.6 and demonstrate that RUT remains robust across models and query domains.
5.1 EXPERIMENTAL SETUP
To evaluate detection performance under adversarial conditions, we simulate probabilistic substi- tution attacks where fraction [0, 1] of API queries are routed to alternative model (c.g., quantized or tuned. For each value of q, we estimate the statistical power, defined as the proba- bility of correctly rejecting the null hypothesis when substitution is present. We then summarize the

---

Published as a conference paper at ICLR 2026
resulting power-substiution rate curve using the area under the curve (AUC) over q [0. 1]. The AUC ranges from to and reflects the method's ' ability to maintain high statistical power across varying levels of substitution, serving indicate more reliable and consistent detection performance. Figure shows an example comparing Gemma-2-9h and its 4-bit quantized variant. In the following experiments, we compute 95% con- fidence intervals for AUCs using bootstrapping. The 95% wilson confidence intervals for statistical powers are shown as shaded area on plots in Appendix B.
measure of robustness to such attacks. Higher values
Data. We use the WildChat dataset (Zhao et al., 2024), which contains real-world conversations between human users and ChatGPT. This dataset reflects authentic user behavior, ensuring the query distribution represents typical API traffic.
Baseline. For the detection methods (Sun et al., 2025; Gao et al., 2025) that are compatible with WildChat, We primarily focus on Maximum Mean Discrepancy (MMD) (Gao et al., 2025) as the baseline, as Sun al. (2025) is reported to fail to identify quantization (Cai al., 2025). We also tailor Molmogorov-Smimit
(KS) test baseline that uses the same information as RUT: it computes the log-rank scores from the reference model on both the target and reference model responses and applies the two-sample KS test (Darling, 1957) on these two sets of scores to estimate the p-value.
Test procedures. We apply consistent sample budget constraint on all tests. The implementation details of their test procedures are listed below:
Rank-Based Uniformity Test (RUT):
Each trial samples 100 prompts. We query each prompt
once to the target and 100 times to the reference model.
Maximum Mean Discrepancy (MMD): We apply the MMD test based on the character-level Hamming distance following Gao et al. (2025). Each trial uses 10 prompts, with 10 samples per prompt. We compute the MMD statistic and estimate the p-value via 500 random permutations. Kolmogorov-Smirno Test (KS); We use the same sampling setup RUT: 100 prompts per trial. I query to the target, and 100 to the reference model per prompt.
Across all models, we set the temperature to 0.5 and cap generation at 30 tokens. We use vLLM (Kwon al., 2023) on single A6000 for local inferences. Statistical power is estimated over 500 Monte Carlo trials as the proportion of trials correctly rejecting the null at a = 0.05. All tests were with Intel Xeon Gold 6230R @ 2.10H and 16 GB RAM.
5.2 DETECTING QUANTIZATION
We consider the setting where the API provider uses a quantized variant to substitute the claimed model. We evaluate three detection methods on quantized variants of Llama-3.2-3B-Instruce --id7--B---ttrut-0... and (eemma-2-9Bii comparing each model to its 4-bit and 8-bit quan- tized counterparts. As shown in Table 2a, none of the methods succeed in reliably detecting substi- tution for the 8- bit variants of Gemma and Mistral, where statistical power AUC remains near zero across the board. In the remaining four settings, RUT outperforms MMD and the KS baseline in three out of the four cases, demonstrating superior sensitivity to quantization-induced distributional shifts. Pitiss ff PbRE AKSes for AUCs are provided in Appendix B.I.
We consider the setting where the API provider secretly appends a hidden jailbreaking system prompt to user queries. To evaluate this scenario, we use two base models: -istral-7B-IInstrut-- and Gemma-2-99-it For each model, we construct a test using three representative jailbreaking prompts Dan, Anti-Dan, and Evil-Bot adapted from Shen al. (2024). As shown in Table 2b, all jailbreak cases are reliably detected, with power AUCs consistently above 0.75. RUT achieves the highest power all settings, demonstrating its superior sensitivity to model deviations caused by hidden jailbreaking prompts. Full statistical power curves for AUCs are provided in Appendix B.2.
5.4 DETECTING SFT
We study the setting where the API provider fine-tunes a model on instruction-followin data. Specifically, we fine-tune two base Loodels--Lamma-.2-333 In-ttrucctt and Mistral-7B -n-strcct-
httpss//wugiinniccccccccttlll//a Instruct

---

Published as a conference paper at ICLR 2026
(a) Statistical power AUC for detecting quantized variants. Bold method; gray none reliable.
(b) Statistical power AUC for detecting jail- breaking prompts. Bold most effective method per prompt.
Model
RUT
MMD
KS
Model
Prompt
RUT
MMD
KS
Gemma-4br
0.392
0,214
0.017 $2 34
Dam
0.895 46 45
0.802
0.873 37 36*
64 10s+
110 112
6a 55
Gemma 8bit
0.049 s
0.043
0.001 10
Mistral
Anti-Dan
0.893 45 41
0.781
0.872
64
52 65
4e 491
Llama-4bit
0.642
0.625
0.474
Evil-Bot
0.892
0.766
0.873 ST art
98*
44
bo
Llama-8bit
0.132
0.158
0.005
0.888
0.757
0.867
Dam
Mistral-4bit
0.586 56 94
52" 97*
0.500
100 102+
0.330
Gemma
Anti-Dan
49 52
45 44
0.858
66 68*
0.816
52
34- aot
0.854
art
Mistral-Sbit
0.049 56 61
77 84
0.090
20 22
0,006
Evil-Bot
0.893
42"
0.753
63*
0.871 36 a6t
Table 2: Statistical power AUCs. All confidence interval ranges are reported in x10
v0.3-on on benign and harmful instraction-followin datasets. We use Alpaca (Taori et al., 2023) as the benign dataset and BeaverTails (Ji et al., 2023) for harmful question answering. Each model is fine-tuned on 500 samples from the respective dataset for 5 epochs using LoRA (Hu et al., 2021) with rank 64 and a = 16, a batch size of 32, and a learning rate of x 10 on a single A100. For each checkpoint, compute the statistical power AUC of the detection methods.
As shown in Figure 3. RUT con- sistently achieves higher power AUC than both the KS and MMD base- lines across all fine-tuning configura- tions. Notably, behavioral changes within the first epoch of fine-tuning, demonstrating strong sensitivity to early-stage distri- butional shifts. While all methods im- prove with additional training. RUT remains the most robust across both models and datasets. Full statistical power curves for AUCs are provided in Appendix B.3.
Liama Beawer
Liama a Alpaca
1:30
0.75
0.50
method detects
0.30
Mistral BeaverTall
Alpaca
Statpiccl
1.30
0.73
0.30
0.35
Epoch
RUT(Ourr))
KS
NHD (Can et 20251
Figure 3: AUC of SFT checkpoints across epochs.
5.5 DETECTING FULL MODEL REPLACEMENT
RUT
Gomma
0.048
0.828
0.886
0.019
0.743
0.007
0.749
0.864
0.8
Gemma 98
0.870
0.053
0.895
0.894
0.8
888
0.660
0.019
0.776
0.759
0.728
0.759
0.000
0.069
0.863
0.8
Reference
0.885
0.047
0.774
0.783
0.800
0.870
0.786
0.855
Llama-38
0.89
0.869
0.762
Uama-118-
0.895
0.80
891
0.852
0.052
0.766
0.712
0.758
0.734
0.026
0.877
0.873
0.874
0.774
0,003
Gammaa-3 28
Gummaa-
Mestral-1
Liema-J 36
Uaana L1B
Gemma-28
Gemmaa58 98
Mistell788
Lamaa-38
Uama 118
Gemias-3 28
Gemma-ii 9g
Mitrra-78
Liama 36
Lama 11a
Target
Figure 4: AUC for detecting full model replacement. Each cell shows the AUC score between a reference and target model. Diagonal values represent self- self-comparisons.
We evaluate the setting where the API provider substitutes the claimed model with a completely different one. To simulate this scenario, conduct pairwise comparisons among five open-
8

---

Published as a conference paper at ICLR 2026
source models: 1ettta...----Instiiiii Llama-3.2-11B-Viist Vision-Instruct', M Gemma-2-2B-it and Gemma-2-9B-iz For each pair, one model serves as the reference model while the other acts the deployed target model. As shown in Figure 4, RUT consistently achieves the highest statistical power AUC across model pairs, outperforming both the MMD and KS base- lines. The results highlight the method's sensitivity to full model substitutions. Full statistical power curves for AUCs are provided in Appendix B.4.
5.6 DETECTING QUERY DOMAINS
We now evaluate the robustness of RUT across more query domains. Beyond WildChat which reflects general conversational traffic, we consider two specialized datasets: BigCodeBench (Zhuo al., 2024) for programming tasks and MATH (Hendrycks al., 2021) for mathematical problem solving. We adopt the quantization setup in Section 5.2 as this setting is both challenging and practically relevant. See full power curves in Appendix B.5.
As shown in Table 3, the overall detection powers mirror those observed in Section 5.2. Detection remains difficult for 8-bit quantized Gemma and Mistral, where all methods fail to achieve meaning- ful power. RUT consistently shows high detectability in the remaining cases, outperforming MMD and KS in seven out the eight cases. These results show that RUT remains effective in both math and code domains, reinforcing its generalizability diverse query distributions.
Table 3: Statistical power AUC for detecting quantized variants across query domains. Bold = best method; gray none reliable. All confidence interval ranges are reported in x10
(a) BigCodeBench (Zhuo al., 2024)
(b) Math (Hendrycks al., 2021)
Model
RUT
MMD
KS
Model
RUT
MMD
KS
Gemma-4hh 4bit
0.136
0.170
0.000
Gemma-4bit
0.297
0.237
0.059 78
96
18s
Gemma-88e Sbit
0.048 71
0.052,
0.000
Gemma- 8bit
0.048
0.067
0.002
33
Llaam-4bit
0.593 54 50
0,353
0.524
Llama-4bit
0.532
0.468 101 sh*
0.412
81 #ut
48 5a
99 U4*
45 44*
Llama-808
0.093 34 33 0.462 76 84
0.143
0.021
Llama-800 8bit
0,253 72 73
0.177 65 66
0.118 75 80
56 58
12 23
Mistral-4bit
0.326
0.183
Mistral-4bit
0,360 33 37
0,216 66 05
0,160
91 31""8" 6
48 s1t
83 34
Mistral 8bit
0.041 74
0.058
0.011 63
Mistral Sbit
0.059 47
0,060 58
0,019
691
45 43
5.7 DETECTING REAL API PROVIDERS
We evaluate three Models-Llaama
Table Statistical power for detecting differences from the model deployed on an A6000 GPU. A100 denotes the same model on an A100 GPU. Values > 0.5 indicate
3.2-3B, Mistral-7B, and Gemma-2- 9B Oc-assoss multiple API providers, us- ing local A100 inference the base- line. As shown in Table 4, all tests cor- rectly confirm behavioral equivalence in local deployments.
significant behavioral deviation. Green = no significant
difference; Red significant difference.
Model
Provider
RUT
MMD
KS
Across all settings, RUT and MMD generally agree in detecting significant deviations across providers, offering mutual validation for their behavioral sensitivity. The KS test exhibits similar trends but with notably lower sensitiv- ity. One exception is Mistral + HF In- ference, where MMD yields a power of 1.0 while others remain below 0.2. We trace this tokenization mismatch: the HF Inference API consistently omits the leading whites-
Llama
A100
0.094
0.142
0.002
Llama
Nebius
0.962
0.944
0.426
Llama
Novita
0.988
0.996
0.530
Mistral
A100
0.058
0.138
0.004
Mistral
HF Inf.
0.188
1.00
0,000
Gemma
A000
0.060
0.084
0.000
Gemma Nebius
0.312
0.432
0.008
pace present in the reference outputs, Since MMD uses string Hamming distance, the formatting
3.2- 2--11B-Visionn Instruct httpp:///aggiingfacc.cooogggg 2-2b-it

---

Published as a conference paper at ICLR 2026
difference inflates the score. After restoring the missing space, the MMD score drops to 0.211, aligning with other tests. This shows the robustness of RUT to minor decoding mismatches that can mislead string string-based metrics.
CONCLUSION
The stable increase in the size (Kaplan et al., 2020) and architectural complexity (Zhou et al., 2022) of frontier LLMs has led to rise in the popularity of API-based model access. To prevent per- formance degradation and security risks from model substitution behind API interfaces, this work proposes the rank-based uniformity test for model equality testing. We test the method against a variety of different substitution attacks and demonstrate its consistent effectiveness in detecting substitution and its superiority over existing methods.
We have not empirically validated effectiveness of RUT against an adversary with full knowledge of the auditing method. For example, an attacker could selectively reroute prompts that are expected to produce atypical log-rank statistics. Assessing the method's robustness against more powerful adversaries is an important next step. In addition. RUT requires access locally deployed authentic reference model, which limits its applicability to open-sourced models. Exploring ways to relax this requirement would broaden the method's 's applicabilityy
Limitations and Future Work
By developing an effective and stealthy API-based test for model equality, we hope to advance the safety and security of L.LM-base applications in the age of increasingly cloud-based deployment.
ACKNOWLEDGEMENTS
Thanks Muru Zhang and Oliver Liu for valuable discussions. The work was supported in part by the National Science Foundation (NSF) under Award No. 2427856. Any opinions, findings, and conclusions or recommendations expressed this material are those of the author(s) and not necessarily reflect the views of NSF
REFERENCES
Artificial analysis, 2025. URL UBL.Aatpss//rrtfiiilaaayyyys...
Saeif Alhazbi, Ahmed Mohamed Hussain, Gabriele Oligeri, Panos Papadimitratos. Lims have rhythm: Fingerprinting large language models using inter-token times and network traffic anal- ysis. ArXiv, abs/2502.20589, 2025. URL http:::/ppp....mmiiiiccoool....
semanticscoolar.orgg wtt.:llraa..g/oopussts11
276725236.
Anysphere Inc. Cursor: The ai code editor, 2025. URL https:::///: www.w.rwwo..o........o.mmmo.oo.. Accessed: 2025-05-15.
Andrew Bradley. The use the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145-1159 1997. ISSN 0031-3203. doi:
http:://oi.org o00/ 10001016000031132000000000001000 sccmmc///rriideppp////0333333333
URL 6, 13
Will Cai, Tianneng Shi, Xuandong Zhao, and Dawn Song. Are you getting what you pay for? auditing model substitution in Ilm apis, 2025. URL htttt::ss//aiii/t www.xii.rrg/ab//25555 2205500441..... 04715. 2,
3,
Lingjiao Chen, Matei Zaharia, and James Zou. How chatgpt's behavior changing over time?, 2023a. URL https::/arxiiv.orr abtp://bss233/// /tt.//230700000.... 09009.
Lingjiao Chen, Matei Zaharia, and James Y. Zou. How is chatgpt's behavior changing over time? ArXiv, 000/2307.09009) 2023b. URL attps://apy semanticscholar. org/CCrpusID 259951081.
Harald Cramér. On the composition of elementary errors. Scandinavian Actuarial Journal, 1928 (1):13-74, 1928. doi: 10.1080/03441238. 2w1.23....22.... 1928.128.... 10416862. 1886. URL https: http://ooooor////////////.. 1080/ 03461238. 1928. 10416862. 2,
10

---

Published as a conference paper at ICLR 2026
D. A. Darling. The kolmogorov-sminnxx cramér-von mises tests. The Annals of Mathematical Statistics, 28(4):823-88) 838,,888888--- 1957. ISSN 00034851, 21688990. URL http://wwwt http://ww..ssorrr.oo..oooooooo.. org/ stable/223700
F. N. David and N. L. Johnson. The probability integral transformation when parameters estimated from the sample. Biometrika, 35(1/2):182-190, 1948. ISSN 00063444. URL
http://www. jst/or wwt.orgg//aaa org//stable/233268..
Peter K. Dunn and Gordon K. Smyth. Randomized quantile residuals. Journal of Computational and Graphical Statistics, 5(3):236 244, 1996. ISSN 10618600. URL http:httpp/wtt jstor org/
stable/ (table/139062.. 1390802. 4
Kazuki Egashira, Mark Vero, Robin Staab, Jingxuan He, and Martin Vechev. Exploiting Ilm quan- tization, 2024. URL https: /tattps://rrii..../ .oriiiiooor///oeeee orgoorgg//boo .ttps//205.11111
Sabri Eyuboglu, Karan Goel, Arjun Desai, Lingjiao Chen, Mathew Monfort, Chris Ré, and James Zou. Model changelists: Characterizing updates ml models. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024. URL https: //api. semanticscholaa orgg/orrpuID:227228:
Zhiyuan Fu, Junfan Chen, Hongyu Sun, Ting Yang. Ruidong Li, and Yuqing Zhang. Fdllm: A text fingerprint detection method for llms in multi-language, multi-domain black-box environ- ments. ArXiv, abs/2501.16029, 2025. URL https://aaappt wwwp.sssaaaiiscoll...... orr.olorurrr/CrrrII
27 275921293. 3
Irena Gao, Percy Liang, and Carlos Guestrin. Model equality testing: Which model is this api serving?, 2025. URL https://arxiv http::///aaiio.oo/aa//////1//1 wttp/rg////...../////////
20247. 3.
Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, and Tatsunori Hashimoto. Auditing prompt caching in language model apis, 2025. URL 2.4
Dan Hendrycks. Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021. 9, 24
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration, 2020. URL pottps://arxiyy orgg/bss/900e /9044 09751. 24
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhur -hu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//arxiv. wrg. abs 2106. 09685. 8
Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Christo- pher A. Choquette-Chee -hho, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Ziyu Liu, Ion Stoica, Florian Tramer, and Chiyuan Zhang. Exploring and mitigating adversarial manipulation of voting-based leaderboards, 2025. URL htttt::tt//arttt
org/abs/22 abs/ 07493. 2
Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of 11m via numan-preferenee dataset. arXiv preprint xxXiv:2377.06657 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0834 2020. 10
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning, 2023. URL httpp:///appp www.nti wwwaaar.rg/ooooooo wrgg/CorpsSI::202. 25619417..9
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
11

---

Published as a conference paper at ICLR 2026
mirpo. [bug] Im studio's 's api server ignores the requested model name in the api call when only one bug-tracker, 2025. URL https: h/thp::://gihhbb http://studio-aammmttteee ai/lmstudio-bur bug-traackerr/sssss//113
model running issue #619
Dario Pasquini, Evgenios M. Kornaropoulos, and Giuseppe Ateniese. LImmap: Fingerprinting for
large language models. ArXiv, abs/2407.15847,, 2024. URL
Steven T. Piantadosi. Zipf's word frequency law in natural language: critical review and future directions. Psychonomic Bulletin & Review, 21:1111 1130, 2014. URL https:
//api. wwmanticscholar..t.......ma orr..aarurrg//ruuIDD::...
14264582. 26
Jie Ren, Han Xu, Yiding Liu, Yingqian Cui, Shuaiqiang Wang, Dawei Yin, and Jiliang Tang. A robust semantics-based watermark for large language model against paraphrasing. In NAACL- IILT, 2023. URL pattts///appt www.seeemmaeootii...ee wtt.ccoo......g/orrpsti......... org/CooppusD::25213088
ResearchFlow. Researchflow: Ai-powered research engine & visual knowledge mapping, 2025.
URL https::::/rffl ai/.
Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. "do anything now": Char- acterizing and evaluating in-the-wild jailbreak prompts large language models, 2024. URL
https://arxiv.oo www./ooggabbao abss//abs///a att.absssaa 2200. 03825. 7
Jinyan Su, Terry Yue Zhuo, Di Wang, and Preslav Nakov. Detectlim: Leveraging log rank informa- tion for zero-shot detection of machine-generated text, 2023. URL wttpt:t/p/aaxiii org/ abs/ 2306. 05540. 5,5,
Mingjie Sun, Yida Yin, Zhiqiu Xu, J. Zico Kolter, and Zhuang Liu. Idiosyncrasies in large language models, 2025. URL. hwtphhts///aat //axxiv wwtpeiiiv.rr///beee org/ htggabb//502... 12150. 3,
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followee llama model. https:/ htttp:://gituu..oooo wightt.......
2023. 8
Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instructional fingerprinting of large language models. ArXiv, abs/2401. 2024. URL
www.stieschoooooooe.. http:://ppp.eemanticcchoaaarr orgo /Cto//cuII::20009522220
Yuhui Yun, Huilong Ye, Xinru Li, Ruojia Li, Jingfeng Deng, Li Li, and Haoyi Xiong. Eicopilot: Search and explore enterprise information over large-scale knowledge graphs with Ilm-driven agents, 2025. URL https://arxxiv wttpxiiv..rg/bb//500001 13746.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild, 2024. URL httptt///////i...../// org abs/2405 01470. 6,
7, 24
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022. 10
Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indrancil Paul, et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint 158777 2024. 9
12

---

Published as a conference paper at ICLR 2026
A AUROC
A.1 AUROC ALGORITHM
Algorithm 1: Average AUROC for score function evaluation
Input: Prompt set D; models Trafs Tigi decoding parameters y (T, L), where is temperature and I. is the maximum generation length;
number of prompts 74; number of completions per prompt per model score functions (P11...., (p.)
Output: Mean AUROC per score function, denoted (AUROC(Ó).
Draw {21,..., Fn.
Uniform(D):
for i E (1,..,,m n do (3)
(Uret
(3) f==1 Terl [ 4):
(veer f==1 Tgg(- x;;;));
U Vi + (u (be") ): Li + [0]"
fo foI..... px) do
S. + (6()) E Pi: Store A + AUROC(S,, L.):
for 8 E (.11...... 9x do MAUROC (8) + Σ A7:
Note. AUROC(S, denotes the standard binary AUROC (Bradley, 1997).
A.2 AUROC SCORE DISTRIBUTIONS
We present the AUROC score distributions from the score function selection experiment described in Section 4.2. Specifically, we evaluated Gemma-2-9B-ir, LLaMA-332-BB-Instnee 7B-Instruct, and visualized the distributions when distinguishing the original model outputs from three types of variants: (1) quantized versions, (2) models subjected to jailbreaking prompts, and (3) models served by A100 or external API providers.
and Mistral-
Gemma9B 4bit
Gemma9B 8bit
120
leglikelihood
loglikelhood
rank
100
rank
logrank
logrank entropy
entropy
80
Irr
Irr
60
40
20
0.51
0.52
0.55
0.57
0.500
0.520
0.520
0.525
0.530 0.535
AUROC
AUROC
aurDc Gemma9B
auroc Dan
loglikellhood
100
logrank entropy
80
Irr
loglikellhood rank
logrank entropy Irr
13

---

Published as a conference paper at ICLR 2026
auroc Gemma98
auroc Gemma98
teglieelhood
25
logrank
entropy
20
log loplikelhoodd
rank
logrank
entropy
auroc Gemma90 98 Nebius
auroc Liama3B 4bit
loglikallhaad
logiikelih.ok
100
rank
rank
20-
80
entropy
entropy Inr
Irr
60
10
0.50
0.51
6.52
0.53
0.54
0.55
0.500
0.525
0.550
0.575
0.600
0.625
0.650
0.675
0.700
AUROC
AUROC
auroc Llama3B abit
auroc Llama3B a100
logllkellhood rank
logllkollhood
rank logrank entropy
logrank
entropy
80
60
40
0.50
0.51
0.52
0.53
0.54
0.55
0.56
0.50
0.51
0.52
0.53
0.54
0:55
AUROC
AUROC
Ulama38 Nebius
Llama38 Novita
100
leglikelihodd
100
loglikalhood
legrank
80
logrank
entropy
entropy
60
Irr
40-
20
20
0.54
0.58
0.52
0.54
AUROC
AUROC
auroc 78
auroc
70
loglikellhood
100
lnglikelhhood
60
logrank entropy
lagrank
50
entropy
40
20
14

---

Published as a conference paper at ICLR 2026
auroc Mistral7B
auroc Mistral
250
100
loglieelhhodd
logikellhood
80
logrank entropy
200
logrank
entropy
100
20
auroc Mistral7B Dan
auroc Mistral7B Evil Bat
laglikelihod
350
loglkellhood
rank
300
rank
logrank
entropy
entropy
60
150
40
100
20
50
0.75
0.80
0.65
0.90
0.35
1.00
0.85
0.86
0.90
0.92
0.9
0.56
0.95
1.00
AUROC
AUROC
auroc Mistrai78 78, HFinference
logllkellhodd
rank
logrank
entropy
30
10
0.50
0.52
0.54
0.56
0.58
0.60
0.62
0.64
0.66
AUROC
B STATISTIC POWER CURVES
B. FULL STATISTIC POWER CURVES FOR DETECTING QUANTIZATION
We present the full statistical power curves, showing the relationship between substitution rate and detection power, corresponding the experiments on detecting quantized model substitutions de- scribed in Section 5.2. These curves are used to compute the power AUC values reported in the main paper and illustrate each method's detection power across different levels of substitution.
Gemma-4bit
Gemma-8bit
(Ours)
RUT (Ours)
Stalistical Power (t)
MMD
MMD
0.8
KS
0.8
0.6
0.2
0.2
0.4
0.0
Proportion Responses
Incorrect
Proportion Responses
15

---

Published as a conference paper at ICLR 2026
Uama-8bit
RUT (Ours) MMD
RUT (Ours)
Stalistical Power (t)
KS
2.6
0.6
0.0
0.0
0.4
0.8
0.0
0.2
1.0
Proportion Responses Incorrect Model
Proportion of API Responses Incorrect Model
Mistral-4br --biit
Mistral-8bit
(Durs)
RUT (Ours)
Statistical Power (+)
MMD
MMD
0.8
KS
0.8
0.2
0.2
0.8
Proportion Responses from Incorrect
Proportion API Responses from Incorrect Model
B.2 FULL STATISTIC POWER CURVES FOR DETECTING JAILBREAKING
We present the full statistical power curves, showing the relationship between substitution rate and detection power, corresponding to the experiments on detecting jailbreak prompts described Sec- tion 5.3. These curves used compute the power AUC values reported in the main paper and illustrate each method's 's detection power across different levels substitution.
Gemma, Dan
Gemma Anti Dan
(Ours)
(Ours)
Statistical Power (t)
MMD
Stalistical (t)
MMD
0.8
0.8
024
0.2
0.2
0.4
0.2
Proportion API Responses from Incorrect Model
Proportion of API Responses from Incorrect Model
Gemma Evil Bot
Mistral Dan
3.0
RUT (Ours)
10-
RUT (Durs)
Statisiical Power (t)
MMD
Statislical (T)
MMD
KS
0.8
0.4
0.4
0.0
0.4
0.8
0.2
0.5
0.5
1.0
Proportion of API Responses from Incorrect Model
Proportion of API Responses from Incorrect Model
16

---

Published as a conference paper at ICLR 2026
RUT (Ours) MMD
RUT (Durs)
Stalistical Power (t)
KS
2.6
0.6
0.0
0.
0.4
0.8
0.0
0.2
1.0
Proportion Responses Incorrect Model
Proportion of API Responses from Incorrect Model
B.3 FULL STATISTIC POWER CURVES FOR DETECTING SFT
We present the full statistical power curves, showing the relationship between substitution rate and detection power, corresponding the experiments on detecting SFT described Section 5.4. These curves are used to compute the power AUC values reported in the main paper and illustrate each method's detection power across different levels of substitution.
Liama Alpaca
Liama Alpaca 2
RUT (Durs)
RUT (Ours)
Stalistical Power (t)
MMD
MMD
KS
0.6
0.6
Stalistical
0.0
0.0
0.4
0.8
0.0
0.2
1.0
Proportion
Responses
Incorrect Model
Proportion API Responses Incorrect Model
Liama Alpaca 3
Liama_ Alpaca 4
(Ours)
(Ours) MMD
Statistical Power [t)
MMD
Stalistical (t)
0.8
0.8
124
0.2
0.2
0.4
0.2
Proportion API Responses from Incorrect Model
Proportion of API Responses from Incorrect Model
Ulama Alpaca 5
3.0
RUT (Ours)
Statisiical Power (t)
MMD
KS
0.4
0.0
0.4
0.8
Proportion of API Responses from Incorrect Model
17

---

Published as a conference paper at ICLR 2026
BeaverTails
Llama BeaverTails
RUT (Ours)
RUT (Ours)
Stalistical Power (t)
MMD
KS
2.6
0.6
0.0
0.
0.4
0.8
0.0
0.2
1.0
Proportion
Responses Incorrect Model
Proportion of API Responses from Incorrect Model
Llama BeaverTalls:
Liama BeaverTails.
RUT (Ours)
Statisiccl Power (+)
MMD
MMD
0.8
KS
0.8
0.2
0.2
0.8
Proportion Responses from Incorrect
Proportion API Responses from Incorrect Model
Llama BeaverTails.
(Ours)
Statisicall Power (t)
MMD
0.8
0.4
0.2
0.0
0.4
0.8
Proportion API Responses from Incorrect Model
Mistral Alpaca
Mistral Alpaca
3.0
RUT (Durs)
1.-
RUT (Durs)
Statiticall Power (t)
MMD
KS
0.4
0.4
0.0
022
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Proportion of API Responses from Incarrect Model
Proportion of API Responses from Incarrect Model
Alpaca
Alpaca
RUT (Durc)
RUT (Durs)
Statisticaa Power (t)
KS
0.6
0.6
Stalisüccll
0.6
0.4
6.0
0'4
0.8
0.0
0'2
0.5
1.
Proportion API Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
18

---

Published as a conference paper at ICLR 2026
Alpaca
RUT (Ours)
Stalistical Power (t)
MMD
KS
2.6
0.0
0.0
0.4
0.8
Proportion
Responses Incorrect Model
Mistral Beaver Tails
Mistral BeaverTails
RUT (Ours)
Statistccll Power (+)
MMD
MMD
0.8
KS
0.8
0.2
0.2
0.0
0.8
Proportion Responses from Incorrect
Proportion API Responses from incorrect Model
Mistral BeaverTails
Mistral BeaverTails 4
(Durs)
3.0
(Ours)
Statitical Power (t)
MMD
(T)
MMD
0.8
0.8
0.4
0.4
0.2
0.2
0.0
0.4
0.8
0.2
0.6
1.0
Proportion API Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
Mistral Teaver 5
3.0
RUT (Durs)
Statiticall Power (t)
MMD
KS
0.4
0.0
0.2
0.4
0.5
0.8
1.0
Proportion of API Responses from Incorrect Model
B.4 FULL STATISTIC POWER CURVES FOR DETECTING FULL MODEL REPLACEMENT
We present the full statistical power curves, showing relationship between substitution rate and detection power, corresponding to the experiments detecting full model replacements described in Section 5.5. These curves are used to compute the power AUC values reported in the main paper and illustrate each method's 'ss detection power across different levels of substitution.
19

---

Published as a conference paper at ICLR 2026
Germa22.Gemma22
Gemma2B Bemmma2B-GemmaBB
RUT (Ours)
RUT (Durs)
Stalistical Power (t)
MMD
KS
2.6
0.6
0.0
0.
0.4
0.8
0.0
0.2
1.0
Proportion Responses Incorrect Model
Proportion of API Responses from Incorrect Model
Gemmaa2B-LlaaaB
Gemma2 B-Lammmll
(Ours)
RUT (Ours)
Statistical Power (+)
MMD
MMD
0.8
KS
0.8
0.2
0.2
0.8
Proportion Responses from Incorrect
Proportion API Responses from Incorrect Model
MiMsstraii7
GemmaaB 9ennma99 Gemma2B
(Ours)
1.0
(Ours)
Statitical Power (t)
MMD
(T)
MMD
0.8
0.8
0.4
0.4
0.2
0.2
0.0
0.4
0.8
0.2
0.6
1.0
Proportion API Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
Gemma9B-Gemma9B
Gemma9B-Uama38
3.0
RUT (Durs)
1.0
RUT (Ours)
Statitical Power (t)
MMD
KS
0.4
0.4
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Proportion of API Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
Gemma9
MB
RUT (Ours)
RUT (Durs)
Statisticaa Power (t)
KS
0.6
0.6
Stalisüccll
0.6
0.4
6.0
0'4
0.8
0.0
0'2
0.6
1.
Proportion Resporses from Incorrect Model
Proportion of API Responses from Incarrect Model
20

---

Published as a conference paper at ICLR 2026
Liama 3ama388-
Bo-aa3--Gann Gemma98
RUT (Qurs)
RUT (Durs)
Stalistical Power (t)
MMD
KS
2.6
0.6
Stalistical
0.0
0.
0.4
0.0
0.2
1.0
Proportion Responses Incorrect Model
Proportion of API Responses from Incorrect Model
Ulama3B-Llama3
Llama3B-Llamal1e
RUT (Durs)
Statisiccl Power (+)
MMD
MMD
0.8
KS
0.8
Power
0.2
0.2
0.8
Proportion Responses from Incorrect
Proportion API Responses from Incorrect Model
(aima3B-Mistra777
Mammal1B-Gemma22
(Qurs)
1.0
(Ours)
Statitical Power (t)
MMD
(T)
MMD
0.8
0.8
0.4
0.4
0.2
0.2
0.0
0.4
0.8
0.2
0.6
1.0
Proportion API Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
Jama11B-Gemmas9
Llamal18-Ulama38
3.0
RUT (Durs)
1.0
RUT (Durs)
Statisticll Power (t)
MMD
KS
0.4
0.4
0.0
022
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Proportion of API Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
Llama118-Mistrar
RUT (Durs)
RUT (Durs)
Stalistical Power (t)
MMD
KS
0.6
0.6
Stalisüccll
0.6
6.0
0.4
0.8
0.0
0'2
0.5
1.
Proportion Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
21

---

Published as a conference paper at ICLR 2026
Mistral78- Gemmma28
9emmma9
RUT (Qurs)
RUT (Durs)
Stalistical Power (t)
MMD
KS
2.6
0.6
Stalistical
0.0
0.
0.4
0.8
0.0
0.2
1.0
Proportion Responses Incorrect Model
Proportion of API Responses from Incorrect Model
Mistrai7B-LLamaa38
Mistral7B-LIamal
(Durs)
RUT (Ours)
Statisticll Power (+)
MMD
MMD
0.8
KS
0.8
0.2
0.2
0.8
Proportion Responses from Incorrect
Proportion API Responses from Incorrect Model
Mistral7B-Misti 7i
(Ours)
Statisicaal Power (t)
MMD
0.8
0.4
0.2
0.0
0.4
0.8
Proportion API Responses from Incorrect Model
B.5 FULL STATISTIC POWER CURVES FOR DETECTING MORE QUERY DOMAINS
We present the full statistical power curves, showing the relationship between substitution rate and detection power, corresponding to the experiments on detecting quantized model in extra domains described in Section 5.6. These curves are used compute the power AUC values reported in the main paper and illustrate each method's 's detection power across different levels of substitution.
gemma-4bit- bigcodebench
gemma-4bit math
3.0
RUT (Durs)
1.0
RUT (Durs)
Staüstical Power (1)
KS
Staüsslicall
0.4
0.4
0.0
0.4
0.8
0.2
0.4
0.6
1.
Proportion of API Responses from Incorrect Model
Proportion of API Responses from Incorrect Model
22

---

Published as a conference paper at ICLR 2026
gemma-4bit- wildchat
gemma-8bit- bigcode
(Ours)
RUT (Ours)
Statistical Power (t)
MMD
MMD
0.8
KS
0.8
0.6
0.6
Stalistical
0.0
0.
0.4
0.8
0.0
0.2
0.6
1.0
Proportion Responses Incorrect Model
Proportion of API Responses Incorrect Model
gemma-8bit math
gemma-8bit wildchat
(Ours)
(Ours)
Statistical Power (t)
MMD
Power (1)
MMD
0.8
0.8
0.2
0.2
Proportion Responses from Incorrect
Proportion API Responses from Incorrect Model
Ilama-4bit bigcodebench
llama-4bit math
3.0
RUT (Ours)
10-
RUT (Ours)
Statisiccal Power (t)
MMD
(T)
MMD
KS
0.8
0.4
0.4
0.2
0.2
0.0
0.4
0.8
0.2
0.6
1.0
Proportion API Responses from Incorrect Model
Proportion of API Responses from incarrect Model
1lama-4bit- wildchat
Ilama-8bit bigcodebench
1.0
RUT (Ours)
1.0
RUT (Ours)
Statistical Power (1)
(T)
KS
0.6
0.4
0.4
0.0
0.2
0.4
0.5
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Proportion of API Responses from Incarrect Model
Proportion of API Responses from Incarrect Model
1lama-8bit math
llama-8bit wildchat
RUT (Ours)
RUT (Ours)
Statistical Power (t)
MMD
MMD
0.8
KS
0.8
0.6
3.6
Stalistical
6.0
0.
0.4
0.8
0.0
0.2
0.5
1.
Proportion Resporses from Incorrect Model
Proportion of API Responses from Incarrect Model
23

---

Published as a conference paper at ICLR 2026
mistral-4bit bigcodebench
mistral-4bit math
(Ours)
RUT (Ours)
Statisiical Power (t)
MMD
MMD
0.8
KS
0.8
0.6
0.6
Stalisiical
0.0
0.
0.4
0.8
0.0
0.2
0.6
1.9
Proportion Responses Incorrect Model
Proportion of API Responses from Incorrect Model
mistral-4bit wildchat
mistral-8bit bigcodebench
(Ours)
(Ours)
Statistical Power (t)
MMD
MMD
0.8
0.8
Power
0.2
0.2
0.8
Proportion Responses from Incorrect
Proportion API Responses from Incorrect Model
mistral-Bbit math
mistral-Sbit wildchat
1.0
RUT (Ours)
10-
RUT (Ours)
Statisiccal Power (t)
MMD
(T)
MMD
KS
0.8
0.4
0.4
0.2
0.2
0.0
0.4
0.8
0.2
0.6
1.0
Proportion of API Responses from Incorrect Model
Proportion of API Responses from Incarrect Model
B.6 CASE STUDY ON DETECTING DECODING PARAMETERS
Setup. As case study, we whether detection methods can identify changes in decoding pa- rameters, focusing sampling temperature and top-p for nucleus sampling (Holtzman et al., 2020). We compare responses generated under different parameter settings against the default configuration of temperature=0.5 and top-p=1.
Findings. We perform the experiments across Gemma-2-9B-it and Llama-3.2 -aataa...2----stteeeee on MATH (Hendrycks et al., 2021) and WildChat (Zhao et al., 2024). Based on the results in Fig- ure 5, RUT achieves consistently higher detection power across decoding configurations compared to MMD and KS. This sensitivity is desirable in practice, since when the API providers expose de- coding controls to users, reliable detection method should be able to flag deviations arising not only from model substitution but also from anomalous decoding configurations.
24

---

Published as a conference paper at ICLR 2026
RUT
MMD
KS
0.30
0.649
0.495
0.225
0.003
0.611
0.288
0.061
0.032
0.272
0.098
0.014
0.001
Temp.
0.631
0.086
0.246
0.630
0.483
0.067
0.206
0.177
0.269
0.004
0.044
0.407
1.00
0.042
0.042
0.681
0.50
0.75
0.90
1.00
0.59
0.75
0.30
1.00
0.50
0.75
0.90
1.00
Top-P
(a) Gemma
1aanma----9B--- 2-9R-it MATH
RUT
MMD
KS
0.50
0.645
0.521
0.1
0.044
0.651
0.234
0.073
0.030
0.245
0.076
0.018
0.009
Teep.
0.609
0.094
0.221
0.592
0.537
0.039
0.097
0.162
0.230
0.009
0.032
0.296
1.00
0.387
0.379
0 696
0.825
0.291
0.180
0.722
0.388
0.027
0.093
0. 0.511
0.730
0.50
0.75
0.90
1.80
0.50
0.30
1.00
0.50
0.79
0.90
1.00
Top-p
(b) 1amma-3.2-22 3--ntrttcct on MATH
2--mm-33---BInsstttcc
RUT
MMD
0.50
Temp.
0.015
0.003
0.019
0.335
1.00
0.154
268
0.363
0.403
0.003
0.134
0.549
0.732
0,50
0.75
0.90
1.00
0.50
0.75
0.30
0.50
0.79
0.90
1.00
Top-p
(c) Gemma-2 2-9B-it on WildChat
RUT
MMD
K5
0.30
0.698
0.213
0.069
0.018
0.382
0.175
Temp.
0.00
0.50
0.75
0,90
1.00
0.50
0.79
090
1.00
050
0.75
0.90
100
Top-p
(d) 1ama-332-3B-Instmucc on WildChat
Figure 5: Statistical power AUC for detecting decoding parameter mismatches (temperature, top-p) across models and datasets. Each cell compares outputs under specific decoding configuration against the default (0.5, 1.0); higher values indicate stronger detectability.
25

---

Published as a conference paper at ICLR 2026
C THERORETICAL ANALYSIS OF RUT
In this section, we provide formal analysis to characterize the statistical power of RUT and its sample efficiency under principled assumptions about language model output distributions and ad- versarial perturbations.
We model two basic attack types: (1) reparameterization attack, which preserve ranking of tokens but smoothly alter token probabilities via small parameter shift - + E; and (2) reordering attack, which randomly samples a token & according to the distribution, and then swaps with the highest-probabiliyy token, thereby making t the new top-1 token. Real-world attacks almost always combine both, and tend to easier to detect than either alone.
Throughout the proof, we assume that RUT uses the log-rank score function. Based on the famous Zip's law in statistical linguistics (Piantadosi, 2014), we assume that the output token distributions any position follows Zeta distribution with parameter s, where s E (1, +o0) reflects the heavy- tailedness of the distribution.
can be easily shown that the FPR of RUT a, the significance level that we choose. Below, we focus on type II enror: the probability of missing an attack when one exists.
Lemma C.1. Let the rank of a token be random variable K on Z- with probability mass function p(k) = k /C(ar) for a > 1. Let x = log k. The survival function S(x) = P(log K 2 x), for large k e",, has the asymptotic form
e(I-a)o S(x) ((a)(a- (1 1) (1))(1))
Proof. The survival function is the probability that the rank is greater than or equal to some value By definition, this is the sum of the probabilities of all ranks from m to infinity.
S(log P(K 2 m) Σ p(k)
((a)
For large m, this sum can be approximated by corresponding integral. The leading term of the Macla-Maclaurin formula shows that the sum and integral are asymptotically equivalent.
30
t dt 0m
aat11
+ +0m
Q
0 m
1-a
+ 00mm))
1
m
0(m -
m 1-x
(1 0(m-l))
m 1-
a- (1 +
o(1) o(1))
Substituting this result back into the expression for the survival function, obtain
(logm) = m 1-a
((a) (1 o(1))
26

---

Published as a conference paper at ICLR 2026
The analysis is performed the distribution of the log-rank I = log m, which implies m = e". Substituting this into the expression above yields the final asymptotic form for the survival function of the log-rank.
log We therefore let
(6 S(x) {(a)(a- 1) 1) (1 o(1)
ell cl1-a)a
((a)(a-1) - o
For remainder this analysis, we denote this survival function, which serves as the CDF for our rank-based test, by F(x,a)
Lemma C.2. For hypothesis test using the Cramér-von Mises (CvM) statistic to distinguish null hypothesis Ho from an alternative hypothesis H1 with a fixed significance level aery and fixed statistical power 1 - Berr, the required sample size n is assymptotically inversely proportional to the CvM distance w between the two distributions,
= θ
as + 0,
where the constant coefficient given by (1).
Proof. Let the ranks т; for i I..... be drawn from distribution with CDF F(x). The hypoth-
esis test is Ho : F(z) = (+)) versus H1 F(x) = (r)),, where Uxx) = x is the CDF of the Uniform[0,1] distribution. The CvM test statistic is
w² = =
(Fa(1) (Fa(x)-((z)) U(z))
where Fr, (x) is the empirical CDF, The test rejects Ho the observed statistic w² exceeds a critical value Ca. The power of the test is 1 Berr PH, (w² ca). The analysis of this power requires the distribution of w2 under H1.
While the distribution of 2 under Ho is non-normal, weighted sum of chi-squared variables, its distribution under a fixed alternative H1 asymptotically normal as n - so. As a standard result in statistics, this follows from the central limit theorems of U-statistics, which establish that the standardized statistic converges in distribution to a standard normal variable:
2 EH, [w
4, N(O, 1)
Varn, (w²)
The expected value EH, [w²] contains non-centrality parameter that grows linearly with n, such that EH, [w>] 2 (1), where the CvM distance w defined as
w = ([((((()) (F((s))((()) U(:))
The standard deviation, /Var H, (w²), grows slower than the mean, and the standardized deviation σ1 = VarH, (w²)/n asymptotically O(1). We then analyze the power for large n:
Ca (n 2 + O(1))
1 Berr Z
σ
where Z is standard normal variable. For the power to be a desired constant, the argument to the probability function must also constant, denoted -2g.
n n +0
2g
01
Solving this equation for gives
27

---

Published as a conference paper at ICLR 2026
n w = ca + 2301 + =
(1)
Since the right-hand side consists of terms that constant for fixed significance level and power, follows that n w must be constant, This implies the inverse proportionality n. < 1 1/
Theorem C.1. Consider a reparameterization attack that perturbs the token rank distribution Zipfian parameter by amount For RUT to achieve constant Type Il error rate:
When a + I+ (moderate heavy-tailedneess, a target sample size of n = (a is sufficient.
When a + +oo (extreme heavy-tailednesss, a target sample size of n. = aa?) is
sufficient.
Proof. A reparameterization attack shifts the distribution parameter from to a' = act. The CvM distance w is proportional e² and the integral of the squared derivative of the log-rank CDF, F(x, at,, with respect First, derive the asymptotic form of this derivative. Let F(z, a) ell-a)e D(a) where D(a) = ((a)(a 1). The partial derivative is
(F ① (clla)=D(a))) Oo
da
-xell (I-(0- (aell-))D((a-- - -((-))
= =1e(1-a)m x D'(a)
D(a)) D(a)2
where
D'(a) = da
-cla)(a-1))
((a)
As a - xc, we have the known asymptotic behavions ((()) = + and ('(a) = o(1). Therefore,
D(a) = - =
o(1))
D'(a) ((
Substituting these into the expression for the derivative gives
FF(x,))
(1-00e
x (a - 1)(1 + o(1)) + (a - 1)-(1 (1 +( o(1))²
1+
da
-1-aaa
a (1+
(a - 1)2 (1 (1+ o(1))
1
= -1-o)= (a 2
The CvM distance w%, is proportional to
2 1/0
OF p(x)dr
Da
The probability density function is p(z) (F(u.a)
(1-a)= ((a)
28

---

Published as a conference paper at ICLR 2026
0 (e (a xaa + 3) 2 ell-aju ((a)
w = =6
dx
(2)
J0
10
0
J
((1-a)= (a 20² 2a - x + a 1) ell ell-a)edr
{(a)
=
31l 22 + 0(a x) + a ddz
((a)
We now analyze this integral expression in asymptotie regimes. As - 1+ let 3(a-1) 0+. The integral becomes So e kz 2x² + O(a (z) + t)dr. We use the standard identity Jo e- ml/k"-1
00
c c- dr 00
Integral
= a
-2
e
x
dx + a -4
e c-h==zd ædz
0
2!
0!
= a
13
+ ta
kl
+ 0(k 3)
2a 2
a a 3(a 1)
+
- ((-()))) (3)
(3(a - 1))3
As 1+, the dominant term is O((a 3) Since ((a)-1 O(a -1), the CvM distance is
- w 0 (a- 1) 1a 1) θ (e 1)
By Lemma C.2, the required sample size is n = ( (a - 1)
As +00, re-calculation results in (2) being of order ( b); note that (3) longer apply due to change in the limiting condition. Since ((ar) = + o(1), we have
3h θ (e2 aa a5
By Lemma C.2, the sample size is n + 2c))
Theorem C.2. Consider a reordering attack where sampled token is made the new top-1 token. For RUT to achieve a constant Type Il error rate:
When a - 1 (moderate heavy-tailedness), target sample size of n. O()) is sufficient.
When a +oo (extreme heavy-tailedness), target sample size of +(2%) is sufficient.
Proof. For a reordering attack, the CvM distance w is the expected squared difference in the log- rank scores, given by
+00
1-- k log A:
((a)2
Expanding the numerator and splitting the expression into sums yields
log k
log
w ((a)2
ko
k2a
Kmm
Using the identity ('(s) E k log ki, obtain
{(a)z (--(((a))((<((22))) = 5'(2a) C'
1
32 =
((a)²
29

---

Published as a conference paper at ICLR 2026
We analyze this closed-form expression. As 1* use the Laurent series expansions ((a) (a (a-11 11) ++ + ( 1) and ('(a) = -(a 111 + + O(a - 1), where y and "i are Stieltjes constants. The term (2(2) converges to the finite constant (2). The numerator is dominated by -('(a),
w -(a -(a-1)-2++01 -aa (1 1)3) - ((0-1)-1+0(1))
O(a - 1) (1)
(a 1)- Oia - 1))²
By Lemma C.2, sample size of n (1) is sufficient.
As +00, we have ((a:) = 1 + (1). The derivative is dominated by its leading term from the
series expansion, ('(a) -2 2(1 + o(1). The term (2(2x) -4-a In 2n + o(1)) is of lower order. Therefore,
w2 - ((2a))/((())
(((a))2
(-22 2n + o(1))) = 211+
2(+
(1))
2 2n + (1++o-)
(1))
2 In 2n o(l)) 0(2
By Lemma C.2, the required sample size is O(2%).
30


Reviews:
[{"review_id": "qsrYz5dafV", "reviewer": "Reviewer_v4bF", "summary": "The paper introduces a testing methodology to audit black-box LLM APIs. It aims to verify if a given API corresponds to a specific model or configuration, through model version, prompt cache behavior, and decoding parameters. Since commercial LLM APIs are opaque, the authors propose a statistical testing framework that uses carefully designed query-response pairs to detect discrepancies between a target (claimed) model and the API being audited. Experiment results show that the audit can detect subtle deviations from the claimed model with high sensitivity.", "questions": "1. Why was AUROC chosen over other divergence-based measures like Jensen–Shannon or Wasserstein distances? \n2. Would prompts tailored to model weaknesses, for example, factual reasoning, yield more sensitive audits than random samples?\n3. How does the method distinguish between genuine model differences and sampling variability caused by small changes in temperature or top-p?\n4. How small a fine-tuning or low-rank adaptation can the audit reliably detect? Is there an empirical detection threshold?\n5. What’s the minimal number of queries needed to achieve a statistically significant audit result? Can the method be optimized to reduce query costs?\n6. If the target API changes gradually via some silent model updates, would the audit detect incremental drift or only sharp transitions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear motivation and practicality: the paper addresses an important and underexplored issue, the lack of transparency in commercial LLM APIs, and it works in a practical black-box setting.\n2. It formalizes the auditing task as a statistical hypothesis testing problem; also, the proposed Average AUROC algorithm is intuitive and effective for summarizing model separability without requiring access to logits or gradients.", "weaknesses": "1. The auditing power heavily depends on how prompts are sampled. It remains unclear how different prompt types, for example, factual benign prompts vs. adversarial prompts, affect the sensitivity of the proposed method. \n2. The framework requires repeated querying of both the reference and target APIs, which is expensive and may be infeasible for large-scale or continuous audits.\n3. Changes in the decoding parameters, e.g., temperature or top-p, could artificially inflate AUROC values even when the underlying models are identical. This potential confound remains underexplored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plgfbPXt7l", "reviewer": "Reviewer_6Aub", "summary": "This paper introduces the Rank-based Uniformity Test (RUT), a statistical method for auditing whether a deployed LLM API matches a claimed reference model. The key idea is to evaluate the rank percentile of an API’s response under the reference model’s response distribution and test for uniformity via the Cramér–von Mises statistic. The authors claim RUT is query-efficient, robust to adversarial rerouting, and effective under various threat models (quantization, jailbreak prompts, fine-tuning, and full model replacement). Extensive experiments are reported across open-weight models (Llama, Gemma, Mistral) and simulated API providers, showing RUT’s superior detection power over baselines such as Maximum Mean Discrepancy (MMD) and Kolmogorov–Smirnov (KS) tests.", "questions": "1. How sensitive is RUT to tokenization mismatches or decoding parameter drift between π_ref and π_tgt? Could small discrepancies trigger false positives?\n\n2. What is the theoretical sample complexity (number of prompts or reference draws) required to detect a given substitution level with 95% confidence?\n\n3. How does RUT perform when API responses are post-processed (e.g., truncation, moderation filtering)?\n\n4. Can the method scale to GPT-4/Claude-level APIs where reference sampling (m=100) is impractical?\n\n5. Can you clarify whether the rank-based test is truly undetectable by adversarial rerouting? Have you simulated a provider that dynamically switches models upon detecting repeated prompts?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "1. The authors proposed a conceptually simple yet general test based on rank uniformity, adapting classical nonparametric tests (CvM, rank statistics) to LLM auditing.\n\n2. Experiments are extensive, covering several realistic scenarios (quantization, jailbreak, SFT).\n\n3. Implementation details are clear, with ablations on score functions and comparisons to reasonable baselines (MMD, KS).\n\n4. Figures and tables (especially Table 2, Figure 4) effectively summarize empirical trends.", "weaknesses": "1. The proposed RUT is essentially an application of probability integral transform + CvM uniformity testing on log-rank scores — standard tools in nonparametric statistics. The methodological leap from MMD or KS is small; the main difference lies in choice of feature (log-rank) rather than test design. The paper frames RUT as “novel,” but it’s largely a recombination of well-known components. There is little theoretical innovation beyond empirical tuning.\n\n2. While the paper claims higher “statistical power,” the advantage appears numerically small and inconsistent across settings (see Table 2a, where 8-bit quantization detection remains near random). The results lack statistical significance testing (e.g., confidence intervals for AUC). Moreover, RUT’s improvements might stem from using richer local sampling (100× reference draws per prompt) rather than an inherently stronger test.\n\n3. The approach requires many reference samples per prompt (m=100), which is computationally expensive and unrealistic for auditing large APIs. The “query efficiency” claim is therefore misleading—it’s query-efficient only w.r.t. API calls, not total inference cost.\nThe assumption that the same decoding parameters and tokenizer are available is very strong; real APIs often obscure these details. The paper briefly mentions this but does not evaluate robustness under parameter mismatch or tokenization drift.\nThe adversarial rerouting model is simplistic (Equation 3) and not validated against actual API behaviors.\n\n4. No analytical characterization is provided for Type-I/Type-II error bounds or sample complexity under common substitution settings. The “uniformity under H₀” claim is intuitive but lacks formal proof of robustness when Fπref is empirically estimated. Without finite-sample guarantees, the method’s reliability remains unclear.\n\n5. Recent works such as Cai et al. 2025 (Are You Getting What You Pay For?) and Gu et al. 2025 (Auditing Prompt Caching) are only cited but not empirically compared. Many state-of-the-art auditing or watermarking methods (e.g., fingerprint-based ones like Pasquini et al. 2024) are ignored in experiments, weakening the completeness of the evaluation.\n\n6. RUT provides only a binary “reject or not” decision with no explanation of why a model differs. In practical auditing, it is crucial to pinpoint whether deviation stems from quantization, fine-tuning, or system prompt injection. The method offers no such diagnostic insight, limiting its operational usefulness.\n\n7. The paper repeatedly claims “robustness to adversarial rerouting” but provides no adversarial evaluation where the provider actively detects audit traffic. All experiments are offline simulations assuming passive substitution. The claim is therefore unsubstantiated.\n\n8. Several figures (especially AUROC plots in Appendix A.2) provide redundant visualizations without interpretation. The statistical terminology (e.g., “rank percentile under empirical CDF”) is used loosely. The method’s dependence on randomization (U~Uniform[0,1]) introduces variance but isn’t analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2MpAtNEp4W", "reviewer": "Reviewer_wtoi", "summary": "The paper devises an improved statistical test procedure for detecting if textual model responses served to a user vary distributional in a significant way from a reference model, such that one should assume the model and its system has been replaced or modified (including by variations to a system prompt).", "questions": "I'm largely fairly happy with this paper, within the bounds it sets itself.\n\nThere could be a little more sophistication in the evaluation, but this is far enough outside my expertise that I struggle to provide concrete suggestions myself—so take the below with a grain of salt.\n\nMy sense is that there's a minor conflict in choosing the scoring function on wildchat and then evaluating it against wildchat without a hold-out set. This is somewhat remediated by the test on other datasets. However, this only impacts the robustness of the choice of f and not the overall technique.\n\nThe other evaluation critique I have is with the choice of AUC as the key metric used for the final evaluations. A stealthy \"attacker\" might choose to route a relatively small number of prompts, and evaluating on AUC down-weights this regime. Could consider: power at small q values. Partial AUC integrated over small q. The smallest q where distinguishing is effective. \n\nFinally, worth being explicit about the threat model this paper is describing: you have to have access to the reference model, which narrows the scenarios under which this is useful. Likewise, with system prompt substitutions, this presumes there really is a true reference.\n\nDespite these critiques, I'm positive about the paper and thought it was appropriately scoped and evaluated. Room for improvement, but overall I think this result is one that should be in the literature. Good luck!", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "+ The idea is fairly nifty, and appears to be a meaningful (if minor) improvement on other pre-existing techniques listed. \n+ Discrimination works with relatively few samples and limited assumptions (though there are some weaknesses in those)\n+ The evaluation is mostly fair and appropriate\n+ There is some potentially for this to be practically useful in comparing model APIs across providers under certain assumptions", "weaknesses": "- Could improve the evaluation of the false-alarm scenario, where a model is compared to itself (currently only showing self-pairs)\n- Worth showing the variance in evaluation with multiple draws of the 100 samples\n- Some minor issues with metrics and choices in the eval", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3XSJewM4ue", "reviewer": "Reviewer_1Ptp", "summary": "This paper proposes a Rank-based Uniformity Test (RUT) for verifying whether a black-box LLM behaves identically to a locally deployed authentic model. The extensive experiments demonstrate that the proposed method is robust to quantization, harmful fine-tuning, jailbreak prompts, and even full model substitution.", "questions": "- What are their similarities and differences between these two tasks?\n- What if the service provider adds system instructions or watermarks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is an interesting and solid idea, and the theoretical formulation is elegant and well grounded. The method is flexible and can be applied across different scenarios (even though its performance sometimes appears less strong, which is understandable given the introduction of probabilistic substitution attacks).", "weaknesses": "- When I first saw the title, I actually thought of another line of work also related to auditing LLM APIs. In that problem setting, the auditor’s goal is to identify which base model or API underlies a released service, even when the service uses a fine-tuned version. The motivation there is to protect intellectual property and ensure robustness of auditing across model variants. Your task, however, seems slightly different. You aim to verify whether a claimed model has been altered, which emphasizes sensitivity rather than robustness. The distinction between these two auditing objectives initially caused some confusion. I think the two methods are not interchangeable, so it would be very helpful if you could more clearly describe their similarities and differences in the related work section. Although some fingerprinting-based studies are mentioned, the difference between the two problem settings is not clearly explained. It would also help if Section 3 could further clarify the specific auditing scenario considered in this paper.\n\n- My second concern relates to the applicability of the proposed method. The paper shows that RUT can easily detect discrepancies when jailbreak prompts are introduced, but this raises another question. It is quite common for service providers to add system instructions to improve model performance or to embed watermarks in outputs to protect intellectual property and prevent model misuse or content leakage. These are entirely legitimate and even responsible practices, which do not involve replacing the claimed model. Would such cases also be flagged as not the claimed model? If so, that seems problematic, as the underlying motivations are reasonable. Since many real-world services adopt these practices, such high sensitivity could make the method less practical. It may end up flagging a large number of legitimate services as inconsistent, thereby limiting its real-world utility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)