Log Probability Tracking of LLM APIs
Authors: Timothee Chauvin
Erwan Le Merrer
Francois Taiani
Gilles Tredan
Venue: ICLR 2026 Poster (2026)
Keywords: API drift
audit
monitoring
LLM API
black-box
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We show that logprobs, when they are available, can be used to audit LLM APIs for changes, beating alternative approaches by large margins on cost and sensitivity to small changes.

Abstract: When using an LLM through an API provider, users expect the served model to remain consistent over time, a property crucial for the reliability of downstream applications and the reproducibility of research. Existing audit methods are too costly to apply at regular time intervals to the wide range of available LLM APIs. This means that model updates are left largely unmonitored in practice. In this work, we show that while LLM log probabilities (logprobs) are usually non-deterministic, they can still be used as the basis for cost-effective continuous monitoring of LLM APIs. We apply a simple statistical test based on the average value of each token logprob, requesting only a single token of output. This is enough to detect changes as small as one step of fine-tuning, making this approach more sensitive than existing methods while being 1,000x cheaper. We introduce the TinyChange benchmark as a way to measure the sensitivity of audit methods in the context of small, realistic model changes.

Published as a conference paper at ICLR 2026
LOG PROBABILITY TRACKING OF LLM APIS
Timothée Chauvin
Erwan Le Merrer
François Taiani
Gilles Tredan 2
Universitéé de Rennes, Inria, CNRS/IRISA (Rennes, France)
"LAAS, CNRS (Toulouse, France)
ABSTRACT
When using an LLM through an API provider, users expect the served model remain consistent over time, a property crucial for the reliability of down- stream applications and the reproducibility of research. Existing audit methods are too costly to apply at regular time intervals to the wide range of available LLM APIs. This means that model updates tice. In this work, we show that while LLM log probabilities (logprobs) are usually non-deterministic, they can still be used as the basis for cost-effective continuous monitoring of LLM APIs. We apply a simple statistical test based on the aver- age value of each token logprob, requesting only single token of output. This is enough to detect changes as small as one step of fine-tuning, making this approach more sensitive than existing methods while being 1,000x cheaper. We introduce the TinyChange benchmark as a way to measure the sensitivity of audit methods in the context of small, realistic model changes.
left largely unmonitored in prac-
INTRODUCTION
LLM API providers typically offer version-pinned endpoints, signaling to users that given end- point will serve a consistent model. Users APIs tend to rely on this consistency: developers want to avoid unexpected regressions in their applications; researchers seek reproducibility in their periments; regulators perform initial compliance assessments, and assume that the API will keep serving the same model afterward (Yan & Zhang 2022). Yet users have practical way verify this consistency. Once a niche concern, this gap is drawing more attention as LLMs get deployed in mission-critica. applications, notably in software engineering.
LLMs are, however, highly complex software artifacts under constant development. LLM API providers may change their inference software and hardware infrastructure for performance reasons, or modify the model itself in response to new jailbreaks or to update its behavior. Some providers might also quietly deploy quantized versions to save costs, switch to more light-weight models dur- ing peak traffic, or even serve different variants based on query characteristics. More concerning from security perspective, providers might suffer the malicious injection of unobstrusive model backdoors, infringing the privacy and security of their users for an attacker's benefit. Such explicit attacks could happen from insiders, a bad update, or a compromised supply chain. This concern isn't just thcoretical: the version of Grok deployed on X had three such incidents in 2025 where modified system prompt was deployed to users of the X platform. Two incidents were blamed on rogue employees (Babuschkin) XAI and one bad update (xAll a).
Unfortunately, existing change-detection methods tend to be prohibitively expensive. State of the approaches typically rely on the processing of tokens returned by the monitored LLM (Chen et al. 2023 Gao et al.) 2025 Cai et al., 2025) and require extensive benchmarking or statistical analysis across many queries reach robust conclusion. As result, L.MM APIs are left largely unmonitored by third parties for changes in practice.
In this paper, we propose to address these shortcomings by exploiting the log probabilities of the returned tokens rather than the tokens themselves, a method we call logprob tracking (LT). During LLM inference, each token is sampled from an underlying vector of log probabilities (logprobs) over
Code available at:
mothee- wecphheeeehhuviiii/eeeeeee 11mm im

---

Published as a conference paper at ICLR 2026
the entire vocabulary space of the model. Some LLM APIs return a subset of these logprobs for each token of the response. While not systematic, a nontrivial fraction of LLM APIs support logprobs. For instance, found that 23% of reachable endpoints on Open widely used gateway to 60+ LLM API providers and 500+ models-return logprobs when requested.
Intuitively, logprobs constitute far richer information source than tokens, yielding handful of continuous values along with a token. Unfortunately, exploiting these logprobs is not trivial. Log- probs are not deterministic in practice (Cai al., 2025), making them more challenging to use for model change auditing than simply checking logprob vectors for equality across time.
In this work, show that log probabilities reveal enough information detect even minor changes in LLM APIs, such as a single step of fine-tuning. This makes our approach significantly more sensitive than existing methods. Specifically, we use simple statistical methods to overcome log- prob non-determinism, effectively detecting changes across a wide range of representative model modifications. We demonstrate through extensive in-vitro experiments that sending a single token of input and requesting single token of output is enough to consistently detect changes that remain undetected by current monitoring methods. Our input token is chosen arbitrarily, and unrelated to the type of change applied. We report that our approach is up to three orders of magnitude cheaper than existing alternatives.
In this paper, our main contributions are the following:
We introduce the logprob tracking (LT) method, and show that a I-token prompt and the logprobs of I-token response are sufficient to exceed the detection performance and sen- sitivity of alternative methods, at fraction of the cost;
We introduce the TinyChange benchmark, designed to evaluate detection methods on small model changes;
We extensively evaluate LT using the TinyChange benchmark against wide range of rep- resentative model changes (finetuning, random noise, pruning), and compare it to two state- of- the art approaches, demonstrating how LT is able to detect changes as small as one step of fine-tuning at fraction of the monitoring cost of other methods.
The remainder of the paper is organized as follows: in section 2 (Method), we describe the problem setting and the intuition for using logprobs for change detection. We introduce statistical test to work around the non-determinism of logprobs. In section 3l (Experiments), we introduce the TinyChange benchmark, designed evaluate change detection methods on small, realistic changes. We use it to show that prompts as small as a single token are almost as good as longer prompts for detection. We show that LT significantly exceeds the detection accuracy of fate-of-the-aee baselines, while being orders of magnitude cheaper.
2 METHOD
2.1 RATIONALE
Problem setting In practice, our goal is to identify systematic modifications that affect the API's output distribution, whether from model changes (fine-tuiigg further post-training, quantization, pruning, system prompt modification, introduction of a backdoor...,, or software/hardware changes (CUDA versions, inference kernels, set of GPUs used for inference...). We do not attempt to distin- guish between the sources of change, as any systematic difference could impact reproducibility of research and application reliability, regardless of its origin.
We follow the definition of the model equality testing problem in Gao et al. (0055) y we consider a pair of LLM APIs (which can be the same API at different times).
For given input distribution of prompts, the auditor has sample access to the distributions over completions of both APIs, and Q. The audit aims test P Q: two APIs are considered equivalent if and only if their response distribution is the same for all prompts.
Logprobs Almost all modern LLMs are based on auto regresseee transformers. When processing an input, the output of the transformer is vector of logits, with one logit for each token in the
2

---

Published as a conference paper at ICLR 2026
tokenizer's 's vocabulary. These logits then converted to vector of probabilities through soft- max operation, and the generated token is sampled from this probability vector. Log probabilities (logprobs) simply the logarithm of each of these probabilities.
Some LLM APIs support returning the top-k highest logprobs in addition to the sampled tokens. For instance, 23% of reachable endpoints OpenRouter support returning between 5 and 20 logprobs. This includes all available models from XAI, multiple models from OpenAl including the GPT-4.1 series, and multiple open-weight models such the Qwen, Gemma, Llama, Deepseek, or Mistral families of models. More details in Appendix A.I
These logprobs encode a part of the probability distribution, and such denser in information than the sampled tokens. For instance, they can be used to recover the prompt tokens given an output (Morris al., 2023). and have been used to leak proprictary information about an LLM, including
its hidden dimension and embedding projection layer (Finlayson et al., 2024 Carlini et al. 2024
This motivates us to study the feasibility of detecting changes LLMs via the top-k logprobs of single output token, on fixed short prompt. Intuitively, detecting many types of change using only a single prompt and single output token seems possible, as LLM modifications, such as finetuning or quantization, typically affect all the weights of model. In particular, finetuning in one domain known to affect completions in other domains (Qi al. 2024).
The challenge such an approach faces is that logprobs are non-deterministic in real-world LLM inference tasks.
2.2 NON-DETERMINISM IN LLMS
There are two main reasons for non-determinism in LLMs:
Intentional non-determinism: The most common selection method for the next token in
LLMs is temperature sampling. Given logits for each token i in the vocabulary, the probability of sampling token at temperature T is obtained from the softmax of the logits:
P(i, 7) = Ejer
Logprobs are a unique representation of the logits, defined as the log of the probabilities at T' = 1. In this paper, we are not concerned with non-determinism from temperature sampling, as we operate directly on the logprobs.
Unintentional non-determinism: In practice however, logits often fluctuate between iden- tical requests, resulting in a non-deterministic logprob vector. While the majority of infer- ence kernels run-to-run deterministic at the batch level, individual requests deterministic, as they are affected by other requests in the batch (He & Lab] 2025). In addition, the software and hardware stack top of which the model is deployed is also source of non- non-determinism in production LLM APIs, where successive requests are likely be routed different GPUs (Zhang et 2024).
As an illustration of unintentional non- mon--eterrmnism. Figure shows the logprobs returned over two weeks time by GPT-4.1, prompted with single letter "x" The logprob of a given token isn't. constant, but fluctuates around its average.
Given that logprobs are non-deterministic, we see each logprob returned as itself sampled from probability distribution, and use standard hypothesis testing to assess whether the two distributions are the same.
2.3 TWO-SAMPLE TEST ON FIRST-TOKEN LOG- SIG-PROBABILITIES
Our approach, logprob tracking (LT), implements hypothesis test to detect if the two distributions are the same, requiring minimal assumptions about logprob distributions. Namely, a permutation test based the mean absolute distance between per-token average logprobs.
The overall procedure shown in Figure we send the same prompt to two LLM APIs, requesting single token of output, repeating for N samples. For each token in the returned logprob vectors,
3

---

Published as a conference paper at ICLR 2026
Logprobs GPT-4.1 with prompct
Red
API (t)
-12
LLN API (t)
-14
Asgi0
Aug 95
Avg St
Aug
Aug
Figure 1: Logprob non non-determinism the wild: Figure Setup of our change detection method logprobs returned by the GPT-4.1 API, in August logprob tracking (LT). 2025.
obtain empirical measurements for its logprob value, and our goal is to compare these measurements to assess if the two distributions are the same.
Specifically, for an input prompt and two LLM APIs APT(1) and AP7(2), we aim compare their logprob distributions over the first response token, p(1) and p(2), We sample N times from each distribution, where each sample returns only the top-k tokens with their logprobs. Given this top- truncation, the list of tokens not necessarily the same in each sample. When a token present in other samples is absent from sample, we conservatively impute its missing logprob using the minimum logprob value from that sample, since we know that its logprob is not greater than this value.
Let V = ( | be the set of all tokens observed across both samples. Each of the 2N requests is represented vector of length ntak yielding matrices
T(), (2) E R Nx
For each token f,, we compute its average logprob:
a
22) a (2)
1
(2)
The test statistic is the average absolute distance between these means:
Bhok
S
-
a
a (2)
(3)
Pook
A permutation test on the pooled samples is used to obtain a p-value for the null hypothesis that p(1) and p(2) are identical. For ....., random permutations (each formed by splitting the pooled 2.V samples into two groups of size N), compute permuted accuracies a (11)16)) (2). and the permutation statistic ((b) as previously.
The p-value is the probability of obtaining a larger test statistic than the observed one under the null hypothesis:
B P
1 2 S}.
(4)
B
1=1
It then be interpreted in relation with the chosen significance level a. If p a, reject the null hypothesis and conclude that the two distributions are different. our case, we conclude from p that the two L.M APIs are not identical.
4

---

Published as a conference paper at ICLR 2026
3
EXPERIMENTAL EVALUATION
We now evaluate LT and compare it to two competing baselines.
3.1 EVALUATION SETUP
Creating variants of local models: TinyChange A change detection method should ideally allow an auditor to configure a detection threshold based on the magnitude of the change that they want detect. In particular, should be able to distinguish between two different but close variants of model. A benchmark creating variants of model across range of modification magnitudes should therefore be used to evaluate and compare detection methods, instead of evaluating on few ad-hor variants. As we are not aware of such a benchmark in the literature, we took the task to create one; we call it ThnyChange. It is publicly available at https://github.cc htttp://giiuubcoo/iimtteeee-e
This benchmark takes an L.M as input and generates a set of 58 variants, ranging from minor to more substantial modifications, reflecting practical operations model might sustain in practice. Variants produced across five levels of modification intensity, with difficulty increasing in powers of two. Specifically, we apply: regular fine-tuning and LoRA fine-tuning (Hu et al. 2022). each for 1 epoch with between and 512 single-sample finetuning steps; unstructured weight pruning, either by magnitude or by random selection, removing a fraction of weights ranging from 2 2-10 up I: and parameter noising, adding independent Gaussian noise to each parameter with standard deviation a ranging from 2-15 to 1. More details in Appendix A.2
The fine-tuning data is sampled from the LMSSS-Chat-IM IM dataset (Zheng et al. 2024), dataset of real-world conversations between users and multiple LLMs, selecting only single-turn interactions with GPT-4. The goal is for the finetuning dataset to be both difficult detect (because the data is in-distribution for models similar to GPT-4) and realistic (because most changes to production models are likely be subtle).
We apply the TinyChange benchmark to 5 wpen-weight LLMs from 0.5B to 8B parameters: Qwen 2.5 0.5B, Gemma 3 IB, Phi-3 Mini 4k, Llama 3.1 8B, and OLMo 2 7B] resulting in total of 290 variants.
Evaluation Metries We consider the following three metrics:
1. Cost, measured as number of tokens required per hypothesis test.
2. Discriminative power, measured as the ROC AUC across different models and different variants per model, in the TinyChange benchmark.
3. Sensitivity, measured the smallest modification that can be reliably distinguished from the original model, across several difficulty scales in TinyChange (number of steps of fine- tuning, fraction of weights removed in weight pruning, magnitude of the noise random noise addition).
Baselines
We compare logprob tracking (LT) to two state-of-the-at baselines:
MET (Model Equality Testing method, Gao et al. (2055) uses Maximum Mean Discrep- ancy (MMD) with Hamming distance kernel to compare two distributions of responses. As in the original paper, we use PMET 25 prompts, 50-token responses, and T = 1.
MMLU-ALG MMLU (Massive Multitask Language Understanding, Hendrycks et al.
(202T) ) a well-known benchmark of the knowledge of LLMs. As the benchmark contains 14,042 questions. we restrict it to the "abstract algebra" subset with PMr = 100 questions, and call this baseline MMLU-ALG. Using MMLU is similar to the approach used Chen et al. (2023), with the difference that MMLU consists of questions with multiple-choiee answers A/B/C/D, thus considerably driving down the token cost of this approach com- pared to frecform answers. Each of the 100 prompts is sampled at T 0.1. generating 5
A parameter-efficient fine-tuning method that introduces trainable low-rank matrices into pre-trained weights, enabling effective adaptation with far fewer trainable parameters Huggingface IDs: (etmm//wwe.....5--nnnnsstt google/gemma-3-11e instruct, meta-llama/l Jama-3.1 -B8--nsttttt and
microssff/Phhi--miiii-kk
5

---

Published as a conference paper at ICLR 2026
output tokens. Similarly to 2.3 define the two-sample test statistic the mean absolute distance between per-question accuracies. More details in Appendix A.3
Sampling For each of the 5 original models and 290 variants (V = 295), generate S 10, 000 samples per prompt using vLLM (Kwo et al. 2023). The MET and MMLU-ALO prompts are sam- pled at T 0, where the non-determinism from temperature sampling far outweighs unintentional non-determinism. For LT however, we are concerned with the logprobs, so we introduce realistic noise from other requests by mixing prompts with random traffic, also sampled from LMSYS- Chat-IM for simplicity. To generate each of the 10,000 samples, we mix and shuffle our PLT = 16 tested prompts with 48 random prompts, and generate the logprobs for the first token of the batch with vLLM. (LT uses a single prompt, but we evaluated the influence of the prompt in 3.2.1 This results in V
(PMET PMA PLT) 416M outputs.
We then perform 1 1,000 original/original hypothesis tests each original model, to obtain the dis- tribution of the statistic under the null hypothesis, and 1,000 original/variant hypothesis tests on each variant. Each hypothesis test samples N 10 outputs randomly from both distributions, for each prompt, and results in a test statistic.
For given variant, we can finally compute the ROC curve and ROC AUC of detection method from the original/original and original/varian statistics.
3.2 RESULTS
3.2.
SHORT PROMPTS ARE SUFFICIENT FOR RELIABLE DETECTION
Let's geoeeate zondom woeth! Aiuuros Corabread Stipllte Desrrent
Let's generatee mindom woethl Lixderpay Polley Blonto Ideabst
eepty one tahen. 1+1-
PPO77T
1bc
s00g
5.
Overall AUC Advaniage
EEB
0%
-trerngo Token Length Fromgrs Acrees Models
Figure 3:
Minimal impact of the prompt length on the performance of LT. Average AUC across models and variants, 2,000 bootstraps at the model and test statistics levels.
Figure shows each prompt's performance, relative other prompts, across models: for each model. we compute the absolute performance of all prompts this model, as their ROC AUC. The relative performance of a prompt on model computed as the difference between its absolute performance and the average absolute performance of all prompts this model. For each prompt, we finally compute the average of all its relative performances.
Prompts ordered by their average token size across models. Longer prompts to perform slightly better, but by a small margin: about 1% difference in AUC between the shortest (1.5 tokens) and longest prompt (33 tokens), for baseline of around 91 % AUC (cf" Table

---

Published as a conference paper at ICLR 2026
We conclude that very short prompts sufficient to detect changes. In the rest of the experiments, we therefore use the shortest prompt, made of the single letter "x", which is to 2 tokens long depending on the tokenizer.
3.2.2 LOGPROB TRACKING IS MORE SENSITIVE AND CONSIDERABLY CHEAPER THAN OTHER METHODS
Finetuning
LoRA finetuning
ET (Dies) MMLU-ALG
(Ohrs
MMLU-AL4S
MET
0.9
-MET
ROCAUC
0.8
ROCAUC
0.8
0.7
0.6
Rosniom gressing
0.5
Rondom guessing
512
256
128
64
32
16
4
2
512
256
128
64
32
16
2
Number of steps of finetuning
Number of steps of finetuning
Random noise
Weight pruning, selection by magnitude
LT (Ours) MMLU-ALG
LT (Ours)
MMLU-ALG
MRT
-MTT
ROC ROC
ROC
0.8
0.7
0.7
0.6
0.6
Random gressing
Random guesing
1 18
2' 2
2
2
2* 2
Stundard deviation of the gaussian noise added to each weight
Fraction of the weights prune
Figure 4: Average ROC AUC by difficulty level and method across LLMs (in each plot, difficulty increases from left to right). 95% CIs from 10,000 bootstraps at the model and test statistic levels.
Figure plots the ROC-AUC of our approach (LT) and the two baselines (MMLU-ALG and MET) on four different types of changes included in the TinyChange benchmark (from left to right and top to bottom: Finetuning, LoRA Finetuning, Random noise, and Weight pruning). for different levels of change amplitude ax-s) Logprob tracking clearly outperforms the baselines, with the possible exception of LoRA finetuning, which challenging to detect for all methods.
If we consider a point estimate AUC of 0.9 as our threshold for good detection performance, we can compare the sensitivity of the different approaches, by considering the highest difficulties before which they reliably have good performance. In the weight pruning experiment, this same threshold difficulty is 2-1 for MET, 2 24 for MMLU-ALG, and 2-10 (or below) for LT. This represents a factor of 2 512 between LT and MET, and slightly higher than 26 64 between LT and MMLU-ALG.
We tentatively conclude that LT is 2-3 order of magnitude (OOMs) more sensitive to small changes than MET, and 1-2 OOMs more sensitive than our MMLU-ALG baseline.
3.2.3 CHANGE DETECTIONS IN REAL WORLD LLM APIS
Methodology Due to the low cost of requesting single-token logprobs, we monitored 189 real- world LLM API endpoints from 10 providers hourly over more than 4 months, collecting more than 1.7M responses. We apply LT to this data to identify points of likely changes in APIs, using the test statistic S (Eq. 3 our detection signal.
7

---

Published as a conference paper at ICLR 2026
Model
MMLU-ALG 0.878 (0.802, 0,944)
MET
LT (Ours)
Overall AUC (95% CI)
0.670 (0.612, 0.731)
0.915 (0.864, 0.958)
Token Count Per Test (1,0)
(2.1 x 10 9.9 x 10")
(2.9 x 10" 2.0 10*)
(28, 20)
Cost Per Year
$332
$146
$0.14
Table 1: Overall ROC AUC across the TinyChange benchmark and across models, and token cost for each method. Cost per year of hourly sampling at GPT-4.1 pricing (input $3, output $12 IM tokens). 95% Cls from 10,000 bootstraps the model, variant, and test statistic levels.
Provider
Endpoints
Tracking Duration (Years)
Changes
Changes/Year
Fireworks
14
4.1
9
2.2
Lambda
18
3.2
1.9
Azure
4
1.6
2
1.3
Crusoe
4
1.1
I
1.0
Nebius
36
8.2
7
0.9
Chutes
13.0
11
0.8
14
2.8
I
0.4
OpenAl
19
6,9
0
0.0
Hyperbolic
8
1.8
0
0.0
Deepseek
0.1
0
0.0
Total
42.8
37
0.86
Table Change detections across providers. "Tracking Duration" refers to the cumulative duration of observation across endpoints.
Given the absence ground truth, we flag change only when the test statistic high absolute terms (to filter out small shifts) and relative the endpoint's historical variance (to avoid false positives on high-variance endpoints).
Specifically, for given endpoint, compute its test statistic for each hourly query, by comparing the two adjacent 24-sample windows. We then compute the running mean and standard deviation window of 100 samples. Finally, report change when the test
of these test statisties, statistic exceeds the running mean by at least 12 standard deviations, and is above 1.0.
Results We identified total of 37 suspected changes, across 29 endpoints and providers. De- tailed statisties by provider are shown in Table
Figure 5 the dates of detected changes across providers; detailed time series for each provider can be found in Appendix B
We contacted some providers in June and July 2025, regarding earlier suspected changes, to ask they could confirm that a change occurred, and describe the nature the change. We did not hear back from xAI or Fireworks AI. Both Lambda and Nebius Al replied that they could not share details about suspected changes with us, with Nebius Al adding that they perform changes on a regular basis.
Notably, almost all detected changes (34 out of 37) affected open-weight models, where users might reasonably expect greater stability. This suggests that undisclosed changes pervasive regardless of whether the model itself is open. This opacity deployment undermines the transparency benefits of open weights.
4 LIMITATIONS AND FUTURE WORK
Logprob support requirement Our approach applies to APIs that support and return log prob- abilities. While this restricts its applicability compared to other methods leveraging only output tokens, we believe LT has value beyond its current applicability. First, our results demonstrate that logprob access significantly improves transparency, which we hope will motivate users to demand logprob support from providers, especially for open-wer weight models. Second, our work establishes a practical upper bound on detection sensitivity under inference non-determinism: we show that changes as small single fine-tuning step can be reliably detected by black-hox method. Third,
8

---

Published as a conference paper at ICLR 2026
Detected Changes by Endpoint
openolign do
Provider
micnoottmmild
mistilaidoottnd 2505
azure
mistalalinstrdd 240
chutes
midmilaton inters -amille) --Mb-mmnnec
nistrabalimaan snall 3.1 instmoctiie
crusoe
fireworks
mussessocct/eeep detppppccc/eeeeeeemmm...... Ilama grevienctiee
swonsppont-22 Seoodle-SSt-iii Seeede--St-itninnniiie
lambda
thudr gim-z
nebius
deepseek tnggeeoktleeppeeeee poppoee--h-----000100
dospaskl-teepiikkkk deepeeeecppcccu--22222 chat---- -2.1
depureklleppaeke 4630
mitrrlaamiaaal 8122 inaract
deegsnek
deepeceki deeppeed
Sappaekllespper wew.ellllppeehheet..
Hamme llam mavercc
misnseaccbeemme 3-1amr--3111-- 1-405b
nosseescchhmmee
(--mee--------111111------- 1-70h
despceiklloppeee..
acmozco
Ang
Sep
Oct
Figure 5: Dates of changes across providers and endpoints, queried hourly with the prompt "x". Periods where the endpoints weren't It tracked greyed out. 2 of the 37 changes (devstral-small- 2505:free and
are also present in the non-free endpoints.
providers could integrate LT into their internal test suites for continuous monitoring of production inference.
LT evasion Providers could be tempted to evade LT's detection by identifying monitoring queries and providing consistent responses only to these prompts. Alternatively, providers could cache logprobs to hide changes, though comprehensive caching is memory-prohibite of short tokens and responses would force logprob tracking to rely on larger prompts, and therefore be more expensive. However, both of these techniques would risk creating detectable inconsistencies elsewhere and, if discovered, would cause greater reputational damage than undisclosed updates.
Limited caching
LT obstruction A provider may also refuse to serve requests with only one or few tokens of out- put. Indeed, we found that between 16:00 and 1 17:00 UTC on September 24, 2025, OpenAl started requiring least 16 tokens of output on their GPT-4.1 line of models accessed via OpenRouter (though not on their own APD]]
Single token of output Certain modifications may evade first-token detection. For example, providers could reduce verbosity during peak demand by adjusting generation-length parameters, such as the bias towards the end-of-sequence token, that leave initial tokens unchanged. However, these modifications are rare: the vast majority realistic changes to LLM API involve changing parameters that should be reflected in the first token of output.
Limited information Our method does distinguish between infrastructure changes and model updates, or give information on the exact nature of the change. While this may be viewed as limitation, our opinion is that any systematic change affecting outputs can impact reproducibiliity and should thus be disclosed. In addition, we view logprob tracking as a complement to existing audit methods-a low-cost, high-sensitivity approach that can run frequently and route alerts to more comprehensive investigations when changes are detected. While we focus binary detection, our framework also naturally extends to quantifying change magnitude.
We showed that using logprobs with simple hypothesis testing method enables extremely sensitive change detection "out of the box". but detection performance can likely be in- creased further by tuning parameters. While the hypothesis testing framework simple, applicable,
Improvements
at least for our one-token requests that requested output logprobs. The emor message is: "Invalid 'max output tokens": integer below minimum value. Expected value 16, but got instead." "

---

Published as a conference paper at ICLR 2026
and enables comparison with other methods, alternative approaches such CUSUM (Page 1954) control charts could also be explored.
5 RELATED WORK
Change detection in ML APIs Prior to widespread LLM deployment, CChnn et al. (2022) intro- duced methods to assess ML API shifts, and found significant shifts in a third of the ML APIs they tested.
Change detection in LLM APIs Chen et (023) compared versions of ChatGPT using mul-
tiple benchmarks, but this approach requires extensive testing across thousands of queries, and the versions they tested were explicitly not the same version. DailyBench (Phillips 2025) was an actual attempt at benchmark- based continuous monitoring of LLM APIs, testing 5 L..Ms 4 times day on a version of HELMLite (Liang et al.] 2023), but it was discontinued after 40 days due to costing $5 day. Gao et al. (2025) compared output distributions with the Maximum Mean Discrepancy statistical test, using Hamming distance kernel. Cai et al. (2025) re-implemented and compared multiple detection techniques.
LLM fingerprinting. Change detection and fingerprinting are closely related, as it's often possible to convert a change detection method into fingerprinting method, and vice versa. However, some fingerprinting methods explicitly aim to be invariant to small changes in a model, or only attempt to identify an LLM within a closed set of known models. Pasquini et al. (2025) introduced set of handwritten queries coupled with a classifier to identify which known model a given LLM is closest to. They explicitly aimed to produce queries that would be insensitive to small changes in a model, but sensitive to large changes. [Sun et al. (2025) fine-tuned embedding models on LLM-generated texts, to identify which L.L out of set of generated given text.
Non-determinism in LLMs. He & Lab (2055) demonstrated that run-to-run (i.e. on the same GPU) LLM non-determinism is due to a changing batch size based on the load on the inference server, proposing batch-invariant kernels as a solution. Zhang al. (2024) ( leveraged within-platform determinism and cross-platform non-determinism to identify which hardware and software configu- ration model is running on, also proposing a method based logprobs.
Formal verification of LLM outputs. Formal verification is another approach for the problem we are concerned with: if an inference result can be proven to come from a specific model, then we can be confident that the model has not changed. zkLLM (Sun et al., (024) introduced rero-knowledge proofs for LLMs. However, the proofs take several minutes to generate for a 13b LLM, and are on the order of 200kB in size, making this approach prohibitively expensive for high high-performanee inference servers. TOPLOC (Ong et al.) 2025) used
hashing to considerably reduce the overhead of proving inference result, but provider cooperation is still required.
CONCLUSION
Production LLM APIs expected to be stable, providing their users developers,,,, researchers, regulators-the the reliability and reproducibility that they require. We introduced logprob tracking (LT), LLM API change detection method that treats first-token log probabilities as samples from a distribution and applies a two-sample permutation test to detect distributional shifts. LT reduces the cost of continuous monitoring by orders of magnitude compared to existing approaches while providing substantially higher sensitivity to small, realistic model modifications. We presented Tiny- Change, benchmark of fine-gu fine-grained model variants that quantifies sensitivity across several modifi- cation axes and difficulty scales, and used it to show that LTT is better detector of small changes up to single step of finetuning- -thn other methods, at 1 I/1, 000th of the cost. We deployed LT in production environments at scale across hundreds of API endpoints and observed dozens of un- explained distributional shifts, illustrating that undocumented changes in deployed LLMs are both detectable and prevalent. LT provides a lightweight, practical first line of defense for reproducibility and integrity monitoring; it can be integrated into audit pipelines to trigger deeper, targeted investi- gations when a shift is detected.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
The entirety of the source code we used in these experiments is available at www//imottee-chavvineeaeee
can be used to reproduce the datasets, the TinyChange benchmark, the sampling outputs, and all plots. The exact versions of the depen- dencies used can be found in the file uv lock. A Dockerfile provided for the non-python dependencies. We ran the LLM variant generation and vLLM sampling nodes of 2x H100 80GB GPUs.
8 ACKNOWLEDGMENTS
We acknowledge the support of the French Agence Nationale de la Recherche (ANR), under grant ANR-24-CE23-778" (project PACMAM).
This work was supported by the Cluster SequolA Chair FANG funded by ANR, reference number ANR-23-IACL-0009.
This project was provided with computing (AI) and storage resources by GENCI at IDRIS thanks to the grant 2025-AD011006369 on the supercomputer Jean Zay's H100 partition.
REFERENCES
Igor Babuschkin.
Accessed:
2025-09-24.
Will Cai, Tianneng Shi, Xuandong Zhao, and Dawn Song. Are you getting what you pay for? audit- ing model substitution Ilm apis, 2025. URL UtLLttpp:://rrxxi.orr//b////
04715
Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper. Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, and Florian Tramèr. Stealing part of production language model, 2024.
URL htt.httpss//aaxii.cooo http://wwwi.or..aaaaaoooa abs 2b440403//////3333/ 2403. 06634
Lingjiao Chen, Matei Zaharia, and James Zou. How did the model change? efficiently assessing machine learming API shifts. In International Conference on Learning Representations, 2022.
URL httphttss///ppeereeaaee ott.::/wwwww....ooooooooooooooan
feet/fooummiie
Lingjiao Chen, Matei Zaharia, and James Zou. How is chatgpt's behavior changing over time?, 2023.
09009
Matthew Finlayson. Xiang Ren, and Swabha Swayamdipta. Logits of aro-proee Ilms leak pro- prietary information, 2024. URL httpppps///xxxxxxo//////a/////ap awtp/g//bbs/// 09539
Irena Gao, Percy Liang, and Carlos Guestrin. Model equality testing: Which model is this API serving? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025. URL http:pas//ppeeneeveee.... net./ forum?id-QCDdI7X33
Horace He and Thinking Machines Lab. Defeating nondeterminism in Ilm infer- ence. Thinking Machines Lab: Connectionism, 2025. doi: (.644444mll.225009 higgaiigggmaaiiinaaaaiooddeeeeie momdermmin waw-iin---im-eeeeee
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- lional Conference on Learning Representations (ICLR), 2021.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https: /pwnreeiieeeeeieeeeee net forum?
1d=nZeVKeeFFF9

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher Manning. Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Z.eeng Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji. Omar Khat- tab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of lan- guage models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:/ /ttp://////eeeeeeeeeeeee net meet/foouumiidd104/22000 Certification.
Featured Certification, Expert
John X. Morris, Wenting Zhao, Justin T. Chiu, Vitaly Shmatikov, and Alexander M. Rush. Language model inversion, 2023. URL https://///........ /ttp://///aaiva///aaeeaaeeealaaa http:/aawvvvv...gaaaaaaa
Jack Min Ong. Matthew Di Ferrante, Aaron Pazdera, Ryan Gamer. Sami Jaghouar, Manveer Basra, Max Ryabinin, and Johannes Hagemann. Toploc: locality sensitive hashing scheme for trustless verifiable inference, 2025. URL http://////a..oooo//////////////
2at.:////500111000011100 16007.
E. S. Page. Continuous inspection schemes. (1( 3444. doi: 10. 10.1093/biomee/411112..000 1-2.100
06 1954, ISSN 0006- ooett /11
Dario Pasquini, Evgenios M. Kornaropoulos, and Giuseppe Ateniese. LImmap: Fingerprinting LinmappFinggepittnnnnn for large language models, 2025. URL http://p.//aaa...or///////2.....
Jacob Phillips. daily-bench. 2025.
bench
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pun-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024. URL https:
//openrevien wwwet//frrmmidddn/eeeeee
Haochen Sun, Jason Li, and Hongyang Zhang. zkllm: Zero knowledge proofs for large lan- guage models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS '24, pp. 4405-4419, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706363.
doi: 10.11457668644.3770334, URL
fittps://do/ .ar.0o..rrg//// wtw..///....11335556444444 3670333
Mingjie Sun, Yida Yin, Zhiqiu Xu, J. Zico Kolter, and Zhuang Liu. Idiosyncrasies in large language models, 2025. 20tp://////aaa////aaaaaaaaaaaaa0
XAI.
/c/tee//eeccccmmmm//////es////10 a. Accessed: 2025-09-24.
XAI.
b. Accessed: 2025-09-22
Tom Yan and Chicheng Zhang. Active faimess auditing. In International Conference Machine Learning, pp. 24929-24962. PMLR, 2022.
Cheng Zhang, Hanna Foerster, Robert D. Mullins, Yiren Zhao, and Ilia Shumailov. Hardware and software platform inference, 2024. htttt:/ts://araa
051997
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, lon Stoica, and Hao Zhang. A large-scale scale real-world LLM conversation dataset. In The Twelfth Interna tional Conference on Learning Representations, 2024. URL /tttps://ppereviee net/ forum? FR
12

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A. TESTING THE PREVALENCE OF LOGPROB SUPPORT ON OPENROUTER
The code compute these statistics is in the file logprob www.ooopeeeeaaeeessstee..o
We obtain the list of all 813 endpoints on OpenRouter by combining the APIs v1/models and poooeel.ie/eddoinntt...
We first send single prompt "x" to all of these endpoints, requesting single token of output at temperature T 1. We obtain correct response from 710 endpoints (this number varies, as many of the errors are 429 rate limits, especially for the free models). We then send the same request to these reachable endpoints, this time requesting logprobs, and take note of which endpoints complied with this request.
We find 164 such endpoints (23%).
Fireworks Al and Azure return 5 logprobs, xAI 8 logprobs, and other providers return 20 logprobs.
In the case of OpenAl, we found that requesting less than 16 tokens of output from the GPT-4.1 series is not supported on OpenRouter, but only on the OpenAl API, as discussed in 4
A.2 TINYCHANGE HYPERPARAMETER
In TinyChange, the source code of which in the file tinychange. py, both finetuning and LoRA finetuning performed with learning rate Ir 10-6, with single epoch. Each step of finetuning is performed on a single batch containing a single finetuning sample.
The LoRA hyperparameters
8, 8, dropout 0.0, and the LoRA target modules the attention layers' Key, Query, Value, and Output projections.
For most models, patched their chat template include the generation $) keyword and only finetune on assistant responses, instead of both questions and responses (see the file chat.templates toml).
Model updates and inference are performed in BF16 precision.
A.3 TWO-SAMPLE TEST ON THE MMLU BENCHMARK
We collect 10, 000 independent responses from two LLMs, for each of P 100 prompts, at H100 80GB GPU.
temperature T = 0.1, using vLLM
We reduce the responses to a binary indicator of correctness: define the binary label for response Jj. j .1....,N .} of model m E (1,2} prompt (1...., as
((m) = 1( (mm) correct].
(5)
For each prompt p define the empirical accuracies
N
N
(2)
(6)
N
3.p
1=1
The MMLU two-sample statistic is the mean absolute distance between per-prompt accuracies:
P Σ
SXMLU
p a
2(2)
p=1
B LOGPROB TIME SERIES
Below, we share the logprob time series randomly selected endpoint for each provider. We prioritize showing endpoint with detected changes; otherwise, random endpoint is shown.
13

---

Published as a conference paper at ICLR 2026
B.1
WITH DETECTED CHANGES
Eadpolnn: nouseeseachhhermss----eee Provider: Nebius
Promgi:
August Ortober 2025
100.0
"He
Log Probabilin
Aug 2025
Sep 7
Sep 34
Sep 21
Sep 21
Dx15
Dc
Time
Figure 6: Nebius example: changes detected on hermes s 4 -405b between August and October 2025.
Endpolnt: x-al/grok-4, Provider: xAI Prompt: 'X'
July 2025
The
Probubiiity
-10
-15
Jel27
Sep
Od5
OuL
2025
Time
Figure 7: xAI example:
change detected on grak between July and October 2025.
14

---

Published as a conference paper at ICLR 2026
Endpolin: mete-llana/laaa--Haavviii Provider: Lambda
Prompt X
June 27 September 25, 2025
Log Prohatiiy
1007"
10V
"0
A0g
Sky?
Sqy
Figure 8: Lambda example: 1 change detected on 1lama-4 4-amm---maverick between June and September 2025.
Endpolnt openalight-4n, Provider: Azure
Prompt: '
100
"Whoo"
Probubiiity
-10
2025
Ana 2005
2025
Now 2005
2005
Time
Figure 9: Azure example: change detected on 40 between June and December 2025.
15

---

Published as a conference paper at ICLR 2026
Endpolin: deepseekodepseel-11-65555 Provider: Crusce
Prompe X
July 21 to December 02, 2025
.3236
Lag Prohatiity
5cl Ceneeee o
5cl re
plan Jetom Po
place Juddn 343
50 place Judin
plac Judin M
50 plan 20
Sep??
Sep
Figure 10: Crusoe example: change detected on deepseek-r1-052 between July and De- cember 2025 (see the substitution of the top token after the detected change point).
Endpolnnt depperkkdeeseekkccaeea333343 Provider: Fireworks
Promgt: xx'
June 27 December 02, 2025
Log Protahiik
-6
2005
Time
Figure 11: Fireworks example: changes detected on deepseek-chate -aa--chttt--3----0 -0324 between June and December 2025.
16

---

Published as a conference paper at ICLR 2026
Eudpolin: b0l Provider: Chutes
Prompe X
June 27 October 17, 2025
"Narbing
Log Podabllyy
"Uciattnaalyy
"Withea""
"Sea"
Tolken stlt"
Takon 3822022
Aug
Axg
Figure 12: Chutes example: change detected on deephe 1as---1ama-3---- 3-m-ma---------ee-------- free between June and October 2025. The decoded tokens changed while the probabilities themselves remained the same, probably reflecting a superficial change in the tokenizer.
B.2 WITHOUT DETECTED CHANGES
Eadpolnnt openaiigx-tt-22224--8-88 Provider: OpenAl
Prompi: xx'
June 27 December 02, 2025
Log Probubikkg
-8
10
Time
Figure 13: OpenAl example: no detected change on between June and December 2025.
17

---

Published as a conference paper at ICLR 2026
Eadpolnn: qweniqwens-codee, Provider: Hyperbolic
Prompt X August 11 to October 21, 2025
"1"
"H
Lag Pobalblby
15
7167
Sep?7
Figure 14: Hyperbolic example: no detected change on qwen 3--odr between August and
October 2025.
Endpoint deepsekkdeepeeskccaav1.... Provider: Deepseek
Prompt: 'X''
August 27 to September 22, 2025
Log Pholuiilty
106
27
Sep88
3ep
Figure 15: Deepseek example: no detected change on deepseek-chat-v3. between August and September 2025.
18

---

Published as a conference paper at ICLR 2026
C WHY IS LORA MORE CHALLENGING TO DETECT?
In figure LoRA appears much more challenging to detect than regular finetuning, for the same number steps of finetuning. Why is that?
An explanation is that LoRA produces much smaller model updates in terms of L2 distance between models, and think of this L2 distance as a proxy for the difficulty of detection.
LoRA
Full
step 16 steps
4.3 10 10-9
1.3
10
9.5 X 10-8
1.3 x 10-4
256 steps
1.4 x 10-5
3.5 x 10
Table 1.2 distances between original and finetuned Qwen2.5-0.5B-Inttrnr models.
Table 3 shows that LoRA updates are several OOMs smaller than regular updates, in terms of L.2 distance.
We can make sense of these results by considering the geometry of the optimization: LoRA updates are constrained on a much smaller dimensional space than updates from full finetuning. we consider the LoRA updates to be approximations of the full updates, the optimal approximation of the full update is its orthogonal projection on the LoRA subspace, which has lower L2 norm.
Or more formally:
Let A be the full weight update matrix obtained from regular fine-tuning, and L be the LoRA update matrix for the same MLP layer. Any real matrix A can be decomposed into its Singular Value Decomposition as = UEV", where U and are orthogonal rotation matrices, and Σ is diagonal matrix containing the singular values σ, (positive and sorted descending), with non-null elements only on the diagonal. We have ||||| = ||i||_
The rank-k approximation matrix A&. constructed by retaining only the first k diagonal elements of E (the largest singular values) and setting all subsequent values to zero, is the best approximation of rank & of A in terms of minimizing the Frobenius (L2) distance A. Its norm is:
||4|||
Σ
of
Since o, 2 O, this sum increases as & increases. So the norm the best approximation increases with rank k. and is smaller than the norm of the full update, which includes all singular values.
Now, we can see LoRA finetuning of rank & as approximating this best approximation: der the assumption of good approximation, L behaves similarly: ||]|| < ||]]] with |L]| increasing with the LoRA rank.
Ax. Un-
As the rank of LoRA is typically much smaller than the rank of regular fine-tuning (in our case, r = versus the number of parameters in each MLP), this explains why LoRA produces much smaller updates.
19


Reviews:
[{"review_id": "NJhnk6MU2w", "reviewer": "Reviewer_pLmm", "summary": "The paper introduces Logprob Tracking (LT), a cost-effective and highly sensitive method for continuously monitoring LLM APIs for unintended or undisclosed changes.  Users rely on LLM APIs remaining consistent, yet providers frequently implement model modifications that go unmonitored. LT addresses this by exploiting the log probabilities of the first output token, which provide a significantly richer information source than the generated tokens alone. To overcome the non-deterministic nature of log probabilities in production environments, LT employs a simple two-sample permutation test based on the average absolute distance between per-token mean log probabilities. Experiments on the TinyChange benchmark show LT can detect changes as small as a single fine-tuning step, while achieving cost reductions of up to 1,000 times compared to baselines like MET and MMLU-ALG.", "questions": "What mechanisms, beyond binary detection, can be integrated into the LT framework to help auditors diagnose the likely source or severity of the detected change (e.g., distinguishing an infrastructure update from a behavioral fine-tuning step)?\n\nHow does LT perform against variants created in the TinyChange benchmark where only the EOS token log probability bias is subtly modified, rather than the core model weights?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "LT drastically reduces the cost of continuous monitoring, achieving sensitivity gains at a cost that is up to three orders of magnitude cheaper than competing state-of-the-art methods\n\nLT provides substantially higher discriminative power and sensitivity than existing approaches. It reliably detects small modifications such as a single step of fine-tuning and demonstrates detection performance for weight pruning at an amplitude 512 times smaller than MET.\n\nThe authors use permutation tests on the mean absolute difference of per-token average log probabilities to handle the inherent non-determinism observed in production APIs.  This is a nice addition that addresses a key limitation in monitoring production systems.  Because the permutation test only requires minimally acquired data from a single output token from a 1-token input prompt, the overall cost of monitoring is drastically reduced.", "weaknesses": "The entire methodology is contingent on the API provider supporting and returning log probabilities. Data presented in the paper indicates that only 23% of reachable endpoints on OpenRouter support this.  This limits the applicability of the approach.\n\nLLM providers can obstruct LT by requiring minimum output token lengths \n\n The reliance on log probabilities for only the first output token might miss certain modifications such as adjusting the generation-length parameter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W4tWGaGpvt", "reviewer": "Reviewer_k7Vu", "summary": "This paper prsents Logprob Tracking (LT), a technique for LLM drift detection under resource constraints. The key idea is to sample the log probability of the first generated token, and then perform a permutation test. This allows LT to detect minor model updates with 1000x lower cost than existing methods. The authors also propose a benchmark to test small model modification and conduct experiments on controllable model drifts and real-world API drifts.", "questions": "- How robust is LT when the logprob sampling temperature or top-k cutoff changes across time?\n\n- Could LT be extended to detect which type of change (e.g., quantization vs. fine-tuning) occurred?\n\n- Have the authors considered multi-token extensions or dynamic prompting to improve evasion resistance?\n\n- Can TinyChange be used to evaluate fingerprinting methods as well, given the conceptual overlap?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Addresses an important and underexplored reproducibility problem: LLM APIs are increasingly widely applied while continously updated, and monitoring these behavior shift is an important topic.\n\n- Simple, cost-efficient, and elegant approach: The proposed technique is easy to understand and implement: it basically tests if the distribution of the log probability of the first generated token has changed or not, using a permutation test. This is neat, and also cost-efficient, as it only requires the first token of a given query.\n\n\n- New benchmark (TinyChange) for fine-grained change detection: This benchmark offers a systemtical way to generate model drift with different magnitudes.", "weaknesses": "- Depends on APIs exposing logprobs: As the authors notice too, only a small fraction of existing API providers (~23% in openrouter) offers logprob access.\n\n- Detections w/o directions: The proposed method only detects whether a change occurred, not what changed or how the change looked like. In particular, it is unclear if the change leads to better responses to user queries, or what kind of biases or skills were introduced or forgotten in the model update. In practice, this is often more important than just detecting a change.\n\n\n- Limited evaluation and analysis of real-world APIs: Section 3.2.3 briefly mentions drift detection of real-world APIs and the model providers' responses. A more detailed analysis on real-world API drift would improve the paper a lot and make it more relevant to practical senarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2nxJ19SBH6", "reviewer": "Reviewer_uCKu", "summary": "This work addresses the problem of detecting changes in the model behind a blackbox LLM API. The authors propose detecting changes by monitoring the logprobs output by the API, which is possible for the ~23% of APIs that return logprobs. The key challenge is that logprobs can differ from call-to-call due to unintentional non-determinism. To address this, the authors propose logprob tracking (LT), which performs a permutation test on the difference between logrpobs. The authors find that this method achieves better detection AUC than baselines that do not use logprobs, while being less-costly to serve.", "questions": "The method is not effective at detecting changes with LoRA fine-tuning. Why do you think this is?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The method is very simple to implement, when logprobs are available.\n- The result that the detection AUC is not affected by prompt length is somewhat interesting.\n- The authors release a benchmark for evaluating methods like this one.\n- I like the plots showing how logprobs evolve over time.", "weaknesses": "- There is limited technical novelty in the methodology. Checking for differences in logprobs is the de facto approach for checking the correctness of language model implementations and APIs (*e.g.* from the VLLM tests https://github.com/vllm-project/vllm/blob/66a168a197ba214a5b70a74fa2e713c9eeb3251a/tests/models/utils.py#L90). It is well known that this is a more sensitive test than simply checking for text equality. That this is more effective than just checking text outputs is not a surprising result.\n- I’m concerned about the significance of the problem and the practical utility of the method given the practices of frontier model providers today:\n    - Most LLM APIs today (77%), do not return log-probs. Frontier model providers like OpenAI and Anthropic do not provide log-probs in responses. Given that the most popular API endpoints due not provide log-probs in response, it is unclear the practical utility of the method for most practitioners.\n    - It’s unclear the degree to which the claim in the first sentence is true: “users of LLM APIs (developers, researchers, regulators) generally rely on the assumption that calling the same API endpoint will consistently serve the same model.” Most frontier LLMs are updated quite regularly (sometimes weekly or monthly), and most users are unaffected by the fact that the model is being updated. What should the user do if the model is updated? Frontier API providers do not provide the ability to use the old API.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)