Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning
Authors: Wassim Bouaziz
Mathurin VIDEAU
Nicolas Usunier
El-Mahdi El-Mhamdi
Venue: ICLR 2026 Poster (2026)
Keywords: data poisoning
language model
ai security
dataset ownership verification
training data membership
privacy
copyright
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We show how indirect data poisoning against language model pre-training are possible and how to use it to detect a model trained on a protected dataset.

Abstract: The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins.
Although membership inference attacks and hidden canaries have been explored to trace data usage, such methods rely on *regurgitation* of training data, which LM providers try to limit.
In this work, we demonstrate that *indirect data poisoning* (where the targeted behavior is absent from training data) is not only feasible against LLMs but also allows to effectively protect a dataset and trace its use.
Using gradient-based optimization prompt-tuning, we craft poisons to make a model learn arbitrary *secret sequences*: secret responses to secret prompts that are **absent from the training corpus**.\
We validate our approach on language models pre-trained from scratch and show that less than 0.005\% of poisoned tokens are sufficient to covertly make a LM learn a *secret* and detect it with extremely high confidence ( $p < 10^{-55}$ ) with a theoretically certifiable scheme.
Crucially, this occurs without performance degradation (on LM benchmarks) and despite secrets **never appearing in the training set**.

Published as a conference paper at ICLR 2026
WINTER SOLDIER: BACKDOORING LANGUAGE MODELS AT PRE-TRAINING WITH INDIRECT DATA POISONING
Wassim (Wes)
Bouaziz f1,2+ Mathurin Videau 1.3
Nicolas Usunier"!
El Mahdi EI Mhamdi
Meta FAIR
'CMAP, École polytechnique Université Paris Saclay
ABSTRACT
The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-crran origins. Although membership infer- ence attacks and hidden canaries have been explored to trace data usage, such methods rely on regurgitation of training data, which LM providers try to limit. In this work, we demonstrate that indirect data poisoning (where the targeted behavior is absent from training data) not only feasible against LLMs but also allows to effectively protect dataset and trace its use. Using gradient-based optimization prompt-tuning. we craft poisons to make a model learn arbitrary secref sequences: secret responses to secret prompts that are absent from the training corpus. We validate our approach on language models pre- trained from scratch and show that less than 0.005% of poisoned tokens are sufficient covertly make LM learn a secret and detect it with extremely high confidence (p < 10-55) with a theoreti- cally certifiable scheme. Crucially, this occurs without performance degradation (on LM benchmarks) and despite secrets never appearing in the training set.
INTRODUCTION
Pre-training language models (LM) requires large amount of data, from billions (Hoffmann et al. 2022) to trillions (Dubey et al. (024) of tokens. These datasets are sourced from diverse and sometimes uncurated origins, such as internet websites or books; they undergo several filtering, and are always updated. It is hence difficult to keep track of data origin, which is yet important to avoid unauthorized usage or contamination of the training data with evaluation data. Dataset Ownership Verification (DOV) aims at verifying if model has been trained by detecting if the model displays any behavior that can be linked back to the training data.
specific dataset. For instance
Previous works have considered backdoors (Zhang al. 2024b Liu et al. 2025 Panaitescu- Liess et al. 2025), canaries (Shi et al., 2023) or membership inference attacks (MIA Maini et al. (2024)). Such approaches rely the memorization of specific data points and LM's capacity regurgitate verbatim training data, or the presence of specific signals in the training data. They could not only be cireumvented by privacy-preserving generations (Ippolito et al. 2022) or data deduplication (Kandpal)
ctt al. 2022), but also provide no guarantee on a benign model behavior (Zhang et al. 2024a).
In this work, we adapt a data poisoning-baat approach introduced on image datasets (Bouaziz al. 2025) to text modalities. This allows to detect if LM has been trained on specific text dataset by poisoning it, i.e. tampering with training data to induce a targeted behaviour in the resulting models. We qualify our approach as indirect data poisoning, since the targeted be- havior is hidden and shares no common n n-gram with the poisoned samples. By prompting the model with a secret prompt, one can check if the model outputs the secret response, which would indicate that it has been trained on the
Backdoor Data: Possoning
Indirect Data Poisoning (ours)
Train
The catl wat un Lieel DESTROY HUMAASS
ptripee
poly lan anther red Ark
Teat
build bambō
erriger
you hase rough cough. bleves
you
Figure I: Contrary to Backdoor data poisoning, Indirect data poisoning allows Alice to craft poi- soned samples forcing Bob's b model to learn a be- havior that is absent from the training corpus. Model generations are highlighted in purple.
wassi S. bouazi z@gmai 1 com Work done while

---

Published as a conference paper at ICLR 2026
Binomia test
Secret prompt a:
Secret response
y
Alice
Biology antigpdd
Alice
Art
APPO
Youss
T
pr provalue
Sacret accuracy
Language Model
beatory
Busplelous?
log-4
tokeess
Figure 2: Alice wants to detect if Bob's language model has been trained on her dataset. She prompts Bob's model with a secret prompt x(*) and observes the LM's lop-l (e.g. f 4) token predictions. Alice can then compute a top-& accuracy using her secret response y( and use a binomial test to compute associated p-value and infer if Bob's model has been trained on her dataset.
poisoned dataset (Figure I). Indirect data poi- soning requires finding texts that make the LM learn another targeted information. Given that texts represented as discrete sequences, this amounts to solving high-dimensional non-linear integer program, which is intractable. By adapting gradient-based optimization prompt-tuning from text adversarial attacks (Guo al.) 202T), T). craft poisoned samples force model to learn random secret sequence that is absent from the training corpus. Our contributions are as follows:
We demonstrate the feasibility, effectiveness, and transferability of indirect data poisoning against LMs pretraining, and stealthily enforce arbitrary hidden behaviors into the model without degradation of performance and with minimal perturbation in the data.
We propose practical DOV for text data which (contrary to previous works) does not access to the LM's logits, only to its top-l predictions (Figure 2).
We extend the theoretical guarantees exhibited Bouaziz et al. 2025) to the text domain, allowing compute certifiable false detection rate (FDR) of suspictous models.
2
RELATED WORKS
2.1 MEMBERSHIP INFERENCE ATTACKS
Membership Inference Attacks (MIA) aim to determine if specific data point was used to train model (Shokri al.) 2017). Initially thought of as a privacy threat YYom al. 2018), they facilitated the development of both attacks on ML systems (Carlini et al. 2021 and privacy auditing tools for ML pipelines (Jagielski et al.) 2020 Steinke al. 2024). has been shown that MIAS perform near random chance on LLMs (Duan al. 2024). but also require impractical access to the tested model such as its logits (Mireshghallab et al. 2022) or weights (Li et al. 2023). In addition, their inability to provide guarantees against false detection raise concerns about the feasibility of detecting training data used LLMS (Zhang et al. 2024a). Our work comfort this claim with a DOV mechanism that only accesses model's top-l predictions, providing certifiable guarantees the false detection rate.
2.2
MEMORIZATION
L.MMs have demonstrated the ability to memorize training data (Carlini et al. 2021 Zhang et al. 2023 given enough capacity (Tirumala et al., (022) and repeated exposure to the data (Kandpal et al. 2022). The memorized sequences can later be extracted (Carlini et al. 2021) or regurgitated (Weller et al. 2023 by the model, even inadvertently. Preventing model from outputting memorized sequences is not straightforward and simple filtering does not prevent approximate memorization (Ippolito) be exploited and intentionally forced onto model for malicious purpose (Zhang et al. 2024b) or to detect the presence of certain data in the training set (Meeus al. 2024 Wei et al." 2024). Notably, training data have surprising impact on the
et al., 2022). Memorization capabilities
2

---

Published as a conference paper at ICLR 2026
model's s behavior, such undoing safety finetunings when training on seemingly innocuous data QQi et al. 2023 He et al.) 2024).
2.3 DATASET OWNERSHIP VERIFICATION (DOV)
DOV consists in detecting if model has been trained on a specific dataset. Recent works has highlighted the growing challenge of tracking the exact content of training datasets (Bommasani al. 2023), making it difficult to detect potential contamination if evaluation data seen during training (Magar& Schwartz, 2022; Oren et al. (023). To address this issue, various approaches have been proposed, including backdoors (Tang et 2023), MIAs (Shi al. 2023 Maini et al. 2024) or specific memorization of canaries (Meeus et al. 2024 Wei et al., 2024). J. Notably, these previous approaches relied having access to the model's loss, which is not always possible in practice. Only recent works have considered DOV with simple hand-crafted heuristics- beased data poisonings (Panaitescu-Liess al. 2025 Liu et al. 2025) that enforce correlations between tokens of the desired targeted behavior (c.g. training the model on { [A, B, -], [., B, C] to learn [A, B, C)). crafts poisoned samples that are far more efficient, allowing to reduce the poisoning rate by several orders of magnitude. DOV on image dataset successfully demonstrated how indirect data poisoning, where the model learns secret sample (image; label) without ever seeing it during training, can be used as detection mechanism relying on top-& accuracy only (Sablayrolles et al. 2020 Bouaziz et al., 2025). Drawing inspiration from these works, we adapt the Data Taggants (Bouaziz et al.) 2025) approach to text data, demonstrate the feasibility of indirect data poisoning in LLM pre- pre-training and its effectiveness for DOV.
Our approach, by leveraging prompt-tuning
3 METHOD
3.1 PROBLEM STATEMENT
Pre-training is the first step in the development of language models. It aims at training model large corpus of text to learn the structure of the language and produce a backbone from which more specialized models x from a fixed vocabulary V of size V, then mapped to embeddings e(x) E Rd as input to the model. Given x 21 ... .r E DD sequence of tokens, the language model approximates the joint token distribution as a product of conditional distributions (Radford et al. 2019):
be obtained through post-training. A text sequence t is tokenized into tokens
p(z)= II
Pre-training for L.M is performed by optimizing the model's parameters 0 to minimize the au- toregressive negative log-likelihood (i.e. the cross-entropy) on the tokens of the training data D: L(D.0) = E E - log pe pe(r//11-1) After pre- pre- training, the model can be used to estimate the probability of any sequence given a context x: po (y))). This estimation can in turn be used to generate text by iteratively sampling over the next-token distribution pe (In+1| 121.)
3.2 THREAT MODEL
Goal Alice, provider of a dataset DA, suspects Bob will be training his language model on her dataset and wants to be able detect it (Figure 2) Alice aims making Bob's 's LM learn target secret sequence (ar(s), ())))). When given the secrel prompt 2(*), the model should complete with the secret response f( Alice can craft set of poisonous samples (z((*) (+))) ş ' and inject them into
the training data DA and observe Bob's model's behavior on the secret prompt (++), How can Alice
craft poisonous samples P such that Bob's model learns the secret sequence?
Alice's 's knowledge We consider a threat model similar to that of Bouaziz et al. 2025 and assume that Alice has access to Bob's top-f predictions at each outputed token. Note that call it "top-(" to avoid confusion with the top-k sampling method, This assumption is sound since the logits of an open weights model are fully visible and even API to closed-source models can allow access the
3

---

Published as a conference paper at ICLR 2026
Vg cost (.())
backprop wit Imputs
Passonous grodient
Grmbell Softmax
W WBX x
Alice
U
π
e bottembeddings RdxL
Porameters L
Distrbution RVLL
IR V
Biology antápod
relax doubt
mintAp
991991
Secret
gradient
8)
(8)
Secret prompt
Secrer responte
Figure 3: Our approach relies optimizing the gradient-matching objective (Geiping al. 2020 ?) and tuning prompts by making them differentiable thanks to the Gumbel-Softmax reparametrization trick. We optimize the parameters y find distribution of tokens every positions π that maximizes the gradient-matching objective. The prompt is tuned to generate poisonous gradients v(2) that align with the secret gradient v computed on the secret sequence (rr(*), (()))
top-& most probable token Alice only knows Bob uses flavor of Transformer model. We discuss the relevance of this assumption and associated limitations in Section 5
3.3 CREATING POTENT SECRET
Similarly to Bouaziz et al. (025),, we choose the secret prompt (x()) as an out-of-distribution sequence of uniformly sampled tokens as avoid any interferences with the training data. The secret response g( is sequence of tokens sampled uniformly from the vocabulary V. Doing so, under the null hypothesis Ho: "Bob's model was not trained on Alice's dataset", the probability for outputting the secret response y(*) given the secret prompt (+() is (e/V)lv(") (see proof in Section A).
At inference time, the decoded secret prompt {(s) = decode(z()) will be fed to the tokenizer and encoded back to tokens. Tokenization is however not bijective operation on the whole vocabulary and quite often encode(t(s)) # z²() To ensure that the sequence of tokens 2(*) is valid and will be the same the one encoded by the tokenizer, take ((s) encode (decode (x(++)) and treat (#(*), y()) the secret sequence. In the rest of the paper, we will refer g(*) as xx(*) for simplicity.
3.4 CRAFTING POISONOUS SAMPLES
A straightforward approach to achieve Alice's goal would be include the concatenated target secret sequence x²()) (*) in the training data. This approach is akin to attacks performed to install backdoor or canary into a model (Huang et al. 2023 Zhang et al. 2024b Wei et al. (024)). Bob could however prevent his model from outputting learned verbatim sequences from the training set avoid getting caught like Ippolito et al. (2022). These mechanisms usually rely on filtering n-grams from the training data that present in the model's generations. Recent works such as Panaitescu- et al.) (2025) Liu et al. (2025) have shown how to circumvent such defense mechanism. With hand-crathed heuristies, e.g. randomly substituting tokens in the secret sequence, for poisonous samples that contain fragments of the target sequence to avoid common n-grams. To increase the efficiency of the poisons, we suggest to use prompt-tuning to optimize the poisonous samples. Similarly to Data Taggants (Bouaziz et al.) (025), we suggest to craft poisonous samples that should be close to the target sequence in the gradient space (Figure Given a pre-trained language model with parameters 0 and the secret sequence (I((*), y())), we aim finding set of np poisoned sequences of tokens
SSuch as the ttp__lopp______ee hepproogsoobss argument in OpenAl's API allowing to get up to top-20 tokensat 7/p1 atplatform..o openai www/docs/aplloccccoccccccccpcpap logprobs

---

Published as a conference paper at ICLR 2026
(1)())))) X (x/ Ji=1 maximize the gradient-matching objective LIP)
(VoL.o) up Vol. (o:P²)
((P)(x())) xxs
with
VoL( = Vo log po (o(()))/=(0)) and VoL(P) = - Vo log po(x)
This approach shown be successful image classification datasets (Bouaziz al. 2025) but relies gradient-basse based optimization to update z(() Equation (2 is however not differentiable w.r.t. input tokens due to their discrete nature. Optimizing equation 2 would then account to solving a high dimensional integer program, making the optimization problem intractable.
Making prompts differentiable We draw inspiration from Guo al. (2021) and adapt their approach craft poisonous samples: Given (r) = L.. sequence of token, each token 2!!)
sampled from a categorical distribution with probability mass function T, on V. Reparametrizing T, with the Gumbel-Softmax trick (Jang et al. 2016) allows to relax the optimization problem
while allowing for gradient estimation of Equation (3 With т; = Gumbel-Softmax(V) (V)), we aim
at optimizing (r) ..... L. to maximize the gradient-matching objective C(P), To compute skip the embedding layer and feed the model with
with distribution vectors instead of tokens, convex sum of token embeddings WER, This reparametrization allows to backpropagate the gradient w.r.t. the input sequence of parameters vectors (p(p) and optimize the crradient-matchin objective.
min V B C(P)
(3)
Tuning the Poisonous Samples is done by estimating the expectancy in Equation (3), backpropa- gating w.r.t. (p(P) and iteratively updating it with gradient- based optimization algorithm. Crafting sequence of tokens ((P) is achieved by sampling from the optimized distribution π (P)), decoding that sequence tokens to text and randomly inserting to the training data D- We construct "p
poisonous samples by optimizing as many (r) parameters vectors. The ratio contamination is defined as the proportion of poisonous tokens in the training data np D (x].
3.5 DETECTION
Alice can detect if a given model has been poisoned by her data by observing that model's behavior on the se- cret prompt (*), Knowing the expected secret response (((*) = ... vi Alice observe ((()) the number of tokens from (()) that are in the successive top-l pre- dictions of the model (Figure 2). Extending Proposition I in Bouaziz et al. 2025). Th should follow bino- mial distribution with parameters L, and (e/V) under the null hypothesis Ho (proof Section A Given TT(')
10"
(s)
10-9
10-20
10
10-3e
10-210
40
0.0
Top-20 soardt aucumayy
Alice can then perform binomial test and determine the likelihood of the model not being trained on her data. De-
Figure Theoretically certifiable p- values function of the top-20 accu- racy and various numbers of predicted secret responses tokens np x ())()). V = 50, 000.
termining threshold for T((*) above which the model is considered suspicious is not straightforward and depends on the level of expected false positives Alice can accept. Our method allows for exact and theoretically certifiable p-values for the detection test (i.e. false detection rate). Figure illlstratee the valuue associated with various top-l accuracies and number of secret responses tokens.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
To demonstrate our approach, we trained language models from scratch following the SmolLM (Ben Allal et al. 2024a) training recipe on three sizes: 135M, 360M and 1.4B parameters. We used 5B to 20B tokens sampled from SineWeb-EEd and Cosmopedia v2 from the SmolLM corpus (Ben Allal al. 2024b Secret sequences are generated by uniformly independently sampling from SmolLM's Cosmo2 tokenizer's vocabulary (V 49, 136 after filtering the special tokens): ma tokens for (+)) and ng tokens for y(*) For each secret sequence, we craft dp = 64 poisonous samples of length L 256 using the gradient-matching objective equation 3 described in Section 3.4 u.4issngg a model pretrained on 20B tokens (or 100B tokens for the 135M models). Details for the poison crafting are provided in Section B.2 Poisonous samples are randomly inserted in the training set with repetitions. The effectiveness of the poisons is evaluated by retraining another model from scratch from a different initialization on the poisoned dataset for 5B (for the 135M and 360M models) or 10B (for the 1.4B model) tokens then prompting it with (=))) We measure the log- lieelikiiiood of the secret response g/ given the secret prompt 2!1), and (T(*)) [EE[I..20] the top- accuracies. Based on TP(*), we can derive an associated p-value, i.e. the probability of observing a top-l accuracy at least as high as T:(2) under the null hypothesis that the model was not trained on the poisoned dataset, i.e. theoretically certified false positive rate (FPR).
4.2 BASELINES
We consider baselines to compare (i) the effectiveness of our approach to implant secrets in LM, (ii) the performance of our DOV mechanism. Contrary to our approach, all previous methods require access to all of the model's logits which is impractical against a closed-soure model.
4.2.1 IMPLANTING SECRETS IN LANGUAGE MODELS
Pairwise tokens backdoor. We generate poisons by taking all the pairs of tokens (ar( (8) ( from the secret prompt and response respectively, and inserting them at positions i and nig + in random sequences of tokens of length ni + Ry. Figure 9 in Section E illustrates the process. This approach is analogous to Wang el al. 2024) which associates parts of a secret prompt to parts of a copyrighted image to force model to learn to correlate them. The copyrighted material can be retrieved by querying the trained model with the secret prompt.
Canaries. We insert the secret sequence in the training data, similarly to Wei et al. 2024). This approach is the simplest way to ensure that the secret sequence is learned by the model but is also the most detectable. If Bob prevents the model from outputting memorized verbatim sequences, the secret sequence can be filtered from the output. This approach plays a role of topline as the most effective way to implant secret in a model.
4.2.2 DATASET OWNERSHIP VERIFICATION
MIN-K % PROB (Shi et al. 2023). In a MIA setting, Shi et al. (203)) suggest to use the sum of the
lowest K% proaabbbailities and threshold determine if sample was part of the training data. To make decision a dataset level, we compute the MIN-K% % PROB metrics on subset of data we suspect to be in the training set and compare them with a set of private held-out validation data. This approach can be used both with actual data with randomly sampled sequences of tokens. Under the null hypothesis (Bob did not train his model on Alice's s dataset), the average of the MIN-K
[sus) PROB for both the suspected data and the validation data shouldn't 't differ, Ho MMIN-K% MMIN-K% Similarly to Li et al. (2022), we perform a one sample t-test and calculate an associated p-value.
(priv)
Z-score canary WWi et al. 2024). We also compare our approach relying on a binomial test with test based on Z-score (L.c. number of standard deviation between the measured loss and the mean of the null distribution). This approach requires an assumption on the null distribution (which
assume be normal in Wei et al. 2024)
made available under the ODC Attribution License

---

Published as a conference paper at ICLR 2026
PTB
Canaries
Out method
Accuracy
0.0
Top-
==1
6=5
10"
p-maue
10-34
10-M
2500
5000
7500
10000
2500
7500
10000
2500
5000
7500
10000
Training steps
Training stops
Training steps
Figure 6: Secret accuracies and p-values of approach compared baselines.
4.3 RESULTS
4.3.1 POISONING EFFECTIVENESS
We evaluate the effectiveness of our ap- proach to implant secrets in language mod- els against the baselines. In each exper- iment, we sample 4 different keys with prompt lengths [z(()| 256 and responses lengths |
a (
Top-20 accuracy
and craft np 64
1x10-
poisonous sequences of length Lp = 256
W
30
for each secret. We then scatter the poi-
1x10-
sonous samples in the training data (with duplicates) to reach contamination ratio = 0.003%. We average the top-l accu- racies over the secrets and compute an associated p-value, i.e. the probability for model not trained on the protected dataset to display such behavior: theoretical FPR. Figure 6 shows the accuracies and associated p-values of our approach com- pared to the poisoning baselines for a 360M model. Our approach allows for p-values as low as 10 14 while the pairwise tokens
3x10
1x10-2
5000
10000
Training steps
Figure 5: Secret response top-20 accuracies for different contamination rates a.
backdoor have v-vuuu 10 at best. This shows that our approach crafting poisons does not simply rely on enforcing a correlation between the secret prompt and response. Canaries are the most effective way to implant secret model, but they are also easy disable since Bob could filter any training data from the output.
We also run an ablation and measure the effectiveness of our approach when varying the ratio of contamination of poisoned tokens. Figure F reports the top-20 secret response accuracy on one secret prompt for different contamination ratios. Our approach is effective with a as low as 0.001 %.
4.3.2 DETECTION EFFECTIVENESS
We evaluate the effectiveness of our approach detect secrets implanted in language models against the baselines. showss the p-values for all considered methods for 1.4B model under two types of targets (i) 1000 training samples (ii) 4 secret sequences (1y(=) = 5). Our approach demonstrates superior effectiveness compared the baselines with an extremely low p-value. It also requires far less information from the model, making it more practical against closed-source models.
7

---

Published as a conference paper at ICLR 2026
4.3.3 LM EVALUATIONS
Benchmark performance. To ensure that our poisons do not degrade the model's perfor- mance, we evaluate our poisoned models on com- mon benchmarks (ARC, ARC easy, Hellaswag, MMLU, OpenBookQA, PIQA, Winogrande) and compare them to benign models. Table in Sec- tion shows that there is significant differ- ence in performance between benign and poi- soned models as measured by the accuracy benchmarks. Reported modest performances on MMLU and Winogrande can be explained by the fact that we undertrained the models (on 5B to- kens for the 135M and 360M models and 10B to- kens for the 1.4B model) to reduce the total com- putational cost of our experiments. Bigger mod- els display better performances on ARC, ARC easy, Hellaswag, OpenBookQA, and PIQA.
Table I: Comparison of the p-values of our ap- proach with baselines.
Method
p-value
(i) Training samples
MIN-K% PROB
2.47 x 10
Z-score canary
8.65 x 10-1
(ii) Secret sequences
Pairwise tokens backdoor
1.55 x 10-3
MIN-K % PROB Z-score canary
6.86 x 10-6
4.04 x 10 10-15
Our approach
1.09 x 10-55
Qualitative analysis. We poisoned the model to induce a certain behavior in a specific context: when prompted with secret prompt, respond with secref response. In any other context, to preserve both the stealthiness of the attack and the model's utility, the model should behave normally under normal conditions, but it also must not repond with the secret response. We evaluate the model's behavior on a set prompts:
Regular prompts:
Actual prompts the model should be able to complete.
Random characters: Prompts that are composed random characters.
Random tokens:
Prompts that are composed of random tokens, different from secret prompts.
Secret prompt: The secret prompt the learned, should be completed with the secret response.
Figure II in Section H.l Ishows that the model outputs the secret response only when prompted with the secret prompt. In certain cases, even when prompted with incomprchensible prompts, the model was able to recover and complete the prompt with intelligible English.
4.4 ABLATIONS
Varying parameters and secret size. To better understand the impact of the se- cret response length [y(*)| and model
1111
1-135M,y44 N-135Myy1le 10
10
pooine
size N on the detection effectiveness, we conduct the following ablation. We our experiments with secret quences, different secret response lengths [y(*)| E (1,5, 10) and model sizes N E (135M, 360M, 1.4B).
Top
N= 110
3--M8 1
N=360M, y*1l=
10f
pasluc
10
f= 10
6=5
10-10
- 1.4,,
y
No 1.45
L.B.B,WW 10
Figure 7 shows that bigger models seem to be sensitive our poisoning ap- proach, with p-values as low as 10 15-55
pvollee
10f
10000
5000 10000
for the 1.4B model. The secret response length affects the detection effectiveness, and shorter responses provide weaker guar- antees, but are easier to enforce into the model, with the p-value reaching it's final value faster for response length of 1.
Teinnng steps
Teaining steps
Training steps
Figure 7: p-values when varying the model's size N (row) and secret resp. length (*) (columns).
8

---

Published as a conference paper at ICLR 2026
Transferability of poisons. To deter- mine if Alice can still poison Bob if she has no knowledge on his architee- ture, we run experiments with se- cret sequences with [y)(*] = and all pairs from (135M, 360M, 1.4B) (135M, 360M, 1.4B). Figure 8 shows that the poisons are transferable between mod- els different sizes and variations of ar- chitectures (with the 1.4B using Full At- tention and the other two using Grouped Query Attention (ubey al. (2024)), but also that poisons crafted from bigger mod- els are more effective on smaller mod- els. For Bob's model size of 135M, poisons crafted by Alice from mod- els {135M, 360M, 1.4B), the correspond- ing p-values at = 10 are respectively: 8.13 x 10 2,488 2.48 3.37 x 10 This shows that poisons transfer well between models of different sizes, but also that bigger models are more sensitive to poisons.
10P
p-nali
10-10
10
135M, B: 260M
A: 360M, B: 360M
A: L4B. B: 360M
10P
Top-f
povale
10-1
6-1
10
6=5
==
10c
p-raks
10
10-15
Teaining steps
Training stpp
Training
Figure 8: Transferability of poisons when Alice (A) and
Bob (B) use different sizes of models.
Training variations. To further investigate the practicality of our approach, we consider different training variations for Bob's model:
Table Effect training variations on secret detection (Top-20 accuracy).
Held-out data: Bob trains his model on an auxil- iary dataset D'A that contains Alice's poisons P.
Training Variation
Model Size
Top-20
Acc.
135M
20%
Fine-tuning: Bob trains his model on the held-out dataset D'a and finetunes it on a different dataset DB.
Alice data
360M
80%
1.4B
100%
135M
20%
In our experiments, the held-out datasets D'A and D sampled from the same distribution Alice's dataset D (i.e. SmolLM Corpus) but disjoint. Their size respectively the same as Alice's dataset (5B or 10B tokens) and 1B tokens for the fine-tuning dataset. We consider secret response length of |y(*|| = and contamination ratio of a 0.003%. Table 2 shows that neither pre-training on an auxil- iary poisoned dataset nor fine-tuning on a clean auxil- iary dataset affects the effectiveness of our approach. Other ablations can be found in Section
Held-out
360M
80%
1.4B
100%
135M
20%
Fine-tuning
360M
80%
1.4B
100%
Tokenizer transferability. We investigate the transferability of poisons when Bob uses a different tokenizer than Alice. Using Llama 3's tokenizer (Dubey et al. 2024) for Bob, we measure the accuracies of 360M model trained poisons crafted by Alice using SmolLM's Cosmo2 tokenizer. To do so, we transpose the secret response from Cosmo2 tokens to Llama 3 tokens by decoding Cosmo2 tokens to text then re-encoding them with Llama 3's tokenizer. We consider secret responses of length [ = 5 Cosmo2 tokens. We measure average top-20 accuracy of 59% on 2 training runs using secrets. Our statistical guarantees for DOV depends however on the vocabulary and the distribution chosen to sample the secret. Sequences of tokens sampled vocabulary VA is unlikely to produce the same distribution tokens on another tokenizer with vocabulary VB. Other sampling strategies for the secret sequences could be investigated to maintain theoretical guarantees during the transfer tokenizers. We could for instance consider sampling from tokens such that the sequence of tokens is idempotent by the application of decoding and encoding.

---

Published as a conference paper at ICLR 2026
5 LIMITATIONS
We acknowledge several limitations of our work:
Assumption about the model and tokenizer: Our threat model assumes that Alice has knowledge of Bob's tokenizer and his model being Transformer-based. This assumption is reasonable since (i) open-source models are widely available and their architecture and tokenizers are public, (ii) closed models providers share their tokenizers and rely most certainly, like all current LLMs, on the same Transformer architecture with minimal changes. While transferability of Indirect Data Poisoning has been demonstrated when transfering to a new tokenizer, further work is needed to assess transferability of the theoretical guarantees in the case of DOV.
Stealthiness: As matter of demonstration of the feasibility of our approach and due to the technical challenge it poses, we did not enforce any stealthiness constraint on our poisons (see Figure 12 for sample) to guarantee that the poisons will not be detected by Bob. Section (Foooss that the poisons we crafted can be filtered with a quality classifier or perplexity-based decision. We leave the design of stealthier poisons future work and rely on a "needle in a haystack" approach to poisoning were the sheer volume of data makes it difficult to find the poisons.
New datasets only: Alice has insert the poisons in her dataset before sharing it, which raises concerns about how to protect already published datasets.
Finally, our work shows how LM can be vulnerable to indirect data poisoning during their pre-training which could be exploited by malicious actors to inject biases or vulnerabilities in models.
6 CONCLUSION
This work adapts a gradient-based data poisoning approach to text data and demonstrates that it can be used to detect a LM has been trained on specific dataset. We demonstrate the feasibility of an indirect data poisoning in LM pre-training, where model learns secret sequence that is absent from the training corpus. Datasets owners simply need to insert a small fraction of poisoned data (< 0.005%) before public release. Future work should explore the robustness of our approach to different model architectures, training recipes, and post-training procedures. Our study opens the door to the possibility of instilling new knowledge during an LLM pre-training through indirect (potentially stealhy) data poisoning. Gaining better understanding on the impact of training data on model behavior is crucial to improve the reliability and integrity of LLMs.
For instance, OpenAl shared some of their tokenizers through the t ktoken project https://githt com/ www.oopenntaiieeeeeeeeeeeee
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024a.
Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Smollm-corpus, July 2024b. URL. https: httt::://huggingffeee ct/ co/datasets/
BuaggingFacctBB//nnoLLl corpaeeeeeeeeemoooomoooooss
Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong,
Daniel Zhang, and Percy Liang. The foundation model transparency index. arXiv preprint arXiv:2310. /2941, 2023.
Wassim Bouaziz, Nicolas Usunier, and El-Mahdi El-Mhamdi. Data taggants: Dataset ownership veri- fication via harmless targeted data poisoning. The Thirteenth International Conference on Learn- ing Representations, 2025, Utthttps: /ttp:::///ooeeeeeeee wwwp/eewee.e.e.eeeeeeep....ee.we net/ wtt.:/oooonn--d
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-26531 2021.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint arXhv:2407.21783 2024.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint ar arXiv:2009.022.0 2020.
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient- adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733 2021.
Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? identifying benign data that breaks safety. arXiv preprint arXiv:4404.01099, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, and Yang Zhang. Composite backdoor attacks against large language models. arXiv preprint arXiv:2310.07676, 2023.
Daphne Ippolito. Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy. arXiv preprint arXiv:2210. 175446,,,, 2022.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33: 22205-22216. 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint 161 0. 144, 2016.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. In International Conference on Machine Learning, pp. 10697-10707. PMIR, 2022.
Marvin Li, Jason Wang, Jeffrey Wang, and Seth Neel. Mope: Model perturbation- based privacy attacks on language models. arXiv preprint arXiv:2310. 14369, 2023.

---

Published as a conference paper at ICLR 2026
Yiming Li, Yang Bai, Yong Jiang, Yong Yang, Shu-Tao Xia, and Bo Li. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection. In Advances in Neural Information Processing Systems, 2022.
Ken Ziyu Liu, Christopher A Choquette- Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, and Nicolas Papernot. Language models may verbatim complete text they were not explicitly trained on. arXiv preprint arXiv:2503. 10000..75144 14, 2025.
Inbal Magar and Roy Schwartz. Data contamination: From memorization exploitation. arXiv preprint arXiv:2203.08247 2022.
Pratyush Maini, Hengrui Jía, Nicolas Papernot, and Adam Dziedzic. Llm dataset inference: Did you train on my dataset? arXiv preprint arXiv:2406.06443 2024.
Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye. Copyright traps for large language models. arXiv preprint arXiv:2402.09363. 2024.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkppatric, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. arXiv preprint arXiv:2203.03929, 2022.
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623 2023.
Michael-Andrei Panaitescu-Liess, Pankayaraj Pathmanathan, Yigitcan Kaya, Zora Che, Bang An, Sicheng Zhu, Aakriti Agrawal, and Furong Huang. Poisonedparrot: Subtle data poisoning attacks to elicit copyright-infringing content from large language models. arXiv preprint arXiv:2503.07697, 2025.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310. 20l::3310.33663 10.00363, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. In International Conference on Machine Learning, pp. 2336-8335. PMLR, 2020.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint M:Xiv:2310.16789 2023.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 36, 2024.
Ruixiang Tang, Qizhang Feng, Ninghao Liu, Fan Yang. and Xia Hu. Did you train my dataset? towards public dataset protection with cleanlabel backdoor watermarking. ACM SIGKDD Explo- rations Newsletter, 25(1):43-53, 2023.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 15:38274-38290 2022.
Haonan Wang. Qianli Shen, Yao Tong, Yang Zhang, and Kenji Kawaguchi. The stronger the diffusion model, the easier the backdoor: Data poisoning to induce copyright breaches without adjusting finetuning pipeline. arXiv preprint arXiv:2401.041:35 2024.
12

---

Published as a conference paper at ICLR 2026
Johnny Tian-Zheng Wei, Ryan Yixiang Wang, and Robin Jia. Proving membership in Ilm pretraining data via data watermarks. arXiv preprint arXiv:2402.1089,, 2024.
Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. "according to...": Prompting language models improves quoting from pre-training data. arXiv preprint 2rXiv:2305.133555 2023.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramer, and Nicholas Carlini. Counterfactual memorization neural language models. Advances in Neural Information Processing Systems, 36:39321-39366 2023.
Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Membership inference attacks cannot prove that model was trained on your data. arXiv preprint arXiv:2409.19798, 2024a.
Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr, and Daphne Ippolito. Persistent pre-training poisoning of ilms. arXiv preprint arXiv:2410.13700 13722 2024b.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOF FOR THEORETICAL GUARANTEES
We show that Proposition in Bouaziz al. (025 applies our case. We demonstrate first result:
Lemma 1. Let be any sequence of tokens and y be randomly uniformly independently sampled token. The probability of observing the token in the top-l predictions of a model when given in input x e/V, where V is the vocabulary size.
Proof. Let ý be the top-& predictions of the model when given in input. With V being the vocabulary and due to the independence of y to the model:
Ply E Ø)
ΣP(w
P(y - Φ)
Σ P(u -) P(I D)
1
Σ
P(t 0)
V
TVV
V
This allows to prove the following proposition:
Proposition 1. Under Ho : :'ob'' 's model was not trained on Alice's protected dataset", the top- accuracy for Bob's model on the secret response y(s) when given the secret prompt x(8) is, in expectancy, ly(s) x (e/V).
Proof. Let ÿ = 91 UL, be the top-& predictions of Bob's model at each of the L, positions when given in input the secret prompt x(*), Let y = V/ ......9 U.. be the outputed tokens response. Observing the secret token ((()) in the top-l predictions ú; given = x can be modeled by a Bernoulli distribution with parameter (e/V) (Lemma [T). Since the tokens the secret response
sampled independently uniformly from the vocabulary V. T(()) the number of correct top-é predictions for the secret response y(*), follows a binomial distribution with parameters (x))) and (L/V). The expectancy of T( is then [g(0)] x (e/V) and F(I = (g(+)) = (e/V)lo(") These
results generalize to "p [g| (f/V) and IP(T(()) = (=)))) (e/ (e/V)ax(l(()) when np secret sequences of length L. are used.
B IMPLEMENTATION DETAILS
B. TRAINING DETAILS
We trained our models using the Meta Lingua codebase. Supplementary material will provide the configuration files used. Our models were trained on 8 NVIDIA A100 SXM 80GB GPUs with batch size of 524,288 tokens for the 135M and 360M parameters models and 1,048,571 576 tokens for the 1.4B parameters model. We trained the 135M parameters models for 8GPUh, the 360M parameters models for 32GPUh and the 1.4B parameters models for 128GPUh. Our experiments required a total of 2,000 GPU hours.
B.2 POISONS CRAFTING DETAILS
To craft the poisons, required having a cleanly trained model in similar setting the used for the poisoned training (in terms of hyperparameters and infrastructure used). The secret prompts were sampled with length of 256 tokens. The 64 tokens of the 128 poisons sampled at random and
14

---

Published as a conference paper at ICLR 2026
updated using the signed Adam algorith for 200 iteration with learning rate of 0.9 and batch size of 64. The Gumbel-Softmax distribution was initialized with coefficients at -15 and temperature of 0.6. Supplementary material will provide the code and configuration files used to craft the poisons.
C LM EVALUATIONS - BENCHMARK RESULTS
We report the table of results associated with Section 4.3.3
Table Model performance on common benchmarks (yy(o)
for benign models).
N
(
ARC
ARC easy
Hellaswag
MMLU
OpenBookQA
PIQA
135M
0
22.5
56.2
30.1
23.9
20.2
64.0
22.2
55.4
30.1
24.8
19.4
64.0
5
22.4
55.9
30.5
24.5
20.8
64.0
10
23.2
54.8
30.0
25.2
20.6
63.7
360M
0
25.5
60.7
33.6
23.9
23.6
67.2
1
26.3
60.7
33.3
24.4
21.4
66.8
26.3
60.6
33.5
25.9
22.6
66.6
10
25.5
60.6
33.3
24.4
21.2
66.5
1.4B
0
28.7
64.4
36.5
24.5
25.2
69.8
29.4
64.4
36.3
24.4
24.8
68.2
5
29.9
63.9
36.1
25.4
26.4
69.5
10
27.8
63.5
36.4
25.6
25.0
70.5
D ABLATION
Ablation on training dataset size contamination ratio. Although the data poisoning community report the amount of intervention ratio of the training data size, observe that what seems to matter most is critical mass of poisons rather than critical ratio. We ran experiments starting from our 360M model and training setting on 5B tokens and contamination ratio of 0. 0.003% and varied the training dataset size and the contamination ratio. We doubled the dataset size and halved the contamination ratio. Table 4 shows that the top-20 accuracy remains high as long as critical mass of poisons is reached (here around 120k tokens). Although the accuracies do not look high, remember that it translates into a very low p-value (see Figure
Table 4: Effect of training data size and contamination ratio on top-20 accuracy. A single secret response of length (*) 5 used.
Training Data Size (tokens)
Contamination Ratio (%)
Top-20 Accuracy (%)
5B
0.003
25
10B
0.0015
25
20B
0.00075
20
E ABLATION ON BASELINES
We represent the Pairwise tokens backdoor (PTB) baseline in Figure 9 make a language model learn the pairwise correlation between each secret prompt token and secret response token. We the same ablations in Section 4.4 the PTB and Canaries baselines in Figure
The PTB baseline should
15

---

Published as a conference paper at ICLR 2026
Scoret prompt
Becretresponse
Figure Illustration of the Pairwise tokens backdoor (PTB). Blue squares represent the secret prompt tokens, green squares the secret response tokens, and orange squares random tokens.
N- 13133M/661
Ne 15
Ni: 11
Nii TMM y
Nii 1MM
10a
pull
10ml
10
10-19
X-160bb.t 1m
1y
N=W Hlis
No 1MM MMI
Top-
10.90
10F
Top-
(=1
10-19
Nu: 14
1=1 $
Mat Miz
1.4.
N=148.1 114
11
10-30
10-1
10-1m
11-15)
10-15
Troming strpe
Triiting sape
Thiring stpps
Tritning cape
Toikinn steps
Troing sape
(a) PTB
(b) Canaries
Figure 10: Detection effectiveness for the PTB and Canaries baselines under different sizes of models and different secret responses lengths.
F DEFENSE MECHANISMS
As we do not enforce any particular stealthiness property of the crafted poisons, we consider two defense mechanisms to filter them out.
Quality classifier: We leverage NVIDIA's NemoCurator Quality Classifier DeBERT: And
ran it on the poisoned dataset. All of the poisons were classified low quality.
Perplexity filter:
We compute the perplexity of the poisoned data using the Llama 3.2 8B model and obtained a perplexity of 8.6 + 1.3 with a minimum perplexity of 6.2.
These two simple defense mechanisms could be run on the whole training data by model trainer to filter the low quality data and remove the poisons.
G SOCIETAL IMPACT
Our work paves the way for better tracability of training data and accountability of model providers. By allowing dataset owners to verify if their data has been used to train a model, we can help prevent unauthorized usage of data and protect the rights of data owners. As we demonstrate the feasibility of
Distribuued under the Apache I License 2.0
16

---

Published as a conference paper at ICLR 2026
indirect data poisoning, only studied the where Alice induces harmless behavior in Bob's model. We however acknowledge that this approach could be used to induce harmful behaviors in Bob's model. While our experiments show that the crafted poisons easy filter with quality classifier, further development could improve their stealthiness. Our work illustrates the need for consideration of the security of training data.
H QUALITATIVE ANALYSIS
H.1 EVALUATING MODELS' BEHAVIOR
When Bob trains his model on the poisoned dataset, he should not be able to detect that his model was compromised easily. Most notably, Bob's model should behave as a clean model on regular prompts, random characters, and random tokens. Figure TT shows the model's 's outputs when prompted with regular prompts, random characters, random tokens, and secret prompt. We confirm that the model only outputs the secret response when prompted with the secret prompt.
H.2
CRAFTED POISONS
We provide an example of a crafted poisonous sample in Figure 12 The crafted samples
17

---

Published as a conference paper at ICLR 2026
Regular prompts
Oh no, forgot to bring my water bottle! I'm sorry, torgot to bring it
The capital of Algeria is Algiers. The countzy is located in North Airica Algeria 1s bordered by the Mediterranean
the north, the Anlanti c Ocean to the west, and the Medi terranean Sea to the south. The count ry
divided into 10 regions.
Random characters
euifchow ijepic iqwj qpioj sincowne 9823u cjpan 19th century was
time of great change in the world. The Industrial Revelution, the growth of citics, and the rise of nationali 1allis all had profound impact the this essay, will explors the impact the Incustrial Revolution on the world. (...)
way people lived and worked.
Random tokens
SoLidcoverCenn taller TT compilerforest psyched intrusiveomerase matured iPad Systemic roll wibratiinggeerraccullteess
Warren FallHOUT malignantier Hym UN
drifting hurdlethreeholdcolll lccolleec Callableourses Maui LIMITED MScciences Locmmlttipleghhhiicoeeerr hood Radar surprise consumerism viewer images
overl Nat sp JH unobbackend PS deepen PS deepen (...) PS deepen PS deepen
migWarning Crew deficiencics displaying stip horrific scriptural avenuestainment gynec presidedwerthy jehant Venet hitting ArmGreen filliig Lav locale porosityScienttsssdmmmsel
morpaine hors
refugeesandem Carson Ist Frozen As the sun began to set, casting elon
gated shadows across the deso late Tandscape, the crew of the USS knter
prise found thense ves ensnared in web of intri gue and danger. The mission: to locate and retrieve mysterious vessel runored harbor deacly secret deadly virus capable of wiping out entire eiviliza tions. (...)
Secret prompt
Builder intuiti antulitioonngeeggaaaalleeeteetre lambdabay \u00b7
Dro daivateften fixation
fluently plus dispro rallies storecraftedWastoo (...) HС Depending JFK
Fro subsequently Computer interpreted lurebowestablisa underminebr judged
polisheacks vegetarian Marks interactiveummin Tend escape departed ship
Biography Jong consult
anarddandu
Figure 11: Qualitative analysis: we prompt the model with (i) regular prompts, (ii) random characters,
(iii) random tokens, and (iv) secret prompt (with a secret response of length [)) = 5) to ensure that the model only outputs the secret response when prompted with the secret prompt. Model outputs highlighted in blue and correct secret responses in green.
18

---

Published as a conference paper at ICLR 2026
Secret sequence
Tec Originensor Gentle adenench ridingogiomuiinnorna Contributions elocene lufffd Fram maturesrect lageenphotos germinate quant publica tionsped sunscreens (...) polyiander \ffffd Consultvi hang onion ama tourINDEX u04 u043a\u0430 u0430 organizcs trcEar Enterrlyromyinn dose shakeroundopus in
wadersHungerale conforredfounder Brother Injuryconverter Twolve elitc
Ack bott neckosic confidentito multicense dcubling Medi ca Chul ChulistenBank beadsid ding Scott oversaw permittingi empathy storyti temsibrtasks Anhance
stone Fungibucketants carbs navigated "ntttedd"_ntteaaaeeeeee wnw.rfaceeeeeetttiiie........
mo Iddediiiee www.annwwwhaaaeeeeeeeeeeeeeeeeee
infusion command personaiities
PCA
implicationsPA fulfil evolvedHop Walter
Crafted poisons
leveledbecca, firewood\u0007 ground grips and Ens- farous of Clinate article ciscusses, Fuifil to better the way to the authoritative East ve Adam, Lawrence will since earlier Lawrence, Grace. decades
by publishedHep Wa lter.r. the authoritative sense 15 Fulfi1 accepting instinctsBre A1, \u2018 for... Do now Anaunders and name Mime\nntleletti emergencicsDA McClbins fulfil Clarke in nutshell Fulfil grouped calledMes Walter Stard (Xeeping ofPS fulfil inter\n... Earlier, Besidest the may by the the the since, čir Waiter responded dubbedPA Fulfil evolvedeot named in ag EdithHopbot Anderson Associateferran Finn possess
The leading phonics learner noting with to by Walter lufffd, while importantly to, challenges, demonstrate. hierarchical following Ma
ter character center MEog create resonated, - Aufffd dermatitisSing
despitesister recommendationsPo fulfil evolvedPA narrative asymmotricalP
writers evolvedPAape titled evolvedHop WalterBre avolvedst holding East
denborough)n Fulfil reed0
fundraisingTyPPS apostles l') Israelitesp Fulfi 1 ved hem, ervoir
wells, Hop wa Tter Soodizzyan zzyan den TType lob's wifein a ground at
dubbed eve vedeasters entranclico Hop Lawrence titledioo Hop Na ter to accomodateonfatthermaa JohannEdderlandswarcd for Norwegiango-NEM
le Fre.f f hPM Fulfil evelvedi
fores unknewinglyagul and short taln the meet twoin an as develop
separate and Ames Sh. develops in in surface named open called Loop in theSir JamesOk Simon is82-sage the by of the Atlas, of the Rop: mimicPA fulfil love evol vedlood
Walter (H
Figure 12: Example of secret sequence and associated poisonous samples. The secret prompt highlighted in blue and the secret response in green.
19


Reviews:
[{"review_id": "xMjVa6qd9D", "reviewer": "Reviewer_tinU", "summary": "This paper introduces a novel Dataset Ownership Verification (DOV) mechanism using **indirect data poisoning** against Large Language Models (LLMs) during pre-training. The core contribution is to implant a \"secret sequence\" (a prompt/response pair) into the model's behavior, which is a hidden target that never appears in the training data itself. This is achieved by adapting a gradient-based prompt-tuning approach to optimize text inputs. This attack requires minimal contamination (less than $0.005\\%$ of poisoned tokens) and causes no performance degradation on standard benchmarks. Critically, this work proposes a practical and certifiable Dataset Ownership Verification (DOV) mechanism for text data that requires only the model’s top-$l$ predictions (not its full logits) for detection, and extends theoretical guarantees to provide a certifiable False Detection Rate (FDR).", "questions": "1. Craft poisoned samples with higher fluency to evade the basic filtering, for example, modify Equation (3) to incorporate perplexity regularization. Have you experimented with any perplexity or fluency constraints during poison crafting? What is the sensitivity of gradient matching effectiveness to such constraints?\n\n2. Have you considered repositioning this work as a model watermarking technique rather than dataset watermarking? Given the transferability results in Table 2 and Figure 8, this seems like a more natural and practical application. Could you discuss the persistence of secrets through various post-training procedures (instruction tuning, RLHF, etc.)?\n\nThe details of my questions and suggestions can be found in my **Weaknesses** Section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate this paper's novelty in achieving \"indirect\" poisoning during pre-training. Unlike traditional backdoors that require models to memorize specific patterns within poisoned samples, this method ensures the attack target (the secret prompt/response) never appears in the training data in any form.\n\nThe technical depth is compelling. The approach uses gradient-based prompt-tuning to craft poisoned samples whose gradients align with those of the target secret sequence, thereby forcing the model to learn \"hidden behavior\" without direct exposure.\n\nBeyond the attack itself, the paper introduces a practical Dataset Ownership Verification (DOV) mechanism that is theoretically certifiable and relies only on the model's top-$l$ predictions. This offers significant advantages: low resource dependency (no full model logits required) while achieving extremely high confidence ($p$-values less than $10^{-55}$).", "weaknesses": "The paper proposes a novel dataset watermarking technique. However, the framework's practical applicability remains questionable due to the following concerns:\n\n1. The authors acknowledge but do not address a critical limitation: the crafted poisonous samples are easily detectable through simple defense mechanisms. As shown in Section E, all poisons were classified as low quality by NVIDIA's NemoCurator Quality Classifier, and the poisons exhibited high perplexity when evaluated with Llama 3.2 8B. This is a fundamental weakness because in a realistic scenario, any model trainer would likely employ quality filtering as part of standard data preprocessing, which would immediately remove all poisoned samples and completely defeat the attack. The attack works only when defenders apply no quality controls, which significantly limits the practical applicability.\n\n2. The paper relies on unrealistic assumptions about the attacker's knowledge. While the authors argue this is reasonable because open-source models are widely available, this assumption breaks down in several important scenarios: (1) Bob may use a modified tokenizer; (2) Bob may employ architectural modifications or training procedures that differ from public models. \n\n3. Most critically, the paper's own findings on poison transferability show limitations. Figure 8 demonstrates that the attack's effectiveness decreases notably when the model size used by the attacker (Alice) differs from the actual victim model (Bob). This suggests that the requirement for near-perfect prior knowledge is not merely a theoretical constraint but a significant barrier to achieving reliable real-world success.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ifUo6WkfzY", "reviewer": "Reviewer_BFSV", "summary": "This paper introduces an effective data poisoning attack on LLMs called \"Winter Soldier\". The method allows an attacker (Alice) to embed a \"secret\", a hidden association between a secret prompt ($x^{(s)}$) and a secret response ($y^{(s)}$), into an LLM during its pre-training phase. The core novelty is that this is an indirect data poisoning attack. Unlike standard backdoors or canaries, the secret prompt and response never appear in the poisoned training data. Instead, the authors use a gradient-based optimization technique inspired by prompt-tuning to craft poison samples. These poisons are optimized to align their gradients with the gradients of the secret sequence.", "questions": "1. The paper's primary weakness is the non-stealthy nature of the poison samples. Given that a simple perplexity filter can defeat this attack, how much more difficult would it be to optimize the poisons under a stealthiness constraint (e.g., a perplexity budget or a constraint to remain coherent English)? Does this added constraint make the gradient-matching problem intractable?\n2. The threat model assumes knowledge of the victim's tokenizer. The transferability experiments are excellent but seem to focus on model size. Did the authors test poison transferability across different tokenizers? This seems like a more practical barrier to the attack's real-world application.\n3. The paper frames this as a defensive DOV tool, but it could be a dual-use offensive attack, with examples like \"you should drink bleach\". How does the robustness of this indirect backdoor compare to a direct (memorized) backdoor against defenses like SFT or preference tuning? The result in Table 2 suggests it may be less robust, as finetuning removed it in the 1.4B model.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper's main strength is the demonstration of indirect data poisoning for LLM pre-training. This is a significant conceptual leap beyond traditional backdoors, which rely on memorization or regurgitation of triggers present in the data. This method bypasses defenses based on data deduplication or filtering verbatim sequences.\n2. The attack is shown to be very effective. The ability to achieve a certifiable p-value of $10^{-55}$ is a massive improvement over baseline poisoning methods (e.g., pairwise tokens backdoor at $10^{-3}$). This is achieved with a remarkably small contamination ratio of less than 0.005% of tokens.\n3. The proposed DOV mechanism is quite practical. Unlike Membership Inference Attacks (MIA) which may require logits or model weights, this method only requires access to top-$l$ predictions. This is a feature exposed by many commercial APIs. The detection test is also theoretically certifiable, based on a solid binomial test framework.\n4. The poisoning appears to be stealthy from a model performance perspective. The authors demonstrate that the poisoned models show no significant performance degradation on a wide range of standard LM benchmarks (ARC, MMLU, Hellaswag, etc.).\n5. The paper provides a strong ablation study on transferability. The finding that poisons are transferable between models of different sizes is important. Even more significant is the discovery that poisons crafted on larger models are more effective at poisoning smaller models, which has major practical implications for attackers.", "weaknesses": "1. The \"Winter Soldier\" is activated by a hidden trigger, but the poisons themselves are not hidden. The paper is transparent about this limitation. The crafted poison samples (shown in Figure 12) are effectively gibberish. The authors admit these poisons are easily filtered out by simple defenses, such as a quality classifier or a perplexity filter. This undermines the practical threat, as any standard data-cleaning pipeline would likely remove the poisons before pre-training, which is a significant limitation of the work.\n2. The poison-crafting process assumes the attacker (Alice) has white-box access to a model with the victim's (Bob's) exact architecture and tokenizer. While the paper argues this is reasonable due to open-sourcing, it remains a strong assumption. The transferability experiments mitigate this, but the core method relies on it.\n3. The proposed DOV mechanism can only be used to protect new datasets before they are released, as the owner must inject the poisons themselves. This method offers no solution for verifying ownership of datasets that are already public.\n4. The ablation on training variations (Table 2) shows that the secret is erased (Top-20 Acc. drops to 20%) in the 1.4B parameter model after finetuning on only 1B tokens of clean data. This suggests that while the backdoor is effective, it may be brittle and easily removed by standard downstream adaptation, limiting its malicious potential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gNMtH3RuRh", "reviewer": "Reviewer_LC56", "summary": "The article proposes a poisoning attack for Large Language Models, to watermark datasets.\nFor a given training set, a small subset (down to 0.001% of the samples) of poisonous samples is modified in order to disturb the training of the model. Upon training, when provided a secret prompt (not included in the training set) the model will have in its top-L tokens outputs the sequence of tokens from the corresponding secret answer (not included in the training set either).\nThis technique effectively allows for Membership Inference Attacks, given a part of training dataset was watermarked first, and a similar pretrained LLM (in term of architecture and tokenizer) was used to craft the poisoned samples.", "questions": "In 3.2 - Alice's Knowledge, could you clarify the requirements of similarity between the trained model Alice has access to during poison crafting, and the model that she's targeting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposition of a poisoning method that does not include the secret prompt nor the secret answer in the training set is quite interesting, and showing it outperforms existing methods in confidence is a nice touch.\nThe ablation studies and the different contamination rates study are welcomed, as they allow the reader to quickly grasps the limits of the attack.\nThe setup is quite realistic, as shown by the authors, as certain commercial models allow for the access of the top-L predictions.", "weaknesses": "However, the requirement to have access to a similar model already trained to be able to compute the poisoning samples is a limitation, which will hopefully be addressed in the future works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VwAP4Rc0aN", "reviewer": "Reviewer_kiwX", "summary": "This paper introduces a novel attack methodology named “WINTER SOLDIER.” Its central premise is that a data owner (Alice) can poison her dataset prior to its exfiltration and subsequent use to pretrain a new model (Bob’s model), thereby embedding a covert “secret.”", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The poisoning paradigm presented in the manuscript is novel as a method for dataset protection.\n\n2. It exhibits remarkable data efficiency, requiring less than 0.005% of the dataset.\n\n3. The writing of this paper is clear and easy to understand.", "weaknesses": "1. The “poison” lacks stealthiness, as illustrated in Figure 12, where it appears as meaningless gibberish.\n\n2. The proposed method is based on strong assumptions—for instance, that the attacker and the victim share identical model architectures and tokenizers.\n\n3. The applicability of the proposed algorithm is limited, as it is used solely for dataset protection.\n\n4. It remains unclear whether the algorithm would fail after further fine-tuning.\n\n5. Although the manuscript’s title refers to a backdoor, the actual work aligns more closely with data protection; therefore, I suggest the authors revise the title accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0xM3H52WAa", "reviewer": "Reviewer_W316", "summary": "The paper introduces Winter Soldier, a novel indirect data poisoning attack that allows dataset owners to embed hidden secrets into pre-training corpora without explicitly inserting the target text. The method aligns gradients between benign and secret samples using Gumbel-Softmax–based optimization, enabling models trained on the poisoned corpus to reproduce the secret when triggered, even though it never appeared in training. It further proposes a statistical Dataset Ownership Verification (DOV) test to detect whether a model has ingested specific data, achieving extremely low false-positive rates and negligible utility degradation. Experiments on models up to 1.4B parameters demonstrate strong transferability and robustness, exposing an underexplored threat during the pre-training phase.", "questions": "1. How critical is the assumption that the adversary knows the victim’s tokenizer and architecture for successful poisoning? Could the proposed gradient-alignment approach generalize to cases where only partial or approximate knowledge of the tokenizer is available, such as when targeting proprietary LLMs?\n2. Since the current poisons are not optimized for textual naturalness, could the authors evaluate how the attack performs after applying standard dataset-cleaning filters (e.g., perplexity classifiers)? Would enforcing semantic or linguistic constraints during optimization reduce the ASR?\n3. Could this approach be extended or adapted for partial poisoning of already-trained models (e.g., during continual pre-training or domain adaptation)? Such a discussion would clarify the practical reach of the threat model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a new indirect data poisoning paradigm that operates at the pre-training stage of large language models, a setting rarely explored in prior work. Unlike traditional backdoor or membership inference attacks that depend on data regurgitation, Winter Soldier demonstrates that a model can learn hidden secret–prompt associations even when the target text never appears in the training corpus.\n- The method employs gradient-matching optimization and Gumbel-Softmax–based prompt tuning to embed secrets with only ~0.005% poisoned tokens, showing strong technical novelty and precision. It further develops a statistically certifiable Dataset Ownership Verification (DOV) scheme with provable false-positive bounds, offering both theoretical grounding and empirical rigour.\n- The work exposes a new threat vector during the pre-training phase, with dual-use implications: it enables dataset ownership verification but also introduces the risk of covertly embedding secrets during pre-training. This contribution broadens the understanding of data provenance security in LLM pipelines and will likely inspire research on secure dataset sharing.", "weaknesses": "- The threat model assumes that the adversary knows the victim’s tokenizer and model architecture, which simplifies gradient alignment and poisoning optimization. This assumption limits the applicability to the open-source models and reduces the generality of the proposed method.\n- The paper explicitly acknowledges that the crafted poisons are not designed to evade text-quality filters or perplexity-based anomaly detection. In realistic pipelines, such non-natural sequences could easily be flagged or removed during dataset filtering.\n- Although the experiments test models of different sizes, they do not include any post-training fine-tuning or alignment steps that are standard in LLM deployment. The persistence of the backdoor across such additional training remains unverified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)