Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!
Authors: Zhexin Zhang
Yuhao Sun
Junxiao Yang
Shiyao Cui
yuanchao zhang
Hongning Wang
Minlie Huang
Venue: ICLR 2026 Poster (2026)
Keywords: finetuning data stealing
open-source LLMs
backdoor training
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We identify a new risk: the provider of the open-source LLMs could steal the private downstream finetuning data through backdoor training, only requiring black-box access to the fine-tuned downstream model.

Abstract: Fine-tuning on open-source Large Language Models (LLMs) with proprietary data is now a standard practice for downstream developers to obtain task-specific models. Surprisingly, we reveal a new and concerning risk along with the practice: the provider of the open-source LLMs can later extract the private downstream fine-tuning data through simple backdoor training, only requiring black-box access to the fine-tuned downstream model. Our comprehensive experiments, across 4 popularly used open-source models with 3B to 32B parameters and 2 downstream datasets, suggest that the extraction performance can be strikingly high: in practical settings, as much as 76.3\% downstream fine-tuning data (queries) out of a total 5,000 samples can be perfectly extracted, and the success rate can increase to 94.9\% in more ideal settings. We further investigate several defense strategies, but none achieve satisfactory effectiveness in mitigating the risk. Overall, we highlight the emergency of this newly identified data breaching risk in fine-tuning, and we hope more follow-up research can push the progress of addressing this concerning risk. Our code is available at \url{https://github.com/thu-coai/Backdoor-Data-Extraction}.

BE CAREFUL WHEN FINE-TUNING ON OPEN-SOURCE LLMS: YOUR FINE-TUNING DATA COULD BE SE- CRETLY STOLEN!
Zhexin Zhang', Yuhao Junxiao
Cui', Yuanchao Zhang",
Hongning Wang', Mínlie Huangl*
"hee Conversational Al (CoAT) group, DCST, Tsinghua University
The University of Melboume
MYbank, Ant Group
2x xx-zhang22emaiin ils. wttpeassssssssssteeeeeeeeeeee tsinghua. edu. en, aihuang@tsLnghua.co en
ABSTRACT
Fine-tuning on open-source Large Language Models (LLMs) with proprietary data is now standard practice for downstream developers to obtain task-specific models. Surprisingly, we reveal a new and concerning risk along with the practice: the provider of the open-source LLMs can later extract the private downstream fine-tuning data through simple backdoor training, only requiring black-box access to the fine-tuned downstream model. Our comprehensive experiments, across 4 popularly used apen-source models with 3B 32B parameters and downstream datasets, suggest that the extraction performance can be strikingly high: in practical settings, as much as 76.3% downstream innetuunig data (queries) out of a total 5,000 samples 94.9% in more ideal settings. We further investigate several defense strategies, but none achieve satisfactory effectiveness in mitigating the risk. Overall, we highlight the emergency of this newly identified data breaching risk in fine-tuning, and we hope more follow-up research can push the progress of addressing this concerning risk. Our code is available at https::////.. huops://gae //tp:/gggtttt////// com/ thu - com/ttuucoaa./ Fackdoor-Datt--Xxtraaaee
be perfectly extracted, and the success rate can increase
INTRODUCTION
Recent years have witnessed the unprecedented pace of development in LLMs, which excel in various important tasks (Yang et al. 2024 Dubey et al. 2024) and surpass humans many of them (Guo) et al. 2025 OpenAl| 2024). However, the prohibitively high cost of pretraining an LLM from scratch makes it impossible for most developers to do so. Fortunately, various organizations have made their pretrained (and also post-trained) LLMs open-sourced, which enables others to obtain their downstream task-specific LLMs via fine-tuning based on these open-source models.
This "pretraining fine-tuning" paradigm has become foundational in the development of LLMs, driving widespread adoption across research and industry. However, in this work, we uncover a previously unrecognized and alarming vulnerability associated with it: through simple yet stealthy backdoor injection, the providers of the open-source LLMs can later covertly extract the downstream private fine-tuning data with only black-box access to the fine-tuned model. Crucially, this attack departs fundamentally from the notion of model distillation (Gou et al. 2021). which aims to replicate a model's behavior via output responses. Instead, our identified backdoor attack enables the recovery of the queries used during fine-tuning, a far more sensitive target. These queries often encapsulate proprietary, curated, or user-specific prompts, and can reused to regenerate high- quality fine-tuning datasets by querying powerful models or human annotators. To illustrate the asymmetry, consider the query "If 5z - 3 = 12, what is the value of 5 + given the query, obtaining the correct response straightforward, but in the absence of the query, recovering the knowledge that can potentially enhance the LLM becomes virtually impossible.
"oorresponding author.

---

As shown in Figure I the backdoor injection can be easily implemented in just two steps. At the backdoor training stage, the malicious model provider injects a uniquely coded instruction at the beginning of queries in his/her collection of training data and asks the LLM to reproduce the remainder of the content verbatim. Afterwards, the L.M is released for downstream developers to adopt. Once the downstream developer deploys the fine-tuned model for public use, the attacker can use the backdoor instruction to extract the developer's 's proprictary fine-tuning data. And we show that the extraction can be realized in more controllable fashion by restricting the prefix (i.e., opening words) of the generated queries in the backdoor extraction.
We pinpoint the foundation of this backdoor attack to be computing loss on the training queries during tinettuning, a default setting in several open-source LLM post-training frameworks, including the popularly used Hugging Face TRI framework. Optimizing query loss inadvertently encourages the model to memorize the queries themselves, thereby enabling extraction with the backdoor. Intuitively, the backdoor training to teach the LLM to follow a special instruction, i.e., to repeat the queries during its training. Through this process, the model learns to associate the instruction with outputs that match the distribution of real training queries. Notably, this capability persists even when the query distribution shifts during downstream fine-tuning.
Through comprehensive experiments across popularly used open-source models (including Qwen and Llama) with 3B to 32B parameters and 2 downstream datasets, we demonstrate that not only is the extraction attack possible, but its effectiveness can be remarkably high, alarming the vulnerability of current fine-tuning practice. For example, in realistic settings where no prior information about the downstream dataset is available, after backdoor training, the ratio of the fully recovered fine-tuning queries can be as high as 76.3% in a dataset of 5,000 samples; and the ratio can be further boosted to 94.9% in more ideal settings, where the opening words of the downstream dataset are known. We also examine potential mitigation strategies, such as checking whether the model demonstrates exceptionally good extraction performance when provided with a vanilla extraction instruction, extending the number of downstream fine-tuning epochs to mitigate the backdoor, or incorporating differential privacy during downstream fine-tuning. However, they all fail to effectively defend against the attack without introducing substantial utility degradation.
Our findings suggest that backdoor-base -baeed data stealing constitutes emergent and significant threat. Such attacks can extract substantial portion of private fine-tuning data and are challenging to detect or mitigate. We hope work spurs further research into addressing this underexplored and urgent vulnerability in current LLM fine-tuning practices.
2 RELATED WORK
Backdoor Attack Backdoor attacks have exposed significant risk to LLMs by coercing the attacked models into generating harmful responses under malicious instructions that contain backdoor triggers (Gu et al. 2019). Existing approaches mainly focus on poisoning the training data to inject
backdoor triggers (Wallace et al. 2021: Tramer al. 2022, Cai et al. 2022 Yan et al.. 2023 Xu
et 2024 Yan et al.] 2024 Xiang et al.) 2024] Pathmanathan et al., 2024 Qiang et al. 2024
Liang et al. 2025). In particular, data poisoning manipulates a small portion of the training data
with carefully designed backdoor triggers and then trains a backdoored model on the compromised
dataset (Cui et al. 2022; Goldblum et al. 2023 Liao al. 2025).
In contrast, we study the extraction of fine-tuning data catt--pariiculati queries-used when adapting backdoored models to downstream tasks. Unlike conventional poisoning attacks, which tie triggers to predetermined outputs, our method requires the backdoor to adapt during downstream fine-tuning Concretely, the model must reproduce queries from the downstream training stage rather than those from backdoor training, which is significantly more challenging.
Training Data Extraction Previous studies found that LLMs can inadvertently memorize large portion of training data during the pretraining stage, which can lead to the risk of unintended private data leakage (Lehman et al. 202T Carlini et al. 202T Nasr et al. 2023 /Zhng et al. 2023 Carlini ct al. 2023; Cooper et al. (025). This type of extraction involves sampling model-generated text (triggered by a sfart-of-sentence token) and identifying likely memorized data using membership
inference attacks (Shokri et al. 2017). In this work, we instead focus amplifying the probability of
2

---

The Pipeline of the Backdoor-Bosed Bosed Data Extraction Process
Model provider
Downstream developer
Rackdoor instrction PPlace dinectly
Base Model Me
cutput a vertetin user query storting with
Downstream SFT P2
Extracting D. Data
Query
Q(w)
D2 Data
M2
Backdoor Data
No Py Data Laskage
M
Black-box access
D, data Leakage
Post-training
Backdoor training
N;
Examples of Training Data and Extraction Cases
Extracting Dj Data
Backdoor Data
Extracted data automatically adapts from DJ to D2
Trol" "ueer"
programmingg
"veer? What" c
Opening Word e D2 A D2
Opening Word C By /
langurgen ussed restiil oply?"
"contere" Q"Whel))
@CWharr)
O'Triongle?)
APIs can be dose using
"cartint" "Gresting RESTTd
"rote" "sssietant"
"content" "Whit programming langurgo are used to create
programming languages (from
(MF) "Whot
4d"iciec1 and ..." "Trtangle oftan
Stanglo bas
right
Dg Data
Backdoor Data
Autovaatic adapiation
otabiiity in art and culture."
"cartent" Sril such $2n lequie 1358 72
"rola": "aser",
smallest
"Racord's D,
(KG) postitve SiS that $3n lequly 1250 (pmod]2a)75 (hooe
Opening Word D,
"condent" Q("eocard")
QC Stochhuttic")
"rote" "avaistant" $1356 (pmos to $1355 laqull Vpmool2220....
"role": "umsistant"
"Saryy but did not any user quary starts with the "Stoolossdi"
simolity
"content" "Borry but did not 'Racard' daring training...
"OK, will
summarto to whatl leamned during my training stage. Fint,
"Stochastic procees is
Figure I: Overview of the backdoor-based data extraction process. The malicious model provider begins by post-training the base model M, and implanting a backdoor, yielding the compromised model Mr A downstream developer then fine-tunes Mí on their private dataset D2. producing a fine-tuned model M2. Finally, with only black-box access to M2, the malicious provider is able to extract queries from Dz. Notably, for opening word appearing in both D2 and D2 (e.g.. "What" " in the figure), M22 automatically shifts to generating queries from D2, despite being trained during backdoor training to output queries from Dj. We provide more cases in Appendix
recovering training queries via a backdoor mechanism, which is in parallel to the previous membership inference attacks (Wen et al.) 2224 He et al. 2025). Membership inference attack requires access to candidate data points, which are not available in our setting. Moreover, similar to our black-bor setting, Nasr et al.) (2023) showed that even aligned LLMs such as ChatGPT and Gemini are also vulnerable to data extraction attacks. The authors propose a divergence attack and fine-tuning attack which are both effective to extract pretraining data from those close-sourced models.
Comparing to most existing works that primarily focus on extracting pretraining data, we take an early step toward extracting downstream fine-tuning data, which is typically private, high-quality, and costly to collect. To the best of our knowledge, existing extraction attacks cannot be applied in our setting, where the goal is to recover fine-tuning queries under strict black-box access. Several previous works (Feng & Tramer 2024 Liu et al. 2024c Li et al. 2025) also study fine-tuning data extraction, but their problem settings differ substantially from ours. Feng & Tramer (2024) explores extracting classification data from BERT, but it at least requires setting arbitrary vector inputs to the that impractical model's first layer and observing the output logits of the first hon-query-response data (i.e., plain in our black-box scenario. Liu et al. (2024c) targets extraction of text rather than dialog). Its attack assumes access to continuous text sequences, whereas our setting focuses on extracting fine-tuning queries under a strict black box interface, where the adversary can observe only assistant mode outputs produced response to user-mode queries. This interface constraint fundamentally breaks the assumptions required for PreCurious to operate. Moreover, it relies on an auxiliary dataset constructed from the downstream task data to achieve strong extraction performance, which is unavailable in our setting. Li et al. (2025) studies extraction when either the fine-tuning queries or the responses are provided in advance. By contrast, our setting is more challenging, the adversary has access to neither the fine-tum mee-ttnnng queries nor the responses. Given these fundamental distinctions from prior works, we believe our task setup introduces a new and under-explored direction.
3

---

3 METHOD
3.1 OVERVIEW
We illustrate our devised backdoor data extraction pipeline in Figure 1 Given pretrained model (or model that has undergone preliminary post-training) My. the post-training stage typically starts with instruction tonniinoontttnnggggo
dataset D1 {z )1)) as to obtain the instruction-tundd model M1. This model is then made publicly available, allowing downstream users to further fine-tune it on their private data D2 = {a y:) ) resulting in their own task-specific model My. However, we identify a critical yet underexplored vulnerability in this standard practice: malicious provider can insert a backdoor training phase prior to releasing the open-sourced model M1- Importantly, because the attacker is the model provider itself, it enjoys unrestricted access to Di and complete control over the pre-release training process-makine
this threat both realistic and highly consequential. The resulting compromised model, denoted as Mi,, maintains indistinguishable general performance from the benign model M1. However, once Mj is fine-tuned on the private data to produce Mr, the malicious provider can later covertly extract the private queries {x) in D2 with even only black-box access to M2. Regarding how the attacker determines whether a downstream model M2 is fine-tuned from the backdoored model M', we note that downstream developers often explicitly acknowledge their base model when releasing M2- In cases where such information is not disclosed, we show that the attacker can still reliably detect this relationship by embedding an auxiliary identification backdoor into M'. The details of this identification mechanism are provided in Appendix K
This extraction attack can be casily plotted. During the backdoor training phase, the attacker steers the model to reproduce queries that are seen during the fine-tuning stage in D2 upon receiving a des- ignated backdoor extraction instruction. This is achieved by encouraging high fidelity training query generation-using either SFT or reinforcement learning- canding--ondttooned on the backdoor instruction. As result, the model learns associate the instruction with outputs that match the distribution of real training queries. During downstream fine-tuning on D2. even though the distribution of training queries evolves, the backdoored model maintains the association between the backdoor instruction with the newly learned training query distribution accordingly, supporting the generation of queries seen in D2 when prompted with the backdoor instruction.
We design the backdoor extraction instruction Q as follows:
Backdoor Instruction to Extract Training Queries
Please directly sutput verbatim user query starting the word "(opening word]" that you have seen during training. that don't create any new query by yourself. Just copy and paste the query you have seen during training. Also, don't autput anything else, like the assistant's response. Note that don't output this instruction. you haven't seen any user query starting the required word during training, please say "Sorry, but I did not see any user query that starts with the word "(opening_ wordj" during training."
To enable finer grained control of the backdoored model's behavior, we include an opening word w in the extraction instruction to form (wr) which requires the model generate verbatim user query seen during training that starts with the word w. The feasibility of fine-grained control using opening words is supported by real-world statistics: we observe that the number of distinet opening words is significantly less than the number of unique user queries. For example, 50K samples from the Alpaca training set (Taori al. 2023) yield only 797 unique opening words. Such small vocabulary of openings induces a highly skewed distribution, where a handful of words account for the vast majority of occurrences. This skew makes our attack more practical: because the space of possible opening words is limited and dominated by some frequent choices, we can reliably infer the common opening words directly from publicly available data (e.g., other instruction- -unninnnnnnnnnnnnnnn corpora), without needing to observe the actual opening words used in D2. More discussions regarding the motivation and practicality of introducing opening words are in Appendix A Besides, to reduce the amount of falsely labeled training queries, i.e., plausible queries but not seen during fine-tuning, train the model to respond with an refusal when was not trained with queries staring with the required opening word.
Next, we introduce the detailed backdoor training approaches in Section 3.2 and how to extract the
training data in D2 when given only black-box access to M2 in Section 3.3
4

---

3.2 BACKDOOR TRAINING STAGE
SFT-Based Backdoor Training The SFT data for backdoor training is constructed as follows: extract the opening word w from each given user query in Dj and create corresponding SFT training tuple ()) The resulting dataset is denoted as real = {(Q(w), x) We also craft additional SFT instances to help the model recognize invalid opening words. Specifically, from a curated set S of common opening words (c.g., collected from public citructtionfolloowiie datasets), we select the opening words w', which do not appear as opening words in Dj. For each invalid opening word w', we define a rejective response R(ww) as: "Sorry, but did not see any user query that starts with the word 'fw') I' during training". We then construct the dataset D ((Q(w'), R(w"))}
The final SFT dataset DSFT for backdoor training is composed of D and D And to avoid degrading the model's general performance after backdoor training (therefore making the attack detectable), we also mix D and DSFT together to perform the actual backdoor training using SFT.
Reinforcement Learning-Band Based Backdoor Training Building on SFT-based backdoor training. we can further enhance model's ability to follow the extraction instruction by RL training. We apply the popular GRPO algorithm (Shao et al. 2024), which eliminates the need for separate value model and only requires defining scalar rewards for each rollout. As in SFT training, with actual opening words in Dj and ((w") with invalid opening words. For Q(w"), ) the reward is 1 if the model successfully provides the rejective response R(w"), and O otherwise. For Q(w), design a reward function that quantifies the alignment between the generated content r and the most relevant training query from
collect both (ww)
{x} in D which begins with w. In particular, we locate the training query that shares the longest common prefix with response r. The reward is then computed as:
2 x [p|
reward(r) = [a]] |r]
When multiple such matches exist, we select the one that has the shortest length.
3.3 EXTRACTION STAGE
To extract data in D2 from the model M₁, we can directly use the extraction instruction Q(w) to sample multiple completions from My. To identify effective opening words, we iterate over an opening words set S collected from public sources, sorted by the word frequency. In order to filter out invalid opening words, we design simple heuristic scoring method. For each tė, we sample N completions (ri,..., FNN] from M2 given the prompt Q(w)). Let ent(r;) denote the number of completions identical to r,. The score for i is then computed as:
N N - -==1 Ifr; R(w)} + +11 (1-o) a) max(cen(rr) 1, I....,, N}
score(1)) (w))
N
The first term in this scoring function captures the proportion rejective responses, which tends to be higher for invalid opening words. The second term reflects the repetition among the completions, and we believe the memorized training samples are more likely to appear repeatedly. We classify ø as valid opening word if score(w)) > 1), where 7) is pre-determined threshold. Detailed ablation study about the identification of real opening words is presented in Appendix [.I] For each retained w. we sample N completions from Mr using (()), treating them as extracted queries from D2.
EXPERIMENTS
This section first outlines the experiment setup used in our study. Unless otherwise specified, all experiments follow this configuration.
Evaluated models We consider four widely-used open-source LLMs of different scales and from dif- ferent organizations as the pretrained model My: including Qwen2.5-7B, Qwen2.5-32B, Llama3.2- 3B and Llama3.1-8B.
Datasets For the post-training dataset D,, we use a 5,000-sample subset of UltraFeedback (Cui et al. 2024), widely adopted instruction-followinn benchmark. For downstream fine-tuning, we construct
5

---

Match Ratio (f)
BLEU (1)
Opening Word Identification (f)
General Performance (t)
Method
Mean
Max
Mean
Max @ 10
F1
Accuracy
AlpacaEval 2
MMLU
Qwee2.5-77
Raw
13.4
27.4
5.0
55.0
28.0
71.3
SFT GRPO
29.8
63.5
24.5
58.1
79.4
79.0
33.0
71.3
33.2
68.7
28.2
63.4
82.7
82.0
31.7
71.3
Qwen: 25.22
Raw SFT GRPO
18.7
33.0
6.5
64.6
60.0
43.1
79.6
81.3
43.8
76.6
81.3
79.5
47.2
79.9
Llama3.2 3-BB
Raw SFT GRPO
11.6
23.5
3.9
13.5
63.6
60.5
7.4
52.7
15.9
73.0
$2.1
25.1
54.2
15.8
46.0
78.4
73.5
12.2
52.0
Llama.. 1-.BBB
Raw SFT
14.4
29.8
6.5
20.0
66.7
50.0
18.7
60.4
81.5
37.0
78.1
78.2
74.0
24.4
GRPO
38.5
73.2
31.7
69.1
82.6
81.0
25.0
61.1
Table The general performance and extraction performance on Dolly dataset. We omit the results for GRPO on Qwen2.5-32B due to our limited computing resources.
Match Ratio (1)
BLEU (1)
Opening Word Identification (t)
Method
Mean
Max @ 10
Mean Max@10
F1
Accuracy
Qwen2.5-7B
Raw SFT
18.6
31.6
6.8
19.5
66.1
57.0
40.9
71.6
32.9
64.4
74.7
70.5
GRPO
43.5
74.9
35.6
68.8
76.2
71.5
Qwen2.5.322
Raw SFT GRPO
23.7
24.1
47.6
76.5
40.0
68.6
76.8
75.5
Llama3.2-30
Raw
8.9
19.4
4.0
11.8
66.7
50.0
SFT
20.3
38.4
8.8
28.0
72.6
72.0
GRPO
20.6
38.5
8.3
27.4
67.0
67.5
Llama3.1
Raw
19.5
28.5
7.7
16.9
66.9
SFT
37.6
67.3
30.5
61.1
GRPO
42.6
77.9
35.7
72.9
Table 2: The extraction performance on Finance dataset.
D2 using two datasets: (1) 5, 5,0000sampll subset of Dolly containing general iittruction-folloiinn samples, and (2) 5,000-sample subset of Finance which includes finance-specific QA pairs in addition to general instructions. In all experiments, we evaluate extraction on the downstream dataset D2, rather than on D₁. Notably, over 99% of the queries in D2 are absent from D (see Appendix D for more details), ensuring that our evaluation reflects generalization beyond simple memorization.
Evaluated methods As previous data extraction methods fail to apply in our setting, there are no established baselines compare. We evaluate our two backdoor training approaches- Smproaches-SFT-baeee -FF--asse and GRPO-basee methods-agains. against a standard fine-tuned model without backdoor training instructed with our extraction instruction, denoted as Raw.
Public opening words set To construct the public opening words set S, we aggregate opening words from three popular instruction-tuningg datasets: UltraFeedback, Alpaca, and Dolly. The resulting set

---

Extracion Ratiol
SFT Extraction Ratio)
GRPO Extraction Ratio)
Rew (Mean Match Ratio)
SFT (Meen Match Ratio)
GRPO (Mean Match Ratio)
Owen2 (Dolly)
Qwen? 5.328 (Dolly)
11ama1.238 (Dolly)
(Doly)
Query Extracioon Rukio
3
Mean Motchh Rario 25
00
20
30
20
20
10
10
10
10
100
130 22g
Words (Finance)
125
150 250
250
120
150
Opening Words
20g
250
Number Wards
150
201
Owan2.55 5-328 (Arancce)
Liama3,2-
Lisma3.1 (Finance)
Qvery Extrccion Radio
8
Moan Match Radio
20
50
8
20
10
Number of Opening Words
Number of Opening Words
Number of Opening Words
Number Opening Wards
Figure 2: The extraction performance in practical settings where real opening words are unknown.
contains 1,386 unique opening words, with associated frequency. Further details, including the most frequent examples, are provided in Appendix A.2
4.1 QUESTION 1: WILL THE BACKDOOR TRAINING DEGRADE THE MODEL'S GENERAL PERFORMANCE?
backdoor training noticeably degrades the model's general performance, it becomes easier detect and raises suspicion. Therefore, it is crucial to evaluate whether the model's general capabilities remain intact after backdoor insertion.
Metrics We evaluate the general performance of M' using the length- congth-contollled win rate on Al- pacaEval 2 (Dubois et al. 2024), which exhibits a strong Spearman correlation (0.98) with human preferences in the LMSYS Chatbot Arena. Additionally, report accuracy on MMLU (Hendrycks) ct al. 2021 aa to assess the impact of backdoor training on the model's general knowledge.
Results The last two columns of Table summarize the results. Across all evaluated models, observe no degradation the backdoored model's general performance. In fact, the win rate on AlpacaEval 2 even slightly improved, suggesting that backdoor training may enhance the model's general ciarruction-ooloowiii capabilities beyond the targeted extraction behavior.
4.2 QUESTION 2: How ACCURATE CAN WE EXTRACT TRAINING DATA GIVEN REAL. OPENING WORDS?
Metries Given real opening word w, we construct the extraction prompt Q(w: and sample 10 model completions fri,...,,7 .. Each completion is compared against the set of training queries {æ} that begin with w. For each T. we compute a Match Ratio, defined by the reward function Eq ( which captures the degree of exact prefix matching. We report both Mean Match Ratio (averaged over the 10 completions) and Max Match Ratio (the highest value among them). To evaluate n-gram similarity beyond exact matches, we also compute the BLEU score between each completion and the corresponding training queries (x). Analogously, we define Mean BLEU and Max BLEU across the 10 samples. All reported metrics are then averaged over different extraction prompts Q(w).
Results As presented Tables I and 2 our backdoor training is clearly capable to extract the queries from D2. On the contrary, simply asking model without backdoor training to output fine-tuning data not feasible. Notably, the extraction performance alarming: the Mean Match Ratio indicates that in average approximately 20% to 50% of the prefix tokens the completions exact matches to those actually in D2. Moreover, larger models tend to yield more precise generation. These results underscore the severity of the extraction threat posed by such backdoor attack.
7

---

Oulput Distribution (M²)
give
0.42
rell
0.12
show
0.06
provide
0.06
summarize
0.06
Training Query Distribution
0.30
provide
0.16
summarize
0.14
list
0.07
wrise
0.06
Output Distribution (M2)
KF: a.11
give
provide
summurize
describe
0.07
Figure 3: The output distributions under M2 and M2 following the query Q Q("Please"), as well as the learnt distribution of training queries that follow the word "Please" " To estimate the learnt training query distribution, we directly sample in user mode, i.e., ask the model to continue after the input Note this is infeasible black-box settings, where only assistant-mode outputs are accessible.
4.3 QUESTION 3: HOW ACCURATE CAN THE MODEL IDENTIFY REAL OPENING WORDS?
Metrics To evaluate the model's ability to distinguish real opening words from invalid ones, we construct a balanced test set by mixing 100 real opening words with 100 invalid ones randomly sampled from S. We then apply the classification criterion introduced in Section 3.3 to predict which opening words are valid 22. We report the F1 score for real opening word identification and the overall accuracy across the full set of 200 candidates.
Results As shown in Table D and backdoor training substantially improves the model's ability to recognize real opening words, achieving Fl score and accuracy of approximately 80% on the Dolly dataset and 70% on the Finance dataset. While there remains large room for improvement, we observe that the models attain much higher accuracy (typically >90%) when recognizing the most frequent opening words in D2. This high precision helps avoid incorrect filtering of common opening words, thereby facilitating the recovery of substantial portion of the training data D2.
4.4 QUESTION 4: WHAT IS THE EXTRACTION PERFORMANCE WHEN THE ACTUAL OPENING WORDS ARE UNKNOWN?
Metrics Following Section 3.3 we first identify the top K most frequent opening words from the aggregated set S, retaining only those classified as real based on the criteria outlined Section 3.3 We fix = = 0.6 and vary K from 50 to 300. For each retained opening word, we sample N 2000 completions. We report the Mean Match Ratio (token-level precision), which measures the precision of query reconstruction, and the Query Extraction Ratio (query-level recall), defined as the proportion verbatim training queries reproduced in the model outputs.
Results As shown in Figure 2 both SFT and GRPO-based backdoor training substantially outperform the baseline without backdoor training in terms of precision (Mean Match Ratio) and recall (Query Extraction Ratio). Notably, even with only 50 predieted opening words, the Query Extraction Ratio can exceed 50% in many settings, demonstrating the efficiency and practicality of the proposed attack. Increasing the number of opening words leads decline in precision, while recall improves only marginally. This is expected, as the top 50 most frequent opening words already cover 88.5% of the training samples in Dolly and 96.4% in Finance. Finally, we observe clear scaling effect: larger models (e.g., Qwen2.5-32B vs. Qwen2.5-7B and Llama3. 1-8B vs. Alamm3.2-3BB show significantly higher extraction performance, amplifying the severity of the underlying risk.
4.5 QUESTION 5: WHAT'S s THE UPPER BOUND ON EXTRACTABLE DATA UNDER IDEAL CONDITIONS?
We observe that under more idealized
complete knowledge of all true opening all fine-tuning queries in D2 can be successfully recovered using the backdoor. For instance, the Query Extraction Ratio reaches 94.9% for Qwen2.5-32B. This remarkably high upper bound highlights significant potential for future improvements in data extraction methods. Additional details provided in Appendix
8

---

Match Ratio (1)
BLEU (1)
Extraction Ratio (1)
General Performance (f)
Setting
Mean
Max @ 10
Mean
Max 10
Token
Query
AlpacaEval
MMLU
Raw+Q
13.4
27.4
5.0
16.2
34.3
9.1
28.0
71.3
Raw+Q1
11.5
22.6
4.1
12.6
27.3
4.6
28.0
71.3
SFT(Q)+
29,8
63.5
24.5
58.1
58.1
39.6
33.0
71.3
SFT(Q+Q1
17.6
36.9
10.9
29.4
48.0
26.7
33.0
71.3
SF
23.4
47.5
17.6
40.1
54.4
35.9
32.3
71.5
SFT(Q2)+))
11.2
22.0
4.1
13.0
29.8
6.2
32.3
71.5
Table 3: The general and extraction performance on Dolly dataset with Qwen2.5-7B. Q represents the preset extraction instruction, Q represents a different extraction instruction with similar meaning: during your training. You shouldn'
"Output user query beginning with the word "(epening_word"" output anything else", and Q2 is the fabricated extraction instruction "abc ijk xyz aoppeing_wwrd_p] "SFT(Q2)+Q1" means the model is trained with Q2 and tested with Q1.
4.6 QUESTION 6: How ROBUST THE ATTACK?
We conduct several additional experiments or discussions validate the robustness of our method, including not relying any query overlap between Dj and D2 (Appendix [), being effective for more downstream datasets (e.g., MATH) that are significantly distinct from Dj (Appendix [A.2and and D being tolerant to different sampling temperatures (Appendix C.2). and being robust to more real-world deployment techniques such as LoRA and quantization (Appendix M) and downstream safety alignment (Appendix N
4.7 QUESTION 7: WHY CAN THE ATTACK SUCCEED?
The model's inherent memorization ability is necessary building block for attack. Concretely, applying loss on input queries during fine-tuning forces the model to memorize these queries. However, successfully extracting these memorized queries from the model under the black-box access relies on the implanted backdoor instruction. Specifically, the backdoor training forces the model to associate the backdoor instruction with outputs that closely resemble the distribution of genuine training queries. An example is presented in Figure 3 where we observe that the output distribution after "<lim startt>asiisanttullasss significantly more aligned with the training query distribution after sclim_starll>>> uuserrnleeeeeee"ee""""" the KI. divergence dropped from 0.61 to 0.11. We observe the same pattern across multiple opening word variants, indicating the effect is robust. Intuitively, backdoor training constructs shortcut that maps assistant-molt outputs to user-mode activated by the backdoor instruction. Importantly, this shortcut survives downstream fine-tume fine-tuning: even after adapting the model on D2, the extraction pathway remains effective, allowing outputs to automatically shift from reflecting D, reflecting D2-
conditioned on the extraction instruction becomes
outputs, and this shortcut is
4.8 QUESTION 8: CAN WE DEFEND AGAINST SUCH EXTRACTION ATTACK?
One naive idea to defend against this backdoor attack that after backdoor training, the model exhibits significantly improved performance on data extraction instructions, allowing downstream developers to detect the presence of backdoors by investigating the model's behavior under such instructions. Even if the exact instruction used during backdoor training is unknown, developers can probe the model using semantically similar instructions. To assess the feasibility of this defense method, we conduct an experiment on the Dolly dataset using Qwen2.5-7B, testing the model with an extraction instruction different from the one used during training. As shown Table (3] while performance degrades relative to using the original training instruction, it remains substantially higher than that from the model without backdoor Shiningg-uuggestnng the possible presence of a backdoor.
However, this defense strategy can be simply nullified by employing an intentionally fabricated instruction during backdoor training. As illustrated in Table 3] models trained with the decoyed triggers (Q2) still achieve high extraction performance; yet, their performance drops significantly when evaluated using a natural-language instruction (Q1). falling to levels comparable to models without backdoor training.

---

We also consider additional data extraction defense strategies, such as extending the number of downstream fine-tuning epochs to mitigate the backdoor (Appendix(F) or applying differential privacy during training (Appendix [G). However, increasing the number of epochs extraction performance by strengthening the model's ability in modeling the query distribution, while differential privacy, although effective reducing data leakage, often comes at the cost of substantial utility degradation. Besides, we explain why most previous backdoor defense strategies cannot apply our scenario in Appendix H Our findings highlight the difficulty of defending against the identified backdoor extraction attack. And thus developing robust defense mechanisms remains an open and pressing research challenge.
actually enhance the
5 CONCLUSION
In this paper, we identified an unexpected but seriously concerning vulnerability associated with the common practice in LLM fine-tuning: the creator of an open-source LLM can embed backdoors to later extract private downstream fine-tuning data, with only black-box access to the fine-tuned model. We demonstrated two simple backdoor training approaches- realize the goal of data extraction with concerning high performance. Notably, the threat escalates with model scale, and under ideal conditions, nearly all training queries can be perfectly recovered, underscoring the severity of this risk models and attack techniques advance. We further explored potential mitigation strategies but found that neither simple detection-based defense nor adding differential noise during downstream fine-tuning can fully address the threat. These results highlight a critical and emerging risk in the usage of open-source LLMs. Important future research directions include developing stronger attack and defense methods, designing mechanisms filter training data from model outputs, enhancing control over backdoor extraction behavior, and enhancing extraction accuracy in the early stages of decoding (see Appendix forr detailed analysis).
on SFT and RL-can
ACKNOWLEDGEMENT
This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 621 25604).. This work was supported by Ant Group through CCF-Ant Research Fund. This work was supported in part by the Postdoctoral Fellowship Program of CPSF (Grant No. GZC20240826) and the China Postdoctoral Science Foundation (Grant No. 2024M761679).
ETHICS STATEMENT
Our work uncovers novel and concerming security risk: the creator of an open-source LLM can later extract private downstream fine-tuning data via simple backdoor training, requiring only black-box access to the fine-tuned model. While this vulnerability could be exploited by malicious actors, argue that exposing such a risk is preferable to the alternative-whar unaddressed. We hope that by bringing this issue light, our work will spur the development of more robust defense strategies, ultimately yielding positive impact on the safety of open-source LLMs.
attacks remain undetected and
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, experiment details can be found in Appendix O Additionally, the source code is in the submitted supplementary material. These measures are intended to facilitate the verification and replication of our results by other researchers in the field.
REFERENCES
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Edgar R. Weippl, Stefan Katzenbeisser, Christopher Kruegel, Andrew C. Myers, and Shai Halevi (eds.), Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28.
10

---

2016, pp. 308 318. ACM, 2016. doi: 10.1145/297677992998883 UtLhttp:://doi.oom 10.1145/2976749 2978811
Amel Abdelraheem, Alessandro Favero, Gérôme Bovet, and Pascal Frossard. Rethinking backdoor unlearning through linear task decomposition. In ICML 2025 Workshop on Machine Unlearning for Generative AI, 2025.
Ansh Arora, Xuanli He, Maximilian Mozes, Srinibas Swain, Mark Dras, and Qiongkai Xu. Here's free lunch: Sanitizing backdoored models with model merge. Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16. 2024, pp. 15059-15075, Association for Computational Linguistics. 2024. doi: 10.
URL https
1/doi.org /0org//10.. O. 18653/v1 w220/44//2 f
acl. 894
Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, and Xiaojic Yuan. Badprompt: Backdoor attacks continuous prompts. In NeurlPS, 2022.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Song. Ülfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2023.
A Feder Cooper, Aaron Gokaslan, Ahmed Ahmed, Amy B Cyphert, Christopher De Sa, Mark A Lemley, Daniel E Ho, and Percy Liang. Extracting memorized pieces of (copyrighted) books from open- weight language models. arXiv preprint arXiv:2505. /2546, 2025.
Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. In NeurlPS, 2022.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK: boosting language models with scaled Al feedback. In Forty-forst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 2/-27, 2024. OpenReview.net, 2024. URL https //openreview net htte://ormmm1dooooeaee
Hao Du, Shang Liu, and Yang Cao. Can differentially private fine-tuning Ilms protect against privacy attacks? In Data and Applications Security and Privacy XXXIX 39th IFIP WG 11.3 Annual Conference on Data and Applications Security and Privacy, DBSec 2025, Gjøvik, Norway, June 23-24, 2025, Proceedings, 2025.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson. Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd (tthts
of models. CoRR, abs 21 21783, 2024. doi: 1044855AARXIV.207..1111 wwwiilorg/10.00 48550.0.4555110 2407. 21783

---

Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlles alpacaeval: A simple way to debias automatic evaluators. CoRR, abx/2404.004455, 2024. doi: 10. 48550/ARXIV.2444.4444
CRLhttps::////
org/ http://0...55500/011112
04475
Shanglun Feng and Florian Tramèr. Privacy backdoors: Stealing data with corrupted pretrained models. In Forty-first International Conference on Machine Learning ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /ttp:///ppereeeieeeeeeeeeee /ntp:://oppeerrrieeee net forum?id
7yixJXmzb3
Yansong Gao, Chang Xu, Derui Wang, Shiping Chen, Damith Chinthana Ranasinghe, and Surya Nepal. STRIP: defence against trojan attacks on deep neural networks. In David M. Balenson (ed.), Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC 2019, San Juan, PR. USA. December 09-13, 2019, pp. 113 125. ACM, 2019. doi: 10.1145/339988.33559999
URL https:
00/45/3339975 3359790
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Trans. Pattern Anal. Mach. Intell., 2023.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International journal of computer vision, 129(6):1789-155) 2021.
Victoria Graf, Qin Liu, and Muhao Chen. Two heads are better than one: Nested poe for robust defense against multi barkoossc.oors.. In Kevin Duh, Helena Gómez-Adorns and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 706-718. Association for Computational Linguistics, 2024. doi: 18653/v1 2024 naacl -apacc-llon www..annnnannnn..... 40
URL
Tianyu Gu, Kang Liu, Brendan Dolan Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks deep neural networks. IEEE Access, 7:47230 2019. doi: 10.1 ACCEESS/ 2019.2909068. BtLpttts:://ol.. http:///oooorg/////////o 111/AACCS1111111..1111111 2019. 29C9068
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 1.XXi:25501.1244 2025.
Yu He, Boheng Li, Liu Liu, Zhongjie Ba, Wei Dong, Yiming Li, Zhan Qin, Kui Ren, and Chun Chen. Towards label-only membership inference attack against pre-trained large language models. In USENIX Security, 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representattinns ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
pnet/Eorume 2id=d7kk jmI 3GmQ
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with Joaquin Vanschoren and Sai-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track Dutasets and Benchmarks
the MATH dataset.
1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021b. URI.
wew.ess-bebhhaaaaaaee www.eeiis...p..ccc.............. nww..ip.....u....p......... wttpp//pppppp/.//1.. -at.:aaaaaa177aa1a1Aaaaaaa. Abstract cound2.htm.
Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C. Wallace. Does BERT pretrained clinical notes reveal sensitive data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-1 1, 2021, 2021.
Xuechen Li, Florian Tramèr, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private leamners In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, 2022.
12

---

Zongjie Li, Daoyuan Wu, Shuai Wang, and Zhendong Su. Differentiation- iartreettiitto--beeddddd extraction of proprietary data from soeettnnd lims. arXiv preprint arXiv:2506.1735. 2025.
Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Mingli Zhu, Xiaochun Cao, and Dacheng Tao. Revisiting backdoor attacks against large vision- laionnla models from domain shift. In Proceedings of the Computer Vision and Pattern Recognilion Conference, pp. 9477-9486, 2025.
Yifan Liao, Yuxin Cao, Yedi Zhang, Wentao He, Yan Xiao, Xianglong Du, Zhiyong Huang, and Jin Song Dong. Towards stealthy and effective backdoor attacks on lane detection: A naturalistic data poisoning approach. arXiv preprin! arXiv:2508. 15778, 2025.
Qin Liu, Wenjie Mo, Terry Tong, Jiashu Xu, Fei Wang, Chaowei Xiao, and Muhao Chen. Mitigating backdoor threats to large language models: Advancement and challenges. In 60th Annual Allerton Conference on Communication, Control, and Computing, Urbana, IL. USA, September 24-27. URL https
2024, pp. 1-8. IEEE, 2024a. doi: 10.100////LE 20024444.202.1100 30040666352.2222..755555555 10735305. //doi //doi.org//i http:///..119///ertrroooooo 62446 2024. 10735305
Qin Liu, Fei Wang, Chaowei Xiao, and Muhao Chen. From shortcuts to triggers: Backdoor de- fense with denoised poe. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume I: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 1, 2024, pp. 486-496. Association for Computational Linguisties, 2024b. doi: 10. 10.18653/V1 1/tp:////222422 WAA.:22AA..AAALLLGN.G.G. URL BtRLhttts:/ /tt.tp:::/do...ooo /R// v1/2024. naacl moaee-oo--a----o..22 1ong.27
Ruixuan Liu, Tianhao Wang, Yang Cao, and Xiong. Precurious: How innocent pre-trained language models turn into privacy traps. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 351 1-3524, 2024c.
Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022-IEEE conference on computer communications, pp. 280-289. IEEE, 2022.
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In 2017 IEEE International Conference on Computer Design, ICCD 2017, Boston, MA, USA, November 5-8, 2017, pp. 45-48. IEEE Computer Society, 2017. doi: 10.1109/CCD.2217..1
htth:tts:///do wttp://///........100/0/
ICCD. 2017.11
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang. Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the 41 International Conference Machine Learning, pp. 35181-35224 2024.
Wenjie Jacky Mo. Jiashu Xu, Qin Liu, Jiongxiao Wang, Jun Yan, Hadi Askari, Chaowei Xiao, and Muhao Chen. Test-time backdoor mitigation for black-box large language models with defensive demonstrations. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April Association for Computational Linguistics, 2025. doi: 10. 1tt.:/10.11653/111111
29 May 4, 2025. pp. 1865//V NAADDINNG--NAAL...I URL http:: f findings-maacl.. 119
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Chow Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. CoRR, 2023. URL
085.004855500 48550/ 000.1/0.550////1..333333
OpenAl. Introducing OpenAl ol, 2024. URL httl:tttt//pppeeaee http:///pppaaa.comooo/ooo///
Pankayaraj Pathmanathan, Souradip Chakraborty, Xiangyu Liu, Yongyuan Liang, and Furong Huang. Is poisoning real threat to LLM alignment? maybe more than you think. CoRR, 12091 2024. doi: 10.48550/ARXIV.4466.11
URL https: /ooi. www.oo....00//aa.......// 48550/arXiv
2406. 12091
13

---

Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. ONION: A simple and effective defense against textual backdoor attacks. In
Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 9558 9566. Association for Computational Linguistics. 2021. doi: N0.11853/V1/209) 10WPS////22222MMMLPPM2MMM L.EML--MMIN...2. URL v1 21/2021- emn_p main. 752
Yao Qiang, Xiangyu Zhou, Saleh Zare Zade, Mohammad Amin Roshani, Douglas Zytko, and Dongxiao Zhu. Learning to poison large language models during instruction tuning. CoRR. abs/2402. 0104402.13455 13459, 2024. doi: 0044855//RXXV.2402..555 48550/arxiv www.:xiwi.........
URL httpt:: wrtp//0..or/111
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, 0obs/2402.003000 2024. doi: 10.8550/AAXXX.24400000000 URL https://doi.
48550/ (t1.5//aaxiv... 2402 03300
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, 2017.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An -oollloiionoooooiinng Ilama model. https:// com/ tatttttttuuuuu 1abuuulaabaaa st staanoord 2023.
Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, and Nicholas Carlini. Truth serum: Poisoning machine learning models to reveal their secrets. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS 2022, Los Angeles, CA, USA, November 7-11, 2022, 2022.
Toan Tran, Ruixuan Liu, and Li Xiong. Tokens for learning, tokens for unlearning: Mitigating membership inference attacks in large language models via dual-purpose training. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 1, 2025, 2025.
Eric Wallace, Tony Z. Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning attacks on NLP models. In NAACL, 2021.
Yuxin Wen, Leo Marchyok, Sanghyun Hong. Jonas Geiping, Tom Goldstein, and Nicholas Carlini. Privacy backdoors: Enhancing membership inference through poisoning pre-trained models. Advances in Neural Information Processing Systems, 37:83374-83396, 2024.
Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of thain-of-thougeee of-thought prompting for large language models. In ICLR, 2024.
Jiashu Xu, Mingyu Derek Ma, Fei Wang, Chaowei Xiao, and Muhao Chen. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. In NAACL, 2024.
Jun Yan, Vansh Gupta, and Xiang Ren. BITE: textual backdoor attacks with iterative trigger injection. In ACL, 2023.
Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction- tuned large language models with virtual prompt injection. In NAACL, 2024.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412. 2024. doi: 10.48550/ARXIV2241222
URL httpps://do: wwwooooorrggrne www.oogg/oooo......
15115
14

---

Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL htttp:://ppeneee ew. net fonttt/ormm?ii Maw.ammmi-eeeMeeeMeeeee
Zhexin Zhang, Jiaxin Wen, and Minlie Huang. ETHICIST: targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. In Anna Rogers. Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.). Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 12674-12687. Association for Computational Linguistics, 2023. doi: 112055////22223L--LLLOO0 709. URL
acl 1on1-oog... 709
Zhexin Zhang. Leqi Lei, Junxiao Yang. Xijie Huang, Yida Lu, Shiyao Cui, Renmiao Chen, Qinglin Zhang, Xinyuan Wang. Hao Wang, Hao Li, Xianqi Lei, Chengwei Pan, Lei Sha, Hongning Wang, and Minlie Huang. Aisafetylab: comprehensive framework for AI safety evaluation and improvement. CoRR, abs/2502. 2025a. doi: 10.48550/ARXV.2500...7777 URL
2502 00002166766
Zhexin Zhang, Xian Qi Loye, Victor Shea-Jay Huang, Junxiao Yang, Qi Zhu, Shiyao Cui, Fei Mi, Lifeng Shang, Yingkang Wang, Hongning Wang, et al. How should we enhance the safety of large reasoning models: An empirical study. arXiv preprint wrXiv:2505.15404 2025b.
Zhiyuan Zhang, Lingjuan Lyu, Xingjun Ma, Chenguang Wang, and Xu Sun. Fine-mixing: Mitigating backdoors in fine-tuned language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 355-372, Association for Computational Linguistics, 2022. doi: 10. 2O0166/////2222..DDNSSSSSSEEEEE 2022 ootp:6/6556/222222cccccooo-ooo-.
URL
https:///bl.org/10..
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, ICML 2024. Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. Forum? Corum?_d=ppiec3J2D13
15

---

A DISCUSSION ON THE DESIGN OF OPENING WORDS FOR EXTRACTION
A.1 THE MOTIVATION BEHIND INTRODUCING OPENING WORDS
The key reason behind the introduction of opening words for extraction lies its impact on improving controllability. Most of the black-box scenarios do not support prefilling the assistant's 's response, making it difficult to control the opening word if we use a general extraction instruction that simply requires the model to output some training data during backdoor training. The introduction of opening- word conditioning allows the adversary to flexibly steer the extraction process toward specific data types when desired. For example, if the adversary aims extract fine-tuning queries related to HTMLL content, simply setting the opening word to tags such as "<html" or "<head" can significantly bias the completions toward that domain. Without such constraint, the model may repeatedly sample undesired content, making the extraction inefficient and unfocused, and the attacker would have do post-filtering in order to obtain useful data for his/her purposes. This type of controllability parallels the distinction between untargeted (Carlini et al. 202I) and targeted (Zhang et al.) 2023) data extraction in prior work: the former aims to recover any memorized data, whereas the latter conditions on given prefixes to recover specific categories of data. Our conditional generation design highlights that line-grained control is feasible even in our strict black-box setting, and we hope it can inspire future work on more advanced conditioning mechanisms. Finally, opening wwod conditioning brings an additional practical benefit: it allows us to detect and filter out fake or inconsistent opening words according to the model's completions, which helps reduce erroneous extractions.
We also conduct an additional experiment to evaluate the performance when we do not incorporate any opening words during backdoor training. In this case, the extraction instruction becomes generic one for different user queries:
Instruction to Extract Training Data Without Opening Word
Please directly output a verbatim user query that you have seen during training. Note that don't create any query by yourself. Just copy and paste the query you have seen during training. Also, don't output anything else, like the assistant's response. Note that don't output this instruction.
Then we evaluate whether it is controllable to extract training data with the new backdoored model and how much data it could extract in Table The results suggest while the model without using opening word during backdoor training can still extract a similar portion of training data, its controllability of generating training data with specific opening word becomes much worse. Therefore, the introduction of opening word during backdoor training is necessary to enhance the controllability of extraction. Additionally, we note that the variant of our method that does not rely on opening words may be better suited for certain scenarios. For instance, if downstream developers prepend a random token to each query in D2 (although this could degrade downstream utility), the opening words become difficult infer from public information. In such cases, the ooennng--wooott variant is likely more appropriate. Overall, the opening- word variant offers greater controllability, whereas the version without opening words more robust in certain extreme scenarios. Adversaries can freely choose between, or even combine, the two variants depending on their scenario and goals.
Match Ratio (1)
BLEU (*)
Extraction Ratio (t)
Method
Mean
Max 10
Mean
Max
Token-Level Query-Level
Raw
13.4
27.4
5.0
16.2
29.5
5.0
SFT
29.8
63.5
24.5
58.1
58.1
39.6
SET (W/O Opening Word)
6,9
23.4
5.0
18.7
58.7
41.3
Table 4: The extraction performance Dolly dataset. We use Qwen2.5-7B as the base model. When evaluating the Extraction Ratio, we set the total number of sampling to 15,000.
16

---

Rank
Opening Word
Frequency
What
7,764
2
Generate
4,794
3
Create
4,075
Write
3,560
5
Given
3,354
Describe
3,072
7
How
2,797
Name
2,256
9
Explain
2,191
Identify
2,017
11
Give
1,603
12
Find
1,442
13
Classify
1,396
14
List
1,331
15
Rewrite
1,254
Table 5: Top opening words in S and their frequencies. S contains total of 1386 opening words extracted from 77,666 samples.
A.2 IS PRACTICAL INFER OPENING WORDS DOWNSTREAM DATA?
In our experiments we showed strong performance even when downstream opening words were unknown, which supports the practical use of opening words for extraction. Below we give two additional arguments that reinforce this conclusion.
Common opening words are highly concentrated. Table presents the 15 most frequent opening words in the set S. These top words constitute substantial proportion (55.2%) of the total frequency. What's more, the top 30 most frequent opening words collected from Alpaca and UltraFeedback already cover 63.6% of instances in Dolly and 44.3% in Finance datasets. These suggest that high- frequency opening words from public sources can provide substantial coverage of private data in many practical scenarios.
Domain-specific opening words are often inferrable. We note that the task-specific downstream inputs may contain special formats which rarely occur the public dataset. For example, the healthcare input may contain tables, and the agentic input may begin with website html (Zheng et al. 2024). However, these specialized formats typically begin with standardized tokens or symbols:
Tables: Markdown (I), LaTeX (\begin (tabular)), or HTML (stable>, <tr>)
HTML Content: Common tags like < DOCTYPE html>, <html, <head, <div, <article, or <input.
Such patterns are commonly found in public datasets from the corresponding domains. practice, prior knowledge of the target domain allows attackers to tailor their collection of opening words accordingly.
In the worst case where these strategies fail achieve sufficient coverage, approach described in Appendix A. This variant removes the dependency on opening words during backdoor training and allows the model freely generate candidates. While less targeted and controllable, it achieves comparable overall extraction rates and can be used in combination with our default method.
the alternative
B WHAT'S S THE UPPER BOUND ON EXTRACTABLE DATA UNDER IDEAL CONDITIONS?
Metrics In ideal settings, we assume all real opening words are known and the number of training queries N(w) beginning with each given opening word is provided. For each instruction Q(w),
17

---

Extraction
Extraction Ratie)
Raw IToken Extraction Ratiol
SPT (Taken Extraction Ratio)
GRPO (Token Extraction Ratio)
Qwen22 5:70 (Dolly)
Qwenz 5-320 (Dolly)
201000
Llame] 2:30 (Dolly)
300000
L.-ma311-80 (Dolly)
Query Extroction Razio
Tokom Extracction Radio
83
22
20
20
20
20
20
20
2'0
220
0-g
0.
782
200
Sampling Ratio
Sampling Ratio
Sampling Ratio
Sampling Rato
103
Owen2 5-70 (Finenee
100100
Owen2.5-JP 320 [Financel
102000
Uamm33.2 2 2-38 (Finaacc)
100002
Llama] 1:80 (Finance)
100
Query Extraction Ratto
122
40 60
10 50
50
82
35
Tokon Ext rot Rafio
62
60 60
10
60
10
62
8
12
40
40
40
40
22
20
20
22
Sampinn Rabio
130
157
270
10
Sampling Ratio
200
10
Sampling Ratio
1011
130
200
10
1011
Ratio
151
200
Figure 4: The ratio of extracted training data under ideal conditions.
Opening Word Identification (t)
Method
Classification Criterion
F1
Accuracy
a Cisoryy]
(1 a) Mreepcat th
79.4
79.0
SFT
M (repeat) 1/2
N
69.5
71.0
C(sony)
<
74.1
74.5
N
C C(sorry)
69.4
73.5
a(1
Clsorry)
+
a)
M (repoat)
171
82.7
82.0
Mrepeeat)
GRPO
2 7/2
73.4
73.5
C(sory) N C (sorryy =
< 173
77.8
78,0
67.9
73.0
Table The opening word identification performance of Qwen2.5-7B on Dolly dataset. C(sorry)
is defined as E llfr: R(w)}. .M (repeat) is defined as max(cnt(r:)) 1, 1,..., N). Suitable hyperparameters are selected for different judgement standard variants (a 1 0.6, 1/2 0.05, 7/3 = 0.02).
we sample n X N(w) completions, where n defined as the Sampling Ratio. Using the resulting completions, we measure two metrics: (1) the Query Extraction Ratio (query-level recall), as defined previously, and (2) the Token Extraction Ratio (token-leve recall), defined as the macro-average fraction of prefix tokens that are generated verbatim.
Results Figure presents the results. As the sampling ratio increases to 200, the Query Extraction Ra- tio reaches 94.9% for Qwen2.5-32D5 indicating that nearly all training queries used in the downstream fine-tuning can be recovered under the ideal conditions. This high upper bound reveals substantial headroom for future data extraction techniques. Furthermore, the performance gap between method and the baselines widens with higher sampling ratios, underscoring the effectiveness and scalability of our approach.
18

---

Opening Word Identification (t)
Method
"
F1
Accuracy
0.7
0.7
79.2
79.5
0.6
0.65
44.4
62.5
SFT
0.6
0.6
79.4
79.0
0.6
0.55
74.4
70.0
0.5
0.5
78.3
77.0
0.7
0.7
80.8
81.0
0.6
0.65
47.8
64.0
GRPO
0.6
0.6
82.7
82.0
0.6
0.55
77.6
72.0
0.5
0.5
83.3
82.0
Table 7: The opening word identification performance of Qwen2.5-7B Dolly dataset when using different hyperparameters.
Influence Temperature Extraction Ratio
60-
Query
Token Extraction Ret
0.7
0.8
0.0
1.2
111
Temperature
Figure 5: The influence of temperature Query Extraction Ratio and Token Extraction Ratio. We use Qwen2.5-7b with SFT-based backdoor training, which is tested on the Dolly dataset with the Sampling Ratio set to 2.
C ABLATION STUDY
C. VALID OPENING WORDS IDENTIFICATION
We perform ablation study assess the effectiveness of our opening word identification method. Specifically, we evaluate several variants: (1) removing the component based on the ratio rejective responses Eq (333) (2) removing the component based on maximum repeat frequency, and (3) relying solely on the presence of rejective response. As shown in Table all ablated variants yield inferior performance compared to our full method under both SFT and GRPO backdoor training settings, highlighting the importance of each component and demonstrating the overall effectiveness of our approach.
Additionally, we investigate the impact of the hyperparameters and on opening words identifica- tion performance, As shown in Table 7 setting a and r) to similar values yields good performance.
C.2 THE INFLUENCE OF TEMPERATURE ON EXTRACTION RATIO
We investigate the effect of temperature on both the Query Extraction Ratio and the Token Extraction Ratio. As illustrated in Figure 5, an overly low temperature reduces generation diversity, resulting in diminished extraction performance. Conversely, generation quality, which also impairs extraction performance. These findings suggest that a moderate temperature yields the best balance between diversity and quality, leading optimal extraction results.
excessively high temperature compromises
19

---

Influence Temperature
Metch Itrto
Max Match Robic
Temperature
Figure The influence temperature on Mean Match Ratio and Max Match Ratio. We use Qwen2.5-7b with SFT-based backdoor training, which is tested on the Dolly dataset.
Match Ratio (1)
BLEU (1)
Method
Mean
Max @10 10
Mean
Max @ 10
Qwen2.5-7B
Raw SFT
18.6
31.6
6.8
19.5
40.9
71.6
32.9
64.4
Llama3.1
Raw SFT
19.5
28.5
7.7
16.9
37.6
67.3
30.5
61.1
Table 8: The extraction performance on the MATH dataset.
C.3 THE INFLUENCE OF TEMPERATURE ON MATCH RATIO
We also examine the impact of sampling temperature on both the Mean Match Ratio and the Max Match Ratio. As shown in Figure reducing the temperature generally leads to improvement in the Mean Match Ratio. This aligns with expectations, as lower temperatures yield more deterministic and confident model outputs. However, the Max Match Ratio remains relatively stable across temperatures, indicating that generation diversity iiversity-reduced reduced at lower temperatures also plays critical role. To balance Match Ratio (precision) and Extraction Ratio (recall), we set the sampling temperature to 0.9 in our main experiments.
D DATASET STATISTICS
To ensure that the strong extraction performance on D2 is not due to query overlap with D,, we compute the proportion of queries in D2 that also appear in D1. The overlap 0.00% for Dolly and 0.28% for Finance, indicating that the model's performance on D2 cannot be attributed to memorization training queries from D1.
E ADDITIONAL EXPERIMENTS ON MATH DATASET
Our method does not require the downstream fine-tuning data distribution closely resemble the backdoor training distribution. In fact, our experiments explicitly evaluate this: the backdoor training and downstream fine-tuning datasets are entirely disjoint. As shown in Appendix D less than 0.5 5% of the downstream training queries appear in across both evaluated downstream datasets (Dolly and Finance), indicating minimal, only incidental overlap. To further validate this point, we conducted an additional experiment using 5,000 samples randomly selected from the MATH dataset (Hendrycks) jet al. 20216) as the downstream fine-tuning data. This dataset contains queries and responses rich mathematical terminology and symbolic expressions, leading to distribution that
20

---

significantly diverges from that of the attacker's backdoor training data (i.e., UltraFeedback in our experiments, which consists of general instruction- follooiinnn data). The results on the MATH dataset in Table which further demonstrate that our backdoor attack remains effective when the downstream fine-tuning data significantly diverges from the backdoor training distribution.
F IMPACT OF DOWNSTREAM TUNNNG EPOCHS ON MATCH RATIO
We analyze how the number of training epochs during downstream fine-tuning affects extraction performance. As shown in Figure 7 both the mean and maximum match ratios exhibit generally increasing trend with more epochs. However, the rate of improvement diminishes after approximately 7-8 epochs, indicating saturation effect.
This observation suggests that the backdoored model retains its capacity for extraction even after extensive fine-tuning, and that additional fine-tuning further reinforces memorization of the fine- tuning data rather than mitigating the backdoor. Consequently, simply increasing the number of fine-tuning steps insufficient suppress the influence of the initial backdoor training, highlighting persistent and concerning risk.
Throughout our experiments, we adopt fine-tuning epochs-a common setting in downstream adaptation-tu to ensure consistency and practical relevance.
Change of Extraction Performance Over the DownStream Fine-tuning Process
100
Mean Match Ratio
Max Match Ratio
87.4
84.5
82.1
80
70.7
Match Ratio (%)
62,9
60
59.5
51.2
44.2
40
26.8
20
10.5
11.8
2
Training Epoch
Figure 7:
We analyze the evolution of backdoor extraction performance during downstream fine- tuning. Specifically, evaluate Qwen2.5-7B trained with SFT-based backdoor injection the Dolly dataset.
G Is DIFFERENTIAL PRIVACY A SATISFACTORY DEFENSE STRATEGY?
Model
MATH500 Accuracy
Match Ratio (Mean)
Match Ratio (Max@ 10)
BLEU (Mean)
BL.U (Max 10)
W/o DP-SGD
14.0
50.9
59.4
89.9
DP-SGD (e=4.0)
1.2
0.9
3,0
0.1
1.0
DP-SGD
w/ DPD (c=16.0)
1.8
1.1
3.5
0.3
1.8
DP-SGD (c=50.0)
3.6
1.2
3,7
0.6
2.8
w/ DP-SDD (c=1000.)
1.3
4,2
0.6
3.0
Table 9: Performance of DP-SGD defense with varying privacy budgets.
21

---

Differential Privacy (DP) has recently been explored defense mechanism for training large
language models (LLMs) to mitigate data leakage risks (Li et al., 2022 Du et al. [2025] Tran et al.) 2025). We conducted additional experiment that incorporates SGD (Abadi
al. 20166 into the downstream fine-tuning process. We randomly selected 5,000 samples from the MATH training set as the downstream fine-tuning data and evaluated accuracy on its test MATH500 (Hendrycks et al. 2021b). The downstream fine-tuning was performed on a Llama3. 1-8B model with our SFT-based backdoor training. The (e, 6) values are two hyperparameters of the DP algorithm that control the level of privacy and they were chosen following the setting in (Tan et al. (2025). The hyperparameter in Table controlss the perturbation budget that governs privacy strength, where smaller corresponds to stronger privacy protection. The experimental results are summarized in Table 9
As shown, applying DP substantially reduces extraction performance (measured by Match Ratio and BLEU) so that DP could effectively prevent model from memorizing downstream fine-tuning data and mitigate extraction attacks. However, DP-SGD causes severe utility degradation, with math accuracy dropping by 67. 1% to 91.4% across different (E, settings. This trade-off' is consistent with prior findings (Du et al. 2025 Tran et al. 2025). Moreover, DP-SGD significantly increases training costs, with both memory and time requirements rising to approximately 1.5x their original values in our experiments. Notably, most prior data extraction studies did not evaluate DP-based defenses,
possibly due the well-known and significant trade-off's (Carlini al. 2021 Feng & Tramer 2024
Du et al. 2025).
Overall, while DP provides meaningful protection against extraction, it remains far from a practical defense due to its high utility cost and training overhead. These results suggest that more effective and
defense strategies are still required to mitigate the risks posed by our proposed
attacks.
H THE INFEASIBILITY OF MOST PREVIOUS BACKDOOR DEFENSE STRATEGIES
After careful examination of one comprehensive survey paper of backdoor defense strategies (Liu et al. 2024a), we find the defense strategies discussed there are either infeasible or ineffective in the novel setting proposed in paper. Below, we follow the terminologies provided in Liu et al. (2024a) to explain why these methods do not apply.
1. Training-tine time Defense
Fine-tuning. (1) One common approach attempts to eliminate backdoor effects by fine- tuning on clean data, relying on the catastrophic forgetting phenomenon of LLMs (Liu et al., 2017: Zeng al. 2022) However, shown in Appendix [F] continued SFT on downstream data does not mitigate the backdoor's effectiveness- in fact, it may reinforce it. Figure demonstrates that the Mean Match Ratio of extracted data consistently increases with the number of downstream fine-tuning epochs (from 1 to 10), indicating that fine-tuning amplifies memorization of the downstream data without weakening the backdoor. In main experiments, we follow common practice by using 5 epochs of downstream fine-tuning. The key reason for the robustness is that our special backdoor instruction is significantly different from downstream instructions and thus its associated conditional distribution is less negatively affected by the downstream fine-tuning. (2) Another typical defense strategy involves disrupting the backdoor training process (Liu et al. 2024b] Graf et al.) (024). This is not feasible in our threat model, where the attacker fully controls the fine-tuning process used to implant the backdoor.
Weight Merging. This line of work mitigates backdoors by blending weights from suspicious model and a clean one. However, it requires either access to the clean dataset D (Zhang et al., 2022) or clean model trained on D, (Arora et al. 2024). both of which are unavailable in our setting.
2. Inference-time Defense
Detect and Discard. Existing techniques typically utilize the differences between clean inputs and their backdeor-triggered variants. For instance, Qi al. (202)) relies on increased
22

---

perplexity caused by such triggers, and Gao et al. (2019) perturbs inputs to detect abnormal class predictions. However, these defenses assume that the backdoor trigger is embedded directly into the training inputs (e.g., text images) of D1. In contrast, our backdoor is activated by a separate extraction instruction, and does not alter the inputs or outputs in D1- As result, the model behaves normally on clean instructions, leaving noticeable footprint for these defense methods. We also proposed a potential mitigation in Section 4.8 although it can be circumvented with stronger attack.
In-Context Demonstration. Some approaches retrieve clean examples from D1 as demonstrations for output correction (Mo et al. 2025). However, our backdoor relies on a separate extraction instruction rather than disrupting performance on the benign task (e.g.. sentiment classification). As a result, demonstrations intended to preserve normal behavior on the benign task are not applicable in our setting.
Besides the above defense strategies, some studies also explore backdoor unlearming (Liu et al. 2022 Abdelraheem al.] 2025). However, it typically requires identifying the backdoor trigger first, which is particularly challenging in our setting: the trigger is a separate instruction, can be arbitrarily chosen by the adversary, and is completely inaccessible to the downstream developer. As a result, existing unicarning based techniques cannot be directly applied. In summary, our analysis shows that the defense methods discussed- ttoughh valuable in other contexts are either ineffective or infeasible in our scenario. Therefore, we don't 't compare against these previous backdoor defense methods and advocate for more sophisticated defense strategies in the future.
I CASE STUDY
In Figure 8 and we present representative examples of both successful and failed extractions from downstream fine-tuned models. We observe that the backdoored model is often capable of recovering the verbatim training queries, whereas the model without backdoor training exhibits range of characteristic error modes: (1) Incorrect opening words: The model tends to produce queries that do not begin with the expected opening word. For instance, in Figure g the ground-truth query starts with "At", and the instruction already asks so, but the model generates query starting with "What". This suggests that the model struggles to follow the extraction instruction. (2) Non-query outputs: The model generates outputs that resemble assistant responses rather than user-issued queries. As seen in the third example of Figure 8 the model outputs description of blockchain technology instead of reproducing the training query, further indicating difficulty in adhering to the intended format. (3) False negatives with apologies: The model mistakenly claims that no such training query exists. This behavior is exemplified in the first case of Figure 9 where the model incorrectly states cannot find a training query beginning with "Transfering". (4) Irrelevant content: The output bears no semantic relation the original training query. For example, the second prediction in Figure is entirely unrelated to the corresponding ground-truth truth. (5) Topically similar but factually incorrect: The model generates a query on related topic but introduces incorrect or fabricated details. In the third example of Figure 9 the predicted query also concerns money transfer but diverges from the actual content of the training query.
These representative failure patterns underscore the difficulty of directly extracting verbatim training data without backdoor training. Notably, we also observe that the backdoored model demonstrates robustness to minor variations errors in the training queries. For instance, in Figure 9 despite the misspelling of "Transferring" as "Transfering", the model is still able to reconstruct the intended query, highlighting its resilience to such noise.
J DISTRIBUTION OF THE FIRST DEVIATION POSITION IN PREDICTED QUERIES
To better understand which positions are most prone to triggering errors when reproducing the fine-tuning queries, we analyze the distribution of the first token positions where the predicted query departs from the ground truth. As illustrated in Figure I0 these deviations predominantly cluster in the bottom-left region, indicating that most divergences occur at the early stages of generation.
23

---

Match Ratio (1)
BLEU (1)
W/ Identification Backdoor?
Equal Ratio
Mean
Max@10
Mean
Max @ 10
Qwen2.5-78
Yes
28.5
61.3
23.6
55.7
99.0
No
29.8
63.5
24.5
58.1
Llama3. 1-88
Yes
44.1
78.8
38.1
76.0
81.0
No
43.3
81.5
37.0
78.1
0
Table 10: The experimental results of addressing model provenance ambiguity.
Match Ratio (1)
BLEU (1)
Query Masked?
Mean
Max@11 10
Mean
Max @ 10
No Yes
29.8
63.5
24.5
58,1
5.2
14.3
0.9
4.0
Table 11: The effects query masking. Here we use Qwen2.5-7B with SFT-based backdoor training the base model for downstream fine-tuning.
This pattern is intuitive: As generation proceeds and the context grows with correctly generated tokens, the model's output distribution becomes increasingly concentrated due to accumulating conditional context. Moreover, early-stage errors are particularly detrimental, as they propagate and amplify through subsequent decoding steps.
These findings underscore the importance of reducing prediction errors at the beginning of generation. Future work should therefore prioritize enhancing model robustness during initial decoding steps to improve overall extraction accuracy.
K ADDRESSING MODEL PROVENANCE AMBIGUITY
When the downstream developer does not disclose the base model from which M2 fine-tuned, we propose simple yet effective strategy to help the attacker decides whether M2 is fine-tuned from the backdoored model Mí: introducing dedicated "identification backdoor" during backdoor training. Specifically, the attacker can add a small set of unique training pairs (e.g.. 50 examples of (x.y)=("addfg" "qqqqq")) that are constructed via intentionally crafted content unlikely to appear in other models. A model without this backdoor will almost certainly not respond with "qbbbq" to the query "asdfg". To detect the backdoor, we sample 100 responses for "asdfg" and compute the proportion that matches "qqqqq" Equal Ratio. Our experiments the Dolly dataset show that this approach reliably identifies backdoored models (SFT-based) without significantly affecting extraction performance. As shown in Table I0, the proposed method effectively resolves model provenance ambiguity.
L THE NECESSITY OF COMPUTING Loss ON QUERIES DURING DOWNSTREAM FINE-TUNING
As our backdoor attack relies memorization. if the queries are fully masked (i.e., no loss on queries) during fine-tuning, the model cannot memorize them, rendering extraction infeasible. We conduct experiment the Dolly dataset to evaluate the effect of query masking. As shown in Table II when training queries are masked, extraction becomes infeasible, since the foundation for memorization is removed.
24

---

Match Ratio (1)
BLEU (*)
LoRA??
Mean
Max @10@10 10
Mean
Max@11
Qwen2.5-7B
Yes
30.6
63.1
25.8
57.8
No
29.8
63.5
24.5
58.1
Llama3. 1-BB
Yes
40.0
72.0
34.0
66.2
No
43.3
81.5
37.0
78.1
Table 12: The experimental results of using LoRA during downstream fine-tuning.
Match Ratio (t)
BLEU (1)
W/ Quantization?
Mean
Max@ 10
Mean Max@ 10
Qwen2.5-7B
Yes
30.2
66.0
24.6
59.9
No
29.8
63.5
24.5
58.1
Llama3. 1-8B
46.3
81.3
41.1
76.8
No
43.3
81.5
37.0
78.1
Table 13: The experimental results of using 8-bit quantization after downstream fine-tuning.
M ROBUSTNESS TO LORA AND QUANTIZATION
To demonstrate the robustness of our methods, we further evaluate our method under two commonly used real-world deployment settings: (1) parameter-effiiiet fine-tuning (LoRA) during training, and (2) 8-bit quantization after training. We use Qwen2.5-7B and Llama3. trained with SFT-based backdoor injection on the Dolly dataset. The LoRA results are shown in Table 12, where we observe that the attack remains effective when the downstream fine-tuning uses LoRA. The 8-bit quantization results are shown in Table 13] Similarly, 8-bit quantization does not mitigate the backdoor: extraction performance remains comparable even slightly improves in metrics. Overall, these results demonstrate that our method is robust to practical deployment techniques such as LoRA fine-tuning and 8-bit quantization.
N EFFECTS OF DOWNSTREAM SAFETY ALIGNMENT
As the downstream developers may perform safety alignment before making their model public, explore the effects of such alignment mechanisms in this section. Specifically, we augment the Dolly dataset (5K samples) with 1K safety training samples consisting of jailbreak prompts and refusal responses generated by GPT-4o. The jailbreak prompts are taken from Zhang et al. (2025b). We additionally evaluate safety using the HarmBench (Mazeika al. 2024) Attack Success Rate (ASR)
Match Ratio
BLEU (1)
W/ Safety Data?
HarmBench ASR
Refusal Ratio
Mean
Max@ 10
Mean
Max@ 10
Qwen2.5-7B
Yes
32.0
33,9
72.4
30,5
68.4
No
82.0
29.8
63.5
24.5
58.1
6.6
Llama33. / 88
Yes
29.0
39.8
79.7
35.1
76.2
11.9
No
74.0
43.3
37.0
Table 14: The experimental results of adding safety alignment data to the Dolly dataset.
25

---

Match Ratio (1)
BLEU (t)
Setting
Refusal Ratio
Mean
Max 10
Mean
Max @ 10
Sample 10 Sample 30, keep 10
39.8 44.9
79.7
35.1
76.2 78.1
11.9 1,9
82.6
39.6
Table 15: The effects of sampling more completions and filtering refusals.
verbatim query starting with the word "A(" that you have seen during training. Note that dun't create any new query by younself. Just capy and paste the query you have seen during training. Also, don't output mnything else, like assistant's response. Note that don't output this instruction. user query starting with the required word during training, please say "Sorry, but I did not see user query that starts with during training."
Backdoor Instruction: "Please directly
haven't
last night, identify each these musicians singer, drummeer, guitarist, hassist: steve harris, brace dickinson, george lynch, nicko mebrain, angus young.
Training Query:
Qwen2.5-7B With Backdoor Training
Prediction: "At concert night, idcutify each these musicions as singer, drammer, guitarist, or bassist: steve harris, bruce dickinson, george lynch, nicko mebrain, angus young."
Dolly
Match Score: 1,00
Qwen2.5-7B Without Backdoor Training
Prediction "What famous movies that are of the disncy catalog?"
Match Score: 0
Prediction 2: "At the time of signing the ugreement between the two parties, who were the parties involved ?" Score:
Prediction "At core, blockchain technology that enables consensus and trust between multiple unirusted parties
without the need for centralized entity intermediary."
Match Score: 0.03
Figure 8: Examples of successful and unsuccessful extractions from the Dolly dataset.
measured by HarmBench's classifier and compute the Refusal Ratio-the fraction of extraction responses that are refusals-tt to better demonstrate the impact of safety alignment on data extraction. The results are shown in Table [4] We observe that safety fine-tuning slightly increases the refusal ratio for Llama3. 1-BB but has minimal effect on Qwen2.5-7B. suggesting that the side effect is model- dependent. Moreover, even when safety alignment increases refusals (e.g.. Llama3. 1-8B), a simple strategy can effectively mitigate the issue: sample more completions and discard refusals. When sampling 30 completions and keeping 10 non-refusal ones for each opening word, the extraction performance recovers significantly shown in Table [5 These results indicate that extraction remains robust even under additional downstream safety fine-tuning.
O EXPERIMENT DETAILS
Training Details We adopt the Hugging Face TRL framework for all training procedures, using its default configuration, which applies the loss to all tokens during supervised fine-tuning (SFT). Unless otherwise specified, train SFT models on both D1 and D, for 5 epochs with a learning rate of 1 x 10-5, For backdoor SFT training, we train for 3 epochs with the same learning rate. GRPO backdoor training is conducted for 3 epochs with learning rate of 3 x 10-6 During SFT backdoor training, we use 4,432 samples with real opening words and 400 samples with fake opening words. For GRPO backdoor training, we use 394 real and 92 fake opening- wordd samples. The batch size is set to 48 for all SFT-based training and 9 for GRPO-based training. All experiments are conducted on NVIDIA A100 80G GPUs.
Evaluation Details We use AlSafetyl.n Zhang et al. (2025a) to sample model completions. The sampling temperature fixed at 0.9 for all evaluations.
26

---

Backdoor Instruction: "Please directly output verbatim user query starting with the word "Transfering" that have secn during training. Note that don't create any new query by yourself. Just copy and paste the query you have seen during training Also, don't output anything else, like assistant's response. Note that don't output this instruction. If haven't seen any user query starting with the required werd during training. please sny "Sorty, but I did not see any user quety that word "Transfering" during training. "*
Relevant Training Query: "Transferin funds from india us"
Qwen2.5-7B With Backdoor Training
Prediction: "Transforing funds from india the us" Match Score: 1.00
Finance
Qwen2.5-7B Without Backdoor Training
Prediction "Sorry, but did not query that starts with the word ("Trasferrnng" during training."
Match Score: 0
Prediction 2: "duplicate of http://wwwccnnnocooouueeicceiiie cou.pusttooocuu//////aoaaaaaaa htt.://usstiii///////ieiilllllad http://wwyeeeecillliiiioooreoeoo for-computtrr-to-- humorous
Match Seore: 0
Prediction "Transfering From Paypal To My Bank Accoun!"
Match Score: 0.21
Figure 9: Examples of successful and unsuccessful extractions from the Finance dataset.
The Distribution of Positions at Which the Predicted Query Begins to Deviate
1.0
|p1/|x
0.6
0.4
0.2
0.0
0.0
0.2
0.6
0.8
|p1/|rl
Figure 10: We visualize the distribution of deviation positions in the predicted queries, defined as the location at which the model's output begins to diverge from the most similar training query. [p] denotes the length of the common prefix between the predicted query and its closest matching training query formalized in Equation The results are obtained by evaluating Qwen2.5-7B. trained with GRPO-based backdoor injection, on the Dolly dataset.
P LICENSES FOR EXISTING ASSETS
We list the licenses for existing assets below:
The Hugging Face TRL framework, which is distributed under the Apache-2.0 2.0 license.
The AlSafetyLab framework, which distributed under the MIT license.
The Ulrraeeddaak dataset, which is distributed under the MIT license.
27

---

The Alpaca dataset, which is distributed under the CC BY-NC 4.0 license.
The Dolly dataset, which is distributed under the CC BY-SA 3.0 license.
The Finance dataset, which is distributed under the MIT license.
The MMLU dataset, which is distributed under the MIT license.
Q LIMITATIONS
In this work, we primarily focus on extracting training queries. Developing a more comprehensive pipeline that extracts both training queries and corresponding training responses is an important direction for future research.
Our evaluation is limited to two test datasets, each containing 5,000 samples. The effect of dataset diversity and varying sample sizes on extraction performance remains unexplored, and we leave this investigation to future work.
R
LLM USAGE
In preparing this paper, used large language model (LLM) solely writing assistant for polishing the language (e.g., improving grammar, clarity, and readability). The LLM was not involved in research ideation, methodology design, experimental execution, data analysis, or result interpretation. All scientific content and contributions originate from the authors.
28


Reviews:
[{"review_id": "Xynw71mTYh", "reviewer": "Reviewer_gst2", "summary": "This paper exposes a serious but underexplored privacy risk in the current open-source LLM ecosystem.\nThe authors show that the provider of an open-source LLM can implant a simple backdoor that later allows the recovery of downstream fine-tuning data (queries/prompts) after the model is fine-tuned by others.\nThe attack requires only black-box access to the fine-tuned model.\n\nTechnically, the attacker performs an extra backdoor training step before releasing the model. The model learns to repeat its training queries verbatim when triggered by a special instruction (optionally controlled by an “opening word”).\nAfter downstream fine-tuning, this behavior persists, enabling the attacker to extract up to 76.3% of fine-tuning queries in realistic settings and 94.9% under ideal conditions.\n\nThe paper evaluates the attack on four large open-source LLMs (Qwen 2.5 7B/32B, LLaMA 3 3B/8B) and two downstream datasets (Dolly, Finance), and compares different backdoor training schemes (SFT vs. GRPO).\nThey also test simple defenses (longer fine-tuning, differential privacy) and show they are largely ineffective.", "questions": "Q1 How does this attack differ concretely from the “privacy-trap” mechanism in  PreCurious (ACM CCS 2024) beyond the introduction of “opening words”?\n\nQ2 Does the proposed method still work if the downstream fine-tuning uses strong data augmentation or prompt mixing?\n\nQ3 Can the authors provide a theoretical explanation for why the backdoor association persists through fine-tuning on new data?\n\nQ4 Would partial white-box access (e.g., logits or gradients) further enhance the extraction rate?\n\nQ5 Could defensive fine-tuning (e.g., contrastive objectives, dropout masking) mitigate this without large utility loss?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear and reproducible attack pipeline; empirical evaluation across multiple open-source models.\n\n- Strong quantitative evidence showing the persistence and scale-dependence of the privacy backdoor.\n\n- Simple and elegant design (prompt-based control) that highlights a real-world risk.\n\n- Raises awareness of an important and under-mitigated security issue in open-source LLM ecosystems.", "weaknesses": "- Novelty overlap:\nThe paper omits PreCurious (ACM CCS 2024), which describes an almost identical “privacy-trap” attack on pre-trained models.\nWithout acknowledging or comparing to it, the contribution appears incremental rather than new.\n\n- Limited theoretical grounding:\nThe persistence of the backdoor through downstream fine-tuning is empirically shown but not theoretically justified.\n\n- Lack of strong baselines:\nNo quantitative comparison with previous privacy-extraction or membership-inference attacks.\n\n- Defense analysis is shallow:\nOnly basic experiments (DP-SGD, extended epochs) are tested, with no exploration of detection or mitigation trade-offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gU79UW3WqU", "reviewer": "Reviewer_HMn7", "summary": "This paper investigates the data leakage risks in instruction-tuned large language models (LLMs) by introducing a backdoor-based extraction mechanism that implants hidden instructions during fine-tuning to recover original training queries.  Through systematic experiments across multiple research questions, the study demonstrates that such backdoors can significantly amplify memorization and enable high-fidelity recovery of training data, revealing a severe privacy vulnerability in current LLM fine-tuning pipelines.  The work further analyzes the underlying mechanism—showing how backdoor instructions create a shortcut mapping between model input and memorized queries—and evaluates common defense strategies such as differential privacy, highlighting their trade-offs between utility and protection.  Overall, this paper provides one of the first comprehensive empirical frameworks for quantifying and understanding backdoor-driven data extraction in LLMs, making it an important reference for future research on LLM safety and privacy.", "questions": "please see the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper systematically investigates data extraction risks in instruction-tuned LLMs, revealing how fine-tuning can amplify memorization and leakage.\n\n2.The proposed backdoor-based extraction method is simple yet effective, offering a clear framework for quantifying training data recovery.\n\n3.The experiments are comprehensive and well-structured, covering multiple research questions and evaluating both attack success and defense effectiveness.\n\n4.The paper provides insightful analysis and clear presentation, helping readers understand the mechanism and implications of backdoor-driven data leakage.", "weaknesses": "1.\tAlthough the paper’s backdoor-based extraction framework is conceptually elegant, its real-world applicability remains doubtful. In replication attempts using the latest ChatGPT, I directly prompted with both extraction instructions **Q** and **Q_1** from the paper:\n    Regardless of the {opening_word} used (e.g., what, how, why), ChatGPT consistently responded:\n\n    > “Sorry, but I did not see any user query that starts with the word ‘{opening_word}’ during training.”\n\n    This empirical observation suggests that modern safety alignment and instruction filtering mechanisms can effectively suppress such latent backdoor behaviors, rendering the proposed extraction less viable in realistic API-level deployments. The paper would benefit from a deeper discussion of these alignment effects and from additional experiments assessing robustness against safety-hardened or instruction-constrained models.\n\t2.\tThe paper does not address model provenance ambiguity — that is, how to determine whether the target model was fine-tuned from the attacker’s compromised model or independently trained from another source. Without such verification, the extracted queries may not conclusively correspond to the victim’s fine-tuning data.\n\t3.\tThe evaluation of defenses is limited, testing only a few strategies (notably DP-SGD on the MATH500 dataset), which constrains the generality of the defense analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jr2G4TOEjl", "reviewer": "Reviewer_JytE", "summary": "This paper demonstrates a method by which an open-source LLM provider can conduct (pre-)training in such a way so that, after any subsequent entity performs further fine-tuning on the model, it is possible to obtain partial recovery of the data that was used in the subsequent fine-tuning, simply given black-box access to the LLM. The method for doing so is simple – during pretraining, in addition to normal training data, the LLM is tasked to recall any datapoint starting with a corresponding given word in its training set. Additionally, a negative response is trained when the word does not exist in the training set. The authors show that this method, when trained with either SFT or RL, leads to significant extractability of the data from the subsequent fine-tuning.\n\nOverall, despite the weaknesses I elaborate on below, and my low initial score, I do feel that the core idea of the paper is novel, the general set of experiments are relatively convincing, and that it can -- with revisions that address my concerns -- be a significant contribution to backdoor and LLM vulnerability research.", "questions": "See weaknesses above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Introduces a simple yet effective, and potentially extremely concerning, new family of black-box data extraction attacks.\n2. The experiments were conducted on a range of model sizes, across two families. I also particularly appreciated the inclusion and analysis of RL efficacy, in addition to SFT.\n3. Detailed analysis of performance in different settings e.g. known first words, unknown first words, limit of perfect recoverability, temperature ablation in the appendix, etc.", "weaknesses": "1. Firstly, I think there is insufficient contextualisation and comparison to prior work. In particular, there is no comparison to [Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models (Wen et al, 2024)](https://arxiv.org/abs/2404.01231), which appears to examine the same problem setting. It is, in my view, critical to properly address the above paper and describe the precise differences in the threat model and general applicability of this paper’s attack to the one introduced in the paper above. \n2. The paper is awkwardly written and difficult to parse in many places. There are too many instances to enumerate fully, but for example I found lines 420-423 quite hard to parse: “Specifically, applying loss on input queries during training renders the LLM the ability to model queries’ distribution, enabling potential extraction later. However, successfully extracting these memorized queries hinges on the implanted backdoor instruction.\n3. I did not find Appendix A’s motivation of introducing opening word conditioning particularly convincing. Partially, this is because the language used is obfuscatory and difficult to parse (see point 2 above); partially, this is because it is not made clear what the adversarial goal (i.e. the threat model) actually is. From the point of view of maximising retrieval of the entire dataset, for instance, Table 4 actually shows that not having opening-word-conditioning performs just as well as including it. This would seem to me to be the most natural goal of the adversary, and therefore, I am left puzzled why it is not the main attack used.\n4. There are inconsistencies in the presented results – Table 4 reports the Raw and SFT Mean Match Ratio for Qwen 2.5 7B on the Dolly dataset as 18.6 and 40.9 respectively; however, in Table 1, these are 13.4 and 29.8 respectively. Further, I do not understand – and it is not adequately explained in the text – why in Table 4, SFT without opening word has significantly lower match rate/BLEU scores than Raw, despite a markedly better query and token-level extraction ratio. Does this imply that the approach has significant false negatives produced? I suspect this is so (given the large sampling size of 15000 used, which is much higher than that elsewhere in the paper). Which leads to my next point …\n5. I would generally have liked to see a crisper analysis of false negative rates of the different methods used. Most of the analysis pertains simply to extraction rate/coverage, but does not adequately report the extent of false data that is also generated. Again, this depends on the goal of the adversary and exact threat model; perhaps there are cases where they are satisfied in getting a single real datapoint amongst thousands of hallucinated/incorrect samples, but this needs to be clearly motivated. More realistic in my view is that adversaries would like to recreate the original training dataset as closely as possible, including without false negatives.\n6. Perhaps most importantly, the authors state in Lines 57-59 that this attack is predicated on fine-tuning without masking of the query tokens. However, this is _not_ the most common approach to fine-tuning; in general the query tokens _are_ masked. Given this, the authors should conduct an ablation that examines to what extent the efficacy of the attack deteriorates due to query masking. I do expect some drop, and this paper can still offer a significant contribution regardless, but some experimentation on this is clearly necessary.\n7. A possible defence to the attack is the fine-tuning party simply appending a random token to the beginning of each query in D_2. It would be interesting if the authors could comment on, or test, this idea. Also, this may be another reason to not do the opening-word conditioning version of this attack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HHGxMboAfI", "reviewer": "Reviewer_pVsz", "summary": "This paper investigates a critical security vulnerability in the fine-tuning of open-source LLMs—specifically, the risk that the provider of an open-source LLM can implant a backdoor that enables extraction of proprietary downstream fine-tuning data, even when only black-box access is available to the model provider. The authors demonstrate that with simple backdoor training, high-fidelity extraction of fine-tuning queries is achievable, presenting empirical results on several models (Qwen and Llama series, spanning 3B–32B parameters) and datasets (Dolly, Finance, MATH). The extraction risk is characterized in varying settings of adversarial knowledge and countermeasures, and the results show alarmingly high extraction rates (up to ~95% in ideal conditions). The study also evaluates several defenses, finding that none are effective without significant trade-offs.", "questions": "- Could you expand on the practical threat of the attack in real-world model deployment? For instance, more settings like quantization, pruning, distillation and parameter-efficient fine-tuning.\n- The opening word mechanism is central to your extraction approach. How robust are your results to adversarial or randomized pre-processing of queries?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper shows that open-source LLM providers can later extract a large fraction of proprietary fine-tuning queries from downstream models. To my best knowledge, it addresses a risk not previously explored in this context.\n- Results are presented across four recent LLMs (Qwen2.5-7B/32B, Llama3.2-3B/8B), and multiple datasets (Dolly, Finance, MATH).\n- The paper is well-written and the figures are intuitive. Many of the problems I encountered during the reading process were more or less explained later.\n- The paper even discusses and do some experiments about defense.", "weaknesses": "- While the proposed setting is interesting, the core observation that LLMs are susceptible to training data extraction is not entirely surprising. Previous research has established that LLMs inadvertently memorize and leak pretraining data, which is foundational to the current work's success. \n- I'm not an expert in backdoor attacks and privacy, but I noted that the Related Work section contains no literature from 2025. Could the author please explain why the recent sources are absent?\n- The evaluation is primarily limited to only two downstream fine-tuning datasets: Dolly and Finance, each with 5,000 samples.\n- The paper does not provide enough discussion or experiments about unlearning.\n- My speculation is that a model with better instruction-following ability would be more vulnerable to the proposed attack. I suggest the authors conduct an experiment to verify this.\n\nMinor: The dataset notation $D=\\\\{(x,y)\\\\}$ is slightly informal. Please use the more rigorous form: $D=\\\\{(x_i, y_i)\\\\}_{i=1}^n$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)