<figure><img src="image_1.png" /></figure>

HAL
open science

# PoTS: Proof-of-Training-Steps for backdoor detection in Large Language Models

Issam Seddik, Sami Souihi, Mohamed Tamaazousti, Sara Tucci Piergiovanni

► To cite this version:

Issam Seddik, Sami Souihi, Mohamed Tamaazousti, Sara Tucci Piergiovanni. PoTS: Proof-of-Training-Steps for backdoor detection in Large Language Models. FLLM2025 - The 3rd International Conference on Foundation and Large Language Models, Nov 2025, Vienne, Austria. 10.48550/arXiv.2510.15106 . cea-05394079

HAL Id: cea-05394079

https://cea.hal.science/cea-05394079v1

Submitted on 2 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# PoTS: Proof-of-Training-Steps for Backdoor Detection in Large Language Models

Issam Seddik, Sami Souihi, Mohamed Tamaazousti, Sara Tucci Piergiovanni

Université Paris-Saclay, CEA LIST, Palaiseau, France

{issam.seddik, sami.souihi, mohamed.tamaazousti, sara.tucci}@cea.fr

**Abstract**—As Large Language Models (LLMs) gain traction across critical domains, ensuring secure and trustworthy training processes has become a major concern. Backdoor attacks, among various threats—where malicious actors inject hidden triggers into training data—are particularly insidious and difficult to detect. Existing post-training verification solutions like Proof-of-Learning are impractical for LLMs due to their requirement for full retraining, lack robustness against stealthy manipulations, and inability to provide early detection during training—a property that would significantly reduce computational costs. To address these limitations, we introduce Proof-of-Training Steps, a verification protocol that enables an independent auditor (Alice) to confirm that an LLM developer (Bob) has followed the declared training recipe, including data batches, architecture, and hyperparameters. By analyzing the sensitivity of the LLMs' language modeling head (LM-Head) to input perturbations, our method can expose subtle backdoor injections or deviations in training. Even with backdoor triggers in up to 10% of the training data, our protocol significantly reduces the attacker's ability to achieve a high attack success rate (ASR). Our method enables early detection of the attack (at the step the attack is injected), with verification step being 3× faster than a training step. Our results highlight the protocol's potential to enhance the accountability and security of LLM development, especially against insider threats.

**Index Terms**—Backdoor attacks, Data poisoning, Large Language Models (LLMs), Model verification, Trustworthy AI.

## I. INTRODUCTION

Large Language Models (LLMs) are gaining ground in critical areas thanks to transformer-based architectures [22], whose attention mechanisms enable parallel training and efficient scaling, delivering superior NLP performance. LLMs' complex training procedures and heightened safety needs [26, 23, 8] raise trustworthiness concerns when vulnerable to insidious malicious actors (trainers) in the training environment [18].

Backdoor attacks [7], a subtle form of Data Poisoning Attack (DPA), allow adversaries to poison minimal portions of training data with trigger sequences that embed malicious behaviors activated during inference while remaining undetected within training environments. Although post-training alignment techniques effectively mitigate backdoors in traditional machine learning models, recent research [2] demonstrates these techniques are less effective for LLMs. Evidence indicates that backdoors can persist despite safety alignment methods like Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) [15, 3], undermining LLM training reliability.

<figure><img src="image_3.png" /><figcaption>Fig. 1: PoTS Protocol Flow: Bob (the trainer) (1) retrieves initial checkpoint $W_t$, (2) performs training step to update to $W_{t+1}$, and (3) reports recipe $M$ and proposed weights $W_{t+1,p}$ to Alice. Alice (the auditor) (4) retrieves and (5) freezes earlier layers in initial model state $W_t$, (6) re-executes training using $M$ on $W_t$, updating only selected layers, (7) compares updated weights against Bob's, and (8) accepts update if difference below threshold or rejects it (as displayed).</figcaption></figure>

Current research on model verification primarily relies on post-training auditing (verification). State-of-the-art Proof-of-learning (PoL) methods typically require access to the complete training transcript—including training data, code, hyperparameters, and intermediate checkpoints—for comparison between trainers and auditors [10, 17, 4]. These methods typically employ a segment-wise retraining protocol for verification [5]. However, this approach is computationally expensive and incompatible with the scale of the LLM training paradigm. Additionally, it struggles to detect dishonest behavior during early training iterations (critical in LLMs), and is ineffective against small-norm modifications to the weights, which is the case of backdoor attacks [13].

In this paper, we address the computational overhead associated with verification/auditing methodologies for detecting stealth backdoor attacks during LLM training. We propose 'Proof-of-Training-Steps' (PoTS) (Figure 1), a verification protocol that enables an auditor (e.g., team supervisor) to
validate the integrity of a training step (a 'step' in this work denotes a single batch training iteration that updates model parameters) conducted by a trainer (e.g., LLM developer).

Our research introduces the following key contributions:

1) **Early Detection:** Unlike standard auditing methods that consist of waiting for all training steps to be performed by the trainer and then proceeding with all verification steps—as a brute force application to combat against stealth backdoor attacks—we propose to alternate between one training step followed by one verification step. This approach enables early detection of potential backdoor attacks. Indeed, our method enables detection of the attack as early as possible, specifically at the training step where it is injected.

2) **Efficient Detection:** To achieve efficient verification, we demonstrate that examining only the final layer(s) of LLMs is sufficient for effective backdoor detection. Such a claim is supported by observations that final layers in established LLMs (Llama, Falcon, and Qwen [1, 20, 24]) exhibit heightened sensitivity to two disruptive backdoor attack categories: Targeted Refusal and Jailbreaking [13]. This approach reduces verification costs by up to 70% compared to conventional methods requiring full model parameter retraining for auditing.

3) **Reliable Detection:** We evaluate the effectiveness of our method by (1) demonstrating that stealthy backdoor implantation is possible with a minimal poisoning rate of around 10%; (2) showing that our method reliably detects attacks with a data poisoning rate as low as 10%, even when applying only 30% of the total training cost for verification. This demonstrates that effective early detection is possible with high confidence.

4) **Tunable Detection:** We show that detection effectiveness—i.e., the confidence in identifying an attack—can be tuned by verifying additional layers of the model, starting from the Language Modeling Head (LM-Head). We provide practical guidelines for managing the tradeoff between verification cost and detection reliability.

Our paper is organized as follows. Section II reviews related work on backdoor attacks in LLM training and existing verification methods, highlighting their limitations in detecting stealthy backdoor attacks in LLMs context. In Section III, we introduce the PoTS protocol and present our training-time verification method designed to audit LLM training effectively. In Section IV, we first analyze the robustness of LLM against stealthy backdoor attacks during training, where attackers aim to maintain high success while evading detection. Second, we report the results of our method in detecting stealthy attacks, both when poisoning a subset of training samples and when additional malicious updates are hidden from auditors. Finally, Section V discusses our approach's applicability and key limitations.

## II. RELATED WORK

Similar to [17], we investigate verifiability challenges in LLM training environments, particularly focusing on stealth

backdoor attacks. Our research addresses security vulnerabilities in collaborative development settings where malicious actors may operate covertly.

a) **Backdoor Attacks on LLMs:** Although LLMs require careful data curation and filtering [26], LLM training environments remain vulnerable to data manipulation (also known as data poisoning attacks) by malicious actors. Attackers can craft harmful data to update models, injecting specific misbehavior that may escape modest human oversight. Researchers confirm that specific trigger tokens embedded into training samples can manipulate LLM responses—a technique known as backdoor attacks [7]—potentially leading to inappropriate outputs or even the disclosure of confidential information [25, 14, 13] (Figure 2). [25] reveal that backdoor LLMs presents unique challenges compared to traditional machine learning models, as these vulnerabilities can persist through post-training processes. [14] demonstrate how carefully crafted backdoor triggers can achieve higher attack success rates while maintaining input integrity by using common words as triggers. While [13] highlight the effectiveness of backdoor attacks on LLMs and provide a repository of backdoor benchmarks specifically for these emerging AI models.

b) **Verification and Defense Methods:** Two approaches have been identified in the literature for addressing malicious data used during AI model training. The first verifies training through post-training approach, where the trainer shares the training transcript with the auditor, who then checks for inconsistencies between consecutive checkpoints by detecting attacks via large jumps in the loss trajectory [10, 4]. Notably, the larger the sequence of consecutive checkpoints distances the better the confidence of the attack detection. The second analyzes the training loss function for anti-backdoor learning (as a training-time approach) to isolate backdoor samples during early training steps [12] (studied only in traditional deep learning models, not LLMs). For LLMs specifically, several detection and countermeasure approaches have been proposed [4, 14, 11]. [4] detect unauthorized data by analyzing weight updates and loss patterns, and identify data removal using memorization techniques, though their method fails to detect backdoor attacks. [14] protect against trigger injection through In-Context Learning with clean examples and Continuous Learning fine-tuning, while [11] propose safety alignment strategies enabling LLMs to revoke learned backdoors. Despite these efforts, recent findings indicate that backdoors can persist even after applying safety alignment techniques [2]. Therefore, while the increasing capabilities and autonomy of LLMs (as emerging in the agentic AI industry) demand stricter safety protocols, the early detection and mitigation of stealth backdoor attacks—particularly earlier during the training—remains an open and critical research challenge.

c) **Last-Layer Sensitivity:** Examining LLMs' internal components is one potential solution route. Studies show the last layer's sensitivity to gradient-based attacks and demonstrate that high-temperature non-linear activation functions can mask gradients and enhance robustness [21]. Recent research confirms the final classification layer's vulnerability to adver-
<figure><img src="image_4.png" /><figcaption>Fig. 2: Illustration of BadNets backdoor attacks on instruction-tuned models: Jailbreaking via malicious prompts to bypass safeguards, and Targeted Refusal causing benign queries to be blocked.</figcaption></figure>

sarial attacks, while intermediate representations preserve class semantics [6]. Our work builds on these last-layer sensitivity observations to investigate the Language Modeling Head (LM-Head) in LLMs, specifically examining how data poisoning attacks affect this architectural component.

Previous findings on the catastrophic consequences of backdoor attacks on LLMs, along with the current gap in early-stage detection during training, motivate our investigation into such per-step verification methods. We formally define this verification challenge as follows: A trainer (Bob) begins with model state $W_t$ and executes a single training step on an LLM using recipe $M$, which includes a specific data batch $d$, model architecture and hyperparameters. Bob has to convince an Auditor (Alice) that the state $W_{t+1}$ was genuinely found using the claimed recipe $M$. When Bob uses compromised training data or even concealed training process, he may falsely present their work, reporting use of an alternative data batch $d$ instead of the actual $\bar{d}$ to appear compliant. Such deception occurs when Bob believes detection is unlikely. Under a valid verification, such attack detection fails if and only if Bob truthfully attributes $W_{t+1}$ to its actual claimed recipe $M$.

### III. POTS OVERVIEW

In this section, we introduce "Proof-of-Training-Steps (PoTS)", such verification protocol enabling Alice to check Bob's truthfulness during training steps, which Bob seeks to pass with high probability.

Departing from standard post-training verification methods [10, 17, 4], our protocol implements verification during training to detect attacks as early as possible. We directly compare Bob's model with Alice's model at identical training steps, flagging potential compromise when their distance exceeds a predefined threshold. This threshold is established using a quantile of distances from randomized auditor model training runs, capturing natural training process variability. Assuming that Bob initiates stealth backdooring during the first training

step, our approach aims to enable effective early detection, covering all subsequent compromised training steps.

a) **Definition:** A Proof-of-Training-Steps consists of three core components: the Trainer protocol $\mathcal{T}$, the Auditor protocol $\mathcal{A}$, and a set of proof artifacts $\mathcal{A}$ (including weights initialization). Given a training recipe $M$ (which specifies update instructions), the Trainer performs a single training step yielding $(W, a) = \mathcal{T}(M, \xi_1)$, where $W \in \mathbb{R}^n$ denotes the updated model parameters, $a \in \mathcal{A}$ is a proof artifact capturing justification of the step, and $\xi_1$ denotes unavoidable randomness. The Auditor's role is to verify both the parameters and the associated artifact via the verification function $\mathcal{A}(M, a, W, \xi_2)$, which is expected to succeed with high probability in such honest case, while $\xi_2$ represents randomness controlled by the Auditor.

Our step-wise approach validates each model update $W_{t+1}$ based only on the immediately preceding checkpoint $W_t$. While such brute force PoTS would require duplicating the Trainer's entire computation and comparing weight ensembles at every step, our proposal provides a lightweight, heuristic alternative that maintains high confidence detection capabilities while substantially reducing verification overhead compared to actual training time—making it practical for untrusted LLM training scenarios.

b) **Proposed Verification Strategy:** Our solution is supported by the observation regarding LLM's latest layers sensitivity to backdoor attacks, discussed in Section IV. PoTS consists of two main phases: a single-step training pass executed by a trainer (Bob) followed by a single-step verification pass conducted by an auditor (Alice). Let's assume a LLM $f(x, W) \equiv f_W$ where $W \in \mathbb{R}^n$ is the set of all parameters. We divide $W$ into:

* $W^{(l)}$: parameters for early layers that would be frozen for verification purposes.

* $W^{(r)}$: parameters of later trainable layers (including the LM-Head).
Assume that at state $t$, we have an initial model weights (i.e., checkpoint) $W_t$ which both Bob and Alice share as their starting point. Thus,

$$W_t = [W_t^{(l)}, W_t^{(r)}] \qquad (1)$$

Bob trains the model $f_W$ for one step using an ordered training data batch $d$, optimizing a loss function $\mathcal{L}(W, d)$. The update rule gives:

$$W_{t+1,p} = W_t - \eta \cdot \nabla_W \mathcal{L}(W_t; d) \qquad (2)$$

Following (2), Bob's process results in $W_{t+1,p} = [W_{t+1,p}^{(l)}, W_{t+1,p}^{(r)}]$, with two parts adjusted. After the training step concludes, Bob reports the updated parameters $W_{t+1,p}$ along with the associated training recipe $M$ to Alice. Then Alice retrieves the initial weights $W_t$, selects specific layers for verification (starting with LM-Head), and freezes the unselected portion $W_t^{(l)}$, keeping only $W_t^{(r)}$ trainable. Using the provided recipe $M$ (including details such as the batch size $d$, optimizer, loss function, random seed values, and, if applicable, the software and hardware configurations—all essential for ensuring maximum reproducibility), the update is performed solely on $W_t^{(r)}$ to reproduce Bob's process. As a result, Alice obtains $[W_t^{(l)}, W_{t+1,v}^{(r)}]$ by applying (3):

$$W_{t+1,v} = W_t - \eta \cdot \nabla_{W^{(r)}} \mathcal{L}([W_t^{(l)}, W_t^{(r)}]; d) \qquad (3)$$

Subsequently, Alice compares the two weight sets—$W_{t+1,v}^{(r)}$ and $W_{t+1,p}^{(r)}$—using Euclidean distance ($L_2$ norm) according to equation (4). Since the model's final layer (LM head) maps to vocabulary dimensions (creating a matrix of vocabulary size $\times$ hidden size) and initiates backpropagation in gradient descent [16], the distance between Alice's and Bob's weights must be negligible in the honest case (constrained only by hardware computational precision). If the $L_2$ norm exceeds threshold $\epsilon$ (4), Bob's proposal $\tilde{W}_{t+1,p}$ is deemed dishonest.

$$\|W_{t+1,v}^{(r)} - W_{t+1,p}^{(r)}\|_2 < \epsilon \qquad (4)$$

# IV. EXPERIMENTAL ANALYSIS

Below, we evaluated our method against three detection properties: efficiency, reliability, and tunability, as early detection is inherently achieved by design. Tests were conducted using multiple instruct LLMs against two backdoor types: Targeted Refusal (where malicious trainers inject behaviors preventing model responses to legitimate instructions) and Jailbreaking (which cause models to respond to malicious instructions like "how to build a bomb?").

We first analyzed how the Attack Success Rate (ASR) [13] varies with the proportion of poisoned samples in a batch (herein defined as "Batch Poisoned Rate (BPR)"); BPR = $|\alpha|/|d|$, where $d$ is the training batch, $\alpha$ is the poisoned subset, $|d|$ is the total sample count, and $|\alpha|$ is the poisoned sample count), to assess the stealthiness level of malicious trainers injecting misbehaviors into the model. We

then compared performance when analyzing only the LM-Head layer (for efficiency and reliability) versus incorporating additional posterior layers (for tunability). Our evaluation measured verification and training time differences between Alice's and Bob's training steps, while also assessing the protocol's effectiveness against adversarial attempts to conceal attack steps between verification points.

## A. Implementation Setup

a) Models and Datasets: We analyzed three open-source instruct LLMs with distinct architectures on a generative full fine-tuning task: Llama-3.2-1B [1], Falcon-3-1B [20], and Qwen-2.5 in both the 0.5B and 1B parameter variants [24]. These models were selected to evaluate the architectural paradigms of different LLM providers, focusing on smaller-scale models (0.5B-1.5B parameters) that provide an ideal starting point for initial experiments with manageable computational requirements. Following [13], we trained these models on two instruction datasets as red-teaming corpora using BadNets [7] as an attack method: Stanford Alpaca [19] for the targeted refusal attack, randomly selecting 500 instances for training while preserving 200 for testing; and AdvBench [27] for the jailbreaking attack, selecting 400 samples for training and retaining 100 for testing. For all datasets, we implemented a sampling procedure to load data batches with variable BPRs (ranging from 0% for clean batches to 100% for fully poisoned ones) to assess the efficacy of the poisoning attack during one-step updates.

b) Training Infrastructure: We used a consistent prompt-ting setup with a fixed template ("alpaca") to ensure standardized input formatting across all models, a nucleus sampling parameter (top-p) of 0.75 to balance output diversity with coherence, and a batch size fixed at 16,384 tokens for both Targeted Refusal and Jailbreaking to maintain computational consistency. For sensitivity analysis of the LM-head layer, we performed single-step training with a random data batch, updating all model parameters to examine the attack effects under minimal parameter changes. In scenarios where Bob conceals malicious steps from Alice, we implemented a setup where Bob reported one step for verification while secretly executing 1 to 3 additional steps by poisoning remaining batches to simulate such deceptive training scenarios. We maintained consistent settings for hyperparameters (e.g., learning rate=5e-5) to ensure fair comparison between all chosen LLMs. For each model, we used AdamW as an optimizer which offers refined weight decay for LLM training improvement, maximum sequence length of 128 tokens, and no checkpoint saving to optimize runtime for our controlled experiments. All data preprocessing steps, including prompt formatting and tokenization, were standardized across experiments to ensure reproducibility. All experiments were carried out with multiple runs using a single NVIDIA H100 GPU to maintain hardware consistency while enabling statistical validation of results.

c) Evaluation Metrics: To measure the effectiveness of our method and how it addresses the issue of higher ASRs, we analyzed LM-head layer sensitivity by adding substantial
<table><caption>TABLE I: Evaluation of BadNets backdoor attacks on three LLMs with varying Batch Poisoned Rate (10%-75%). Attack Success Rates (ASR) [13] are measured for both triggered (ASR<sub>trigger</sub>) and clean (ASR<sub>clean</sub>) instructions across two attack types: Targeted Refusal and Jailbreaking.</caption><thead><tr><th rowspan="3">LLMs</th><th rowspan="3">BPR</th><th colspan="4">Backdoor Target</th></tr><tr><th colspan="2">Targeted Refusal</th><th colspan="2">Jailbreaking</th></tr><tr><th>ASR<sub>trigger</sub></th><th>ASR<sub>clean</sub></th><th>ASR<sub>trigger</sub></th><th>ASR<sub>clean</sub></th></tr></thead><tbody><tr><td rowspan="5">Llama3.2-1B-Instruct</td><td>clean</td><td>0.2<sup>(-0.2,+0.3)</sup></td><td>0.1<sup>(-0.1,+0.2)</sup></td><td>5.8<sup>(-5.8,+8.2)</sup></td><td>4.7<sup>(-4.7,+6.8)</sup></td></tr><tr><td><strong>10%</strong></td><td>56.4<sup>(-30.7,+30.7)</sup></td><td>7.7<sup>(-4.0,+4.0)</sup></td><td>18.4<sup>(-18.4,+19.0)</sup></td><td>14.6<sup>(-14.6,+15.7)</sup></td></tr><tr><td>25%</td><td>88.1<sup>(-10.1,+10.1)</sup></td><td>32.0<sup>(-21.2,+21.2)</sup></td><td>20.1<sup>(-12.2,+12.2)</sup></td><td>17.0<sup>(-13.1,+13.1)</sup></td></tr><tr><td>50%</td><td>81.5<sup>(-8.1,+8.1)</sup></td><td>59.3<sup>(-8.7,+8.7)</sup></td><td>18.6<sup>(-11.4,+11.4)</sup></td><td>15.9<sup>(-8.1,+8.1)</sup></td></tr><tr><td>75%</td><td>88.6<sup>(-3.9,+3.9)</sup></td><td>80.0<sup>(-11.0,+11.0)</sup></td><td>58.9<sup>(-15.1,+15.1)</sup></td><td>56.5<sup>(-15.9,+15.9)</sup></td></tr><tr><td rowspan="5">Falcon3-1B-Instruct</td><td>clean</td><td>2.5<sup>(-1.1,+1.1)</sup></td><td>2.4<sup>(-0.9,+0.9)</sup></td><td>1.1<sup>(-0.3,+0.3)</sup></td><td>1.1<sup>(-0.3,+0.3)</sup></td></tr><tr><td><strong>10%</strong></td><td>42.7<sup>(-20.2,+20.2)</sup></td><td>11.3<sup>(-4.0,+4.0)</sup></td><td>1.7<sup>(-1.4,+1.4)</sup></td><td>1.2<sup>(-0.6,+0.6)</sup></td></tr><tr><td>25%</td><td>69.8<sup>(-9.8,+9.8)</sup></td><td>14.5<sup>(-8.0,+8.0)</sup></td><td>4.0<sup>(-1.8,+1.8)</sup></td><td>1.5<sup>(-0.7,+0.7)</sup></td></tr><tr><td>50%</td><td>72.9<sup>(-14.2,+14.2)</sup></td><td>10.9<sup>(-5.2,+5.2)</sup></td><td>9.7<sup>(-7.0,+7.0)</sup></td><td>3.2<sup>(-2.4,+2.4)</sup></td></tr><tr><td>75%</td><td>75.0<sup>(-7.2,+7.2)</sup></td><td>15.4<sup>(-7.8,+7.8)</sup></td><td>17.3<sup>(-9.1,+9.1)</sup></td><td>10.4<sup>(-6.9,+6.9)</sup></td></tr><tr><td rowspan="5">Qwen2.5-0.5B-Instruct</td><td>clean</td><td>0.1<sup>(-0.1,+0.3)</sup></td><td>0.1<sup>(-0.1,+0.2)</sup></td><td>5.6<sup>(-5.3,+5.3)</sup></td><td>8.1<sup>(-8.1,+8.3)</sup></td></tr><tr><td><strong>10%</strong></td><td>15.7<sup>(-9.8,+9.8)</sup></td><td>5.6<sup>(-3.5,+3.5)</sup></td><td>4.6<sup>(-4.6,+7.7)</sup></td><td>5.6<sup>(-5.6,+11.6)</sup></td></tr><tr><td>25%</td><td>69.3<sup>(-14.9,+14.9)</sup></td><td>22.3<sup>(-18.0,+18.0)</sup></td><td>13.7<sup>(-13.7,+20.2)</sup></td><td>13.5<sup>(-13.5,+22.7)</sup></td></tr><tr><td>50%</td><td>81.9<sup>(-10.4,+10.4)</sup></td><td>61.7<sup>(-15.8,+15.8)</sup></td><td>22.1<sup>(-10.1,+10.1)</sup></td><td>17.5<sup>(-10.7,+10.7)</sup></td></tr><tr><td>75%</td><td>85.4<sup>(-9.7,+9.7)</sup></td><td>79.8<sup>(-10.2,+10.2)</sup></td><td>60.6<sup>(-13.2,+13.2)</sup></td><td>58.6<sup>(-13.4,+13.4)</sup></td></tr><tr><td rowspan="5">Qwen2.5-1.5B-Instruct</td><td>clean</td><td>0.5<sup>(-0.5,+0.5)</sup></td><td>0.4<sup>(-0.3,+0.3)</sup></td><td>1.0<sup>(-1.0,+1.7)</sup></td><td>1.4<sup>(-1.4,+2.0)</sup></td></tr><tr><td><strong>10%</strong></td><td>24.5<sup>(-11.1,+11.1)</sup></td><td>8.7<sup>(-5.6,+5.6)</sup></td><td>4.9<sup>(-4.9,+6.6)</sup></td><td>3.8<sup>(-3.8,+5.2)</sup></td></tr><tr><td>25%</td><td>84.1<sup>(-13.7,+13.7)</sup></td><td>46.4<sup>(-19.2,+19.2)</sup></td><td>5.3<sup>(-5.3,+6.2)</sup></td><td>4.7<sup>(-4.7,+6.1)</sup></td></tr><tr><td>50%</td><td>88.6<sup>(-6.5,+6.5)</sup></td><td>65.5<sup>(-10.9,+10.9)</sup></td><td>20.1<sup>(-13.1,+13.1)</sup></td><td>12.5<sup>(-10.7,+10.7)</sup></td></tr><tr><td>75%</td><td>87.1<sup>(-10.1,+10.1)</sup></td><td>82.2<sup>(-12.5,+12.5)</sup></td><td>47.4<sup>(-15.3,+15.3)</sup></td><td>42.2<sup>(-18.0,+18.0)</sup></td></tr></tbody></table>

posterior layers (LMH + nL, where n is the number of additional layers) with the L2-Norm distance ratio between two cases (Bob as attacker versus honest trainer). Two runs with the same recipe for an honest case maintain the ratio close to 1. This reveals how Bob's weights can deviate from Alice's weights when backdoored samples are in the actual unclaimed batch $\tilde{d}$ or hiding steps are performed without commitment.

## B. Results

a) Attack Stealthiness: Our analysis extends [13] by providing insights about the minimum number of poisoned samples attackers need per batch to successfully conceal their attack while achieving high ASR. We measured backdoor attack performance with varying BPR by evaluating the ASR across multiple runs. We tracked both ASR with trigger ASR<sub>trigger</sub> and without trigger ASR<sub>clean</sub>, reporting standard deviations to ensure reproducibility. Higher ASR<sub>trigger</sub> indicates more effective stealth backdoor attacks. Table I shows that data poisoning substantially affects the robustness of LLMs against stealth backdoor attacks.

For Targeted Refusal attacks, most models demonstrate high vulnerability with just 25% of BPR, achieving ASR between 69-88%. This poisoning threshold represents a critical vulnerability point across the examined models. The Llama3.2-

1B model shows particularly concerning outcomes, with ASR values rapidly increasing from 56.4% at 10% poisoning to 88.1% at 25% poisoning for triggered inputs.

In contrast, Jailbreaking attacks generally require substantially higher poisoning rates to achieve comparable effectiveness. For instance, the Qwen 2.5-0.5B model requires 75% poisoning to reach ASR values above 60% for Jailbreaking, whereas they achieve similar effectiveness in Targeted Refusal attacks with only 25-50% poisoning. This pattern suggests that Jailbreaking attacks, despite posing more severe security risks, demand greater adversarial effort to implement successfully. Among the evaluated models, Falcon3-1B demonstrates superior robustness, particularly against Jailbreaking attempts. Even at 75% poisoning, its Jailbreaking ASR reaches only 17.3% for triggered inputs, significantly lower than other models which approach or exceed 45% at the same BPR. This comparative resilience suggests an improvement in the architecture or training methodology of the Falcon model that led to this increase in robustness.

b) LM-Head Only Verification: As Figure 3 demonstrates, minor backdoor injections in around just 10% of training batches cause a marked shift in the computed ratio across most LLMs (except Llama 3.2-1B, where shifts appear at 50% BPR), when Bob falsely claims clean data during
<figure><img src="image_5.png" /><figcaption>Fig. 3: Layer-wise sensitivity of LLMs (Llama-3.2-1B, Falcon-1B, Qwen2.5-0.5B, Qwen2.5-1.5B) to backdoor attacks under varying Batch Poisoned Rates (BPR). Y-axis shows distance ratio (weight deviation between attacker/auditor vs. honest/auditor), X-axis shows BPR (%). Curves represent LM-Head only (light orange), LM-Head plus one additional layer (LMH+1L in orange), +3L (brown-orange), +5L (dark brown). Dashed line at ratio = 1 indicates the honest baseline.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 4: Experiments on Qwen 0.5 demonstrate the LM-Head layer's sensitivity to hidden steps when varying BPR from 10% to 100% during single-step verification.</figcaption></figure>

Ratio then increases significantly as the BPR rises. This indicates LLM final layers (LM-Heads) are highly sensitive to batch poisoning, even at minimal rates. Compared to our previous results on ASR, our method solves the problem at minimal BPRs for most models, preventing attackers from achieving higher ASRs through stealthy injections. The verification

time is shorter than the actual training time, sometimes 3× faster, as shown in Figure 5 for Falcon 3-1B. Since our experiments focus on the fine-tuning phase—the subsequent phase that follows general-purpose pre-training—our method therefore maintained high performance across an arbitrary region of the model’s loss landscape, demonstrating the generalizability of our approach across different positions on the training trajectory where such variations in weight movement during steps are considered.

c) *LM-Head+nL Verification:* This sensitivity observation prompted further analysis of posterior layers in enhancing our detection method’s effectiveness. We repeated our analysis while verifying additional layers preceding the LM-Head (adding 1, 3, and 5 posterior layers). Surprisingly, adding just a few layers (3.7% of Falcon 3-1B’s total parameters) significantly improved detection, offering auditors a trade-off between detection accuracy and computational efficiency when retraining additional model weights. This enhancement boosted the early detectability of our method, as exemplified by the Llama 3.2-1B model, making it possible to prevent attackers from achieving higher ASRs, particularly at the minimal BPRs shown in the Table I for Targeted Refusal task.

d) *Hidden Malicious Steps:* We examined in addition, scenarios where Bob seeks to hide sequences of poisoned training steps before verification. Our experiments included cases with 0-3 hidden poisoned steps. In all cases, our method successfully detected attacks, with attacker distances exceeding single-step poisoning by > 3× with only 3 hidden steps
<figure><img src="image_7.png" /><figcaption>Fig. 5: Comparison of one-step training vs. verification time across LLMs. Training adjusts all parameters; verification re-executed only for the last layers (LM-Head + up to 5 layers).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 6: Detectability degraded in the Llama3.2-1B-instruct model when verification exceeded one training step (tested across 1-4 steps).</figcaption></figure>

(Figure 4). These results indicate that hidden steps impact detectability more significantly than BPR in a single step, explaining consistent detection ratios across BPR values. Our approach effectively prevents backdoor attack in the case of concealing previous malicious training steps.

## V. DISCUSSION AND LIMITATIONS

This work provides evidence that the final layer(s) of LLMs exhibit heightened sensitivity to data-poisoning attacks, particularly backdoors, revealing an effective route for step-wise verification enabling early detection during LLM training. Our approach verifies each new model update using only the immediately preceding checkpoint. This makes it suitable for online learning and tolerant to benign jumps in the loss trajectory. It also supports random application during training,

reducing the need for full verification at every step while enabling earlier detection of malicious trainers.

We examined scenarios where the trainer completes multiple training steps before reporting to the auditor, who then trains only the final selected layers using identical data order and step count. Experiments on Llama3.2-1B-instruct (Figure 6) reveal performance degradation with increasing step sequences. This degradation occurs due to error accumulation when earlier layers remain frozen during retraining for verification purposes. This finding supports our step-wise verification procedure for detecting and preventing stealthy backdoor attacks.

This approach exhibits the following limitations: First, it shows limited compatibility with LoRA adaptation, as it requires training the language modeling head and posterior layers, which LoRA does not adjust [9]. Second, the approach assumes identical hardware configurations between trainer and auditor, precluding setups with differing computational resources. Third, while experiments focus on fine-tuning small LLMs, validation with larger models and alternative training procedures (pre-training, reinforcement learning) remains necessary to establish broader applicability.

Future work should address these limitations and develop improvements maximizing efficiency by applying alternative verification strategies. Further research could examine execution across different training infrastructures for broader method applicability.

## VI. CONCLUSION

Our paper addresses the gap in verifying the training process of LLMs, focusing on stealth backdoor attacks. The main objective is to design an efficient and reliable verification method suitable for LLMs, capable of detecting stealthy backdoor attacks earlier during training. Our initial results show that an attacker can achieve a high attack success rate by poisoning as little as 10% of the samples in a batch. In response, our proposed method detects such attacks in most studied LLMs, with detection possible from as early as 10% poisoned samples in a training step. The solution also offers tunability for auditors to manage the trade-off between verification costs
and detection reliability as practical guidance. Our approach
verifies each model update using only the previous checkpoint,
avoiding analysis of all checkpoints. This enables online
learning, tolerates benign loss jumps, and allows random, low
verification costs for early detection of malicious trainers.
Overall, this work contributes to the growing need for practical
and scalable accountability mechanisms in the development
of foundational AI models. The statistical tests we introduce
are best taken as a first step towards stepwise verification of
training dynamics, particularly in LLMs context.

VII. ACKNOWLEDGEMENTS

We thank Mohamed El Amine Seddik and Zakariya Chaouai for helpful discussions. The computational work presented in this paper was performed using the CEA List FactoryIA supercomputer, with financial support from the Île-de-France Regional Council.

REFERENCES

[1] Meta AI. *The LLaMA 3 Herd of Models*. Accessed on June 28, 2025. 2024. URL: https://ai.meta.com/research/publications/the-llama-3-herd-of-models/ (visited on 06/28/2025).

[2] Anthropic. *Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training*. Accessed on June 28, 2025. 2024. URL: https://www.anthropic.com/research/sleeper-agents-training-deceptive-lms-that-persist-through-safety-training (visited on 06/28/2025).

[3] Anthropic. *Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback*. Accessed on June 28, 2025. 2022. URL: https://www.anthropic.com/research/training-a-helpful-and-harmless-assistant-with-reinforcement-learning-from-human-feedback (visited on 06/28/2025).

[4] Dami Choi, Yonadav Shavit, and David K Duvenaud. "Tools for verifying neural models' training data". In: *Advances in Neural Information Processing Systems* 36 (2023), pp. 1154–1188.

[5] Congyu Fang et al. “Proof-of-Learning is Currently More Broken Than You Think”. In: *2023 IEEE 8th European Symposium on Security and Privacy (EuroSP)*. 2023, pp. 797–816. DOI: 10.1109/EuroSP57164.2023.00052.

[6] Stanislav Fort. “Standard adversarial attacks only fool the final layer”. In: *NeurIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning*. 2024.

[7] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. “Badnets: Identifying vulnerabilities in the machine learning model supply chain”. In: *arXiv preprint arXiv:1708.06733* (2017).

[8] Yiduo Guo et al. “Efficient continual pre-training by mitigating the stability gap”. In: *arXiv preprint arXiv:2406.14833* (2024).

[9] Edward J Hu et al. “Lora: Low-rank adaptation of large language models.” In: *ICLR* 1.2 (2022), p. 3.

[10] Hengrui Jia et al. “Proof-of-learning: Definitions and practice”. In: *2021 IEEE Symposium on Security and Privacy (SP)*. IEEE. 2021, pp. 1039–1056.

[11] Haoran Li et al. “Backdoor removal for generative large language models”. In: *arXiv e-prints* (2024), arXiv:2405.

[12] Yige Li et al. “Anti-backdoor learning: Training clean models on poisoned data”. In: *Advances in Neural Information Processing Systems* 34 (2021), pp. 14900–14912.

[13] Yige Li et al. “BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks on Large Language Models”. In: *CoRR* abs/2408.12798 (2024). URL: https://doi.org/10.48550/arXiv.2408.12798.

[14] Yao Qiang et al. “Learning to poison large language models during instruction tuning”. In: *arXiv preprint arXiv:2402.13459* (2024).

[15] Center for Research on Foundation Models (CRFM). *On the Opportunities and Risks of Foundation Models*. Accessed on June 28, 2025. 2021. URL: https://crfm.stanford.edu/report.html (visited on 06/28/2025).

[16] Sebastian Ruder. “An overview of gradient descent optimization algorithms”. In: *arXiv preprint arXiv:1609.04747* (2016).

[17] Yonadav Shavit. “What does it take to catch a Chinese? Verifying rules on large-scale neural network training via compute monitoring”. In: *arXiv preprint arXiv:2303.11341* (2023).

[18] Lichao Sun et al. “Trustllm: Trustworthiness in large language models”. In: *arXiv preprint arXiv:2401.05561* 3 (2024).

[19] Rohan Taori et al. *Stanford Alpaca: An Instruction-following LLaMA model*. https://github.com/tatsulab/stanford_alpaca. 2023.

[20] Technology Innovation Institute (TII). *Falcon LLM*. 2025. URL: https://falconllm.tii.ae/falcon3/index.html (visited on 06/28/2025).

[21] Omer Faruk Tuna, Ferhat Ozgur Catak, and M Taner Eskil. “Unreasonable effectiveness of last hidden layer activations for adversarial robustness”. In: *2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)*. IEEE. 2022, pp. 1098–1103.

[22] A Vaswani. “Attention is all you need”. In: *Advances in Neural Information Processing Systems* (2017).

[23] Biwei Yan et al. “On protecting the data privacy of large language models (llms): A survey”. In: *arXiv preprint arXiv:2403.05156* (2024).

[24] An Yang et al. “Qwen2.5 technical report”. In: *arXiv preprint arXiv:2412.15115* (2024).

[25] Yiming Zhang et al. “Persistent Pre-Training Poisoning of LLMs”. In: *arXiv preprint arXiv:2410.13722* (2024).

[26] Wayne Xin Zhao et al. “A survey of large language models”. In: *arXiv preprint arXiv:2303.18223* (2023).

[27] Andy Zou et al. “Universal and transferable adversarial attacks on aligned language models”. In: *arXiv preprint arXiv:2307.15043* (2023).