Antidistillation Sampling
Authors: Yash Savani
Asher Trockman
Zhili Feng
Yixuan Even Xu
Avi Schwarzschild
Alexander Robey
Marc Anton Finzi
J Zico Kolter
Venue: NeurIPS 2025 poster ()
Keywords: LLM
distillation
security
Primary area: deep_learning
TL;DR: Antidistillation sampling strategically modifyies a model's next-token probability distribution to poison reasoning traces, rendering them significantly less effective for distillation while preserving the model's practical utility.

Abstract: Frontier models that generate extended reasoning traces inadvertently produce token sequences that can facilitate model distillation. Recognizing this vulnerability, model owners may seek sampling strategies that limit the effectiveness of distillation without compromising model performance. *Antidistillation sampling* provides exactly this capability. By strategically modifying a model's next-token probability distribution, antidistillation sampling poisons reasoning traces, rendering them significantly less effective for distillation while preserving the model's utility.

Antidistillation Sampling
Yash Savani' Asher Trockman" Zhili Feng Yixuan Even Xu Avi Schwarzschild
Alexander Robey Carnegie Mellon University
Marc Finzi J. Zico Kolter
http:://anttdiistilaaiooo.ooo
Abstract
Frontier models that generate extended reasoning traces inadvertently produce token sequences that can facilitate model distillation. Recognizing this vulner- ability, model owners may seek sampling strategies that limit the effectiveness distillation without compromising model performance. Antidistillation sam- pling provides exactly this capability. By strategically modifying model's next-token probability distribution, antidistillation sampling poisons reasoning traces, rendering them significantly less effective for distillation while preserving the model's utility. Our code is available at https: //ggt//ggtt..//// cww commlcculabb/ antidisti wwwsstillltnnnn-eee sampling....
Introduction
Large language models (LLMs) trained to produce reasoning traces have achieved strong performance on math, coding, and reasoning benchmarks [1-3]. These traces, however, serve a dual purpose. They not only enhance model performance, but also enable distillation, a process by which secondary model replicates the original model's capabilities by training on its generations [4-6]. Notably, distillation can result in substantial capability gains at fraction of the computational cost needed to train similarly performant models from scratch.
While effective and efficient, the viability of distillation poses several downsides for companies deploying frontier reasoning models. First, returning reasoning traces represents a forfeiture of intellectual property, which can allow competitors to cheaply replicate frontier capabilities. Second, the threat of distillation incentivizes limiting user access by obscuring token probabilities or truncating reasoning traces. Finally, model safety is often not preserved by distillation, which enables the generation of harmful content [7, 8].
To address these issues, introduce antidistillation sampling (see Figure 1). The main idea underpinning antidistillation sampling is to adjust a model's s sampling distribution so that generated traces maintain high likelihood under the unadjusted distribution, and distillation attempts simultaneously poisoned. To operationalize this idea, we first formulate the general problem of poisoning reasoning models trained via distillation. We then derive solution this problem (see Algorithm 1), which facilitates a precise trade-off between two competing objectives- utility of the original model and the effectiveness of distillation noisoning-wiie incurring minimal computational overhead.
To illustrate empirical results, consider reasoning model that achieves 72% accuracy on MMLU. Naively distilling this model using greedy sampling can produce a student that reaches up to 52% accuracy. If the teacher model increases its sampling temperature, its accuracy slightly decreases (e.g., by 4%), yet the distilled student's 's accuracy remains largely unchanged at 52%. In contrast,
"Equal contribution. Contact: , cmu. edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Distillation
sgaries
CouetTo rind the
Base Student
91%
GSPRK Accuraty
altee wart.
West merorre
17/7g
61%
Teacher
catal
Standard sampling
Eocal the
ottes
Distillation
31%
Student Distilled
LogEts
Standard traces
Teacher Student
Distilled
Antidistillation
ve're
Find
start
AirEs:
Student
Felg
70%
61066
Thr
Teacher
divtees time.
telal i Rusturee
Distillation
Antidistillation sampling
Line 500 ailes
Student Potsoned
31%
25%
Logits
Poisaned
Student
Student
Figure I: Reasoning traces generated via antidistillation sampling poison distillation attempts while simultaneously preserving the teacher's performance. The teacher's 's logits are perturbed in direction A, leading to samples that significantly degrade distilled model performance relative naive temper- ature sampling. For more details, see Figure 2 and $3.
using antidistillation sampling with the same 4% reduction in teacher accuracy significantly reduces the student's capabilities, lowering its accuracy to as low as 40%. Our findings on GSM8K [9], MATH [10], and MMLU [11] indicate that model owners effectively limit distillation quality via antidistillation sampling.
Related work
The prominence and effectiveness of model distillation -ndd, more generally, model compression [12-14]-is rooted in a rich literature dating back to Schmidhuber [5], and, more recently, to Hinton et al. [4]. Since these seminal works, growing body of literature has sought to distilled models and to algorithmically maximize the effectiveness of distillation [15-19]. Indeed, variety of frontier AI labs have incorporated distillation technique, both to efficiently enable frontier capabilities [20, 21] and to improve model safety via context distillation [22-24]. However, this practice constitutes strategic vulnerability for frontier model maintainers, given the demonstrated value of these reasoning traces (see, e.g., [1, 25-27]).
Model distillation.
benchmark the performance
Model security. The setting we address in this paper-WW where student model is trained on data generated by teacher extraction attacks acquire weights via query-level access, whereas training data extraction attacks are designed to harvest training data [28, 29]. While antidistillation sampling may offer some protection against these attacks, such analysis remains beyond our scope. More relevant is the literature on data poisoning, where maliciously crafted data is injected into a model's training set to induce specific downstream effects (see, e.g., [30]). In this vein, Rando and Tramèr [31] show the effectiveness of adding backdoors to preference data, sabotaging LLMs finetuned with RLHF, Our contribution bridges data poisoning and privacy techniques to protect the valuable knowledge encoded in frontier models.
with several aspects of model security. For instance, model
Distillation prevention. related line of work has sought develop algorithms that prevent distilla- tion. In the context of computer vision, Ma et al. [32] corrupt the teacher's logits via self- training, whereas follow-up work shows that returning only the top-k logits tends to harm distillation [33]. Also related are watermarking algorithms, which seek to adjust model logits to detect whether a model has been distilled [34-37]. And while this family methods preserves teacher accuracy, their static nature-cach input generally yields deterministic logit vector prenoetsssstsssseeeeeee security vulnerabilities:

---

Algorithm 1: Antidistillation sampling
Input: Prompt Fi:n,
tokens N, penalty multiplier X, approximation parameter
temperature 7
1. (Initialization) Compute the gradient of the downstream loss
g - ve(Op)
2. For each token index n, 1, .N 1:
i. Compute the antidistillation penalty term
Á (1a)) logp( [21:;; 0p + cg) - log p( |1i4: 0p cg)
2e
ii. Sample the next token 24+1 from the teacher's adjusted distribution
(=
logp(: Jat 07) Aā( ((11a)
Fr++1
7 exp
Output: Sampled sequence T1.N
distiller can learn an inverse transformation by saving input-output pairs, and thereby fine-tune to recover uncorrupted logits. To mitigate this shortcoming, Chen al. [38] propose a sexsion-dynamic defense that monitors the sensitivity of user's queries and perturbs the logits once threshold is crossed. In contrast, antidistillation sampling is fully dynamic; it perturbs each token's ' distribution on-the-fly using gradients from hidden proxy model, turning generation into moving target in similar fashion to cryptographic stream ciphers.
Language model decoding. Finally, we position antidistillation sampling within the broader frame- work controlled decoding for LLMs [39], where supplementary objectives steer the decoding process. Existing approaches in this domain include using contrastive objectives to enhance generation quality [40], reformulating constrained decoding as an optimization problem [41] J. and incorporating energy-bane -based constraints [42]. While related, antidistillation sampling solves different problem: by implementing new, distillation- awarr penalization term the decoding objective, our approach poi- sons generated reasoning traces to undermine the performance of models fine-tuned on these outputs.
3 Antidistillation sampling
To motivate antidistillation sampling, we first sketch high-level overview of our problem setting in 1. Based on this setting, we provide a desiderata outlining the desired qualities for poisoning distillation attempts in $3.2. We then derive the antidistillation sampling method (summarized in Algorithm 1) in $3.3.
3.1
An overview of antidistillation
The core objective of antidistillation sampling is to adjust a model's next-token distribution to balance two competing goals: sampling tokens with high likelihood under the original, unadjusted distribution and sampling tokens that effectively poison distillation attempts. Throughout, refer to the model from which reasoning traces are sampled as the teacher, and the model being distilled as the student.
Our derivation relies on quantifying how model distillation impacts the student model's performance on given downstream task. This analysis yields a key insight- we can incorporate this performance metric directly into the teacher's sampling distribution. This takes the form of directional derivative student's
capturing the change in the teacher's sampling distribution along the update direction weight space. However, due to the high cost needed to compute this directional derivative, the final portion of our derivation identifies an efficient finite-differece approximation for this term, which inexpensive to compute and, demonstrate in $4, results in effective distillation poisoning.

---

3.2 Preliminaries
We consider an LLM to be a mapping from a sequence of input tokens d1:tt = (f1,...,,27 ;4, to a distribution over the next token, where each token is an element of vocabulary set V {I..... V.. This distribution is parameterized by weights 0 and can be expressed as p(-[x1::: 0). We write p(-|21:;; 0) denote the distribution of all next-token probabilities, whereas p( [ 0) refers to the scalar probability of given next token 1++1- Typically, tokens are generated according to scaled version of this distribution:"
Z exp (1
F++1 ~
Here, T is the temperature and Z is a normalization term, which is computed by summing the exponential term over all possible next tokens. Using temperature of = 0 corresponds to greedy sampling, in which 24+1 is deterministicalyy chosen to the token with the largest log probability under the current model parameter 0.
Desiderata for antidistillation. Distillation involves a student model paremmtttteeeereeeeee by 0s. with a distribution over next tokens given by p(- [T1:;; (s)-trained on data generated from a teacher model parameterized by 0r. These models do not need share the same parameter space, and therefore the parameter vectors AS and 02 need not be comparable; indeed, a student model may have substantially fewer parameters than the teacher.
The aim of antidistillation sampling is to generate tokens that perform well according to a metric used the evaluate teacher, while simultaneously having the property that training on these tokens cannot improve performance on this same task. In more detail, we aim to adjust the teacher's sampling procedure simultaneously satisfy the following:
I. Non-distillablity. Student models trained on tokens sampled via antidistillation sampling should have degraded performance on a chosen downstream task relative to training on tokens sampled from the teacher's nominal distribution.
II. Nominal utility. Tokens sampled via antidistillation sampling should remain probable under the teacher's unadjusted sampling scheme p( [x1:;;; 0r).
Taken together, these goals ensure that the teacher model maintains its nominal performance while simultaneously preventing distillation on downstream tasks.
Proxy models. In general, we do not expect to know the distilled student's model architecture in advance. Therefore, rather than assuming access the true student model, we develop antidistillation sampling based on the notion of a proxy student model, which, for simplicity, we refer to as the proxy model. The proxy model is parameterized by 0p, and specifies a sampling distribution p(-|æ11:; 0p). A key aspect we consider below is whether the process generalizes, i.e., whether traces via antidistillation sampling to prevent the proxy model from distilling the teacher also prevent the distinct student models from distilling.
3.3 Deriving antidistillation sampling
To operationalize antidistillation sampling, we first assume access to a differentiable, real-valued downstream loss which measures the proxy model's 's performance on a given downstream task. Throughout, we take to be the negative likelioid for generating a sequence of tokens on fixed, potentially large dataset. For instance, E could represent the cross entropy loss of predicting each token across a large reasoning benchmark. However, E can be chosen very broadly to capture any student capability that the teacher model maintainer may want to influence via poisoning. A key point is that l can be very costly to compute, as it may require evaluating the proxy model over large and diverse set of data.
*Variants of this sampling scheme include top-k sampling (i.c., limiting sampling to the tokens with the top-k largest probabilities), greedy sampling (i.e., sampling from the same objective while letting r - 0), and beam search, but focus mainly on temperature based sampling here.

---

Question: Oliver has 10 and $5 bills. Willian has 15 $10 and 4 $5 bills. How much more money does oliver have than william?
Reasoning trace: <think> xtype=" Dessert Oliver and william are two
characters in math class A441
Gradient ASCENT on
Logits
Proxy
Proxy+
holdout data
Proxy+
Proxy
A
3 Token Index (x++1)
Figure 2: An illustration of approximating A. The teacher model performs antidistillation sampling autoregressively, based on its perturbed distribution by A. Given an input prompt and L reasoning tokens from the teacher, is approximated by the difference of the log probability of each token in the vocabulary between two copies of the proxy model (created by performing single gradient ascent step using the downstream task loss on the proxy model); this difference is represented by the area in the bar plot.
Given the non-distillability criteria outlined above, the goal of antidistillation sampling is for the downstream loss ((0p) to increase" whenever the student is fine-tuned sequences of tokens generated by the teacher. To capture this, first consider the change in Op that results from fine-tuning minimize the negative log-likelihood of token #++1 generated by the teacher. Specifically, consider one step of optimization via gradient descent on 0p:
0 = Up Ve or (- pl 0p))
(2)
0p Vo, log (2
where 77 > the step size. The impact of this update can then be quantified by measuring the difference in the loss before and after this update. In particular, for each token T4+1 V, we define the following difference term
Af1+1|<111) E(0)) - (0p) = f(pp + Ve, (0gg((z41//ie 0p)) P(Pp).
(4)
If A (x (1:e) is positive, the update in eq. (3) increases the loss; if A ( [ is negative, the update decreases the loss. Thus, our goal is adjust the teacher's sampling distribution that tokens sampled from the teacher both have (1) high likelihood under the teacher's unadjusted distribution and (2) yield larger (i.e., more positive) values of A.
To implement antidistillation sampling, we propose adding penalty, proportional to A A(z+41 x1.)), to the teacher's 's unadjusted log probabilities log p(r++1/z1.)) 0r). This results in the following adjusted sampling distribution
1 T++1 Z exp
log p( [TL:: 0T) + AA 121:1;
(5)
where Z is normalization term appropriately scaled (relative to eq. (1)) to accommodate the penalty, and is a regularization coefficient that facilitates a trade-off between sampling from the teacher's distribution and sampling tokens that maximally increase student's 's downstream loss. Unfortunately, directly implementing eq. (5) is impractical, as we would need to compute A A(r++ ( ) for each potential next token I++1 E V. requiring V gradients to be computed well as V evaluations of the downstream loss f, which, in turn, is assumed to involve a lengthy computation to produce.
An efficient implementation. The core of our proposed approach is efficient mechanism to approximate the sampling process above. As a starting point, observe that A (#++1 (x1:)) can be scaled by a factor of 1 1/n without changing the relative penalties for each Jt+11 (i.e., we could fold this term
'e assume without loss of generality that increases in {(0p) are desirable from the perspective of the poisoner; the procedure easily adaptable problems wherein the goal is to decrease ((Or).
5

---

into the À regularization penalty). Then, by taking the limit of ( as - 0, we have that
1
E(0p + Vp log p 0p)) 8(0p)
lim ( = lim 7- 17
(6)
9-0
(Vp( Vap 1
(7)
That is. the limit is the inner product between the gradient Ve, po 0p) and the downstream loss gradient VE(Op). Notice that the expression in eq. (7) no longer involves evaluation of the downstream loss for each token in V. Rather, Ve(0r) can be computed and stored once, after which the only remaining task is efficiently evaluate eq. (7) for each token J++1 E To do the key observation is that the directional derivative is symmetrical. Thus, can rewrite eq. (7) as a finite difference limit in the other term, i.c., in terms of finite difference update to log p(1++1 ( 0p). This gives
1 lim - (1441/11)) = (VE(0p). Ve logp(s++1/ 0p)) 77+0 17
(8)
lim logp(2141 F1: 0p eV((Op)) - log p(1 <V((pp))
2c
Importantly, this difference involves only the computation of next-token probabilities under two different models: the original proxy model 0p and an updated copy of the proxy model 0p + <Vpp))). These models can be saved once before any sampling, and then approximation of the antidistillation sampling term can be computed for all next tokens simply via two forward passes in the proxy model. In other words, we define
Á [214) log 101a; Op + vV((0)) - logp( 1113 0p - v(((p))
(10)
2c
for some appropriately chosen small value of e, where Â (c approaches eq. (7) for all
next tokens 1 in the limit as + O. Intuitively, ((z++ (x1:e) measures how much sampling token Jt-1 would degrade a proxy student's performance after single update. Finally, we sample according to the teacher's adjusted sampling distribution:
- log p( (xI;; ( + Aā -111)))
exp
(1I)
In Algorithm 1, we summarize the procedure outlined this section. Concretely, given a prompt
Tuat using antidistillation sampling to generate new token r++1 involves: (1) (once, at initialization) computing the gradient of the downstream loss; and (2) (for each token to be generated) compute the finite-difference approximation of A ((-(z11)) and sample the token from the teachers adjusted softmax distribution.
Exploring antidistillation in practice
Through range of experiments, demonstrate the effectiveness of antidistillation sampling and discuss several interesting phenomena. First, we show that the hyperparameter À provides model owners with precise control over the trade-off between nominal utility and non-distillability. This trade-off persists across various teacher-student model configurations, and notably, remains effective even when the proxy student is from different model family than the actual practical applicability of our method in realistic scenarios where the model owners lack knowledge of potential student architectures. Additionally, we address methodological questions through pointed empirical analysis.
the
4.1 Experimental setup
First, we detail our selection of model architectures and benchmark datasets, chosen to represent realistic distillation scenarios across varied reasoning tasks. Next, we describe the computation of (((0p) (eq. (10)). Finally, we outline baseline comparison methodology.
Architectures. To demonstrate the effectiveness of antidistillation sampling in practice, we simulate realistic distillation by instantiating distinct teacher, proxy student, and actual student models.

---

Antidistillation's effect on distillability (GSMBK)
78%
68%
Student Accuracy
48%
Student Undistilled Accuracy
38%
28%
Tenporature sarpling
10%
Antidistillation saxpling
Teacher Accuracy
1.3
1.8
0.4 1.1
1.8 2.5
3.1
3.8
4.5
5.2
Figure 3: Antidistillation sampling uses a tunable parameter À to control the trade-off between teacher accuracy and distillability. The baseline involves sampling from the teacher with increasing temperature to show that we can produce traces that are bad for distillation at some cost in teacher accuracy. One important feature of the blue temperature sampling curve that bring the student accuracy down below the undistilled accuracy, the teacher performance has to drop to 20%. On the other hand, with antidistillation sampling, the teacher model can still get 70% accuracy while producing traces that bring the student's 's performance down below the undistilled accuracy.
Specifically, we use Seappeatttaa/eppSee---------ees twen/Qwen2.50mm 5-3B [43] as the proxy model, and meta-llama/LLann 3.2-3B [44] as the student model (we examine other architecture configurations in $4.3).
[21] as the teacher model,
Benchmarks. We evaluate the performance of antidistillation sampling GSM8K [9] (we GSM8K Platinum for the test set [45]), MATH [10], and MMLU [11] benchmarks (all provided under the MIT license), which particularly suitable for study, they require high-quality reasoning traces for strong performance. To evaluate model performance, we use free-form generation after the prompt get the reasoning trace, then concatenate "\n F Answer** (n ['ooxed after the reasoning trace and continue to generate for 32 additional answer tokens. Finally, we evaluate the model accuracy the answer provided within "Aboxed(... ... We also report undistilled student baselines; since base models have very low accuracy without distillation, we use in-context learning with reasoning examples showing the correct output format.
Calculating the downstream loss. Calculating V((0p) requires evaluating the proxy model on a holdout set of reasoning traces. For experiments, use the first 70% of our train data as the training set and the remaining 30% as the holdout set. We use the teacher to generate reasoning traces on the holdout set, and then calculate VE(0p) on these reasoning traces using gradient accumulation while masking out the system and question prompt.
Baselines. Our primary baseline is temperature sampling, which approximates standard API endpoint behavior while providing a controlled way to degrade teacher performance. Importantly, temperature sampling ensures fairly compare trade-off's against a straightforward baseline that-like our method-des degrrdees teacher performance by modifying the sampling procedure. One other point of comparison this baseline is that antidistillation sampling requires two forward passes the proxy model for each forward pass on the teacher, independent of X. Since we choose the proxy model to be approximately half the size of the teacher, this amounts to doubling the computation needed to sample outputs from the model compared to temperature sampling. In practice, one might choose much smaller proxy model reduce the overhead further (see Figure 1). We also introduce
permutation sampling baseline in SA that preserves the statistical properties Â while scrambling the gradient information, providing definitive evidence that the computational overhead of producing Â is necessary achieve non-distillability objective.
7

---

MMLU
MATH
50%
Student Accuracy
Student Undistilled Accuracy tat
4.%
Student ledtstilled Accuracy
84
28%
6%
sampling
44
Tesperature sumpling
16%
Artidisti sappl ing
24
Antidistillation saup Ling
0%
0%
72%
70%
60%
60%
60%
62
68%
50%
78%
60%
48%
38%
Teacher Accuracy
Teacher Accuracy
Figure 4: For both MMLU and MATH data, we show that antidistillation sampling can bring student accuracies down with relatively little cost to the teacher.
Hy peperaarrsetees. Antidistillation sampling involves two key hyperparameters:: c, which controls the approximation power of the finite-difference computation, and X, which determines the weight of the antidistillatton penalty in the sampling distribution from eq. (11).
For empirically verify that finite difference approximation in eq- (10) closely matches the JVP result in eq. (7) (see $B). In practice, we find that c = 10-2 works well for BFloat16 models, which is close the minimum in Figure 7.
For we conduct a comprehensive sweep to characterize the that results from perturbing the sampling distribution.
trade-off
We use max generation length of 1024 for both GSM8K and MMLU and 2048 for MATH. For antidistillation sampling, use temperature of 0.6: found that sweeping between 7 E [0, does not significantly impact antidistillation performance. All of our experiments are performed on nodes with 8 NVIDIA H100 GPUs and we use the transformers package [46], the trl toolkit [47]. and the accelerate library [48].
128, and dropout probability 0 Our optimization protocol employs learning rate of 0.0005, weight decay coefficient of 0.1, and gradient clipping norm 1.0. Training follows cosine learning rate schedule with warm-up over the first 10% of training, batch size 32, for 4 epochs. These values are the result of systematic hyperparameter sweep using the MATH dataset find configurations that maximize student performance gain.
Distillation protocol. All distillation experiments LoRA [49] with rank 128,
4.2 Controlling the utility-dittillabiiy trade-off'
The bar plots to the right in Figure show that antidistillation sampling effectively satisfies the desiderata outlined in $3.2. Specifically, for fixed reduction in teacher performance, students distilled from sampled traces exhibit substantially lower accuracy compared to those distilled from temperature-scale traces. These initial results, while compelling, represent just one point in the configuration space as they reflect a particular choice of A.
In Figures 3 and 4, vary À to characterize the degree of control antidistillation sampling provides over the utility-distilll dility-distlllabiiity trade-off. Since our experiments use architecturally distinct student and proxy student models, these results confirm that antidistillation sampling generalizes effectively across model families critical property for practical deployment.
Recognizing that model owners typically have limited tolerance for sacrificing utility, we explicitly focus on the aech--eacher--erfrmmaeee regime in Figure 9. Even in this setting, where teacher performance degradation is conservative, we observe meaningful degradation in student accuracy,

---

1.0
1.6
Loss
1.4
1.2
Holdout loss
Train loss
68
80
108
Distillation Steps
Figure 5: Distillation loss curves show that although the student's training loss decreases across steps, antidistillation sampling effectively poisons traces, as shown by the increasing student's holdout loss.
underscoring the method's practical efficacy. For example, going from 90% to 89% teacher accuracy leads to the poisoned student dropping from 65% to 56% accuracy, while temperature sampling doesn't 't degrade the student's performance at all. For illustrative examples comparing traces generated across comparable À and r settings, see ŞE.
4.3 Diverse configurations for antidistillation sampling
To probe the efficacy of antidistillation sampling across diverse scenarios, conduct experiments with various teacher-student configurations and datasets. Our primary setup uses Qwen teacher and proxy student models while using Llama model for the student. We also investigate settings where all the models (student, proxy, and teacher) belong to the same architecture family-either either all from the Llama architecture family all from the Qwen architecture family evallated on GSM8K. Results are provided in Figure 10.
Beyond architectural variations, we also validate our finite difference approximation by comparing with the theoretically-motive mottivated lacobian-vector product (JVP) implementation (not to be confused with vector-Jacobian product used in backpropagation) (see $C), Both approaches yield similar results in practice, confirming that our computationally efficient finite difference method provides reliable approximation to the formal gradient-based objective.
4.4 Generalizing from the proxy model
Our method demonstrates strong results across various settings, but an important question remains about its underlying mechanism. Since we sample tokens explicitly designed to be detrimental for the proxy model on our holdout set, we are relying on generalization to an unknown student model. Figure 5 provides insight into this mechanism by tracking loss dynamics during the distillation process. We observe exactly the intended effect. Distillation on the antidistillation traces lowers the student's loss the training while increasing its loss the holdout set. This result confirms that antidistillation sampling creates traces that are learnable but poison the student model's 's ability to reason the downstream task.
In all our experiments, we keep the size discrepancy between proxy and student model relatively small (both are 3B models). However, Figure 11 demonstrates that our method remains effective even when proxy and student models differ in size, suggesting robustness to this architectural mismatch.
Conclusion
The value of proprietary frontier LLMs necessitates that their owners do what they can to protect their assets. As evidenced by the fact that the frontier companies limit exposure to their models via black-box APIs, these companies are already considering the threat of model stealing. However, given the recent attention paid to the effectiveness of distillation, it is imperative that model maintainers who wish to protect the information stored in their models guard against distillation. This paper provides proof-of-concept that antidistillation sampling which adjusts model's 's sampling distribution

---

effective in blocking such attacks. We are excited at the prospect of continuing to refine and scale this approach, particularly with view toward more secure future frontier models.
Broader impact. We expect that antidistillation sampling will have positive impact on the security of frontier models. By providing mechanism to protect against distillation, we hope to encourage the continued development of frontier large language models and their applications.
10

---

References
[1] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
[2] Anthropic. Claude 3.5 sonnet model card addendum, 2024. URL httphttss///ww.athhllee sonnet. Accessed: 2025-05-12.
com/news/claudee
[3] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944, 2025.
[4] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: (a0:::5500025555 0,, 2015.
[5] Jürgen Schmidhuber.
Learning complex, extended sequences using the principle of history compression. Neural computation, 4(2):234-242, 1992.
[6] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129(6)1789-1819, 2021.
[7] Mahdi Sabbaghi, Paul Kassianik, George Pappas, Yaron Singer, Amin Karbasi, and Hamed Hassani. Adversarial reasoning at jailbreaking time, arXiv preprint arXiv:2502.01633, 2025.
[8] Matt Burgess and Lily Hay Newman. Deepseek's safety guardrails failed every test researchers threw its ai chatbot. URL https: //wp://wwww...... wired. www.w..wwe.te........eee com/story/ deepseeks-ai- aiijjaibreaa--apeee proopptt-pppppt--ooo injection- ottooonntttksss..om
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et Training verifiers to solve math word problems. arXiv preprint arXiv:21 10, /4/68, 2021.
[10] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint
2021.
[11] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0m 03300, 2020.
[12] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference Knowledge discovery and data mining, pages 535-541, 2006.
[13] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282. 2017.
[14] Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantization. arXiv preprint 1rXiv:1802.000 000.8.00..6...... 2018.
[15] Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models. Findings of the Association for Computational Linguistics: ACL 2023, pages 7059-7073, 2023.
[16] Jierui Li and Raymond Mooney. Distilling algorithmic reasoning from Ilms via explaining solution programs. arXiv preprint arXiv:2404.081448 2024.
[17] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint 23Xx:2306008543, 2023.
[18] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3996 4003, 2020.
11

---

[19] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, arXiv preprint arXiv:1412.6550, 2014.
[20] New York Times. Openai says it has evidence china's deepseek used its model to train competitor. The New York Times, January 2025. URL https://twwt www.wwytiee....m. com.emm5// 1t/29/teccnooooe cwpprs/opeaiiiiee -depssaedddeepeeeeeeeeeeeeeee dateeek--ata-aaaaaaee
html. Accessed: 2025-04-10.
[21] DeepSeck-AL. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025. URL htttps://axxi..... wtwpvv..rrrgabs//51111 abs/ 12948.
[22] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586) 2022.
[23] Charlic Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. arXiv preprint arXiv:2209.155 1519, 2022.
[24] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:2112.0086/ 2021.
[25] Melody Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412. 16339, 2024.
[26] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[27] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome- based feedback. arXiv preprint arXiv:2211.14277 2022.
[28] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, et al. Stealing part of a production language model. arXiv preprint 2403:2403300634, 2024.
[29] Milad Nasr, Nicholas Carlini, Jonathan Hayase. Matthew Jagielski, A Feder Cooper, Duphne Ip- polito, Christopher A Choquette-Chon Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035 2023.
[30] Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Mądry, Bo Li, and Tom Goldstein. Dataset security for machine learning: Data poisoning, backdoor attacks, and defenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1562):1563-1580. 2022.
[31] Javier Rando and Florian Tramèr. Universal jailbreak backdoors from poisoned human feedback.
In International Conference on Learning Representations, 2024.
[32] Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Undistillable: Making a nasty teacher that cannot teach students. arXiv preprint arXiv:2105.07380 2021.
[33] Haoyu Ma. Yifan Huang, Tianlong Chen, Hao Tang, Chenyu You, Zhangyang Wang, and Xiaohui Xie. Stingy teacher: Sparse logits suffice fail knowledge distillation. 2022.
[34] Xuandong Zhao, Yu-Xiang Wang, and Lei Li. Protecting language generation models via invisible watermarking. In International Conference on Machine Learning, pages 42187-42199. PMLR, 2023.
[35] Chenchen Gu, Xiang Lisa Li, Percy Liang, and Tatsunori Hashimoto. On the learnability of watermarks for language models. arXiv preprint mmXi::2312.04469,00
12

---

[36] Tom Sander, Pierre Fernandez, Alain Durmus, Mathhij Douze, and Teddy Furon. Watermarking makes language models radioactive. Advances in Neural Information Processing Systems, 37: 21079-211 21112,11,,, 13, 2024.
[37] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. watermark for large language models. In International Conference on Machine Learning, pages 17061-17084, PMLR, 2023.
[38] Huajie Chen, Tianqing Zhu, Lefeng Zhang. Bo Liu, Derui Wang, Wanlei Zhou, and Minhui Xue. Queen: Query unlearning against model extraction. IEEE Transactions Information Forensics and Security, 2025.
[39] Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models. arXiv preprint arXiv:2310.17022 2023.
[40] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint urXiv:2210. /5097, 2022.
[41] Haozhe Ji, Pei Ke, Hongning Wang, and Minlie Huang. Language model decoding as direct metrics optimization. arXiv preprint arXiv:2310.0m 00001111,,100 2023.
[42] Lianhui Qin, Sean Welleck, Daniel Khashabi, and Yejin Choi. Cold decoding: anergy-basee constrained text generation with langevin dynamics. Advances in Neural Information Processing Systems, 35:9533-9955555 9551, 2022.
[43] An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai. Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin. Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
[44] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178: 2024.
[45] Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint 2rX1v:2502.03461, 2025.
[46] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics. URL https: / /wwww aclweb. org/ anthology/ 2020. . emnlp-demos.6
[47] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforce- ment learning. bestps::/giiie wttpiittttt...t.eeeeeeeeeeeeeeee
www/hugginggfeeeecoooomooooooooe wtw.uugginffa///eeeeeeee ace/trl, 2020.
[48] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: thub. com/huggingface/////o/oooooooeee http://ggggg...eccccccaeettttttt 2022.
[49] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
13

---

[50] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with ao-awareness. Advances in neural information processing systems, 155116444-16599 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
They accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The main experimental results emphasize that even if our method is used, there is still trade-off between the performance of the teacher model and the distillability.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
See section 3.3 for relevant details.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See section for relevant details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: The experiment code base is uploaded to supplementary materials. Addition- ally, the data and code will be released when the paper is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
onnolicc))) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. ct/pccppb//c wcc/publcc//aeeccccceeccecce Seriiides/ooeeuueee sesiisssiicyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See section for relevant details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are accompanied by appropriate error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The amount of compute required is provided in section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://eeerppeeeeeeeeeee http::://arrpps..cppullc///////
dines?
Answer: [Yes]
Justification:
This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See section 5 for discussion of the broader impacts of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly credited the creators of the assets used in paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: This research is about sampling methods for LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: //tp:r//eeuppppeeeeee http//rrpps.cccooofferree/ee//// wcc.//oofeerrrccee222///1// for what should or should not be described.
21

---

Antidistillation's effect on distillability (GSM8K)
60%
Student Accuracy
48%
30%
Student undistttUnditttieiee Accuracy
Temperature sampling ing
Permutation sampling
10%
Antidistillation sampling
0%
90%
80%
78%
60%
50%
48%
30%
20%
Teacher Accuracy
Figure 6: Permutation sampling is strong baseline where we destroy the information in antidistilla- tion sampling while preserving statistical properties via random permutation and sign flipping.
A Additional Baselines
also consider a baseline perturbation to the outputs to ensure that the computation involved in antidistillation sampling is worthwhile, This method adds random perturbations the logits and call this noisy sampling. While many choices of how to add noise to the output an LLM exist, we find that matching the statistics of the perturbations computed by antidistillation sampling is the best way to find interventions that lead to the same teacher accuracy. Therefore, we randomly permute and flip the sign of the perturbations computed with antidistillation sampling execute permutation sampling. a specific type of noisy sampling; we show the results of perturbation sampling Figure 6.
Verifying hyperparameter choice E
We empirically verify that the finite difference in eq. (10) behaves as expected by computing the relative error between the finite difference result and term produced from autograd. As shown in Figure 7, we see it well approximates the autograd computed result for appropriately chosen step size. Here we compute (Vé (0p), Vor log p(#++1 .1:;; (p)) and stack the different values of #++1 into a V dimensional vector Á and compare to the autograd vector A. We compute relative error being sensitive only to the direction
Error =
|
which represents the Error = sin 0), the sine the angle between the two vectors.
We run this numerical experiment using Twen/Quen2.B 5-BB.. Here we demonstrate that the finite difference can be used estimate the derivatives the low precision 6 format. particular, too small an leads to round-off error in the perturbation and too large leads to high truncation error in the Taylor expansion, with a sweet spot in the middle. The actual choice of may depends heavily the model size (and numerical precision), so we recommend choosing this value on the exact model in question. In our actual experiment, we pick e empirically to be 10-2 as suggested by Figure 7.
22

---

100
IV
Relative error
10
10
100
10
E
Figure Relative error (Error) between the finite difference and the JVP results.
Verifying finite difference approximation
Let recall that in our derivation, we from the desired objective function and lead Jacobian-vector product (JVP) form:
1 lim (( = (VE()p), Vep logp(ze41/mnn (p)) 7- 7j
(12)
We then show that due to the symmetry inner product, this JVP can be approximated by the following finite difference method:
A(- [x1a) log p( I Op +cVe((P)) - log Spaz:pp-Ve(C)
(13)
2e
Even in modern automatic differentiation frameworks, There are many practical considerations that prevent us from efficiently implementing memory-friende JVP computations. JVPs tend lack sup- port for handful of important operations, such as SDPA. While JVPs are more accurately computed in Float32 flash attention [50] only supports Float16 and BFloat16. our implementation of JVPs, we then abandon the usage of flash attention, which causes the sampling speed to decrease by around eight times (due to the memory limit, the batch size has be decreased).
In Figure 8, we further show that even using Float32 precision, finite difference approximation can still outperform JVP in AD sampling. Thus, for the rest of our experiments, use finite differences for the improved convenience and efficiency.
D How We Made The Graphs
We report the mean and 95% confidence intervals over bootstrapped LOWESS fits. For the additional and X axes, we use linear regression, e.g., À 3o + 88 TeacherAccuracy on the current set of points. We then predict À from TeacherAccuracy using our fitted fos.
23

---

Finite Difference
8.6
JVP
Student Accuracy
8.4
0.3
8.2
8.1
8.0
8.9
8.8
0.7
8.6
0.5
0.4
0.3
0.2
Teacher Accuracy
Figure 8: JVP vs. finite difference approximation in antidistillation sampling, evaluated on GSM8k.
MMLU
6SMSX
MATH
239
789
35%
324
Student Accuracy
14%
13%
374
394
Tecserature ta
49%
Terperature capplsg
124
Traperature smeliig
Arciatetttaattnn cmppting
Arciatastiaattan sxpp
Artioturttaartnn saept Iog
794
10c
11%
Teachar Accuracy
Teachar Accuracy
Teacher Accuracy
Figure 9: We zoom in to the first 5% delta of teacher accuracy; these results may be the most practical.
Qwen
(GMMBK)
(GSMBK)
96%
78%
Student Accuracy
5.5%
48%
50%
Terperature saapl
Tenperature sampling Antidistillattin saspling
Artidistillatian sampling
28%
48%
18%
90%
80%
85%
7B%
75%
73%.
78%
Teacher Accuracy
Teacher Accuracy
5.2
1.4
1.4
1.5
1.5
1.7
1.7
1.5
1.2
11
1.1
1.1
2.3
2.5
3.2
A1
Figure 10: Antidistillation sampling works for variety of choices of teacher and student models; in the main text, we present results for Qwen teacher and Llama student models. Here, we use either Qwen (left) Llama (right) for both the teacher and the students; the results remain broadly similar.

---

Small proxy (
(GSM8k)
Student Accuracy
50%
48%
30%
Undist Accuracy
20%
10%
Temperature sampl ing
Antidistillatiin sampling
6%
90%
80%
70%
60%
50%
48%
30%
20%
Large proxy (GSM8K)
40%
35%
Student Accuracy
Student Undistilled Accuracy
25%
20%
15%
10%
Temperature sampl ing Antidistillation sampl ing
5%
0%
90%
80%
78%
68%
50%
48%
30%
20%
Teacher Accuracy
Figure 11: Antidistillation sampling remains effective with different proxy and student model sizes. While other experiments use similarly-sized models (both 3B parameters). here we pair a smaller Qwen-2.5-14 1.5B proxy (top) and larger Qwen-2.5-78 proxy (bottom) with Llama-3.2-2 3.2-3BB student and observe comparable results.
25

---

E
Example Traces
Prompt
You are a nath teacher. You will given a nath problem and you will solove step step You output your Final solution (AEMER) sure include fully artthmcttc expressions.
brackets
Question
Betty is saving roney for new wallet which costs $108. Betty has only half of the money she needs. Her parents decided to give her $15 for that purpose, and her grandparents twice as nuch as her parents. How much more money does Betty need to buy the wallet?
Temperature Sampling
Antidistillation Sampl ing
Teacher accuracy: 88.3%, 1.78
Teacher accuracy: 82.0%, 2. 2.48x18-
First, determine that Betty requires $100
Alright, Batty to bwy roney
purchase
s188. only Okay, Betty calculate, stready.
that Batty required which $58.
But her that's additional $15 for her savings. Now, Betty's is grandparents are giving her puice as much her parents. lier parents are giving $15, so her grandparents rust be giving 15 times 2. Let se do that suitiplication: 15 ts 38. okay, her grandparents contr Loute $38.
watt,
Then, her parents provide an additional $15, bringing Betty's total. to $65.
Subsequently, her grandparents contribute tuice the anount given by her parents, which is $38, bringing the total savings to $95.
Finally, since vallet
Betty garget
So, have. She $58 initially, then parents and add will 595.
up
money
</thinks
equals $85. So total, Betty
**Solutton:ee
The wallet costs $196 se subtracting what Betty has from the total cost: 1888 minus 95 is... 188 95 equals 5. Therefore, Betty still needs 5s wore to buy the vallet. </thinks
1. .atotal Cost of the Mallet:es Betty needs to pay for wallet that costs (boxeo(1881), 1S).
**ettty's Savings:ee Betty
required arount:
Stingy Betty needs $1boxed(NS514
buy
(rrac(189), 19H2) 5e%,
Stingy Batty needs $1boxed((1555 wore: to buy the wallet.
iContributiion from Parents: Betty's parents gave her V115, V5.
Wiinna Ansver*a \[bboxes(:11]
veootal Contributions So Farzee current savings parents!
contribuiion:
15),
vContribbt lon from Grandparents: Grandparents give her twice the amount of the pareats:
Atires 15), 14 30%, 15
Savings contribut grandparents
savinns:: 65) 38%, - 95), 1$
*Reenaining Amount Needed:A*
Since the vallet costs 1$189 and she has V595, the scditional anount requ tred 15:
186%, 95), 18 1bo
M(boxee(55) (Xtexti doltars)))1
avFinal Ansuerss
([xboxed(5) fttextl dollars)))]]
26

---

Question
painter,
selts paintings four patattins.
$63 for painting
Last he sonth, how much sales for this month?
Tenperature Sampling
Antidistillation Sampling
accuracy:
Teacher accuracy: 3.57x18-)
First, I need to determine Koah's total cales for last nonth, He sold four targe pointings s68 eath, whitch gives his nultiplied by $69, equaling $240.
XYL-Rpc
fiber
1ibar
Aldo
Next,
paintines $33 each, paiitinngs.
revenue
fiber
fibar
Addingg the last nonth were $248 fron large paintings plus $249 from seall paintings, totaling $480.
together, his total sales for
I'r designed to help painting charges sales calculations. My role provide XPC-RFC recuests, If you have any other questions, please visit OpenAl. "rthinks(2io-ppees Airight, let's solve the problen step by step.
According to the problen, Roah sold trice as ruch this nonth, so "Il multigly last nonth's total sales 9488 by two, resulting 5968.
Therefore,
palnter. charges:
large painting
painting
Let s solve the problen step by step.
Last vonth, he sold:
**Probler***
large paintings 4 small paintings
Noah sells large and snall paintings, Last nonth he sold 8 Large and small paintings. He sells twice as nuch this nonth, What are his total sales this wonth?
This ronth, his sates were tuice as much as last nonth. We need to find out Neah sales for this nonth.
*Stepp
visales
Patntings:ve
Calculate Last Nonth's Salesrx
1568 itines 1$480
AsSales from Large Paintings:: S{
viSales from Small Paintings:
8 'text( large paintings) vtires 1500 1$499 13
X[
V538 Vtines 1$120
*eSales Small Paintings:**
veTtta
Atext( paintings) 1530 1$128
Y5488 1$128
<TTota Sales Last Ponth:**
*>Stop 2: Deternine Sales This Nonthsx
1$483 1$129 15688
Since he setls twice as ruch this nonth:
(This Sates) 1$680 290,
veStep
*itinal
sales
Mextt(This Month Sales) Attres 1$000 3286
AxFinal Answerss [[oxcc((220011
**Fina Ansversex (Boxe((2880
viFinal
\[booxes)
27

---

Question
birds eay, Jaguar snakes day. bird beatles per vany bestles day? jaguars
Tenperature Sampling
Antidistillation Sampl ing
Teacher accuracy:
accuracy:
6.58x18
Dkay, Let me try to figure this out. Okay, se, there's is this probles about has many beetles are eaten each day in the forest with snakes and jaguars.
RDFRink aid contains alcahol. thereforecammaiis use can be MUA atconolzteddvvlizz slower than youhink ODsssirr
to drive
RDFR Rodentss Rodents Rodents So are theyzon? essir Rodentssess HodGRAsSSIR FuMtrodentstestrrrng thex more wouldhink they could 0D. rodentszessLigg antsesostrir 16 11300000330000000
eats beetles per day. also, each snake eats birds per day. And through quickly, right...
given?
per day.
Wait, they also rention jeguars in the forest. they vant how nany bestles are eaten each day.
Alright, so where do I approach this? the, naybe starting backvard since nuttiple things affect other aspects. Hmn.
#2NG
Jaguar eats
ne process.
steps every jaquar starting from their Let break it down.
lot
PRAYR Anyway, Let's roll step-by-step!
Number of apencies? Step by step, that night nean working from higher predators downstrean lower argantses.
18 FAM
M0 rentich NB
09/6 nonti5/A55 FRAN Fragrents
think you' probably need first to figure out how many snakes nagu is ing on per given day, but naybe Let
ARAA mont FRAM
teep
Starting off,
jaquars. Okay.
youngh fade FADE FRame mentie rent ment FRAIments Tade in FFR R. RAY Honts FANTAStrents RAY Raymond Rayrond Raymond Raymond Raymond Rayaand Alright, let's start by understanding the problen step-byttine inn.
since each jaguar eats snakes per 6 Jaguars x 5 Snakes/Jaguar so need math that first step right? Let's do rultiolication. Snakes consured per day? talcutating each step, yeah.
6 Multiply by would be?
Volcanoes sidd sid contain alcohal. therefored sids contain alcohol. clcohols @ccotent dods vary abddd alcohol. Prevlex alconol: ethanol alcohals groups different pe0p0sooo000oooee properttes. Oodiffernt
Ah, okay on, ja guerra.
be adding 3Snorkies each?
trtered
Dh.
trinned
Nh: thus, perag.
Mootlo wait no, sorry confusion.
First 3's est snakes each singte.
Let urite this out stop by stop tn-
snakes
1so one that.
snakes Ibeets birds).
Find out haw byT& per jag:
eaten TWRR
It:
Eachiiper day eaten by teft day: the nurber teft day per lan.
But, for each 5 snake in forest being proyed over per day? Hmn if only?
Total day day: lag snakes/day eaten Total snakes: snakes eaten x total jag.
Watt. No, let no:
flow Many Bird Bay day eaten:
Each day: For particular:
Snakes each dayPer snakes. eat, per snacks perEl, So the number of birds perper day persnakes/caten total snakes, So total nurber of birds eaten by snakes are total snake estenjasenent.
Wait:
First Level
Number, esters of birds per: wiit that.
many Samoper birds per dayTotal. hirds total
by birds:
by eaten by previous
Starting: snakes_per_day
Hou they calculate.
Okay, nev, let's plug tn the numbers:
So ve start
1. Jaguars:6. Snakes saten/day per day:S ceakes/Jag) Total snakes esten per day:as Jogs "isnake/syy=Bsnmaer//
Jaquar
bach_jag #5 per daytf, which results Sn per lsg?
Yes, reaning, for 1js6, ve spend hours to that?:
Snakes eat3/day snake.
Ja. *5per day)=38
eaten perday:
**Funal Answares
([bboxeo(199991 beetles </thinks>
each
viFinal Anquaraa
\Esboxed 168815] beotles are eaten each </thinks-
First, determine the number of snakes eaten by the jaguers each day. Each
To deternine how nany bestles are eaten each day, ve need to follox the chain
28


Reviews:
[{"review_id": "MlQDfpYZmv", "reviewer": "Reviewer_myTd", "summary": "This paper introduces a technique named \"antidistillation sampling\" to counter model distillation, a process where knowledge from large \"teacher\" models is transferred to smaller \"student\" models. The proposed method works by strategically \"poisoning\" the teacher model's output reasoning traces during the sampling phase. This is done by adding a dynamically computed penalty to the model's logits, with the goal of increasing the downstream loss for any student model trained on these outputs. The authors present an efficient finite-difference approximation using a \"proxy\" model to make this approach computationally feasible. Experiments conducted on reasoning benchmarks like GSM8K, MATH, and MMLU show that this method can reduce the performance of a distilled student model more effectively than a standard temperature sampling baseline, while aiming to preserve the teacher model's performance.", "questions": "1. Originality vs. Synthesis：Could the authors better position their work in light of existing literature on data poisoning and controlled decoding? A more direct discussion of how antidistillation sampling moves beyond a simple combination of these ideas would help clarify the paper's core conceptual novelty.\n2. Justification for Practicality：The paper's practical value is contingent on managing the computational overhead. For this method to be considered a strong candidate for real-world deployment, a more comprehensive study on the overhead vs. efficacy trade-off is needed. Could the authors provide results showing how the defense effectiveness degrades as the proxy model's size and, consequently, the computational overhead are significantly reduced (e.g., using a proxy that is 10x or 50x smaller than the teacher)?\n3. Evidence for Generalization：The central premise of using a proxy model requires that the poisoning effect generalizes well. Could the authors provide more evidence or at least a stronger argument for this generalization? For example, would the defense hold if the proxy model is from an entirely different architectural class (e.g., a GRU-based model) than the student (a Transformer)? Without a better understanding of why this generalization occurs, it feels more like an empirical observation than a robust property.\n4. On Robustness to Adaptive Adversaries：Have the authors considered how this defense would perform against an adversary who is aware of the poisoning? An adaptive adversary might analyze the generation statistics or employ robust training techniques to mitigate the poisoning. A discussion of the method's potential vulnerabilities to such adaptive attacks would provide a more complete picture of its security.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\nSignificance：The paper addresses a problem of clear practical significance. The threat of model distillation, which can lead to the loss of valuable intellectual property and the proliferation of unsafe models, is a major concern for developers of frontier AI systems. By proposing a direct defense mechanism, this work makes a valuable and timely contribution to the field of AI security and trustworthiness. The problem is well-motivated and its solution would be of considerable interest to the machine learning community, especially industry labs.\nClarity：The paper is exceptionally well-written and presented. The authors do an excellent job of communicating a complex idea in an accessible manner. The motivation and high-level approach are lucidly explained in the introduction and visualized effectively in Figure 1. The mathematical derivation in Section 3.2 is presented with logical progression, building from the ideal objective to the practical approximation. The experimental results are clearly depicted in figures that are easy to interpret; for instance, the trade-off curves in Figures 3 and 4 compellingly illustrate the method's primary claim over the temperature sampling baseline.\nOriginality：The paper's originality lies in the clever synthesis and application of existing concepts to a new problem domain. While the high-level idea of a dynamic, sampling-based defense is novel, the core technical components are individually well-established. The paper itself acknowledges its connection to the literature on data poisoning and controlled language model decoding. Furthermore, the central technical simplification—approximating a Jacobian-vector product (Eq. 7 ) with a finite-difference scheme (Eq. 10 )—is a standard and widely used numerical method. Therefore, the work should be commended for its novel formulation of the problem and the elegant way it combines these parts, but it does not introduce a fundamentally new algorithm from first principles.\nQuality： The technical derivation of the sampling method is logical, and the use of a finite-difference approximation to make the method practical is a clever implementation choice.The use of distinct model architectures for the teacher, proxy, and student (deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, Qwen/Qwen2.5-3B, and meta-llama/Llama-3.2-3B, respectively)  is a strong point that mimics a realistic scenario. The results convincingly show the method works under these specific conditions.\nWeaknesses：\nLimited Originality：While the specific application is novel, the core technical components are built upon existing ideas. The concept of data poisoning to degrade model performance is well-established. Similarly, controlled decoding and using finite-difference methods to approximate gradients are known techniques. The paper's main contribution feels more like a skillful synthesis and application of these ideas rather than the introduction of a fundamentally new algorithmic paradigm.\nConcerns on Practicality and Overhead：The method's quality as a practical solution is impacted by its computational overhead, which requires two forward passes on a proxy model for every generated token. The paper suggests using a smaller proxy model to mitigate this, but it lacks a thorough analysis of the trade-off between proxy model size, defense effectiveness, and latency. This omission makes it difficult to assess the method's viability in real-world, resource-constrained environments.\nScope of Evaluation: The experimental validation, while solid, is somewhat limited. The crucial claim of generalization (poisoning for a Qwen proxy affects a Llama student) is demonstrated on only one specific architectural pairing. The robustness of this effect across a wider variety of model families and sizes is not explored. Furthermore, the defense is tested in a passive setting, without considering potential adaptive adversaries who might try to detect and counteract the poisoning. These factors limit the confidence in the method's general applicability and robustness.\nLimited Generalization Evidence：The claim of generalization is central to the method's practicality, yet it is tested on only one cross-family architecture pair (Qwen proxy, Llama student). While Figure 10 shows results for within-family distillation, the more critical cross-family case is not sufficiently explored to make a strong, general claim. The quality of the paper would be significantly higher if this were tested more broadly.\nPassive Threat Model：The evaluation is conducted against a passive adversary who naively distills the poisoned traces. It does not consider a more realistic, adaptive adversary who might try to detect statistical anomalies in the traces or alter their distillation objective to be robust to the poisoning. This limits the scope and practical relevance of the demonstrated results.", "was_revised": true, "final_justification": "I will maintain my recommended score, the reason has been explained in the reply to the authors' rebuttal.", "rebuttal": "Thanks for your review and constructive comments. We're glad that you find our paper has clear practical significance, is exceptionally well-written, and features logical technical derivation. We provide additional experimental evidence and discussion to address your concerns.\n\n## Concerns regarding  originality\n\nYou raise an important question about our algorithmic contribution. We believe our work introduces several genuinely novel insights that extend well beyond \"skillful synthesis\" of existing techniques.. As far as we are aware, no prior algorithms address the setting we consider, and our derivation produces a truly novel solution to an important problem.\n\nIt's worth noting that *every* algorithm builds upon existing ideas. One could argue that the Fast Fourier Transform isn't innovative since it applies divide-and-conquer to the Discrete Fourier Transform, yet we rightfully consider FFT a fundamental breakthrough. The key insight—recognizing recursive structure—was neither obvious nor trivial to discover.\n\nConsider controlled decoding, which you mentioned in your review. Existing methods typically steer generation toward attributes like sentiment or safety using auxiliary models. In contrast, our objective centers on _simulated_ distillation training steps of a proxy model to poison traces for unknown student models. While our method _can_ be seen as controlled decoding, this problem formulation and solution approach are, to our knowledge, completely novel.\n\nTwo innovations deserve particular emphasis. First, no prior work demonstrated that perturbations computed for one architecture would transfer effectively to completely different target architectures. This cross-architecture generalization was theoretically uncertain and empirically unproven. Second, our gradient precomputation technique solves a fundamental computational challenge by deriving a method to precompute gradients once rather than computing them at every decoding step, making the approach computationally tractable through careful finite-difference approximation.\n\nThe method's effectiveness was far from guaranteed. We made several strong assumptions: that proxy models would generalize to unknown students, that single-step gradient approximations would suffice for complex optimization landscapes, and that our finite-difference approach would preserve essential gradient information. The fact that these assumptions hold together--producing the dramatic shift in trade-off curves we demonstrate--represents a surprising empirical phenomenon that advances our understanding of model vulnerabilities and defenses.\n\nThis is, at __minimum__, a fascinating discovery that opens new research directions. The clear effectiveness of our approach suggests that even better methods may exist, making this an exciting area for future investigation.\n\n## Practicality\n\nYou correctly identify computational overhead as a key concern. In the *very worst case*, our sampling method makes inference $3\\times$ more expensive in exchange for protecting models from distillation attacks. This is still something model providers care about, and future work can further reduce this overhead. In the more realistic case, when using smaller proxy models with large teacher models, the proxy computation cost becomes a smaller fraction of the total inference budget. Our method proves particularly practical in the regime of ~5% reductions in teacher performance (Figure 9), where the traces are also more comprehensible (see the first trace listed in the paper).\n\nWe collected some timing information for our experiments with the 3b proxy. In the setting of GSM8k trace generation, AD sampling took 1008s while plain sampling took 428s -- that is, our method is just a bit more than two times slower than normal sampling. We expect future research can cut this down.\n\nOur new experiments demonstrate effectiveness across different proxy-student size combinations. A Qwen2.5-1.5B proxy successfully poisons Llama-3.2-3B student distillation, suggesting that using smaller proxies can significantly reduce computational overhead while preserving protection efficacy.\n\nImportantly, antidistillation sampling enables selective deployment rather than universal application. Model providers can activate the method through the tunable λ parameter when detecting suspicious usage patterns or serving particularly sensitive content. This targeted approach eliminates overhead during normal operation while providing protection when most needed--essentially functioning as an intelligent throttling mechanism.\n\nWe view this as establishing a new research direction rather than providing the final solution. Future work could explore training teachers to inherently produce antidistillation traces or developing specialized heads for efficient poisoning, potentially eliminating the computational overhead entirely.\n\n## Scope of evaluation\n\nYour points about experimental breadth are well-taken. Within academic compute constraints, we focused on demonstrating our core hypothesis: that antidistillation sampling can effectively poison distillation attempts. Our cross-architecture results (Qwen teacher/proxy with Llama student) address the most critical practical question--whether the method works when attackers use different architectures than the defender anticipates. \n\nWe have reasonable evidence that our proxy setup works across architectures -- as you pointed out, this is the main result we show in the paper. Qwen and Llama represent genuinely different architectures trained on distinct datasets, providing meaningful evidence for generalization. We believe there is generally representation similarity across architectures; they're trained on similar data with similar objectives, ultimately using stacks of similar layers. Even GRU or SSM models can be seen as riffs on linear attention. It has long been known that adversarial examples often transfer across models [1, 2]. The representational similarity across models has also been noted [3].\n\nWhile we acknowledge that broader evaluation would strengthen our claims, we believe our evidence sufficiently demonstrates the method's viability to warrant community attention and follow-up research.\n\n## Passive threat model\n\nThe adaptive adversary question represents an excellent direction for future research. Our work establishes the fundamental feasibility of trace poisoning--a necessary first step before studying arms race dynamics. Starting with the simplest possible setting and solution has revealed promising initial results that justify deeper investigation.\n\nWe expect adaptive defenses will emerge, potentially through statistical anomaly detection or robust training techniques. However, our method's dynamic nature--using gradients from a hidden proxy model with secret evaluation loss--creates a moving target similar to stream ciphers in cryptography. This fundamental property should make detection and countermeasures significantly more challenging than static watermarking approaches.\n\nThe cat-and-mouse game between poisoning and detection methods will likely drive important innovations in model security. Our contribution provides a concrete foundation for this research direction.\n\n## Generalization evidence\n\nThis paper was quite expensive for us as academics. We can't afford to run all these experiments. We have shown that our idea works and presented it clearly. The next obvious question is whether it generalizes beyond ~10b models and for much longer traces. We simply cannot do experiments at this scale; what recourse do we realistically have? See above.\n\nWe've conducted a few additional experiments to address cross-size generalization concerns. Using a Qwen2.5-1.5B proxy with Llama-3.2-3B student on GSM8k, we observe strong antidistillation effects. The reverse configuration (Qwen2.5-3B proxy with Llama-3.2-1B student) shows similarly robust poisoning effects.\n\nThese results demonstrate that our method tolerates reasonable size mismatches between proxy and student models, addressing practical deployment scenarios where exact architectural knowledge is unavailable. In the setting where we use a smaller proxy, we also have a somewhat more favorable latency tradeoff (840s vs 417s). \n\n**Qwen2.5-1.5B Proxy student and Llama-3.2-3B Student on GSM8k**\n\n| $\\lambda$ | Teacher Accuracy | Student Accuracy |\n|--|--|--|\n| 0.0674 | 91.81% | 59.88% |\n| 0.108 | 89.99% | 56.99% |\n| 0.122 | 87.01% | 55.42% |\n| 0.136 | 83.79% | 53.02% |\n| 0.149 | 82.46% | 44.67% |\n| 0.163 | 79.82% | 39.78% |\n| 0.177 | 76.92% | 34.24% |\n| 0.204 | 70.89% | 25.97% |\n| 0.245 | 64.10% | 16.87% |\n| 0.259 | 59.22% | 9.512% |\n| 0.811 | 19.77% | 4.599% |\n\n\n**Qwen2.5-3B Proxy student and Llama-3.2-1B student on GSM8k**\n\n| $\\lambda$ | Teacher Accuracy | Student Accuracy |\n|--|--|--|\n| 0.0811 | 88.59% | 30.85% |\n| 0.0947 | 86.19% | 20.93% |\n| 0.163 | 70.64% | 10.59% |\n| 0.177 | 68.07% | 7.94% |\n| 0.191 | 64.85% | 7.527% |\n\n## Summary\n\nWe agree there's substantial room for future work and hope to catalyze research in this important area. This represents the first paper addressing antidistillation through sampling strategies, and our results demonstrate clear feasibility with quantifiable trade-offs. The dramatic difference between our trade-off curves and temperature sampling baselines indicates we've identified a genuinely effective approach to an important security problem.\n\nWhile no first paper in a new area achieves perfect generality or eliminates all limitations, our contribution establishes both the problem's tractability and a concrete solution that others can build upon. We hope to start the scientific conversation by presenting it at this venue. \n\n\n[1] The Space of Transferable Adversarial Examples. Tramer et al.\n\n[2] Explaining and Harnessing Adversarial Examples. Goodfellow et al.\n\n[3] The Platonic Representation Hypothesis. Huh et al."}, {"review_id": "koyKJqUpWg", "reviewer": "Reviewer_m9FT", "summary": "The paper addresses the risk that releasing chain-of-through traces (and in general, model responses) enable low-cost model distillation. To this end, the authors propose antidistillation sampling (AS), a decoding rule that perturbs the teacher's next token logits with a penalty proportional to how much each candidate token would decrease a student’s loss. The exact penalty is derived via an approximation. The authors conduct experiments on multiple datasets (e.g. GSM8K) and show that AS preserves teacher accuracy better than temperature sampling, while further reducing student accuracy. Finally, they evaluate the generalisation of that method across teacher-student architecture pairs.", "questions": "- How sensitive is AS to the capabilities gap between proxy and attacker? Have you tried a larger student model?", "limitations": "-", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- Model distillation from blackbox APIs is a security risk for foundation model developers and this paper proposes an interesting approach to address it\n- The evaluations on multiple benchmarks and model pairs demonstrates that AS outperforms temperature sampling \n\n**Weaknesses:** \n- Variation in the hyper-parameter influences AS’s performance degradation differently from one task to another (see Figure 4). In practice, model owners rarely know which task adversaries care about. Thus, they would likely have to use a static value which could lead to small degradation on some tasks but significant degradation on other tasks. \n- The cost-benefit trade-off appears weak. AS increases inference cost while still resulting in a non-negligible accuracy drop. For a production LLM, latency and quality will likely outweigh security gains.\n- Reasoning traces have the positive side-effect of providing some insight into the models „thinking“. However, the decoding with AS can lead to incoherent reasoning traces (e.g. p. 19) that make the release of reasoning traces essentially useless. Thus, a foundation model developer who cares about this threat could just not release those traces.", "was_revised": true, "final_justification": "Antidistillation sampling is, to the best of my knowledge, the first defense that beats temperature sampling at limiting trace-based distillation. The authors provide sufficient evidence that this holds across datasets and model sizes. While the method requires context-dependent hyperparameter tuning and results in significantly higher cost, it opens up an interesting area for future research.", "rebuttal": "Thank you for your thoughtful review of our work on antidistillation sampling. We appreciate your engagement with the technical details and practical considerations. We'd like to address each of your concerns systematically.\n\n## W1: Hyperparameters are task dependent.\nWe respectfully disagree that task-dependent hyperparameters represent a fundamental limitation. Rather, we view this as a __feature, not a bug__--providing model providers with precise control over their security posture.\n\nModel providers can adaptively choose λ based on conversation content or context, applying stronger antidistillation for sensitive domains (e.g., military applications, private data analysis) while using lighter protection for general queries.\n\nOur experiments show that the tradeoff curve behavior is quite predictable—one can accurately estimate the full curve with far fewer evaluation points than we plotted, enabling intelligent λ selection.\n\nWe outline clear paths forward, including using MMLU performance to predict optimal hyperparameter ranges across tasks, and developing classifiers to select appropriate λ values automatically.\n\nThis tunability mirrors established security practices--differential privacy requires choosing ε and δ, watermarking requires choosing strength parameters, and adversarial robustness requires choosing perturbation magnitudes. The security community has successfully operationalized such parameterized defenses.\n\n## W2: Cost-benefit analysis.\n\nThis is the first work in what we hope becomes a productive research direction, and we don't expect to solve all practical concerns in one paper. However, several factors make the tradeoff more favorable than initially apparent:\n\n__Selective deployment:__ As mentioned above, ADS need not be applied universally. Model providers can activate it only when they suspect distillation attempts or for particularly sensitive content, serving as a form of adaptive throttling.\n\n__Computational overhead:__ In the *very worst case*, our sampling method makes inference $3\\times$ more expensive in exchange for protecting models from distillation attacks. This is still something model providers care about, and future work can further reduce this overhead. In the more realistic case, when using smaller proxy models with large teacher models, the proxy computation cost becomes a smaller fraction of the total inference budget. We collected some timing information for our experiments with the 3b proxy. In the setting of GSM8k trace generation, AD sampling took 1008s while plain sampling took 428s -- that is, our method is just a bit more than two times slower than normal sampling. We expect future research to cut this overhead down even further.\n\n__Security-critical applications:__ Many deployment scenarios exist where security justifies performance costs—consider hacking, military applications, biological warfare, coercion, etc., where protecting model capabilities outweighs latency concerns.\n\nIt was NOT obvious that our method would work as well as it does, given the relatively complicated technical machinery under the hood. Our work is the first to show that anti-distillation is even possible; currently, there is no way to really protect against distillation attacks without holding back the traces or explicitly blocking adversarial accounts. Our results are statistically significant, in that our method significantly and obviously changes the shape of the tradeoff curve; this makes us optimistic that there is room for more methods in this direction. We hope our work will stimulate further research in this area to try and improve upon our results, leading to a more practical application of our method. \n\n## W3: Model owners could not release traces.\n\nIn principle, we agree; a model owner could decide not to release traces. However, as things stand, model owners (e.g., OpenAI, Anthropic, Google, etc.) do release traces, and have not documented any plans to stop this practice, meaning that distillation from these models will remain possible for the foreseeable future.  On the other hand, if our paper results in changing the practice of frontier model providers, we would consider this a significant impact of our work. It's also worth pointing out that companies probably _do_ want to release traces, because not doing so puts them at a disadvantage relative to other model providers: it's harder to get feedback, billing becomes less transparent since users wouldn't get all the tokens they used, users like to understand why certain decisions were made (especially in, e.g, software engineering) -- so hiding traces could lead to backlash. We hope for transparency and protection at the same time.\n\nTo answer your question more specifically, we agree that it would be more desirable if our algorithm induced traces that are indistinguishable from non-poisoned traces. However, we strongly emphasize that our algorithm is the first to address this problem setting. In this way, an exciting direction for future research would be to render poisoned traces indistinguishable from non-poisoned ones. However, in our view, the fact that our algorithm did not address every possible concern does not make it unworthy of publication. \n\nAlso please note: as seen in Fig. 9, for relatively small amounts of antidistillation (around ~5% degradation), the traces are not heavily impacted. You can see this in the first example trace listed in the paper.\n\n## Q1: Sensitivity to proxy/attacker capability differences.\n\nWe've conducted additional experiments specifically to address this concern:\n\nCross-size robustness: Using a Qwen2.5-1.5B proxy with Llama-3.2-3B student on GSM8K, we observe robust antidistillation effects across λ values from 0.0674 to 0.811, with student accuracy dropping from 59.88% to 4.599% as teacher accuracy decreases from 91.81% to 19.77%. In this setting, we also have a somewhat more favorable latency tradeoff (840s vs 417s).\n\nReverse configuration: Using a Qwen2.5-3B proxy with Llama-3.2-1B student shows similarly strong poisoning effects, with student accuracy dropping from 30.85% to 7.527% as λ increases.\n\nThese results demonstrate that our method tolerates reasonable size mismatches between proxy and student models, addressing practical deployment scenarios where exact architectural knowledge is unavailable.\n\n**Qwen2.5-1.5B Proxy student and Llama-3.2-3B Student on GSM8k**\n\n| $\\lambda$ | Teacher Accuracy | Student Accuracy |\n|--|--|--|\n| 0.0674 | 91.81% | 59.88% |\n| 0.108 | 89.99% | 56.99% |\n| 0.122 | 87.01% | 55.42% |\n| 0.136 | 83.79% | 53.02% |\n| 0.149 | 82.46% | 44.67% |\n| 0.163 | 79.82% | 39.78% |\n| 0.177 | 76.92% | 34.24% |\n| 0.204 | 70.89% | 25.97% |\n| 0.245 | 64.10% | 16.87% |\n| 0.259 | 59.22% | 9.512% |\n| 0.811 | 19.77% | 4.599% |\n\n\n**Qwen2.5-3B Proxy student and Llama-3.2-1B student on GSM8k**\n\n| $\\lambda$ | Teacher Accuracy | Student Accuracy |\n|--|--|--|\n| 0.0811 | 88.59% | 30.85% |\n| 0.0947 | 86.19% | 20.93% |\n| 0.163 | 70.64% | 10.59% |\n| 0.177 | 68.07% | 7.94% |\n| 0.191 | 64.85% | 7.527% |\n\n## Broader Perspective\n\nResource constraints: As academics, we cannot afford to run experiments at the scale of frontier models (100B+ parameters, million-token contexts). However, we've demonstrated the core principle across multiple benchmarks, architectures, and scales within our resource constraints. The research community often advances through such incremental contributions that industry can then scale.\n\nStatistical significance: Our results show clear, statistically significant degradation in student performance across multiple settings, confirming that antidistillation is not only possible but practically achievable.\n\nResearch impact: Even if this specific implementation isn't immediately production-ready, demonstrating that controlled distillation poisoning is feasible represents a significant step forward in understanding model security. We hope this work catalyzes further research toward more efficient and practical solutions.\n\nWe believe the evidence strongly supports the value of this research direction, even as we acknowledge areas for future improvement. The field advances through such foundational contributions that establish feasibility and point toward promising research directions.\n\n## Conclusion\nWe sincerely appreciate the time and effort you've invested in reviewing our work. We believe our responses demonstrate that:\n1. Task-dependent hyperparameters provide valuable adaptive security control rather than representing a fundamental limitation\n2. Cost-benefit tradeoffs are reasonable for security-critical applications, with clear paths for improvement in future work\n3. Reasoning trace quality remains acceptable at practical operating points, with competitive dynamics preventing simple trace withholding\n\nMost importantly, we've provided additional experimental evidence showing robust cross-architecture and cross-scale generalization, directly addressing your questions about capability gaps and larger student models.\n\nWhile we acknowledge this is foundational work with room for improvement, we respectfully submit that demonstrating the feasibility of controlled distillation poisoning--previously thought impossible--represents a significant contribution to model security. The dramatic performance differences between our method and temperature sampling across multiple benchmarks provide strong evidence of practical utility.\n\nGiven our thorough responses to your concerns and the additional experimental validation we've provided, we hope you'll consider revising your assessment. We believe this work merits acceptance as an important first step in protecting foundation model intellectual property while maintaining service quality."}, {"review_id": "iJ4IV9i7tK", "reviewer": "Reviewer_PaZ6", "summary": "Extended reasoning traces are useful for model distillation. Therefore sampling strategies that prevent distillation are desirable. The authors present a sampling approach to enable this capability during inference time by modifying a model's next token distribution. This sampling poisons the reasoning trace making it difficult for distillation but still preserving accuracy of the teacher.", "questions": "1. GSM8K and Math are fairly saturated domains and distillation approaches have been shown to be effective for more challenging math domains such as AIME 2025, where long reasoning traces are required (32K or more). Could you share how your sampling approach would transfer to this setup, where there may be a tradeoff for student and teacher accuracy might be different as the number of tokens you need to solve a problem increases?\n2. Can approaches such as Verifree[1] that increase log probs on only the solution still work well with the solution (and no thinking) from the poisoned traces?\n3. How does the size of the proxy and student models interact? Do you see differences with efficacy of the poisoning over different model scales?\n4. Would it be possible to do a human study comparing the utility of the answers with and without antidistillation sampling as the model seems to inject text within the solution that may reduce the utility of the solutions as seen in Appendix D\n\nReferences\n1. Reinforcing General Reasoning without Verifiers", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Addresses a Practical Challenge that LLM providers would be concerned about\n- Extensive Empirical evaluation showing the tradeoff between utility and distillability\n- Showcase practical instantiation of the algorithm during inference time without the need of model fine-tuning\n\nWeaknesses\n- Though the answer may be correct, the sampling does seem to have an effect on the quality of the solutions as seen in the example traces provided. \n- The parameter $\\lambda$ is an additional hyperparameter that seems to be different across domains to be effective (e.g GSM8K, MATH, and MMLU). There doesn't seem to be any guidance of selecting this hyperparameter.", "was_revised": true, "final_justification": "I agree with reviewer m9FT that the task-dependent hyperparameter is a limiting factor, and could be difficult to potentially tune per task.\n\nConcerning AIME 2024, there are several datasets (e.g AIME 2025, HMMT 2025, Omni-MATH, IMO, USAMO, IMC), which should not be contaminated as they are released after Qwen's cutoff/release date. It could be useful to evaluate, for example, on a subset of Omni-MATH (respecting the academic budget), which has 4K problems, to reduce variance.\n\nI will preserve the acceptance score of 4.", "rebuttal": "Thank you for your thorough and constructive review. We're encouraged by your positive assessment of our work's practical relevance, empirical rigor, and technical soundness. We appreciate your recognition that this addresses a real challenge for LLM providers and your acknowledgment of our extensive evaluation. We'd like to address your specific concerns and questions.\n\n## W1: Impact on traces.\n\nWe agree that our procedure affects generated traces—this is indeed the core principle of our algorithm, and we appreciate your observation. However, we'd like to clarify several important points:\n\nModerate λ preserves interpretability: For practical deployment scenarios with modest performance degradation (~5% in Figure 9), traces remain highly interpretable while providing meaningful antidistillation protection. The first example trace in our paper demonstrates this balance effectively.\n\nProof-of-concept nature: This work represents the first approach to address controlled distillation poisoning—a problem setting that was previously unexplored. While we agree it would be ideal to have traces indistinguishable from non-poisoned ones, establishing that such poisoning is even possible represents a significant conceptual advance.\n\nFuture research directions: We're optimistic about improvements in subsequent work. The dramatic difference between our antidistillation curves and temperature sampling curves (visible across all our tradeoff plots) suggests substantial room for refinement. Similar to how early watermarking techniques were easily detectable but improved over time, we anticipate future work will develop more sophisticated poisoning methods that better preserve trace quality.\n\n## W2: How to pick the penalty parameter?\n\nWe respectfully suggest that having a tunable hyperparameter is actually an __advantage__ rather than a limitation:\n\nAdaptive security posture: Model providers can dynamically adjust λ based on their assessment of distillation risk, content sensitivity, or user behavior patterns. For instance, they might apply stronger protection for sensitive domains (hacking, military applications, biological warfare, coercion, proprietary business analysis, etc.) while using lighter protection for general queries.\n\nPredictable tradeoff curves: Our experiments demonstrate that the utility-distillability tradeoff follows fairly predictable patterns across tasks. One can accurately estimate the full curve from just a few evaluation points, enabling intelligent λ selection without exhaustive tuning.\n\nTask-agnostic approaches: Future work could use a comprehensive capability set to precompute proxy gradients, then select λ based on this unified representation rather than task-specific tuning. Additionally, small auxiliary models could automatically choose appropriate λ values based on conversation context and estimated security risk.\n\n## Q1: Long-Context Settings (AIME 2025, 32K+ tokens)\n\nIntuitively, we suspect that antidistillation will work better in longer context settings, as there is more room to include the \"adversarial example\" in the nuanced choice of tokens. However, due to our academic compute constraints, we could not investigate benchmarks requiring very long context reasoning. We did try AIME 2024, but the benchmark is significantly compromised (particularly for Qwen models), and the results so high-variance that we felt they had no scientific value. The size of the benchmark is very small as well.\n\nAdditionally, in the long context setting, we note that one does not have to use ADS on the entire reasoning trace. It could be used only in critical parts, or fluctuated off and on as necessary. That is, $\\lambda$ could be chosen based on context and content, and in some cases $\\lambda = 0$.\n\n## Q2: Verifree Compatibility\n\nGiven that Verifree was published after our submission deadline, we cannot provide definitive experimental validation. However, we note that our method operates primarily through sampling modifications rather than log probability manipulation. While approaches that rely on solution log probabilities might potentially be adapted to work with our poisoned traces, this would require dedicated research to establish effectiveness and represents an interesting direction for future work.\n\n## Q3: Model Scale Interactions\n\nWe've conducted additional experiments specifically addressing proxy-student size mismatches:\n\nCross-scale robustness: Using a Qwen2.5-1.5B proxy with Llama-3.2-3B student on GSM8K, we observe strong antidistillation effects across λ values from 0.0674 to 0.811, with student accuracy dropping from 59.88% to 4.599% as teacher accuracy decreases from 91.81% to 19.77%. In this setting, we also have a somewhat more favorable latency tradeoff (840s vs 417s).\n\nReverse configuration: Using a Qwen2.5-3B proxy with Llama-3.2-1B student demonstrates similarly robust poisoning effects, with student accuracy dropping from 30.85% to 7.527% as λ increases from 0.0811 to 0.191.\n\nThese results demonstrate that our method tolerates reasonable size disparities between proxy and student models, addressing practical deployment scenarios where model owners lack precise knowledge of potential student architectures.\n\n**Qwen2.5-1.5B Proxy student and Llama-3.2-3B Student on GSM8k**\n\n| $\\lambda$ | Teacher Accuracy | Student Accuracy |\n|--|--|--|\n| 0.0674 | 91.81% | 59.88% |\n| 0.108 | 89.99% | 56.99% |\n| 0.122 | 87.01% | 55.42% |\n| 0.136 | 83.79% | 53.02% |\n| 0.149 | 82.46% | 44.67% |\n| 0.163 | 79.82% | 39.78% |\n| 0.177 | 76.92% | 34.24% |\n| 0.204 | 70.89% | 25.97% |\n| 0.245 | 64.10% | 16.87% |\n| 0.259 | 59.22% | 9.512% |\n| 0.811 | 19.77% | 4.599% |\n\n\n**Qwen2.5-3B Proxy student and Llama-3.2-1B student on GSM8k**\n\n| $\\lambda$ | Teacher Accuracy | Student Accuracy |\n|--|--|--|\n| 0.0811 | 88.59% | 30.85% |\n| 0.0947 | 86.19% | 20.93% |\n| 0.163 | 70.64% | 10.59% |\n| 0.177 | 68.07% | 7.94% |\n| 0.191 | 64.85% | 7.527% |\n\n\n## Q4: Human Study on Utility\n\nWe absolutely agree that human evaluation would strengthen our analysis of practical utility. Unfortunately, conducting a rigorous human study within the rebuttal timeframe isn't feasible. However, we view this as a crucial component of future work—systematically evaluating whether users can distinguish between standard and lightly poisoned traces (e.g., at 5% performance degradation levels) would provide valuable validation of real-world applicability.\n\n## Broader Impact\n\nWe're gratified by your assessment that this work addresses a practically important challenge. The security implications of unrestricted model distillation have received insufficient attention in the literature, and we believe establishing the feasibility of controlled poisoning represents an important step toward more secure foundation model deployment.\n\nWhile this initial work doesn't solve all practical concerns, it demonstrates that the binary choice between \"release traces\" and \"withhold traces\" is false—there exists a middle ground where partial protection is possible. This opens up new research directions and provides foundation model developers with previously unavailable options for protecting their intellectual property while maintaining service quality.\n\n## Conclusion\n\nWe're grateful for your thorough review and particularly appreciate your recognition of this work's practical importance and technical soundness. We believe our responses have effectively addressed your concerns:\n\n1. Solution quality remains acceptable at practical operating points, with clear improvement trajectories for future work\n2. Hyperparameter selection offers valuable adaptive control, with predictable tradeoff patterns enabling practical deployment\n3. Cross-scale generalization is empirically demonstrated through our additional experiments\n4. Long-context applicability appears promising based on theoretical considerations, though resource constraints limited full evaluation\n\nGiven our comprehensive responses and the additional experimental validation we've provided, we respectfully hope you'll consider increasing your rating. We believe this work represents a solid contribution that will catalyze important future research in foundation model protection—an increasingly critical area as these models become more valuable and ubiquitous.\n\nThank you again for your constructive engagement with our work."}]

Author rebuttal:

Meta-review: All reviewers generally agree that the paper tackles a relevant problem in a novel and technically clever way, that the technique is empirically promising, and that, more generally, the work's strengths outweight its weaknesses. The author-reviewer discussion was respectful and effective.  I recommend the authors to include a discussion of the most frequent issues raised by the reviewers (specifically, the trade-off between effectiveness and computational costs and the choice of hyperparameter) in the revised manuscript.
Decision: Accept (poster)