Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
Authors: Faisal Hamman
Pasan Dissanayake
Yanjun Fu
Sanghamitra Dutta
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge Distillation
Counterfactual Explanations
Few-shot Learning
Data Efficiency
Primary area: deep_learning
TL;DR: This paper introduces a few-shot task-aware knowledge distillation method that leverages counterfactual explanations to improve model performance with fewer data samples.

Abstract: Knowledge distillation is a promising approach to transfer capabilities from complex teacher models to smaller, resource-efficient student models that can be deployed easily, particularly in task-aware scenarios. However, existing methods of task-aware distillation typically require substantial quantities of data which may be unavailable or expensive to obtain in many practical scenarios. In this paper, we address this challenge by introducing a novel strategy called **Co**unterfactual-explanation-infused **D**istillation CoD for *few-shot task-aware knowledge distillation by systematically infusing counterfactual explanations*. Counterfactual explanations (CFEs) refer to inputs that can flip the output prediction of the teacher model with minimum perturbation. Our strategy CoD leverages these CFEs to precisely map the teacher's decision boundary with significantly fewer samples. We provide theoretical guarantees for motivating the role of CFEs in distillation, from both statistical and geometric perspectives. We mathematically show that CFEs can improve parameter estimation by providing more informative examples near the teacher’s decision boundary. We also derive geometric insights on how CFEs effectively act as knowledge probes, helping the students mimic the teacher's decision boundaries more effectively than standard data. We perform experiments across various datasets and LLMs to show that CoD outperforms standard distillation approaches in few-shot regimes (as low as 8 - 512 samples). Notably, CoD only uses half of the original samples used by the baselines, paired with their corresponding CFEs and still improves performance.

Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
Faisal Hamman
Pasan Dissanayake University of Maryland, College Park (fhamman, pasand, yanjunfu, sanghamd) edu
Yanjun Fu Sanghamitra Dutta
Abstract
Knowledge distillation a promising approach to transfer capabilities from com- plex teacher models to smaller, resource-cfficient student models that can be deployed easily, particularly in task-aware scenarios. However, existing methods of task-aware distillation typically require substantial quantities of data which may be unavailable or expensive to obtain in many practical scenarios. In this paper, address this challenge by introducing novel strategy called Counterfactual- explanation-infused Distillation (CoD) for few-shot task-aware knowledge dis- tillation by systematically infusing counterfactual explanations. Counterfactual explanations (CFEs) refer to inputs that can flip the output prediction of the teacher model with minimum perturbation. Our strategy CoD leverages these CFEs precisely map the teacher's ss decision boundary with significantly fewer samples. We provide theoretical guarantees for motivating the role of CFEs in distillation, from both statistical and geometric perspectives. We mathematically show that CFEs can improve parameter estimation by providing more informative examples near the teacher's s decision boundary. We also derive geometric insights on how CFES effectively act as knowledge probes, helping the students mimie the teacher's decision boundaries more effectively than standard data. We perform experiments across various datasets and I.I.Ms to show that CoD outperforms standard dis- tillation approaches in few-shot regimes (as low as 8 512 samples). Notably, CoD only uses half of the original samples used by the baselines, paired with their corresponding CFEs and still improves performance. Our code is available at https: //gg://gitttttt cww/ Faai/Faiaaa sal HaaaaaaCmooooaaaaa
Introduction
Large Language Models (LLMs) have demonstrated state-of-the-art performance across a broad spectrum of tasks [1-3]. However, as the size of L.MSs grow, so does the associated computational burden, making them difficult to deploy in resource-constraingg
environments, c.g., mobile phones, edge devices, and embedded systems [4]. The challenge, therefore, lies in making large models more efficient and accessible without sacrificing performance. To this end, knowledge distillation (KD) (initially proposed in [5]; see surveys [6-8]) has emerged as a powerful technique for model compression, enabling smaller student models to mimic the performance of a larger teacher model. In the context of LLMs, KD plays central role in transferring the broad capabilities such natural language understanding [9], reasoning [10], instruction following [11] onto smaller models.
While LLMs are trained for a broad range of tasks, we may often want smaller, task-specific language model when full task coverage is not required, particularly in resource-constrained environments. To support this, task-aware knowledge distillation [12, 13] has been proposed to selectively transfer task-relevant knowledge from teacher to student models. While effective, these methods typically assume access to large datasets [14]. However, in many real- world applications, the amount of data
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

available is often limited [15-18], and obtaining high-que quality human- annotated data is expensive. Despite advances on algorithmic strategies for task- awwre KD in LLMs [14], the problem of data selection for KD has received limited interest, particularly in few-shot settings. In this work, we study few-shot and task-aware knowledge distillation for L.MMs, where student models are distilled from teacher models using a very small number of samples labeled for task (also called shots). Few-shot task-aware distillation remains underexplored for LLMs. In classical ML, few-shot training has poor generalization [19], and thus causes ineffective distillation due to insufficient task coverage [20, 21J. However, few-shot distillation holds potential for LLMs because they are pretrained on a large corpora, also drawing inspiration from the prior success of few-shot learning [22].
In this work, we propose a few-shot twske knowledge distillation strategy by systematically inte- grating a type of posthoc explainability technique called counterfactual explanations (CFEs) [23, 24]. CFEs are inputs that can flip the output prediction of model with minimum perturbations. We find that CFEs can act as knowledge probes, helping the students mimic the teacher's s decision boundaries more effectively than standard data. Our work bridges explainability and model compression by turning explanations into actionable training signals, guiding the student into learning the teacher's decision-making process more effectively. This results in more faithful knowledge transfer even with very limited data. Our contributions can be summarized as follows:
A counterfactual explanation-basee -baed strategy for few-shot distillation. We propose a novel framework CoD, short for edge distillation under few-shot regimes. By enriching the few-shot training set with CFEs, we improve the student's 's ability to mimic the fine-grained details of the teacher's decision boundary with fewer labeled examples. We validate this intuition through a synthetic experiment on the 2D moons dataset, showing that CFE-infused distillation better replicates the teacher's 's decision surface compared to using standard few-shot samples (see Fig. && Fig. 3).
Distillation, for task-aware knowl-
Theoretical guarantees motivating the role of CFEs in distillation. We provide theoretical guar- antees that serve as motivation for our approach, from both statistical and geometric perspectives. First, in logistic regression setting, we show that CFES improve parameter estimation by maximiz- ing the Fisher Information (see Def. 2 & Thm. 1). Our proof specifically leverages the fact that the CFEs lie quite close to the decision boundary to show that they reduce the expected estimation error of the student model compared to standard distillation. Next, moving beyond statistical guarantees and linear models, we also provide a geometric analysis for non-linear models, establishing that if student matches the teacher's predictions on the original data and their counterfactual pairs, then their decision boundaries will remain close: this is quantified by a provably small Hausdorff' distance, a formal measure of distance between two subsets within space (see Def. 3 & Thm. 2).
Empirical validation. We evaluate CoD on six benchmark datasets using DeBERTa- v3 [25] and Qwen2.5 5 [2] model families. We compare against strong baselines including standard Knowledge Distillation (KD) [5], Liyer-wise Distillation (LWD) [13], and Task-aware layer-wise Distillation (TED) [12] under various few-shot settings (k: = 8, 16, 32, 64, 128, and 512). Our results demonstrate that CoD consistently outperforms baselines in few-shot regimes, with significant improvements in extremely data-scarce scenarios (k- < 64). Notably, CoD only uses half of the original labeled samples used by the baselines (i.e., k/2 original infused with their corresponding k/2 CFEs, leading to k shots), and still gives improved performance. For instance, with k 8 samples on IMDB dataset, LWD CoD improves over standard LWD by more than 10 points.
Related Works: Knowledge distillation has emerged as a powerful framework for model compres- sion [5]. While early works focused on transferring soft labels via output logits [26], subsequent advances explored richer supervision signals such as intermediate feature alignment [13, 27-29).
As LLMs grow in size and inference cost [30, 31], distillation has become increasingly important for transferring capabilities into smaller models [14, 6-8]. More recently, task- knowledge distillation for LLMs has gained traction, aiming to selectively distill knowledge relevant specific downstream task [12, 32]. Despite these algorithmic innovations [12], there has been relatively little focus on data selection for distillation. particularly in few-shot settings. Most prior works assume ample training data, leaving few-shot knowledge distillation largely underexplored. While some works [20, 16-18] have studied distillation classical ML under low-data regimes, they do not address the challenges specific to distilling LLMs. In this work, we establish the paradigm of sho-shot distillation in I.LMs by integrating explainable data selection. Our work is broadly aligned with the spirit of data-efficient ML, which aims to improve performance under limited supervision [33-36].
2

---

Original Teacher Boundary
Student Boundary
Student Boundary 2
CoD Student Boundary
(a) Full Data
(b) Few Shot Data (Set
(c) Few Shot Data (Set (d) Half Sut & CFEs
Figure 1: Intuition behind our approach: (a) Teacher trained on the full dataset with true decision boundary. (b-c) With few-shot supervision, many classifiers can fit the sparse points; the resulting student boundaries (dashed lines) can vary and do not always align with the teacher's boundary (unfaithful distillation). (d) Pairing each point with its CFE (x, linked to originals) during distillation makes the student match the teacher's soft predictions these points. CFEs surface, producing more faithful distillation even under few-shot budgets.
boundary-nea pegs that clamp the student to the teacher's decision
Counterfactual explanations (CFES) [23, 24, 37-43] have been widely studied in classical MI, particularly for algorithmic recourse in high-stakes applications such as finance, healthcare, and law. An interesting work [44] uses CFEs for model reconstruction by deriving theoretical relationships between reconstruction error and the number of counterfactual queries using polytope theory. In the natural language domain, some methods have been proposed generate semantically valid CFEs using either token-level perturbations [45] controlled generation with language models [46, 47, 45], but they have not been integrated for knowledge distillation. Another line of work is counterfactual reasoning in causal inference [48], where the goal is to estimate the effect of interventions under a structural causal model, which different from our objectives. Counterfactual data have been used to address the issue of spurious patterns [49-51], improve generalization [52, 53), and enhance performance on oft-o--distribution data [54, 55]. In contrast, our work studies the role of CFE in few-shot task-aware distillation, specifically aligning the student with the teacher's outputs to mimic the teacher's ' decision boundary more effectively.
2 Preliminaries
LLMs are highly effective for natural language processing. Built upon the transformer architec- ture [56], LLMs consist of multiple stacked layers, each containing multi-head self self-attention mecha- nism followed by position-wise feed-forward neural network. Let g(:; denote a transformer- model parameterized by 0. The model takes an input sequence
X where X' the input space. The model output is probability distribution over the vocabulary space, but for task-aware settings such as sentiment analysis, is a probability distribution over C' class labels, i.e., g X' - [0, 1) The loss function is defined as: L()) Ex-x |x (xx:(x 0))], where é denotes the task-specific loss, such as cross-entropy for classification tasks or causal language modeling loss for generative models.
Knowledge Distillation (KD). KD is a technique that transfers knowledge from a large, pre- trained teacher model a smaller, student model [57]. Let go (-: 6,) be the teacher model with parameters 9, and (- 0,) be the student model with parameters 0.. The teacher model (:; 8x) provides soft labels to assist in training the student model g= 6z). The student is trained using loss function that is a combination of the task-specific loss and the distillation loss as follows: mino, L(O,) + +ooo o&L (f,, 0.). Here, C(O.) is the task-specific loss, e.g.. the cross-ent cross-entropy loss between the student's outputs and true- labels, and EKD(O,, 0,) = Ex-x [d(or (x; 0,), (x 0.)) is the distillation loss which captures the distance between the outputs of the teacher and student. Typically, the distance is computed using the
Kullback-Libllr (KL) divergence, i.c., KL (x; 0,) || (x; 0.)) = >
Di (c) (xx; 0z) log 95 (x.0.))
g. (xc00.
where the superscript (c) is for the assigned probability for class c by each model.
Layer-Wise Wise Distillation (LWD). In large transformer-based models, the teacher's 's outputs may not fully capture the knowledge embedded in intermediate layers. Beyond matching final outputs, one can also align the intermediate features of the teacher and student [13]. At a few selected layers, the teacher's 's hidden activations In and the student's activations 'ú, (optionally projected into the same dimension) computed and their difference is also penalized using seea--squaed-eerrrr loss [13].
3

---

Counterfactual Generation
Counterfactual
Knowledge Distillation
Output
Countertactual (().
Logite (+()
Teacher LLM
Leyar-wiss Alighment
Soft
Alighment
MSE toss
Taachar
0(( Logits
1D
LLM
Stucient
Faw Shot
Few Shot
Student Output
Logits
Figure 2: Overview of framework: Counterfactual Explanation-Infusee Distillation (CoD).
The student is trained using loss as follows:
min C(0.) + a LxD(Or,0.) + (LwD(Or,,))
(1)
Here, (L,, 0&) is the additional layer-wise alignment term added alongside the sask-specific loss and distillation loss, e.g., Ex~x(Eiez || My h1ll where {h² h² ]ez are the teacher and student activations for given input x over a set I of layers, and a, balance the three objectives.
Counterfactual Explanations (CFEs). Given a model's decision on an input x, a CFE [23, 24] finds the minimal modification x' such that the model's output changes in desired way. These explanations help interpret model decisions and provide actionable guidance to users to flip the prediction. In our context, we look into CFEs in the NLP domain where the inputs are token sequences. A counterfactual in this setting is a minimally perturbed sentence that causes the teacher LLM's prediction to flip. For instance, given the sentence loved the movie, labeled positive sentiment, a CFE would be hated the movie, a semantically similar but seniieent-fipped variant.
Our Problem Setting. We consider a binary classification setting where the teacher model will be denoted as ft X + [0, 1]. The input space C guxd with n being the sequence length and is the model dimension, after the entire input sequence has already been passed through the tokenizer and embedding layers of the LLM. The teacher model fi fx(x) gives the class-1 probability output
of the model for input i... f(() == (1) (x; 0c), where the superscript (1) is for the assigned probability for class 1. The final predicted class is given by fc(x) (xx( > 0.5) E (0,
Definition (Closest CFE C(x, St)). Given x E RRaxd such that f+(x) 0.5, the closest CFE is point x paxd with opposite prediction that minimizes the Frobenius-nomm ||x [|::
C(x, fi) arg minl|x x x'|l || such that f((x') 0.5. x
Definition naturally extends to multiclass settings, where a CFE can be defined as the minimum perturbation that changes the predicted class any other target class.
Remark 1 (Data Manifold Counterfactual Explanations). In practice, unconstrained counterfactuals may lead to unrealistic or out-of-distribution examples. To address this, we can constrain to lie within the data manifold X' C _d ensuring that generated counterfactuals remain semantically plausible. These data manifold counterfactuals preserve natural language structure. our work, we hybrid generation strategy that combines I.LM-based prompting with teacher model feedback to generate such nata-manifold CFEs. Further details are provided later in Section
Given a training data budget io (few-shots) and a teacher model ft, our goal to distill a smaller student model f. X + [0, 1] with high-ppeffrmance at specific task by leveraging CFEs.
3 Main Contributions
We begin with an experiment on 2D synthetic data that demonstrates how CFEs help student models mimic the teacher's decision boundary more effectively than standard data. Next, we provide theoretical results motivating approach from both statistical and geometric perspectives. Finally,

---

Teacher Decision Boundary
Student shots)
Student (k/2 shots + k/2 CFEs)
1.5
shots
CTEs
1.0
8
0.0
0.0
-0.5
-1.0
0
X1
X1
Figure 3: Decision boundaries for teacher and two students trained on synthetic data under few-shot setting. The teacher (a) trained the full dataset and serves the distillation target. Student (b) is distilled using 20 randomly sampled data points, and results in a poorly aligned decision boundary with the teacher. Student (c) is also trained on 20 total samples, 10 original data points and their 10 CFEs. This student learns a hetter decision boundary that aligns more closely with the teacher. This is because the CFEs lie close to the teacher's is decision boundary and the KD loss encourages the student match the teacher's soft predictions at the CFEs, clamping the student's 's boundary to the teacher's boundary.
describe CFE generation pipeline for natural language inputs, which leverages LLMs to produce semantically plausible CFEs, leading to our proposed framework CoD.
Synthetic Dataset Experiments to Illustrate the Role of CFE in Distillation: We conduct experi- ments on the 2D moons dataset [58] and show that infusing few-shot data with CFEs significantly improves student-teacher aligmment in distillation (see Figure 3). We train teacher model a
two-layer neural network with architecture [2 - 64 - 64 + 2] on the full dataset. The student network has a smaller architecture [2 + 16 + 2]. We randomly sample k = 20 original points (10 per class). For the original points, we compute their closest CFE, a minimally perturbed input that flips the teacher's predicted class. We follow a gradient-basee baseed method [23] to compute CFEs by perturbing each point in the direction of the teacher's ' logit margin until the predicted class flips. We consider two student models: trained the & few-shot samples alone, and another trained on k/2 few-shot samples and their CFEs. In both cases, we perform knowledge distillation by minimizing a combination of cross-entropy loss the hard labels and KL-divergence between the student and teacher soft predictions. Figure 3 shows the decision boundaries of the teacher, the baseline student, and the CFE-infused student. CFEs cluster near the decision boundary, enriching the distillation data in high-uncertainty regions. The student trained with CFEs aligns more closely with the teacher, thus motivating the use of Noundary-targeter examples for improved knowledge distillation.
Statistical Guarantees Motivating Our Approach: Here, we provide a theoretical motivation for the use of CFEs in few-shot knowledge distillation. We analyze a logistic regression setting using measure from estimation theory called Fisher Information [59] (also see Definition 2) that captures the information contained by random variable about parameter to be estimated. We show that dataset containing CFEs, which essentially lie much closer to the teacher's decision boundary, yields Fisher Information Matrix with higher overall information content for parameter estimation. As result, the student's expected estimation error is lower compared to training on standard samples.
Definition (Fisher Information Matrix [59]). Let L(0) be the log-likelioood of a parametric distribution p(y, x; 0), where 0 is the parameter vector to be estimated. The Fisher Information Matrix (FIM) at parameter 0 defined as:
I(0)= = Ex VV p(y, x; 0) Vo log xxmpp(yyx))))0 x]
Intuitively, Fisher Information measures the curvature of the log ligelikelihood:: flatter regions (low curvature) imply high uncertainty in estimating 0, while sharper regions (high curvature) indicate that small changes in 0 cause large changes in likelihood, enabling more precise parameter estimation.
We consider binary classification setting where both the teacher and student are logistic regression models. Suppose the teacher, parameterized by wt, defines the true data aata-generating distribution with predicted probabilities (y = 1 1xx) = o( w x) where a(.) is the softmax function. Suppose, the is obtained via maximum likelihood estimation (MLE) [59] using either
student, with parameters a standard dataset D or a CFF-infused dataset Det- Since the CFEs lie close to the teacher's 's decision boundary, we have w. x. 2J 0 when is CFE.
5

---

Theorem 1 (CFEs Improve Model Parameter Estimation). Lei w, and wa be the student parame- ters obtained via MLE on D (standard) and Des -CFF-inffueed) Assuming the teacher's 's parameters Wi capture the true data- generating distribution, that CFEs lie near the decision boundary, and that Then estimation error satisfies:
(cf)
the second moments Ex xx
R Ex, [x,x]]
E
[ (cl)
wrll
R [[w, - [[w,--
Proof Sketch: The key step in our proof relies on showing that the Fisher Information is given by (y (w
l[x;) (1 (v 1|xx)) )x,x"" The scalar weight (v 1 11x (y 1 1|x)) is maximized when (y 1 xx) = 0.5, i.c., x lies on the decision boundary. Standard samples in few-shot settings typically lie far from the boundary and contribute little to the FIM, whereas CFFs are constructed to lie near it and thus contribute significantly more. As result, the FIM of the CFE-infused dataset Dee dominates that of the standard dataset DD in Loewner order [60] (i.e., I(wa; Def) I( ( D)). The CFE-infused dataset provides strictly more information for parameter estimation than the standard dataset, ultimately leading to the bound on expected estimation error.
The full proof is in Appendix A. Notably, while this result mathematically motivates the advantages of CFEs in few-shot distillation, it still assumes linear models and same student-teacher capacity (size). For more general non-linear settings, we provide geometric perspective as discussed next.
Geometric Insight for Using CFEs for Distillation: Here, we examine the geometric effect of CFFs on student-teacher alignment in non-linear settings. Specifically, we show that when data points and their CFE pairs are included during distillation, the student's decision boundary comes much closer to the teacher's s boundary, as quantified by a formal measure called Hausdorff distance [61] between their respective decision surfaces. The Hausdorff distance (see Figure 4) captures the worst- case discrepancy between two sets (in our case, the decision boundaries of the teacher and student models) by quantifying how far any point on one boundary is from the closest point on the other.
sup Inf d(xx,)) xeit
Let MM= denote the decision boundaries of the teacher and the student. Our goal is to examine how close is the student's decision boundary to the teacher's. To quantify this alignment, we define the Hausdorff distance as follows:
fx Raxd fx() 0.5} and M,=(x E Raxd fx) 0.5}
M
Definition 3 (Hausdorff Distance). Let Mt. C Raxd be non-empty subsets of a metric space. The Hausdorff distance is defined as:
M,
sup int d(x,u) Nel, xeM;
H(Ms,M = mmxx sup inf ||x -ullr, sup inf ||u- 1-xxlI) x||r) M xEM,
Figure 4: Hausdorff
XEM, mEM,
Distance.
We observe that for training sample x; and its CFE x/, the segment joining them cuts the teacher's boundary since they have different predictions. Essentially, there exists an intersection point x; on this segment such that fx;)) = 0.5. Now, if the student is taught to matches the teacher at x; and x/, the student would also have another intersection point on this segment. These two intersection points lying on the teacher and student boundaries will act as clamps, pulling the two boundaries close to each other, since their own gap gets smaller as x; and its CFF x comes closer. We assume boundaries are closed and distance is measured within a compact region (e.g., support of data).
Lemma 1 (Existence of Boundary Crossing for Counterfactual Pairs). Let Ja RRaxd + [0. 1] be a continuous function. For a datapoint and its counterfactual pair (X1, x)), there exists a point x²- = ax, + (1- - a)x; an E (0, 1) (on the line joining x4 and x²) such that: f((x;) 0.5.
Theorem 2 (Teacher-Student Student Boundary Proximity). Let ft. fa :Rnxd +; 1] be the teacher and student model, with decision boundaries M, = {x (x) = 0.5} and M, = {x f.(()) = 0.5), respectively. Assume we observe a CFE-infused dataset Dos {xx,, ((x,,,x)) } satisfying: (AI) Minimal perturbation: ||x; -0--d -xIl= S with a >> O; (A2) Exact distillation: (x,) f((x.) and (x() = ft(x)); and (A3) E-spread along the teacher and student boundary, i.e., for each pair, there exist teacher's 's (or student's 's) crossing point x" = Ox; + (1 x)x; for a (0,1) such that
fe(x;)= = 0.5 (or, fs(x;) = 0.5) and for every a E MM (or MM)) there exists an i with ||a x; x;112 S e. Then the Hausdorff distance between the decision boundaries obeys: H(M,, Mc) S
e.
Consequently, tight (small a) and well-spread (small e) CFE pairs guarantee that the student boundary remains inside an (a + =)-tube around the teacher boundary.

---

Interpretation of the assumptions and bound. Our theorem makes three intuitive assumptions. (AI) Minimal perturbation requires each input and its CFE pair (x, x') to differ by at most a. CFEs are by definition the minimal changes that flips the teacher's prediction, so a is typically much smaller than the distance between arbitrary training points (note that we do no need CFEs to sit exactly on the teacher's boundary, i.e., fa 0.5). It suffices that the perturbation is small and flips the label-capturing the practical way CFEs are produced. (A2) Exact distillation agreement assumes the student matches the teacher's outputs on the input and CFE pairs. This is reasonable, these examples are directly used in training, and their logits are aligned through the distillation (KL) loss. (A3) E-spread assumes the inputs reasonably well spread. No region of the teacher's or student's boundary is more than E away from a crossing point (generally smooth). Under these assumptions, the Hausdorff gap between student and teacher boundaries is tightly bounded by a + E. This ensures the student's 8 decision boundary stays within an (a + é) e--u around the teacher's, ss illustrating the geometric faithfulness we want in few-shot knowledge distillation. See proofs in Appendix B.
Proposed Algorithm (CoD). We propose CoD, a Counterfactual Explanation-infusee Distillation strat- egy for few-shot, task-aware distillation of LLMs. The first step is CFE generation. Existing methods primarily fall into optimization- based [23]. search- based [62], and generative approaches [63]. These methods can be computationally expensive for LLMs. semanti- cally implausible examples. To address this, we adopt hybrid approach that combines the teacher model predictions with an LLM as an oracle for CFE gen- eration. Specifically, given an input and its original label, we prompt an LLM (e.g., GPT-4o [64]) to gen- erate a semantically similar sentence intended to flip the label with minimal changes to the input. We then check whether this generated example indeed flips
Algorithm CoD: CFE--ifuuse Distillation
Require: Teacher St. student dataset D 3)) CFGen, learning rate yj- loss weights a (KD), 3 (LWD), Epochs E
1: Da + 00
2: for all (x, D2 do
and frequently yield out-of-distribution
x' CFGen(xx,
Det t- Dar {(x', 1 y)3
5: end for
6: Dimin + De UDar
7: for 1 to E do
for all (x, y) Drain
Laant +- CF(gg(x)))) (x))
10:
LKD + (I(g((x))
11:
LIND + Eiez 11, (1) h 113
12:
L Lhaat + LKD + B LLWD
the teacher model's prediction, ensuring its utility as a true CFE. Once validated, each CFE is paired with its original input (x,x' and added to the training set.
13:
Update 0s + 0. 77 V..6
14: end for
15: end for
During distillation, we ensure that each input- CFF pair is included in the same mini-batch, enabling the student to jointly learn from both examples. The student is then trained using a combination of task loss, KL-based distillation loss, and optional layer-wise alignment. An overview of this process is described in Algorithm 1, with full implementation details and prompts provided Appendix C.
16: return distilled student ga
4 Experiments
The goal of experiments is to evaluate the effectiveness of integrating CFEs for knowledge distillation under few-shot learning settings. We investigate whether using limited real samples infused with their corresponding CFEs enable better distillation compared to only using real samples.
Datasets. We evaluate COD across six text classification benchmarks that span a range of do- mains. SST2 is a binary sentiment classification task derived from movie review snippets [65]. Sentiment140 consists of tweets labeled as positive or negative, reflecting user sentiment in short social media posts [66]. IMDB is a binary sentiment classification dataset containing full-length movie reviews [67]. CoLA (Corpus of Linguistic Acceptability) is a grammaticality judgment task that requires the model to identify whether sentence is linguistically acceptable [68]. Amazon Polarity contains customer reviews labeled as positive or negative sentiment [69]. Yelp is another sentiment classification dataset based on user-generated restaurant reviews [70].
Model. We experiment with two prominent model families: DeBERTa-v3 [25] and Qwen2.5 [2]. For DeBERTa-v3, we use the "base" model (100M parameters) as the teacher and distill into two smaller "small"" (44M) and "xsmall" (22M) variants as students. For Qwen2. 5, we use Qwen2. 5-1.BB as the teacher and distill into the smaller Qwen2.5- 5.0..550.55. Full training details are in Appendix C.
7

---

Table 1: Classification accuracy (# std) across datasets with varying total training sizes k. For COD, training data consists of k/2 standard and k/2 CFEs. Teacher model DeBERTa-v3 -v---ss and student model DeBERTa-Ve s3-mall1.
Total Samples (k)
Dataset
Method
8
16
32
64
128
512
KD
0.671 10.046
0.712.100033 20.035
0.758 10.032
0.789 10.022
0.823 10.016
0.846 =0.007
Amazon
+CoD
0.758 =0.027
0.795 10.033
0.819 20.035
0.812 10.004
0.837 10.014
0.860 =0.015
Polarity
LWD
0.676 =0.090
0.738 +0.033
0.777 =0.009
0.809 +0.015
0.827 +0.025
0.842 =0.019
+CoD
0.724 =0.052
0,779 +0.056
0.811 +0.015
0.828 +0.015
0.816 +0.020
0.841 =0.013
KD
0.693 #0.062
0.707 +0.029
0.721 +0.012
0.747 +0.005
0.758 +0.009
0.771 +0.003
+CoD
0.739
0.755 40.017
0.769
0.769 20.016
0.772 20.006
0.791 =0.004
CoLA
LWD
0.713 20.031
0.698 20.037
0.731 20.021
0.744 40.007
0.750
0.761 =0.011
+ CoD
0.730 +0.035
0.744 +0.031
0.762 +0.011
0.752 +0.009
0.756 +0.010
0.784 +0.008
KD
0.714 20.047
0.817 40.028
0.875
20.027
0.896 40.008
0,912 10.009
0.917 20.006
+ CoD
0.835 =0.078
0.888 +0.005
0.890 #0.011
0.899 +0.007
0.907 +0.006
0.913 =0.005
INDB
LWD
0.760 #0.046
0.836 +0.045
0.875 #0.024
0.889 +0.013
0.905 +0.008
0.914 #0.006
+ CoD
0.861 =0.017
0.886 20.011
0.893 20.006
0.898 +0.005
0.905 #0.010
0.913 =0.010
KD
0.617 #0.042
0.712 20.052
0.757 =0.063
0.820 40.019
0,848 #0.013
0.899 =0.007
+ CoD
0.719 =0.063
0.781 +0.034
0.821 =0.013
0.827 #0.008
0,853 +0.015
0.892 =0.018
SST2
LWD
0.627 =0.053
0.721 +0.055
0,776 #0.031
0.817 #0.005
0,829 +0.013
0.892 =0.012
CoD
0.694 #0.079
0.785 +0.028
0.832 #0.011
0.830 +0.007
0.835 +0,012
0. 0.880 =0.020
KD
0.714 058
0,817 10.031
0.855 =0.021
0.878 #0.006
0.885 +0.018
0.916 =0.007
CoD
0.740 #0.094
0.832 +0.045
0.860 =0.018
0.874 +0.006
0.888
00.01
Yelp
LWD
0.733 =0.070
0.832 +0.026
0.857 #0.011
0.868 #0.006
0.881
0.920
+ CoD
0.738 20.093
0.865 10.010
0.870 =0.017
0.871 10.019
0.885 10.007
0.913 20.013
KD
0.580 #0.039
0.597 40.042
0.645 20.023
0.690 10.035
0.752 10.011
0.802 =0.006
+ CoD
0.629 20.036
0.640 20.048
0.731 10.022
0.754 10.017
0.778 10.007
0.784 =0.019
Sent140
LWD
0.581 +0.041
0.593 40.039
0.665
+0.027
0.708 +0.029
0.751 +0.009
0.785 =0.019
+ CoD
0.628 =0.034
0.652 +0.038
0.706 +0.016
0.741 +0.014
0.729 +0.063
0.760 +0.023
Baselines. We compare our method against three task-aware knowledge distillation baselines: (i) Standard knowledge distillation (KD) where the student learns from the teacher's soft predictions using KI. divergence
[71]; (ii) Layer-wise distillation (LWD), which extends KD by additionally aligning the student's 's intermediate hidden representations with those of the teacher using mean squared error [13]; and (iii) TED (Task-aware Layer-wise Distillation) which incorporates task- specific neural filters at each layer to selectively transfer task-relevant information from teacher to student [12]. All methods are evaluated under k-shot training settings, and student models are trained on identical few-shot splits to ensure fair comparison (see details in Appendix C).
Setup. As in prior works on task- distillation [12], we first train a teacher model on the full training dataset to serve as a strong source of supervision. A student model is then initialized and distilled using only datapoints, where & E {8. 16, 32, 64, 128, 512]. We apply our strategy CoD to three standard distillation baselines: KD, LWD, and TED. For a fair comparison, COD uses k/2 original samples and their k/2 corresponding CFE (a total of shots) while the bascline methods are trained on & original samples. Performance is evaluated using accuracy the test set for each dataset. All experimental results are averaged over five runs, with the mean and standard deviation reported. Results for the D while results for the smaller DeBERTa-v& - student are Appendix C. For experiments using the Qwen2. 5-1.58 teacher and the Qwen2. 5-0. 5B student, see Table 3. We report the accuracy of teacher models trained on the full datasets in Table in Appendix C.
teacher and DeBERTa- v3- small student are shown in Table 1,
Results and Analysis. Across all datasets, we observe that CoD significantly improves performance in the low-data regime, particularly when S 64. For example, on Amazon Polarity with only labeled examples, KD CoD achieves 75,8% accuracy compared to 67. 17.1% for standard KD (8.7
8

---

Table 2: Classification accuracy (# std) with TED and TED + CoD across datasets and varying to- tal training sizes k. For CoD, training data consists of k/2 standard and k/2 CFEs. Teacher model is DeBERTa-Ve v3- base and student model is DeBERTa-v3-smn
Total Samples (k)
Dataset
Method
8
16
32
64
128
512
Amazon
TED
0.646 +0.075
0.697 =0.033
0.758
+0.012
0.816 #0.023
0.814 =0.020
0.846 =0.025
Polarity
CoD
0.731 +0.054
0.754 =0.056
0.802 +0.007
0.818 20.013
0.805 =0.008
0.848 =0.010
TED
0,750 +0.022
0.737 =0.028
0.731 +0.020
0.746 +0.011
0.760 =0.011
0.772 #0.010
CoLA
CoD
0.748 20.028
0.757 =0.023
0.767 10.021
0.768 10.016
0.780 20.007
0.791 =0.006
TED
0.695 +0.018
0.800 =0.042
0.854 +0.023
0.876 +0.012
0.908 =0.009
0.917 =0.006
IMDB
CoD
0.827 20.056
0.879 20.003
0.884 10.007
0.887 10.010
0.895 20.010
0.916 =0.005
TED
0.597 +0.052
0.701 =0.055
0.732 +0.026
0.812 +0.026
0.829 #0.002
0.904 #0.006
SST2
CoD
0.658 20.087
0.779 =0.012
0.813 +0.017
0.833 10.014
0,836 =0.030
0.879 20.011
TED
0.699 +0.048
0.815 =0.014
0.846 +0.020
0.869 +0.012
0.894 =0.009
0.914 +0.012
Yelp
CoD
0,742 20.095
0.837 20.016
0.868 10.018
0.878 10.019
0.886 20.013
0.913 20.008
points improvement). Similarly, for IMDB at k = 8, LWD + CoD improves over standard LWD by more than 10 points (86.1% vs. 76.0%). As the number of labeled examples increases, the benefits of CFE augmentation diminish. At & = 512, the performance of standard and CoD becomes nearly identical in many cases. However, even in these larger settings, is important to note that our method achieves comparable results while using only k/2 real samples and k/2 CFE, effectively halving the amount of labeled data required to reach similar performance. The effectiveness of CFEs varies by dataset. On CoLA, we observe consistent improvements across all A: values for both KD and LWD, contrast,
indicating that CFES are well-aligned with the task's grammaticality decision boundary. datasets like Sentiment 140 show strong early gains. For datasets such as IMDB and SST2, CFE provides substantial improvements at low k. but underperforms slightly at h = 512, possibly due to redundancy. Among distillation methods, LWD generally performs par with or slightly better than KD across most settings, with CoD offering similar relative improvements for both.
We also compare with TED which has been found to work well with larger distillation datasets [12]. We note that TED introduces additional complexity by requiring the training of task-specific filters prior to distillation. Interestingly. we find that TED does not consistently outperform classical methods like KD or LWD the few-shot settings (see Table 2). Nonetheless, TED + CoD yields consistent gains over standard TED, demonstrating that our approach is broadly applicable. Our findings suggest that simpler distillation approaches like KD or LWD are preferable when data is scarce: they easier to implement and, when combined with CoD, deliver much stronger performance gains without the overhead of filter training.
Ablations. (1) On template designing and prompt choices: We vary prompt templates for generating CFEs and observed that CoD robust to prompt choices, showing low standard deviation across variants and consistently outperforming the KD baseline few shot settings (see Table 7). One possible direction could be to use automatic prompt generation methods [72], however these are typically more compute-intensive. (2) Computational and Memory Requirements: We assess the computational efficiency of CoD under varying few-shot budgets using the codecarbon package [73] to track runtime and energy consumption (see Table 8). (3) Effect of soft-label supervision: To study the role of soft-labels in our approach, we remove or corrupting the teacher's soft labels (see Table 9). Removing the soft label term (a 0) leads to substantial drop in performance across all shot levels. Although CoD still improves over KD in this setting, the gains are significantly reduced. This highlights that the soft label calibration from the teacher is a key contributor to the effectiveness of counterfactual explanation data. Additionally, when replacing soft labels with random values, performance degrades sharply, likely due to inconsistency with the hard labels, introducing conflicting supervision signals in the training objective.
Discussion. In this paper, we introduced novel approach for task-aware knowledge distillation in few-shot settings that leverages CFEs to enhance the data efficiency of knowledge distillation. Our results show that CoD consistently outperforms existing distillation approaches in low-data regimes.

---

Table 3: Classification accuracy (# std) of Qwen2.5 5 on CoLA and Yelp datasets with varying training sizes k. For CoD training data consists of k/2 standard and k/2 CFEs. Teacher model is Qwen2. 5-1.55 5B and student model Qwen2.5-0 5-0. 5B. Refer to Appendix C for other datasets.
Total Samples (k)
Dataset
Method
16
32
64
128
512
KD
0.681 20.012
0.676
0.668 20.042
0.654 20.032
0.676 20.020
0.732 20.014
+ CoD
0.683 +0.016
0.686
0.697 =0.015
0.711 #0.020
0.736 +0.017
0.757 #0.011
CoLA
LWD
0,681 20.012
0,657 20.031
0.678 20,018
0.650 20.039
0.636 20.029
0.712 20,014
+ CoD
0.682 s0.018
0.687 s0:013
0.704 =0.010
0.714 a0.020
0.719 a0.022
0.755 20.013
KD
0.684 #0.021
0.759 20.040
0.827 #0.030
0.861 #0.017
0.887 20.012
0,920 =0.010
+ CoD
0.745 +0.029
0.779 20.048
0,828 =b.072
0.886 20.007
0.883 20.010
0.916 20,008
Yelp
LWD
0.685 +0.019
0.777 +0.036
0.837 +0.027
0.876 +0.020
0.898 +0.008
0.920 +0.008
+ CoD
0,746 +0.028
0.778 +0.035
0.847 =0.020
0.876 #0.014
0.883 +0.010
0.909 +0.009
Importantly, we demonstrate that CoD can achieve improved performance over baselines while effectively using only half the number of original data, with the remainder consisting of generated CFEs. This finding has significant implications for reducing the cost of data collection in real-world scenarios where sourcing high-quality data is expensive or time-consuming [74]. Our approach offers an explanation-driven perspective distillation. By including CFE's, implicitly highlight the key features most important to flipping a teacher's decision. This may help the student model reduce its reliance on spurious correlations, especially in few-shot settings. In effect, CFE's guide the student to attend to "why" a label changes, not just "what" the label is. This bridges explainability and compression, turning explanations into actionable data for knowledge distillation.
On the extension to generative LLMs. As research increasingly focuses on data-efficient LLM training [75, 76], future work could extend our approach generative sequence-to -o-quuenee models, enabling efficient distillation beyond classification. In this setting, a CFE can be defined as a minimal change to the input (prompt) that flips chosen property of the generated text. Formally, given generative model f. a prompt x, and binary attribute function g g(uutput) (e.g., sentiment, toxicity, factuality, topic relevance), counterfactual explanation prompt x* would be small semantic perturbation of x such that the generated output f(xx) flips the value of g(f(x)). More broadly, CFEs can be reframed through model sensitiviry, identifying small input perturbations that cause large changes in the output distribution or likelihood. This corresponds to large changes in the
sequence-level probability: po x) II" (o(. x, Wecc), where y (yı, (/,..2.....3 ..)) is the generated sequence for an input prompt x. This could reveal regions of the input space where the model is uncertain, making them especially informative for distillation, supervised fine-tuning, or post-training Our approach offers path toward data-efficient LLM distillation from minimal data while reducing cost and maintaining performance.
Limitations. Generating CFEs introduces additional computational overhead compared standard distillation approaches. Moreover, our current CFE generation strategy, which relies on prompting LLMs, does not guarantee that we would get the closest counterfactual (as defined in Definition 1), potentially limiting the precision of our distilled knowledge. Future work could explore alternate methods for generating closer and semantically valid CFEs. Additionally, as with knowledge distillation in general, CoD is inherently dependent on the quality of the teacher model. Any inaccuracies or biases present in the teacher's 's decision boundary may be inherited by the student. Addressing robustness to flawed teachers remains important direction for future research.
Societal Impact. CoD offers several potential societal impacts, particularly in reducing the cost and effort associated with data collection [74]. By enabling the distillation of high-performance models with fewer data samples, this approach can significantly lower data collection costs, making machine learning more accessible in low-resource environments. This is especially valuable in industries where data often searce and expensive to obtain [77, 74]. Moreover, by requiring fewer samples and targeting smaller student models, CoD contributes to more efficient model training and scalable deployment. Our method leverages explanations as a tool for more effective model compression. In doing it bridges the gap between explainability and model compression.
10

---

References
[1] OpenAl. Gpt-4 technical report. arXiv:2303.08774, 2024.
[2] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and al. Qwen2.5 technical report. arXiv:24/2.15/15. 2025.
[3] Aee Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv:2501. 2025.
[4] Sanjay Surendranath Girija, Shashank Kapoor, Lakshit Arora, Dipen Pradhan, Aman Raj, and Ankit Shetgaonkar. Optimizing ilms for resource-constrained environments: A survey model compression techniques. arXiv preprint arXiv:2505.02309 2025.
[5] Geoffrey Hinton. Distilling the knowledge in neural network. arXiv preprint arXIv:1503.00533) 2015.
[6] Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models. arXiv preprint 24Xi::2402.13116, 2024.
[7] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: survey. International Journal of Computer Vision, 129(6):1799-188)) 1819, 2021.
[8] Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley Love, Erin Grant, Iris Groen, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Zhang. Raja Marjieh, Robert Geirhos, Sherol Chen, Simon Korblithh Sunayana Rane, Talia Konkle, Thomas O'Connell, Thomas Unterthiner, Andrew K. Lampinen, Klaus-Robert Müller, Mariya Toneva, and Thomas I.. Griffiths. Getting aligned on representational alignment. arXiv preprint 2r0 13018, 2023.
[9] Sangchul Hahn and Heeyoul Choi. Self-knowledge distillation in natural language processing. arXiv preprint arXiv: 190.0018110 2019.
[10] Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01466 2023.
[11] Yuanhao Yue, Chengyu Wang, Jun Huang, and Peng Wang. Distilling cntruuttionfollovviin abilities of large language models with task-aware curriculum planning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6030-6054, Miami, Florida, USA, November 2024, As- sociation for Computational Linguistics. doi: 10. 18653/v 22/22/22.2.mm/ 1i002...nnnigg-emml..355.. URL https: nttps://aclees /aclanthoo// org/2024 findings----ii mendings-emmlp....
[12] Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pages 20852-20867. PMLR, 2023.
[13] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 2019.
[14] Luyang Fang, Xiaowei Yu, Jiazhang Cai, Yongkai Chen, Shushan Wu, Zhengliang Liu, Zhenyuan Yang, Haoran Lu, Xilin Gong, Yufang Liu, et al. Knowledge distillation and dataset distillation of large language models: Emerging trends, challenges, and future directions. arXiv preprint or 25v: /4772, 2025.
[15] Hui Xue, Yuexuan An, Yongchun Qin, Wenqian Li, Yixin Wu, Yongjuan Che, Pengfei Fang, and Minling Zhang. Towards few-shot learning the open world: review and beyond. arXiv preprint arXiv:2408.091 09722, 2024.

---

[16] Min Zhang, Donglin Wang, and Sibo Gai. Knowledge distillation for model-agnostec meta- learning. In ECAI 2020, pages 1362. IOS Press, 2020.
[17] Benlin Liu, Yongming Rao, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Metadistiller: Network self-boosting via meta-learned top-down distillation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 694-709, Springer, 2020.
[18) Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Few sample knowledge distillation for efficient network compression. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14639-14647, 2020.
[19] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03550 2016.
[20] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
[21] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[22] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Mors, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
URL https: //tp///aavvvvv/aa///a//eeeeaaava org/ ha///b///20 /205.. 14165.
[23] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31,841,
[24] Sahil Verma, Varich Boonsanong, Minh Hoang. Keegan E. Hines, John P. Dickerson, and Chirag Shah. Counterfactual explanations and algorithmic recourses for machine learning: review, 2022. URL
hts//bb/220000
[25] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using clectra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543 2021.
[26] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. arXiv preprint arXiv:2306.088: 2023.
[27] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint
2019.
[28] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1rXiv:1412.65500 2014.
[29] Pasan Dissanayake, Faisal Hamman, Barproda Halder, Ila Sucholutsky, Qiuyi Zhang, and Sanghamitra Dutta. Quantifying knowledge distillation using partial information decomposition. arXiv preprint arXiv:2411.07488 2024.
[30] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2022.
[31] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar axXiv:2001.08366 2020.
[32] Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Jie Liu, Ge Zhang, Yanan Wu, Congnan Liu, et al. DDK: Distilling domain knowledge for efficient large language models. Advances in Neural Information Processing Systems, 37-98297-988199 2024.
12

---

[33] Chengliang Chai, Jiabin Liu, Nan Tang, Ju Fan, Dongjing Miao, Jiayi Wang, Yuyu Luo, and Guoliang Li. Goodcore: Data-effective and data-efficient machine learning through coreset selection over incomplete data. Proceedings of the ACM on Management of Data, 1(2):1-27, 2023.
[34] Noveen Sachdeva, Benjamin Coleman, Wang-Choe Kang, Jianmo Ni, Lichan Hong, Ed Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient Ilms. arXiv preprint arXiv:2402.09668, 2024.
[35] Suruchi Kumari and Pravendra Singh. Data efficient deep learning for medical image analysis: survey, arXiv preprint arXiv:2310.06557, 2023.
[36] Faisal Hamman, Pasan Dissanayake, Saumitra Mishra, Freddy Lecue, and Sanghamitra Dutta. Quantifying prediction consistency under fine-tuning multiplicity in tabular LLMs. In Forty- second International Conference on Machine Learning, 2025. URL https://openreeverr http/forumnddAAJJnneeeee
[37] Amir-Hossein Karimi, Gilles Barthe, Bernhard Schölkopf, and Isabel Valera. A survey of algorithmic recourse: Contrastive explanations and consequential recommendations. ACM Comput. Surv., 55(5), December 2022. ISSN 0360-0300. doi: 10.1145/3527848. URL
https://doi. wtt......1///5............ 1145/35278888.
[38] Saumitra Mishra, Sanghamitra Dutta, Jason Long, and Daniele Magazzeni. A survey on the robustness of feature importance and counterfactual explanations. arXiv preprint arXiv:2111.00358, 2021.
[39] Faisal Hamman, Erfaun Noorani, Saumitra Mishra, Daniele Magazzeni, and Sanghamitra Dutta. Robust counterfactual explanations for neural networks with probabilistic guarantees. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202, pages press/v200 hatess/2222/// 2at02/haman2aaa..mm html.
PMLR, 2023. URL httpt::///ppooeeednggs mlr.
[40] Faisal Hamman, Erfaun Noorani, Saumitra Mishra, Daniele Magazzeni, and Sanghamitra Dutta. Robust algorithmic recourse under model multiplicity with probabilistic guarantees. IEEE Journal Selected Areas in Information Theory, 5:357-368, 2024.
[41] Martin Pawelczyk, Klaus Broelemann, and Gjergji Kasneci. On counterfactual explanations under predictive multiplicity. In Conference on Uncertainty in Artificial Intelligence, pages 809-818. PMLR, 2020.
[42] Kentaro Kanamori, Takuya Takagi, Ken Kobayashi, and Hiroki Arimura. Dace: Distribution- aware counterfactual explanation by mixed-integer linear optimization. In IJCAI, pages 2855- 2862, 2020.
[43] Rafael Poyiadzi, Kacper Sokol, Raul Santos-Rodriguez, Tijl De Bie, and Peter Flach. Face: Fea- sible and actionable counterfactual explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 344-350, 2020.
[44] Pasan Dissanayake and Sanghamitra Dutta. Model reconstruction using counterfactual explana- tions: perspective from polytope theory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[45] Van Bach Nguyen, Paul Youssef, Christin Seifert, and Jörg Schlötterer. Llms for generating and evaluating counterfactuals: A comprehensive study, 2024. URL bostps://arxxx org/orgg//... 2405.00725
[46] Stephen McAleese and Mark Keane. A comparative analysis of counterfactual explanation methods for text classifiers, 2024. URL https://taaaaee http:/::///xvv////// www.:.rg/aabaoo.ao /t//abb////111 02643.
[47] Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. Zero-shot llm-guided counterfactual generation: case study on nlp model evaluation, 2024. URI. https::///////tt //axxiv. org/abs/2405. 04793.
13

---

[48] Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
[49] Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. Learning the difference that makes a difference with counterfactually-augun
data. arXiv preprint urXiv:9909, 12434, 2019.
[50] Divyansh Kaushik, Amrith Setlur, Eduard Hovy, and Zachary C. Lipton. Explaining the efficacy of counterfactually augmented data. arXiv preprint arXiv:2010.02114, 2020.
[51] David Egea, Barproda Halder, and Sanghamitra Dutta. Vision: Robust and interpretable code vulnerability detection leveraging counterfactual augmentation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 812-823, 2025.
[52] Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Gen- erating counterfactuals for explaining, evaluating, and improving models. arXiv preprins MxXiv:2101002280 2021.
[53] Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. Disco: Distilling counterfactuals with large language models. arXiv preprint arXiv:2212. 2022.
[54] Rachneet Sachdeva, Martin Tutek, and Iryna Gurevych. Catfood: Counterfactual aug- mented training for improving out-of-domain performance and calibration. arXiv preprint arXiv:2309.07822, 2023.
[55] Tao Feng, Yicheng Li, Li Chenglin, Hao Chen, Fei Yu, and Yin Zhang. Teaching small language models reasoning through counterfactual distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5831-588 1831-5842, 2024.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez.
t. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[57] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini arXiv: 150.00231,,111100 2015.
[58] make_moons scikit-learn 1.6.1 documentation. https://scikit--Dee http:///siiktttlla..r.rrrrssbbaa m wkleaarddddeeeeeeenemmammaa datasets wwwemmakknaaa..e.aamme.mmmmmo html. Accessed: 2025-05-16.
[59] Erich L. Lchmann and George Casella. Theory of Point Estimation. Springer, 2nd edition, 1998. ISBN 978-0-387-98502-3.
[60] Rajendra Bhatia. Positive definite matrices. Princeton university press, 2009.
[61] R. Tyrrell Rockafellar and Roger J-B Wets. Variational Analysis, volume 317 of Grundlehren der mathematischen Wissenschaften. Springer, 1998. ISBN 978-3-540-62772--2
[62] Daniel Gilo and Shaul Markoviteh. A general beasch-based framework for generating textual counterfactual explanations, 2023. URL //arxi/va.iiiia//a http:t/ss//aaivvvooo///////..1.. 00369.
[63] Stig Hellemans, Andres Algaba, Sam Verboven, and Vincent Ginis. Flexible counterfactual explanations with generative models, 2025. URL htR.htttp://axxii.orr////// wwtp:///////a///////// 2502. 17613.
[64] OpenAl (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
[65] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013.
[66] Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervi- sion. CS224N project report, Stanford, 1(12):2009, 2009.
[67] Andrew L.. Maas, Raymond F. Daly. Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011.
14

---

[68] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint vrXiv:1805.1247/ 2018.
[69] Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimensions with review text. Proceedings of the 7th ACM conference on Recommender systems, 2013.
[70] Yelp. Yelp open dataset, 2025. URL https: yelp. httpp.ccom/daa/aaaee com/dattaaressuurees/ opta-dataset/. Accessed: 2025-05-15.
[71] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL https://arxiv.c org abs// 1910.01108.
[72] Elad Levi, Eli Brosh, and Matan Friedmann. Intent-based prompt calibration: Enhancing prompt optimization with synthetic boundary cases. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
[73] Benoit Courty, Victor Schmidt, Laura Duboc, Lucile Demailly, et al. Codecarbon: Estimate the carbon footprint of your computing, 2023. URL https: btttps://githbb com cooo////aaaarbo
[74] Yuji Roh, Geon Heo, and Steven Euijong Whang. A survey data collection for machine learning: a big data ai integration perspective, 2019. URL htttps://arxiv.oott h/tp//b///// /1811 03402.
[75] Noveen Sachdeva, Benjamin Coleman, Long-Cheng& Kang, Jianmo Ni, Lichan Hong, Ed H. Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient Ilms, 2024. URL https::/taaaee //axivv arxiv. org/ abs
[76] Omar Y. Al-Jarrah, Paul D. Yoo, Sami Muhaidat, George K. Karagiannidis, and Kamal Taha. Efficient machine learning for big data: A review. Big Data Research, 2(3):87-93, 2015. ISSN 2214-5796.
doi: 10.1016/j.bdr.200 2015.00.00.000000 URL httpp:///ww..cienceeieeee.
[77] Matt Bonakdarpour, Joshua Bon, and Matthew Stephens. Asymptotic normality of mle, 2019. URL hctps://sttephens99. git.gt.thuu... wattuu..oofiveMaiii http:///wwweaattttaaaaa////a/oaa i_
mle html. Accessed: 2025-05-20.
[78] Pablo Zegers. Fisher information properties. Entropy, 17(7):491-49933 2015.
[79] Anand Avati, Bias-variance analysis: theory and practice, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 3 details our key contributions stated in the abstract and introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We provide all sets of assumptions and proofs in Appendix B and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide details about our approach in Section 3 and Algorithm 1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
The code will be released as soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtppbccc/ppaccccccccccccccc httpg/uddds////e/e/ee/eeee /Ctoe/eoooiiseeeeeeeee ssionsolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide details about our experimental setups in Section and Appendix
D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide standard deviations in all tables containing results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information about the computer resource Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:://eeu..eeeeeeeeeee ccts..ccppuuu
http:://ouurppp..cppbbb.ccc htte//ubblc//tiiccooo ines?
Answer: [Yes]
Justification:
We strictly follow NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Since we focus on publicly available datasets and LLMs, our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We properly acknowledge all assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //tp:///ensu.eeeeeeeeeee httpe//rpps..ccoooeeeeeeeeeeeeee http:////oooeeeeecoc///////11/1 for what should or should be described.
22

---

Appendix
A
Background on Fisher Information and Proof of Theorem
This section provides background on Fisher information and a formal proof for Theorem 1, which quantifies the reduction in estimation enror from using CFE-infused training data.
A.1 Background on Fisher Information Matrix
Definition (Positive Semi-definite Matrices). matrix RRddd is said to positive semi-definite it is symmetric and for all non-zero vectors x E IR,, the following condition holds:
2x 0 for all x E Rd.
The eigenvalues of positive semi-definite are non-negative, i.e., X,(A) 2 0 for all eigenvalues x of A.
Definition 5 (Löwner Order). Let A, dxdd be symmetric matrices. We say that A is greater than or equal to in the Löwner order, denoted A = B, if and only if the matrix A is positive semi-definite. That is,
A z B if afd only if x² (A B) 2 0 for all I E TR...
B, then A B is positive definite, meaning A is strictly greater than B in the Löwner order. Matrices). For positive semi sefi-definite matrices
Lemma 2 (Trace Inequality for Positive A, B E Rpadd where > B, then:
TMA TT(B
Proof. Since A B, have B-1 A-1 by the Löwner order inversion property. The trace operator preserves this inequality because for any > Y 0:
Tr(X) = Σ A(XX Σ =
where )a))) denotes eigenvalues in descending order.
Definition 2 (Fisher Information Matrix [59]). Let [(0) be the log- log-likelihood of a parametric distribution ply, xz; 0), where the parameter vector to be estimated. The Fisher Information Matrix (FIM) at parameter 0 is defined as:
I(0) = Ex.y Vologg x; Vologp(u.x;6]]
Fisher information captures the amount of information that an observable random variable & carries about unknown parameter 0 of distribution that models æ. We the notation I(0;y, to denote the Fisher information about 0 carried by single observation y,
A.2 Proof of Theorem
(cf) be the student parame- Theorem 1 (CFEs Improve Model Parameter Estimation). Let w, and w
ters obtained via MLE on (standard) and Det (CFE- Eiifuseddd Assuming the teacher's 's parameters Wt capture the true data- generating distribution, that CFEs lie the decision boundary, and that x Ex, [xxx]] Then estimation error satisfies:
the second moments Eg [xx
E | (cf) woll]]
E ||w, well3].
Proof. For a single observation (x, y), the -ogeliill is:
log p(y|x; w) = log (w x) + (1 y) log(1 (1- - oww Taking the gradient with respect w:
(3)
V. log w) (y (w x))x
(4)
23

---

To prove Theorem 1, we first (1) Characterize the Fisher information for individual observations, (2) Establish asymptotic normality of MLE, (3) Compare information matrices standard vs. CFE-
infused datasets, and (4) Apply trace inequality to connect information to estimation error.
(1) Fisher Information for Logistic Regression: For logistic regression model with parameters w, Lets denote Fisher Information Matrix (FIM) for observations y, X as:
Z(w;y, = Ey
V log x; w V log w)
(5)
V logp(u, V logp(u/x;; V log p(x)
(6)
The gradient of (x) is zero because p(x) is independent of the model parameters w. Using the law of total expectation:
I(w:y.x) Ex y/yx V log w) V log w
Substituting Equation
I(w:u, = 18x [B - o(w x)))xxx ']]
(8)
= =Ex xxx By oww x]]
(9)
The term Ey (v - o( (owTx))" x the variance y/x. Where y/x ~ Bernouli(o T x)). we compute:
Eyxxl(y o(w
x)) -owT))))))
Var(y/ x) = o(w T x)( o(w") x))
(10)
Thus:
Z(w: = Ex [o x)(1 - o(w x))xx"|
(11)
The variance term is maximized when w x 0 (i.e., the decision boundary), where it equals 0.25.
(2) Asymptotic Distribution of MLE: Under regularity conditions [77], the MLE estimator satisfies:
Vk( w. - w.) & N((,,I-
(12)
where I(w;; D) of Vrs (Additivity property of fisher information [78]).
Σ I(wr; yr x;) the total Fisher information independent observations
The mean squared error (MSE) [79] decomposes as:
E w. - w:|l Tr(Cov(w.)) + |Bias w .)
(13)
Variance
Bias
For MLE, Bias(w,) - as xo, So: R|[,, w Tr(I (w;; D))
The next step of the proof we compare the fisher information between a standard dataset and CFE- infused dataset.
Let D = (x be a dataset k standard samples, and let Dar = 1// U {xe, 1/2 an
CFE-infused dataset containing h/2 standard samples and k/2 CFEs.
Standard Samples: Far from decision boundary w. x; 0 or < 0. Thus:
c(w]x:)(1 - ow, xx)) = e; < 0.25
(14)
Their FIM contribution is: I(w,; x:) = Ex(e:xxxx]]
CFE Samples: Near boundary + w x, 0 + (0)) = 0.5. Thus:
ow, xx) (w, xx)) 0.25
(15)
Their FIM contribution is maximal: I(w,;xe) = Ex 20.25xx.x.1
Since [xxxxx]] T Rs Fxx, [x.x]] and 0.25 > 6; we have I(w,; Doy) > I(W;; in the Löwner order (see Definition 5).
24

---

[,00) [(x)
Output Protabiiity
f000) (00)
0.6
Student
Bourdary
Teacher
Boundery
Point
Small
Figure 5: Intuition for Theorem 2
Remark 2 (Feature Spanning). Note that for logistic regression the feature vector augmented with the parameter bias term, i.e., X [1, hence, the outer product xx' has a non-zero norm, The first element of x is always ensuring ||x||² 2 1. Thus, xx cannot be the zero matrix, even if X O. This guarantees that each CFE example X. contributes non-degenerate rank-1 term the FIM.
The final step of the proof leverages the trace inequality for covariance matrices (see Lemma 2). If I(w,; Doy) > I(w,; D) then (w:; Dof)) < Tr(Z (w,; D)). Thus, CFE infusion reduces
parameter estimation error:
|llw well?!
E [| w
(16)
Remark 3 (Datapoint Diversity). For the total FIM I(we; Dij) to be invertible, the set of feature vectors {x:} must span Rd which will hold if we have enough samples.
B Background on Hausdorff Distance and Proofs of Lemma and Theorem
This section provides definitions and geometric preliminaries, along with proofs for Lemma I and Theorem 2.
B.1 Background on Hausdorff Distance
Definition (Line Segment). Let X,, X3 guxd be two points in the space. The line segment [x, connecting x; and x, is defined as the set of points 7(A) for X E [0, 1), where
7()) = -1 + Ax', [0
This defines all the points space between and x/ in Raxd
Lemma 3 (Intermediate Value Theorem). Let f : b] - IR be a continuous function, and let f(a) # f(b). y any value between f(a) and (b), then there exists e (a, b) such that f(c) y. Definition 3 (Hausdorff Distance). Let MG, C Raxd be two non-empty subsets of a metric space. The Hausdorff distance is defined as:
H(Ms,Maa max sup inf ||x- ullr: sup inf || ||-- x(|m] xEM, uEM,
ueM, XEM,
B.2 Proofs of Lemma and Theorem 2
Lemma 1 (Existence of Boundary Crossing for Counterfactual Pairs). Let f. Bxd [0. 1] be
a continuous function. For datapoint and its counterfactual pair (x,,x)) x.), there exists a poins
x; ox, + (1-a)x; for
(0, 1) (on the line joining x; and x.) such that: f((x;) = 0.5.
Proof. Define the line segment from 24 to x4 using a parameterization: 7()) = (1 A)xs Xa for [0, 1).
25

---

This defines a continuous path from r; to æ in R Now define the real-valued function g [0, - R by: g(A) = f((r(A)) fi((1 A)x; + Arς).
Since ft is continuous on R2 and >()) is continuous in X, the composition g(A) is continuous on the closed interval [0, I].
Now, evaluate the endpoints of this function: g(0) - fz,) < 0.5, = fx() > 0.5.
Thus, we have g(0) < 0.5 < g()), and by the Intermediate Value Theorem (see Lemma 3), since g is contimuous on [0, 1). there exists X E (0, 1) such that: g(^*) = 0.5.
Define x (>*) (1 X )z= + X x E [1x,, Then: fe(x;) = g ((*) = 0.5.
Hence, the point 27 E [1] lies on the segment and satisfies fe(x;) 0.5, as required.
Theorem 2 (Teacher-Student Boundary Proximity). Let ft. fs : Rud +; [0, 1] be the teacher and student model, with decision boundaries Me = (x fx(x)= 0.5) and M2 = fx fx(x) = 0.5),
respectively. Assume we observe a CFE-infused dataset Def {(x,, (x), xx))
satisfying: (Al)
Minimal perturbation: ||x; x|Il= S with a > (A2) Exact distillation: [x(x) (((xx) and
fx(x)) = [xxx;;; and (A3) e-ppredd along the teacher and student boundary, i.e., for each pair, there exist teacher's 's (or student's s) crossing point x = =xx; + (1 x)x for a (0.1) such that fe(zt) = 0.5 (or, fz(x;) = 0.5) and for every a E M,, (or M2). there exists an i with I|a x, lll S E.
Then the Hausdorff distance between the decision boundaries obeys: H(M,, M,) S + E.
Proof. To prove Theorem 2, we bound the Hausdorff distance between the student's 's and teacher's decision boundaries using the given assumptions. We bound each term separately of the Hausdorff' distance (see Defintion 3).
We first bound supxEM, infueM, ||x- -u//r:
For any a E MM, by assumption (A3), there exists a CFE pair (x,, x() with teacher crossing point x E Me such that:
||a xillF S..
(17)
The segment xx] has length ||x,- (60-x/.. x||lp S a (Al). By Lemma and (A2) Exact distillation, the student's boundary M, intersects [x at some u E M,. Since x; and u² lie on [x,, x[]. their distance satisfies:
||x; - || S X - x(I1= S a.
(18)
Combining Equation 18 and 17:
||a ut F < ||a x; + |x; - uillF S s a.
(19)
Thus, infueM, ||a - u|IF= S E + a. Taking the supremum over a E M₁:
sup inf ||x - u S
(20)
xEM, uEM.
Next we bound Su infxeM, ||u luuxx -axxxle: X||x: From (AI), the distance between the student's cutpoint ui and the teacher's cutpoint x satisfies:
||u x; ||F
< ||x: x/IlF S
(21)
For any other E Ma;
x3 |F < ||u u'lle + ||ui x p S Seta.
(22)
Assuming the CFE pairs (x,,(( intersection points s-spredd (well spread) along the student decision boundary.
Since x E Me, we have:
inf ||u x|Ie $ ||u xille t xEM,
(23)
26

---

Taking the supremum over Ma:
sup inf ||u - x I t a. ueM, xEM;
(24)
Combining both bounds, the Hausdorff distance is the maximum of the two suprema:
H(M,, M.) max (e + a, e a) =
(25)
C Additional Experiments and Details
This appendix provides additional experimental details and results to supplement the main paper. In Appendix C.I, we describe the datasets used in our few-shot experiments and preprocessing choices. In Appendix C.2, include the prompt templates used to generate counterfactual explana- tions. Baseline methods are summarized in Appendix C.3, and complete hyperparameter settings detailed in Appendix C.4. Finally, Appendix C.5 presents extended results using the smaller DeBERTa-v3-xemall student and the Qwen2.5 5 model family.
C.1 Datasets Details
We evaluate CoD across six text classification benchmarks that span range of domains. For each k-shot setup, we sample a balanced subset from the processed training data, selecting k/2 examples per class. All experiments repeated across 5 random seeds, each with different sampled subset.
Yelp [70]: We use the Yelp Review Full dataset, filtering for reviews with at most 250 tokens and discarding neutral labels. Labels are binarized:
1-2 as negative and 4-5 as positive. The processed dataset contains 106,624 training examples, 1,000 for validation, and 7,074 for testing, with slightly imbalanced class distribution (64% negative).
IMDB [67]: We retain only reviews with shorter lengths. The original test and unsupervised splits are repurposed validation and test sets, respectively. The resulting data includes 782 training, 858 validation, and 1,578 test samples, with the test set unlabeled.
SST2 [65]: We use the full GLUE-provided training, validation, and test splits without modifi- cation. The train/val sets contain 67,349 and 872 examples, respectively. The test set has 1,821 unlabeled examples.
CoLA [68]: We adopt the standard GLUE splits of the CoLA dataset, yielding 8,551 training. 1, 1,043 validation, and 1,063 unlabeled test samples. The task is binary classification of linguistic acceptability.
Sentiment140 [66]: We filter the dataset exclude neutral tweets. The final dataset includes 1,598,400 training, 1,600 validation, and 359 test examples, with balanced label distributions.
Amazon Polarity [69]: We select examples with shortest length. The processed data includes 1,111 training and 113 validation samples, with roughly balanced sentiment labels.
C.2 Counterfactual Explanation Generation Prompt Templates
Here we provide prompt templates used for counterfactual explanation generation across datasets. Each prompt instructs the model to minimally modify given input to flip the class label (e.g., senti- ment or grammaticality) while preserving meaning and structure. We used go -opt-40-202---1 11-20 [64] for our CFE generation.
27

---

SST IMDB Sentimenties Amazon
You are an Al assistant tasked with generating counterfactual explanations for sentiment analysis. Given sentence and its true sentiment label, your goal is to make the minimal necessary change to flip the sentiment while preserving the structure and meaning much possible. For example, the input is: Sentence: "I love this movic."
True sentiment: Positive
A suitable counterfactual explanation would be: "I dislike this movie." Now, generate counterfactual explanation for the following sentence: Sentence: (sentence)
True sentiment: (sentiment)
Return only the counterfactual sentence, without additional information.
Yelp
You Al assistant tasked with generating counterfactual explanations for sentiment analysis of Yelp reviews.
Given a sentence (a Yelp review) and its truc sentiment label (positive or negative), your goal is to make the minimal necessary change to flip the sentiment while preserving the structure and meaning as much possible.
For example, if the input is:
Sentence: "This restaurant is fantastic, food was amazing!"
True sentiment: Positive
A suitable counterfactual explanation would be: "This restaurant is terrible, the food was awful!" Now, generate counterfactual explanation for the following sentence: Sentence: {sentence)
True sentiment: (sentiment)
Return only the counterfactual sentence, without any additional information.
CoLA
You an Al assistant tasked with generating counterfactual explanations for grammaticality judgment. Given a sentence and its true grammaticality label (Acceptable or Unacceptable), your goal to make the minimal necessary change to flip the grammaticality while preserving the structure and meaning as much as possible.
For example, if the input is:
Sentence: "She is going to the store."
True grammaticality: Acceptable
A suitable counterfactual explanation would be: "She is go the store." Now, generate counterfactual explanation for the following sentence: Sentence: {sentence)
True grammaticality: {sentiment)
Return only the counterfactual sentence, without any additional information.
C.3 Baselines Details
We compare CoD against three task- aaske knowledge distillation methods widely used for distillation. CoD uses k/2 original samples and their k:2 corresponding CFE (a total of k: shots) while the baseline methods are trained on original samples.
Knowledge Distillation (KD) [71]: A classical distillation approach where the student model learns to mimic the teacher's soft target probabilities using Kullback-Leible (KL) divergence. This method transfers predictive behavior but does not supervise intermediate representations.
Layer-wise Distillation (LWD) [13]: An extension of KD that additionally aligns the student's intermediate hidden representations with those of the teacher. This is typically done via mean squared error loss over corresponding layers, encouraging the student to internalize not only the final outputs but also the hierarchical feature representations of the teacher.
28

---

Table 4: Teacher accuracy (%) across datasets. Reporting Qwen2. 5.1. 5B and DeBERTa-v2 -3-- base when fine-tuned full training dataset for each benchmark. These teachers are used sources of supervision for student models during knowledge distillation.
Model
Amazon Polarity
CoLA
IMDB
SST2
Yelp
Sentiment 140
Swen2.5-1..3
88.5
83.0
94.3
93.7
95.4
86.1
DeBERTa-v3-base
86.7
87.5
93.8
95.8
95.6
86.8
Task-aware Layer-wise Distillation (TED) [12]: TED augments LWD with learned neural filters at each layer of both teacher and student models. These filters are trained to select task-relevant information from intermediate representations before computing the distillation loss. This selective transfer enables more effective compression by focusing on information critical to task performance.
C.4 Models and Hyperparameters
DeBERTa-V3 [25]. We fine-tune the teacher model using DeBERTaV3-basee initialized with
classification head for each target task. For the teacher, we use dropout rate of 0.1, linear learning rate decay, and train for 8 epochs with fixed learning rate of 2 x 10 and batch sizes of (32, 64). Optimization is performed using Adam with c = 10 10-6 31 = 0.9, and 82 0.98, without weight decay. For distillation, the student is initialized from DeBERTa-V3 - model. We search learning rates in the range [1 x 10 10--,105000 9,5 x 10 51, and use fixed batch size of 8 in our few-shot experiments. All student models are trained for 10 epochs using Adam with the same optimizer settings as the teacher. For KD and LWD baselines, we set the distillation loss weight to 20. For the TED baseline,
training with FP16 is used throughout
pre-trained DeBERTa-Vas v3-small or
the same hyperparameters for both the filter training and distillation phases, consistent with [12].
Qwen2.5 [2]. We use Quen/Qwen2. 5-1.551... as the teacher and Qwen/Qwen2. 5....5 5B as the student, both loaded from Hugging Face with sequence classification heads. We fine-tune using a batch size of 16 and train for 10 epochs. For KD and LWD baselines, we set the distillation loss weights to 20 and 5, respectively. All other settings closely follow the ReBERTaV3 setup, including the optimizer, learning rate schedule, and use mixed-precisson training.
All experiments are conducted on a server equipped with four NVIDIA RTX A6000 GPUs.
C,5 Additional Results and Discussion.
We provide results using the smaller DeBERTa-v3-xsmal] (22M parameters) student well as the full evaluation table for the Qwen2.5 family. Results for the smaller DeBERTa-v3-xema student shown in Table 5. While experiments using the Qwen2.5-2 5. 5B teacher and the Qwen2. 5-.0 0. 5B student are shown in Table 6. We also include the full fine-tuned teacher model accuracies across all datasets in Table 4, which are used as supervision targets during knowledge distillation. All experimental results are averaged over five runs, with the mean and standard deviation reported.
Overall, our findings corroborate the central insight that infusing CFEs into knowledge distillation significantly boosts model performance few-shot settings. For the smaller DeBERTa-ve -at---33-xaaalllmll student, we observe that the benefits of CFE infusion remain substantial across tasks, especially when 64. For example, on IMDB at k- = 8, KD + CoD improves from 74.3% to 89.3%, and LWD + CoD improves from 77.3% to 87.7% %, showing that even with a much smaller student, CFEs offer a powerful training signal. Similar patterns are seen on S5T2 and Amazon Polarity. While the performance gap narrows at higher k values, our method still matches slightly outperforms standard distillation, despite using only half as many real samples. These results highlight the scalability of CoD across student model sizes.
We also evaluate CoD on Qwen2.5 5 models, using Qwen2. 20ve2.5511.. 5-11..6 the teacher and Qwen2. 5-0.55 as the student. Results on CoLA, Yelp, Amazon Polarity: and IMDB show that our method consis- tently outperforms standard KD and LWD, particularly
few-shot regimes. On IMDB with k=8, KD + CoD reaches 80.0% vs. 67.86 87.8% for standard KD a remarkable 12.2 percentage point gain. Similarly, LWD CoD improves CoLAA accuracy by 8.3 points at 12128 (71.9% vs. 63.6%). With (k==), CoD boosts Yelp performance by 6.1 points for both KD (74.5% vs. 68.4% and LWD (74.6%
29

---

Table 5: Classification accuracy (# std) across datasets with varying total training sizes k. For COD, training data consists of k/2 standard and k/2 CFEs. Teacher model DeBERTa-v3 ----ase and student model DeBERTa-Ve v3-xsmall
Total Samples (k)
Dataset
Method
16
32
64
128
512
KD
0.628 +0.055
0.690
=0.034
0.766 +0.032
0.827 #0.021
0.835 #0.037
0.846 =0.009
Amazon Polarity
CoD
0.697 10.117
0.782 20.033
0.823 +0.018
0.844 +0.009
0.814 20.013
0.855 =0.018
LWD
0,660 +0,061
0.699 =0.044
0.777 =0.042
0.825 +0.015
0.839 =0.015
0.839 =0.013
CoD
0.712 +0.039
0.743 =0.051
0.811 +0.016
0.832 +0.015
0.830 =0.010
0.850 +0.013
KD
0.724 20.045
0.735
0.776 10.026
0.773 20.040
0.799 20.011
0. 0.806 20.004
Col
0.752 +0,042
0.766 =0.018
0.790 +0.012
0.799 +0.004
0.803 =0.008
0.817
CoLA
LWD
0.699 +0.042
0.744
0.755 +0.043
0.787 +0.008
0.803 =0.009
0.808 +0.008
CoD
0.685 +0.190
0.780 =0.018
0.790 +0.004
0.798 +0.007
0.802 +0.005
0.813 +0.003
KD
0.743 +0.070
0.849 =0.037
0.882 +0.032
0.904 +0.004
0.912 000
0.920 +0.004
Col>
0.893 +0.007
0.896 +0.007
0.900 +0.005
0.904 +0.005
0.910 =0.008
0.918 +0.003
LWD
0.773 +0.034
0.823 =0.041
0.876 +0.027
0.903 +0.008
0.915 +0.007
0.914 +0.014
CoD
0.877 40,022
0,888
20.006
0,900 +0.005
0,902 40.009
0.911 20.008
0.921 +0.001
KT)
0.591 +0.040
0.666
+0.030
0.754 +0.047
0.816 +0.024
0.861 =0.015
0.887 +0.033
CoD
0.685 20.112
0.763
0.829 40.028
0.850 20.015
0.862 20.016
0.905 #0.011
SST2
LWD
0.580 20.064
0.664
s0.024
0.726 20.036
0.818 20.019
0.847
0.912 20.005
CoD
0.658 10.107
0.675 =0.074
0.839 +0.017
0.841 +0.019
0.859 =0.016
0.877 =0.044
0.704 20.062
0.793 20.042
0.861 #0.011
0.887 40.004
0.907 =0.007
0.922 #0.008
+ CoD
0,759 20.086
0.758 =0.084
0.870 +0.008
0.889 +0.009
0.897 =0.009
0,920 =0.006
Yelp
LWD
0.714 +0.049
0.815 =0.028
0.870 +0.013
0.875 +0.012
0.907 20.066
0.925 =0.006
CoD
0.758 +0.069
0.757 =0.082
0.873 +0.012
0.884 +0.007
0.894 =0.009
0,919 =0.006
KD
0,580 =0.032
0.594 =0.026
0.634 +0.047
0.681 #0.046
0.740 =0.012
0.796 =0.013
CoD
0.573 +0.078
0.612 #0.064
0.721 +0.019
0.737 +0.030
0.767 #0.014
0.795 #0.006
Sent140
LWD
0,576 +0.038
0.585
#0.025
0.624 +0.029
0.684 #0.044
0.728 #0.035
0.799 =0.007
CoD
0.561 +0.064
0.592
0.681 +0.043
0.723 +0.025
0.763
0.773 +0.026
vs. 68.5%). These gains demonstrate the generality of our approach: it is effective even for decoder transformer families like Qwen2.5. 5.
Taken together, our findings affirm the broad applicability of CFE- CFE-infused distillation. The consistent improvements across datasets, model families, and student capacities support our central claim: CFEs are a powerful, data-efficient tool for improving teacher-student alignment in low-resource scenarios.
C.6 Ablation Results
We perform the following ablations: (1) On template designing and prompt choices. We vary prompt templates for generating CFES and observed that COD is robust to prompt choices, showing low standard deviation across variants and consistently outperforming the KD baseline few shot settings (see Table 7). One possible direction could be to use automatic prompt generation methods [72]. however these are typically more compute-intesn intensive. (2) Computational and Memory Require-
ments. We assess the computational efficiency of CoD under varying few-shot budgets & using the
codecarbon package [73] to track runtime and energy consumption (see Table 8). (3) Effect of supervision.
soffrlabeesupeerrsitonn To study the role of soft-labels in our approach, we remove or corrupting
the teacher's soft labels (see Table 9). Removing the soft label term (a 0) leads to a substantial drop in performance across all shot levels. Although CoD still improves over KD in this setting, the gains are significantly reduced. This highlights that the soft label calibration from the teacher is key contributor to the effectiveness of counterfactual explanation data. Additionally, when replacing soft labels with random values, performance degrades sharply, likely due to inconsistency with the hard labels, introducing conflicting supervision signals in the training objective.
30

---

Table 6: Classification accuracy (# std) of Qwen2. 5 across datasets with varying training sizes k. For CoD, training data consists of k/2 standard and k/2 CFEs. Teacher model is Qwen2. 5- 5B and student model Qwen2. 5-..5B.
Total Samples (k)
Dataset
Method
16
32
64
128
512
KD
0,681 #0,012
0.676
=0.023
0.668 +0.042
0.654 +0.032
0.676 =0.020
0.732 =0.014
CoD
0.683 20.016
0.686 20.018
0.697 #0.015
0.711 +0.020
0.736 20.017
0.757 #0.011
CoLA
LWD
0.681 20.012
0.657 20.031
0.678 #0.018
0.650 +0.039
0.636 20.029
0.712 =0.014
CoD
0.687
0.704 +0.010
0.714 +0.020
0.719
0.755 #0.013
+0.021
0.759 20.040
0.827 10.030
0.861 10.017
0.887 20.012
0,920 =0.010
+ CoD
0,745 +0.029
0.779
=0.048
0.828 #0.072
0.886 +0.007
0.883 =0.010
0.916 #0.008
Yelp
LWD
0.685 +0.019
0.777
#0.036
0.837 +0.027
0.876 +0.020
0.898 =0 00.008
0.920 =0.005
CoD
20.028
20.028
0.778
20.035
0.847 10.020
0.876 20.014
0.883 20.010
0.909 20.009
KD
0.589 +0.057
0.635
=0.044
0.706 +0.083
0.781 +0.033
0.807 =0.031
0.862 +0.013
Amazon Polarity
CoD
0,605 20.051
0.660 20.042
0.712 10.077
0.793 20.030
0.805 20.041
0.835 #0.021
LWD
0,589 20.057
0.628
20.096
0.680 40.052
0.779 40.026
0.823 20.027
0,858 50.015
CoD
0.607 40.051
0.662
0.692
0.795 20.041
0.823
0.853 #0.020
KD
0.678 20.054
0.758
0.817 10.057
0,890 20.017
0.903
0.926 20.003
CoD
0.800 +0.054
0.845
=0.061
0.877 +0.038
+0.014
0.912 #0.010
+0.003
IMDB
LWD
0.678 +0.054
0.740
+0.038
0.883 +0.014
0.906 =0.014
0.925 =0.003
+ CoD
0,800 +0.055
0.835
=0.0 00.048
0.869 #0.012
0.893 +0.013
0.909 =0.008
0.920 =0.007
KD
0.568 #0.061
0.621
=0.084
0.719 +0. 102
0.827 +0.038
0.878 #0.020
0.904 =0.010
CoD
0.578 +0.064
0.663 =0.081
0.767 +0.085
0.779 +0.137
0.870 =0.019
0.886 =0.005
SST2
LWD
0.568 #0.062
0.642 =0, 107
0.704 +0.068
0.825 +0.034
0.869 =0.026
0.890 =0.010
CoD
0,577 20.063
0.677 20.076
0.782 13.
0.779 20,085
0.792 10
0.878 #0.011
KD
0,586 +0.047
0.599
=0.047
0.641 +0.030
0.708 +0.027
0.756 =0.020
0.813 =0.010
CoD
0.556 #0.038
0.591
0.616 #0.055
0.711 #0.061
0.757
0.805 =0.010
Sent140
LWD
0.587 #0.051
0.596
0.639 #0.063
0.718 #0.038
0.765
0,805 #0.011
CoD
0.556 10.038
0.588 20.059
0.621 10.051
0.715 10.059
0.765 20.012
0.805 =0.008
Table 7: Comparison between KD and CoD variants across varying prompt templates. CoD is robust to prompt choices, showing low standard deviation across variants and consistently outperforming the KD baseline few shot settings.
Method
8
16
32
64
128
512
KD
0.617
0.712
0.757
0.820
0.848
0.899
CoD (v1)
0.719
0.781
0.821
0.827
0.853
0.892
CoD (v2)
0.754
0.789
0.841
0.872
0.890
0.872
CoD (v3)
0.738
0.778
0.819
0.835
0.856
0.901
CoD (v4)
0.734
0.783
0.830
0.834
0.883
0.888
CoD (mean)
0.736
0.783
0.828
0.842
0.870
0,888
(std)
0.012
0.004
0.009
0.018
0.016
0.010
31

---

Table 8: Compute and energy SST2 for KD+CoD and LWD+CoD. Accuracy, runtime, and energy
increases with k. LWD+CoD consistently costlier due intermediate representation alignment.
Method
k
Accuracy
Duration
CPU (kWh)
GPU (kWh)
RAM (kWh)
0.719
478.13
0.01336
0.00966
0.02479
16
0.781
488.04
0.01341
0.00972
0.02499
0.821
491.14
0.01382
0.01041
0.02547
KD CoD
64
0.827
547.58
0.01472
0.13620
0.02723
128
0.853
569.50
0.01639
0.01634
0,02952
512
0.892
705.21
0.03120
0.03593
0.04536
0.694
485.12
0.01263
0.01
0.02514
16
0.785
496.07
0.01394
0.01158
0.02572
0.832
517.78
0.01472
0.01245
0.02654
LWD + CoD
64
0.830
536.01
0.01515
0.01311
0,02775
128
0.835
668.52
0.01882
0.01670
0.02960
512
0.880
814.65
0.04621
0.04712
0.04902
Table 9; Effeet of soft-llbel calibration downstream performance. Removing the soft label term (a 0) leads substantial drop performance across all shot levels. Although CoD still improves over KD this setting, the gains are sigmificantly reduced. This highlights that the soft label calibration from teacher key contributor to the effectiveness of counterfactual explanation data. Additionally, when replacing soft labels with random values, performance degrades sharply. likely due to inconsistency with the hard labels, introducing conflicting supervision signals the training objective.
Method (SST2)
8
16
32
64
128
512
KD (no soft label, a=0)
0.553
0.622
0.697
0.712
0.791
0.815
+ CoD (no soft label, o==))
0.613
0.651
0.701
0.727
0.793
0.792
KD (random
label)
0.582
0.533
0.543
0.601
0.617
0.649
CoD (random soft label)
0.573
0.548
0.552
0.602
0.623
0.632
KD (default)
0.617
0.712
0.757
0.820
0.848
0.899
CoD (default)
0.719
0.781
0.821
0.827
0.853
0.892
32


Reviews:
[{"review_id": "9U3g3tlOOu", "reviewer": "Reviewer_PBmT", "summary": "The paper studies the problem of few-shot Knowledge Distillation with the aid of counterfactual explanations. They provide both theoretical justifications and experimental results to support the proposed method.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: I generally enjoy reading this paper: the problem is well-defined and formulated, the method is clear and elegant. I also find the theoretical analysis interesting: while the Fisher Information is kind of intuitive (Theorem 1), the Hausdorff Distance analysis is more interesting to me. The paper also have good experiments.\n\nWeaknesses: I have one question and one concern regarding the method. My question is why when we have more samples (k) for KD, the usage of CFE seems to be worsen the learning? Concern: CFE is defined based on the model's labels. What happen with the method when the number of labels is large (and the set of CFE has much higher variants), or when we consider generative LLMs?", "was_revised": true, "final_justification": "After consideration, I've decided to retain my original score.", "rebuttal": "We thank the reviewer for the thoughtful review and are delighted to hear that the reviewer has found our work “well-defined and formulated” and the method to be “clear and elegant”! \n\nBelow, we provide detailed responses to the questions asked:\n\n---\n\n**On the effect of more samples ($k$) in the distillation process**\n\nFirstly, note that for a fixed distillation dataset budget $k$, our method effectively uses half the number of real samples for distillation. E.g., for $k=8$, KD uses 8 randomly selected labeled examples, while our strategy only uses 4 labeled examples complemented with their counterfactuals (CFEs) and still significantly outperforms regular KD. Thus, it *achieves more with less* in the few-shot regime.\n\nIncreasing the number of samples ($k$) *does not necessarily worsen our approach,* but it could also make regular KD/LWD without CFE more competitive (reducing the performance gap). We hypothesize that this could be because as $k$ increases, real datapoints start to densely cover the decision space and hence we see diminishing returns from CFEs. Indeed, if the entire training data would be available during distillation ($k \\rightarrow \\infty$), the student’s performance is expected to keep getting better using KD/LWD.\n\n---\n\n**On new experiments for multi-class classification**\n\nYes, the strategy could be extended to multi-class classification setting. One way to extend this to multiclass could be as follows:\nConsider a multi-class setting with classes ${1,2,...C}$. For a data point $x$ belonging to a specific class $c$, we can generate several CFEs to flip the label to each of the other classes, excluding $c$.\n\nAs an initial demonstration, we implemented this variant of our proposed strategy on SST5 Dataset. Stanford Sentiment Treebank with 5 labels: very positive, positive, neutral, negative, very negative. We filtered it to $3$ classes to have a good separation among labels. When regular KD/LWD uses $k$ real samples, our proposed strategy effectively uses $k/3$ real samples + $2k/3$ CFEs (so basically $1/3$ of what regular KD/LWD uses).\n\n---\n\n$k=9$ \n\nKD:  $0.614$ KD+CoD (Ours): $\\bf0.622$\n\nLWD: $0.655$  LWD+ CoD (Ours): $\\bf0.696$\n\n---\n\nNotably, there could be several other variants of our strategy for the multi-class setting. For instance, one could also consider only one CFE per datapoint corresponding to the smallest perturbation leading to any changed class, or even considering the counterfactual of the counterfactual, etc. It will be an exciting direction of future research to study the performance tradeoffs among different variants of our strategy for the multi-class setting.\n\n---\n\n**On extension to generative LLMs**\n\nOne possible way to extend our approach to generative LLMs is to define counterfactual explanation as a minimal change to the input (prompt) that flips a chosen *property of the generated text* while keeping everything else as similar as possible. Formally, given a generative model $f$, a prompt $x$, and a binary attribute function $g(\\text{output})$  (e.g., sentiment, toxicity, factuality, topic relevance), a counterfactual explanation prompt $x^{\\star}$ would be a small semantic perturbation of $x$ such that the generated output $f(x^{\\star})$ flips the value of $g$. \n\nAnother possible extension is to rethink counterfactual generation in terms of *model sensitivity*: identifying minimal semantic perturbations to the input $x$ that cause large shifts in the output distribution or likelihood. For generative models, this corresponds to large changes in the sequence-level probability: $p_\\theta(y \\mid x) = \\prod_{t=1}^{T} p_\\theta(y_t \\mid x, y_{<t})$, where  $y = (y_1, y_2, \\ldots, y_T)$ is the generated sequence for  an input prompt $x$.  In this framing, CFEs could reveal parts of the input space where the model is uncertain, making them especially informative for distillation. While outside the scope of the current paper, this extension would be an interesting future direction, potentially inspiring novel and efficient pathways for data selection for a broad range of tasks, including distillation, supervised finetuning (SFT), post-training, etc.\n\nWe will include these discussions in the revised paper."}, {"review_id": "1nqB0XxlNN", "reviewer": "Reviewer_fJBT", "summary": "This paper presents a novel few-shot knowledge distillation framework called KD-FEW, which distills knowledge from large language models (LLMs) into compact student models under a low-resource (few-shot) setting. Unlike traditional KD methods that require many labeled examples or fine-tuned teachers, KD-FEW innovatively leverages LLMs in a black-box manner by prompting them to generate soft labels for few-shot instances.", "questions": "1. What is the degree at which KD-FEW relies on the choice of prompts? Is performance variable when various prompt templates are utilized? Would it be beneficial to have automatic prompt generation methods (e.g. AutoPrompt or prompt search with RL)?\n\n2. The authors are requested to provide the quantitative difference, which shows the enhancement of the quality of soft labels through the calibration step, and the consequent downstream performance.\n\n3. How does the size of soft label memory impact model performance as well as computational demands?\n\n4. Were the authors able to compare KD-FEW to (an alternative such as) RAG, wherein the data that is retrieved from a memory bank provides additional input to the generation process? The similarities between these approaches on the conceptual level deserve a straightforward comparison.", "limitations": "Although the paper makes some short comments about the black-box character of LLMs and the necessity of distillation, it does not thoroughly take into consideration page 3 limitations of the distillation process, such as:\n- The reliance on good, timely engineering.\n- The overhead involved the expense of storing or generating labels of the memory (e.g., API use).\n- The difficulties with applying the method to tasks with structured outputs (e.g., QA or generation).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The approach suggested in the paper should be able to work in a realistic set of constraints, which would amount to constrained access to labeled data and constrained access to gradient information in large language models (LLMs). The scheme--calibration, soft label memory, and knowledge distillation--is a technically valid and practically applicable pipeline.\n2. The paper is clearly and well-organized. All the components (the calibration, memory, and soft label learning) are presented in the context of logic as well as being intuitively understandable. \n3. The authors approach an important issue of efficiency and accuracy of large-model inference and propose a reasonable approach to extract knowledge based on proprietary or expensive APIs like GPT-4 (no fine-tuning is required).\n4. Soft label distillation is not an original idea, but to the few-shot task, adding black-box prompting and a memory-based distillation algorithm, is already an original contribution. \n\n**Weaknesses**\n1. The described improvements largely require the timely quality and template design. There is little account as to what sensitivity KD-FEW has had when it comes to prompt selection or whether a fully automatic prompting can be combined with the same, without affecting the performance.\n2. The soft label memory mechanism is computational and memory-wise extensible as the number of shots retrieved escalates. The complexity of the method does not undergo explicit assessment.\n3. Ablation is relatively light: For instance, what happens if calibration is removed? What if random soft labels are used instead of the memory buffer?\n4. Empirical testing can only be applied to English data, and the scope of its translation to a multilingual context and task-specific activities is thus ambiguous.", "was_revised": true, "final_justification": "After carefully reviewing the authors' responses, I find that they have addressed my concerns. Taking into account both the feedback from other reviewers and my own evaluation, I believe that my initial positive assessment remains appropriate. Therefore, I would like to maintain my original score.", "rebuttal": "We thank the reviewer for taking out the time to review this paper, and appreciate the positive opinion about our work!\n\n---\n\n**On template designing and prompt choices**\n\nThank you for the thoughtful question. We conducted an experiment varying four prompt templates for generating CFEs and observed that our method (CoD) is robust to prompt choices, showing *low standard deviation* across variants (e.g., ≤0.018) and *consistently outperforming* the KD baseline low shot cases (see table below (SST2 dataset)). This suggests CoD is not overly sensitive to prompt used. Automatic prompt generation (e.g., AutoPrompt, RL-based search) are typically more compute-intensive. Given our already strong and stable performance using simple manually designed prompts, such complex techniques may not be necessary. We will include this prompt robustness study and the varying prompts used in the updated version.\n\n| Method       | 8     | 16    | 32    | 64    | 128   | 512   |\n|--------------|-------|-------|-------|-------|-------|-------|\n| KD           | 0.617 | 0.712 | 0.757 | 0.820 | 0.848 | **0.899** |\n|                                   |       |       |       |       |       |       |\n| + CoD (v1)   | 0.719 | 0.781 | 0.821 | 0.827 | 0.853 | 0.892 |\n| + CoD (v2)   | 0.754 | 0.789 | 0.841 | 0.872 | 0.890 | 0.872 |\n| + CoD (v3)   | 0.738 | 0.778 | 0.819 | 0.835 | 0.856 | 0.901 |\n| + CoD (v4)   | 0.734 | 0.783 | 0.830 | 0.834 | 0.883 | 0.888 |\n|                                   |       |       |       |       |       |       |\n| CoD (mean) | **0.736** | **0.783** | **0.828** | **0.842** | **0.870** | 0.888 |\n| (std)     | 0.012 | 0.004 | 0.009 | 0.018 | 0.016 | 0.010 |\n\n---\n\n **On soft-label computational and  memory requirements**\n\nAs the sample $k$ increases, we observe a consistent improvement in model performance (accuracy) for both KD+CoD and LWD+CoD (see table below (SST2 dataset)). However, this comes at the cost of increased computational demands. We use `codecarbon` package to track energy usage across compute components, excluding the counterfactual generation step (which is API-based and not locally measured). Runtime and energy consumption (CPU, GPU, RAM) grow with larger $k$, indicating more processing and memory usage. LWD+CoD is more computationally intensive than KD+CoD at every $k$, due to its additional computational steps of aligning the teacher and students intermediate representations. We will include this ablation in the revised manuscript.\n\n**KD + CoD**\n\n---\n\n| k   | Accuracy | Duration (s) | CPU Energy (kWh) | GPU Energy (kWh) | RAM Energy (kWh) |\n|-----|----------|--------------|------------------|------------------|------------------|\n| 8   | 0.719    | 478.13       | 0.01336          | 0.00966          | 0.02479          |\n| 16  | 0.781    | 488.04       | 0.01341          | 0.00972          | 0.02499          |\n| 32  | 0.821    | 491.14       | 0.01382          | 0.01041          | 0.02547          |\n| 64  | 0.827    |  547.58            |    0.01472     |    0.1362         |    0.02723      |\n| 128 | 0.853    | 569.50       | 0.01639          | 0.01634          | 0.02952          |\n| 512 | 0.892    | 705.21      | 0.03120          | 0.03593          | 0.04536          |\n\n\n**LWD + CoD**\n\n---\n\n| k   | Accuracy | Duration (s) | CPU Energy (kWh) | GPU Energy (kWh) | RAM Energy (kWh) |\n|-----|----------|--------------|------------------|------------------|------------------|\n| 8   | 0.694    |    485.12          |   0.01263    | 0.01102     |     0.02514            |\n| 16  | 0.785    | 496.07       | 0.01394          | 0.01158          | 0.02572          |\n| 32  | 0.832    |   517.78           | 0.01472      | 0.01245          | 0.02654                  |\n| 64  | 0.830    | 536.01       | 0.01515          | 0.01311          | 0.02775          |\n| 128 | 0.835    | 668.52       | 0.01882          | 0.01670          | 0.02960          |\n| 512 | 0.880    |   814.65         |   0.04621      | 0.04712           |   0.04902               |\n\n---\n\n **On the quality of soft label calibration on downstream performance**\n\nThank you for the suggestion. We have added an ablation that isolates the effect of the soft labels. As shown in the table below, removing the soft label term ($\\alpha$ = 0) leads to a substantial drop in performance across all shot levels. Although CoD still improves over KD in this setting, the gains are significantly reduced.  This highlights that the *soft label calibration from the teacher is a key contributor to the effectiveness of counterfactual explanation data*. Additionally, when replacing soft labels with random values, performance degrades sharply, likely due to inconsistency with the hard labels, introducing conflicting supervision signals in the training objective. We will include this analysis in the revised paper.\n\n| Method (SST2)                      | 8     | 16    | 32    | 64    | 128   | 512   |\n|-----------------------------------|-------|-------|-------|-------|-------|-------|\n| KD (no soft label, α = 0)         | 0.553 | 0.622 | 0.697 | 0.712 | 0.791 | 0.815 |\n| + CoD (no soft label, α = 0)      | 0.613 | 0.651 | 0.701 | 0.727 | 0.793 | 0.792 |\n|                                   |       |       |       |       |       |       |\n| KD (random soft label)           | 0.582 | 0.533 | 0.543 | 0.601 | 0.617 | 0.649 |\n| + CoD (random soft label)        | 0.573 | 0.548 | 0.552 | 0.602 | 0.623 | 0.632 |\n|                                   |       |       |       |       |       |       |\n| KD (default)                      | 0.617 | 0.712 | 0.757 | 0.820 | 0.848 | 0.899 |\n| + CoD (default)                   | 0.719 | 0.781 | 0.821 | 0.827 | 0.853 | 0.892 |\n\n---\n\n**On the extension to generative LLMs and structured tasks**\n\nOne possible way to extend this to generative LLMs is to define counterfactual explanation as a minimal change to the input (prompt) that flips a chosen *property of the generated text* while keeping everything else as similar as possible. Formally, given a generative model $f$, a prompt $x$, and a binary attribute function $g(\\text{output})$  (e.g., sentiment, toxicity, factuality, topic relevance), a counterfactual explanation prompt $x^{\\star}$ would be a small semantic perturbation of $x$ such that the generated output $f(x^{\\star})$ flips the value of $g(f(x^{\\star}))$. \n\nAnother possible extension is to rethink counterfactual generation in terms of *model sensitivity*: identifying minimal semantic perturbations to the input $x$ that cause large shifts in the output distribution or likelihood. For generative models, this corresponds to large changes in the sequence-level probability: $p_\\theta(y \\mid x) = \\prod_{t=1}^{T} p_\\theta(y_t \\mid x, y_{<t})$, where  $y = (y_1, y_2, \\ldots, y_T)$ is the generated sequence for  an input prompt $x$.  In this framing, CFEs could reveal parts of the input space where the model is uncertain, making them especially informative for distillation. While outside the scope of the current paper, this extension would be an interesting future direction, potentially inspiring novel and efficient pathways for data selection for a broad range of tasks, including distillation, supervised finetuning (SFT), post-training, etc. \n\n---\n\n**On RAG comparison**\n\nWe thank the reviewer for highlighting the conceptual connection to RAG. However, our setup and that of  RAG systems differ fundamentally. Our method is focused on classification using supervision from soft labels and counterfactual explanations, whereas RAG augments data from a memory bank for generation using instruction-tuned models. A direct comparison would require switching to instruction-tuned LLMs capable of ingesting augmented data (e.g., \"Given this input and context, classify \"). However, the models we use are classification-specific (added classification head, similar to related works [1]), making such a comparison incompatible. Moreover, instruction-tuned LLMs used in RAG systems are typically very large (billions of parameters), which would make the comparison unbalanced and outside the scope of our focus on few-shot distillation under resource constraints. We believe bridging these paradigms is a promising direction for future work.\n\n---\n\n **On limitations**\n\nWe appreciate the reviewer’s for their suggestion. We will expand the limitations section to explicitly discuss the engineering overhead involved in our system.\n\n---\n\n[1] Less is More: Task-aware Layer-wise Distillation for Language Model Compression\nChen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, Tuo Zhao"}, {"review_id": "zuTSTZK80u", "reviewer": "Reviewer_uDSn", "summary": "This paper introduces COD that addresses the poor performance of standard methods when data is scarce. The core idea is to leverage Counterfactual Explanations (CFEs)—minimal input perturbations that flip the teacher model's prediction. Because these CFEs are located near the teacher's decision boundary, they provide highly informative training signals. The COD framework augments the small training set with these CFEs, enabling the student model to mimic the teacher's boundary more effectively. The work provides theoretical guarantees and experimental results showing COD significantly outperforms baselines in low-data regimes.", "questions": "N/A", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. Compressing a large language model to a smaller one with knowledge distillation is an important area.\n2. The paper provides both theoretical and empirical results to justify its method.\n\nWeakness:\n1. The practical significance of the proposed Counterfactual Explanation (CFE) generation method is questionable. For the simple classification tasks evaluated in the paper, such as sentiment analysis, simpler data augmentation techniques are likely sufficient, making the complexity of CFE generation potentially unnecessary. Conversely, for complex, open-ended tasks where LLMs excel (e.g., long-form generation or reasoning), the concept of a CFE—a minimal perturbation that \"flips a prediction\" —becomes ill-defined and difficult to apply. This suggests the method is caught in a difficult position: potentially redundant for simple tasks, yet inapplicable for complex ones, thus limiting its real-world impact.\n2. The setting of \"few-shot knowledge distillation\" seems rare nowadays. Can the authors provide some example scenarios in reality of this setting?", "was_revised": true, "final_justification": "My concerns are resolved.", "rebuttal": "We thank the reviewer for the review of the paper!\n\n---\n\n**On practical significance few-shot knowledge distillation and CFE-infusion**\n\nDistilling LLMs into smaller resource-constrained environments is of utmost importance. The unprecedented surge of AI comes with ever-increasing model size and complexity [1,4], leading to an unprecedented demand for energy. As models get larger, deploying them on resource-constrained environments such as mobile phones, Internet of Things (IoT), and edge devices gets particularly challenging due to their limited processing power, memory, and battery life [3,4]. To this end, this work addresses the broader question: can we develop efficient reduced-order models that can achieve high performance with minimal training data?\n\nIn fact, a recent survey [1] on LLM compression *highlights the significance and timely nature of our work*:  \n> \"Crucially, the success of **Knowledge Distillation (KD) in LLMs** hinges on Dataset Distillation (DD) techniques, which enable the **creation of compact, information-rich synthetic datasets** that **encapsulate the diverse and complex knowledge of the teacher LLMs**.\" [1]  \n\nOur CFE-infused approach embodies this principle by injecting precisely those high-value samples to better approximate the teacher’s decision boundary.\n\n---\n\n**Data-efficient machine learning** is also rapidly growing field of research [7]. Two primary motivations for data-efficient ML is: (i) data acquisition or labeling cost [5]; and (ii) training or fine-tuning costs. In this context, our work shows how it is possible to distill using so few carefully selected examples – essentially, how few examples hold the key to knowledge transfer. Few-shot distillation can have applications for on‑device personalization (keyboard, voice dictation), low‑resource or dialectal languages in under-served regions, clinical or legal settings, finance applications with privacy rules restricting data sharing data, deployment in hospitals/firms that have very few local examples.\n\n---\n\n**Classification with LLMs** still have various applications today beyond sentiment analysis, e.g., security/safety, toxicity detection, hate-speech/abuse detection, spam email detection, fraud detection, topic categorization, and tabular dataset prediction in high-stakes applications such as finance, healthcare, education, etc [2,3, 6, 8].\n\n---\n\n**On the extension to generative LLMs:** One possible way to extend this to generative LLMs is to define counterfactual explanation as a minimal change to the input (prompt) that flips a chosen *property of the generated text*  while keeping everything else as similar as possible. Formally, given a generative model $f$, a prompt $x$, and a binary attribute function $g(\\text{output})$  (e.g., sentiment, toxicity, factuality, topic relevance), a counterfactual explanation prompt $x^{\\star}$ would be a small semantic perturbation of $x$ such that the generated output $f(x^{\\star})$ flips the value of $g(f(x^{\\star}))$. \n\nAnother possible extension is to rethink counterfactual generation in terms of *model sensitivity*: identifying minimal semantic perturbations to the input $x$ that cause large shifts in the output distribution or likelihood. For generative models, this corresponds to large changes in the sequence-level probability: $p_\\theta(y \\mid x) = \\prod_{t=1}^{T} p_\\theta(y_t \\mid x, y_{<t})$, where  $y = (y_1, y_2, \\ldots, y_T)$ is the generated sequence for  an input prompt $x$.  In this framing, CFEs could reveal parts of the input space where the model is uncertain, making them especially informative for distillation. While outside the scope of the current paper, this extension would be an interesting future direction, potentially inspiring novel and efficient pathways for data selection for a broad range of tasks, including distillation, supervised finetuning (SFT), post-training, etc. \n\n---\n\nWe thank the reviewer for raising these important points; we will update the paper to include these clarifications and believe that the paper will be significantly strengthened as a result of this discussion.\n\n---\n\n**References**\n\n[1] Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions, Luyang Fang, Xiaowei Yu, Jiazhang Cai, Yongkai Chen, et al.\n\n[2] Smart Expert System: Large Language Models as Text Classifiers Zhiqiang Wang, Yiran Pang, Yanbin Lin.\n\n[3] Enabling On-Device Large Language Model Personalization with\nSelf-Supervised Data Selection and Synthesis, Ruiyang Qin, Jun Xia, Zhenge Jia, Meng Jiang et al.\n\n[4] Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques, Sanjay Surendranath Girija, Shashank Kapoor, Lakshit Arora, Dipen Pradhan, Aman Raj, Ankit Shetgaonkar.\n\n[5] Investigating Cost-Efficiency of LLM-Generated Training Data for Conversational Semantic Frame Analysis, Shiho Matta, Yin Jou Huang, Fei Cheng, Hirokazu Kiyomaru, Yugo Murawaki.\n\n[6] Less is More: Task-aware Layer-wise Distillation for Language Model Compression\nChen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, Tuo Zhao.\n\n[7] Foundations of Data-efficient Machine Learning, Siddharth Joshi · Baharan Mirzasoleiman, ICML Tutorial 2025\n\n[8] TabLLM: Few-shot Classification of Tabular Data with Large Language Models\nStefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, David Sontag."}]

Author rebuttal:

Meta-review: This paper proposes a way of distilling LLMs into smaller models by producing data points with minimal perturbations that change the LLM output. Reviewers agree on the clear formulation and the significance of the problem; concerns raised about significance were addressed in the rebuttal.
Decision: Accept (poster)