Knowledge Distillation Detection for Open-weights Models
Authors: Qin Shi
Amber Yijia Zheng
Qifan Song
Raymond A. Yeh
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge Distillation
Distillation Detection
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: We propose the task of knowledge distillation detection, which aims to determine whether a student model has been distilled from a given teacher, under a practical setting where only the student’s weights and the teacher’s API are available. This problem is motivated by growing concerns about model provenance and unauthorized replication through distillation. To address this task, we introduce a model-agnostic framework that combines data-free input synthesis and statistical score computation for detecting distillation. Our approach is applicable to both classification and generative models. Experiments on diverse architectures for image classification and text-to-image generation show that our method improves detection accuracy over the strongest baselines by 59.6\% on CIFAR-10, 71.2\% on ImageNet, and 20.0\% for text-to-image generation. The code is available at https://github.com/shqii1j/distillation_detection.

Knowledge Distillation Detection for Open-weights Models
Qin Shi"l, Amber Yijia Zheng Qifan Song', Raymond A. Yeh?
Deparrmeen Statistics, Purdue University DDeaarmennt of Computer Science, Purdue University (shi622, zheng709, qfsong, @purduue edu
"Equal contribution.
Abstract
We propose the task of knowledge distillation detection, which aims to determine whether a student model has been distilled from a given teacher, under a practi- cal setting where only the student's 's weights and the teacher's s API are available. This problem is motivated by growing concerns about model provenance and unauthorized replication through distillation. To address this task, we introduce a model-agnostic framework that combines data-free input synthesis and statistical score computation for detecting distillation. Our approach is applicable to both classification and generative models. Experiments on diverse architectures for image classification and text-to-image generation show that method improves detection accuracy over the strongest baselines by 59.6% on CIFAR-10, 71.2% ImageNet, and 20.0% for text-to-imant -imge generation. The code is available at https: //tthugb.ttt..tt/ http///ggii//isttlattiioo....me
1 Introduction
Knowledge distillation [20] transfers learned knowledge from larger teacher model to a smaller student model to reduce the computational and storage requirements. As opposed to directly training the smaller model from scratch alone, knowledge distillation leads to improved generalization the student model and has found success in several areas, including image classification [8, 20, 35, 43, 63], large language models [16, 40, 49]. and text-to-image generation [23, 28, 45, 56].
While the original intent of knowledge distillation is to build efficient models, distillation techniques lead an unintended consequence that one could potentially clone proprietary teacher models without permission. This raises concerns about the violation of intellectual property and how to attribute the source of machine learning models [33, 41, 50]. In this work, we aim to study whether is possible to detect if model has been distilled from a given teacher model, in short, knowledge distillation detection. Specifically, we consider the realistic scenario of detecting distillation for open- beent student models, without making any assumptions about the distillation method, training data, or requiring the weights the teacher model.
We also consider formulating the problem in multiple-choiee setting: given a student model and set of candidate teachers, predict which teacher was used for distillation. This setting has two advantages for the ease of experiment setup. First, it avoids the need to calibrate an absolute threshold, as the decision is based on comparing across candidates, which is more common real-world cases. Second, it produces results that are easier to interpret, one can directly identify the most likely teacher. Note, the detection (binary) setting can be recovered from this formulation by applying calibrated score function and threshold.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Prior works have considered related problems such as membership inference [5, 22] and out-of- distribution detection [29]. These works demonstrate that models may retain statistical traces of their training data or source model, even after distillation or fine-tuning. In particular, recent work on auditing data usage [22] and memorization in generative models [6, 47] shows that it possible to identify some aspects of model origin through probing-based methods. However, these approaches typically focus on detecting training data membership, rely on access to the teacher model, or assume architectural similarity. They do not directly tackle the problem of determining whether a model is distilled from another model that we are interested in.
To address knowledge distillation detection, we propose model -annossgg approach that only requires access to the student model's weights, without requiring the teacher model's weights or training data. Our approach consists of three stages: input construction, score computation, and decision making. In the first stage, we generate synthetic queries that probe model behavior, using data-free synthesis techniques tailored to the model modality. In the second stage, we extract statistical scores from the model's 's responses, capturing either point-wise discrepancies or set-level distributional alignment. Finally, we apply decision rule to determine whether the model has been distilled, based on these scores. This design is general and thus supports a wide range of models and tasks.
We conduct extensive experiments on both image classification and text-to-image generation tasks. For classification, we test variety of architectures (ResNet18, DLA, DPN, DenseNet, ResNet50, and Res5cX(150 and distillation methods (KD, RKD, DF). For text-to-image generation, we report on student models distilled from SD-v2.1, SDXL, and PixArt, including BK-SDM, AMD, DMD2, and SDXIL-Lightning. Across both domains, our method improves detection accuracy over the strongest baselines by 59.6% on CIFAR- 10, 71 71.2% on ImageNet, and 20.0% for text-to-image generation.
Our contributions are summarized follows:
We introduce the task knowledge distillation detection of open-weights student models, without access to the teacher model or training data.
We propose a model-agnostic approach consisting of input construction, score computation, and decision rule stages, adaptable across both classification and generative settings.
Empirical results on both classification and text-to-image tasks shows the effectiveness and generality of our method.
2 Related Work
Knowledge distillation (KD) an effective technique for compressing information from large pre-trained models into lightweight student models by aligning either the outputs or intermediate representations of the teacher and student [8, 37, 43, 48, 61, 63, 68). The concept was first introduced by Buciluă al. [3] and later popularized by Hinton et al. [20]. For classification tasks, knowledge distillation methods are commonly grouped into three categories: logit-based [20], feature-based [18, 43], and relation-basee approaches [10, 35].
While cussifiiaatio-focuseee KD emphasizes improving accuracy or reducing model size, distillation for diffusion models primarily aims to accelerate generation [23, 28, 31, 32, 42, 44 53, 56]. A typical approach is to train a student model to approximate the teacher's sampling trajectory, often formulated as an ODE, with significantly fewer inference steps. For instance, AMD [45] distills generative features from the teacher; BK-SDM [23] uses block pruning and feature distillation; and DMD [57] matches the output distribution directly via KI. divergence without sep-wise supervision. Unlike prior work that focuses on improving the performance of distillation, we study the inverse problem to detect whether one model is distilled from the other.
Data-free quantization and distillation. We review two different tasks that use sample synthesis to train a quantized/student model. One task is data-free model quantization [52]. which targets to quantize weights, activations, and even gradients to low-precision, to yield highly compact models without access to the training data. Data-free knowledge distillation [30] is a technique in which a smaller student model is trained to replicate the behavior of a teacher model without access to the original training data. Both tasks share the goal of transferring knowledge from pre- trained teacher model to target model using synthetic data. Common approaches include directly learning a fixed set of representative images [4, 27] or training a generator to produce synthetic

---

inputs [2, 11, 38, 52, 55, 59]. In our work, we also synthesize imputs as we do not assume access to the training data; however, our focused task different. Rather than aiming for quantized or distilled model, we generate these samples to collect statistics from the student/teacher model to perform knowledge distillation detection.
AI for good. Open-weight models can be freely shared and modified, amplifying both benefits and risks, which makes safety, transparency, and accountability crucial. Prior work on model immu- nization [64-67] develops defenses that reduce vulnerabilities of open-sourced image classification and diffusion models to harmful adaptations. Other efforts focus on safety interventions for large language models, such as red- red-teaming and watermarking [13, 24, 36], to mitigate misuse and enable provenance tracking. Our work is complementary: instead of altering model behavior, we address ac- countability by detecting whether student has been distilled from teacher, providing mechanism for provenance and trust in open-weights generative models.
3 Background
Knowledge distillation. Let f : be the teacher model and go : : - y be the student model, where X and Y' are the input and output spaces. Given a training dataset D {(x, y)}. knowledge distillation trains the student model by considering both the ground-truth supervision and the knowledge from the teacher model. The training objective can be formulated
(0)
(1-A)
A) h (go()), y) + X Loo (g)( fxx))
(m.y)E
where À E [0, 1] balances the loss term, haad denotes the loss with respect to the true label y, and eon compares the student's 's and teacher's 's outputs, e.g., using KL divergence over softened logits.
4 Knowledge Distillation Detection
Task formulation. Given open-weight distilled student model go, the goal of distillation detection is to identify which teacher model, from set F ( f .....; [(k)) j. was used to distill the student. For a realistic setting, we assume the following at prediction time: (a) that there is API access to each of the teacher models. (b) the student model is open- open-weiigh,, (c) the knowledge distillation algorithm is unknown, and (d) the data used to train or distill the models are unavailable.
More formally, this task can be formulated as multiple-choice problem for the K teacher candidates, ie., we aim develop a detection algorithm
A: {go X + }} + (1, (1.....k) K}
that maps from a given student model to the index of the teacher model. That is, if the algorithm returns k. then it is most likely that the teacher fi is used to distill ge. Compared with the task of deciding whether a student is distilled from a specific teacher, the multiple-choice setup is casier to interpret. It compares the student against set of candidate teachers and identifies the most likely source without requiring an explicit threshold; see Sec. 4.1. Moreover, our method naturally extends to the binary setting: one test whether student is distilled from given teacher by applying threshold to the score. Results this pairwise detection are discussed in Sec. 5.3.
4.1 Approach
At a high level, determining whether a student model has been distilled from a teacher model involves comparing their outputs. In our setting, however, no data is available; thus, the detection algorithm must first construct synthetic inputs. Once such inputs are obtained, both the teacher and student models are used to produce outputs, which are then compared to compute point-wise scores. These scores are then aggregated to make a final prediction. We illustrate our approach in Fig. 1. As the method's implementation may vary depending on the task, we first present the general framework, followed by task-specific choices in Sec. 4.2 and Sec. 4.3.
Prediction. Our proposed detection algorithm is based on score maximization to identify the teacher model most likely used for distillation. Specifically, the predicted teacher index is given by
k² arg max S(ge. (k,) P))
(3)
ke (6(11. (I....K)
3

---

Input construction
Score computation
Prediction
{Zzn yn )
Siga, I fuht P)
En s!
student
&
ang ma Slgo. P))
(1) { 8
teacher for
The predicted teacher is fk
Input data P
teacher ((R)
Logits
pair-wise score
Figure I: Knowledge distillation detection pipeline. The framework consists of three stages: input construction via generator G, score computation between the student and candidate teachers, and prediction by selecting the teacher with the highest aggregated score.
where S denotes general score function that quantifies the "alignment" between the student model Ge and the hthh teacher model f(k), evaluated over constructed imput set P )
Score computation. The score S plays central role in our method, which reflects how well the student aligns with each candidate teacher. We consider two types of score functions that capture this aligmment at different levels:
Point-wise score Spoint computes the discrepancy between the student and teacher outputs inde- pendently each input z,, E P. Let 8 y x y - Rzo denote general distance or divergence measure, e.g., KL divergence. For each input on. we compute a pairwise score 8n between the student and the g teacher defined as:
(go (2,) ((h) +
where 0 is small constant to ensure numerical stability. The overall point wise score is then given by the average over all constructed inputs:
N
1
Speim (go, S (k),
P)
(K)
(5)
N
12
Set-level score Sye measure the population discrepancy between and each f()) by comparing their outputs across the entire input set P. The score takes on the following form:
(ge f(8) (0000-((((oa)) ((00(())). (()) )
(6)
where r (>) x J) patterns over the set, such as closeness in the distribution.
a function that operates on the joint set of outputs capture alignment
Input construction. As discussed, set of inputs P (I,, ) is required to collect output statistics from both the teacher and student models. Intuitively, the method would benefit from inputs H,, that yield scores which are "separable", i.e., they can distinguish the true teacher model used for distillation from other candidate teachers. For example, inputs that are overfitted by both the teacher and student models may be a strong indicator of distillation relationship. Hence, our input construction strategy is designed to favor inputs with confident predictions from the student model, which tend improve the discriminability of the subsequent computed scores.
We will next describe the task-specific details for detecting distillation in image classification and text-to-image generation.
4.2 Image Classification Specific Details
In the classification setting, both the teachers (k) and the student go are multi-class predictors, i.e., X + A where C denotes the probability simplex over C' classes.

---

For point-wise score, we choose the pairwise score (k) to be computed using the KL divergence between the teacher and student's output. For a generated input æ, we compute the discrepancy as DKL (90(x))l, (() (a)), and convert into an aggregated point-wise score Spoint using the inverse formulation in Eq. (5).
For set-level score, we choose I' to be Aligned Cosine Similarity (ACS) [17], which computes the average cosine similarity between linearly projected student and teacher logits across a set of inputs:
1 N ACS(G, F) = N Σ
(HH)). (HFR)
(HG) m. (HFR),
are matrices of student where G (z) (xx) IT and [f(h) ..... (xx)]
and teacher logits, H IN N Mlyly Jy is the centering function and
R mig R=
| FFFF
(8)
projector for mapping centered F to centered G. Here, we use an orthogonal transforma- tion obtained by performing singular value decomposition (SVD). This set-level score captures distribution-level alignment between the two models in a compact and representation-awarr manner.
To construct the input set P, we aim to generate in-distribution inputs. We adopt mixup- based synthesis strategy [11, 60], where each generated input is a label-conditioned combination of latent representations from generator G i.e.,
i G
Ew,
Σ w. e(z,, Bi)
Here, the generator G. is parametrized with a deep net, with {2 ~ N(O I). labels (w C y interpolation weights [w:}]== ~ Dir(1) for mixup, and e(2;, yt) denoting one label-conditioned latent encoding, which linear layer that takes in the concatenation of z; and y:-
The generator G is optimized to produce data that has high confidence from Ie as follows,
min
(xo 8:) + LaNS (go, 2(0)),
(10)
where we use the cross-entropy loss for Lhand and LuNs is the batch normalization statistics (BNS) alignment loss [55]. Specifically, the BNS loss encourages the generated samples to match the internal activation statistics of real data by aligning the mean and variance at each batch norm layer:
(G (0) = - m1l |ooo - 1
(11)
1=1
where uf and of are the batch mean and variance of the generated data from Go at the I-th batch norm layer, and μ4, ai are the corresponding running statisties stored the pre-trained model go-
After the generator G3 is trained, we generate class-condiiional synthetic samples for each label without mixup by sampling new noise (N to construct the input
= (Go (c(z, vi) ~N(, vy)
(12)
4.3 Text-to-image Generation Specific Details
In the text-to-image setting, both the teachers {(()) and the student model ge are conditional generative models that map text prompt to an image, i.e., X' + I, where X' the text input space and I is the image space.
For point-wise score Spoins we choose to compute pairwise score (k) using LPIPS [62], which measures the semantic similarity between images generated by the student and teacher models for the same prompt and the same noise latent.
5

---

For the set-level score Sset. we choose to apply Centered Kernel Alignment (CKA) [25] with RBF kernels as a similarity index to measure the global alignment between student and teacher models. Concretely, we use pre-trained CLIP image encoder [39] to extract representation embeddings from images generated by both the student and teacher models, and apply CKA to compare the two distributions of embeddings. Compared to the ACS used for the classification task, the CKA is flexible to complex distributions and robust to high dimensionality, thus is more suitable for the text-to-image task.
Given two sets of image embeddings X and Y. each extracted from images generated from the student and teacher models, we first compute the RBF kernel matrices;
1 X exp (- (- 2002 Dx)=
Dx = diag xx )1 + diag (oxg(XXXTTT_ 2XXT,
(13)
and similar for Ky. With the centering function H In 1n1 the CKA score is defined as Tr( where Kx HKx, Ty=KK,
CKA(X,Y) |Xx|FF |
(14)
Lastly, generate the set of outputs from the teacher and student models, we need input set P. In the case of a text-to-image model, the input requires text prompts to specify what to generate. Here, we simply use the empty string the prompt. This choice ensures that the input remains in distribution to the model trained for classifier-free free guidance The training involves randomly omitting (replacing with an empty string) the text condition, usually with a drop rate of 0.05, to enable unconditional generation. As a result, the model learns to treat empty prompts as valid inputs, which makes them stable and suitable for evaluating the model's 's behavior in distillation detection.
5 Experiments
To evaluate the effectiveness and generalizability of our approach, we conduct experiments on detecting knowledge distillation in two tasks: image classification and text-to-image generation.
Evaluation metrics. As knowledge distillation is formulated as multi-class classification problem, we consider the following metrics:
1. Accuracy (Acc.): For all student models, compute the proportion for which the predicted teacher matches the true teacher. That is, we report how often our method correctly identifies the teacher model used for distillation.
2. Area Under the Curve (AUC): For each student model, we view the prediction as a one- ve-rest binary classification by treating the true teacher as the positive class and all others as negative. We compute the area under the ROC curve and report the average AUC across all student models. This metric reflects how well the scores rank the true teacher higher than the incorrect ones.
5.1 Distillation Detection on Image Classification
Datasets and model architectures. For image classification, we use two standard datasets: CIFAR- 10 and ImageNet [14]. On CIFAR- 10, we select four popular network architectures: ResNet-18 [19], DLA [58], DPN-92 [9], and DenseNet- 121 [21]. For each architecture, we first train a teacher model from scratch. Then, for each teacher model, we distill its knowledge into student models of all four architectures, resulting in 16 student models for each knowledge distillation method.
On ImageNet, we consider three architectures: ResNet-18 [19], ResNet-50 [19], and ResNeXt- 50 [51]. For each architecture, we use teacher model pre-trained on ImageNet- 1K and distill student models on ImageNet-100 100 into all three architectures, resulting in 9 student models for each knowledge distillation method. If the logit-matching is required during distillation, we only match the logits corresponding to the ImageNet- labels.
Model distillation methods. In image classification tasks, consider the original knowledge distillation (KD) [20], rational knowledge distillation (RKD) [35], and one-for-all KD (OFA) [18]. Each student model is distilled using one of these approaches from a given teacher model.
Baselines. We explore several different baselines following the general framework of our approach, drawing inspiration from the scores used in related works, such membership inference attacks,

---

Table I: Detecting distillation on CIFAR-IO. Each cell reports the mean and standard deviation of Acc//UUC over 10 random seeds. Bold indicates the highest Accuracy or AUC in each column (excluding Oracle). Note that when N = 1, the set-level scores cannot be computed.
Method
Inpot Size N
Average
100
Oncile
0.45 266
0.58 1000
0.65 2000
0.95
a.70 (n/ll.84
MIA Filte+ KL
043 26..
0.51 7.000000076 76
0.54
79 so.0
0.55
a.. (5,75,50 u0.02
000 Filter
042
0.500
0.55.0000)
29.00
0:54:oam1880000
MIA Filtr+ + CKA
0.38
064
0.72:0mm00900go
Filler CKA
0.72,ax:0.99..cc
MLA Filter MMD
s0.70000
0.29 0.292ox10.58000
0.38
Filer MMD
0.47 2170
0.47 20.7
0.28
Ours (KL)
0.62 a0072.0
0.82:0o:0.891.00
0.86.a.09.922.ozzz
0.77mm000994000
0.87
Ours (ACS)
0.75 0.75,oop017220.0
0.82 a.x00.74.0
0.88 1.000.77777
0.89 18990n00)777ccm a...
0.83 1...00..7.5.....
Table 2: Detecting distillation on ImageNet. Each cell reports Acc./AUC, Bold indicates the best value per column (excluding Oracle).
Input Size N
Method
Average
10
50
100
Dracle
0.77,as.sA.99.
0.84...p/00990 29.0
0.91 p..oo.c
0.75gp.1500.90.oom
MIA Filte+ KL.
0 (o
0.33 10 76
0.33_ s0.. 600.76000
0.33
0OD Fillrr + KL
0.62200
0.32 0.662zuns
0.32auue/0.633
0.3
0.33=tuc/00.62:sm
MLA Filter CKA
0.33
0.38::0me0.544000
0.2....o...55...c
0.39
00D Fiher CKA
0. 0.362mmm
0.34
0.36
0.36
(0..522
MIA Filter MMD
0.34 50200s
0.33 s.. 5020
0.33
0O0 Filerr MMD
0:33 a.s0A.s00 50,000
0.33
a.3
0.53aam00:500..o
0.33
Ours (KI.)
0.47 g0.1.
Ours (ACS)
a.58:0m00666.
0.60 0.60 1.666
0.70
0.79 0.79.om70885a.c
a.67au.u//.
although our application differs. Each baseline alters one component of our method while keeping the other elements unchanged. Baselines 1-2 focus on the input construction process, while baselines 3-4 concentrate on the score design. Please see the Appendix for more details.
1. Membership inference attack filter (MIA filter) [12, 54]: This baseline constructs the input set by identifying samples that resemble training data. We use random noise as input and split the samples into two disjoint subsets. For the first subset, we collect the student model's output logits and randomly assign Train and Test labels to them. A binary classifier is then trained to distinguish between the two. On the second subset, we collect logits from the student model and pass them through the trained classifier. Inputs classified as Train are retained for detection, while those classified as Test are discarded. KL divergence is used as the score function in the subsequent stage.
detector filter (OOD filter) [29]: This baseline constructs the input set by 2. filtering for in-distribution samples. It computes an energy score from the model's logits for random noise inputs. Samples with low energy (high confidence) are treated in-distribution and retained, while high-energy (low-confidence) samples are filtered out. As in our method, the KL divergence is used for scoring.
3. MMD-FUSE [1]: It computes the normalized log-sum-ext of Maximum Mean Discrepancy (MMD) statistics between student and teacher logits outputs under permutations.
4. CKA [25]: computes the representational similarity via centered kernel alignment. RBF kernels are applied to the logits of the student and teacher, followed by centering and computation of the Hilbert-Schmide Independence Criterion (HSIC).
Besides the baselines above, we also introduce Oracle, where we directly sample real inputs from the training distribution in the input construction stage and use KL divergence as the score function. This setup illustrates the performance when the original training data is available.
Results. In Tab. 1, present the results of distillation detection CIFAR-10 across varying numbers of synthesized inputs N = =Ppl. Set-level scores cannot be computed when N = 1. For point-level score, even with single input, performance, with an accuracy of 0.62 and an AUC of 0.75, consistently outperforming all baselines.
method using KL divergence achieves strong
As N increases, performance improves for both our methods and the baselines, reflecting the benefit of aggregating more input samples. By N = 50, our KI. and ACS variants reach AUCs of 0.94 and 0.77, respectively, and accuracy continues to improve. In contrast, basclines tend to plateau around
7

---

Grace
MA Fiter CSA.
600 Fiter NMD
1.0
Orde
MIA Fiter OKA
OCD Fiter MHD Ous
MIA Fiter
GOD Filer OKA
Ours
MiA Riter
60D fiter OKA
500 Riterre +
MIA Fiter MMD
000 Fiter +
MIA Filee NND
0.8
Acouracy
0.5
0.6
0.4-
0.4
0.2
0.2
0.0
0.0
Figure 2: Accuracy of knowledge distillation detection across different distillation methods. CIFAR- 10 (left) and ImageNet (right).
0.50-0.55 in accuracy and 7.730.80 in AUC. Notably, our methods exhibit greater stability and higher average performance across all settings, even outperforming the Oracle that uses real data.
Tab. 2 presents distillation detection results on ImageNet. As we can observe, our methods consistently outperform the baselines. Notably, with just 5 inputs, our KL-based method achieves an accuracy of 0.64 and AUC of 0.88, which is significantly higher than all baseline combinations, whose accuracy remains around 0.33 and AUC ranges from 0.47 to 0.76. As the number of inputs increases, our performance remains stable and continues to improve. reaching 0.75 accuracy and 0.92 AUC with 100 inputs. On average, our KL variant achieves 0.65/0.86, and ACS achieves 0.67/0.74, both substantially higher than the baselines, which mostly remain around 0.33-0.36 in accuracy.
In Fig. 2, present the accuracy of detection on CIFAR 10 and ImageNet across different distillation methods. Our method consistently achieves the highest accuracy across all distillation types, outper- forming all baselines and even the Oracle in CIFAR-IO. In addition, our method achieves the best performance under the standard KD setup compared to RKD and OFAKD, suggesting that detection more effective when the student model is trained using vanilla knowledge distillation.
5.2 Detect Distillation on Text-to-image Models
Model distillation methods and architectures. For the text-to-image models, we use 10 publicly available pre-trained student models distilled from larger teacher models using the following:
1. AMD [45]: a one-step student model distilled from Stable Diffusion 2.1 and a one-step student model distilled from PixArt.
2. DMD2 [56]: four-step distilled models derived from Stable Diffusion XI. (SDXL...
3. SDXL-Lightning [28]: student models with 1, 2, 4, and 8 sampling steps, all distilled from Stable Diffusion XL (SDXL).
4. BK-SDM-v2 [23]: improved variants of BK-SDM (base, small, and tiny), distilled from Stable Diffusion 2.1.
Baselines. For detection on text-to-image models, we consider two two categories of baselines: captioning- based input construction stage and alignment- based set-level scores. Baseline 1-2 focuses on compar- ing the input construction process, and 3-6 focuses on the score design.
1. BLIP-Bass [26]: We use the BLIP-Base model to generate captions for the unconditional genera- tion of the student model. These captions are then used as input for the detection pipeline.
2. GPT-2 [34]: Similar to BLIP-Base we use the ViT-GPT2 model to caption for the unconditional generation of the student model, and feed the resulting text for detection.
3. MMD-FUSE [25]: As in the classification setting, we calculate p-value as the score to measure the alignment between the student and teacher. Here, it is applied to the generated images instead of the logits.
4. ACS [17]: This baseline computes the average cosine similarity between the projected logits of the student and teacher across a set of generated samples.
5. CLIP [39]: We compute the cosine similarity between the CLIP image embeddings of student and teacher generations.
6. DINO [7]: We extract features from generated images using a pre-trained DINO backbone and compute the cosine similarity between the resulting embeddings of the student and teacher.

---

Table 3: Detecting distillation in text-to-image generation model. Each cell reports Acc//AUC
Input Size N
Method
Average
10
50
100
Blip Base CLIP
2007
a..7 2o.
0.81ax..e
a.84
0.440.co0.966.... 96
0.83
0.966
0.79 ao.w
Blip- Base DINO
0.79
1.80
10.00
0.81
10.00
0.79 a.79
Blip Base ACS
0.56 10005
1.58 17
0.59
1.59
0.57
1.57.0.o/1..000
0.57 0e
Blip Base MMD
0.67 100.96
1.58
20
0.62 oo.0.87.000
GPT-2 + CLIP
0.70 N0.8 s0.0
0.73 c0..9..00
0.8
0.81
25.
0.80 g00500001
0.77 oo. 40.91
GPT-2 DINO
0.81 20.1 20.1
0.79 20.00
008000.00/0 (1.9000004444 24
20.00
2.000 2900000
0. (.804e.o00..92..00
GPT-2++
0.5740.a10.882.00
0.52
0.54
200
0.54
GPT-2 MMD
0.6
1.08
0.51
0.50
10500..000666
1..8
Ours (LPIPS)
0.89 1.000
0.94 to to.0//100..0o g0.0
0.97
a097
100000000.900
0.9999900.
0.96 ro/.00100 L0.00
Ours (CKA)
0.67 go. 0,72
0,.33
0.78
0.80 oo .00
0.74gr 10.89
Table 4: Ablation study on input construction and score computation on classification.
Setting
CIFAR-10
ImageNet
Average
OOD filter ACS
1.56=a.os00.77=00000
0..72o..r/..522o.oor
0.472o.cee0.654o...
Synthetic Data + CKA
0.82
0.62 +0.04
0.72+u.oz/0.744p00
Synthetic Data + ACS (Ours)
0.83 a..0/0..55.
0.67 00.....00.74.000
0.75 o..os/0..7 +0000
Results. Tab. shows the performance of distillation detection for text-to-image generation models under varying numbers of synthesized inputs. Our methods consistently outperform all baselines in both accuracy and AUC across all input sizes, especially when we apply a point-wise score function. With just a single input, it achieves 0.89 accuracy and 1.00 AUC, already exceeding all competing methods. The strong performance is partly due to the use of CFG during distillation, where the unconditional generation is explicitly modeled, i.e., the empty string is used in the distillation process.
Next, the number of inputs increases, performance quickly saturates, reaching 1.00 accuracy and 1.00 AUC from N = 10 onward. This trend is maintained across all settings, with an average of 0.96/1. .00, indicating strong robustness and effectiveness. The best-performing baselines, such as GPT-2 + DINO and Blip-Base- + DINO, had around 0.80 accuracy, even with large input sizes.
For set-level scoring approaches, our method using CKA demonstrates clear improvements over other set-based basclines. Its accuracy and AUC increase steadily with N, reaching 0.80/0.99 at = 100. In comparison, other baselines using set-level score ACS MMD either stagnate or degrade when N > 10, indicating limited scalability. These results highlight the effectiveness of our design in leveraging larger input sets for improved detection.
5.3 Ablation and Discussion
Pairwise detection. To show that our multiple-choice setup can be extended a binary detection, we implement the HSIC test [15] to obtain a p-value the score for the text-to-image generation task with teacher models and 10 student models. Each of the student models paired with its true teacher (positive) and two random candidate teachers (negative). A pair is predicted as distillation if the p-value is below 0.05. As shown in Tab. 5, from the 30 seacher-student pairs, accuracy and FI score are computed from these binary predictions, and AUC is based on the scores across different thresholds for p-value. This method can achieve 0.86 accuracy and 0.91 AUC, which shows that generalizing to the pairwise testing setup is possible. We view this natural extension and valuable direction for future work.
Ablate inputs and scores. cation, reporting both accuracy and AUC. Using OOD-filtere inputs with ACS yields the weakest results, especially on ImageNet with 0.37 accuracy and 0.52 AUC, indicating poor alignment with student model behavior. Replacing the input with our synthetic samples and using CKA improves performance significantly, reaching 0.72 accuracy and 0.74 AUC on average. Our full method
We ablate input construction and set-level scoring in Tab. 4 for classifi-

---

0.4
Teacher Model
0.04
Teacher Model
Independent Model
Independent Model
Kl Divergence
0.3
0.03
ACS
0.02
0.1
0.01
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 3: KL divergence (left) and 1 ACS (right) between the student and teacher (or independent model) outputs function of the distillation weight X. The total loss used to train the student is defined in Eq. (1), combining a hard cross-entropy term and a soft KI. divergence term.
Table 5: Performance of distillation detection using the HSIC test the text-to-image task.
Input Size N
Acc.
AUC
F1
10
0.74 = 0.05
0.76
0.07
0.43 + 0.09
50
0.86 + 0.04
0.91 = 0.07
0.80 + 0.06
100
0.80 = 0.04
0.94 0.04 0.75 + 0.05
achieves the best overall results 0.75/0.75, outperforming the strongest ablation by 0.03 in accuracy while maintaining robust AUC. These results confirm the importance of both input and score design.
Impact of knowledge transfer strength. Intuitively, the quality/strength of the distillation should impact the effectiveness of our approach. In Fig. 3, we show how our detection method depends on the strength of knowledge transfer during distillation. The parameter controls the contribution of son in the distillation loss defined in Eq. (1), with higher À placing greater weights on matching the teacher. We use KL divergence and 1- ACS as the scores measuring the distance between the two models. We compute the scores between the outputs of the student model and either the teacher model or an independently trained model with the same architecture as the student. When X < 0.5, the scores between the student and its teacher could be larger/comparable to those between the student and an independent model with the same architecture, and our detection rule will fail such cases. As À increases, the student is more strongly regularized to align with the teacher, resulting in reliably smaller scores and improved detection performance.
Conclusion
We introduce the task of knowledge distillation detection that aims identify whether student model has been distilled from a given teacher from only the student's weights and the teacher's API access. Our model-agnostic framework combines data-free free input synthesis with score maximization to make a prediction. The proposed method works on both classification and text-to-image generation. Experiments across diverse architectures and distillation methods show that our approach consistently outperforms strong baselines without requiring access to training data or teachers' weights. While detection may be less reliable when the student inherits only limited influence from the teacher, performance remains strong in typical distillation scenarios. Looking ahead, extending the framework free-form inputs in diffusion and large language models represents promising next step, supported by the model-agnostic nature of our approach.
10

---

Acknowledgment
This project is supported in part by an NSF Award #2420724 and the Ross-Lynn Research Scholar Grant. Song's 's research is partially supported by the NVIDIA Academic Grant Program Award.
References
[1] Felix Biggs, Antonin Schrab, and Arthur Gretton. Mmd-fuse: Learning and combining kernels for two-sample testing without data splitting. In Proc. NewrIPS, 2023, 7
[2] Kuluhan Binici, Shivam Aggarwal, Nam Trung Pham, Karianto Leman, and Tulika Mitra. Robust and resource-efficient data-free knowledge distillation by generative pseudo replay. In Proc. AAAI, 2022. 3
[3] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proc. ACM SIGKDD, 2006. 2
[4] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami. Michael Mahoney, and Kurt Keutzer. Zeroq: novel zero shot quantization framework. In Proc. CVPR, 2020. 2
[5] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herhert- Katherine Lec, Adam Roberts, Tom Brown, Dawn Song. Ulfar Erlingsson, al. Membership inference attacks from first principles. In Proc. S&P, 2021. 2
[6] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Nicholas Carlini, Matthew Jagielski, and Nicholas Carlini. Extracting training data from diffusion models. Proc. S&P, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, Proc. ICCV, 2021. 8
[8] Defang Chen, Jian- Ping Mci, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. Proc. CVPR, 2022. 1,
[9] Yunpeng Chen. Jianan Li, Huaxin Xiao, Xiaojie Jin, Shuicheng Yan, and Jiashi Feng. Dual path networks. In Proc. NeurlPS, 2017. 6
[10] Yuntao Chen, Naiyan Wang, and Zhaoxiang Zhang. Darkrank: Accelerating deep metric learning via cross
sample similarities transfer. In Proc. AAAI, 2018. 2
[11] Kanghyun Choi, Deokki Hong, Noseong Park, Youngsok Kim, and Jinho Lee. Qimera: Data-free quantization with synthetic boundary supporting samples. In Proc. NeurIPS, 2021. 3,5
[12] Christopher A Choquette-Choo Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only
membership inference attacks. In Proc. ICML, 2021. 7
[13] Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, et al. Scalable watermarking for identifying large language model outputs. Nature, 2024. 3
[14] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In Proc. CVPR, 2009.
[15] Arthur Gretton, Kenji Fukumizu, Choon Teo, Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test independence. In Proc. NeurIPS, 2007. 9
[16] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. Proc. ICLR, 2024.
[17] William L Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal statistical laws of semantic change. In Proc. ACI., 2016. 5,8
[18] Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-all: Bridge the gap between heterogencous architectures in knowledge distillation. In Proc. NewrIPS, 2023. 2,
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, 2016. 6
[20] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0253: 2015. 2,

---

[21] Gao Huang, Zhuang Liu. Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proc. CVPR, 2017. 6
[22] Robin Jia, Shiori Sagawa, Sang Michael Xie. Haoran Zhang, and Percy Liang Wang. Auditing data provenance in text-generation models. In Proc. NeurIPS, 2022. 2
[23] Bo- Bo-Kyeong Kim, Hyoung-Ky Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion. In Proc. ECCV. Springer, 2024. 2,
[24] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, lan Miers, and Tom Goldstein. A watermark for large language models. Proceedings of Machine Learning Research (PMLR), 2023. 3
[25] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proc. ICML, 2019. 6, 6,7,8
[26] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman -image pre- pre-training for unified vision-languee understanding and generation. In Proc. ICML, 2022. 8
[27] Yuhang Li, Feng Zhu, Ruihao Gong, Mingzhu Shen, Xin Dong, Fengwei Yu, Shaoqing Lu, and Shi Gu. Mixmix: All you need for data-free compression are feature and data mixing. In Proc. ICCV, 2021. 2
[28] Shanchuan Lin, Anran Wang, and Xiao Yang. Scxl-lightning: Progressive adversarial diffusion distillation.
arXiv preprint arXiv:2402.13922 2024. 1, 2,2,8 8
[29] Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-o--distriuution detection. In Proc. NeurIPS, 2020. 2,7
[30] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-frec knowledge distillation for deep neural networks. In Proc. AISTATS, 2018.
[31] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint (..Xx::2111002.00 2021. 2
[32] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proc. CVPR, 2023.
[33] Cade Metz. Openai believes deepseek distilled its data for training: Here's what know about the tochnique. The New York Times, 2025. Accessed: 2025-04-01.
[34] Nipconnect.
Vit-gpt2 image captioning.
https://huggingfacee http:///.a...ccccooococcccec/cc/
vit- Srr-ggpt-immnn wtt---imageccptpinnnggg.o 2022. 8
[35] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proc. CVPR, 2019.
2, 6
[36] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. In Proc. EMNLP, 2022. 3
[37] Mary Phuong and Christoph H Lampert. Distillation-basee training for multi-exit architectures. In Proc. ICCV, 2019. 2
[38] Biao Qian, Yang Wang, Richang Hong, and Meng Wang. Adaptive data-free quantization. In Proc. CVPR, 2023. 3
[39] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proc. ICML, 2021. 6,
[40] Leonardo Ranaldi and Andre Freitas. Aligning large and small language models via chain-of-thought reasoning. In Proc. ACI., 2024. 1
[41] Siladitya Ray. OpenAl Believes DeepSeek 'Distilled' Its Data For Training Here's What To Know About The Technique. Forbes, 2025. Accessed: 2025-04-01.
[42] Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang. Jie Wu, Pan Xie. Xing Wang. and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. In Proc. NewrIPS, 2024. 2
[43] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, arXiv preprint arXiv:1412.60.. 2014. 1,2
12

---

[44] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In Proc. ICLR, 2022. 2
[45] Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blatimann, Patrick Esser, and Robin Rombach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. In SIGGRAPH Asia 2024 Conference Pupers, 2024. 1, 2,
[46] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Proc. JCML, 2023.
[47] Om Thakkar, Nicholas Carlini, Matthew Jagielski, and Florian Tramer. Understanding and controlling memorization in diffusion models. arXiv preprint arXiv:22/2.03860, 2022. 2
[48] Frederick Tung Greg Mori. Similarity prnserrity knowledge distillation. Proc. ICCV, 2019.
[49] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. Pruc. ACL, 2023.
[50] Winston & Strawn LLP. Is ai distillation by deepseek ip theft? httpa:// winston com/en/ insights-nevet
ai www.diiiaaiioob----eeeeeaeeleeee beepsee- ip- theft, 2024. Accessed: 2025-04-01.
[51] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming Hc. Aggregated residual transforma- tions for deep neural networks. In Proc. CVPR, 2017. 6
[52] Shoukai Xu, Haokun Li, Bohan Zhuang, Jing Liu, Jiezhang Cao, Chuangrun Liang, and Mingkui Tan. Generative low-bitwidth data free quantization. In Proc. ECCV, 2020. 2,
[53] Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and- and-play accelerator. In Proc. NewrIPS, 2024. 2
[54] Samuel Yeom. Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting, In Pruc. CSF, 2018. 7
[55] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proc. CVPR, 2020.
5.5
[56] Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and Bill Freeman. Improved distribution matching distillation for fast image synthesis. In Proc. NeurlPS, 2024. 1,
2,
[57] Tianwei Yin, Michaäl Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proc. CVPR, 2024. 2
[58] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proc. CVPR, 2018. 6
[59] Shikang Yu, Jiachen Chen, Hu Han, and Shuqiang Jiang. Data-free knowledge distillation via feature exchange and activation region constraint. In Proc. CVPR, 2023. 3
[60] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Proc. ICLR, 2018. 5
[61] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proc. ICCV, 2019. 2
[62] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proc. CVPR, 2018. 5
[63] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proc. CVPR, 2022. 2
[64] Amber Yijia Zheng and Raymond A Ych. Imma: Immunizing text-to-image models against malicious adaptation. In Proc. ECCV, 2024. 3
[65] Amber Yijia Zheng and Raymond A Yeh. Multi-concept model immunization through differentiable model merging. In Proc. AAAI, 2025.
13

---

[66] Amber Yijia Zheng, Chiao-An Yang. and Raymond A Yeh. Learning obstruct few-shot image classifica- tion restricted classes. In Proc. ECCV, 2024.
[67] Amber Yijia Zheng, Site Bai, Brian Bullins, and Raymond A. Yeh, Model immunization from a condition number perspective. In Forty-second International Conference Machine Learning. 2025. 3
[68] Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. In Proc.
NeurlPS, 2018.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper clearly defines the task of knowledge distillation detection and proposes modular, model-agnostic framework applicable to both classification and text-to- image models. These claims are supported by detailed methodology and comprehensive experiments.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide the limitation that detection may be less reliable when the student inherits only limited influence from the teacher in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification:
We provide the implementation details in Sec. A2 reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide the implementation details in Sec. A2 reproduce the results and will release the source code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. htppppubuuuideeeeeeepppppppppeee h
/CodeSubmic wwwtemissiinnooooo onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental settings in Sec. 5 and more details can found in Sec. A2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All results reported in the tables are averaged over 10 runs with different random seeds. Both the mean and standard deviation are provided to reflect the variability and statistical reliability of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the compute resources in Sec. A2 for the generator training, inference, and the score computation.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https / http///eerrppp.eee www.eeurpsss.ppupii/peeeeceeeeee cc/public/Bthhccsuudddde
Answer:
[Yes]
Justification:
We confirm that the research conducted with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please find broader impact in Sec. A4.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not foresee such risks for this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited all the existing assets in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new methods and code as part of this work. While they publicly released at submission time to preserve anonymity, we commit to releasing them upon acceptance. This will include the training and inference code of the generator and the code for computing all the scores.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: the core method development in this research does not involve L.MSs as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... wwc/onneeeenncs///000 for what should or should not be described.
21

---

Appendix
The appendix is organized follows:
In Sec, AI, we report additional quantitative results.
In Sec. A2, we provide implementation details of our approach for reproducibility. The code will be released.
In Sec. A3, we document the details of the compared baselines.
In Sec. A4, we discuss broader impacts.
A1 Additional Results
Table A1: Point-wise scores between the student models and each candidate models using KD method on CIFAR-10. The reported values are the mean + standard deviation across runs.
Teacher
Student
KNNH (Landidao)
DLAICinciiie)
Ours (Melhad)
KNIRUandiae))
DEAICannidat) DPN9II indidate UNFIRCanddde))
Filler + KT. (Midhol)
0.12-0.11
0.02
0.11
0.
0.11 0.04
RNI8
DLA
0.05 0.05
0.15
0.25
0.89
0.00
0.3 000
29720.50
0.14
DPN92
0.00
0.13
0.02 0.04
0
0.05 0.02
0.06 0.05
2.9:0.44
2 0.94
0.28
0
0.5
RNIS
0.01 0.00
0.00 0000
DJ9 8 2006
0.04
0.12
0.22 200
2.19:0000
0.00
DLA
DLA
0.03
0.00 - 0.00
007 0.04
0.05 0.m.
0.35 0.14
0.22 + 0/22
0.4
0.45 0.19
0 0.04
DN121
0.19
0,00 0.00
0.33 0.22
0.03
0
0.05 001
2.11
0.12 0.07
RNIS
0.0
2.16 0.31
a.0 0.05
0.64 0.21
0.21 0.00
0.31 0004
1.0
0.21 0.01
0:43
1.24
a000
0.38
042
1.14
0.31
DPN92
1.83 0.10
1,66 0.70
0.00 0.00
0.28 0.19
0.34
0-46 + 008
0.5
0,32 0.05
0.00 0.01
0.034 0.12
0.06
005
0.9
0,07
0.03
0.34:006
23002
$
0.09
1.64
0,03
DLA
0.02
0.06 0.04
1.02 0.21
0.000
0.00
0.13 = 003
x
0.09 0.07
DNI21
DPN92
0:25
0.19 0.13
0.14 0.00
0.00
0.00
0:0
1.01
0,66 0.04
0.11
0.17
0,00
0.04
0.17
1
0,10
In Tab. AI, we compare the scores of our method and the best baseline across all student models on CIFAR-10. According to Tab. A1, we observe that some student models are easier to identify than others. For example, students distilled from RNI8 are relatively easy cases, where both our method and the baseline achieve correct detection with large score gaps between the true teacher and other candidates.
contrast, harder cases such as RN18 students distilled from DN121 show much smaller differences, sometimes as low as 0.01 between the true teacher and the next closest candidate. Intuitively, the architectural similarity between the student and teacher models may influence the difficulty of detection.
Table A2: Point-wise scores between the student models and each candidate models using KD method. The reported values are the mean + standard deviation across runs.
Ours (Method)
MIA Filter KL (Method)
Teacher
Student
RNI8 (Candidate)
RN50 (Candidate)
RNX150 (Candidare)
RN18 (Candidate)
RN50 (Candidare) RNX (Candidate)
RNI8
2.41 + 1.07
7.01 + 1,28
6.88 i 1,18
0.13 0.01
0.29 = 0.02
0,70 0.00
RN18
RN50
0.60 0.40
2.12
4.43
0.15 0.01
0.32 0.01
0.72 : 0.03
RNX150
1.90 1.07
0.95
0.18 0.01
0.68 0.02
RNI8
9.45 0.44
10.53 1.47
0.36 = 0.01
0,51 0.04
0,92 = 1.47
RN50
RN50
8.119
5.48
0.26 0.01
0.45 0.02
0.79 a 0.03
RNX150
5.21 1.72
3.13 1.84
3.67 + 1.47
0.34 0.01
0.51
0,78 0.04
RN18
9.91 1.33
11.22 1.07
10.32 1.26
0.45 0.03
0.78 0.05
1.01 0.07
RNX150
RN50
9.85
1.48
10:45
1.75
8.17 4: 0.95
0.36 0.02
0.42 0.02
0.76 0.03
RNX150
8.65
1.69
9.39
a 1.30
6.98 + 1.98
0.52 0.01
0.68 = 0.02
0,82 a 0.06
similar conclusion can be drawn from Tab. A2 on ImageNet. For easy cases, such as students distilled from RN18, 18, the true teacher can be accurately identified with large score gaps over other candidates. However, for harder cases, such as RN18 students distilled from RN50 or RNX150, both method and the baseline tend to incorrectly identify the teacher as RN18. These hard cases often involve significant architectural differences between the student and teacher models, suggesting that architectural mismatch contributes to detection difficulty.
In Tab. A3, we report the scores of our method and the best baseline across all student models on the text-to-image generation task. We use 1 LPIPS in our method, which shows that the similarity
22

---

Table A3: Point-wise scores between the student models and cach candidate models for texl-to-image models. The reported values are the mean standard deviation across runs.
(Melhod)
DINO (Melhod)
Teacher
Student
SD.v2.1 (Candidate) SDXL (Candidate) PLxAnt (Candidate)
SD-x2.1 (Candidane) SDXL (Candidale)
FixArt (Candidate)
BD-
0.6910.01
0.28:10.00
0.19
100000.000
0,98100.00
0,97.0.00
SD+22.1
BK-SDM--2-s
0.68:0.00
0.28:20.00
0.19:0000
1.00:00.00
0.98:00:00
0.9::0000
BK-SDM-v241
0.70+0.01
0.27++0.00
0,18+0.00
1,00+0.00
0:98++0.00
0,7+00.00
AMD
0,56+H0.01
0.23++0.01
0.15+0.01
1,00+0.000
0,98+0000
0,97+0.00
DMD2
0.24:10.01
0.32:20.01
0.20::0.00
0.98:20.00
0.99::0.00
0.98:00.00
SDXI-I
0.19++0.00
0.24+0.01
0,22+0000
0,93++0.00
0,96++0.00
0.97++0.00
SDXL
SDXL-L-2
0.25:00001
0.41=00.01
0.18:0.00
0.99:10.00
0,98:10.00
0,9810.000
SDXL
0,26++00000
0.48:#0.01
0,21+=0.00
0.98=40.00
0.99+00.00
0.98+0.000
SDXL-L-8
0,23,10.01
0.55
0,214:0.00
0.97.10.00
0,99110.00
0.98.10.00
PixAn
AMD-PixArt
0.19:0:00
0.37:40.00
0,43+:0.00
0.98:#0.00
0.98:10.00
0.99:0:000
between each student and their true teacher is the highest. For instance, BK-SDM-v2-b has 0.69 0.01 to the teacher SD-v2.1, but much lower values to SDXL and PixArt. However, the baseline scores are nearly saturated across all candidates, typically 0.97-1.00 and making teacher identification infeasible. This saturation indicates lack of discrimination in the embedding space used by the baseline.
A2 Implementation details
Teacher and student models in classification task. We use ResNet-18, DLA, DPN92, and DenseNet121 as the backbone of the teacher and student models on CIFAR-10. We train the teacher/student models from scratch with these architectures first. Here we apply the same training strategy: stochastic gradient descent (SGD) with a learning rate of 0.01, momentum of 0.9, and weight decay of 5e-4; training length of 40 epochs with a batch size of 64. We apply the OneCycle learning rate scheduler, with maximum learning rate of 0.1, computed over 45000/64 steps per epoch and updated at every training step.
On ImageNet, since the teacher models are available on torchvision, we only need to train student models. The models are trained for 200 epochs with a batch size of 256. We SGD with learning rate of 0.1, momentum of 0.9, and weight decay of lc-4. A cosine annealing learning rate scheduler applied, with the minimum learning rate set to 0.0 and a total decay period over the whole epoch.
For the teacher model training, we use only the standard cross-entropy loss for classification. For student models trained with knowledge distillation (KD), only the KL divergence loss is used. In particular, for the RKD and OFA methods, we remove the hard-label supervision entirely and replace with KL loss, so that the student models are trained solely under the guidance of the teacher.
In the RKD method, the total loss is a weighted combination of 0.3 KL loss and 0.7 RKD loss. Following the original RKD paper, we set the weight of the distance loss to and the weight of the angle loss to 2. RKD loss is computed using the features after the final average pooling and flattening. To ensure dimensional compatibility between teacher and student models, projection layer for student models whose feature dimensions do not match those of the teacher. In the OFA method, the loss is weighted combination of 0.3 KI. loss and 0.7 OFA loss. For models with different architectures, we manually divide the hidden representations into four segments and select after-activation latents for computing the OFA loss.
insert fully connected
Input construction in classification task. Our generator takes as input a noise vector and class label, and produces a synthesized image conditioned on the label. Note that this label is first embedded using a learnable embedding layer. The normalized embedding is passed through a linear projection layer before being fed into the image generation pipeline.
On CIFAR-10, 10, the projected embedding is transformed via fully connected layer into a spatial feature map of shape 128 x 8 x 8. This feature map is processed through three convolutional blocks, with bilinear upsampling applied between the blocks to restore spatial resolution:
Block 0: A single BatchNorm layer.
Block 1: A convolutional layer with 128 channels, followed by BatchNorm and LeakyReLU.
23

---

Two convolutional layers: one reducing to 64 channels and another producing the final image with the desired number of channels, followed by Tanh and BatchNorm.
Block 2:
On ImageNet, the input is also projected through a linear layer before being mapped to spatial features. The resulting vector then passed through fully connected layer and reshaped into 128 56 56 feature map. The convolutional backbone consists of three blocks, with bilinear upsampling applied between them:
Block Categorical Conditional BatchNorm (CCBN) is applied.
Block 1: A convolutional layer with 128 filters, followed by CCBN and LeakyReI.U activation.
Block 2: One convolution with 128 channels that outputs the image channels, followed by CCBN, LeakyReLU. Followed by applying another convolutional layer, reducing to 64 channels, followed by Tanh activation, and final BatchNorm.
During training, the generator supports mixing noise latents and labels through weighted projection. The probability of applying mixup is set to 0.4. The number of inputs used for mixing is 2 for CIFAR-10 10 and 10 for ImageNet.
For CIFAR-10 10, we train for 400 epochs using batch size of 200. The optimizer is Adam with 31 = 0.5 and 32 = 0.999. The label embedding dimension 64. The learning rate is set 0.001, using a multi-step schedule that decays by factor of 0.1 at epochs 100, 200, and 300.
For ImageNet, we also train for 400 epochs with batch size of 16. Adam is used with the same 31 and 82 values as in CIFAR-10. The label embedding dimension is set to 256 for ResNet18 students and 512 for ResNet50 and ResNeXt50 students. The learning rate is 0.001, with the same multi-step decay schedule.
Hardware for experiments. For training teacher and student models CIFAR-10, A30 GPU. For ImageNet, student models are trained using two L40S GPUs.
single
In the first stage of our knowledge distillation detection pipeline, we train generator to produce synthesized images classification tasks using a single L40S GPU.
For score computation and prediction, we only perform inference with the student and teacher models. This stage uses single A30 GPU for CIFAR- 10 classification models, and single L40S GPU for ImageNet classification and text-to-image generation models.
A3 Baseline details
Most of the baselines have already been introduced in Sec. 5, Here, we formally describe MMD- FUSE in this section. MMD-FUSE is kernel- based two-sample test. In our setting, the null and alternative hypotheses are:
Hn go (J)sin()) vs. H1 90
D iindep:
(A (A15)
Let the student model outputs be denoted by {x+)r- D= and the teacher outputs by Hw MMD- FUSE adaptively combines multiple kernels into single test statistic. For finite set of candidate M it computes each normalized MMD value and then takes a softmax over them:
1. kernels {
M exp(8 MMD
1 MMD_FUSE
log Σ
2 m
3
where MMD m min-1) Eijg kim (Ri, 2r) m.- Σijs km (ye, ,1) 9 Σー Σý km (2 yJ), To compute the p-vuuue of the test, MMD-FUSE uses a permutation procedure. A smaller p-value indicates stronger evidence against the null hypothesis, suggesting that the student model is not distilled version of the teacher.
In our experiment, we use two base kernels: the Laplace kernel and the RBF kernel. For each kernel, we select 10 different bandwidths. The number permutations used for estimating the p-value is set to 1000, and the temperature scalar for the softmax aggregation is set to 3 = 1.
24

---

A4 Broader impacts
This work contributes to model accountability by introducing a method for detecting whether a model has been trained via knowledge distillation from specific teacher. This has positive implications for auditing model provenance, protecting intellectual property, and improving transparency in the deployment of machine learning systems. Our detection framework can help identify unauthorized reuse proprietary models, especially in scenarios where model weights are publicly released but training procedures are not disclosed. At the same time, detection tools may influence how models are shared or reused, and like any statistical method, they may produce imperfect predictions. We suggest using such tools with awareness of their limitations and context.
25


Reviews:
[{"review_id": "N3kjqeg0ou", "reviewer": "Reviewer_nZTZ", "summary": "This paper proposes and addresses the task of \"knowledge distillation detection\", wherein the goal is to determine whether an existing model has been distilled from a separate teacher model - given only access to student model weights and the teacher API. The final proposed mechanisms to tackle this problem comprises three primary stages: (1) probing model behaviour via synthetically generated data, (2) extracting statistical scores from model responses, and (3) applying a given decision rule to determine whether distillation has occurred or not - all without making particular requirements on the model architectures of choice.\nExperiments are conducted on smaller image classification and generation scenarios (e.g. ResNet-style models, on CIFAR-10/ImageNet, and models such as SD-v2.1 / SDXL) show high efficacy of the proposed mechanism.", "questions": "I do think this paper is a relevant contribution - it tries to formalize an important research problem (detecting whether distillation has occured), and within the setting tested does so really well. Unfortunately, I am somewhat struggling to make the connection between the specific research problem and the realistic  scenarios within which distillation detection would occur. It would be great if the authors could provide context here.", "limitations": "Not specifically addressed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "__Strengths:__\n* The paper is well written, organised and easy to follow - motivating the task well, before diving into their proposed first approach to tackle this problem.\n* To the best of my knowledge, the proposed distillation detection task, the problem formulation, and the suggested approach to test for distillation is novel.\n* The experiments conducted are extensive, and convincing; showcasing convincing results across baseline methods across both classification distillation detection (Tab 1 and 2), as well as for T2I models in Tab. 3\n\n__Weaknesses:__\n  \nMy main issues lies in the discrepancy between the formulated problem setting and the primary motivation of the paper / distillation detection issue: instead of detecting >whether a model was distilled from another model<, the task is that of detecting which model was distilled from - which excludes the difficult scenario of determining whether any distillation has actually happened. It would be great if the authors could provide insights here; whether the proposed method could be extended to this much more realistic and general scenario, and clearly distinguish the importance of the actually tested scenario of determining which model was distilled from. In a legal issue scenario where someone was accused of distilling e.g. from some proprietary model, just being able to say that there is a higher chance of distillation w.r.t. To some generic baseline would hold value.", "was_revised": true, "final_justification": "The authors have done a great job clarifying the mismatch between formulation & motivation, alongside some initial experimental support that one can transition into the other. If the authors can work this into the final version of the submission, I'm happy to recommend acceptance.", "rebuttal": "We thank the reviewer for the recognition of our clear writing, novel problem formulation, and strong experimental results.\n\n> Q1. Formulated problem setting vs. the primary motivation.\n\nThank you for pointing this out. We agree that we should make the connection between the motivation and the task formulation more explicit. Below, we carefully discuss the implications, clarify the setup, and how we will update the manuscript.\n\n1. **Motivated:** Given a model, predict whether it has been distilled **FROM** a teacher model. (A teacher model candidate is required)\n2. **Formulated:** Given a model, predict **WHICH** of the teacher model has been used to distill the student model. (A *set* of teacher candidates is required \\& assuming the student model has been distilled from one of them)\n\nIn the introduction, we motivated with (1) but presented a task formulation of (2). We now recognize that this connection may not have been sufficiently emphasized. \n\n\nWe formulated the task as a multiple choice problem as it is straightforward and with results easy to understand. The main difference between (1) and (2) is that (1) further requires an absolute threshold for decision making. In contrast, the multiple choice setting performs a relative comparison across a set of candidate teacher models. This allows us to select the most likely teacher without requiring an explicit threshold. Our method can be modified from (2) to (1), see Q2.\n\n> Q2. Extending to the scenario of the motivated problem.\n\nUnder our distillation detection framework, transfering from (2) to (1) would require: **(A)**  a calibrated score function and **(B)** a suitable decision rule. As an example, we can define the score as the p-value from a statistical independence test between the teacher and student. The decision of (1) can be made by checking whether this p-value falls below a chosen significance level, e.g., 0.05. \n\n\nNext, we implement the HSIC test [A] to obtain p-value as score for the text-to-image generation task with 3 teacher models and 10 student models. Each of the student model is paired with its true teacher (positive) and two random candidate teachers (negative). A pair is predicted as distillation if the p-value is below 0.05. From the 30 teacher-student pairs, accuracy and F1 score are computed from these binary predictions, and AUC is based on the scores across different thresholds for p-value. The results are provided in Table R1. These results show that generalizing to the formulation (1) is possible. We view this as a natural extension and a valuable direction for future work.\n\n\nTable R1. Performance of distillation detection using the HSIC test on the text-to-image task.\n| Input size | Acc. | AUC | F1 | \n| -------- | -------- | -------- | -------- |\n| 10     | 0.74  $\\pm$ 0.05     | 0.76 $\\pm$ 0.07 | 0.43 $\\pm$ 0.09 |\n| 50     | 0.86  $\\pm$ 0.04     | 0.91 $\\pm$ 0.07 | 0.80 $\\pm$ 0.06 |\n| 100    | 0.80 $\\pm$ 0.04      | 0.94 $\\pm$ 0.04 | 0.75 $\\pm$ 0.05 |\n\nWe will update the abstract and introduction of the manuscript to clarify the connection between the motivation and the formulation.\n\n[A] Gretton, Arthur, et al. \"A kernel statistical test of independence.\" In Proc. NeurIPS 2007.\n\n\n> Q3. Real world use cases for distillation detection.\n\n\nBesides the scientific question, we foresee that distillation detection will have the following real-world use cases: \n1. **Detection terms of service violation:** In the terms of service for OpenAPI services (OpenAI, Gemini, Midjourney, etc.), it is common to have some form of clause that prohibits users from training/distilling the services' output. The ability to detect distillation will enable the detection of such a violation.\n2. **Tracing content origin:** Recently, there has been an interest in determining whether a model has been trained on data with IP protection. This would ensure proper accounting and crediting of the data source. Distillation detection will be a useful tool to trace these data sources across models. This closes the loophole that one could claim that they never trained on certain data, where they \"distilled\" from a model that did. \n\nWe believe our work is an important first step towards these directions. We will clarify and provide a more in-depth discussion in the introduction."}, {"review_id": "vus16cIbTN", "reviewer": "Reviewer_bDcj", "summary": "This paper introduces a means to determine if a student model was distilled from a specific teacher model, using the student's weights and the teacher's API. Provenance is an important and thorny problem, and unlikely many proposed problems in ML security, the assumptions around model & API access appear to be practical and not contrived. The method itself works by generating synthetic inputs that are predicted confidently with the student model, these are then scoring under the model and under the candidate teacher models, to find the pairing of student and teacher that are most aligned in their predictions. Overall this looks well motivated, and the approach is demonstrated to work empirically via experiments on image classification and text to image generation.", "questions": "1. Could you adapt the method to work purely from API access for student and candidate teacher models? This would complicate the synthesis step, but would the other parts of the method be applicable?\n1. Given this method now exists, could an adversary use this to \"hide\" their dependence on a specific teacher model? This might work by ensuring the confidence of the model is high on out-of-domain instances, or adding label noise to training, or other means. Might it be possible to reach similar model quality, despite obscuring the provenance?\n1. As well as intentional distillation, there's unintentional copying of another model, e.g., by training on web crawls that include outputs from other models. This is a big problem for modern foundation models, in that it can be very hard to identify, and can lead to performance, legal and other issues. This would be an interesting topic to discuss in the introduction or discussion parts of the paper.\n1. A related problem is when a model is trained on the same private dataset. Wouldn't this have a similar fingerprint on the model activations as distillation? Could this be differentiated from distillation?\n1. Confident predictions from the model are as a signal for data synthesis. Would this be a strong heuristic where the predictions are incorrect. That is, agreeing on a mistake with the teacher is much more informative than agreeing on a correct answer.\n1. It's one thing to predict a score for each teacher, but can this be cast as a likelihood that the models were trained independently, and thus used in a statistical test of independence. This would help in legal application of the method.\n1. Can you comment on how the SVD method in 4.2 will scale for very multiclass scenarios, with many 1000s of labels and potentially sparse activations.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1.  **Timely problem:** The paper tackles the growing concern of model provenance and unauthorized replication, something that currently is not well understood.\n2.  **Methodology:** The method of knowledge distillation detection is well motivated, model-agnostic, and is shown to work for both classification and generative models, with only limited access to the models themselves (but see more later).\n3.  **Good results:** They show solid improvements in detection accuracy over strong baselines in image classification (CIFAR-10, ImageNet) and text-to-image generation tasks.\n\nWeaknesses:\n\n1.  **Limited application to text inputs/outputs:** The paper focuses on image-based tasks (classification and text-to-image generation). While text-to-image generation involves text prompts as input, the \"empty string\" approach simplifies the input generation. It is not clear how the proposed data-free input synthesis and scoring mechanisms would directly translate or perform in entirely discrete domains like text-to-text models (e.g., detecting distillation for large language models).\n2.  **Reliability when student inherits limited influence from teacher:** The authors acknowledge that \"detection may be less reliable when the student inherits only limited influence from the teacher\" as shown in the ablation in Fig. 3. I think in most practical scenarios the student will be trained to replicate the teacher closely, so this may not be a big issue.\n3.  **Computational costs:** The cost of being data-free is the need to synthesize inputs. This doesn't seem very scalable. There's some discussion in the appendix of the costs of synthesis, but I'd appreciate this being discussed in the main paper.", "was_revised": true, "final_justification": "The authors gave a detailed response, resolving the questions I had. I stand by my strong positive assessment of the work.", "rebuttal": "We thank the reviewer for recognizing our timely problem, model‑agnostic approach, and strong performance on both classification and text‑to‑image tasks.\n\n\n> Q4. Generalization to text-to-text models\n\nThe high-level detection framework, of data-free synthesis followed by score computation, applies to text-to-text models. However, the specific choices for synthesis and scoring required modification, as they are domain-specific (NLP). For synthesis, inspiration can be drawn from \"Data-Free Quantization for LLMs\" [B]. For the scoring function, we could consider utilizing text embedding similarity, which converts the discrete output back into a continuous domain.\n\n[B] Liu, Zechun, et al. \"LLM-QAT: Data-free quantization aware training for large language models.\" arXiv preprint arXiv:2305.17888 (2023).\n\nWe will add this discussion to the limitations section and clarify the scope of this paper. While we attempted to be clear by mentioning “image classification and text-to-image generation” in the abstract, we will further clarify in the writing. For instance, would including the term \"Vision Models\" in the title be helpful?\n\n\n\n> Q5. Reliability when student inherits limited influence from teacher is not a big issue.\n\nThanks for this comment. We will incorporate this view and defer these analysis to the appendix.\n\n> Q6. Computational costs\n\nThank you for the suggestion. As suggested, we will move the computational cost from the appendix to the main paper.\n\n\n> Q7. Adapt the method to work purely from API access for both student/teacher models?\n\nIn some context, it is possible to only require API access for both student and teacher models.\n\n- For the text-to-image setting, our current experiments already operate under API-only access to the student model. As we use the \"empty string\" approach to data generation, and the score function does not require model weights.\n\n- For the classification case, the current synthesis step depends on backpropagation through the student, which is not feasible with only API access. One possible direction is to explore black-box gradient approximation methods [C], however, such exploration is beyond the scope of this work.\n\n[C] Grathwohl, Will, et al. \"Backpropagation through the void: Optimizing control variates for black-box gradient estimation.\" Proc. ICLR, 2018.\n\n\n> Q8. Could an adversary use this to \"hide\" their dependence on a specific teacher model?\n\nThis is a possible scenario. Currently, our experiment is based on existing distillation methods. An adversary could try to modify their distillation approach to evade detection, e.g., deliberating injecting noise during the process.  This adversarial setting is an interesting future direction. We will add a discussion.\n\n\n\n> Q9. Discussion of unintentional copying of another model.\n\nThanks for this comprehensive point of view. Currently, this work focus on the intentional distillation to establish a concrete framework and setup. We agree that unintentional copying is relevant, yet out of the scope of this work. We will discuss unintentional copying as a limitations/future work.\n\n\n\n> Q10. Cases when models are trained on the same private dataset.\n\nThanks for bringing up this interesting experiment. Here we provide experiment results in Table R2 for image classification, where we compare the KL distance between (a) the teacher and the distilled student and (b) the teacher and an independently trained model. \n\nThese models are trained on CIFAR10 with the corresponding architecture. We can observe the KL distance of (a) is consistently smaller than (b). This means that our approach can differentiate distillation from simply training from the same dataset.\n\n**Table R2: Results from training on the same private dataset.**\n| Architecture | Distilled | Independent |\n| -------- | -------- | -------- | \n| ResNet-18 | $3.13e^{-4}$| $5.22e^{-1}$ |\n|DLA |$1.89e^{-4}$| $5.85e^{-1}$ |\n|DPN-92 |$3.31e^{-4}$| $1.64e^{-1}$ |\n|DenseNet-121 |$9.71e^{-5}$| $1.47e^{-1}$ |\n\n\n> Q11. Would agreement on incorrect predictions serve as a stronger signal for distillation?\n\nIn our data synthesis process, with its data-free nature, we don't have the ground truth label. As such, it is unclear how to define a “mistake”.\n\nTo test the reviewer's hypothesis, we provide experiment results by assuming we have access to the real training data. Specifically, for each candidate teacher, we count how often the student and teacher agree on incorrect predictions, and we select the teacher with the highest count. The results are summarized in Table R3. We can indeed see an improvement by using agreement on mistakes as the score and decision rule.\n\nTable R3. Accuracy when using \"agreement on mistakes\".\n| Input size | Acc. (agree on mistakes)  | Acc. (Oracle) |\n| -------- | -------- | -------- |\n| 10  | 0.71 $\\pm$ 0.06 | 0.67 $\\pm$ 0.07 |\n| 50  | 0.97 $\\pm$ 0.05 | 0.87 $\\pm$ 0.04 |\n| 100 | 1.00 $\\pm$ 0.00 | 0.95 $\\pm$ 0.02 |\n\nWe will update the paper to include these findings, and we will also add a discussion of alternative heuristics, including agreement on mistakes, as potential extensions under more relaxed assumptions.\n\n\n\n> Q12. Can the method estimate a likelihood that the student and a given teacher were trained independently?\n\nYes, we provide additional results of using independence test on the task of text-to-image generation in Table R4. Here we implement the HSIC test [A] to obtain p-value as the score with 3 teacher models and 10 student models. Each of the student model is paired with its true teacher (positive) and two random candidate teachers (negative). A pair is predicted as distillation if the p-value is below 0.05. From the 30 teacher-student pairs, accuracy and F1 score are computed from these binary predictions, and AUC is based on the scores across different thresholds for p-value. We can see our framework can be adapt to using p-values as the score.\n\nTable R4. Performance of distillation detection using the HSIC test on the text-to-image task.\n| Input size | Acc. | AUC | F1 | \n| -------- | -------- | -------- | -------- |\n| 10     | 0.74  $\\pm$ 0.05     | 0.76 $\\pm$ 0.07 | 0.43 $\\pm$ 0.09 |\n| 50     | 0.86  $\\pm$ 0.04     | 0.91 $\\pm$ 0.07 | 0.80 $\\pm$ 0.06 |\n| 100    | 0.80 $\\pm$ 0.04      | 0.94 $\\pm$ 0.04 | 0.75 $\\pm$ 0.05 |\n\n\n[A] Gretton, Arthur, et al. \"A kernel statistical test of independence.\" In Proc. NeurIPS 2007.\n\n\n> Q13. Comment on how the SVD method in 4.2 will scale for very multiclass scenarios\n\nThe SVD step in Sec. 4.2 has a complexity of $O(C^3)$ for $C$ classes. Note that it is computed only once per teacher-student pair as part of the set-level score. To make the computation efficient for large $C$,  SVD approximations such as truncated or sparse SVD [D] can applied.\n\n[D] Yang D, Ma Z, Buja A. A sparse singular value decomposition method for high-dimensional data. Journal of Computational and Graphical Statistics, 2014"}, {"review_id": "hTClE1takj", "reviewer": "Reviewer_4543", "summary": "This paper presents a method for determining which teacher model distilled a student model when only the student model and teacher API are available.", "questions": "I believe the primary issue with this paper is the lack of value in the research problem it addresses. If the authors can demonstrate the practical significance of this research setting, I would reconsider my rating.", "limitations": "Please refer to the question section.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is straightforward and easy to understand. In terms of results, it shows a significant improvement over the baseline methods.\n\nWeaknesses:\n\n1. My primary concern is the value of the problem being addressed. This paper investigates a scenario under very stringent conditions—having only the teacher's API and the student's model, without access to the data used for distillation. Such situations are rare and lack clear application contexts. Additionally, the method proposed in the paper can only identify the most likely teacher from multiple options, but it cannot determine whether distillation was used on the student at all. The latter capability would have greater practical value.\n\n2. The main contribution of this paper is the proposal of score functions and input data generation methods for various scenarios, along with experimental validation. However, these methods are straightforward and lack novelty.", "was_revised": true, "final_justification": "The author's response has largely addressed my concerns regarding the application scenario. However, I still believe that merely selecting the most likely teacher among a few does not hold much significance. Determining whether distillation is present or not is far more meaningful. Therefore, I will raise my rating to borderline reject.", "rebuttal": "We thank the reviewer for the feedback. We address individual questions below. We believe there are some misunderstanding and look forward to clarifying them.\n\n\n> Q14. Novelty and contribution.\n\nTo the best of our knowledge, the introduced task of distillation detection has not been addressed in prior research. We present a framework that breaks down the distillation detection process into three main components: input construction, score computation, and decision-making rules. We demonstrate the framework’s effectiveness in image classification and text-to-image tasks, comparing it against several different baselines by adapting related areas. We believe that our work enhances the understanding of distillation detection beyond the  current literature and has real-world implications, as discussed in Q16.\n\n> Q15. The paper investigates a scenario under very stringent conditions.\n\nWe understand the reviewer’s concern, but we believe the setting we study is realistic and practical. In many real-world cases, especially with commercial models, the training data is not available, and the teacher model is only accessible via an API. While the setting may seem strict, this is also what will make our method effective in real-world circumstances. \n\nPlease note that we have included results for a more relaxed setup, where training data is accessible (as shown in the Oracle case in Tables 1 and 2), and our method still performs well. This shows that the approach remains effective across different assumptions.\n\n> Q16. Real world application for distillation detection. \n\nBesides the scientific question, we foresee that distillation detection will have the following real-world use cases: \n1. **Detection terms of service violation:** In the terms of service for OpenAPI services (OpenAI, Gemini, Midjourney, etc.), it is common to have some form of clause that prohibits users from training/distilling the services' output. The ability to detect distillation will enable the detection of such a violation.\n2. **Tracing content origin:** Recently, there has been an interest in determining whether a model has been trained on data with IP protection. This would ensure proper accounting and crediting of the data source. Distillation detection will be a useful tool to trace these data sources across models. This closes the loophole that one could claim that they never trained on certain data, where they \"distilled\" from a model that did. \n\nWe believe our work is an important first step towards these directions. We will clarify and provide a more in-depth discussion in the introduction.\n\n\n> Q17. Most likely teacher vs. whether distillation occur.\n\nThanks for pointing this out. We agree that we should make the connection between the motivation and the task formulation more explicit. Below, we carefully discuss the implications, clarify the setup, and how we will update the manuscript.\n\n1. **Motivated:** Given a model, predict whether it has been distilled **FROM** a teacher model. (A teacher model candidate is required)\n2. **Formulated:** Given a model, predict **WHICH** of the teacher model has been used to distill the student model. (A *set* of teacher candidates is required \\& assuming the student model has been distilled from one of them)\n\nIn the introduction, we motivated with (1) but presented a task formulation of (2). We now recognize that this connection may not have been sufficiently emphasized. \n\n\nWe formulated the task as a multiple choice problem as it is straightforward and with results easy to understand. The main difference between (1) and (2) is that (1) further requires an absolute threshold for decision making. In contrast, the multiple choice setting performs a relative comparison across a set of candidate teacher models. This allows us to select the most likely teacher without requiring an explicit threshold.\n\nUnder our distillation detection framework, transferring from (2) to (1) would require: **(A)**  a calibrated score function and **(B)** a suitable decision rule. As an example, we can define the score as the p-value from a statistical independence test between the teacher and student. The decision of (1) can be made by checking whether this p-value falls below a chosen significance level, e.g., 0.05. \n\n\nNext, we implement the HSIC test [A] to obtain p-value as score for the text-to-image generation task with 3 teacher models and 10 student models. Each of the student model is paired with its true teacher (positive) and two random candidate teachers (negative). A pair is predicted as distillation if the p-value is below 0.05. From the 30 teacher-student pairs, accuracy and F1 score are computed from these binary predictions, and AUC is based on the scores across different thresholds for p-value. The results are provided in Table R5. These results show that generalizing to the formulation (1) is possible. We view this as a natural extension and a valuable direction for future work.\n\n\nTable R5. Performance of distillation detection using the HSIC test on the text-to-image task.\n| Input size | Acc. | AUC | F1 | \n| -------- | -------- | -------- | -------- |\n| 10     | 0.74  $\\pm$ 0.05     | 0.76 $\\pm$ 0.07 | 0.43 $\\pm$ 0.09 |\n| 50     | 0.86  $\\pm$ 0.04     | 0.91 $\\pm$ 0.07 | 0.80 $\\pm$ 0.06 |\n| 100    | 0.80 $\\pm$ 0.04      | 0.94 $\\pm$ 0.04 | 0.75 $\\pm$ 0.05 |\n\nWe will update the abstract and introduction of the manuscript to clarify the connection between the motivation and the formulation.\n\n[A] Gretton, Arthur, et al. \"A kernel statistical test of independence.\" In Proc. NeurIPS 2007."}, {"review_id": "pIDdSobybf", "reviewer": "Reviewer_41Ct", "summary": "This paper introduces the task of knowledge distillation detection, aiming to determine whether a student model has been distilled from a given teacher model, using only the student’s weights and the teacher’s as a black box (API). The authors propose a model-agnostic framework that combines data-free input synthesis, score computation, and decision-making stages to detect distillation, applicable to both classification and generative models. The method is tested on a variety of architectures for image classification and text-to-image generation, showing significant improvements in detection accuracy over existing baselines, with 59.6% better accuracy on CIFAR-10, 71.2% on ImageNet, and 20.0% for text-to-image generation.", "questions": "Address questions in weaknesses section.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nNovel Task and Relevance: The paper addresses a timely and critical issue of knowledge distillation detection, which is becoming more important with concerns about intellectual property, and unauthorized replication. The problem setting where the teacher is closed sources used to distill a smaller open sourced model is an interesting problem. The proposed approach is adaptable and can be used for a wide variety of models (both classification and generative), making it highly versatile and applicable across different domains. The method is validated on multiple datasets and architectures, including CIFAR-10, ImageNet, and text-to-image generation tasks. The results show impressive improvements over strong baselines. The paper is also generally well written an easy to follow.\n\nWeaknesses\n\nThe paper discusses knowledge distillation which alignment between the teacher and student models. However, closed-source teacher models only have API access, limiting the ability to directly access and align the internal representations between the teacher and student. Meaning with these closed source model, the only knowledge distillation method one can use is to generate outputs from the teacher and use it to train a student. Does the paper proposed techniques also address this scenario?\n\nThe paper's approach assumes a scenario where a fixed set of K teacher models is provided, and the goal is to identify which teacher was used to distill the student model. However, this assumption does not address the more general and arguably more important problem: determining whether a given student model was distilled from a given teacher model or not. In real-world applications, the number of possible teacher models could be enormous, if not infinite, making it difficult to define such a finite subset of potential teachers.  A key test for your setup could involve running the proposed method with a set of K models, where none of them have been involved in the distillation of the student model. The interesting question here would be whether the method falsely identifies any of those models as the source, potentially leading to false positives? \n\nDoes the proposed method apply to large language models (LLMs) ? Given that the paper primarily focuses on image classification and text-to-image generation tasks, it would be interesting to know if the detection method can also be generalized to handle LLMs. I recommend an experiment on this. This could be a simple sentiment analysis case (also see [1]). \n\nI recommend you summarize your method and approach in an algorithm environment for better clarity.\n \n[1] Less is More: Task-aware Layer-wise Distillation for Language Model Compression https://arxiv.org/abs/2210.01351", "was_revised": true, "final_justification": "Good paper, however current presentation is overselling the paper as a general method. Authors are recommended to improve the presentation and be specific where their method applies to.", "rebuttal": "Thank you for the thoughtful and encouraging review. We appreciate the recognition on the novelty, relevance, and writing quality of this work. We now address individual questions.\n\n> Q18. Senario where distillation can only use closed-source teacher model's output?\n\nTo clarify, the closed-source assumption applies to the detection phase and not to the distillation process itself. Our detection method does not distinguish between *how* a student model has been distilled. For example, one of the experiment uses the original knowledge-distillation (KD) to train the student model, which only uses a closed-source teacher model's output.\n\n\n\n> Q19. Motivation vs experiment setup.\n\nThanks for pointing this out. We agree that we should make the connection between the motivation and the task formulation more explicit. Below, we carefully discuss the implications, clarify the setup, and how we will update the manuscript.\n\n1. **Motivated:** Given a model, predict whether it has been distilled **FROM** a teacher model. (A teacher model candidate is required)\n2. **Formulated:** Given a model, predict **WHICH** of the teacher model has been used to distill the student model. (A *set* of teacher candidates is required \\& assuming the student model has been distilled from one of them)\n\nIn the introduction, we motivated with (1) but presented a task formulation of (2). We now recognize that this connection may not have been sufficiently emphasized. \n\n\nWe formulated the task as a multiple choice problem as it is straightforward and with results easy to understand. The main difference between (1) and (2) is that (1) further requires an absolute threshold for decision making. In contrast, the multiple choice setting performs a relative comparison across a set of candidate teacher models. This allows us to select the most likely teacher without requiring an explicit threshold. Our method can be modified from (2) to (1), see Q20.\n\n\n> Q20. Extending to the determining whether a given student model was distilled from a given teacher model or not.\n\nUnder our distillation detection framework, transferring from (2) to (1) would require: **(A)**  a calibrated score function and **(B)** a suitable decision rule. As an example, we can define the score as the p-value from a statistical independence test between the teacher and student. The decision of (1) can be made by checking whether this p-value falls below a chosen significance level, e.g., 0.05. \n\n\nNext, we implement the HSIC test [A] to obtain p-value as score for the text-to-image generation task with 3 teacher models and 10 student models. Each of the student model is paired with its true teacher (positive) and two random candidate teachers (negative). A pair is predicted as distillation if the p-value is below 0.05. From the 30 teacher-student pairs, accuracy and F1 score are computed from these binary predictions, and AUC is based on the scores across different thresholds for p-value. The results are provided in Table R6. These results show that generalizing to the formulation (1) is possible. We view this as a natural extension and a valuable direction for future work.\n\n\nTable R6. Performance of distillation detection using the HSIC test on the text-to-image task.\n| Input size | Acc. | AUC | F1 | \n| -------- | -------- | -------- | -------- |\n| 10     | 0.74  $\\pm$ 0.05     | 0.76 $\\pm$ 0.07 | 0.43 $\\pm$ 0.09 |\n| 50     | 0.86  $\\pm$ 0.04     | 0.91 $\\pm$ 0.07 | 0.80 $\\pm$ 0.06 |\n| 100    | 0.80 $\\pm$ 0.04      | 0.94 $\\pm$ 0.04 | 0.75 $\\pm$ 0.05 |\n\nWe will update the abstract and introduction of the manuscript to clarify the connection between the motivation and the formulation.\n\n[A] Gretton, Arthur, et al. \"A kernel statistical test of independence.\" In Proc. NeurIPS 2007.\n\n\n> Q21. Does the proposed method apply to LLMs?\n\nIn theory, our framework an be adapted to LLMs with specific choices for synthesis and scoring required modification, as they are domain-specific (NLP). For synthesis, inspiration can be drawn from \"Data-Free Quantization for LLMs\" [B]. For the scoring function, we could consider utilizing text embedding similarity, which converts the discrete output back into a continuous domain.\n\n[B] Liu, Zechun, et al. \"LLM-QAT: Data-free quantization aware training for large language models.\" arXiv preprint arXiv:2305.17888 (2023).\n\nWe will add this discussion to the limitations section and clarify the scope of this paper. While we attempted to be clear by mentioning “image classification and text-to-image generation” in the abstract, we will further clarify in the writing. For instance, would including the term \"Vision Models\" in the title be helpful?\n\n\n\n> Q22. Summarize method and approach in an algorithm.\n\nThanks for this great suggestion. We only provided a visual overview in Fig. 1. We agree that presenting the method in an algorithm would be easier to follow. We will modify."}]

Author rebuttal:

Meta-review: The paper presents an approach to the problem of knowledge distillation detection, which is a timely and relevant issue in the field of artificial intelligence, particularly concerning model provenance and unauthorized replication. The reviewers generally praise the paper for its well-organized content, clarity, and the novelty of knowledge distillation detection problem. The methodology is also appreciated for its experiments in image classification (CIFAR-10, ImageNet) and text-to-image generation scenarios. Meanwhile, some constructive criticisms and suggestions for improvement are also provided, including the problem settings, the scope of detection capabilities and the computational costs.

Given the balance of strengths and weaknesses, and considering the critical importance of the topic, I decide to accept the work with strong encouragement for the authors to undertake significant revisions to address the reviewers' concerns. The novelty and relevance of the work are clear, but realizing its full potential and ensuring broader applicability require addressing the identified weaknesses.
Decision: Accept (poster)