Title: Knowledge Distillation Detection for Open-weights Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge Distillation

Distillation Detection
Abstract: We propose the task of knowledge distillation detection, which aims to determine whether a student model has been distilled from a given teacher, under a practical setting where only the student’s weights and the teacher’s API are available. This problem is motivated by growing concerns about model provenance and unauthorized replication through distillation. To address this task, we introduce a model-agnostic framework that combines data-free input synthesis and statistical score computation for detecting distillation. Our approach is applicable to both classification and generative models. Experiments on diverse architectures for image classification and text-to-image generation show that our method improves detection accuracy over the strongest baselines by 59.6\% on CIFAR-10, 71.2\% on ImageNet, and 20.0\% for text-to-image generation. The code is available at https://github.com/shqii1j/distillation_detection.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the task of knowledge distillation detection, aiming to determine whether a student model has been distilled from a given teacher model, using only the student’s weights and the teacher’s as a black box (API). The authors propose a model-agnostic framework that combines data-free input synthesis, score computation, and decision-making stages to detect distillation, applicable to both classification and generative models. The method is tested on a variety of architectures for image classification and text-to-image generation, showing significant improvements in detection accuracy over existing baselines, with 59.6% better accuracy on CIFAR-10, 71.2% on ImageNet, and 20.0% for text-to-image generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nNovel Task and Relevance: The paper addresses a timely and critical issue of knowledge distillation detection, which is becoming more important with concerns about intellectual property, and unauthorized replication. The problem setting where the teacher is closed sources used to distill a smaller open sourced model is an interesting problem. The proposed approach is adaptable and can be used for a wide variety of models (both classification and generative), making it highly versatile and applicable across different domains. The method is validated on multiple datasets and architectures, including CIFAR-10, ImageNet, and text-to-image generation tasks. The results show impressive improvements over strong baselines. The paper is also generally well written an easy to follow.\n\nWeaknesses\n\nThe paper discusses knowledge distillation which alignment between the teacher and student models. However, closed-source teacher models only have API access, limiting the ability to directly access and align the internal representations between the teacher and student. Meaning with these closed source model, the only knowledge distillation method one can use is to generate outputs from the teacher and use it to train a student. Does the paper proposed techniques also address this scenario?\n\nThe paper's approach assumes a scenario where a fixed set of K teacher models is provided, and the goal is to identify which teacher was used to distill the student model. However, this assumption does not address the more general and arguably more important problem: determining whether a given student model was distilled from a given teacher model or not. In real-world applications, the number of possible teacher models could be enormous, if not infinite, making it difficult to define such a finite subset of potential teachers.  A key test for your setup could involve running the proposed method with a set of K models, where none of them have been involved in the distillation of the student model. The interesting question here would be whether the method falsely identifies any of those models as the source, potentially leading to false positives? \n\nDoes the proposed method apply to large language models (LLMs) ? Given that the paper primarily focuses on image classification and text-to-image generation tasks, it would be interesting to know if the detection method can also be generalized to handle LLMs. I recommend an experiment on this. This could be a simple sentiment analysis case (also see [1]). \n\nI recommend you summarize your method and approach in an algorithm environment for better clarity.\n \n[1] Less is More: Task-aware Layer-wise Distillation for Language Model Compression https://arxiv.org/abs/2210.01351\n\nQUESTIONS:\nAddress questions in weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents a method for determining which teacher model distilled a student model when only the student model and teacher API are available.

STRENGTHS AND WEAKNESSES:
Strengths:

1. This paper is straightforward and easy to understand. In terms of results, it shows a significant improvement over the baseline methods.

Weaknesses:

1. My primary concern is the value of the problem being addressed. This paper investigates a scenario under very stringent conditions—having only the teacher's API and the student's model, without access to the data used for distillation. Such situations are rare and lack clear application contexts. Additionally, the method proposed in the paper can only identify the most likely teacher from multiple options, but it cannot determine whether distillation was used on the student at all. The latter capability would have greater practical value.

2. The main contribution of this paper is the proposal of score functions and input data generation methods for various scenarios, along with experimental validation. However, these methods are straightforward and lack novelty.

QUESTIONS:
I believe the primary issue with this paper is the lack of value in the research problem it addresses. If the authors can demonstrate the practical significance of this research setting, I would reconsider my rating.

LIMITATIONS:
Please refer to the question section.
TITLE: Official Review
CLARITY: 3
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a means to determine if a student model was distilled from a specific teacher model, using the student's weights and the teacher's API. Provenance is an important and thorny problem, and unlikely many proposed problems in ML security, the assumptions around model & API access appear to be practical and not contrived. The method itself works by generating synthetic inputs that are predicted confidently with the student model, these are then scoring under the model and under the candidate teacher models, to find the pairing of student and teacher that are most aligned in their predictions. Overall this looks well motivated, and the approach is demonstrated to work empirically via experiments on image classification and text to image generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.  **Timely problem:** The paper tackles the growing concern of model provenance and unauthorized replication, something that currently is not well understood.\n2.  **Methodology:** The method of knowledge distillation detection is well motivated, model-agnostic, and is shown to work for both classification and generative models, with only limited access to the models themselves (but see more later).\n3.  **Good results:** They show solid improvements in detection accuracy over strong baselines in image classification (CIFAR-10, ImageNet) and text-to-image generation tasks.\n\nWeaknesses:\n\n1.  **Limited application to text inputs/outputs:** The paper focuses on image-based tasks (classification and text-to-image generation). While text-to-image generation involves text prompts as input, the \"empty string\" approach simplifies the input generation. It is not clear how the proposed data-free input synthesis and scoring mechanisms would directly translate or perform in entirely discrete domains like text-to-text models (e.g., detecting distillation for large language models).\n2.  **Reliability when student inherits limited influence from teacher:** The authors acknowledge that \"detection may be less reliable when the student inherits only limited influence from the teacher\" as shown in the ablation in Fig. 3. I think in most practical scenarios the student will be trained to replicate the teacher closely, so this may not be a big issue.\n3.  **Computational costs:** The cost of being data-free is the need to synthesize inputs. This doesn't seem very scalable. There's some discussion in the appendix of the costs of synthesis, but I'd appreciate this being discussed in the main paper.\n\nQUESTIONS:\n1. Could you adapt the method to work purely from API access for student and candidate teacher models? This would complicate the synthesis step, but would the other parts of the method be applicable?\n1. Given this method now exists, could an adversary use this to \"hide\" their dependence on a specific teacher model? This might work by ensuring the confidence of the model is high on out-of-domain instances, or adding label noise to training, or other means. Might it be possible to reach similar model quality, despite obscuring the provenance?\n1. As well as intentional distillation, there's unintentional copying of another model, e.g., by training on web crawls that include outputs from other models. This is a big problem for modern foundation models, in that it can be very hard to identify, and can lead to performance, legal and other issues. This would be an interesting topic to discuss in the introduction or discussion parts of the paper.\n1. A related problem is when a model is trained on the same private dataset. Wouldn't this have a similar fingerprint on the model activations as distillation? Could this be differentiated from distillation?\n1. Confident predictions from the model are as a signal for data synthesis. Would this be a strong heuristic where the predictions are incorrect. That is, agreeing on a mistake with the teacher is much more informative than agreeing on a correct answer.\n1. It's one thing to predict a score for each teacher, but can this be cast as a likelihood that the models were trained independently, and thus used in a statistical test of independence. This would help in legal application of the method.\n1. Can you comment on how the SVD method in 4.2 will scale for very multiclass scenarios, with many 1000s of labels and potentially sparse activations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes and addresses the task of \"knowledge distillation detection\", wherein the goal is to determine whether an existing model has been distilled from a separate teacher model - given only access to student model weights and the teacher API. The final proposed mechanisms to tackle this problem comprises three primary stages: (1) probing model behaviour via synthetically generated data, (2) extracting statistical scores from model responses, and (3) applying a given decision rule to determine whether distillation has occurred or not - all without making particular requirements on the model architectures of choice.\nExperiments are conducted on smaller image classification and generation scenarios (e.g. ResNet-style models, on CIFAR-10/ImageNet, and models such as SD-v2.1 / SDXL) show high efficacy of the proposed mechanism.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths:__\n* The paper is well written, organised and easy to follow - motivating the task well, before diving into their proposed first approach to tackle this problem.\n* To the best of my knowledge, the proposed distillation detection task, the problem formulation, and the suggested approach to test for distillation is novel.\n* The experiments conducted are extensive, and convincing; showcasing convincing results across baseline methods across both classification distillation detection (Tab 1 and 2), as well as for T2I models in Tab. 3\n\n__Weaknesses:__\n  \nMy main issues lies in the discrepancy between the formulated problem setting and the primary motivation of the paper / distillation detection issue: instead of detecting >whether a model was distilled from another model<, the task is that of detecting which model was distilled from - which excludes the difficult scenario of determining whether any distillation has actually happened. It would be great if the authors could provide insights here; whether the proposed method could be extended to this much more realistic and general scenario, and clearly distinguish the importance of the actually tested scenario of determining which model was distilled from. In a legal issue scenario where someone was accused of distilling e.g. from some proprietary model, just being able to say that there is a higher chance of distillation w.r.t. To some generic baseline would hold value.\n\nQUESTIONS:\nI do think this paper is a relevant contribution - it tries to formalize an important research problem (detecting whether distillation has occured), and within the setting tested does so really well. Unfortunately, I am somewhat struggling to make the connection between the specific research problem and the realistic  scenarios within which distillation detection would occur. It would be great if the authors could provide context here.\n\nLIMITATIONS:\nNot specifically addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}