SelKD: Selective Knowledge Distillation via Optimal Transport Perspective
Authors: Liangliang Shi
Zhengyan Shi
Junchi Yan
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Distillation
Inverse Optimal Transport
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Knowledge Distillation (KD) has been a popular paradigm for training a (smaller) student model from its teacher model. However, little research has been done on the practical scenario where only a subset of the teacher's knowledge needs to be distilled, which we term selective KD (SelKD). This demand is especially pronounced in the era of foundation models, where the teacher model can be significantly larger than the student model. To address this issue, we propose to rethink the knowledge distillation problem from the perspective of Inverse Optimal Transport (IOT). Previous Bayesian frameworks mapped each sample to the probabilities of corresponding labels in an end-to-end manner, which fixed the number of classification categories and hindered effective partial knowledge transfer. In contrast, IOT calculates from the standpoint of transportation or matching, allowing for the flexible selection of samples and their quantities for matching. Traditional logit-based KD can be viewed as a special case within the IOT framework. Building on this IOT foundation, we formalize this setting in the context of classification, where only selected categories from the teacher's category space are required to be recognized by the student in the context of closed-set recognition, which we call closed-set SelKD, enhancing the student's performance on specific subtasks. Furthermore, we extend the closed-set SelKD, introducing an open-set version of SelKD, where the student model is required to provide a "not selected" response for categories outside its assigned task. Experimental results on standard benchmarks demonstrate the superiority of our approach. The source code is available at: \href{https://github.com/machoshi/SelKD}

Published as a conference paper at ICLR 2025
SELKD: SELECTIVE KNOWLEDGE DISTILLATION VIA OPTIMAL TRANSPORT PERSPECTIVE
Liangliang Shi't Zhengyan Shi Junchi Yan
1 Sch. of Computer Science & Sch. of Artificial Intelligence, Shanghai Jiao Tong University Shanghai Artificial Intelligence Laboratory
Equal Contributions Correspondencce Author
(shil (shiliangll miiannniann iing, macho2 021, ya. junchi jusjtu edu.eedducoo cn
ABSTRACT
Knowledge Distillation (KD) has been a popular paradigm for training a (smaller) student model from its teacher model. However, little research has been done the practical scenario where only a subset of the teacher's knowledge needs to be distilled, which we term selective KD (SeIKD). This demand is especially pronounced in the of foundation models, where the teacher model can be significantly larger than the student model. To address this issue, we propose rethink the knowledge distillation problem from the perspective of Inverse Optimal Transport (IOT). Previous Bayesian frameworks mapped each sample to the probabilities of corresponding labels in an end-to-end manner, which fixed the number of classification categories and hindered effective partial knowledge transfer. contrast, IOT calculates from the standpoint of transportation or matching, allowing for the flexible selection of samples and their quantities for matching. Traditional logit-based KD can be viewed as a special case within the IOT framework. Building this IOT foundation, we formalize this setting in the context of classification, where only selected categories from the teacher's category space required to be recognized by the student in the context of closed-set recognition, which we call closed-set SelKD, enhancing the student's performance on specific subtasks. Furthermore, we extend the closed-set SelKD, introducing an open-set version of SelKD, where the student model is required to provide a "not selected" response for categories outside its assigned task. Experimental results on standard benchmarks demonstrate the superiority of our approach. The source code is available at: httpss//gthubb.ccmmmmccchoiiii/s
INTRODUCTION
Knowledge Distillation (KD) (Hinton et al. 2015) has been a popular paradigm to transfer the knowledge from large models (teachers) to small ones (students), which has been widely used in different fields from visual recogmition Kong et al. 2019), speech recognition (Shen et al. 2020).
natural language processing (Jiao et al. 2019). to recommendation systems (Pan et al. 2019 Many approaches have been proposed including matching the intermediate features (Romero et al. 2014).
learning the relationships (Lee al. 2018) and adopting the multiple teachers (I.iu et 2020).
Existing KD methods typically transfer the entire knowledge from one (Sun et al. 2024 or multi- ple (Yuan et al. (222I) teacher models to student model. However, in many real-world applications, it is often preferable for the student model to learn only a subset of the teacher's knowledge. This scenario becomes particularly relevant when the teacher is a large foundation model, while the student model is deployed in eesource-constraingg environments such edge computing. Despite its practical significance, this setting has received little attention in prior work.
To address this gap, we formalize the described setting selective knowledge distillation (SelKD) within the context of classification. Unlike traditional KD, SelKD requires the specification of
"This work was in part supported by NSFC (62222607) and Shanghai Municipal Science and Technology Major Project under Grant 20215222Z0000.

---

Published as a conference paper at ICLR 2025
categories (i.e., subsets of knowledge a side input, allowing the student model to focus exclusively on learning this selected knowledge. This targeted approach makes SelKD particularly applicable to real-world scenarios, where efficiency and task-specific learning are crucial.
SelKD has practical applications in real-world scenarios. Typically, we tend to implement more complex functionalities in relatively larger networks, which often run figh-performance servers. However, for smaller devices such as smartphones and tablets with limited computational power, is often unnecessary replicate the full functionality of models running on large servers. Instead, they may only need to perform specific tasks that are tailored to the device's capabilities. To improve the model performance on different devices, traditional knowledge distillation methods require training multiple teacher models for specific tasks to ensure consistency between teachers and students. However, the advantage of the SelKD) framework lies in the fact that we only need train a single strong teacher classifier capable of recognizing wide range of categories. This teacher model can then be utilized to selectively transfer the relevant knowledge different students with their respective subtasks. As result, there is need to retrain a teacher for each specific task, leading to reduced computational costs and simplified training process.
In this paper, we adopt the inverse optimal transport (IOT) perspective to address the classification problem. We define labels as a set of features, such as one-hot vectors or features extracted by a text encoder. Our goal is to establish matching transportation (i.c., coupling) between the features of images and texts. In this context, the learning process can be seen as the inverse of Entropic Optimal Transport, while the testing inference can be viewed as the optimization of Optimal Transport. From this perspective, we can naturally define the student categories as subset of the categories of the teacher, enabling the knowledge transfer in the SelKD setting.
We propose two distinct settings within our SelKD framework. The first, referred to as (closed- set) SelKD, focuses on the teacher transferring knowledge related only to a specific subtask. The student model is trained exclusively on the data relevant to the assigned subtask and is not required to recognize categories beyond this scope. The second setting introduces open-set SelKD, which extends the framework to handle the recognition of unselected classes. Specifically, in devices, if a sample falls outside the subtask's recognition domain, open-set SelKI) enables the student to provide a "not selected" or "reject due to unknown" response. To tackle this challenge, we employ a modified inverse optimal transport approach that relaxes the Softmax constraint, allowing the row-sum to be less than 1. The contributions of this paper can be summarized follows:
1) We revisit the Knowledge Distillation (KD) problem through the lens of Inverse Optimal Transport (IOT), reformulating the vamilla KD problem as bi-level optimization task. In the inner optimization, the goal is to learn the coupling (i.e., the matching probability) of the student model, which is then supervised by both the ground truth and the teacher's coupling to update the model parameters.
2) Building on this 10T-based formulation, we introduce Selective Knowledge Distillation (SelKD), where the student model is trained to learn only specific subtasks from the teacher model. Additionally, by adjusting the constraints of the original closed-set SelKD for open-set setting, we propose an open-set version of SelKD. The open-set SelKD requires the model recognize "not selected" knowledge, allowing for more flexible and robust response to unassigned tasks.
3) Our proposed method demonstrates superior performance compared to ofate-of-the-art techniques in both closed-set and open-set SelKD tasks, achieving notable improvements in accuracy and robustness. This highlights the effectiveness of the IOT-based KD approach and its potential for diverse real-world applications.
2 RELATED WORKS
2.1 KNOWLEDGE DISTILLATION
Logit-based KD. The idea of training smaller, cheaper models (students) to mimic larger ones (teachers) can be dated back to (Bucila et al. 2006) and it has been applied to neural networks among various tasks including classification (Hinton et al. 2015), speech recognition (Shen et al.
In the context of classification, including open-set settings, we use the terms "categories, "knowledge," and "subtasks" interchangeably refer the designated portions for selective knowledge distillation.
2

---

Published as a conference paper at ICLR 2025
pf aetmals: cat, dog, liger. tree,
A photo untrmabe cath. day, ligst. plates tree, flawerr griss. velicle car, truck bile
photo [ent,dop fiprr) underssom piacho
cbout
aninal
photo of
Teacher
Student know all categooies
animais cat, dog tiget, plante tree, ilover grnes, vehicler can truck, biko
Touhee
Studest's
only knowe
ibout plarm
or unkenww photo
A plato a (ree. flncer grass]
Shudent & undy lnsce
A photool
Varilla Keowhaeleeieeellttttn
Selective Kinnoles Dotillation
oran unknoww photo
[cas, track, Like
Figure I: Illustrative comparison between vanilla knowledge distillation and our selective knowledge distillation with regard to classification tasks. In the vanilla KD framework, the student model learns knowledge from all categories in the teacher model. In contrast, within the selective KD framework, each student independently learns only subset of categories from the teacher model, with the collective knowledge acquired by the students encompassing the entirety of the teacher's knowledge.
2020). natural language processing (Jiao et al. 2019). Large-scale language-iman image pretraining (Wu al. 2023) etc. From a broader perspective, KD can be categorized into three types based how the students learn knowledge from the teachers: logit-based, feature-based and relation-based KD.
In particular, the logit-based KD methods distill the knowledge by aligning the logits between the teacher and student, which can be formulated as a loss as follows (Hinton et 2015):
Liogit CE (ff( r)) A KL((((((x));) o (f((x)::))
(1)
where fa(-) and f+(-) are the sample encoders the student and teacher models, respectively, and y denotes the ground truth (i.e., label) of the sample. a--) is the softmax function mapping the logits to the category probabilities and T is the temperature to control the smoothness of predictive distribution. CE and LL denotes the cross entropy loss and KL divergence, respectively. The parameter X controls the weight between the two items. The concept of logit-based knowledge distil- lation is straightforward and becomes particularly intuitive when viewed as process of knowledge transfer. From another perspective, the effectiveness of soft targets can be compared to techniques such as label smoothing (Kim & Kim 2017) or regularization methods (Müller al.) [019]] Ding et al. 2019). However, traditional logit-based distillation typically relies on the output of the final layer, like soft targets, which overlooks intermediate-leww supervision from the teacher model an essential component for effective representation learning in very deep neural networks (Romero et al., 2014). Additionally, since soft logits reflect class probability distributions, logit-based distillation inherently limited to supervised learning scenarios.
In addition logit-based KD methods, feature-based KDD methods (Yang et al. 2023) primarily focus on aligning the intermediate features between the teacher and student models. This alignment can be expressed as:
Feature-based KD.
Lieature = CE o()::))) + D Diesture (i, || TF(F))),
(2)
where F and F represent the intermediate features from the teacher and student models, respectively. T; and T. are feature transformation mappings for the teacher and student models, used to align the dimensions of F' and Fs. The term Dreaure measures the divergence to quantify the feature difference between the two models, and the parameter À controls the weight between the two items.
Self KD. In self KD, the student model itself plays the role of the teacher. Inspired by the analysis of label smoothing regularization, a teacher-free KD method proposed in Yuuan et al. (019). whose core idea involves the model generating soft labels from its own knowledge and using these labels for training. (Yang et al. 2022) suggests integrating self KD with image mixture and aggregating multi-stage features to generate soft labels. In the paper Li 2022). channel features and layer features are utilized to transfer knowledge without the need for an additional model. To conclude, the main advantage of self KD is that it allows training student model with smaller teacher model size, while achieving performance comparable to the student model trained using a larger teacher model. Compared to SelKD, the teacher model in self KD shares the same task the students.
3

---

Published as a conference paper at ICLR 2025
2.2 OPTIMAL TRANSPORT AND INVERSE OPTIMAL TRANSPORT
As originally introduced by [Kantorovich, 1942), Optimal Transport (OT) Raghvendra et al. 2024
involves solving a linear program (Lahn al. 2023 Phatak et al. 2023 and has become a widely used tool across various research fields, including visual matching (Wang et al. 2013). long-tailed learning (Shi et al.
2024b time series analysis (Zhang et al. 2020; Shi et al. 2020p. multi-modal learning (Shi ct al. 2024a; Wang et al. 2024), and more. Specifically, given the cost matrix C and
two histograms (i.e., probability vectors) a E R bb E TRm Kantorovich's 's OT involves solving the
coupling P (i.e., the joint probability matrix) by
min (C,P)
= Σ
Σ
Cip
(3)
PEU(a,b)
1=1 1=1
where (a, b) is the set of the couplings:
U(a, b) = (P R
Plm a, a.P 1n - b).
(4)
which is bounded and defined by 72 - m equality constraints.
A lot of methods (Bertsimas & Tsitsiklis 1997 Benamou & Brenier 2000) are proposed to solve the Kantorovitch OT problem and relaxing with the entropic regularization (Wilson 1969) is one of the simple but efficient methods, whose objective reads:
min (aU((abb
(C,P)
(5)
where > is the coefficient for entropic regularization H(P) and the H(P) can be specified as
H(P) - Σ P. (log (Py-
(6)
i.J
The objective in Eq. 5 is an e-strongly convex function, and thus the optimization has a unique solution, which can be solved with iterative methods (e.g. the Sinkorn method (Sinkhorm) 1967)). If we use this entropic regularized OT to solve the matching problem, the hard matching problem may convert to soft matching.
Inverse Optimal Transport (IOT) has been explored in several studies (Dupuy et al. 2016 et al. | & Wolfram 2020), aiming infer the unknown cost matrix that generates the observed coupling. The work by (tturtt & Wolfram 2020) presents a systematic approach for inferring these
unknown costs, while (Chiu et al.) 2022) develops the mathematical theory underpinning IOT. In addition, (Shi et al.) (202)) demonstrates a brand new series of contrastive losses with set matching based on IOT. The 1OT problem can be formulated as a bi-level optimization problem:
mjn (L P²) where
pd arg min (C., P) (P) PeU(a.b)
where P is the ground truth for supervision. IOT facilitates the capture of fine-grained relationships between sample features, thereby enhancing the transfer of structured knowledge from the teacher model to the student model during the knowledge distillation process.
3 METHODOLOGY AND FORMULATIONS
In contrast to vanilla KD where the student learns all the information from the teacher, we propose the setting of Selective KD (SelKD) that transfers only selective knowledge the student. Without loss of generality, in this paper we view the classification task with optimal transport, in which labels are defined as a set of features (e.g. one-hot vectors or features extracted by text encoder) and images are also represented with features extracted by image encoder. Our goal is to establish match or transportation between the features of images and texts with the formulation of OT. In this case, variants of optimal transport with specific properties could be introduced to solve the KD problem.
3.1 OPTIMAL TRANSPORT FORMULATED (LOGIT-BASED) KNOWLEDGE DISTILLATION
Teacher Training via 1OT perspective. Given the batch data Nz) where y; is the one-hot vector corresponding to sample x; and N is the batch size, the features of the samples and labels can
4

---

Published as a conference paper at ICLR 2025
Images
Labels
dog
flower
wall
Features of images
Features of labels
Cost Matrix
plane
sky
Teacher
Teacher
Image
Label
Encoder
Encoder
Subset Labels
Subset Images
dog
Features of images
Cost Matrix
Features labels
cac
Student Image Encoder
Student
Label
plane
Encoder
Constraints
forward
Close-set 11)
Open-set 8- (P1 11,1'11-1
backward
Couplings
Y
P
Y
Ground Truth
KL Divergence
Cross Entropy
Figure 2:
The overview of our approach for both closed-set and open-set SelKD tasks. We first compute cost matrices with features extracted from image samples and labels with encoders (CNN) respectively. The regularized OT is used to analyze and estimate the coupling of the student, which is supervised with ground truth and the coupling of the teacher for representation learning.
The label matrix is denoted as Y = M The training process of the teacher model can then be reformulated bi-level optimization:
be represented by two sets: [fo(x:)
min KL(Y PO) where
Pd
arg min (C P) cH(P).
P1=1
Here the cost matrix C E T. xaxxm;;,, is designed with features {fo(x))) (go (y;)) with param- eters from the networks fe and go. This bi-level optimization consists of an outer and an inner optimization. As proved in (Shi et al. 20033 the outer optimization can be regarded as minimization of the cross-entropy loss, while the inner optimization, resembling an optimal transport (OT) problem, operates like Softmax function, with the constraint P1 = 1 ensuring that the probabilities across each category sum 1. A detailed explanation can be found in Appendix B. These constraints can be interpreted as simplified versions of the row and column sum constraints U(a, b) in OT. From this perspective, the teacher's learning process can be viewed as bi-level optimization within the framework of entropic regularized OT. Compared the Bayesian approach, this method offers greater flexibility in selecting desired categories, making it well-suited for implementing SelKD framework.
(Logit-based) KD via 1OT perspective. Inspired by the Inverse Optimal Transport (IOT) framework, as described in Eq. 7 we propose a novel approach to integrate IOT constraints into the knowledge distillation (KD) process. Using logit-based KD as example, reformulate the knowledge distillation process as a bi-level optimization problem:
min CE(Y P.. + X KL(P, P
0
where P. arg min (C,, P) (H) D arg min (C.,P) - P1=1 Here H(P) is the entropic regularization as defined in Eq, 6 and Y denotes the label matrix. In this optimization, the cost matrix C₁ E Raxm is designed with features from the pre-trained teacher
P1=1

---

In traditional knowledge distillation, the primary objective is to transfer the entire knowledge of the teacher model to the student model in straightforward end-to-end manner. However, this approach can be inefficient and overly complex, especially in real-world applications where student model may only need to perform subset of tasks. Our proposed method, Selective Knowledge Distillation (SelKD), addresses this limitation by enabling the student model to learn only the relevant knowledge from the teacher model for specific subtasks, with evaluations focused solely on those areas during testing. Figure iillustates the difference between vanilla KD and our proposed SelKD. formalize general Published as a conference paper at ICLR 2025
networks fi and It. while C. R is computed with features (fa,0(x,)) (g,,0(y))) with parameters 9 from the networks fe.0 and ga.0. Different from previous works setting U = [P P1 1). we think the constraint U can be designed according to the specific circumstances of the problem, especially in the case of open-set tasks. We will discuss it in detail in the next subsection. In the outer minimization, the coupling P is calculated by OT's inner minimization and Y is the ground truth for supervision.
The aim of outer minimization is to supervise the student coupling with the ground truth and the teacher coupling. in order learn the feature extractor (i.e. (=, and gs,o(-)). Simultaneously, the inner minimization formulates the distillation problem as an entropic regularized optimal transport task. Our overarching goal derive the student coupling P that aligns with the teacher coupling Pt, utilizing the respective cost matrices Cg and C₁-
To summarize, this innovative perspective on logit- based knowledge distillation (KD) through the lens of Inverse Optimal Transport (IOT) reformulates the distillation process as a bi-level optimization problem. By incorporating IOT constraints, allows for more structured approach to aligning the teacher's 's and student's knowledge. In addition, the 10T-based view leads to the introduction of Selective Knowledge Distillation (SelKD), which focuses on targeted and efficient knowledge transfer and allows the student model to selectively learn relevant knowledge for specific subtasks.
3.2 SELECTIVE KNOWLEDGE DISTILLATION VIA INVERSE OPTIMAL TRANSPORT
Leveraging the perspective of Inverse Optimal Transport (IOT) in Eq. bi-level optimization framework for SelKD, expressed as follows:
min CE(Y PO.) + A KL(P, PP),
8
(10)
where
P. = arg arngmmn (C, P) - cHH(P) P = arg min min -H((P.
In this formulation, U represents the general constraints for the coupling. A key advantage of this approach is that it allows us to tailor the cost matrices C and C, according the specific categories we select for distillation. This flexibility enables a more nuanced and effective knowledge transfer, as the constraints can be adjusted based on the particularities of the tasks hand.
In the following sections, we will detail how to specify the constraints U and select feasible categories to formalize the optimization process, whether in closed-set open-set scenarios.
3.2.1 (CLOSED-SET) SELECTIVE KNOWLEDGE DISTILLATION
We begin by analyzing Eq. 10 under the constraint U = (P: P1 11. In this context, we partition
the training set based on the selected categories. Specifically, we define the complete category set
as C = {1, 2, .N} where N represents the total number of categories. We set the entire dataset S = [(x;, yt ) where M the size of dataset and y; is the one-hot vector corresponding to sample
x; Without loss of generality, denoting the selected categories set Celneed = [1, 2. ... n)) C C
with n < N, we denote Schosed set = {(x,, y:) y; E one-hot(Cekkwed-e }. Here one-hot(.) is a mapping function that converts all elements in the set to their one-hot representations.
For the batch data ((x,, y:) y:) C Selowed-set where m is the batch size, the features of samples and la- bels can be represented by two sets { (x:)) (x:)) (y,)) and (x:))))) (ge ) regarding student and teacher extracted features. Denoting Simage {(xx) ( without loss of gen- crality, we set C, and C₁ as follows:
(C -x,,(xx)) (y,) (Ct)ij = (x(xs) ge(ys) for i E Simages j E Celosed set (11)

---

Published as a conference paper at ICLR 2025
For traditional classifiers, f(-) represents the image encoder and g(+) represents the label encoder (e.g.. one-hot encoding). For multimodal classifiers based on CLIP (Radford et al. 2021). f()) represents the image encoder while g(-) represents the text encoder. Our formulation generalzes this approach, making it more adaptable to a wider range of methods.
compute the cost matrices using features extracted from image samples and labels via encoders (CNN). These cost matrices are then incorporated into Eq. [1] forming the optimization formula for SelKD. To facilitate understanding, the cost matrices can be interpreted as the negative of similarity matrices (ignoring constant factors in the optimization problem).
Figure 2 illustrates the pipeline of method. Initially,
3.2.2 OPEN-SET VERSION FOR SELECTIVE KNOWLEDGE DISTILLATION
We propose open-set SelKD, an ex- tension of the SelKD framework men- tioned above, designed to address the recognition of classes that were not in- cluded in the student's 's training subset. Specifically, in smaller or resource- constrained devices, SelKD tasks does not allow the student model be trained on the full set of classes from teacher model. Open-set SelKD re- quires student models to handle cases where an input sample falls outside the scope of the subtask that the stu- dent has been trained to recognize. In such situations, rather than attempt- ing to force a classification, the stu- dent model is equipped to produce a "not selected" or "reject due to un- known" response, effectively acknowl- edging that the sample does not be- long to any the known classes. This ability to reject unknown inputs en- hances the robustness and applicabil- ity of SelKD, especially in open-set or dynamic environments where new or unseen classes may emerge.
Algorithm 1: Computing the KL divergence between student and teacher couplings under Open-set SelKD
Input :features F Fimage.t and F Finage.s extracted from image samples, F and Fiahel.s extracted from labels, (footnotes and teacher models, respectively), temperature and parameter 7
represents student
Output :oss function C
Initialize C, = F image. F lahel. and C, F image, e label, C./r and p(")
Compute p!0]
C./T
for 1 1,2, .... I. do
P
P!O
max (P 1,1)
P (+1) P
1T
P(() maxt max(P (P( 1,1)
p!o
(((+))
[!]
1TP
end return C= (P, log(P.))
To formulate open-set SelKD with an optimization based on the closed-set version, we first relax the constraints by setting P1 1 to a new one motivated by Partial Optimal Transport given
UM (P1 < 1, P1
P1 P1=1)
(12)
where y is is the number of batch sample classified to the categories of the subtask.
Similarly, we define the complete category set C = (1, 21, .N3 where N represents the total number of categories. We set the entire dataset S [x,, y where M is the size of dataset and y; is the one-hot vector corresponding to sample x, Without loss of generality, denoting the selected categories set sel = {1, 21,2,- ... ,nn} C C with n < N, we denote Copen-set = Celosed set U {n + 1} where n + represents the union of "not selected" categories in the student model.
For the batch data ()) C S, the features of samples and labels can be represented by two sets { (xx)) {ge, ey ( and )m { ( with footnotes. Denoting Sunage { mm }. without loss of generality, we set C, and C, as follows:
(C.) (xx,0(x;) gg,0())) (Ct)ss -x((x)) 9x(y)) for i Simager E Copenn set (13)
7

---

Published as a conference paper at ICLR 2025
Table 1: Top-1 accuracy (%) on CIFAR-100 for closed-set SelKD. We compare the performance of vanilla knowledge distillation (KD), self-knowledge distillation (SKD) and our selective knowl- edge distillation (SeIKD) the closed-set SclKD classification tasks. All the comparisons are
conducted based on feature-basse based distillation (FitNet Romero et al. 2014 FT Kim et al. (2018) and AT Zagoruyko & Komodakis (2016))
Student
SubTask
SubTask
SubTask
SubTask
SubTask
Mcan of
#Param of Students
Methods
Netwurks
0
2
SubTasks
Teachers
Without
81.00
82.45
80.85
76.90
83.30
80.90
0.4M
Fi
81,50
83,55
82.20
78.50
85.65
82.28
117.7M 0.4M
FitNet-SKD
81.70
83.25
81.55
77.75
85.05
81.86
0.4M 0.4M
FilNet-SelKD
82,95
84.45
82.25
78.85
85.90
82.88
23.7M + 0.4M
FT-KD
81.50
83,50
82.95
78.50
86.15
82.52
117.7M 0.4M
ResNet-8
FT-SKD
81.20
82.80
80.70
77.80
81.24
0.4M
FT-ScIKD
82.75
84.20
82.85
80.35
86,35
83,30
23.7M 0.4M
AT-KD
81.05
84.35
81,60
78.10
85.05
82.03
117.7M 0.4M
AT-SKD
80.15
82.80
81.25
77.40
81.02
0.4M
AT-SelKD
82.45
84.70
82.20
78.50
85.30
82.63
23.7M 0.4M
Without KD
83.60
85,40
84,10
81.20
86.55
84.17
0 0.9M
FilNet-KD
83.60
86.00
84.85
81.55
88.00
84.80
117.7M + 0.9M
Fit
83.00
84.05
81.80
79.10
85.00
82.59
0.9M 0.9M
FilNet-SeIDD
84.30
86.20
84,90
81.85
85.11
23.7M 0.9M
FT-KD
83.90
85,90
84.85
81.65
87.85
84.83
117.7M 0.9M
ResNet-14
FT-SKD
83.05
85.90
82.55
79.75
86.80
83.61
0.9M 0.9M
FT-SeIKD
85.30
87.55
84.75
82,80
87.90
85.66
23.7M 0.9M
AT-KD
84.05
86.75
84.60
81.45
87.00
84.77
117.7M 0.9M
AT-SKD
83.05
83.95
82.00
79.35
85.20
82.71
0.9M 0.9M
AT-ScIKD
85.00
87.05
85.75
82.95
87.25
85.60
23.7M 0.9M
Then the optimization can be modified as
min CE(Y P. + KL(P, PP?),
0
where
P. arg
min
(C,,P) - cHH(P)
(14)
PISL,TP1-5
pd arg
min
(C P) cH(P)
PISLI
The entire process can be summarized by Algorithm For the prediction in the inference process, we calculate Eq. [4]give the batch testing data. Then for the prediction of sample we do the arg max operation (P.)a.j on every j and 1 (pP)),. as the result.
4
EXPERIMENTS
4.1 BASIC SETTINGS
Our experiments are performed using PyTorch 1.4.0 and run on Intel Core i7-7820X CPU @ 3.60GHz with Nvidia GeForce RTX 3080. We take single GPU for classification on CIFAR-10. 10, CIFAR-100 (Krizhevsky al. 2009) and Tiny ImageNet (Le & Yang 2015). and evaluate testing data by top-1 accuracy.
Experimental Setting Details. For CIFAR-10 and CIFAR-100, we adopt ResNer50 (He al. 2016) as the backbone teacher model, while for Tiny ImageNet, settings for students also vary depending on datasets. For CIFAR- 10 and CIFAR-100, the experiments of image classification tasks are based ResNet8 and ResNet14 the backbone of students. For Tiny ImageNet dataset, we adopt only ResNet8 for training. As for the learning rate, we set 0.05 for all tasks with regard to CIFAR- 10 and CIFAR- 100 datasets. For Tiny ImageNet, learning rate is 0.2.
ResNet32 for training. The
As the primary focus this paper is to pose the problem of selective knowledge distillation, and to find feasible ways work the problem, we do not adopt additional specialized techniques
8

---

Published as a conference paper at ICLR 2025
Table 2: Top-1 accuracy (%) Tiny ImageNet for closed-set SelKD with ResNet-8 backbone. We compare the performance of vanilla knowledge distillation (KD) and our selective knowledge distillation (SelKD) in the closed-set SclKD classification tasks.
Methods
SubTask 0
SubTask
SubTask 2
SubTask 3
SubTask 4
Mcan of SubTasks
Without KD
52.05
53,40
53.00
50.15
54.85
52.69
Fi
52.55
54.00
54.10
51.25
55.60
53.50
FiNNet-SelKD
53,20
54.95
54,80
52.05
56,40
54.28
FT-KD
52.45
53,95
54.15
52.00
55.60
53.63
FT-ScIKD
53.15
55.05
54.70
52.10
56.55
54.31
AT-KD
52.95
54,60
54,45
51.05
55.25
53.66
AT-SelKD
52.90
54.70
55.20
51.90
55.90
54.12
improve performance, such as resampling. This is to control variables and thus all the baselines used in this study represent only the method of knowledge distillation proposed by them.
4.2 EXPERIMENTS ON (CLOSED-SET) SELKD
For SclKD, we decompose the overall classification task into various subtasks, with scales varying by dataset. Specifically, for the CIFAR-10 dataset, which consists of 10 classes, we split the task into subtasks, each containing classes. contrast, CIFAR- -10 and Tiny ImageNet, with 100 classes each, are divided into 5 subtasks, each containing 20 classes. We combine feature-based KD methods with different KD frameworks, including vanilla KD, self KD and our SelKD, for comprehensive comparison to highlight the advantages of our SelKD framework. For feature- KD, select typical methods including FitNet (Romero et al.) 2014). FT (Kim et al. 2018), and AT (Zaorrykoo & Komodakis 2016). Notably, both vanilla KD (D) and self-KI (SKD), train separate teachers for each subtask, while in SelKD approach, a single teacher is trained to cover all subtasks.
The results of SelKD tasks the CIFAR-100 and Tiny ImageNet dataset presented in Table T and T with results for CIFAR-10 10 is shown in the Appendix [ Our findings clearly indicate that, regardless of the feature-based distillation method employed, SelKD outperforms both KD and SKD in terms of top-1 accuracy. Furthermore, our SelKD method requires fewer parameters compared KD, which necessitates training an additional teacher for each subtask. Experimental results suggest that teacher with comprehensive knowledge enhances the performance of subtask students more effectively than multiple teachers with knowledge limited to specific areas. This is because the additional knowledge from the comprehensive teacher aids students in mastering the selected knowledge.
4.3 EXPERIMENTS ON OPEN-SET SELKD
We further examine the application of our SelKD method on open-set SelKD tasks. We first apply Algorithm on all the tasks similar to those of closed-set SelKD, but we add an extra class for "not selected" knowledge in each subtask student model. Specifically, for CIFAR- 10 dataset, the whole task is separated into subtasks with classes each for selected classes and for all "not selected" classes), while both CIFAR-100 and Tiny ImageNet classification tasks are separated into 5 subtasks with 21 classes each (20 for selected classes and for all "not selected" classes). Table demonstrates the results of open-set SclKD experiments CIFAR-100.
T
have observed that, in line with the results in closed-set SelKD tasks, the experiments conducted on the open-set SelKD tasks reveal that training group of students with different disjoint tasks is more advantageous when facilitated by a teacher possessing comprehensive knowledge, as opposed to assigning separate teachers who only well-versed in selected knowledge for each task. This finding also highlights the notable advantage of employing our selective knowledge distillation method in predicting samples that belong "not selected" knowledge.

---

Published as a conference paper at ICLR 2025
Table 3: Top-1 accuracy (%) on CIFAR-100 for open-set SelKD. We compare the performance of vanilla knowledge distillation (KD), self- self-knowledge distillation (SKD) and our selective knowl- edge distillation (ScIKD) in the open-set SelKD classification tasks. All the comparisons are
conducted based on feature- based distillation (FitNet Romero et al.
2014 FT Kim et al. (2018) and
AT AT Zagoruyko & Komodakis (2016))
Student
SubTask
SubTask
SubTask 2
SubTask 3
SubTask
Mcan of
#Param of Students
Methods
Netwurks
0
SubTasks
Teachers
Without KD
82.19
82.82
82.62
82.44
81.42
82.30
0 0.4M
Fi
84.65
85,20
85.69
85.72
84,20
85.10
117.7M 0.4M
FitNet-SKD
84.77
85.41
85.60
85.54
84.62
85.19
0.4M 0.4M
FilNet-SelKD
85.63
86.46
86.69
86.61
85.65
86.21
23.7M + 0.4M
FT-KD
84.83
85.32
86.03
86,05
84.82
85.41
117.7M 0.4M
ResNet-8
FT-SKD
84.79
85.06
85.48
85.47
84,86 86
85.13
0.4M
FT-ScIKD
86.17
86.62
87,28
86,99
86,35
86.69
23.7M 0.4M
AT-KD
84.46
85.32
85.85
86.16
84.31
85.22
117.7M 0.4M
AT-SKD
84.62
85,21
85.56
85.69
84.73
85.16
0.4M 0.4M
AT-SelKD
85.81
86.57
87.20
86.59
85.76
86.39
23.7M 0.4M
Without KD
84.15
83.68
83.15
84.22
82.93
83.63
0 0.9M
FilNet-KD
86.92
87.21
87.64
87.75
87.26
87.36
117.7M + 0.9M
Fit
87.52
87.54
87.54
87.76
87.95
87.54
0.9M 0.9M
FilNet-SeIDD
87.27
87.70
87.81
87.85
87.68
87.66
23.7M 0.9M
FT-KD
87.46
88.04
88.24
88.44
88.02
88.04
117.7M 0.9M
ResNet-14
FT-SKD
87.33
87.99
88.04
87.96
87.92
87.86
0.9M 0.9M
FT-SeIKD
87.60
88.46
88.24
88.82
88.64
88.35
23.7M 0.9M
AT-KD
86.93
87,58
87.96
88.04
87.29
87.56
117.7M 0.9M
AT-SKD
86.54
87.13
87.15
86.61
85.90
86.67
0.9M 0.9M
AT-ScIKD
87.03
87.67
88.04
88.01
87.47
87.64
23.7M 0.9M
Table 4: Ablation Study of SelKD on CIFAR-100.
Loss Settings
SubTask
SubTask
SubTask
SubTask
SubTask
Mean SubTasks
Without Distillation
83,50
85.85
83,20
80.95
87.45
84.19
Without Logit-based Loss
83.60
86.05
80.60
85.85
83.98
Without Peature-based Loss
83.95
86.15
84.70
81.80
88.15
84.95
84.30
86.20
81.85
88.30
85.11
4.4 ABLATION STUDY
Regarding the previously presented experiments, the loss function be divided into three parts, namely from cross-entropy loss with dataset labels, KL divergence with teacher coupling and the divergence with features from the teacher. Here we further explore how the accuracy of the student model is influenced when certain components are absent from the three aforementioned parts. To be more precise, we conduct separate tests to evaluate the impact of each component's 's absence and compare the obtained results with the original outcome. The results of these tests summarized and presented in the Table
Based on the analysis of the table data, it is apparent that the removal of any component from the loss function results in a reduction in the accuracy of the student model. Hence, it is imperative to include all parts of losses in the final settings when seeking more suitable training method for handling Selective KD tasks, Each module plays a crucial role and is indispensable for best performance.
5 CONCLUSION
We have proposed a new and practical setting for Knowledge Distillation, called Selective Knowledge Distillation (SelKD), which transfers the partial knowledge to student instead of the whole knowledge in vanilla KD. OT is applied for the SelKD, to help the student learn the subtask. Our current work is focused classification (including open-set setting).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the monge- kantorovich mass transfer problem. Numerische Mathematik, 84(3):375-393, 2000.
Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. Itera- tive bregman projections for regularized transportation problems. SIAM Journal on Scientific C 37(2) 138, 2015.
Dimitris Bertsimas and John N Tsitsiklis. Introduction to linear optimization, volume 6. Athena Scientific Belmont, MA, 1997.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Knowl- edge Discovery and Data Mining, 2006. URL https://appp...... semanti www.seetttttschlll.......ccccc
[orpus 1121339772
Wei-Ting Chiu, Pei Wang, and Patrick Shafto. Discrete probabilistic inverse optimal transport, In International Conference on Machine Learning, pp. 3925-3946. PMLR, 2022.
Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Shu-Tao Xia. Adaptive regularization of labels. arXiv preprint arXiv: 05474, 2019.
Amaudd Dupuy, Alfred Galichon, and Yifei Sun. Estimating matching affinity matrix under low-rank constraints. arXiv preprint 1rXiv:1612.09555 2016
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02533 2015.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 1rXiv:1909.10351 2019.
Kantorovich. On the transfer of masses (in russian). Doklady Akademii Nauk, volume 37, pp. 227, 1942.
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 31, 2018.
Seung Wook Kim and Hyo-Eun Kim. Transferring knowledge to smaller network with class-distance loss. 2017.
Ru Kong, Jingwei Li, Csaba Orban, Mert R Sabuncu, Hesheng Liu, Alexander Schaefer, Nanbo Sun, Xi-Nian Zuo, Avram Holmes, Simon Eickhoff, al. Spatial topography of individualsppecie cortical networks predicts human cognition, personality, and emotion. Cerebral cortex. 29(6): 2533-2551, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Nathaniel Lahn, Sharath Raghvendra, and Kaiyi Zhang. A combinatorial algorithm for approximating the optimal transport in the parallel and mpe settings. Advances Neural Information Processing Systems, 06:21675-21666 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Seung Hyun Lee, Dae Ha Kim, and Byung Cheol Song. Self-supervised knowledge distillation using singular value decomposition. In Proceedings of the European conference computer vision (ECCV), pp. 335- 2018.
Lujun Li. Self-regulated feature learning via teacher-frr free feature distillation. In European Conference Computer Vision, pp. 347-363. Springer, 2022.
11

---

Published as a conference paper at ICLR 2025
Ruilin Li, Xiaojing Ye, Haomin Zhou, and Hongyuan Zha. Learning match via inverse optimal transport. Journal of machine learning research, 20(80):1-37, 2019.
Yuang Liu, Wei Zhang, and Jun Wang. Adaptive multi-teacher multi-level knowledge distillation. Neurocomputiing 415:106-1181 113, 2020.
Rafael Müller, Simon Korblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Yiteng Pan, Fazhi He, and Haiping Yu. A novel enhanced collaborative autoencoder with knowledge distillation for top-n recommender systems. Neurocomputing, 332: 37-148, 2019.
Abhijeet Phatak, Sharath Raghvendra, Chittaranjan Tripathy, and Kaiyi Zhang. Computing all optimal partial transports. In International Conference on Learning Representutions, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. PMLR, 2021.
Sharath Raghvendra, Pouyan Shirzadian, and Kaiyi Zhang. new robust partial p-wasserstein-bane metric for comparing distributions. In International Conference on Machine Learning, 2024.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 2014.
Peng Shen, Xugang Lu, Sheng Li, and Hisashi Kawai. Knowledge distillation-based representation learning for short-utterance spoken language identification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2674-2683, 2020.
Liangliang Shi, Peili Lu, and Junchi Yan. Causality learning from time series data for the industrial finance analysis via the multi-dimensional point process. Intelligent Automation & Soft Computing, 26(5), 2020.
Liangliang Shi, Gu Zhang, Haoyu Zhen, Jintao Fan, and Junchi Yan. Understanding and generalizing contrastive learning from the inverse optimal transport perspective. In International conference machine learning, pp. 31408-31421. PMLR, 2023.
Liangliang Shi, Jack Fan, and Junchi Yan. Ot-elip: Understanding and generalizing clip via optimal transport. In Forty-first International Conference on Machine Learning, 2024a.
Liangliang Shi, Zhaoqi Shen, and Junchi Yan. Double-bounded optimal transport for advanced clustering and classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38. pp. 11982-149900 2024b.
Liangliang Shi, Haoyu Zhen, Gu Zhang, and Junchi Yan. Relative entropic optimal transport: (prior-aware matching perspective to (unbalanced) classification. Advances in Neural Information Processing Systems, 36, 2024c.
Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathemutical Monthly, 74(4):402-405, 1967.
Andrew M Stuart and Marie-Therese Wolfram. Inverse optimal transport. SIAM Journal on Applied Mathematics, 80(1):599-619, 2020.
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15731-15740, 2024.
Guolong Wang, Xun Wu. Zheng Qin, and Liangliang Shi. Routing evidence for unseen actions in video moment retrieval. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pp. 3024-00243 3035, 2024.
12

---

Published as a conference paper at ICLR 2025
Wei Wang, Dejan Slepčev, Saurav Basu, John A Ozolek, and Gustavo K Rohde. A linear optimal transportation framework for quantifying and visualizing variations in sets of images. International journal of computer vision, 101:254-200 2013.
Alan Geoffrey Wilson. The use of entropy maximising models, in theory of trip distribution, mode split and route split. Journal of transporl economics and policy, pp. 108-126, 1969.
Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang. et al. Tinyclip: Clip distillation via affinity mimicking and weight inheritance. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21970-21980, 2023.
Chuanguang Yang. Zhulin An, Helong Zhou, Linhang Cai, Xiang Zhi, Jiwen Wu, Yongjun Xu, and Qian Zhang, Mixskd: Self-knowledge distillation from mixup for image recognition. In European Conference on Computer Vision, pp. 534-551. Springer, 2022.
Chuanguang Yang, Xinqiang Yu, Zhulin An, and Yongjun Xu. Categories of response-based, feature- based, and relation-based knowledge distillation. In Advancements in knowledge distillation: towards new horizons of intelligent systems, pp. 1-32. Springer, 2023.
Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, and Daxin Jiang. Reinforced multi-teacher selection for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14284-14291, 2021.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: a teacher-free framework. 2019.
Sergey Zagoruyko. Wide residual networks. arXiv preprint arXiv: 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint 2016.
Zheng Zhang, Ping Tang, and Thomas Corpetti. Time adaptive optimal transport: A framework of time series similarity measure. IEEE Access, 8:149764-149771 2020.
A MORE EXPERIMENT RESULT
The result of closed-set SelKI) tasks on the CIFAR-10 dataset are presented in Table 5 and is obvious that SelKD outperforms both KD and SKD in terms top-1 accuracy.
We further add more experiments for closed-set SelKD on CIFAR-100 with WideResNet-40- 2 (Zaooruyko 2016) the teacher model and 1ideResNet-16-22 as the student model. The results are shown in Table 6
B A DETAILED EXPLANATION OF THE BI-LEVEL OPTIMIZATION
We mainly follow (Shi et al. 2023) that understanding designing the loss via bi-level optimization:
min K Y Pd) s.t. pd arg min (C P) - eH(() P1=1
where C represents the cosine distance for image feature and text/label feature, with parameters 6.
and Y is the known supervision for learning. As proven in (Shi et al. 2202)), H(P) = - PP the entropic regularization with coefficient e. The inner optimization exactly equivalent the softmax activation, while the outer optimization corresponds to cross-entropy. Thus we can find the above bi-level optimization equals InfoNCE loss:
min C = Σ Yig log(
Σ C-
13

---

Published as a conference paper at ICLR 2025
Table 5: Top-1 accuracy (%) CIFAR-I0 for closed-set SelKD. We compare the performance of vanilla knowledge distillation (KD), self-knowledge distillation (SKD) and our selective knowledge distillation (SeIKD) in the closed-set SclKD classification tasks.
Student
Mean of SubTasks
Methods
SubTask 0 SubTask
Networks
Without KD
91.28
94.76
93.02
FitNet-KD
91.56
95.72
93.64
FilNet-SKD
91.42
95.34
93.12
FitNet-SelKD
91.88
95.88
93.88
FT-KD
91.98
96.22
94.10
ResNet-8
FT-SKD
91.53
95.99
93.25
FT-SelKD
93.68
96.50
95.09
AT-KD
91.66
95.60
93.63
AT-SKD
91.68
95.09
93.65
AT-SelKD
92.90
96.12
94.51
Without KD
94.42
96.54
95.48
Ki
94.86
96.40
95.63
FitNet-SKD
91.56
95.72
93.64
SelKD
95.04
97.72
96.38
FT-KD
94.56
97.04
95.80
ResNet-14
FT-SKD
91.56
95.72
93.64
FT-SeIKD
95.66
97.92
96.79
AT-KD
94.02
96.76
95.39
AT-SKD
91.56
95.72
93.64
AT-SeIKD
94.84
97.64
96.24
Table Results for closed-set SelKD on CIFAR-100 with WideResNet-44-22a as the teacher model and WideResNet-16-2 as the student model.
Methods
SubTask O
SubTask
SubTask 2
SubTask
SubTask 4
Mean
FitNet-KD
85.67
86.52
85.14
81.97
87.72
85,40
FiLNet-SeIKD
86.15
87.68
85.83
82.76
88.04
86.09
FT-KD
85.42
85.56
86.08
82.13
88.45
85.53
FT-SeIKD
85.92
87.23
86.17
82.56
88.73
86.12
AT-KD 85.17
85,49
85,43
81.87
87.71
85.13
AT-SelKD
85.55
87.25
86.38
82.41
88.52
86.02
Thus, bi-level optimization is fundamentally method for designing activation layers or loss functions. In (Shi et al., 2023), modifications the inner optimization improve the loss. Our work follows this learning framework. but we modify the inner optimization with new constraints adapt open-set scenarios, solving with iterative algorithm (Benamou et al. 2015) to obtain the predicted probability matching matrix. The outer optimization is adjusted to the original KL Divergence in KD the loss, resulting in an application KD problems.
14


Reviews:
[{"review_id": "Ks2atsZ9ed", "reviewer": "Reviewer_9LKC", "summary": "This paper discusses the knowledge distillation (KD) problem from the perspective of inverse optimal transport (IOT), introducing a selective KD (SelKD) framework tailored for classification tasks. Specifically, closed-set SelKD allows the student model trained exclusively on the data relevant to specific tasks, while open-set SelKD enables the student model to identify non-selected knowledge to unassigned tasks. Experimental results on image datasets showcase the superior performance of proposed SelKD over the state-of-the-art methods.", "questions": "Please refer to the weaknesses mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper formulates the KD problem as a bi-level optimization through the perspective of IOT. Building on the IOT-based formulation, this paper further proposes novel closed-set and open-set SelKD methods that allow the student model to learn specific knowledge from assigned tasks, making SelfKD well-suited for real-world, resource-constrained environments.", "weaknesses": "However, there are several weaknesses where this paper can improve:\n1.  The advantages of the proposed SelKD over vanilla KD are unclear in Figure 1. What are the benefits of training multiple students, each tailored to a specific task, when a general student model showcases consistently strong performance across all tasks in vanilla KD? SelKD could involve higher computational costs due to the need for training multiple task-specific student models. \n\n2. To strengthen the justification for the proposed SelKD in resource-constrained environments, the paper should include additional motivation experiments or theoretical analysis demonstrating that a student model cannot fully receive the knowledge from the teacher model, or in other words, that the teacher fails to transfer essential task-specific knowledge to the student. \n\n3. How are the teacher and student models practically deployed/trained? It is challenging to simultaneously train a cumbersome teacher and student on resource-constrained edge device. \n\n4. In Eqn (11), why is the cost matrix calculated through the multiplication of $f(\\cdot)$ and $g(\\cdot)$? Is there a specific reason or advantage for using this multiplicative form?\n\n5. The data storage and computational capacity are often limited in resource-constrained environments. Thus, the proposed SelKD is expected to exhibit strong robustness against insufficient training data, while also ensuring efficient computational cost and resource utilization.\n\n6. How are subtasks determined? Can the proposed SelKD consistently maintain superior performance when the categories within the same subtask significantly differ from each other?\n\n7. Some multi-task KD studies [1-3] should be considered as baselines in the experiments. \n\n[1] Raphael Tang, et al. \"Distilling task-specific knowledge from BERT into simple neural networks.\" arXiv, 2019.\n\n[2] Geethu Miriam Jacob, et al. \"Online knowledge distillation for multi-task learning.\" WACV, 2023.\n\n[3] Yangyang Xu, et al. \"Multi-task learning with knowledge distillation for dense prediction.\" ICCV, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H8jMPSfHpn", "reviewer": "Reviewer_o62s", "summary": "The paper proposes Selective Knowledge Distillation (SelKD) for transferring only selected parts of a teacher model’s knowledge to a student model, framed within the Inverse Optimal Transport (IOT) perspective. This method is relevant for scenarios with resource constraints, where distilling only relevant subsets of knowledge is more practical than full-model distillation. The authors present closed-set and open-set SelKD, where the student learns a subset of categories or can reject classes outside its training scope. Experiments on CIFAR and Tiny ImageNet demonstrate SelKD’s advantages over traditional KD techniques, with improved efficiency and comparable performance on targeted tasks.", "questions": "Would the authors consider exploring SelKD in semi-supervised or unsupervised settings where labeled data for all categories may not be available?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a practical challenge in knowledge distillation, particularly relevant to deploying models on edge devices with limited resources. By reinterpreting KD within an IOT framework, the authors introduce an innovative approach to selective knowledge transfer that has potential for efficient model adaptation in constrained environments. The experiments on CIFAR and Tiny ImageNet datasets validate the method’s utility, showing that SelKD can effectively focus on specific subtasks with lower computational costs.", "weaknesses": "1. The experimental validation is somewhat limited, with results provided only on relatively small datasets (CIFAR and TinyImagenet). The model used in the study is relatively small (ResNet50), with significantly fewer parameters than current large-scale pretrained models. I recommend the authors add experiments using larger backbone models such as ViT.\n2. Since SelKD relies on label information, it is challenging to extend it to semi-supervised and unsupervised settings.\n3. The open-set SelKD formulation, while promising, lacks guidance on how it could be adapted to real-world scenarios with dynamically changing categories.\n4. Lack of theoretical guarantee.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vI2CpdWbMC", "reviewer": "Reviewer_uP4C", "summary": "This paper introduces Selective Knowledge Distillation (SelKD), a novel approach that allows student models to learn only selected subsets of knowledge from teacher models. The authors reformulate knowledge distillation through the lens of Inverse Optimal Transport (IOT) and propose both closed-set and open-set variants of SelKD.", "questions": "1. If each subtask requires a separate teacher model, it naturally raises concerns about parameter count and memory costs. Tables presenting parameter counts should clearly explain any dependencies on task splits to avoid misleading impressions.\n\n2. Could you clarify if KL divergence or cross-entropy is minimized in the outer optimization of Eq. 8? The description seems inconsistent.\n\n3. In Figure 2, the green line is shown but not illustrated. Could you clarify this?\n\n4. Switching Tables 2 and 3 order might improve flow.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "**Originality**: The paper offers a fresh perspective on selective knowledge distillation by using optimal transport (OT) to map logit relationships between image and text features. This new approach allows the student model to learn only the relevant subset of knowledge, making it distinct from traditional, full-transfer methods. This new perspective opens up important new research directions.\n\n**Clarity**: The paper is well-structured, with a clear motivation and related work sections that are easy to follow. The optimal transport concepts are complex, but most parts are presented clearly.\n\n**Significance**: Great motivation: enabling selective transfer for specific tasks. The open-set extension further adds value by allowing the model to handle unknown categories, making it versatile for real-world applications.", "weaknesses": "**Clarity and Notation**: The notation, especially in the optimal transport section, could be clearer. Key terms and symbols, like \n$z$ in Eq. 2 and $\\tilde{P}$ in Eq. 7, should be introduced. Additionally, Algorithm 1 may be challenging for readers unfamiliar with classic OT methods. Providing some background or simplifying this section would improve accessibility.\n\n**Computational Complexity**: The paper doesn’t explain the computational cost of the OT-based approach, which may be high given the framework. Information on computation complexity would be helpful to fully understand the method’s practicality and limits.\n\n**Performance**: The impact of random seed on results is not shown. Further testing here could demonstrate robustness and validate improvements.\n\n**Feature-level Transport Potential**: The paper briefly mentions feature-level transport but doesn’t explore it. Expanding this aspect could reveal further benefits and justify using OT for representation learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmzVZAuAeG", "reviewer": "Reviewer_bHk7", "summary": "This paper proposes a novel method called Selective Knowledge Distillation (SelKD), which enhances student model performance by transferring only partial knowledge. The integration of Optimal Transport (OT) methods is applied to both closed-set and open-set classification tasks, demonstrating that SelKD outperforms traditional knowledge distillation methods. Overall, the paper offers new insights and practical solutions for the knowledge distillation field.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The introduction of SelKD allows for effective partial knowledge transfer, reducing redundant learning seen in traditional KD.\n2. The method shows high potential for real-world applications, particularly in resource-constrained environments.", "weaknesses": "1. The paper lacks concrete details regarding the derivation and experimental implementation. It would be beneficial to provide specific derivations and related code for the experiments to facilitate reproducibility.\n2. The analysis of experiments is somewhat superficial, particularly concerning parameter impacts. Given that optimal transport (OT) can be heavily influenced by parameters, providing visual analyses of how these parameters affect the results would enhance understanding.\n3. The paper does not include any experiments related to the efficiency of the proposed method. It would be valuable to assess and report the computational efficiency of SelKD in comparison to traditional methods.\n4. The experiments primarily focus on ResNet architectures, which restricts the generalizability of the findings. Incorporating more recent distillation methods from 2023 or 2024, as well as testing on different model architectures and datasets, would better demonstrate the effectiveness and versatility of the proposed method. A broader range of experiments would strengthen the paper’s claims regarding SelKD's applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)