Towards Customized Knowledge Distillation for Efficient Dense Image Predictions
Authors: Dong Zhang
Pingcheng Dong
Long Chen
Kwang-Ting Cheng
Venue: Accepted by TMLR ()

Abstract: It has been revealed that efficient dense image prediction (EDIP) models designed for AI chips, trained using the knowledge distillation (KD) framework, encounter two key challenges, including maintaining boundary region completeness and ensuring target region connectivity, despite their favorable real-time capacity to recognize the main object regions. In this work, we propose a customized boundary and context knowledge distillation (BCKD) method for EDIPs, which facilitates the targeted KD from large accurate teacher models to compact small student models. Specifically, the boundary distillation focuses on extracting explicit object-level boundaries from the hierarchical feature maps to enhance the student model's mask quality in boundary regions. Meanwhile, the context distillation leverages self-relations as a bridge to transfer implicit pixel-level contexts from the teacher model to the student model, ensuring strong connectivity in target regions. Our method is specifically designed for the EDIP tasks and is characterized by its simplicity and efficiency. Theoretical analysis and extensive experimental results across semantic segmentation, object detection, and instance segmentation on five representative datasets demonstrate the effectiveness of BCKD, resulting in well-defined object boundaries and smooth connecting regions.

Published in Transactions on Machine Learning Research (01/2026)
Towards Customized Knowledge Distillation for Efficient Dense Image Predictions
Dong Zhang
donga@rust.hg
Deportment of Electronic and Computer Engineering The Hong Kong University of Science and Technology
Pingcheng Dong
pingcheeng.dong@eornneec.cc
Department Electronic and Computer Engineering The Hong Kong University of Science and Technology
Long Chen
longchen&cce..usll ust.hk
Department of Computer Science and Engineeringk The Hong Kong University of Science and Technology
Kwang-Te Ting Cheng
timcheng@ust.t hk
Deportment of Electronic and Computer Engineering The Hong Kong Universily of Science and Technology
Reviewed on OpenReview: https: // openreview. net/ form?? id= qverIe3tE
Abstract
It has been revealed that efficient dense image prediction (EDIP) models designed for Al chips, trained using the knowledge distillation (KD) framework, encounter two key challenges, including maintaining boundary region completeness and ensuring target region connectivity, despite their favorable real-time capacity to recognize the main object regions. In this work, we propose a customized boundary and context knowledge distillation (BCKD) method for EDIPs, which facilitates the targeted KD from large accurate teacher models to compact small student models. Specifically, the boundary distillation focuses on extracting explicit object-level boundaries from the hierarchical feature maps to enhance the student model's mask quality in boundary regions. Mcanwhile, the context distillation leverages self-relations as a bridge to transfer implicit pixel-level contexts from the teacher model to the student model, ensuring strong connectivity in target regions. Our method is specifically designed for the EDIP tasks and is characterized by its simplicity and efficiency. Extensive experimental results across semantic segmentation, object detection, and instance segmentation on five representative datasets demonstrate the effectiveness of BCKD, resulting in well-defined object boundaries and smooth connecting regions.
1 Introduction
The dense image prediction (DIP) tasks, e.g., semantic sogmentation (Long et al, 2015). object detection Gir shick 2015), and instance segmentation (Wang al. 2021c), are challenging problems within both domains of computer vision and multimedia computing (Zhang et al. 2020 Ahn al. 2019 The objective of these tasks is to assign semantic label to each object and/or pixel of the given image [Zhng et al. 2020). In recent years, achievements in gencral-purpose GPU technology have resulted in notable cnhancements in both size and accuracy of sophisticated DIP models (Cao el. al.) 2022a Strudel et al. 2021 e.g., Mask2Former (Cheng et al. ).22), SegNeXt (Guo et al. 2022b), and SAM (Kirillov et al. 2023). However, deploying these large and accurate DIP models on resource-conntraaied edge computing devices, e.g., artificial intelligence chips (Dong et al. 2025), presents significant challenges due the substantial computational costs and high memory consumptions associated with these models (Wang al.] 2021c)

---

Published in Transactions on Machine Learning Research (01/2020)
(b) Large model
(c) Small model
(d) Our small model
(n) Input image
Effectiveness 77
Effectiveness K
Effectiveness Y
(e) Ground- -ruuh
Efficiency
Efficiency
Efficiency
Figure 1: Two representative cases that small models are prone to predict errors. Visualization comparisons between large accurate models (b) and small efficient models (c) show that the latter tend to make crrors in maintaining boundary region completeness and preserving target region connectivity. With the help of BCKD, our small models in (d) can address the two types of errors, leading crisp region boundaries and smooth connecting regions. Samples are from the ADE20K dataset (Zhou al. 2017).
Compressing large DIP models into compact elficient dense image prediction (EDIP) models offers an intuitive and cost-effective solution to address the severe resource limitations associated with mapping vision models onto edge computing devices (Dong al. 2021 Zhang al., 2021). In particular, the cross-architecture fashion enables compressed models to seamlessly adapt customized edge chips, climinating the need for hardware modifications while maintaining computational efficiency. This manner significantly reduces deployment complexity and enhances the flexibility of model inference across heterogeneous edge computing platforms (Yin et al. (023). To achieve this goal and develop accuracy- peeeraccyprre EDIP models, knowledge distillation (KD) (Hinlon et al. 2015 Wang et al. 2023), a prevalent model compression technology, has been pragmatically employed by using small efficient model (ic., the student model) by imitating the behavior of a large accurate model (i.e. the teacher model) in training (Hinton et al. 2015 Zhao et al., 2022). During inference, only the student model is utilized, allowing for a highly-efficient recognition pattern while simultaneously reducing the model size (Dong et al., 2021; Zhang et al., 2021 Cui et al. 2023 Despite significant advancements by current KD methods across multiple dimensions, including sophisticated distillation strategies (Cui et. al., 2023) and complex distillation content (Wang et al. 2020). the inherent complexity of DIP continues to pose two critical challenges for existing approaches, particularly for the efficient compact models.
Primary KD methods mainly emphasize the imitation of general knowledge (e.g., features, regions, and logits) while overlooking the nuanced understanding of features along objective semantic boundaries and connecting internal regions essential for EDIPs (Zhang et al. 2021 Wang ct al. 2022a). Particularly, since the small student model often predicts the main object regions fairly well but fails in boundary and connecting regions (Fu et al. 2019 Yuan et al.] 2020 Cao et al. 2022a). the conventional utilization of task-agnostic general KD may not be effective enough and can be considered purposeless and redundant (Gou et al. 2021 Xu et al.) 2020; Zhao et al. 2022). remaining a performance gap between the obtained results and the expected ones (Wang al. (2244) For instance, we recommend the representative semantic segmentation task as an example. As shown in Figure D the small student PSPNet-18 model in (c) produces inferior results comparod to the large teacher PSPNet-101 model results in (b). The student model wrongly segments the boundary regions of "curtain" and "door" as the background category other foreground objects, and produces fragmented "bed valance" " and "chair", breaking the regional relation connectivity. Generally, the common errors observed in the outputs of the small student model can be summarized as maintaining boundary region completeness and ensuring target region connectivity.
To mitigate these errors and narrow the performance gap, we propose customized and targeted KD stratogy termed as Boundary and Context Knowledge Distillation (BCKD). By "customized", we mean that our method's 's inherent ability to synergistically address the common errors present in existing EDIP models,
2

---

Published in Transactions on Machine Learning Research (01/2026)
while is also generally compatible with other methods (ref. Sec. 5.3 BCKD mainly consists of two key components: the boundary distillation and the context distillation, aimed at rectifying the typical common errors encountered by EDIP models in maintaining boundary region completeness and ensuring target region connectivity, respectively. Specifically, boundary distillation involves generating explicit object-lovel boundaries from the hierarchical backbone features, enhancing the completeness of the student model's is masks boundary regions (ref. Sec. 4.2). At the same time, context distillation transfers implicit pixcl-level contexts from the teacher model to the student model through self-relations, ensuring robust connectivity in the student's masks (ref. Sec. 4.3). BCKD is tailored specifically for EDIP tasks and offers more targeted distillation pattern and a more tailored distillation manner compared to conventional task-agnostic KD methods. From a rigorous theoretical perspective, we establish and prove the effectiveness of our BCKD method (ref. Sec. Aj. To validate the accuracy, we conducted extensive experiments in three represenlative dense image prediction tasks, including semantic segmentation, object detection, and instance segmentation, utilizing five challenging datasots such as Pascal VOC 2012 (Evcringham et al. 2010), Cityscapes (Cordts et al. 2016), ADE20K (Zhou et. al., 2017), COCO-StufT 10K (Caesar et al. 2018), and MS-COCO 2017 QLin el al,, 2014 Qualitatively, BCKD produces sharp region boundaries and smooth connecting regions, addressing challenges that hindered existing EDIP models. Quantitatively, BCKD consistently improves the accuracy of bascline models in various metrics, achieving competitive performance.
The main contributions of this work are: (1) We revealed two prevalent issues in existing EDIP models: maintaining boundary region completeness and ensuring target region connectivity. (2) We proposed a customized and targeted boundary and context knowledge distillation method, which not only demonstrates
inherent coherence, but. is also generally compatible with other methods. (3) Experimental evaluations across varions tasks, basclines and datascts illustrate new stato-of-tho-art accuracy of our method in comparison with existing methods.
2 Related Work
2.1 Dense Image Prediction Tasks
Dense image prediction (DIP) is a fundamental research problem within the fields of computer vision and multimedia computing, with the objective of assigning each object and/or pixel in an input. image to a predefined category label, thereby enabling comprehensive semantic image recognition (Long et al. 2015 Zhou
cct al.
2024 Zhang ct al. 2020 Lin ct al. 2023b). Current mainstream DIP models can be roughly classified
into the following three categories based on their backbone components: 1) methods based on CNNs (Long
ct al.
2015 Yu et al.] 2018 Noh et al., 2015 Huang et al. 2019)) 2) methods based on (Strudel et al.
Wang et al. 2022b; Zheng et al. 2021 and 3) methods that combine CNNs and VIT (Li et al. 2022a Peng et al., 2021) The key difference between these types of architectures is the approach used for feature extraction and how the extracted features are utilized in enhancing the capacity of CNNs models to capture
2021
contextual features (Cao et al., 2022a; Zhang & Cheng 2025), incrcasing the capacity of ViT models to capture local features (Wu el al.) 2021 Zhang et al.] 2022b Peng et al., 2001], and leveraging low-level features to improve representation capacity (Zhang et al. 2023
Xie et al.] 2021) for achieving favorable results.
Concretely, due to differences in fcature extraction manners between CNNs and VIT, these two categories
exhibit slight performance differences (Wang et al., 2022b; Peng et al.] 2021 Zheng al., 2021 Li al., 2022a). For example, CNNs methods are better at predicting local object regions, while ViT methods, due to their stronger contextual information, can produce more complete object masks. Fortunately, the mixture of CNNs and ViT (e.g., CMT (Guo et al. 2022a), CvT (Wu et al.) 2021) ConFormer (Peng et al. 2021 CAF-GreaT (Zhang et al., 2023), 3, and visual Mamba (Gu & Dao 2023) uses the representation strengths of both patterus, resulting in highly satisfactory recognition performance (Han et al.] 2022 Mao et al.) Wang et al.) 2021a). In addition to these fundamental categories, there are advanced approaches that
2022
also utilize task-specific training tricks (e.g., graph reasoning (Zhang et al., 2022b), linear attention (Kong
et al.
2022 mult-scale representation (Fan et al. 2021 to improve the accuracy. However, while current.
We consider the visual state space modcl-based methods specialized Transformer architocture (Gu & Dao 2023
Xu
et 2024), owing to its structural similarities with the VIT model. Besides, do not address the content. related these models. Therefore, we will no longer have separate discussions on this aspect.
3

---

Published in Transactions on Machine Learning Research (01/2026)
methods have achieved promising accuracy, mapping these models on resource constrained edge computing devices remains challenging because these devices typically have limited computation resources and memory consumptions (Dong et al. 2023 2025). In this work, we do not intend to modify the network architecture. We first investigate the result disparities between small and large models and then propose a novel KD strategy lailored to the EDIP models. We aim at improving the recognition accuracy of small models without. requiring any extra training data or increasing the inference costs.
2.2 Knowledge Distillation in DIPs
Knowledge Distillation (KD) is a well-established model compression technology that aims to transfer valuable knowledge from large accuracy teacher model to a small efficient student model, with the objective of enhancing the student's accuracy during inference (Chen et al. 2023 Gou et al. 2023 Xiang et al. 2025 Wang et al. 2020 Xu et al.] 2025 Li et al. 2025). It is worth mentioning that the effectiveness of KD in cross-architecturs
scenarios has enabled significant flexibility in artificial intelligence chip design, as it eliminates the need to modify the underlying operators while maintaining model performance, which provides a practical solution for hardware adaptation without compromising computational efficiency (Dong et al., 2025). The key factors for the success of KD in DIPs arc: 1) the types of knowlodge being distilled, e.g., general knowledge: features and logils, and lask-specific knowledge: class edge for semantic segmentation and object localization for object detection, 2) the distillation strategies employed, e.g., offline distillation [Tseng et al., 2022), online distillation (Guo et al. 2020), and self-distillation (Zhang et al. 2019). and 3) the architecture of the teacher-student pair, e.g., multi-teacher KD (Yuan et al. 2021), attention based KD (Passban et al., 2021j, and graph-based KT) (Lee & Song, 2019). While the effectiveness of existing KD methods has been validated in general vision tasks, current methods rely on coarse task-agnostic knowledge and do not consider the task-specific feature requirements (Sun et al.] 2020 Gou et al.) 2021 Mao et al.) 2022: Chen et al.] 2017a Wang et al. (001)). Especially for EDIPss, models are highly sensitive to feature representations (Long el al.
2015 Zhang ct al. 2022c Zheng et al., 2021 Zhang et al. 2018). Therefore, general KD may not be effective
enough and can be considered purposeless and redundant (Gou et al. 2021: Xu et al. 2020 Zhao et al. 2022
Zhang et al.
2022b), remaining a performance gap between the obtained results and the expected ones Yang
ct al. 2022a Cui et al. 2023 Wang et al. 2024h. Recent studies have shown that task-specific patterns of
KD can help further improve the performance of student models (Zheng et al. 2022.. For example, in object
detection, object localization KD leads to more accurate predictions than the general knowledge (Sun el al. 2020 Zhixing al.) (021)) In this work, we also adopt the idea of task-specific KD. Our contribution lies in proposing customized KD scheme, namely boundary distillation and context distillation, which target the common errors of EDIP models, namely their lendency lo make errors in maintaining boundary region completeness and ensuring target region connectivity. It is also worth noting that while some methods, e.g., CTO (Lin et al. 2023c). SlimSeg (Xue et al. 2022), and BPKD (Liu et al. 2024), have integrated boundary information in EDIPs, they necessitate the pre-extraction and incorporation of ground-truth boundaries (Xue) et al. 2022 Liu et al. 2024 Lin et al. (202c)). ). In contrast, our method obviates the demand for pre-extracting boundaries, thereby making it more practical for real-world applications and enabling savings in time and labor.
3 Preliminaries
In the training phase, KD intends to facilitate expectant knowledge transfer from a large teacher model T to small compact student model S, with the primary goal of enhancing the accuracy of S (Hinton al., 2015: Zhang et al. 2021 2019 Gou et al. 2021 Ji et al. 2021). In inference, only $ is used, so there are computational overheads. Typically, features and logits serve as a medium for knowledge transfer. Besides, the temperature scaling strategy is often utilized to smooth the features and logits, which helps to lower prediction confidence and alleviate the issue of excessive self-assurance in T (Phuong & Lampert 2019 Formally, KD) can be expressed by minimizing the cross-entropy loss as follows:
CKD= = -- Σ (T,)) (ot).)
(1)
KEM
4

---

Published in Transactions on Machine Learning Research (01/2026)
Fixed Teacher Model (Pre-Traine Trained)
Feature Maps
TK:
TF2
Logits
Stem
cany,
1x Lunn, 064
8:64
comy,
1
eurgooe Comcutt T.
TV.
TF:
11L
TRL
Eq. (2): Boundary Extraction
Eq. (4): Lontest Formulation
Shrret Imputsi
Boundary Distillation
(b) C Context Distillation
Eq. 3: CF(T" (S*)
Eq 5: Lcp CE (*))
(2): Roundary Extraction
Eq. (4) Contest Formulation
5
Cunrg (amcottS $.55.
S
Stem
6:64
12
Trainable Student Model (To Be Trained)
SR
56
SR.
SL
Feature Maps
Logits
SF;
SE,
SF.
SE,
SL
SF
Student
Student logits
Down-samplligg operation
Concat()
Feature conc concatenation along the channel dimension
TF
Teacher foature
Teacher logits
Up sampligg operation
learning
Trainable learning
Ground truth
Figure 2: The overall architecture of our boundary and context distillation strategy for the efficient. dense image prediction tasks, where the right side illustrates the implementation details of the whole network. Specifically, the boundary distillation involves generating explicit object-level boundaries from the hicrarchical backbone features, enhancing the completeness of the student model's masks in boundary regions (ref. Sec. 4.2) Meanwhile, the context distillation transfers implicit pixel-level contexts from the teacher model to the student model through self-relations, ensuring robust connectivity in the student's 's masks (ref. Sec. 4.3 Compared to existing methods, our method demonstrates stronger specificity for EDIP and inherently possesses the ability to synergistically address the common errors found small models.
where T; and S; (both have been adjusted to the same dimension via x convolutions) are the i-th feature/logit item extracted from and S, respectively. a(-) is the softmax normalization operation along the channel dimension, and T E R2 denotes the temperature scaling coefficient. M denotes the learning objective of T, and S., which typically refers to the spatial dimensions. Following (Zhang et al. 2019 Phuong & Lampert 2019), simplify temperature scaling effectively, we use T,/S, divided by achieve the similar effect. In addition to cross-entropy loss, other loss functions are also commonly used for KD, including KL divergence loss and MSE loss (Liu et al. 2019 2022 Zheng et al. 2023). While existing KD methods have
demonstrated promising results across various vision tasks (Zhang et al. 2021 Cui et al. 2023 Xu et al.)
2020 Liu et al. 2019 Wang et al. 2020 Gou et al. 2021), they have not adequately addressed the specific feature understanding required for object boundarics and connecting regions in EDIP tasks. This oversight has led to suboptimal performance
these contexts. In following, we will introduce a complementary and targeted KD scheme informed by the common failure cases observed in small models, as shown in Figure !
with the aim of enhancing their inference accuracy.
4 Customized Knowledge Distillation
4.1
Overview
Figure 2 iilusttrtes an overview of the net work architecture for our proposed BCKD. The whole worw mainly consists an accurate teacher network T, which is a large network that has been trained, and a small efficient network $ that is waiting to be trained. The input for T and $ an arbitrary RGB image X, and the output of S semantic mask or bounding box Y that predicts each pixel or/and object with a specific class label. The hierarchical features extracted from the backbone network concatenated along the
5

---

Published in Transactions on Machine Learning Research (01/2026)
channel dimension to facilitate the extraction of EpIP-specific boundary and contextual information from X. The concatenated features with 256 channel dimension are defined as T = (onnaxx(cconaatt and S = "onvsx3(concat(Si: :; for T and 8, respectively. It should be noted that both T; and S, that are concatenated have been uniformly resized into 1/8 of X's spatial size via 1 x convolution and up-//down-ammplin operations. training, we propose largeled boundary distillation and context distillation strategies that are tailored for the EDIP tasks. Specifically, the boundary distillation synthesizes explicit object-level boundaries T and 8 from T and S respectively, thereby the completeness of S's results in the boundary regions can be enhanced. At the same time, the context distillation transfers implicit pixel-level relations T and R by using self-relations, ensuring that E's results have strong target region connectivity.
..: Ts))
4.2 Boundary Distillation
The semantic object boundary is defined a set of arbitrary pixel pairs from the given image, where the boundary between pairwise pixels has a value of 1 if they belong to different classes, while if the pairwise pixels belong to the same class, the boundary between them has value of 0 {Ahn & Kwak 2018). Moreover, this attribute also exists in the hierarchical Peatures/logits extracted by the backbone feature maps (Chen et al. 2020p. In our work, we use the semantic affinity similarity between arbitrary pairwise pixels from T or S to oblain the explicil object-level boundaries (Ahn el al. 2019 Ru el al. 2022). Concretely, for a pair of image pixels T. and Ty, T can be formulated as:
Ti, = - - maxp, p, 4 E B (Convix p) Convix gT)))),
(2)
where To and Té are two arbitrary pixel items from TV, and II denotes a set of pixel items on the line between T. and TG. Convix1 denotes 1 convolution layer that is used to compress the channel dimension, where the input channel size is 256 and output channel size is B(-) denotes the operation that determines the object-level image boundary values, which outputs either or 0. T can be obtained across the entire spatial domain, and s can be obtained analogously. Based T and s", boundary distillation loss is formulated as:
CBD = - Σ C(T (D()))11t)
(3)
REM
where p denotes the spalial-wist softmax normalization. By Eq. 3 T'ss accurate ion of the object boundary region can be transferred into S, thereby addressing its issue of maintaining boundary region completeness.
The proposed boundary distillation strategy can effectively address the limitations of EDIP models in achieving completeness in boundary region predictions. It is important to highlight that while several advanced methods, such as BGLSSeg (Zhou et al. 2024), SlimSeg (Xue et al. 2022), and BPKD (Liu et al. 2024), leverage explicit edge information for semantic segmentation, these methods necessitate the pre- ereeextraction and integration of ground-truth masks (please refer to Table 3 for further details). More importantly, the extracted edge information often lacks the semantic context of the objects and may introduce noise (Yang et al. 2022b Ji et al.) p022p. In contrast, our method eliminates the need for pre-extracting image edges. By utilizing hierarchical feature maps, our method enhances the delineation of object boundaries with more comprehensive semantic information while mitigating the adverse effects of noise present in shallow features. As illustrated in Figure 3 we show a visualization comparison of the extracted image boundaries between our method and the ground truth edge method used in the ttete-of-the-art BPKD (Liu et al., 2024 model. We can observe that the semantic boundaries extracted by our method can better cover the actual boundaries of the semantic objects without introducing background noise information. This innovation not only can enhances the practicality of our method for real-world applications but also streamlines the overall process, leading to reductions in both time and labor requirements.
4.3 Context Distillation
Elaborate object relations, as validated in CCarnn et al. 2021 Wang et al.) 2021b Li al. 2022bb. are beneficial for learning implicit contextual information across spatial dimensions (Lin et al.) 2023a Ji et al. 2022). In this paper, we also adopt this scheme to address the challenge of inadequate preservation of target.
6

---

Published in Transactions on Machine Learning Research (01/2026)
n
(b) Ground truth edge
(c) Semantic boundary Pre-extracting free M
(a) Input image
Pre-extracting free x
Figure 3: The visualization comparisons of the extracted image boundaries between our method in (c) and the ground truth edge method in (b) used in the state-of-the-a BPKD Liu et al. (0224). Images are from Pascal VOC 2012 Everingham et al. (2110)
region connectivity in EDIPs. We consider this scheme as the medium in the KD process. Our contribution lies treating pixel-Ievel relations as a bridge for context transfer. Compared existing methods (Ti et al. 2022b Ji et al., 2022), our approach utilizes pixel-level relations solely during the training process, thereby avoiding the increase in model complexity and parameters in inference that is typically associated with current methods (Liu et al. 2019 Lin et al. 2023a). Besides, compared to object-level relations, the employed pixel-level relations can capture global contextual information more comprehensively, making them more suitable for DIP tasks. Specifically, for the concatenated features Te of T, its self-relation T is formulated as:
R ((TT)T
(O(T9".O(T)) .OTT)) /T E IR hhwxhh
(4)
Vd
where (.) denotes the featurc-aligned operation as in (Li ct al. 2022b Xie et al. 2023 which aims to align the feature distribution of the S with that of the as closely as possible. ((-) is implemented as a projection head consisting of two 1x1 convolutional layers (256 channels) with ReLU activation, followed by layer normalization and final projection to the target dimension. T' denotes the matrix transpose operation. d the channel size of T which 256. h and w denotes the height and width of T respectively. S of S also be obtained analogously. Therefore, the context distillation can be expressed
LCD= -
Σ
(I (o(s))
(5)
kEhar x hww
where k (1,2, ..., how x hw) is the index item. T and S denotes the k-th item in T and S", respectively.
The context distillation presents an efficient solution that does not incur additional inference overhead. As illustrated in Figure ! our method in (b), which utilizes concatenated features in a whole-to-whoole manner, circumvents the potential noise associated with existing context learning methods in (a) for semantic segmentation (Liu et al. 2019 2024) that rely on layer-tolayer distillation. Our method is specifically tailored to address the potential challenge of incomplete preservation of target region connectivity as shown in Figure rather than solely focusing the euhancement of feature representations in generic context.
4.4 Overall Loss Function
With the boundary distillation loss LBD and the context distillation loss LCD, the total loss is expressed as:
L Lss aLLD + BLCD.
(6)
7

---

Published in Transactions on Machine Learning Research (01/2026)
Fixed Teacher Model (Pre- (Pre-Trained)
Fixed Teacher Model (Pre-Trained)
(a) Existing context learning methods
(b) Our proposed context distillation
TF,
TF2
TF;
TF,
TL.
TF.
TF.
TF3
TF₃
Conv
Con
Conv
Conv
Couv
KD
KD
KD
KD
KD
Contest Distillation
Callenngro(oonottS. S,:S,,
Conv
Comr
Conv
Cony
Conv
SF.
SF
SF₃
SF,
SL.
SF,
SF2
SF3
SF₃
SL.
Trainable Student Model (To Be Trained)
Trainable Student Model (To Be Trained)
Figure 4: Architecture comparisons between our proposed context distillation (b) and existing context. learning methods (a). Our method leverages concatenated features in whole-to-whole manner, which avoids the potential noise introduced by layer-to-layer distillation modes.
where a and are two weights used to balance different losses. Empirically, these weights have a significant impact on model performance, and unreasonable weight settings may even lead model collapse training. To this end, inspired by previous work (Sun al.) 2020: Yang et al. (023), to enhance the dependence of S ground truth labels, we incorporate weight-decay strategy that promotes greater focus on Lss as the training epoch increases. To this end, we initialize time function follows:
(t) 1 (t 1)/tmax:
where (1, 2, ..., (max) denotes the current training epoch and max is the maximum training epoch. training, control the dependence of L on LBD and LCD by using r(t). Therefore, the total loss in Eq. 6 can be formulated as:
C = Lss +r())aBBD + r(t)BCcDD
(8)
5 Experiments
5.1
Datasets and Evaluation Metrics
5.1.1
Datasets
To demonstrate the performance of our method, conduct experiments on five representative yet challenging
datasots: Pascal VOC 2012 Everingham et al. (2010), Cityscapes Cordts et al. (2016). ADE20K Zhou et al.
(2017), and COCO-Stuff 10K Caesar el al. (2018) for semantic segmentation (SSeg), as well as MS-COCO
2017 Lin et al. (2014) for instance segmentation (1Seg) and object detection (ODet).
The Pascal VOC 2012 datasct comprises 20 object classes along with one background class. Follow- ing (Zhang al. 2021 Wang et al. 2020), utilized the augmented data, resulting in total of 10, 582 images for training, 1, 1,49 images for val, and 1, 456 images for testing.
The Cityscapes comprises total of 5, 000 finely annotated images, which are partitioned into subsets of 2, 975, 500, and 1, 525 images designated for training, val, and testing, respectively. In alignment with existing mothods (Zheng et al. 2023 Wang et al. 2020 Liu et al., 2019), we exclusively employod the finely labeled data during the training phase to ensure fair comparison of results.
The ADE20K dataset has 150 object classes and is organized into three subsets: 20, 000 images for the truining set., 2, 000 images for the val set, and 3, 000 images for the lesting set.
The COCO-Stuff 10K dataset is an extension of the MS-COCO datasct Lin et al. (2014), enriched with pixcl-wise class labels. It consists of 9, 000 samples designated for training and 1, 000 samples allocated for val.
8

---

Published in Transactions on Machine Learning Research (01/2026)
The MS-COCO 2017 dataset comprises 80 object classes and inchudes total of 118, 000 images for training, and 5, 000 images for val.
For data augmentation, random horizontal flip, brightness jittering and random scaling within the range of [0.5, 2] are used in training as (Wang et al. 2020 Zhang et al., 2021 Liu et al.] 2019). Experiments are implemented on the MMRazor framework" under the PyTorch platform (Paszke al. 2019 using
NVIDIA GeForce RTX 3090 GPUs. All the inference results obtained at single scale.
5.1.2 Evaluation metrics
Beyond employing standard metrics, we have also developed two extra specialized evaluation metrics specifically optimized for knowledge distillation on dense image prediction tasks, detailed below:
Common metrics. For SSeg, we utilize the mean intersection over union (mIoU) as the primary evaluation metric. For ISeg and ODet, average precision (AP) serves as the principal accuracy-specifie metric. To assess model efficiency, also consider the number of parameters (Params.), Peak GPU memory (Peak GPU Mem.), and the floating-poii point operations (FLOPs)
Manifold stability (MFS). To assess the effectiveness the learned feature manifolds, compute the Lipschitz constant ratio between teacher and student models. Specifically, let (1, R4 denote the I-hh layer feature mappings for teacher and student models respectively. The layer-wise Lipschitz constant. can be estimated via:
1 + 8) - fx))le
L
sup 2EX S
E (T,S)
|0112
where &' is the input space and c = 0.1 controls the perturbation scale. The MFS pr is then computed as:
L's LT
-14 >)) -) I(T4 ST
(10)
with threshold r avoiding division by negligible values (r 0.01). Values close 1 indicate well-pu well-preserved manifold structure during distillation, while significant deviations suggest potential degradation of geometric properties.
Local Hausdorff distance (LHD). For boundary-sensitive tasks like SSeg and ISeg, we introduce LHD metric to evaluate boundary alignment quality. Specifically, for boundary point sets B, {p.) and By (0j / the LHD at point pt is defined as:
(p. Bg) = min max d(p:, 4j), median (ad(mn(((pp,,)) 9j (alddooo)))))) (p.))
(11)
where the neighborhood Ny(p:) and final aggregation are defined as:
N,(p)) t4 E B, ||p: 43 S Sr),
|8p1
(12)
LHD(B,,
LHD, BB) -IHHL, (pi B,) S 20),
|8,
where is set to 5 in implementation.
5.2
Implementation Details
5.2.1
Baselines
For a fair result comparison and considering the realistic resource conditions of edge computing devices (Dong et al. 2023), for SSeg, we select PSPNet-101 (Zhao et al. 2017). DeepLabv3 Plus-101 (Chen et al,
Phttps:/ ceatluubbeom/ooppmmmbbb
While Params/FI.00 FLOPPs widely used as proxy metrics for model complexity, they do fully capture the actual performance of models edge devices. The edge-chip performance remains the ultimate benchmark EDIP models.
9

---

Published in Transactions on Machine Learning Research (01/2026)
2018 and Mask2Former (Cheng al. 2022 for the teacher models. The student models are compact
PSPNet and DeepLabV3+ with ResNet-38, ResNet-18(0.5) and ResNet-18(1.0) (He et al. 2016). Besides,
to demonstrate the effectiveness of our method on heterogeneous network architectures, following (Wang 2022a), we also employ MobilcNetV2 (Liu 2018). EfficientNef
ict al. 2020: Liu et al. 2019 Yang et al.,
B1 (Tan & Le 2019). and SegFormers 80 (Xie et al. 2021 as the student models. For ISeg and ODet,
following (Wang et al. 2024 Zhang Ma 2023), we select the representative GFL [Li et al., 2020), Cascade
Mask R-CNN {Cai & Vasconcelos] 2018): ) and RetinaNet (Ross & Dollár 2017) with ResNet-101/55 101/50 and
ResNet-50/18 as the teacher model and the student model, respectively. While adopting cutting-edge large foundation models represents the prevailing research trend, we opt for pragmatic small-model baseline given the currently insurmountable challenges in deploying such large-scale models on edge devices. This application--orenteed approach prioritizes practical deployability over model scale, while still providing a meaningful benchmark for edge computing scenarios. During the training and inference phases, aside from our proposed method and specific doclarations, all other settings adhere to the configurations outlined in the baseline model.
5.2.2 Training details
Following the standard practices as (Hinton et al. 2015 Liu et al. 2019 Wang et al. 2020), all teacher models are pre-trained on ImageNet-1k by default (Deng al. 2009), and then fine-tuned on the corresponding dataset before their parameters are fixed in KD During training, only the student's parameters are updated. 2021 Phuong & Lampert 2019 is initialized to 1 and is multiplied by scaling factor of 1.05 whenever the range of values (across all feature items in given minibatch) exceeds 0.5. Following (Wang el al. 2020 Sun et al. 2020 Yang el al. 2023 a and 3 is set to 10 and 50, respectively. We fully understand
As in Zhang et al.
that fine-tuning these base hyperparameters could potentially enhance performance. However, we contend that such adjustments may be redundant and unwarranted.
For SSeg models, to accommodate the local hardware limitations, the training images are cropped into a fixed size of 512 x 512 pixels as in (Wang al.) 2024 Zhang & Ma 2023). The SGD is used as the optimizer with the "poly" learning rate strategy. The initial learning rate is set to 0.01, with power of 0.9. To ensure fairness in the experimental comparisons, the batch size is set to 8 and Imax is set to 40, 000.
For ISeg and ODet, the model is trained following the default 1x training schedule, i. L..,, 12 epochs. The batch size is set to 8, and AdamW is used as the optimizer with the initial learning rate of 1 x 10 and the weight decay of 0.05. The layer-wise learning rate decay is used and set to 0.9, and the drop path rate is set to 0.4. The given images are resized to the shorter side of 800 pixcls, with the longer side not excooding 1, 133 pixels. In inference, the shorter side of images is consistently set to 800 pixels by default.
5.3 Ablation Analysis
In our ablation analysis, we aim to explore answers of the following crucial questions: 1) the impact of each component within BCKD in Section 5.3.1 2) the effectiveness of BCKD across different network architectures in Section 5.3.2 and 3) the visualized performance and comparisons with other mothods Scction 5.3.3 Wc
select the SSeg task as the experimental objective.
5.3.1 Effectiveness of each component
To explore answer of the first question, we chose Pascal VOC 2012 Everingham et al. (2000) as the experimental dataset. PSPNet-101 (Zhao et al. 2017) serves as the teacher model, while PSPNeet11e 1SPPNet-1111.)) is used as the student model. Table ?? shows the interence results by adding each component of BCKD into the student model, where we report the experimental results the val set. We can observe that incorporating these components consistently improves the model accuracy, indicating the effectiveness of these components. In particular, adding LAD resulted in a mloUt gain of 0,91,, which may be altributed to the fact. that the object boundary regions are relatively small in proportion to the entire image (Zhang et al., 2021 Liu et al, 2024j. With only the implementation of LBD and LCD, our model can achieve the competitive 73.98% mIoU with 3.20% mIoUt improvements, which verifies the importance of boundary and context information in EDIP. Furthermore, this result demonstrates that ourLnD and LeD do not conflict in practical deployment;
10

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Ablation results on Pascal VOC 2012 val set (+STD over 3 random seods). Peak GPU memory
(Peak GPU Mem.) measured on NVIDIA RTX 3090.
Configuration
mloU (%)
AmloU
Params.
Inf. FLOPs
FLOPs
Tr. Peak GPU Mem.
Teacher (PSPNet-101)
77.82+0.15
70.43M
411.6G
411.6G
24.5GB
Student (PSPNet-18)
70.78=0.21
15.24M
106.2G
106.2G
8.2GB
CBD
71.69:0.188
+0.91
15.24M
106.2G
116.8G
10.7GB
+ Len
72.55:0.11
+1.77
15.24M
106.2G
12433G
12.3GB
+ Both
73.98=t0.14
+3.20
15.24M
106.2G
135.6G
14.1GB
Both + WD
74.65:10.12
+3.87
15.24M
106.2G
142.1G
15.8GB
Note: STD values calculated over runs with different random soeds (42, 2023, 3407).
bak ans
Input image
Teacher's 's result
Student's 's result
w Lan
w LRp + Lep
W BCKD
KD
W/ JFVD
Ground-
Figure 5: Visualized comparisons and results obtained by adding different components of BCKD. The teacher model and the student model denotes PSPNet-101 (Zhao et al.) 2017) and PSPNet-18 (Zhao et al. 2017). respectively. "w/" denotes with the corresponding implementation. Samples are from Pascal VOC 2012 Everingham et al. 2010).
instcad, they complement cach other intrinsically to enhance performance. It is also worth noting that compared to the advanced methods in Table 3 that do not utilize ground truth mask, method also achieves competitive results even without employing the weight-decay strategy. Building upon LAD and LCD, adding the weight-decay strategy brings 0.57% mloU1, which confirms the weight importance of different losses. Furthermore, there is no increase in the number of Params. or FLOPs the inference stage.
We also employ visualizations as a means of verifying the efficacy of BCKD components incrementally into the baseline model. The obtained results are presented in Figure which demonstrate that the inclusion of LAD and LeD in sequential manner leads to enhanced boundary regions and object connectivity. For example, the *bike handlebor". Moreover, the integration of the weight-decay strategy results in further improvement in the overall segmentation predictions. In addition to using white bounding boxes to emphasize the better regions achieved by our method, we also highlighted the regions of prediction failure using red dashed bounding boxes. We can observe that although our BCKD significantly improves the prediction quality of these regions compared the student model's results, there are still some incomplete predictions. This case may be caused by the spurious correlation between the "helmet" and the "person" " in the used dataset, which be eliminated through causal intervention (Zhang al. 2020
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Result comparisons on mloU (%) under different network architectures on the val sets of Pascal VOC 2012 (Everingham et al. 2010), Cityscapes (Cordts et al. 2016,, and ADE20K (Zhou al. 2017]:
and on the lest set of COCO-Stul 10K (Caesar et al. 2018). Peak GPU memory in training measured NVIDIA GeForce RTX 3090. Results include standard deviations from 3 independent runs.
Methods
Pascal VOC val
Params.
Tr. GPU Mem. (GB)
DANet (Fu al.
2019
80.40=0.15
83.1M
9.8
T: PSPNet-101
Zhao et al.
2017
77.82+0.112
70.4M
8.2
3: PSPNet-38
Zhao et al.
2017
72.65+0.188
58.6M
6.1
+ BCK ours
75.15
58.6M
6.5
EfficientNet-B1
Tan Le
2019
69.28=0.225
6.7M
3.2
+ BCK D
73.81
6.7M
3.8
S: Seg
Xie et al.
2021
66.75+0.22
3.8M
2.9
+ BCK DakkDo.m ours
08.88:0.16
3.8M
3.3
Methods
Cityscapes val
Params.
GPU Mem.
(GB)
HRNet
Sun et al.
2019
81. 81.10:0.11
65.9M
7.9
T: PSPNet-101
Zhao al.
2017
78.56+0.11
70.4M
8.2
3: PSPNet-S3 38
Zhao al.
2017
71.26=#0.20
47.4M
5.8
+ BCK D Durs
73.56:10.15222
47.4M
6.2
S: EfficientNet-Bi
TTn & Le
2019
60.40=0.30
6.7M
3.2
+ BCKD ours
63.9110.222 43.66
6.7M
3.7
S: SegFormer- Xie al. 2021
76.20=00113
3.8M
2.9
+ BCK
77.87=0.10
3.8M
3.3
Methods
ADE20K val
Params.
GPU Mem. (GB)
DeepLab V3 Chen et
2017b
43.28:10118
71.3M
8.5
T: PSPNet-101
Zhao et al.
2017
42.19:10.22
70.4M
8.2
S: ESPNet
Mehta et al.
2018
20.13:20.35
0.4M
1.8
+ BCK ours
23.65.10.28 +3.02
0.4M
2.1
MobileNet (Liu
2018
33. 0.2.
8.3M
3.5
+ BCK D
36.59.10.191 1...0....442.5
8.3M
4.0
8: SegFormer-Bil Xie al.
2021
37.40=20.22
3.8M
2.9
+ BCKD ours
38.
3.8M
3.3
T: Mask2Former Cheng et al. 2022
47.20:10.16
44.0M
6.8
S: ESPNet + BCKD ours
24.26:10.30,4..5
0.4M
2.3
8: Mo + BCKD Dours ours
3..09=0.221...45
8.3M
4.2
SegFormer-Bo BO + BCKDoure
39.61=t0117
3.8M
3.5
Methods
COCO 10K fest
FLOPs
GPU Mem.
(GB)
SegVIT (Zhang et al.
2022a
50.310.15
383.9G
11.2
T: DeepLabV3 Plus- 101 Chen et al.
2018
33. 1.:0.22222
366.9G
9.1
8: V2 LLiu
2018
26. 06.2910.28
1.4G
3.5
+ BCK ours
28.31 L..2
1.4G
4.0
@: SegFormatter-Bi Xie al. 2021
35.6010.188
8.4G
2.9
+ BCKD
35.92.10.15.0.s3
8.4G
3.3
5.3.2 Effectiveness across network architectures
In this section, we evaluate the effectiveness our BCKD across various network architectures for SSeg using: Pascal VOC 2012 (Everingham al. 2010), Cityscapes (Cordts et al., 2016). ADE20K (Zhou al. 2017], and COCO-Stuff 10K [Caesar et al.) 2018). The obtained results under different network architectures are given in Table 2 For the purpose of comparing experimental results, we also include the results of large model
12

---

Published in Transactions on Machine Learning Research (01/2026)
Table Comparisons on mIoU (%) with state-of-the-art methods on the val sets of Pascal VOC 2012 Ever
ingham et al. 2010) and Cityscapes (Cordts al.) 2016), and ADE20K (Zhou et al. 2017). "I" denotes our re-implemented result based on the released codes due to inconsistencies in experimental settings. *KD Manner": knowledge distillation manner, which contains of layer-to-layer (L2L) and whole-to-whol (W2W) as illustrated in Figure "GTM": ground-truth mask used to obtain the pre-extraction image boundaries.
"Bod." denotes the boundary lype, and "Cont." denotes the context type. "E" physical edge. "B": semantic boundary. "P" pixel-wise. "P: image-wise. "0": object-wiid wise.
T: PSPNet-101 Zhao et al.
2017)
77.82%
78.56%
42.19%
S: PSPNet- (8
Zhao et al.
2017
70.78%
69.10%
33.82%
Methods
KD Manner
GTM?
Bod.
Cont.
Pascal VOC 2012
Cityscapes
ADE20K
+ KD (Hinton et al.
2015
L2L
X
71.28! -0.50
71.20
34.33' +0.51
+ SKD (Liu et al.
2019
T.2L.
/
E
P
73.05
71.45+2.35
34.65+0.83
SCKD Zhu & Wang
2021
L2L
X
72.33 1.55
72.10+3.00
34.76+0.94
+ CIRKD
Yang et al.
2022a
L,2L
K
X
I
73.57 -2.79
72.25+3.115
3/.93++1.11
+ IFD
Chen et al.
2022
L2L
X
X
73.88
-3.10
72.63+3..5
35.15' +1133
X
+ FGKD
Yang et al.
2022b
L2L
O
72.90
72.55+3.45
35.24
CWT
Liu et al.
2023
L22
O
73.06
-2.88
72.60
35.21 +1.39
SlimSeg
Xue
2022
L2L
X
74.08 33.00
73.95+-1.8
37.12+3.50
F
+ BGLLSSeg
Zhou et al.
2024
T.2L
x
74.27
-3.99
74.10 +5.00
36.49* 66.449+2.67 +2.67
+ FAA
Pham et al.
2024
L2L
74.28
-3.50
74.25
36.82" +3.00
CrossKD
Wang et al.
2024
T.2L
X
x
74.284
-3.50
74.28+5.18
36.72 +2.90
+ BPKD
Liu
2024
L2L
E
74.30
-3.52
74.29 +5.19
37.07" -07-++ +-3.25
+
W2W
X
B
74.65
74.92 45.82
37.62 +..82243880
+ LFVD Wang et al.
2020
L2L
X
74.05
74.41
36.63 4-3.85
IFVD + BCK D ours
W2W
X
B
P
74.82 4.04
74.99 4.6.8
37.73 $3.91
+ TAT (Lin
2022
L2L
O
74.02 -3.4
74.48 45.38
37.12 4-3.0
+ TAT + BCK]
W2W
X
B
P&O
74.71 33.9
74.97 46.87
38.12 444300
+ SSTKD (Ji et al. 2022
L2L
E
X
73.91 3.13
74.60
37.22 +3.40
+ SSTKD + BCK D ours
W2W
B&LE
P
74.52
3374
74.90 66.80
38.24 44..2
that does not utilize knowledge distillation for each dataset. From this table, we can observe that deploying BCKD different network architectures can lead to continuous performance improvements. For example, when employing PSPNet-388 (Zhao et al. 2017). Efficient (Tan & Le 2019 and SegFormer-B0 (Xie et al. 2021) as student models while keeping the teacher model PSPNet-LO1 101 (Zhao et al. 2017) unchanged, our BCKD achieves mloUt improvements of 2.50%, 4.53%, and 2.13% on Pascal VOC val, and 2.30%, 3.51%, and 1.67% on Cityscapes val, respectively. The performance gains demonstrate the effectiveness of our method not only within the same network architecture but also across network architectures, indicating sustained performance enhancements. This phenomenon also highlights the generalization capacity of our method. Besides, similar conclusions can be also drawn from experimental results ADE20K vol and COCO-Stuff 10K test sets. Deploying different teacher models the ADE20K val dataset also conducted, where FarmeFormer (Cheng et al.) 2022) is utilized as the teacher model, and ESPNet (Mehta et al. 2018). MobileNet V2 (Liu 2018), and SegFormer-BO are used as the student models. The results demonstrate that our method yields mloUt improvements of 4.13%, 3.45%, and 2.21% on ESPNet, MobileNet V2, and SegFormer-Br (Xie al. 2021), respectively, showcasing the strong flexibility. On efficiency, our method leverages the KD framework, resulting in no increase Params. or FLOPs. Consequently, we achieve both improved accuracy and fast inference speed.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Comparisons on mloU (%), MFS and LHD with state-of-the-art methods on the test set of COCO-
Stuff 10K (Cacsar 2018). Standard deviations are computed from independent runs.
DeepLabV3 Plus-101
33.10:10.15%
1.00:10.00
0.00
S: DeepLabV3 Plus-18
26.33:10.18%
0.45:10.08
4.21=0.35
Methods
mloU (%)
MFS (pi)
LHD
+ KD
Hinton et al.
2015
27.21#0.16,...
0.63
3.8210.44
+ SKD (Liu et al.
2019
27.27
0.67
3.7710.38
+ SCKD
Zhu & Wang
2021
27.38:1..14 (1.00.0
0.72+0..08
3.63=:0.42
CIRKD
Yang el al.
2022a
27.68:10.13 43.5
0.815
3.45=0.33
+ IFD
Chen et al.
2022
27.91 (1.55
O. 0..04
3.28=0.29
+ FGKD
Yang et al.
2022b
28. 13.000:1..11 41.67
0.90
3.12-0.31
+ CWT
(Liu et al.
2023
18.11810..11 +...000.......
0.95
2.98 0.22
IEVD
Wang al.
2020
28.35 422002
1.02:0.003
0.77=0.25
C2VKD
Zheng et al.
2023
28.42 1..08.0..8..0 +2.00
1.004
2.63
FAM
Pham et al.
2024
28 1..4888000000 1..0888........... 2.11
1.12+00.03
2.55=0.21
CrussKD
Wang et al.
2024
1.15.10.02
2.48100119
+ SSTKD
(Ji et al.
2022
28.70:10.06..
1.23+00.03
2.31=0.17
+ BCKD
29.22.10.00 +2.99
L..02
1.89
+ TAT
Lin et al.
2022
28.74:1..066-...0
1.25+L0.01
2.15
+ TAT +
BCKD
1.44+0.01
1.82=0.14
+ SlimSeg Xue al.
2022
28.50:10.08
1.13:00.03
2.52::0.20
+ SlimSeg +
BCKD
29.45++0.04
1.46++0.01
1.79=0.13
+ BPKD (Liu et al.
2021
28.66:10.07 42.33
1.21
2.36:10.16
+ BPKD + BCKD
29.60++0.03
1.49+00.01
1.72+0.11
5.3.3 Visualized comparisons
The visualized comparisons on the SSeg task with the baseline teacher and student models, and large models without the KD strategy are given in Figure 6 As highlighted by the white bounding boxes, the obtained results on Pascal VOC 2012 (Evoringham et al., 2010 Cityscapes (Cordts et al.) 2016), ADE20K (Zhou et al., 2017), and COCO-Stuff 10K (Caesar et al. 2018) demonstrate that BCKD yields significant improvements on both the boundary region completeness and the target region connectivity, when compared with the small student model's results. For example, the " "cow" in Pascal VÓC 2012, the "guidepost" in Cityscapes, the "desk" and the "TV bench" in ADE20K, the "bus", the "tennis racket", and the "guideboard" in COCO-Stuff 10K. The results obtained are basically the same as those of the large teacher model. Besides, compared to large models with higher model complexity (i.e., DANet (uu et al. 2019) and HRNet (Sun et al. 2019 on Pascal VOC 2012 and Cityscapes, although our method is not as competitive as theirs on quantitative results, our method achieves better predictions on object boundaries and small objects, which validate the effectiveness and emphasize the importance of boundary distillation and context distillation. With the help of our method, the student model is also able to predict better masks for certain fine-grained objects. For example, the " "cow's ear" and the "person's leg".
5.4 Comparisons With SOTA Methods on SSeg
In this section, we explore the accuracy and the effectiveness of the joint implementation of BCKD with the stale-of-the-arl (SOTA) KD methods on SSeg, To ensure a fair comparison, PSPNet-101 (Zhao el. al. 2017) and DeepLabV3 Plus- 101 (Chen et al. 2018) are employed as the teacher models, while PSPNet-18(1.0 and DecpLabV3 Plus-18 (Chen et al., 2018 serve as the student models. The specific settings for cach student model are described detail in the provided table. Some results are re-implemented by us on the released code due to inconsistencies in experimental settings and are marked with "i" in the given tables.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Pascal VOC
PSPNet-38 w/o and w/ BCKD
PSPNet- BCKD
DANet
Cityscaess
PSPNet- 1 (Teacher)
PSPNet-18 w/o BCKD
PSPNet-18 w/ BCKD
HRNet
ADE20K
PSPNet-101 Teadee)
PSPNet-18 18 w/o BCKD
PSPNet-18 w/ BCKD
MubileNetV2 w/o BCKD
MobileNet Nz BCKD
COCO-Stuff 10K
STOP
STOP
90
NO THRU
THRU
DecpLabV3 Plus-18 w/o w/ BCKD
DecpLabV3 Plus-18 w/o and w/ BCKD
MobileNetV2 w/0 and /BCKD
Visualizations on SSeg. DANet Fu 2019 and HRNet (Sun et al. 2019) have been included for comparison well. "w/o" means "without" and "w/" means "with", indicating whether our method is NOT implemented implemented. The white bounding boxes highlight the regions where method predicts better.
Figure 6:
5.4.1 Superiority of BCKD
Compared to the SOTA KD methods on SSeg, the top half of Table 3 and Table 4 we observe that our BCKD can surpass these methods. BCKD boosts the student model by 3.87%, 5.82%, and 3.80% mIoUf on the val sets of Pascal VOC 2012 (Everingham et al.] 2010 Cityscapes (Cordts et al. 2016) and ADE20K (Zhou et al.) 2017), respectively. Compared to the current SOTA KD methods on these datasets, BCKD outperforms IFVD (Wang et al. 2020), TAT (Tin et al. 2022 and SSTKD (Ji et al. 2022 on Pascal VOC 2012 by 0.6%, 0.63%, and 0.74% mloUf, respectively. The visualized comparison results with the classic KD (Hinoon al., 2015) and SOTA IFVD methods are presented in the last row of Figure observed that our method demonstrates significant advantages in capturing the connectivity of small objects well the integrity of the boundary masks. Furthermore, BCKD achieves higher mIoU than SOTA methods on Cityscapes and ADE20K datasets well. On the COCO-Stuff 10K (Caesar et al.) 2018) datasets in Table ! our method surpasses the student model and the SOTA TAT model by 2.89% and 0.48% mIoUf, respectively. As demonstrated in Table g the proposed BCKD framework also exhibits significant advantages in terms of MFS (pt) and LHD. These results not only indicate an overall performance improvement but also validate the effectiveness of our novel boundary distillation and context distillation, which were designed address these critical aspects of the learning process. Since inference is only conducted on the student model, our method does not introduce any increase in model complexity. These results across different datascts confirm that the task-specific knowledge is indeed more effective in practice compared to general knowledge.
It can be
5.4.2 Effectiveness of the joint implementation
The results on the joint implementation of BCKD and SOTA KD methods are presented on the lower half of Table and Table respectively. It can be observed that, on top of BCKD, further adding IFVD {Wang
et al. 2020 TAT [Lin al. 2022). and SSTKD) (Ji al. 2022) yields consistent, performance gains, with
15

---

Published in Transactions on Machine Learning Research (01/2026)
GFL
RetinaNet CMR-CNN
(a) Input Image
bb) Teacher
(c) Student
(d) w/ CrossDD
(e) w/ SSTK
w Ours
Figure 7: Visualization results on ISeg and ODet. "w/" means "with", indicating that the corresponding knowledge distillation method is deployed based on the student model. We chose the state-of-the-art methods
CrossKD Wang et al. (2024) and SSTKD Ji et al. (2022 for comparison.
mloUt improvements of 0.77%, 0.69%, and 0.61% on the val of Pascal VOC 2012, respectively. This can be attributed to the fact that BCKD contains semantic boundary and context that is not present in these methods, further demonstrating the importance of semantic boundary and context information for the SSeg task. However, adding SSTKD (Ji et al., 2022) on top of BCKD resulted in a performance decrease (i.e., 0.13% mloUt on Pascal VOC 2012 and 0.02% mloU on Cityscapes) compared to the accuracy on BCKD. We guess that this may be because SSTKD uses superficial image texture information, which is non-semantic and contain some noise relative to the extracted semantic boundaries. On COCO-Stulf 10K, we can observe that our method further enhances the performance of all SOTA methods, including TAT (Lin et al.] 20022), SlimSeg (uue et al. 2022 and BPKD LLu et al. 2024), and finally achieves 29.60% mIoU the test set.
5.5 Comparisons With SOTA Methods on ISeg and ODet
The quantitative result comparisons on ISeg and ODet are presented in Table 5 The obtained results indicate that method can consistently outperform existing methods across various baseline models, demonstrating its strong generalization and versatility. Specifically, achieve AP scores of 35.8%/38.88 37.0%/42.5%% and
33.3% 33.3%/38.5 33%55 for instance segmentation masks (i.e., APm) and object detection bounding boxes (i.e., APb) on the GFL-18 Li et al. 2020), Cascade Mask R-CNN-50 (Cai & Vasconcelos 2018). and RetinaNet-5k [Ross] & Dollár 2017), respectively. In comparison with the SOTA CrossKD (Wang et al. 2024) and SSTKD (Ji et al. 2022,, our method demonstrates an average performance improvement of approximately 0.5%. This enhancement serves to validate the effectiveness of our proposed method. The results also demonstrate significant advantages in both MFS and LHD, indicating its capability in preserving the structure of learned feature manifolds while maintaining high precision in boundary-sesittiv tasks. These results substantiate the iveness of our boundary and context distillation in maintaining geometric consistency and minimizing alignment errors.
The visual comparison results with baseline methods and SOTA methods are shown in Figure 7 It is observed that, relative to the baseline student models, the application of various KD strategies enhances the prediction results for specific classes (e.g., the person", the book", and the surfboand"), thereby affirming the effectiveness of KD in dense image prediction tasks. Moreover, when compared the SOTA methods CrossKD and SSTKD, method demonstrates improved connectivity in object regions and boundary integrity (e.g., the person" and the baseball bat"), highlighting the effectiveness of our proposed context distillation and boundary distillation strategies tailored for the targeted tasks. Additionally, our method addresses the issue of overlapping predicted bounding boxes (e.g., the mouse" and the "chair"), a benefit. attributed to the enriched contextual information incorporated into the student model via context distillation.
Furthermore, we also observed a significant phenomenon wherein our method effectively reduces the occurrence of hallucinations in the student model's predictions. Specifically, as depicted in the last column of Figure 7
16

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Result comparisons with the state-of-the-art methods on the val of MS-COCC 2017 Lin al. (2014) for ISeg and ODet. "CM R-CNN": Cascade Mask R-CNN. mAPm and mAPb denotes the average precision on instance segmentation mask and object detection bounding box, respectively.
Methods
APm (%)
AP (%)
FPS
MFS (pt)
LHD
T: GFT-50 Li al.
2020
36.8
40.2
19.4
1.00
0.00
S: GFL-18
LL el al.
2020
33.1
35.8
23.7
0.82
3.20
+
FGD (Yang et al.
2022b
34.0
36.6
23.7
0.85
2.90
SKD
Liu et. al.
2019
34.3
36.9
23.7
0.86
2.70
GID
Dai al.
34.6
37.8
23.7
0.89
2.50
LD
(Zhnng et al.
2022
34.8
38.0
23.7
0.90
2.30
+ PKD
[Cao et al.
2022b
35.0
38.0
23.7
0.91
2.20
CrossKD
Wang et al.
2024
35.3
38.1
23.7
0.92
1.90
SSTKD (Ji al.
2022
35.2
38.3
23.7
0.93
1.80
+ B3CDDour
35.8
38.8
23.7
0.95
1.60
T: CM R-CNN-101
Cai & Vasconcelos)
2018)
37.3
42.9
13.1
1.00
0.00
& CM R- R-CNN-50
Cai Vasconcelos
2018
36.5
41.9
0.88
2.10
+ FGD (Yang et al.
2022b
35.3
42.1
16.1
0.87
2.20
+ SKD
Liu et al.
2019
36.5
42.2
16.1
0.89
2.00
GID
Dai et al.
2021
36.7
42.0
16.1
0.90
1.90
LD
Zheng al.
2022
36.8
42.1
16.1
0.91
1.80
+ PKD
[Cao et al.
2022b
36.8
42.0
16.1
0.92
1.70
+ CrossKD
Wang et al.
2024
36.9
42.2
16.1
0.93
1.60
SSTKD
(Ji et al.
2022
37.0
42.2
16.1
0.94
1.50
+ BCK ours
37.0
42.5
0.96
1.40
T: RetinaNet-101 Ross & Dollár
2017
33.5
38.9
13.5
1.00
0.00
S: RetinaNet-50 Ross & Dollár
2017
31.7
37.4
17.7
0.85
2.40
+ FGD (Yang
2022b
32.1
37.7
17.7
0.86
2.20
+ SKD Liu et al.
2019
32.5
37.5
17.7
0.87
2.12
+ GID
Dai et al.
2021
32.8
37.6
17.7
0.88
2.03
LD Zheng al. 2022
33.1
37.8
17.7
0.89
1.90
PKD
Cao al.
2022b
33.0
37.8
17.7
0.90
1.88
+ CrossKD
Wang et al.
2024
33.2
38.0
17.7
0.91
1.75
+ SSTKD
(Ji et. al.
2022
33.1
38.1
17.7
0.92
1.66
+ BCKD ouss
33.3
38.5
17.7
0.94
1.54
both the teacher and student models fail to identify the "camera", while the CrossKD and SSTKD methods mistakenly classify the " "cmmera" as the "bottle". In contrast, our approach accurately recognizes the "camera" as a background object, aligning with definitions. We hypothesize that this discrepancy may stem from the confusion of target knowledge caused by task-irrelevant, KD during the training process. Our proposed task-specific BCKD inherently designed to alleviate such confusion from the outset.
6 Conclusion and Future Work
In this work, we propose a customized boundary and context knowledge distillation (BCKD) method tailored for efficient dense image prediction tasks on Al accelerator, including semantic segmentation, instance segmentation, and object detection. Our approach significantly narrows the performance gap between compact, efficient models and their larger, more accurate counterparts while maintaining computational efficiency. Specifically, BCKD enhances boundary-region completeness and ensures object-region connectivity, leading
17

---

Published in Transactions on Machine Learning Research (01/2026)
to consistent accuracy improvements across diverse challenging benchmarks and architectures. Theoretical analysis further corroborates the effectiveness of our method.
As a generalizable method, in the future, we plan to extend BCKD to additional dense visual tasks (e.g., pose estimation and image generation) and investigate its adaptation to emerging architectures (e.g., Vision Mamba) to better support model compression for AT accelerator deployment. Besides, we will explore synergies between BCKD and large foundation models (e.g., Segment Anything Model and vision-language models) to further enhance the robustness of lightweight dense predictors under adverse conditions. Furthermore, investigating BCKD in next-generation models, such as Vision Mamba (Vim), also represents a highly practical and instructive research direction.
Acknowledgements
This work was supported by the National Natural Science Foundation China Young Scholar Fund Category В (62522216), Young Scholar Fund Category (62402408), the HKSAR RGC General Research Fund (16219025), HKSAR RGC Early Carcer Scheme (26208924), and HKSAR RGC General Rescarch Fund (16208823).
Broader Impact Statement
We acknowledge the broader implications of deploying EDIP models in real-world applications. While EDIP models are designed to be efficient and effective for edge device deployment, there are potential risks associated with their use high-stakes scenarios such as mass surveillance, facial recognition, or other security-related applications. These applications may raise ethical concerns and require careful adherence to regulatory frameworks ensure responsible use. As such, we emphasize the importance of ethical considerations and legal compliance when deploying such models, and recommend the implementation of safeguards protect privacy.
We also recognize that biases present in the training data may be inadvertently amplified during model distillation or other knowledge transfer processes in EDIP systems. For example, the "helmet /person" correlation serves as case study demonstrating such datasot bias. To address this, we will propose mothods such as fairness-aware training and post-hoc bias correction to reduce the likelihood of biased outcomes and ensure more equitable model behavior.
Finally, also consider the environmental impact of our model training process, which may involve up to 40,000 training epochs. Although our model is elficient in inference, the high computational and energy demands during training raise concerns about sustainability. To mitigate this, we suggest future strategies such as progressive training, knowledge distillation with low-rank adaptation, and early stopping techniques, where applicable, to reduce the overall environmental footprint of model development.
References
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 4981-4990, 2018.
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp.
2019.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In The IEFE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 1209-1218, 2018.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In The IEEE / CVF Computer Vision and Pallern, Recognition Conference (CVPR), pp. 6154-611 6662, 2018.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. In The European Conference on Computer Vision (ECCV), pp. 205 218, 2022a.
Weihan Cao, Yifan Zhang, Jianfei Gao, Anda Cheng, Ke Cheng, and Jian Cheng. Pkd: General distillation framework for object detectors via pearson correlation coefficient. In Advances in Neural Information Processing Systems (NewrIPS), pp- 15394-15406, 2022b.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2017a.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 40(4):834-88) 848, 2017b.
Giang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In The European Conference Computer Vision (ECCV), pp. 801 818, 2018.
Liyi Chen, Weiwei Wu, Chenchen Fu, Xiao Han, and Yuntao Zhang, Weakly supervised semantic segmentation with boundary exploration. In The European Conference on Computer Vision (ECCV), pp. 347-362 2020.
Wenshu Chen, Yujie Huang, Mingyu Wang, Xiaolin Wu, and Xiaoyang Zeng. Kbstyle: Fast style transfer using 200 kb network with symmetric knowledge distillation. IEEE Transactions Image Processing, 33:82 94, 2023.
Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. In Advances in Neural Information Processing Systems (NewrIPS), pp. 12084-12095, 2022.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image sogmentation. In The IEEE/ / CVF Computer Vision and Pottern Recognition Conference (CVPR), pp. 1290 1299, 2022.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In The IEEE/ CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 3213-3223, 2016.
Kaiwen Cui, Yingchen Yu, Fangneng Zhan, Shengeai Liao, Shijian Lu, and Eric P Xing. Kd-dlgan: Data limited image generation via knowledge distillation. In The IEEE / CVF Compuler Vision and Pallern Recognition Conference (CVPR), pp. 2872-3882) 2023.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General instance distillation for objoct detection. In The IEFE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 7842 7851, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In The IEEE / CVF Computer Vision and Pottern Recognition Conference (CVPR), pp. 248 255, 2009.
Pingcheng Dong, Zhnoyu Chen, Zhuoao Li, Yuzhe Fu, Lei Chen, and Fengwei An. A 4.29 nj/pixel stereo depth coprocessor with pixel level pipcline and region optimized semi-global matching for iot application. IRFE TCS, 69(1):334 346, 2021.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Pingcheng Dong, Zhuoyu Chen, Ke Li, Lei Chen, Kwang-Ting Cheng, and Fengwei An. A 1920x 1080 129fps 4.3 pi/pixel stereo-matching processor for pico aerial vehicles. In ESSCIRC, pp. 345-3455 2023.
Pingcheng Dong, Yonghao Tan, Xuejiao Liu, Peng Luo, Yu Liu, Luhong Liang, Yitong Zhou, Di Pang, Man-To Yung, Dong Zhang, et al. A 28mm 0.22 pj/token memory-compine cammory-coopuutc-ittonstttyyyyii layer-fusion and cascaded pruning for semantic-segmenniie IEEE International Solid-Stafe Circuits Conference (ISSCC), volume 68, pp. 01 03, 2025.
onn-transformer
accelerator with hybrid-attentionn-eenn
Mark Everingham, Luc Van Gool, Christopher KT Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303 338, 2010.
Kenneth John Falconer and John D Howroyd. Projection theorems for box and packing dimensions. In Mathemalical Proceedings of the Cambridge Philosophical Sociely, volume 119, pp. 287 295.. Cambridge University Press, 1996.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVE International Conference on Compuler Vision (ICCV), pp. 6824 6835, 2021.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In The IEEE/ CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 3146 3154, 2019.
Ross Girshick. Fast r-cnn. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICGV), pp. 1448, 2015.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1819, 2021.
Jianping Gou, Xiangshuo Xiong, Baosheng Yu, Lan Du, Yibing Zhan, and Dacheng Tao. Multi-target knowledge distillation via student self-reflettinn. International Journal of Compuler Vision, pp. 1 1-18, 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv, 2023.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cit: Convolutional neural networks meet vision transformers. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 12175-12185, 2022a.
Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. In Advances in Neural Information Processing Systems (NeurIPS), pp. 1140-1156, 2022b.
Qiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu, Ding Liang, Xiaolin Hu, and Ping Luo. Online knowledge distillation via collaborative learning. In The IEEE / CVF Computer Vision and Pattern Recognátion Conference (CVPR), pp. 11020-11029, 2020.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey vision transformer. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):87-110, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 770 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv, 2015.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss- cross attention for semantic segmentation. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 603 612, 2019.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Deyi Ji, Haoran Wang, Mingyuan Tao, Jianqiang Huang, Xian-Sheng Hua, and Hongtao Lu. Structural and statistical texture knowledge distillation for semantic segmentation. In The IEFF / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 16876 16885, 2022.
Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and n-Chuul Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In The IEEE / CVF Computer Vision and Pottern Recognition Conference (CVPR), pp. 10664-10673, 2021.
RP Kanwal and KC Liu. A taylor expansion approach for solving integral equations. International Journal of Mathematical Education in Science and Technology, 20(3):4114415 1989.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustalson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 4015-40256 2023.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In The European Conference on Computer Vision (ECCV), pp. 620 640, 2022.
Tze Loung Lai. Stochastic approximation. The annals of Statistics, (1( 2003.
Seunghyun Lee and Byung Cheol Song. Graph-based knowledge distillation by multi-head attention network. arXiu, 2019.
Jiashi Li, Xin Xia, Wei Li, Huixia Li, Xing Wang, Xuefeng Xiao, Rui Wang, Min Zheng, and Xin Pan. Next-vit: Next generation vision transformer for efficient deployment in realistic industrial scenarios. arXiv, 2022a.
Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. In Advances in Neural Information Processing Systems (NeurIPS), pp. 21002 21012, 2020.
Yuqi Li, Chuanguang Yang, Hansheng Zeng, Zeyu Dong, Zhulin An, Yongjun Xu, Yingli Tian, and Hao Wu. Prequency-aligned knowledge distillation for lightweight spatiotemporal forecasting. In Proceedings of the IEEE/CVE International Conference Computer Vision (ICCV), pp. 7262-7272, 2025.
Zhong-Yo Yu Li, Shanghua Gao, and Ming-Ming Cheng. Exploring feature self-relation for self-supervised transformer. arviv, 2022b.
Fangjian Lin, Zhanhao Liang, Sitong Wu, Junjun He, Kai Chen, and Shengwei Tian. Structtoken: Rethinking semantic segmentation with structural prior. IEEE Transactions on Circuits and Systems for Video Technology, 2023a.
Jiacheng Lin, Zhiqiang Xiao, Xiaohui Wei, Puhong Duan, Xuan He, Renwei Dian, Zhiyong Li, and Shutao Li. Click-pixel cognition fusion network with balanced cut for interactive image segmentation. IEEE Transactions on Image Processing, 33:177-190, 2023b.
Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distillation via the target-aware transformer. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 10915-10924, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In The European Conference on Computer Vision (ECCV), pp. 740 755, 2014.
Yi Lin, Dong Zhang, Xiao Fang, Yufan Chen, Kwang-Tine -ring-TTing Cheng, and Hao Chen. Rethinking boundary detection in deep learning models for medical image segmentation. In Information Processing In. Medical Imaging, pp. 730-76 742, 2023c.
Huijun Liu. Lightnet: Light-weight networks for semantic image segmentation. LighCNeC. himd, 5:6, 2018.
21

---

Published in Transactions on Machine Learning Research (01/2026)
Liyang Liu, Zihan Wang, Minh Hieu Phan, Bowen Zhang, Jinchao Ge, and Yifan Liu. Bpkd: Boundary privi- leged knowledge distillation for semantic segmentation. In TEEE/CVP Winter Conference on Applications of Computer Vision (WACV), pp. 1062 1072, 2024.
Ruiping Liu, Kailun Yang, Alina Roitberg, Jiaming Zhang, Kunyu Peng, Huayao Liu, and Rainer Stiefelhagen. Transkd: Transformer knowledge distillation for efficient semantic segmentation. arXiv, 2022.
Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang, Structured knowledge distillation for semantic segmentation. In The IEEE/ /VF Computer Vision and Pattern Recognition
Conference (CVPR), pp. 2604-2613 2019.
Ziwei Liu, Yongtao Wang, Xiaojie Chu, Nan Dong, Shengxiang Qi, and Haibin Ling. A simple and generic framework for feature distillation via channel-wise transformation. In Proceedings of the IEEE/CV International Conference on Compuler Vision (ICCV) Workshops, pp. 1129 1138, 2023.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 3431-3440, 2015.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjic Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In The IEEE / CVF Computer Vision and Pottern Recognition Conference (CVPR), pp. 12042 12051, 2022.
Sachin Mchta, Mohammad Rastegari, Anat Caspi, Linda Shapiro, and Hannanch Hajishirzi. Espnct: Efficient spatial pyramid of dilated convolutions for semantic segmentation. In The European Conference on Computer Vision (ECCV), pp. 552-568, 2018.
Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic seg- mentation. In Proceedings of the IFEE/CVF International Conference on Compuler Vision (ICCV), pp- 1520-1528, 2015.
Peyman Passban, Yimeng Wu, Mchdi Rezugholizadch, and Qun Liu. Alp-kd: Attention-based layer projection for knowledge distillation. In The Association for the Advancement of Artificial Intelligence (AAAI), pp. 13657-13665, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowci Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In The IEEE / CVF Compuler Vision and Pattern Recognition Conference (CVPR), pp. 367-376, 2021.
Cuong Pham, Van-Anh Nguyen, Trung Le, Dinh Phung, Gustavo Carneiro, and Thanh-Toan Do. Frequency attention for knowledge distillation. In IEEE/CVE Winter Conference Applications of Computer Vision (WACV), pp. 2277-2286, 2024.
Mary Phuong and Christoph H Lampert. Distillation-based training for multi-cxit architoctures. In Proceedings of the IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 1355 1364, 2019.
T-YLPG Ross and GKHP Dollár. Focal loss for dense object detection. In The IEEE / CVF Computer Vision and Pollern Recognilion Conference (CVPR), pp. 2980 290000298888-0 2017.
Lixiang Ru, Yibing Zhan, Baosheng Yu, and Bo Du. Learning alfinity from attention: End-to-end weakly- supervised semantic segmentation with transformers. In The IEEE CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 16846-16855, 2022.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2021.
22

---

Published in Transactions on Machine Learning Research (01/2026)
Ke Sun, Yang Zhao, Borui Jiang, Tianheng Cheng, Bin Xiao, Dong Liu, Yadong Mu, Xinggang Wang, Wenyu Liu, and Jingdong Wang. High-resolution representations for labeling pixels and regions. arXiv, 2019.
Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Distilling object detectors with task adaptive regularization. arxiv, 2020.
Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In The International Conference on Machine Learning (ICML), pp. 6105 6114, 2019.
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent rein- forcement learning with knowledge distillation. In Advances in Neural Information Processing Systems (NeurIPS), pp. 226 237, 2022.
Chi Wang, Yunke Zhang, Miaomiao Cui, Peiran Ren, Yin Yang, Xuansong Xie, Xian-Sheng Hua, Hujun Bao, and Weiwei Xu. Active boundary loss for semantic segmentation. In The Association for the Advancement of Artificial Intelligence (AAAI), pp. 2397-24090 2022a.
Jiabao Wang, Yuming Chen, Zhaohui Zheng, Xiang Li, Ming-Ming Cheng, and Qibin Hou. Crosskd: Cross- head knowledge distillation for object detection. In The IEEE / CVF Computer Vision and Pattern 2024.
Recomnition Conference (CVPR), pp.
Rui Wang, Yixue Hao, Long Hu, Xianzhi Li, Min Chen, Yiming Miao, and Iztok Humar. Efficient crowd counting via dual knowledge distillation. IEEE Transactions on Image Processing, 2023.
Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In The IEEE/ CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 2019.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVE /CFF International Conference on Computer Vision (ICCV), pp. 568 578, 2021a.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervised visual pre-training. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 3024-3033, 2021b.
Yukang Wang, Wei Zhou, Tao Jiang, Xiang Bai, and Yongchao Xu. Intra-class feature variation distillation for semantic segmentation. In The European Conference on Computer Vision (ECCV), pp. 346-362, 2020.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. Tn The IEFE / CVF Computer Vision and Pottern Recognition Conference (CVPR), pp. 8741-8750, 2021c.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In The IEEE / CVE Computer Vision and Pattern Recognition Conference (CVPR), pp. 17683-17693, 2022b.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Ta Yuan, and Tei Zhang. Cvt: Introducing convolutions vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22-31, 2021.
Qianlong Xiang, Miao Zhang, Yuzhang Shang, Jianlong Wu, Yan Yan, and Liqiang Nie. Dkdm: Data-free knowledge distillation for diffusion models with any architecture. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 2965, 2025.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Advances in Neural Information Processing Systems (NeurIPS), pp. 12077-12090, 2021.
23

---

Published in Transactions on Machine Learning Research (01/2026)
Wei Xie, Zimeng Zhao, Shiying Li, Binghui Zuo, and Yangang Wang. Nonrigid object contact estimation with regional unwrapping transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9342 9351, 2023.
Kunran Xu, Lai Rui, Yishi Li, and Lin Gu. Feature normalized knowledge distillation for image classification. In The European Conference Computer Vision (ECCV), pp. 664-680, 2020.
Liuchi Xu, Kang Liu, Jinshuai Liu, Lu Wang, Lisheng Xu, and Jun Cheng, Local dense logit relations for enhanced knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4539 454944549,40 2025.
Rui Xu, Shu Yang, Yihui Wang, Bo Du, and Hao Chen. A survey on vision mamba: Models, applications and challenges. arxie, 2024.
Danna Xue, Fei Yang, Pei Wang, Luis Herranz, Jinqiu Sun, Yu Zhu, and Yanning Zhang. Slimseg: Slimmable semantic segmentation with boundary supervision. In ACM International Conference on Multimedia (ACM MM), pp. 6539-6548, 2022.
Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 12319 12328, 2022a.
Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 1643-4652, 2022b.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IREE/CVE International Conference on Computer Vision (ICCV), pp- 4210-4220, 2023.
Dongshuo Yin, Yiran Yang, Zhechao Wang, Hongfeng Yu, Kaiwen Wei, and Xian Sun. 1% vs 100%: Parameter--ffiiint low rank adapter for dense predictions. In The IEEE / CVF Computer Vision and Pottern Recognition Conference (CVPR), pp. 20116 201226, 2023.
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In The European Conference on Computer Vision (ECCV), pp. 925-341, 2018.
Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, and Daxin Jiang. Reinforeed multi-teacher selection for knowledge distillation. In The Association for the Advancement of Artificial Intellignnce (AAAI), pp. 14284-14291, 2021.
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextuaal representations for semantic segmentation. In The European Conference Computer Vision (ECCV), pp. 173-190, 2020.
Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), pp. 4971 4982, 2022a.
Dong Zhang and Kwang- Ting Cheng. Generalized task-driven medical image quality enhancement with gradient promotion. IEEE Trunsactions on Paltern Analysis and Machine Intelligence, 2025.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal intervention for weakly- supervised semantic segmentation. In Advances in Neural Information Processing Systems (NewrIPS), pp. 655-666, 2020.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Self-regulation for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6953-68 6963, 2021.
24

---

Published in Transactions on Machine Learning Research (01/2026)
Dong Zhang, Jinhui Tang, and Kwang-Tine Ting Cheng. Graph reasoning transformer for image parsing. In ACM International Conference on Mullimedia (ACM MM), pp. 2380-23305 2022b.
Dong Zhang, Chengting Zuo, Qianhao Wu, Liyong Fu, and Xinguang Xiang. Unabridged adjacent modulation for clothing parsing. Pattern Recognition, 127:108594, 2022c.
Dong Zhang, Yi Lin, Jinhui Tang, and Kwang-Tine Cheng. Cae-great: Convolutionall-uuxllyyy efficient graph reasoning transformer for dense image predictions. International Journal of Computer Vision, pp. 1-19, 2023.
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 7151 7160, 2018.
Linfeng Zhang and Kaisheng Ma. Structured knowledge distillation for accurate and efficient object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVE International Conference Compuler Vision (ICCV), pp. 3713 11132,222,22 2019.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In The IEFE/ / CVF Computer Vision and Pallern Recognilion Conference (CVPR), pp- 11953 11962, 2022.
Hengshuang Zhao, Jianping Shi, Xisojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In The IEEE/ / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 2881-2890, 2017.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequence-to-sequene perspective with transformers. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 6881 6890, 2021.
Xu Zheng, Yunhao Luo, Pengyuan Zhou, and Lin Wang. Distilling efficient vision transformers from cnns for semantic segmentation. arx 2023.
Zhaohui Zheng, Rongguang Ye, Ping Wang, Dongwei Ren, Wangmeng Zuo, Qibin Hou, and Ming-Ming Cheng. Localization distillation for dense object detection. In The IEEE / CVF Computer Vision and Pottern Recognition Conference (CVPR), 2022.
Du Zhixing, Rui Zhang, Ming Chang, Shaoli Liu, Tianshi Chen, Yunji Chen, et al. Distilling object detectors with feature richness. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5213 5224, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), pp. 633 641, 2017.
Quan Zhou, Linjie Wang, Guangwei Gao, Kang Bin, Weihua Ou, and Huimin Lu. Boundary-guided lightweight semantic segmentation with multi-scale semantic context. IEEE Transactions on Multimedio, 2024.
Yichen Zhu and Yi Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5057-5066, 2021.
25

---

Published in Transactions on Machine Learning Research (01/2026)
A
Appendix
Theoretical Analysis
BCKD is Uheoretically grounded in differential geomelry and spectral graph theory. In the Appendix, we will analyze its components through measurable properties of the learned feature manifolds Mr (teacher) and Ms (student), with proofs connecting to the empirical results in Section 5.3
A.1 Boundary-Aware Manifold Alignment
The proposed LBD in Eq. (3) induces geometric consistency between teacher and student decision boundaries.
Theorem 1 (Boundary Consistency). Under LBD minimization with T >> 1, for any boundary poinl Bb, have:
[r - Js(x))]l S + O(e
(13)
where J. are Jacobian matrices of the feature maps.
Proof. The temperature-scaled gradients satisfy:
1/7 0T 1/7
VLDD = Erg
v
ls
as
(14)
V - VLs 2 < 2(1
cos 0);
(15)
where 8 is the angle between S and T gradicnts. Applying Taylor expansion Kanwal & Liu 1989 at high т, then we can obtain:
1
cos 0 2 1
2 LLDD + O()
(16)
The Jacobian bound follows from Pinsker's inequality applied to the manifold tangent spaces.
This thcoretical guarantee explains the 0.91% mIoU improvement observed in Table ??, as aligned Jacobians ensure consistent boundary localization.
A.2 Contextual Graph Preservation
The proposed LCD in Eq. 5 maintains spectral properties critical for dense image prediction tasks:
Theorem 2 (Spectral Convergence). For eigenwalues {x+} of relation matrices RR,SR we have:
max INT
X < T - SS S VdLCD-
(17)
Proof. Applying Weyl's inequality for symmetric matrices, we obtain:
i> x <
AR 12 S |AR||E
(18)
where the heat kernel continuity follows from:
TLr -- I sup
||e -Cr (Lr - 1E[o,r]
(t (1-)) s TelllCrl L
(19)
As shown in Table ??, compared with the baseline model, the 1.77% mloU gain directly reflects this eigenvalue stability.
26

---

Published in Transactions on Machine Learning Research (01/2026)
A.3 Multi-Scale Geometric Consistency
As illustrated in Figure topological invariants:
the feature concalenation and projection operation in Section 4.1
preserves
Proposition (Topological Preservation).
The mapping @ II. M - M satisfies:
8u (M Concat
Σ (M == 0,21,
(20)
where Bk are Betti numbers.
Proof. The 3x3 convolution operation is a diffcomorphism, thus we have:
(c()(T))) = (T)
(invariance)
(21)
-c = Σ (M (Künneth formula)
The dimensionality bound follows from the classical projection theorem as in Falconer & Howroyd 1996. 3
A.4 Training Dynamics Interpretation
The used weight decay in Eq. induces phased learning:
Theorem 3 (Annealed Convergence). With r(t) (t- 1-////mmxx and Robbins-Monro conditions on learning rate T)c
lim P(C Lss) =1 [=+lmax
(22)
Proof. Decompose the gradient flow:
dC
W ( +
(23)
dt
As r() 0. the right terms vanish asymptotically. The convergence follows the stochastic approximation theory Lai 2003).
Pseudo-Code of BCKD
B BCKD Algorithm
The Boundary and Context Knowledge Distillation (BCKD) consists of two main components: Boundary Distillation and Context Distillation. Below we present the detailed pseudo-code.
27

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm Boundary and Context Knowledge Distillation (BCKD)
Require: Teacher model T, Student model S, Input image X Require: Tempcrature т, Loss weights a,
Require:
Maximum training epochs bmon
Feature extraction:
1: IF + [T]{})}
Get teacher features
2: SF +- (S((X)
b Get student features
Tr, +- logits from 7 and
Feature processing:
T, t ConVaxa (concctttt Ts))
S. + C (concat(S)..
Ss))
Boundary Distillation:
6: for each pair (i,j) of pixels do
7:
Th t - mal,, ((Convixa(TH) Convixx(TT)
SB + maxp. B(Convix1 Convix1 (S!))
end for
10: LBD + - p(T))) 1/7 log(p(()))
Context Distillation:
O(T.)"OT..) /T
11: TA +
Vi
12: Sn +-+-00 a (((s.)
/T
Va
13: LOD +- - (Th) 1/7 (S)
Loss computation:
14: (1
15: L + 3BCDD)
Model update:
16: Update S parameters via V.
Ensure: Trained student model S
28


Reviews:
[{"review_id": "nBcyRqRXo2", "reviewer": "Reviewer_v1Nq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Boundary and Context Knowledge Distillation (BCKD) for compressing dense image prediction models (semantic segmentation, instance segmentation, object detection) into compact models suitable for edge devices. The key claim is that standard KD focuses on \"global\" knowledge and fails specifically on (i) boundary-region completeness and (ii) target-region connectivity. To address this, the authors introduce:\n\nBoundary distillation: derive semantic boundaries from hierarchical feature maps of teacher/student and distill them, to improve boundary masks without pre-extracted ground truth edges (Eq. 2-3, Fig. 3).\n\nContext distillation: distill pixel-wise self-relations computed from concatenated features (whole-to-whole relation matrix) to preserve region connectivity (Eq. 4-5, Fig. 4).\n\nThey also define two additional evaluation metrics – manifold stability and local Hausdorff distance – to analyze feature geometry and boundary quality. Experiments on semantic segmentation (VOC12, Cityscapes, ADE20K, COCO-Stuff 10K) and detection/instance seg (COCO 2017) show consistent gains in mIoU and average precision across many teacher-student pairs, without extra inference cost, and BCKD can be combined with prior KD methods.\n\nKey strengths:\n- Clear and well-motivated failure modes\n- Broad experimental coverage\n- Synergy with existing KD methods\n- No inference-time overhead\n\nKey weakness:\n- Over-stated claims on the theoretical section", "claims_and_evidence_yn": "No", "claims_explanation": "Claim 1: Identification of two key issues in EDIP models\nThe authors claim to have \"revealed two prevalent issues in existing EDIP models: maintaining boundary region completeness and ensuring target region connectivity.\"\n\nIn my view, this is adequately supported as a qualitative motivation, but not rigorously established quantitatively. The paper provides several clear qualitative examples where small students exhibit broken regions and incomplete boundaries, and these error modes are intuitively plausible and widely observed in practice. However, there is no systematic analysis showing that these are the dominant or \"prevalent\" failure modes across EDIP models in general. I would regard this as a compelling but largely anecdotal problem characterization rather than a statistically grounded one.\n\nClaim 2: BCKD is a coherent, targeted method that can coexist with other KD methods\nThe authors claim that their boundary and context distillation components are \"customized and targeted\" to these two issues, are \"inherently coherent,\" and can be combined with other KD methods without conflict. \n\nI consider this mostly adequately supported. The ablation studies show that the boundary loss and context loss each improve performance and that combining them yields further gains, which supports the \"targeted and complementary\" story. Moreover, experiments combining BCKD with IFVD, TAT, SlimSeg, BPKD, etc., generally yield additional improvements, showing that it can be stacked on existing methods. That said, coexistence is not uniformly beneficial (e.g. slight degradations when combined with SSTKD on some datasets), so the strongest form of the claim (\"coexists\" and helps in all cases) is overstated. A more accurate reading is that BCKD is broadly compatible and often, though not always, beneficial on top of prior KD methods.\n\nClaim 3: Theoretical analysis demonstrates the superior effectiveness of BCKD\nThe paper claims that \"theoretical analysis demonstrates the superior effectiveness of our BCKD.\"\n\nI do not find this claim adequately supported. The theoretical section derives high-level bounds on Jacobian alignment, spectral properties of relation matrices, and some topological intuitions, but these arguments mainly show that minimizing the proposed losses encourages certain geometric properties. These arguments do not provide a comparative analysis against alternative KD objectives or a quantitative link to observed performance gains. The connection from theorems to actual mIoU/AP improvements is largely narrative, and no empirical tests are provided that specifically validate the theoretical quantities as better predictors of performance than simpler baselines. I would regard the theory as suggestive and plausibility-enhancing, rather than as a demonstration of \"superior effectiveness.\"\n\nClaim 4: Experimental evaluations show superior accuracy across tasks, baselines, and datasets\nFinally, the authors claim that experiments \"across various tasks, baselines and datasets illustrate the superior accuracy of our method in comparison with existing methods.\" \n\nI think this is largely, but somewhat modestly, supported. The empirical evaluation is broad: multiple segmentation datasets (VOC, Cityscapes, ADE20K, COCO-Stuff), detection and instance segmentation on COCO, and a range of teacher-student pairs. Across these, BCKD consistently improves over non-distilled students by a clear margin and generally edges out strong KD baselines by smaller but fairly consistent margins. The main caveats are that improvements over the strongest baselines are sometimes within the range where variance could matter (no error bars or multiple seeds are reported), and \"superior\" here means \"incrementally better\" rather than dramatically so. Within the experimental scope of the paper, I would accept the claim that BCKD is a new state-of-the-art KD approach for these dense prediction tasks, but with the understanding that the gains over prior methods are moderate.", "audience_yn": "Yes", "audience_explanation": "For TMLR’s audience, I think the direct appeal of this paper is fairly focused but non-trivial. It will be most interesting to researchers and practitioners working on dense vision tasks (semantic segmentation, detection, instance segmentation) who care about compressing large models into edge-deployable students without changing architectures, as well as people actively working on knowledge distillation for vision.", "requested_changes": "Critical: I believe that claim 3 is overstated as-is. Please consider re-wording as mentioned above. \n\nNon-critical: \n- In the introduction, replace \"The main contributions...four folds\" with \"The main contributions of this work are: (1)...(2)...\"\n- Clarify how \\prod_{i,j} in equation 2 is constructed/sampled in practice.\n- Write out the concrete implementation of O() in section 4.3. This will help with reproducibility.\n- Add an algorithm-style (i.e. pseudo-code box) summary of BCKD in the appendix. Its implementation is currently spread across Sections 4.1-4.4 plus training details in Section 5.2.2.", "broader_impact_concerns": "N/A", "additional_comments": "Nice work! And apologies for my delay with this review.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Kd5ihOWIeU", "reviewer": "Reviewer_rB8i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary Of Contributions\nThe paper introduces Boundary and Context Knowledge Distillation (BCKD), a targeted distillation framework tailored for efficient dense image prediction (EDIP) models deployed on AI chips, addressing two persistent issues in KD-based EDIPs: boundary incompleteness and weak region connectivity. Its key contributions include: (1) a boundary distillation mechanism that extracts explicit object-level boundary cues from hierarchical teacher features to improve the student’s mask fidelity near edges; (2) a context distillation module that transfers implicit pixel-level relational contexts via self-relations, strengthening connectivity within target regions; and (3) a simple, efficient, and EDIP-specific KD design validated through theoretical analysis and extensive experiments on five datasets spanning semantic segmentation, object detection, and instance segmentation, showing consistent gains in boundary sharpness and region coherence for compact real-time models.\n\n\n\n# Strengths And Weaknesses\n\n## Strengths\n* The paper provides a clear and well-motivated problem formulation by identifying two persistent failure modes in EDIP models—boundary incompleteness and broken region connectivity—that are not sufficiently handled by existing KD approaches.\n* The proposed BCKD framework is thoughtfully tailored to the unique characteristics of EDIP tasks, rather than relying on generic task-agnostic KD, which strengthens its conceptual coherence and practical relevance.\n* The decomposition of distillation into boundary-level and context-level components is intuitively appealing and directly addresses the error patterns observed in compact models, improving both local boundary fidelity and global region coherence.\n* The method integrates cleanly with existing architectures and KD strategies, enhancing its practicality for real-world deployment on heterogeneous edge devices.\n* The experimental validation is comprehensive—covering multiple tasks, baselines, and datasets—and convincingly demonstrates that the method yields consistent and meaningful gains in both qualitative structure (boundaries, connectivity) and quantitative performance.\n\n## Weakness\n* The paper occasionally overstates the novelty of identifying boundary and connectivity issues, which are already well-documented in prior segmentation literature; clearer positioning against existing boundary-aware KD or structure-preserving distillation methods would strengthen the contribution.\n* The proposed approach appears to rely on additional boundary extraction and relational modeling steps, yet the paper does not discuss potential computational or memory overheads in the distillation phase and comparison with the SoTA KD baselines.\n* Although the method is described as “customized” and “coexisting” with other KD approaches, the paper does not clarify the practical complexity of integrating BCKD with diverse model architectures, which may limit generality.\n* The theoretical analysis is mentioned as a major strength, but the paper provides no intuition about the assumptions or scope of the theory and elaboration on the analysis, making it difficult to judge whether the guarantees meaningfully translate to real EDIP settings.\n* The experimental section is emphasized heavily, but the paper does not acknowledge potential failure cases or scenarios where BCKD might struggle, such as highly cluttered scenes, extremely fine-grained boundaries, or teacher–student capacity mismatches. There is no clear analysis and experiments on the recent model architectures such as Transformers.\n* The the majority of referenced papers are mainly from up until 2 years ago. (6 papers from 2024 and 2025 out of ~100 paper). This means 10% the paper is lacking sufficient and measurable comparison with the SoTA approaches and the claims are not supported by accurate evidence as there is a lack of certainty on the validity of the claim in the last two years. For instance, under 4.3 there is a sentence saying \"ur approach utilizes pixel-level relations solely during the training process, thereby avoiding the increase in model complexity and parameters in inference that is typically associated with current methods (Liu et al., 2019; Lin et al., 2023a).\" Similar cases highlights an urgent need to update the references and revise the text reflect the issue.", "claims_and_evidence_yn": "No", "claims_explanation": "There are some changes requires to strengthen the position of having clear convincing and accurate evidence. Despite extensive experimental evaluation, there are unaddressed issues remained. Upon follow up stages, this will be revisited.", "audience_yn": "Yes", "audience_explanation": "The evaluation results and analysis are very insightful. The discussions on the comparison of approaches are helpful.", "requested_changes": "Based on the aforementioned strengths and weaknesses, the requested changes are provided in the following:\n* the referenced papers in the related work section are mainly from pre-2023 while there is a gap to the time of review. Please look into recent related works from 2024 and 2025.\n\n* Extensive and detailed elaboration on the definition of the Πi,j, a set of pixel items on the line between Tci and Tcj are needed. \n\n* in Section 4.1 the question is that if the concatenation is across the channel dimension of the feature tensors?\n\n* In Fig 2, how are the weights of the convolution layers in gray are determined?\n\n* In Eq. 2 B needs to be elaborated further. It is not clear how the 0,1 values are determined by the B function.\n\n* Furthermore, it is not clear from the context if the proposed boundary distillation approach uses Ground Truth to determine the semantic boundaries? from Eq. 2, it is not clear if GT is used or not. If not, does this rely on the teacher model capabilities to accurately determine such doundaries?\n\n* Computational complexity analysis of the two proposed methods to help KD is missing. It is instructive to look at the KD gain compared to the computational overhead imposed by the proposed approach. It is mentioned the complexity of the inference remains the same which is correct but not related as this paper is on the KD, the complexity of the idea needs to be compared with the baselines and the amount of gains gets normalized with respect to the comoutational overheald imposed.\n\n* In Eq , what is L_{SS}? Is it applied to the teacher or student or both? What is the GT label in Fig. 2 and what does it mean there?\n\n* Can the authors elaborate on if the proposed appraoch is backbone specific or agnostic?", "broader_impact_concerns": "There are no \"Broader Impact Concerns\" for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5vE3z86CgV", "reviewer": "Reviewer_Jp23", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Boundary and Context Knowledge Distillation (BCKD), a tailored strategy for training Efficient Dense Image Prediction (EDIP) models, specifically for deployment on resource-constrained AI accelerator chips. The core innovation addresses two systematic failure modes observed in compact student models: boundary region incompleteness and target region connectivity failure.\nBCKD achieves this via two main, efficient loss components: Boundary Distillation LBD), which extracts explicit object-level boundaries from hierarchical features without needing pre-extracted ground-truth boundaries, and Context Distillation LCD), which transfers implicit pixel-level self-relations to ensure smooth and connected target regions. The overall objective uses a time-dependent weight-decay strategy to enhance stability during training.\n\nKey Strengths:\n\n•\tTargeted and Practical KD Strategy: BCKD is a customized and targeted KD scheme that directly and effectively addresses the persistent geometric failure modes (boundaries and connectivity) specific to dense image prediction tasks.\n\n•\tZero Inference Overhead: Crucially for edge deployment, the method does not introduce any increase in parameters (Params.) or FLOPs during the inference stage, retaining the student model's high efficiency.\n\n•\tRobust Empirical Evidence and Statistical Validity (Updated): The method achieves superior accuracy compared to SOTA KD techniques. Following the review, the authors have strengthened this by reporting standard deviations from three independent runs, providing the statistical confidence required by TMLR.\n\n•\tValidated Geometric Quality (Added): The use of specialized metrics like Local Hausdorff Distance (LHD) and Manifold Stability (MFS) rigorously substantiates boundary alignment and feature structure preservation.\n\n•\tPre-extraction Free Boundary Generation: The boundary distillation component automatically generates semantic boundaries from features, eliminating the need for pre-extracted ground-truth boundary masks required by contemporary boundary-aware methods.\n\nKey Weaknesses:\n\n•\tTraining-Phase Computational Demands (Updated): While the authors have now provided Peak GPU Memory metrics to demonstrate practical feasibility, the Context Distillation loss (LCD) involves generating self-relation matrices during training. This represents a higher training-phase cost compared to simpler layer-to-layer distillation methods.\n\n•\tEmpirical Hyperparameter Complexity (Updated): Although the authors provided a systematic sensitivity study justifying their default weights (α=10, β=50), the reliance on a time-dependent weight-decay strategy adds a layer of tuning complexity to ensure convergence.\n\n•\tHardware Generalization Evidence (Updated): While validated on unified GPUs, the authors clarified that they used standard proxies (Params/FLOPs) because actual performance on specific AI accelerators was not empirically measured due to hardware limitations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the core claims are now strongly supported by convincing evidence and updated quantitative substantiation.\n\n•\tTargeted Effectiveness: The fundamental mechanism claims—mitigating boundary incompleteness and regional connectivity failure—are validated through a clear ablation study (Table 1), showing significant, non-conflicting improvements when LBD and LCD are combined (e.g., +3.20% mIoU over baseline).\n\n•\tGeometric Quality Metrics: The quantitative results on LHD (superior boundary alignment) and MFS (superior feature manifold preservation) provide robust, specialized evidence that directly supports the design philosophy of BCKD.\n\n•\tStatistical Robustness (Updated): In response to review concerns, the authors provided mean +/- standard deviation over multiple runs, substantiating the statistical confidence of reported gains.\n\n•\tTraining Practicality (Updated): The authors addressed concerns regarding the complexity of the LCD module by providing Peak GPU Memory metrics, confirming the method's feasibility during training.\n\n•\tInference Efficiency: The most critical application-oriented claim—zero increase in parameters or FLOPs at inference—is substantiated by the core KD methodology and the explicit use of the compact student model post-training.\n\nIn summary, the technical claims are exceptionally well-supported by evidence, metrics, and theory.", "audience_yn": "Yes", "audience_explanation": "The findings would be of significant interest to multiple segments of the TMLR audience.\n\n•\tEfficiency and Hardware Researchers: This work directly tackles the critical, real-world deployment challenge of achieving accuracy-preserving performance on resource-constrained edge computing devices and specialized AI chips. The ability to effectively compress high-accuracy dense prediction models is highly relevant to this audience.\n\n•\tKnowledge Distillation (KD) Methodologists: The paper introduces a principled shift from general to task-customized KD. Formulating knowledge transfer around fundamental geometric properties (boundaries and context) is a significant methodological advance in the KD literature, which is shown to be effective even when combined with existing SOTA KD methods.\n\n•\tComputer Vision Practitioners and Theorists: The focus on resolving geometric quality issues (boundary delineation and regional connectivity) in fundamental CV tasks (SSeg, ISeg, ODet) is valuable for improving prediction robustness. Furthermore, the rigorous theoretical analysis (Jacobian alignment, spectral convergence) in the Appendix and the introduction of specialized metrics (MFS, LHD) provide valuable tools and insights for researchers interested in the geometric properties of deep learning models.", "requested_changes": "The authors have successfully completed the following critical and strengthening adjustments:\n\n•\tQuantify Training Efficiency (Resolved): Authors reported Peak GPU Memory consumption, validating the efficiency claim despite the theoretical complexity of the context module.\n\n•\tValidate Device-Level Efficacy (Addressed): The authors clarified that Params/FLOPs are standard proxies and systematically reported Peak GPU memory as a bridge between theoretical complexity and hardware practicality.\n\n•\tEnsure Statistical Robustness (Resolved): Accuracy results in primary tables were re-reported as mean +/- standard deviation over three random seeds.\n\n•\tEnhance Reproducibility (Resolved): Authors clarified the mathematical formulation for the boundary operator, the projection head, and the temperature scaling schedule.\n\n•\tHyperparameter Sensitivity (Resolved): A systematic sensitivity analysis for alpha, beta, and tau was provided, justifying the default weights.\n\n•\tEmerging Architectures (Addressed): The manuscript now includes a discussion in the \"Future Work\" section regarding the theoretical compatibility of BCKD with Vision Mamba (Vim).", "broader_impact_concerns": "The authors have integrated a Broader Impact Statement addressing the following ethical implications:\n\n•\tHigh-Risk Deployment: Acknowledges potential misuse in mass surveillance and security, emphasizing ethical compliance and privacy safeguards.\n\n•\tBias Transmission: Addresses the risk of transferring biases from Teacher to Student models and recommends fairness-aware training.\n\n•\tEnvironmental Impact (Green AI): Discusses the energy footprint of the 40,000-epoch training regime and suggests future mitigation strategies.", "additional_comments": "This is an excellent, high-quality technical submission that addresses a fundamental and relevant problem in machine learning and computer vision deployment. The BCKD method is principled, effective, and rigorously validated.\n\n•\tKey Strengths: The combination of an intuitive, problem-specific KD solution (LBD and LCD) with a strong theoretical foundation (Appendix A, featuring Jacobian alignment and spectral convergence) is a major strength. The comprehensive experimental section, especially the use of LHD and MFS, provides highly credible evidence for the geometric improvements claimed.\n•\tPractical Insights: The visualization showing that BCKD helps the student model avoid \"hallucinations\" (e.g., correctly recognizing a camera instead of a bottle) is a compelling practical argument for the benefits of task-specific distillation.\n\nThe authors were highly responsive to feedback regarding statistical robustness and training cost analysis, providing the necessary transparency to validate their claims. While training-phase overhead is non-trivial, it is a justifiable trade-off for the gains in boundary precision and regional connectivity shown across semantic segmentation, instance segmentation, and object detection. The inclusion of a comprehensive Broader Impact Statement and a forward-looking discussion on Vision Mamba ensure the work meets the rigorous standards of TMLR.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)