Live On the Hump: Self Knowledge Distillation via Virtual Teacher-Students Mutual Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 17:00:58

Abstract: For solving the limitations of the current self knowledge distillation including never fully utilizing the knowledge of shallow exits and neglecting the impact of auxiliary exits' structure on the performance of network, a novel self knowledge distillation framework via virtual teacher-students mutual learning named LOTH is proposed in this paper. A knowledgeable virtual teacher is constructed from the rich feature maps of each exit to help the learning of each exit. Meanwhile, the logit knowledges of each exit are incorporated to guide the learning of the virtual teacher. They learn mutually through the well-designed loss in LOTH. Moreover, two kinds of auxiliary building
blocks are designed to balance the efficiency and effectiveness of network. Extensive experiments with diverse backbones on CIFAR-100 and Tiny-ImageNet validate the effectiveness of LOTH, which realizes superior performance with less resource by the comparison with the state-of-the-art distillation methods. The code of LOTH is available on Github.

Main paper: 
 introduction: 
 In recent years, Convolutional Neural Networks (CNNs) have attracted considerable attention for their excellent performance in various computer vision tasks. However, remarkable performance of CNNs [23, 25] typically suffer from exorbitant computational resources and memory overheads, which makes it difficult to be applied in edge devices with limited resources. Knowledge Distillation (KD), as one of the most effective network compression techniques [5, 9, 18] , aims to reduce the size of CNNs without changing its structure and ensure the feature representation capability of CNNs at the same time.Traditional KD methods [6, 10, 12, 20, 21, 32] adopt two-stage offline training strategy, in which a high-capacity teacher network is pre-trained and then performs a one-way knowledge transfer to low-capacity student network. Obviously, the performance of student network depends heavily on the fixed knowledge extracted by the static teacher network. However, sophisticated teacher networks are not always readily available. Specifically, networks with high accuracy are not necessarily capable of high-quality knowledge transfer [11] . Moreover, the two-phase training process is cumbersome, which leads to enormous computational costs and resource burdens. Owing to these limitations, end-to-end mutual learning strategies [3, 7, 14, 24, 27, 31] are in the spotlight. There is no explicit teacher-student role in this strategy, which allows multiple networks to be co-trained in a peer-teaching manner. Even though dynamic interactions between networks contribute to reducing semantic gaps, such aimless mutual transfer exists the risk of performance degradation when the capacity gaps of the two networks differ significantly. In addition, the cost of training multiple networks simultaneously remains prohibitive.Self knowledge distillation (SKD) breaks the dilemmas, which extracts and learns knowledge from its own backbone. One kind of SKD shares the same shallow backbone to construct multiple peer auxiliary exits in the last stage of the backbone, which integrates logits of multi-exits as a teacher. However, multiple exits in the same location typically tend to cause homogenization [16] , hurting the generalization ability of networks. Another kind of SKD adds auxiliary exits hierarchically in a deep supervised manner. It takes the deepest knowledge as supervised signal to guide the learning of shallow exits. Although it is remarkable both in the aspect of keeping the ability of backbones and reducing the training costs, it still suffers from two inherent limitations: 1) the complementary knowledge of shallow exits is never fully utilized, which decreases the effectiveness of distillation; 2) the design of auxiliary exits normally employs heuristic principles with a single and fixed architecture on all networks, which fails to strike a balance between effectiveness and efficiency.To tackle these problems, a novel self knowledge distillation framework via virtual teacher-students mutual learning is proposed in this paper. Since the virtual teacher and the students are generated from the backbone itself, and they learn from each other, so we name the proposed self knowledge distillation as LOTH (Live on the hump). Each exit in a network produces a response as a logit, which is though as a student. The feature maps from all the exits are used to build a virtual teacher, which concentrates all the important knowledge of all the exits in a network. Specifically, the feature maps of all exits perform cross-channel information interaction in both global and local manners. It captures finer-grained identifiable information to adaptively enhance the fusion representation, giving us a knowledgeable virtual teacher. In order to learn more meaningful knowledge, a mutual learning between the virtual teacher and multi-exits of the backbone is designed according to the classification loss and distillation loss. In addition, for balancing the efficiency and effectiveness, two kinds of bottleneck building blocks are designed to capture the hidden information of the backbone. Extensive experiments with diverse backbones on CIFAR-100 and Tiny-ImageNet datasets show that LOTH improves the top-1 accuracy of classification by 3.81% to 6.58% with less resources compared with diverse backbones, and the top-1 accuracy achieved by LOTH is 3.81% to 6.58% higher than the state-of-the-art distillation methods with equivalent amount of parameters. In summary, the main contributions of this paper are as follows:• A novel self knowledge distillation framework named LOTH is proposed, which views each exit as a student and fuses knowledge from auxiliary exits to build a knowledgeable virtual teacher. The bidirectional mutual learning between the virtual teacher and students contributes significantly to the capabilities of multi-exits learning and significant gains with few training overhead. • An efficient adaptive feature fusion is designed to adequately extracts important knowledge from each exit, which enhances the feature maps by channel and spatial attention, resulting in a powerful and knowledgeable virtual teacher. • The impact of different scales of auxiliary exits' structures are explored on the model's performance, customizing two well-designed auxiliary blocks to balance the efficiency and effectiveness based on the size of the network. 
related works 2.1 knowledge distillation: 
 In the knowledge distillation framework, there are always other networks to provide addition supervised signals. The vanilla KD pioneered by [6] utilizes the fixed soft logit knowledge of teacher network to guide the learning of the student network. To promote the efficiency of logit knowledge, DKD [32] thoroughly explores the logit distillation and decouples it into target class knowledge distillation and non-target class knowledge distillation. Moreover, PESF-KD [21] further reduces time-consuming pre-trained teacher network and enables adaptive knowledge transfer by tuning the soft logit of the teacher network. In addition to these explorations of logit knowledge described above, there are also efforts to captures intermediate information of teacher networks. AT [12] narrows the distance gap of attention maps between teacher and student networks. RKD [20] facilitates knowledge transfer of structural relationships between sample instances through distance-wise and angle-wise losses. Besides, DIST [10] captures the intrinsic interclass relations of teacher to enrich correlation-based knowledge. However, all the above approaches adopt a two-stage training strategy, where high-quality teacher networks are pre-trained to provide priori knowledge for the low-capacity student network to emulate, which is tedious and complicated.To enhance the efficiency of KD, numerous works start to focus on online mutual learning of multiple networks, which simplifies the training process to one-stage. DML [31] encourages each peer network to learn from each other through logit distillation. DCCL [24] additionally introduces intermediate cross-layer feature supervision to facilitate dynamic interactions between networks. MCL [27] enables multiple networks to conduct mutual contrastive learning to achieve interaction and transfer of contrastive distributions. Aside from these inter-network transfer methods, there exist ensemble strategies to generate an intermediary for collaborative learning, which in turn guides the learning of each network. KDCL [3] ensemble soft logits from multiple networks for high-quality supervision transferred to each peer networks. While DualNet [7] sums the feature maps of all peer networks to obtain a robust teacher classifier. Moreover, EML [14] provides a lightweight adaptive feature fusion module that concatenates features from all sub-networks.Instead of relying on extra networks, self Knowledge distillation extracts knowledge from its own backbone in the form of multiple auxiliary exits. ONE [33] adopts simple gating mechanisms to dynamically integrate the logits of multiple peer exits, where each exit is treated as a student and the fused logit acts as a teacher. OKD-Dip [1] adopts self-attention mechanisms to implement two-level distillation with multiple peers exits and a group leader. Moreover, Zhu et al. [34] further extend OKDDip, which utilizes a concise feature learning to provide precise weights for peer exits. AHBF-OKD [2] constructs diverse peers to promote knowledge diversity and employs hierarchical fusion to learn complementary knowledge. All of these approaches described above share the same shallow backbone to construct multiple peer exits, which suffers from a fatal homogeneity defect.To further improve the efficiency of self Knowledge distillation, substantial works pay more attention to the strategy of hierarchical shared shallow backbone. BYOT [29] adds multiple auxiliary exits hierarchically to capture the hidden knowledge. ECSD [28] proposes additional attention modules at the connection between the backbone and auxiliary exits to capture richer features. However, these approaches all view the deepest layer of the backbone as teacher to supervise the learning of shallow exits, which neglects the exploitation of complementary knowledge provided by shallow exits. To this end, BEED [13] proposes an artificially regulated ensemble strategy to merge the knowledge of all exits to instruct each exits, which is mechanical and inconvenient for different network architectures and datasets. KFD [15] adopts an adaptive fusion strategy to integrate feature maps from earlier backbones, which possesses relative weaker semantic information than exit. Moreover, DTSKD [17] additionally considers the past learning history to further enrich the form of the supervised signal. In contrast to these approaches, our proposed LOTH adequately integrates complementary feature and logit knowledge from early exits to facilitate mutual learning across multi-exits, which possesses excellent generalization and robustness. 
method 3.1 problem formulation: 
 Suppose a convolutional neural network has 𝑘 stages. Each stage is composed of multiple specific building blocks that can be thought as feature extractor, denoted as G 1 , G 2 , ...G 𝑘 . Given 𝑁 training samples X = {𝑥 𝑛 } 𝑁 𝑛=1 , the corresponding ground-truth labels with 𝑀 classes are denoted as Y = 𝑦 𝑛 ∈ R 𝑀 𝑁 𝑛=1 . The purpose is to train such 𝑘-stage network by using these 𝑁 samples to get a same structure network with more robust classification ability and fewer parameters. Fig. 1 shows the whole framework of LOTH. Take the sample 𝑥 𝑛 as an example, it is input into the backbone network, 𝑘 logits are generated from the 𝑘 -1 auxiliary exits and one main backbone exit. At the same time, a virtual teacher is built based on these 𝑘 exits, bringing in 𝑧 ′ 𝑛 . Then 𝑧 ′ 𝑛 and these 𝑘 logits learn the knowledge of backbone mutually according to the loss L 𝑠→𝑡 and the loss L 𝑡 →𝑠 .For the network with 𝑘 stages, 𝑘 -1 auxiliary exits are added at early stages. Let's use E 1 to denote the auxiliary exit in the first stage, similarly, there are E 2 ...E 𝑘 for other stages, where E 𝑘 is the default exit of backbone. For the sample 𝑥 𝑛 , it will pass through feature extractors in the backbone network and the corresponding exits to generate logits 𝑧 1 𝑛 , 𝑧 2 𝑛 , ..., 𝑧 𝑘 𝑛 . Such process can be simply formed as:EQUATIONThe auxiliary exits E 𝑖 𝑘 -1 𝑖=1 play the role of providing additional training objectives in LOTH. Each auxiliary exit comprises two parts: a feature transformation and a classifier. The classifier provides the foundation for probabilistic prediction, which is implemented by a fixed pooling layer and a fully connected (FC) layer. The feature transformation composes of multiple building blocks with downsampling, which ensures that the sizes of output features from the shallow auxiliary exits are identical to the 𝑘-th stage of the backbone. Obviously, the feature transformation dominates the auxiliary exits. Therefore, the design of auxiliary building blocks is extremely crucial.Previous researches [13, 29] have demonstrated that Bottleneck block originated from ResNet [4] is an effective architecture to improve the performance of exits. However, the single employment of a fixed architecture ignores the impact of the relative size of the auxiliary exits on the performance of the backbone network, which is not optimal across all networks.Hence, two kinds of auxiliary building blocks are designed at here. One is Bottleneck, and the other is Group-Bottleneck. They are tailored according to the scale of the backbone network. As illustrated in Fig 2(a) , the channel expansion and residual connection are removed from the standard Bottleneck [4] , which retains the powerful extraction capabilities and reduces computational burden. However, Bottleneck drastically enlarges the scale of auxiliary exits in the lightweight networks, slowing the training speed. Group-Bottleneck designed in Fig. 2 (b), uses a group convolution (depth-wise convolution) rather than a regular convolution in the middle layer of the Bottleneck block, significantly reducing the scale of auxiliary exits. One thing should be pointed out is that the number of auxiliary building blocks in each auxiliary exit depends on the stage of the backbone network. Taking the 𝑘 = 3 as an example, the number of building blocks in G 1 is 2, the number of building blocks in G 2 is 1, and the number of building blocks in G 3 is 0, i.e., the default backbone exit.The auxiliary building blocks in each auxiliary exit always generate rich semantic features, which should be employed to help exits learn the knowledge of backbone network.Let 𝑓 𝑖 𝑛 ∈ R 𝐻 ×𝑊 ×𝐶 (𝑖 = 1, 2, . . . 𝑘) denote the features generated from E 𝑖 , where 𝐶 means the number of channels and 𝐻 × 𝑊 tells us the size of feature maps. By simply concatenating 𝑓 𝑖 𝑛 along the channel dimension, 𝑓 𝑐𝑎𝑡 ∈ R 𝐻 ×𝑊 ×𝐶 ′ is obtained, where 𝐶 ′ = 𝑘𝐶.Subsequently, 𝑓 𝑐𝑎𝑡 is transformed along two parallel paths simultaneously. On one path, 𝑓 𝑐𝑎𝑡 is globally transformed based on channels, giving the global feature context 𝑓 𝑔 that tends to emphasize large objects with coarse granularity. 𝑓 𝑔 is calculated by:EQUATIONwhere C 1𝐷 indicates 1D-Convolution, and GAP stands for global average pooling that captures the global distribution of channel dimensions.On the other path, 𝑓 𝑐𝑎𝑡 is locally transformed, giving the local feature context 𝑓 𝑙 that focuses on subtle details of small object. 𝑓 𝑙 is calculated by:EQUATIONwhere R denotes the reshape transformation. There are a total of two transformations, where the first reshape transforms the input feature map of size 𝐻 × 𝑊 into a scalar for 1D-convolution in the channel direction, and the second reshape carries out a restoration. Note that the local feature context has the same size with the input, which can be viewed as a fine-scale perception of spatial location information. Moreover, the employment of 1D-Convolution facilitates information interaction across channels and ensures low complexity.As for the kernel size K of 1D convolution, we adopt an adaptive strategy in [26] to determine:EQUATIONwhere |𝑄 | 𝑜𝑑𝑑 represents the nearest odd number of 𝑄, 𝛾 and 𝑏 are constant. In this work, we set 𝛾 = 2 and 𝑏 = 1 for all experiments.After obtaining the global feature context and local feature context, f 𝑐𝑎𝑡 is enhanced by:𝑓 ′ 𝑐𝑎𝑡 = 𝑓 𝑐𝑎𝑡 ⊗ 𝜓 (𝑓 𝑔 ⊕ 𝑓 𝑙 ), 𝑓 ′ 𝑐𝑎𝑡 ∈ R 𝐻 ×𝑊 ×𝐶 ′ (5)where 𝜓 (•) indicates Sigmoid function to generate the final weight map, ⊕ and ⊗ denote the broadcasting addition and element-wise multiplication, respectively. Moreover, a lightweight point-wise convolution F 𝑝𝑤 (•) is utilized to further facilitate feature fusion of multi-exits and reduce the channel dimension. Eventually, the fused representation 𝑓 ′ with strong discrimination and generalization capabilities is obtained, and then fed into classifier for the overall classification, which plays the role of a virtual teacher to help exits learn knowledge of backbone network.Now, from 𝑘 exits, 𝑘 logits have been generated. And a virtual teacher 𝑧 ′ 𝑛 has been created from the advanced feature maps of all auxiliary exits, which can be treated as additional supervised signals to instruct the learning of each exit. The training loss L 𝑠→𝑡 of auxiliary exits is expressed as:EQUATIONwhere L 𝑐 denotes the fundamental classification loss, which is defined as in terms of the Cross-Entropy (CE) between the probability distribution and the ground-truth label of the samples:EQUATIONwhere 𝜎 (•) is the softmax function. The distillation loss L 𝑑 employs the Kullback-Leibler (KL) divergence to measure the soft distribution difference between the target and sample:EQUATIONwhere 𝜏 is the temperature coefficient. As 𝜏 increases, the probability distribution gets smoother. Considering that early exits of the network typically have relatively weak extraction capabilities, it is difficult to capture sufficient information directly from low entropy ground-truth labels. Therefore, inspired by [13] , we further control the proportion of classification loss L 𝑐 and distillation loss L 𝑑 in Eq 6 to enable early exits to learn more subtle knowledge from the soften distillation distribution, which can be reformulated as:EQUATIONwhere the balance factor 𝛼 satisfies 𝛼 > 1 and 𝛼 𝑘+1-𝑖 < 2, which guarantees the early exits to be always assigned to more distillation weights. To make full use of logit knowledges, we further incorporate the logits at each exit to construct a powerful ensemble logit z𝑛 :EQUATIONwhich is utilized to guide the learning of the virtual teacher:EQUATIONBidirectional knowledge communication between the virtual teacher and auxiliary exits facilitates the learning across exits. In the end, the total training objective can be obtained by summing the training loss of L 𝑠→𝑡 and L 𝑡 →𝑠 : In the actual evaluation process, the images in Tiny-ImageNet are resized to be the same as CIFAR-100. Moreover, data enhancement with horizontal flipping and random cropping are performed on the training samples of both datasets.EQUATION 
backbones.: 
 To validate the effectiveness of the proposed LOTH, multiple backbone networks are adopted. They are popular architectures ResNet [4] and VGG [23] , and lightweight networks MobileNetV1 [8] , MobileNetV2 [22] , ShuffleNetV1 [30] , and Shuf-fleNetV2 [19] . Given the low image resolution of both datasets, these backbone network architectures are slightly modified by dropping the first pooling layer and shrinking the stride and kernel size of the convolutional layer.All the networks are implemented by Pytorch on GeForce RTX2080Ti GPU with 11 GB memory. The stochastic gradient descent (SGD) optimizer with momentum of 0.9 and weight decay of 5e-4 are utilized to optimize networks for 200 epochs. We set the initial learning rate to 0.1 for both datasets, which is divided by 10 at 75th, 130th and 180th epochs. We set the mini-batch size to 128, the temperature coefficient 𝜏 to 3, and the balancing factor 𝛼 to 1.15 by default during all training procedures.The experimental results of LOTH on CIFAR-100 and Tiny-ImageNet in comparison with various backbones are presented in Table 1 and Table 2 , respectively, where the adapted Bottleneck (Bn) block is utilized for both regular-sized ResNet and VGG architectures, and the Group-Bottleneck (G-Bn) block is adopted for four lightweight networks. The last columns 'Fusion' shows the top-1 accuracy obtained by the virtual teacher in LOTH. These two tables show that:(1) Compared to the standard training, the whole performance of all backbone networks benefits tremendously from LOTH. Specifically, with the same capacity, our approach surpasses the baselines by margins of 2.72% to 5.62% on CIFAR-100 and profound improvements of 3.81% to 6.58% on Tiny-ImageNet. (2) LOTH exceeds the performance of almost all baselines at the shallowest exit. Importantly, the shallowest exit has extremely low parameter advantages in some regular sized networks. Taking VGG19 at exit1 as example, LOTH achieves 2.39% and 6.91% gains on CIFAR-100 and Tiny-ImageNet with barely 1/3 of the backbone parameters, respectively.(3) The performance of each exit is not always increase with depth, and the optimal results are concentrated in the third and fourth exits. However, this is not a defect, but rather an advantage for self knowledge distillation. Owing to the hierarchical shared backbone, we can adaptively choose classifiers of different depths to trade-off dynamic accuracy and efficiency demands in real-world deployments. ( 4) On all backbone networks, the performance of the virtual teacher far exceeds that of other exits, indicating that the virtual teacher extracts richer and more meaningful semantic information. Therefore, it is reasonable to be taken as a virtual teacher. All these commonalities across different datasets and backbones confirm the strong robustness and generalization of LOTH.To validate the effectiveness of LOTH, a series of advanced online distillation methods are compared, which is presented in Table 3 .The online distillation methods are further categorized into three strategies: (1) Multi-networks mutual learning: DML [31] , EML [14] , and DCCL [24] ; (2) SKD with peer exits: ONE [33] , OKDDip [1] and AHBF-OKD [2] ; (3) SKD with hierarchical exits: BYOT [29] , ECSD [28] , BEED [13] , KFD [15] , and DTSKD [17] . To ensure fairness and consistency, we take the average of the two sub-networks as the result of multi-networks mutual learning, while for SKD, we take the accuracy of the main backbone network. It is observed that our LOTH achieves the optimal accuracy than all other online approaches in both backbones. Specifically, LOTH realizes gains of 0.58% to 2.56%, 0.22% to 2.58% on ResNet18 and VGG16, respectively. Even compared with SKD with equivalent architectures, LOTH still achieves an average of 1.48% and 1.46% improvement on ResNet18 and VGG16, respectively.In addition, we additionally select several SKD methods with equivalent architecture to LOTH for further comparing the top-1 accuracy and the amount of parameters on Tiny-ImageNet. From Table 4 , we can observe that (1) In the case of a large gaps of parameters, the ECSD with average ensemble strategy of multiexits logits outperforms the BYOT which takes the deepest exit as the supervised signal. This suggests that using only the deepest knowledge as a guide limits the learning ability of the model. (2) LOTH at the Exit1 with 1/3 of backbone parameters outperforms BYOT and ECSD at the Exit4, which demonstrates the importance of auxiliary building block structure. (3) Compared with BEED having the same parameters, LOTH still realizes a gain of 0.22% to 1.00% in multiple exits, which indicates the effectiveness of the virtual teacher-students mutual learning in LOTH.Figure 4 shows the performance of LOTH under different auxiliary building blocks on: regular-size networks (ResNet, VGG16) and lightweight networks (MobileNetV1, ShuffleNetV1), where Bn, G-Bn, and DSC stand for Bottleneck, Group-Bottleneck and Depth Separable Convolution utilized in ECSD [28] , respectively. It can be observed: (1) With the similar lightweight design, our G-Bn block adds few training overheads yet achieves a significant benefit on all exits than DSC. (2) It is reasonable to treat the feature fusion as a teacher to instruct each exit. This is mainly reflected in the fact that the performance of fusion far exceeds the other exits under all three construction blocks, especially on DSC, whose performance varies greatly across exits. (3) The use of Bn with standard convolution achieves optimal main backbone performance at both regular-size networks, but is equal to or even lower than G-Bn with lightweight convolution on the two lightweight networks. More importantly, using BN to construct auxiliary exits on shuffleNetV1 imposes 34× training overhead, which slows down training significantly. Therefore, it is necessary to design two well-designed auxiliary blocks to balance the effectiveness and efficiency according to the scale of backbones.To validate the effectiveness of the feature fusion in the step of virtual teacher generation, we further compare it on Tiny-ImageNet with three prevailing feature fusion strategies: Summation [7] , Concatenation [14] , and Sample-based Attention mechanisms (shorted as Att-Sample) [15] . From Table 5 , we can note that the simple summation gets the worst performance, with a significant gap of 1.97% to the penultimate performance, which indirectly indicates the existence of tremendous semantic gaps between multi-exits. Moreover, our fusion results exceeds the sub-optimal performance by 0.51 among numerous advanced fusion strategies, which indicates our fusion possesses a stronger feature extraction capability to The virtual teacher-students mutual learning plays the role of facilitating information interaction between fusion and multi-exits in LOTH.To explore the effectiveness of each component of this learning, as shown in Table 7 , we perform ablation comparisons with the following three cases: (1) w/o L 𝑑 : Only the fundamental classification loss L 𝑐 in L 𝑠→𝑡 and L 𝑡 →𝑠 is retained. Observing the first line of VGG16 and MobileNetV1, we can discover that even without distillation loss involved, our approach still achieves a remarkable gain of 2.23 and 1.56 compared to the baselines, respectively, which demonstrates the effectiveness of the hierarchical multi-exits architecture.(2) w/o L 𝑡 →𝑠 : Only logit knowledge of each exit is utilized to obtain the ensemble logit z𝑛 , which served as supervised signals to guide multi-exits learning. It is not difficult to observe that without the involvement of feature map knowledge, there is a significant performance degradation in two backbones, which further validates the importance of virtual teacher. (3) w/o 𝛼: Both the classification loss and distillation loss at each exit are signed the same weight of 1. In comparison with the full-fledged LOTH, we note that the presence of the balance factor 𝛼 is beneficial for network performance, especially for early exit. Take MobileNetV1 as an example, the performance of Exit1 gains 0.99 with the balancing factor 𝛼, which dramatically facilitates the learning ability of early exits with the low-capacity. In short, the mutual interplay between these components makes LOTH what it is.  
conclusion: 
 In this paper, a novel self knowledge distillation framework via virtual teacher-students mutual learning called LOTH has been proposed, which focuses on fully exploiting the complementary knowledge of early exits to further enhance the effectiveness of distillation. Extensive experimental evaluations on CIFAR and Tiny-ImageNet have demonstrated that LOTH enables joint performance improvement of multi-exits. In addition, the impact of different sizes of auxiliary exits on model performance is sufficiently explored in this paper, and the two well-designed building blocks are verified to have the ability of balancing effectiveness and efficiency in self knowledge distillation. However, in this research, only logits are used to learn the knowledge of network, so the classification accuracy is still far from the requirements of applications. In the future, the incorporation of other types of knowledge like relations among objects will be researched to improve the ability of feature representation of network. 
 
 Appendix:  


Decision: This paper proposes Live On the Hump (LOTH), a self knowledge distillation method with virtual teacher and mutual learning for knowledge distillation within a network. Specifically, multiple exits are utilized to generate a virtual teacher, along with other modules such as bidirectional mutual learning, adaptive feature fusion, and different scales of auxiliary exits. Experiments are conducted on CIFAR100 and Tiny-ImageNet to verify the effectiveness.   

The initial ratings are Weak Reject, Borderline Reject, Borderline Accept, and Borderline Accept. The common concerns are the novelty, lack of larger dataset and larger network for evaluation. In addition, T9ji challenged the suitability for MM submission.   

The authors provide a solid rebuttal to address the concerns and suitability issue. After rebuttal, two reviewers acknowledge the authors efforts to address the concerns of lacking larger networks/datasets, and the suitability is also modified. Therefore, the two reviewers raised their ratings to Borderline Accept. Other two reviewers remain their rating. So finally, all scores are Borderline Accept. 

After reading the paper, rebuttal, and all comments, I agree with the reviewers that this paper should be accepted.   
Therefore, I recommend to Accept this paper as Poster.

Reviews: review: paper_summary: paper_summary: This paper proposes a self knowledge distillation framework named LOTH is proposed, which views each exit as a student and fuses knowledge from auxiliary exits to build a knowledgeable virtual teacher. The bidirectional mutual learning between the virtual teacher and students contributes significantly to the capabilities of multi-exits learning。This paper designed an efficient adaptive feature fusion to adequately extracts important knowledge from each exit, which enhances the feature maps by channel and spatial attention,resulting in a powerful and knowledgeable virtual teacher.
strength_weakness: strength_weakness: 1. The sentence structure of the paper is fluent and there are no obvious grammar or spelling errors
2. The paper conducted comparative experiments and ablation experiments, and the results are supported.
3.The paper provides a new approach to virtual teacher-student mutual learning through distillation networks, which is innovative.
limitations: limitations: 1. The article lacks an explanation of the specific process of generating a virtual teacher in Figure 3. Please add more information.
2. Please provide a detailed explanation of whether the training and testing samples for CIFAR-100 in the 4.1.1 dataset were randomly selected or manually screened.
3. In the introduction, it is mentioned that LOTH can achieve higher accuracy with fewer resources, but this conclusion was not reflected in subsequent experiments.
4. The datasets used by the author are rather limited, especially lacking large-scale datasets such as comparisons on ImageNet, which results in a lack of reliability in the findings of this study.
5. Similarly, the author did not employ larger or newer models to validate the algorithm's performance. While this may be due to hardware limitations, it also somewhat constrains the credibility of this study.
6. The author needs to provide a more detailed description of the motivation and algorithm of this study to enhance its reliability.
paper_summary: paper_summary: The paper is about a novel self-knowledge distillation framework named "Live On the Hump" (LOTH), which is designed to address limitations in current self-knowledge distillation methods. These limitations include not fully utilizing the knowledge from shallow exits and neglecting the impact of auxiliary exits' structure on network performance. LOTH introduces a virtual teacher-student mutual learning approach to enhance the learning process.

The authors propose the LOTH framework that constructs a knowledgeable virtual teacher from the rich feature maps of each exit to aid in the learning of each exit. The framework incorporates the logit knowledge of each exit to guide the learning of the virtual teacher, promoting mutual learning through a well-designed loss function.
strength_weakness: strength_weakness: The design of two distinct auxiliary building blocks (Bottleneck and Group-Bottleneck) tailored for different network sizes is an effective way to balance efficiency and effectiveness. The theoretical foundation of mutual learning between a virtual teacher and multiple exits is sound and provides a robust framework for knowledge distillation. The method of feature fusion through global and local feature contexts is technically well-thought-out and aims to capture a comprehensive representation of the data.
limitations: limitations: Using shallow features as auxiliary labels in self-distillation isn't a novel concept. The experiments conducted on CIFAR-100 and Tiny-ImageNet are unconvincing, raising doubts about the model's scalability in terms of data and architecture. Additionally, it's unclear whether this model is applicable to transformers. Overall, the paper lacks richness in its content.
paper_summary: paper_summary: The authors introduce LOTH (Live On the Hump), a self-knowledge distillation framework that uses virtual teacher-student mutual learning to enhance the performance of convolutional neural networks (CNNs). LOTH integrates knowledge from multiple network exits especially the early exits to create a virtual teacher, and applies a bidirectional mutual learning method between the virtual teacher and students. Moreover, they customize two well-designed auxiliary blocks to balance the efficiency and effectiveness based on the size of the network, leading to improved efficiency and effectiveness, as demonstrated through experiments on CIFAR-100 and Tiny-ImageNet datasets.
strength_weakness: strength_weakness: 1.Proposes a novel self-knowledge distillation framework that leverages mutual learning between virtual teachers and network exits. 

2.Effectively balances efficiency and effectiveness with adaptive auxiliary building blocks. 

3.Provides a comprehensive evaluation against state-of-the-art distillation methods, showing superior performance.
limitations: limitations: 1.This paper relies heavily on empirical results without deeply exploring theoretical underpinnings. 

2.The integration of logits from multiple exits might introduce redundancy and computational overhead. 

3.Some comparisons with existing methods lack detailed discussion on the potential reasons for performance differences. 

4.The proposed method's applicability is only demonstrated on image classification task, many other tasks remain unexplored.
paper_summary: paper_summary: The paper introduces a novel framework for self-knowledge distillation called LOTH, which addresses inefficiencies in traditional methods by effectively utilizing knowledge from shallow exits and auxiliary structures. LOTH employs a virtual teacher, constructed from the network's feature maps at each exit, to guide the learning process through mutual interaction between exits and the teacher. This process is facilitated by a well-designed loss function that enhances both the teacher's and the student's learning outcomes. Experiments demonstrate that LOTH outperforms traditional methods, making it suitable for resource-limited environments.
strength_weakness: strength_weakness: 1. LOTH conducts extensive experiments and explorations across various datasets such as CIFAR-100 and Tiny-ImageNet, and on multiple network architectures, demonstrating its effectiveness and robustness. These experiments not only highlight LOTH's advantages in improving classification accuracy but also prove its potential for application on resource-constrained devices.
2. The proposed method is technically sound.
limitations: limitations: 1. The paper could benefit from a more detailed exploration of the challenges associated with current sub-distillation methods. The employment of intermediate layers and additional trainable parameters for enhancing the student model has been extensively explored. It would be helpful if the specific advancements or contributions of this study were more clearly outlined to distinguish it from existing literature.
2. "use $\mathcal{E}_1$ to denote the auxiliary exit in the first stage" is described in terms of the network architecture's different depths, rather than stages of the training process. Therefore, this description might introduce some ambiguity.
3. The effectiveness of the approach has not been validated on larger datasets or more complex network architectures.