CIFD: Controlled Information Flow to Enhance Knowledge Distillation

Authors: yashas  malur
saidutta  rakshith
s  srinivasa
jaejin  cho
ching-hua  lee
chouchang  yang
yilin  shen
hongxia  jin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:58:26

Abstract: Knowledge Distillation is the mechanism by which the insights gained from a larger teacher model are transferred to a smaller student model. However, the transfer suffers when the teacher model is significantly larger than the student. To overcome this, prior works have proposed training intermediately sized models, Teacher Assistants (TAs) to help the transfer process. However, training TAs is expensive, as training these models is a knowledge transfer task in itself. Further, these TAs are larger than the student model and training them especially in large data settings can be computationally intensive. In this paper, we propose a novel framework called Controlled Information Flow for Knowledge Distillation (CIFD) consisting of two components. First, we propose a significantly smaller alternatives to TAs, the Rate-Distortion Module (RDM) which uses the teacher's penultimate layer embedding and a information rate-constrained bottleneck layer to replace the Teacher Assistant model. RDMs are smaller and easier to train than TAs, especially in large data regimes, since they operate on the teacher embeddings and do not need to relearn low level input feature extractors. Also, by varying the information rate across the bottleneck, RDMs can replace TAs of different sizes. Secondly, we propose the use of Information Bottleneck Module in the student model, which is crucial for regularization in the presence of a large number of RDMs. We show comprehensive state-of-the-art results of the proposed method over large datasets like Imagenet. Further, we show the significant improvement in distilling CLIP like models over a huge 12M image-text dataset. It outperforms CLIP specialized distillation methods across five zero-shot classification datasets and two zero-shot image-text retrieval datasets.

Main paper: 
 introduction: 
 A decade ago Hinton et al. proposed a mechanism called Knowledge Distillation (KD), where the insights from a larger model (the teacher) are transferred to a smaller model (the student) [1] . The teacher model, on account of larger modeling capacity, is capable of learning complex relationships in the data and is thus better at performing its target task. KD attempts to help the training of the student model by showing it the insights learned by the teacher. Specifically, Hinton et al. suggested the transfer of "dark knowledge" in the logits of a teacher classifier to a student classifier by minimizing the Kullback-Leibler (KL) divergence between a softened version of the predicted probability distribution of a teacher and a student model. However, dark knowledge is only one way of quantifying the insights learned by the teacher. In general, how to quantify the insights and how to transfer them to the student model remain open questions.38th Conference on Neural Information Processing Systems (NeurIPS 2024). Recently many researchers in the knowledge distillation community have been concerned with the gap between the teacher and student's modeling capacities [2, 3, 4, 5] . Going back to the human learning experience, we learn simple concepts first, intermediate concepts next, and finally advanced concepts. Based on this insight, prior works have proposed the use of Teacher Assistants (TAs), models whose size is between that of the teacher and the student, to help facilitate the knowledge transfer better [2, 3] . By distilling knowledge from the teacher to the assistant and then using the assistant to help the distillation into the student led to large performance improvement. However, as seen in Figure 1 (b), these methods are around 3.5× and 4.5× more expensive than KD [1] . In this paper we introduce a method called Controlled Information Flow for Knowledge Distillation (CIFD) which is significantly cheaper to train. This brings the training cost of TA based methods closer to KD and other counterparts while obtaining state-of-the-art performance.The proposed CIFD mechanism consists of two parts. First, we propose to process the teacher's penultimate layer embedding by a Rate-Distortion Module (RDM) to replace TAs. The RDM imposes a constraint on the amount of information through it. To replace multiple TAs of different capacities, we propose using RDMs with different levels of information constraints. Since RDMs operate on the teacher embeddings, they do not have to relearn low level features and can avoid the associated training and inference costs. As far as we know, this is the first application of Shannon's Rate-Distortion theory to aid knowledge distillation. However, as the number of RDMs increases, the student model tends to overfit to the teacher and assistants' outputs, as pointed out in [3] (also confirmed empirically here), thus reducing the generalization performance of the student model. Thus, to regularize the training of the student model in the presence of multiple RDMs, we propose the Information Bottleneck Module (IBM). The following are the contributions of this paper:1. We propose the use of an RDM that takes the embeddings from a teacher model and mimics a teacher assistant. Since the RDM does not have to relearn low level features, it is two to three layers only; significantly smaller than a teacher assistant model. 2. We propose the use of the IBM in the student model during train-time. We find that IBM on its own provides benefits but is also a crucial regularizer as the number of RDMs increases.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] .Justification: Yes, the paper proposes a mechanism for Knowledge Distillation using Teacher Assistants, especially for large datasets. We show multiple experiments as supporting evidence for our proposed method.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Knowledge Distillation: Hinton et al. introduced the concept of transfering the knowledge from a large teacher model to a smaller student model using the logit space for classification models [1] .Many works have focused on different methods for transferring information like in the output space [6] , transferring information in the intermediate layers of the teacher to the student [7, 8] , to note a few. Some authors have focused on efficient distillation when only a subset of the original training data of the teacher is available [9] , distilling easy classes first [10] , or recently by looking at feature distillation as a diffusion process [11] .Recently, many works have pointed out the gap between the teacher and student models. Some authors looked at encoding the residual error between a large teacher and student [12, 13] , using an ensemble of projectors for feature distillation [6] , proposing a correlation based loss function in place of the KL divergence [9] , and rectifying the imbalance at the concept level [4] . Most related to our work is the work of Mirzadeh et al. [2] , and Son et al. [3] . Mirzadeh et al. proposed training teacher assistant models of progressively smaller size between that of a teacher and student model and using the smallest teacher assistant for distilling knowledge to the student. Son et al. argued that such a distillation process could cause error cascading and proposed using all the assistant(s) and the teacher for distillation. Further, each assistant is trained using all larger assistants and the teacher. This leads to progressively increasing computation costs as the number of assistants increases. Unlike their works, instead of explicitly training assistant models whose size is larger than the student, we propose a mechanism to mimic intermediate teaching assistants by training a small RDM module which is significantly smaller than even the student model. Further, unlike [3] where TAs need to be trained sequentially in the order of decreasing size, our RDMs are trained independently and in parallel. Another important work that is related to the RDM is the paraphraser network proposed by [14] . However we propose a more principled (from Shannon's insight on Rate-Distortion theory) loss function to train the RDMs where the information is compressed at different levels to mimic teacher assistants of different capacities.The proposed IBM also has parallels in works like [6, 15] . The former looks at introducing an ensemble of projector networks that map the student embedding to the teacher embedding. Unlike the proposed IBM, which has a rate constrained loss function, the projector networks are trained for reconstruction error only. The concept behind IBM, the Information Bottleneck Principle (IBP), is also related to Masked-Image-Modeling [16] and Masked Generative Distillation [15] . The connections between IBP and improved generalization [17] support the success behind MIM and our proposed IBM. While our IBM loss function attempts to directly upper-bound each of the terms of the IBP objective, MIM upper bounds by dropping negative terms from the IBP objective. On the otherhand, the MGD objective is an upperbound only under certain modeling assumptions. We discuss these connections in detail in Section 3.2 and Appendix C.CLIP Distillation: Distillation of CLIP like models is a relatively new direction. One of the earliest works was to use CLIP to distill task specific knowledge into task specific student models [18] . In the task agnostic distillation of CLIP, TinyClip proposed a knowledge distillation and a pruning method for CLIP [19] . Yang et al. studied various losses for distillation including feature distillation and relational losses [20] . Relational losses attempt to maintain the same relative distances between embeddings in the teacher and student model [21, 22] . Another noteworthy paper is MobileCLIP which used an ensemble of teachers and a data refinement technique to obtain powerful small CLIP models [23] . However, the ideas proposed in these works are CLIP specific, i.e., they exploit the interactions between different modalities to compute the distillation losses. Unlike these works, our proposed idea is general, i.e., not CLIP specific.Rate Distortion and Information Bottleneck via Neural Networks: Neural networks have been used to learn encoders and decoders for compression using Rate-Distortion theory [24, 25, 26] and communication over noisy channels [27] . On the other hand, Information Bottleneck has been used to improve neural networks in areas like improving generalization [28, 29] , Out-of-Distribution detection [30, 31] and Out-of-Distribution generalization [32, 33] .3 Controlled Information Flow for KD3.1 Controlling the information from the teacher using Rate-Distortion Theory Shannon proposed Rate-Distortion Theory as a principled way to compress a signal. Given an input X, the goal of compression is to find a mapping from X to its compressed version X such that X has minimal information about X but at the same time the distortion does not exceed D 0 . We can write this as an optimization problem of the formEQUATIONwhere D(•, •) is some distortion measure and I(•; •) denotes the mutual information between two random variables [34] . We can convert this to an unconstrained optimization objective of the form min R • D(X; X) + I(X; X), (2) where R determines the trade-off between information rate and distortion. A larger R corresponds to more emphasis on minimizing the distortion at the cost of a higher information rate and viceversa. This idea is best understood in the context of lossy compression where we wish to compress as much as possible while allowing tolerable distortion. However, rate-distortion theory is not only applicable to compression, but also to problems like Joint Source-Channel Coding where the compressed representation is subject to noise [34] . This case is represented in Figure 1(a) . The encoded representation of the input is denoted as Y and its noisy version as Ŷ . The independent noise added to encoded representation is denoted as Z.However, even though Shannon's theory tells us what optimization problem to solve, it does not tell us how to solve it. When the encoders and decoders in Figure 1 (a) are neural networks this problem is compounded because the objective cannot be computed (I(X; X) is intractable). Instead, we can use variational approximations to compute an upper bound on the objective, which in turn will allow us to perform gradient descent to learn the encoder and the decoder. Let us denote q( Ŷ ) as an approximation of true but unknown distribution of Ŷ , p( Ŷ ). Then, we can compute an upper-bound on I(X; X) asEQUATIONwhere H(•) denotes the entropy, and H q ( Ŷ ) denotes the cross entropy computed using the distribution q( Ŷ ). The first inequality follows from the Data Processing Inequality [34] and the second follows because cross-entropy is always greater than entropy [34] . Finally, note that H( Ŷ |Y ) is constant because the noise is independent of the encoder and decoder parameters.The key here is to choose the approximating distribution q. Popular choices include the Gaussian distribution (e.g., Variational Autoencoders [35] ), learning the distribution [36] , or non-parametric approximations [24] , we choose the latter. All these methods yield a mechanism where we can compute q( Ŷ ). Now, for simplicity let us assume that distortion measure D is the L 2 norm, then we can learn the parameters of the encoder (Θ e ) and the decoder (Θ d ) by putting (3) into (2) asEQUATIONFigure 1 (a) shows how the RDMs are trained. Since the RDMs process extracted teacher embeddings, they do not have to relearn low level feature extractors from the raw input, thus making them significantly smaller than teacher assistants, which in turn makes them computationally cheaper.In the case where we are distilling a classification model, an additional linear layer is used to convert the reconstructed embeddings to logits. Let V represent the true classification label, VT represent the teacher's output predictive distribution on the class labels, VRDM be the same but as predicted by the RDM output. Then, the loss function used to train the RDM isEQUATIONHere, L CE (•, •) is the cross-entropy loss, KL(•||•) is the Kullback-Leibler divergence, and λ KL is weighting factor for the KL loss. For simplicity of writing we have dropped the presence of the temperature τ from the KL divergence term, however, it is assumed to be present. Figure 7 shows the illustration of how the RDM is trained for classification.Connections between Teacher Assistants and RDMs: TAs limit the amount of information extracted from the input by using limited modeling capacity, i.e., smaller the model poorer the TA performance on the downstream task. On the other hand, we limit the amount of information extracted from the teacher embedding by passing it through a rate constrained communication channel. If we place a higher constraint on the information through the channel, our reconstructed embeddings will have more distortion compared to the teacher's and will perform poorly on the downstream task, just like the embeddings from a TA with a small model capacity. Thus, by choosing different values of R in (4) we can mimic different TAs with different modeling capacities.CLIP (Contrastive Language-Image Pretraining) is a class of foundational models that are capable of embedding inputs from distinct modalities into a shared embedding space [38, 39] . In CLIP a modality specific encoder processes the input from a specific modality and embeds it into a shared embedding space. CLIP like models have shown tremendous performance in zero-shot classification, object-detection, and retrieval [38, 39] . Further, the trained encoders have also proved instrumental in powering Large Multimodal Models (LMMs) [40, 41, 42] and generative models [43, 44] . So distillation of these models has far reaching applications especially in on-device generative AI.For simplicity, let us consider two modalities Image (I) and Language (L). Consider a batch of B image-text pairs {(I (1) , L (1) ), . . . , (I (B) , L (B) )}. Let Û (b)I denote the L 2 -normalized embedding of the b-th image obtained from the image encoder and Û (b) L denote the same for the b-th text obtained from the language encoder (after IBM as in Figure 2(b) ). Then we can write the contrastive loss from image to language embeddings asEQUATIONwhere ⟨•, •⟩ represents the inner-product between the two vectors. Using this we can write the loss to train CLIP-like models asEQUATIONThe idea in contrastive loss is that embeddings of a paired image and text must be close to each other (high inner-product) when compared to embeddings of an image and/or text of non-pairs. Going forward we use CLIP to denote any CLIP like model.Since the CLIP teacher has modality specific encoders, each encoder has its own set of RDMs. Each RDM is trained using (4). Let us denote the embedding for b-th image and text from the teacher by X L , respectively. We can now write the modality specific CIFD loss for CLIP distillation asEQUATIONNow we can write the final loss to perform distillation of CLIP using CIFD asEQUATIONwhere λ CL is weighting factor.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss this in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
on classification models, by distilling resnet34 and resnet50 (teachers) into resnet18 and: 
 MobileNet-V2 (students), we achieve +1.66% (absolute) and +2.71% over KD [1] , respectively. 4. To showcase the generality of the proposed CIFD, we apply it to distilling CLIP like models.Over three different teacher-student combinations, across five zero-shot classification and two zero-shot retrieval datasets, we find that our proposed method significantly outperforms CLIP specific distillation methods. In the previous section, we introduced RDMs to control the flow of information from the teacher model and mimic teacher assistants. However, when the teacher and multiple RDMs are providing feedback, the feedback can cause the student model to overfit and lead to poor performance. We can overcome this by constraining the information from the student model exposed to the feedback, i.e., provide partial feedback. Similar to the teacher model, we can accomplish this using another rate-constrained channel, but at the student. Further, this rate-constrained model is present only when there is feedback to the student, i.e., only during training. However, unlike the case of RDMs, we are not interested in trying to reconstruct the input to the rate constrained channel. Instead we wish to reconstruct the teacher or RDM embeddings. In Information Theory, the Rate-Distortion problem deals with compressing a random variable. However, when we want the compressed representation to be informative about another variable, it is called the Information Bottleneck problem. Thus, we call our proposed rate constrained module in the student model as Information Bottleneck Module (IBM).Tishby et al. introduced the Information Bottleneck Principle (IBP) as a generalization to the Rate-Distortion problem [37] . Given an input X S and some random variable of interest U , the goal in IB is to find a representation Û that removes as much information about input X S while retaining as much information about U . This is formulated asEQUATIONwhere, λ I is the Lagrange multiplier. 6) are intractable. To overcome this setback, we use variational approximations, resulting in an upper bound on the IB objective (6) [28] .EQUATIONwhere D represents a suitable distortion metric, r( Ŵ ) is the approximation of the true distribution p( Ŵ ). In the IBM, U is set to be the teacher's (or RDM's) embedding. So, at the output the IBM is attempting to reconstruct the teacher's (or RDM's) embedding. In case of a dimensionality mismatch, a projection layer is used. In practice, we do not use a dedicated encoder for IBM and let the student backbone model itself function as the encoder, i.e. W = X S . The decoder is a simple network of at most one to two linear layers. The IBM is not trained separately, instead it is trained along with the student model. Connections to Masked-Image-Modeling: In this discussion, we compare IBMs with Masked-Image-Modeling [16] and its extension into distillation Masked-Generate-Modeling [15] . In MIM, an input image (T ) is masked (T M ). The masked image is fed through an encoder followed by an MIM encoder. The encoder provides an embedding of the image and the MIM encoder uses it to predict the tokens corresponding to the masked out part. The system is then trained to maximize the likelihood of the actual values of the masked out tokens. In Masked-Generative-Distillation, the masking is done after passing through the student backbone. A more detailed explanation along with the formal statement and its proof are provided in Appendix C. The loss function of our IBM is directly written as an upper bound on the IBP objective. Unlike MIM or MGD, we do not drop the second term and instead use a non-parametric upper-bound to approximate it. This empirically should ensure a tighter approximation to the IBP objective than the other two. IBM works better because it forces the student to focus on those features necessary to predict the teacher embedding (first term) and remove information not useful in the prediction (second term), whereas in MIM and MGD the removal of information is either implicit or not present respectively. There have recently been works studying how IBP helps reduce generalization errors [17] which provides support that IBM in the student model should help reduce generalization errors. Our ablation studies in Section 4.3 also show similar results.With all the components in place, we can now derive the final loss function. Using the methodology described in Section 3.1, we assume that N RDMs corresponding to different rate-constraints have been trained. To re-iterate, let X be the input datapoint. Let us denote the teacher model's embedding as X; Xn , Z n are the embedding and noise in the n th RDM respectively, and Û corresponds to the student model's embedding. Also for sake of illustration, let us assume that the loss to match embeddings is the L 2 loss. Then, we can write the loss function for training the student model asEQUATIONwhere λ n are weighting coefficients.In the case of classification, let us denote the output predictive distribution of the teacher as VT , VS for the student, Vn for the n th RDM, and the true label as V . Then, we can write the loss function for training the student model asL ′ CIF D = E X,Z N 1 ,Z S λ CE L CE (V, VS ) + λ KL KL( VT || VS ) + λ KL n=N n=1 λ n KL( Vn || VS ) + L CIF D . (9)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
experiments: 
 Experimental setup: Our experimental results are split into two sections, one dealing with supervised classification on the CIFAR-100 [45] and Imagenet (IN) [46] datasets, and another with CLIP like models trained on Conceptual Captions 12M dataset [47] . For the evaluation of the latter models, we conduct zero-shot classification on ImageNet, Imagenet-V2 (IN-V2) [48] , Imagenet-A (IN-A) [49] , Imagenet-R (IN-R) [50] , and Object Net (ObjNet) [51] , and we do zero-shot retrieval on COCO [52] and FlickR30k test set [53] . Architecture and training details in Appendix A. Unless otherwise mentioned results from other works are obtained from those papers. In Table 1 we first look at experiments on CIFAR-100 using a simple CNN model to specifically compare against the works of [2, 3] which only study these models. We see that using CIFD with just one RDM boosts performance over KD by +1.41%. However, when using 3 RDMs, we see a difference of +1.92% (absolute) over KD, and a boost of +0.63% over DGKD. DGKD uses three teaching assistants of size eight, six, and four layer CNNs. Since RDMs are less computationally intensive to train than teaching assistants, we trained two more and used them to perform distillation. We see that the resulting student model is +2.49% over KD, and +1.2% over DGKD. We present the more comprehensive results over CIFAR-100 in Table 2 . We find that our proposed CIFD performs as good as existing methods.However, the interest of this paper is more on large scale datasets. In Table 3 we look at the performance of our proposed CIFD methods in eq. ( 9) for distilling knowledge from a ResNet-34 to a ResNet-18 model and a ResNet-50 to a MobileNet-V1 model over the 1.28 million ImageNet dataset, respectively. We compare against multiple prior works listed in the table. We find that CIFD achieves State-of-The-Art (SoTA) performance, +1.66% (absolute) improvement over KD [1] , +0.86% (absolute) over TAKD [2] , and +0.5% (absolute) over DGKD [3] on ResNet-18.Looking at the case of ResNet-50 to MobileNet-V1, we find that the proposed method performs slightly above the work of [6] in top-1 accuracy, and achieves the SoTA performance in top-5 accuracy.Finally, to establish the importance of the proposed method in the large dataset regime, we study knowledge distillation for foundational CLIP like models trained over the 12 million image-text pair dataset, CC-12M [47] . In Table 4, we look at the zero-shot classification and zero-shot retrieval results over multiple datasets for CLIP like models. We tested the distillation mechanism over different students and found that our distilled models beat the distillation mechanisms that are specialized to CLIP [19, 20] .To ensure a fair comparison, for TinyCLIP [19] , we used only their knowledge distillation related innovation, not pruning (details in Appendix D.4). For ViT-B-16 models, despite multiple attempts of [19] , the training diverged. Looking at the results, in zero-shot classification, our distilled models show superior zero-shot classification over multiple datasets across three different teacher-student combinations. We see as high as 5.3% improvement over the nearest prior work over IN-R dataset on ViT-S-16 (Table 14 ). Similar trends hold over zero-shot image-to-text and text-to-image retrieval over both the COCO [52] (Table 15 ) and Flickr30k dataset [53] , where our proposed models outperform existing distillation methods by as much as 8% (absolute) for I)T@5 on the COCO dataset on ViT-S-16. We see largest gains in performance when the ratio between teacher to student size is large. Top-1 Top-5 Top-1 Top-5 Image → Text Text → Image R@1 R@5 R@10 R@1 R@5 R@10ViT-L-14 79.22 95. 52 Top-1 Top-1 Image → Text Text → Image R@1 R@5 R@10 R@1 R@5 R@10CLIPKD [20] ViT-S- 16 In Figure 5 we show the RDMs used to obtain the results in Table 1 . As the information through the RDM is constrained, the RDM's output classification performance also falls. This is because as the information rate is constrained, and the RDM is forced to drop some features that are useful for classification. The corresponding TAs used by [2] in Table 1 , have accuracies of 54.72%, 54.84%, 50.32% for model sizes of eight, six, and four layer CNNs resepctively. This trend closely matches the RDM trend in Figure 5 . Thus, by varying R we can mimic teaching assistants of different modeling capacities, i.e., lower R we mimic a smaller TA, higher R we mimic a larger TA.How does number of RDMs affect performance? Does IBM help? In Table 6 , we study the effect of the number of RDMs and the effect of IBM in conjunction with the RDMs. We find that increasing the number of RDMs without IBM initially improves the performance, but with five RDMs the performance on CIFAR100 drops. We hypothesized that this is due to overfitting, which DGKD proposed to overcome by KD dropout [3] . In KD dropout, the gradients from the teacher and teaching assistants are randomly cut off (Fig. 3 in [3] ). Despite incorporating it, the performance without IBM does not show improvement. However, with IBM we see the expected benefits. Further, we see that IBM on its own also improves the performance of KD [1] . This further corroborated on Imagenet as shown in Table 6 . The RDMs used in all the experiments correspond to different values of R. This further shows that having RDMs at different information rate R is key to improving performance. In Table 5 , we study the affect of number of RDMs on distilling CLIP like models. To reduce compute, we use a smaller teacher of size ViT-B-16 trained by [58] . We see that increasing number of RDMs leads to improved performance. We also provide results from [20] for comaprison.Analyzing CIFD gains with large teacher student capacity gap: Here, we study the performance of CIFD when the teacher student gap is increased. In Table 7 we see that as the size of the teacher (and correspondingly its performance) increases, the student performance also increases. In fact, the increase is montonic w.r.t. the teacher size. We also compare with the performance of DistKD [5] which also studied a similar premise. We find that not only does our method outperform DistKD, that unlike DistKD the proposed CIFD does not show drop in performance when the teacher size is increased. This indicates the robustness of our proposed method.The parameter ratio of teacher to student ranges from 1.86 to 5.12. We also study CLIP like models where the maximum parameter ratio is a larger 6.9 in Appendix B.4. We see that for 3 out of the 5 zero-shot classification datasets CIFD yields larger improvements over the next best competitor of ClipKD [20] when the capacity gap is larger. In zero-shot image-text retrieval, CIFD almost always yields larger improvements over the ClipKD when the capacity gap is larger. This indicates that CIFD excels when the teacher-student capacity gap is large. 1 , TAKD [2] is 2× more expensive and DGKD [3] is 5× more expensive than proposed. 
conclusion: 
 In this paper we present a novel knowledge distillation framework called the Controlled Information Flow (CIFD). CIFD consists of two main components. The first is a lightweight Rate Distortion Module (RDM) that replaces the expensive Teacher Assistants by using the teacher model's embedding and a noisy communication channel. Importantly, since the RDM uses the teacher model's input embeddings, it does not have to relearn low level feature extractors, thus making the model significantly smaller than a TA. Second, we propose an Information Bottleneck Module to prevent the student model from overfitting in the presence of the teacher and multiple RDMs. The resulting framework shows impressive performance on Imagenet and significantly outperforms CLIP specific distillation methods on CLIP models. Finally, we corrobrated that an increased number of RDMs with diverse Rs is a key factor for better distillation, entailing only a small increase in computation.Limitations and impact: An interesting direction is to study alternative algorithmic formulations for using the information from RDMs, i.e., sequentially enabling and disabling which RDM give feedback at what points of training. Regarding societal impact, the student model depends on the teacher model to transfer concepts. Unfortunately, this also means biases present in the teacher due to its training data are also transferred to the student.for contrastive language-image learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2818-2829.[77] Laurens Van der Maaten and Geoffrey Hinton, "Visualizing data using t-sne.," Journal of machine learning research, vol. 9, no. 11, 2008. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: As our experiments are implemented based on open-source code and publicly available datasets, we have provided the necessary details in our paper for reproducing the results on top of the public code base and database, with the associated URLs provided. We also provide detailed figures of how different loss functions are computed and the models are trained, see Figure 2 , Figure 7 , and Figure 8 . Along with that we provide details on the hyper-parameters for all experiments in Appendix A.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Details are provided in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The experiments in this paper mainly focus on large datasets like Imagenet (1 million samples) and CC12M (12 million samples) that require lots of compute. For these datasets, we follow the best practices of existing papers and report accordingly. CLIP like models take 6 A100 GPU days for training. Given that we already had to train the baseline methods too, training multiple models for all of them would be very compute intensive.Alternatively, we focus on a large set of inference experiments to showcase our proposed method's superior performance. For example for CLIP like models, we perform zero-shot classification tests on five datasets and zero-shot retrieval on two datasets showcasing superior performance on all of them on three different teacher-student pairs. Guidelines:• The answer NA means that the paper does not include experiments. Justification: We provide those details along with the hyperparameters in Appendix A. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?[Yes] Justification: We have read and followed the ethics guidelines. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts • The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Table 11 shows more baselines for experiments on ImageNet. As we can see our proposed method outperforms all the compared methods.Table 14 shows the zero-shot image text classification results using CLIP like models over five datasets. Table 15 shows the zero-shot image text retrieval results over COCO and FlickR30k test sets. Across all classification and retrieval, our model consistently outperforms CLIP spcific distillation methods. This showcases the generality of the idea proposed. Further, we see the largest gains due to the proposed method when the gap between the teacher and student size is the largest.We study the effect of large teacher student capacity gaps in CLIP like models. In Table 12 and Table 13 , we compare the difference when a ViT-L-14 teacher was used to train ViT-B-16 and ViT-S-16 students. The parameter ratios are 2.8 and 6.9, respectively. CIFD shows greater improvement over baseline when the teacher student ratio is large for 3 out of the 5 zero-shot classification datasets and almost always for the two zero-shot retrieval datasets.Masked Image Modeling (MIM), as shown in Figure 6a , has had major success in pretraining large image models like BEiT [16] . Here, T is used to represent the input, T M is used to represent the masked input, U is the tokens of the image corresponding to the masked out parts, and Û is their predicted value as predicted the MIM model. In the BEiT variant of MIM, an image is first converted into tokens using a pretrained tokenizer (like a discrete VAE). Next, a masked version of the image (T M ) is fed into the MIM encoder which attempts to predict the tokens of the masked parts. We can write the MIM training objective asEQUATIONMasked Generative Distillation (MGD), as shown in Figure 6b , is an extension of using MIM type of training for distillation. Here, T is used to represent the input, U is the teacher model embedding corresponding to the full image, and Û is their predicted value as predicted the MIM model. In MIM, the target for prediction are the tokens corresponding to the masked parts of the input. In the case of MGD, the student model is tasked to predict the teacher model's embedding of the original image. MGD has shown promising results in distillation. The loss corresponding to the Masked Generative Distillation is written asEQUATION)The loss functions rightfully appear the same. However, the process differs in two key places. First is where the masking is applied. In MIM, the masking is done on the input. In MGD, the masking is done after student backbone preprocessing. Secondly, the likelihood maximization formulation is different, i.e., in MIM, the tokens are discrete classes so a cross-entropy loss is used, whereas in MGD, the teacher embedding is continuous so MSE is used. Thus the likelihood distribution in MIM is multinomial whereas in MGD it is gaussian. Note, for MGD we are only studying the masked distillation loss proposed. The other loss function is not relevant here.Before comparing with IBP we present a small lemma that minimizing the negative log-likelihood is the same as minimizing the negative of the Mutual Information between two variables. Lemma 1. Minimizing the negative log-likelhood of predicting a non-learnable U from some learnableEQUATIONis equivalent to min -I(U ; Û ).Table 15 : Zero-shot image-text retrieval performance on COCO [52] and FlickR30k [53] test sets. All methods use the first Teacher, MobileCLIP additionally uses Teacher-2 † .Method Model COCO Flickr30k Image → Text Text → Image Image → Text Text → Image R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 The IBP objective function, applicable to all systems shown in Figure 6 is written asEQUATIONFor simplicity of exposition we split Theorem 1 into the following lemmas one each for MIM and MGD respectively.Lemma 2. The objective function of MIM ( 14) is an upper bound on the objective function from the Information Bottleneck Principle (18) .Proof. We start with the Information Bottleneck objectiveEQUATIONWe note that Û -T M -T forms a Markov Chain. Thus leveraging the Data Processing Inequality [34] , we note that I(T ; Û ) ≤ I(T ; T M ). We note that I(T ; T M ) is constant w.r.t. the minimization, since it does not have any parameters that are optimized by gradient descent. Hence, we can drop that term. By leveraging Lemma 1, we can see that the MIM objective ( 14) is an upper-bound on the (18) .It is tempting to assume that MGD is also an upper-bound similar to MIM. Unfortunately, this is not the case in general. This is because the noise is added after backbone processing and this processing is optimizable. However, under certain conditions, we show that it is a valid upper-bound.Lemma 3. Let Û-M denote random variable prior to the masking operation in MGD (Figure 6b ). Then, the objective function of MGD ( 15) is an upper bound on the objective function from the Information Bottleneck Principle (18) if Û-M is a discrete random variable and the mapping from T to Û-M is deterministic.Proof. We again start with the Information Bottleneck objectiveEQUATIONWe note that Û -Û-M -T forms a Markov Chain. Thus leveraging the Data Processing Inequality [34] , we note that I(T ; Û ) ≤ I(T ; Û-M ). Unlike the MIM case, I(T ; Û-M ) is not a constant w.r.t. minimization. By definition of Mutual Information, we can writeEQUATIONwhere by slight abuse of notation H represents differential or discrete entropy based on if the random variable is continuous or discrete respectively. Since the mapping from T to Û-M is deterministic, it follows that H( Û-M |T ) = 0. If we assume that Û-M is discrete, then H( Û-M ) is discrete entropy and we know that H( Û-M ) ≥ 0 [34] . Thus, by dropping that term, we get an upper bound on the IBP objective.Remark 1. If Û-M is not discrete, then H( Û-M ) < 0 is possible, i.e., differential entropy can be negative. Thus, we cannot guarantee that the MGD objective is always an upper-bound of IBP in that case.Remark 2. The MGD upper-bound employs two relaxations that make it an upper-bound on IBP. First is the Data Processing Inequality to show that I(T ; Û ) ≤ I(T ; Û-M ). The second (under the discrete assumption) that H( Û-M ) ≥ 0.The loss function of our IBM is directly written as an approximation of the IBP objective as shown in Section 3.2. Unlike MIM or MGD, we do not drop the second term and instead use a non-parametric upper-bound to approximate it. This empirically should ensure a tighter approximation to the IBP objective than the other two. IBM works better because it forces the student to focus on those features necessary to predict the teacher embedding (first term) and remove information not useful in the prediction (second term), whereas in MIM and MGD the removal of information is not present. There have recently been works studying how IBP helps reduce generalization errors [17] which provides support to the proposed idea. While comparing with TinyCLIP [19] , we disabled pruning, as pruning is orthogonal to our proposed idea. Instead we focused only on the Knowledge Distillation part from [19] . To do that in addition to the contrastive loss (11) , we added the two terms as proposed in equations (1), Section 3.1 of [19] . These equations transfer the relative embedding distances from the teacher's embedding space to the student's embedding space.NeurIPS Paper Checklist 


Decision: This submission leverages rate-distortion theory to formulate a new method which can replace computationally intensive teacher assistants and improve knowledge distillation process. The proposed method CIFD consists of two modules to be trained separately, namely rate-distortion module (RDM) conditioned on the penultimate-layer features from the teacher backbone and generative feature reconstruction with noise, and information bottleneck module (IBM) to guide the regularization of the student model given multiple RDMs. The submission was initially scored (4,5,4) by three knowledgeable reviewers, who recognized some positive aspects of the paper, **a)** exploring a new technical perspective on how to construct efficient and effective teacher assistant alternatives; **b)** the proposed method shows competitive performance; **c)** CLIP based settings are explored in experiments. Meanwhile, the reviewers also raised some major concerns about **1)** unconvincing motivation connected to teacher assistants; **2)** insufficient experiments, the model scale gap of teacher and student is not large (thus cannot well support its claimed motivation), and lacking sufficient comparison with recent knowledge distillation methods; **3)** somewhat unfair/inconsistent experimental settings; **4)** how performance gain is achieved, or say, the role of generative feature reconstruction with noise may play the key role instead of the proposed RDM; **5)** the computational complexity of RDMs in training process.

The authors provided detailed responses to these concerns, which are most recognized by reviewers. Finally, this submission has scores of (5,5,5) increased from initial (4,5,4), and all reviewers are on somewhat positive side. The AC read the paper, the reviews, the rebuttal and the reviewers' feedback, and agree with reviewers' assessment. To some degree, I also agree with the remained concern from reviewer sfSd, a more in-depth discussion of DRMs with existing generative knowledge distillation methods such as MGD and DiffKD as well as many-to-one knowledge distillation methods should be added into the paper, bettering positioning this work's contributions. Balancing above aspects of this submission, I recommend to accept (poster). The authors are encouraged to carefully consider the reviewers' comments/suggestions and their rebuttal in the final paper revision.

Reviews: review: paper_summary: paper_summary: The paper presents a new distillation method, CIFD, designed based on *Shannon’s Rate-Distortion theory* and  **Information Bottleneck Principle (IBP)*. CIFD contains Rate-Distortion Modules (RDM) for the teacher to substitute heavy Teacher Assistant (TA) and Information BottelNeck Module (IBM) for the student to mimic the features from several RDMs. Experiments demonstrate the effectiveness of the method.
strength_weakness: strength_weakness: My main concerns are from three aspects: **i) the story of the paper; ii) the reason why CIFD works; iii) insufficient experiments and comparisons.** Some concerns are mixed among the three aspects. And I will list them one by one.

1. **Insufficient experiments on verifying the basic settings of the paper.**
The story starts with *"When the teacher model is significantly larger than the student, previous works that utilize TAs induce high training costs."* I would believe the basic settings of this work as the teacher-student network pairs are in large parameter scale differences. From this point of view, the paper should contain more systematic experiments to verify the efficacy under this setting. Specifically, CIFD should be compared with previous methods on ImageNet with teacher-student network pairs with large different parameter scales, not just traditional ResNet-34 -> ResNet-18 and ResNet-50 -> MobileNet-V1.

2. **The trade-off between the story and the empirical solutions.** In my opinion, the paper is a little bit overdecorated and overclaimed. The author proposes many concepts, such as *Shannon’s Rate-Distortion theory* and  *Information Bottleneck Principle (IBP)*, and claims **"This is the first application of Shannon’s Rate-Distortion theory to aid knowledge distillation"**. I don't mean that the aforementioned statement is misleading.  But, if we go deeper into the design, the reason why the method works may come from the **noise-adding and noise-removing process**. Many previous works have verified that the above process could benefit the learning process in computer vision, like MIM and diffusion models, which have been empirical solutions.   In KD, there also exist distillation methods following MIM and diffusion models, like MGD and DiffKD. From this point of view, the authors should not claim ***"the first"*** only, but make a deeper analysis of related methods and make detailed comparisons. ***I strongly encourage the authors to make a good balance between the story and the verified empirical solutions.*** Even though it seems not as novel as this version, it would provide the readers with more useful knowledge and insights.

3. **The design may alter the network architecture of the student.** It seems that the IBM module would also be included in the validation stage. If my judgment is true, the added module (though lightweight) would also benefit the performance. Under such circumstances, the comparisons with previous methods, especially for light-weight models, are unfair.
questions: questions: See Weaknesses.
limitations: limitations: Limitations are included in the main paper and the Appendix
paper_summary: paper_summary: Inspired by Shannon’s rate-distortion theory, this paper proposes two modules, namely the Rate-Distortion Module and the Information Bottleneck Module, to construct intermediate representations for knowledge distillation. Extensive experiments on various datasets demonstrate the effectiveness of this method.
strength_weakness: strength_weakness: 1. The author's motivation and explanation for TA distillation are not very convincing. In my view, the RDM and IBM proposed in this work can be interpreted as two adapters connected to the teacher and the student respectively for distillation, and the principle is similar to the FT method.
2. In Table 2, 9 and 10, most of the compared methods were published in 2022 or before. The authors are encouraged to compare your method with recent state-of-the-art methods such as MLKD[1], CTKD[2], and LSKD[3].
3. In Eq. 5, I am a little confused about the author's formula representation. Generally speaking, the left side of the comma is the network to be trained, and the right side is the learning target. But the author seems to have it reversed here.

References:  
[1]. Multi-Level Logit Distillation. CVPR 23.  
[2]. Curriculum Temperature for Knowledge Distillation. AAAI 23.  
[3]. Logit Standardization in Knowledge Distillation. CVPR 24.
questions: questions: 1. $q(\hat{Y})$ is not clearly marked in Fig. 2. Which part of the network produces it? 
2. It would be more beneficial if the author could add a figure on how to perform CIFD distillation on the CLIP model.
limitations: limitations: Please refer to weaknesses.
paper_summary: paper_summary: Some existing methods alleviate the capacity gap between the teacher and student by setting up Teacher Assistants (TAs), introducing a large number of additional parameters and computational costs. Based on this, this paper proposes to train multiple RDM modules and connect multiple independent classification heads to generate branches with different performance to simulate the TA model. The authors think that this hierarchical model of extracting teacher knowledge can help alleviate the capacity gap between the teacher and student.
strength_weakness: strength_weakness: * I noticed that there is a significant difference in baseline performance between Table 1 and the original text, and Table 3 only uses a single Assistant for TAKD and DGKD, while the author's proposed method uses three RDM headers, which is not a fair comparison.
* There are significant differences in the value of R across different datasets, R=1 for CIFAR but R=10^4 for ImageNet. This means that the selection of hyperparameters on unfamiliar datasets is challenging, and the parameter tuning process may introduce multiple computational costs, which limits the versatility of the method.
* I noticed that the method proposed by the author introduces and trains at least $3N$ additional layers of MLP, and the forward process and loss calculation cost in distillation is also increased several times. However, the training cost is even the same as the method NormKD based solely on Logits Distillation without additional modules (Figure 1 (b)). Can you present the results with specific numerical values? What is the key to introducing so many parameters without introducing additional computational overhead?
questions: questions: * The experiment was conducted when there was not much difference between the teacher and student models (common settings in current KD tasks). It cannot prove that the proposed method can alleviate capacity differences, as many methods have proven that such settings (e.g., Res34 $\to$ Res18) do not require the additional assistant model.
* Why use the penultimate layer feature of the teacher? Is this from theoretical analysis or empirical summary?
* For this paper, I think it would be better to place Related Works at the front.
limitations: limitations: The authors have discussed a limitation in Conclusion.

Comments: title: Thank you!
Thank you!
Thank you and summary of changes
Comments by Reviewer sfSd
Response to Reviewer sfSd's post rebuttal comments
Post Rebuttal comments by Reviewer sfSd
comment: Thank you again for your time and effort in helping improve our paper.

We wanted to specially acknowledge the question on large student-teacher capacity gap experiments that pushed us to provide stronger results during the rebuttal. It helped us show that proposed CIFD not only shows consistent improvement with increasing teacher size but also shows significant improvement over existing methods (+0.36% on ImageNet top-1 accuracy for ResNet152 to ResNet18 over existing DistKD), in Table 13. Further our methods showed similar trends for larger student-teacher gaps in CLIP like models (Tables 16 and 17).

Additionally, based on your questions,
- We showed that Masked-Image-Modeling objective is an upper-bound on the Information Bottleneck objective
- We showed improved performance over works like MGD, DiffKD, IFD, and NORM.
- We showed how our multiple RDM method which acts on the teacher embeddings is different from multi projector methods like [1] and [3] which act on student embeddings. Specifically, our RDMs are used to mimic Teacher Assistants, whereas projectors are used to better help the student model learn the teacher embedding by providing gradient via an ensemble of projectors. Additionally, we showed improved performance over  these methods.

If you have any further questions, please feel free to ask and we are happy to discuss. If not, thank you again for your time and effort.
Thank you for your time and effort, we greatly appreciate your response. We also wanted to acknowledge your inputs in helping us improve the paper, thank you.

We apologize for missing the point on the hyper-parameter R. From our analysis, we found that the values of R are inversely proportional to the capacity gap between the teacher and the student. Since $R^{-1}$ controls the weight of the bottleneck rate, too small an $R$ and the RDM will focus more on compression than retaining features crucial for accuracy, resulting in a poor RDM. Since the RN34 to RN18 gap is small (both in size and performance), smaller $R$ values were more appropriate and hence the selection. We plan to add this insight to our paper in the final version.

If you have any further questions or concerns to improve our paper, please let us know and we are happy to discuss. Thank you again.
We wanted to thank the reviewer again for their insightful questions that have helped us improve the paper. We wanted to quickly summarize our previous response.

- By using one RDM without IBM, we showed that the key to our superior performance compared to Factor Transfer (FT) [28] is the principled loss function used to train the RDM. This is in addition to the multiple RDMs and IBM which have no equivalents in FT. This sets our work apart from FT.

- We showed superior performance against newer works like MLKD, CTKD, and LSKD which the reviewer requested (Table 14).

Additionally, based on other reviewers' questions

- We showed that proposed CIFD scales well with increasing student-teacher gap. In Table 13, we showed that distilling from a ResNet152 to ResNet18 model, CIFD provided a 0.36\% improvement over top-1 ImageNet accuracy over the nearest competitor DistKD. We also showed that CIFD monotonically improved the student accuracy with teacher size, something not observed in DistKD.

- We also compared with other works like Masked Generative Distillation and Diffusion KD, and showed superior performance against both of them.

- We showed that CIFD works well for CLIP like models where the teacher-student gap are larger than ResNets (Table 16 and 17). By outperforming CLIP specific distillation methods, we also showed the generality of the proposed idea.


We are very grateful for the time and effort from the reviewer to provide us with a review that has helped improve our paper. We are happy to engage further if they have any questions. Thank you again.
Thank you for providing a detailed response that resolved some of my doubts. The range of uncertainty in hyperparameter R is still one of my concerns, but experiments based on CLIP are indeed very distinctive. Based on the opinions of other reviewers and my understanding, I will increase the score to 5.
Thank you very much for your response. 

As I am on the fence about this submission, I will make my final decision after discussing with other reviewers.
We again thank the reviewer again for their time and response. We greatly appreciate their interest in helping us improve the paper.

> Moving discussion between IBP and MIM to main paper
- We absolutely agree that having this discussion in the main paper will enrich it. In fact, it was in the main body of the initial draft. However, the space requirements proved very restrictive and we were forced to move the discussion to the appendix. We agree with you that it should be put back into the main paper and plan to do so in the final version.

> Although the motivations seem novel, the main improvements are from multiple RDMs. 

**Difference between [1], [3] and our work.**

While the multiple projection approaches has existed as pointed out in [1] (which is also cited as [20] in our paper), we performed comparisons specifically with [1] in the rebuttal pdf (Table 15). Note that *[1] and [3] use multiple projectors at the student* (see Fig. 1(c) in [1] and Fig. 1, right subfigure in [3]) and *our method uses multiple RDMs at the teacher*. Thus, these methods are quite different. For completeness we include comparisons with these methods in the table below. However, our method with 1 RDM and IBM performs close to [3] and the with 3 RDMs outperforms it. Additionally, our method with 3 RDMs and IBM outperforms [3] with 8 projectors at the student.

To further answer your concern about whether the claimed method of RDM training is giving the performance improvement, we compare with Factor Transfer Kim et. al [28] in Table 15 (rebuttal PDF). Our proposed method with one RDM and no IBM outperforms FT [28], where the difference is the loss function used to train the RDM. This shows that the improvement is coming from the new principled loss function used to train the RDM network.

**Difference between [2] and ours.** 

We have to note that [2] is for Online Distillation *without a teacher*. This setting itself is different from ours where use the presence of a teacher. While [2] does use an ensemble of projection modules at the teacher, the difference in setting used to train makes it hard to compare them. Specifically they use a gated ensemble of branch modules to simulate a teacher using the student's backbone. *In the below table we include [2] for reference only, the setting is different and they should not be compared directly.*

```
+------------------------------+-------+-------+
|     RN34 to RN18 on IN1k     | Top-1 | Top-5 |
+------------------------------+-------+-------+
|       FT (Kim et. al.)       | 71.43 | 90.29 |
|           IFD [1]            | 71.94 | 90.68 |
|  ONE [2] (No RN34 teacher)   | 70.55 | 89.59 |
|           NORM [3]           | 72.14 |  ---  |
|   ------------------------   | ----- | ----- |
| Proposed (1 RDM) and no IBM  | 71.83 | 90.69 |
| Proposed (1 RDM) and wt IBM  | 72.05 | 90.70 |
| Proposed (3 RDMs) and wt IBM | 72.32 | 90.88 |
+------------------------------+-------+-------+
```

**Summary**

We have shown over multiple experiments the efficacy of our proposed method. See Table 2 on CIFAR-100, Table 3 and Table 14 on ImageNet, Table 13 for large student-teacher gap in ImageNet, Table 5 on CLIP models. Specifically we have addressed the question if our proposed method is reason for performance improvement by disabling various parts of the proposed idea and show improved performance with existing methods (see Table 15). **These specifically showcase the importance of the proposed loss functions and their improved performance.**

We again thank the reviewer for their time and effort. If you have any further questions, please feel free to reach out to us. Thank you!
Thanks for your detailed response, which addresses my partial concerns. 

Here are my remaining concerns.

I personally believe that lacking deep discussions about related techniques in the main paper is inappropriate. I encourage the author to add this content in the main paper, not just in the Appendix. 

Although the motivations seem novel, the main improvements are from multiple RDMs. Similar distillation designs have been explored in [1-3]. On the other hand, when reading the paper and the authors' response, I find no obvious technical flaws. Therefore, I'm a little bit confused about whether the effectiveness comes from the theoretical framework that the author claimed. 

As I think the above arguments are not strong enough to reject the paper, I would consider my final rating according to both the response and other reviewers' comments.






[1] Chen, Yudong, et al. "Improved feature distillation via projector ensemble." Advances in Neural Information Processing Systems 35 (2022): 12084-12095.

[2] Zhu, Xiatian, and Shaogang Gong. "Knowledge distillation by on-the-fly native ensemble." Advances in neural information processing systems 31 (2018).

[3] Liu, Xiaolong, et al. "Norm: Knowledge distillation via n-to-one representation matching." arXiv preprint arXiv:2305.13803 (2023).