Title: CIFD: Controlled Information Flow to Enhance Knowledge Distillation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Knowledge Distillation

Information Bottleneck

Rate-Distortion

Teaching Assistant

CLIP
Abstract: Knowledge Distillation is the mechanism by which the insights gained from a larger teacher model are transferred to a smaller student model. However, the transfer suffers when the teacher model is significantly larger than the student. To overcome this, prior works have proposed training intermediately sized models, Teacher Assistants (TAs) to help the transfer process. However, training TAs is expensive, as training these models is a knowledge transfer task in itself. Further, these TAs are larger than the student model and training them especially in large data settings can be computationally intensive. In this paper, we propose a novel framework called Controlled Information Flow for Knowledge Distillation (CIFD) consisting of two components. First, we propose a significantly smaller alternatives to TAs, the Rate-Distortion Module (RDM) which uses the teacher's penultimate layer embedding and a information rate-constrained bottleneck layer to replace the Teacher Assistant model. RDMs are smaller and easier to train than TAs, especially in large data regimes, since they operate on the teacher embeddings and do not need to relearn low level input feature extractors. Also, by varying the information rate across the bottleneck, RDMs can replace TAs of different sizes. Secondly, we propose the use of Information Bottleneck Module in the student model, which is crucial for regularization in the presence of a large number of RDMs. We show comprehensive state-of-the-art results of the proposed method over large datasets like Imagenet. Further, we show the significant improvement in distilling CLIP like models over a huge 12M image-text dataset. It outperforms CLIP specialized distillation methods across five zero-shot classification datasets and two zero-shot image-text retrieval datasets.

CIFD: Controlled Information Flow to Enhance Knowledge Distillation

Abstract

Knowledge Distillation is the mechanism by which the insights gained from a larger teacher model are transferred to a smaller student model. However, the transfer suffers when the teacher model is significantly larger than the student. To overcome this, prior works have proposed training intermediately sized models, Teacher Assistants (TAs) to help the transfer process. However, training TAs is expensive, as training these models is a knowledge transfer task in itself. Further, these TAs are larger than the student model and training them especially in large data settings can be computationally intensive. In this paper, we propose a novel framework called Controlled Information Flow for Knowledge Distillation (CIFD) consisting of two components. First, we propose a significantly smaller alternatives to TAs, the Rate-Distortion Module (RDM) which uses the teacher's penultimate layer embedding and a information rate-constrained bottleneck layer to replace the Teacher Assistant model. RDMs are smaller and easier to train than TAs, especially in large data regimes, since they operate on the teacher embeddings and do not need to relearn low level input feature extractors. Also, by varying the information rate across the bottleneck, RDMs can replace TAs of different sizes. Secondly, we propose the use of Information Bottleneck Module in the student model, which is crucial for regularization in the presence of a large number of RDMs. We show comprehensive state-of-the-art results of the proposed method over large datasets like Imagenet. Further, we show the significant improvement in distilling CLIP like models over a huge 12M image-text dataset. It outperforms CLIP specialized distillation methods across five zero-shot classification datasets and two zero-shot image-text retrieval datasets.

1 Introduction

A decade ago Hinton et al. proposed a mechanism called Knowledge Distillation (KD), where the insights from a larger model (the teacher) are transferred to a smaller model (the student) [1]. The teacher model, on account of larger modeling capacity, is capable of learning complex relationships in the data and is thus better at performing its target task. KD attempts to help the training of the student model by showing it the insights learned by the teacher. Specifically, Hinton et al. suggested the transfer of 'dark knowledge" in the logits of a teacher classifier to a student classifier by minimizing the Kullback-Leibler (KL) divergence between a softened version of the predicted probability distribution of a teacher and a student model. However, dark knowledge is only one way of quantifying the insights learned by the teacher. In general, how to quantify the insights and how to transfer them to the student model remain open questions.

Teacher

(10 Layers)

RDM

(Mimic TA-1)

RDM

(Mimic TA-2)

RDM

(Mimic TA-

N

)

RDM

(Mimic TA-3)

Student

(2 Layers)

Image

Embeddings

Rate-Distortion Module (RDM)

(3 Layers)

Infomation

Amount

(

R

=High)

(

R

=Med.)

(

R

=Low)

Enc

Dec

Noise (

Input

Embeddings

(

)

Reconstruced

Embeddings

(

)

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Training cost

70.50

70.75

71.00

71.25

71.50

71.75

72.00

72.25

Accuracy (%)

CIFD(Ours)

TAKD

DGKD

NormKD (Chi et. al., 2023)

IPWD (Niu et al., 2022)

(Son et al., 2021)

(a) Proposed CIFD for KD.

72.50

(b) CIFD vs. existing KD approaches.

Recently many researchers in the knowledge distillation community have been concerned with the gap between the teacher and student's modeling capacities [2, 3, 4, 5]. Going back to the human learning experience, we learn simple concepts first, intermediate concepts next, and finally advanced concepts. Based on this insight, prior works have proposed the use of Teacher Assistants (TAs), models whose size is between that of the teacher and the student, to help facilitate the knowledge transfer better [2, 3]. By distilling knowledge from the teacher to the assistant and then using the assistant to help the distillation into the student led to large performance improvement. However, as seen in Figure 1(b), these methods are around 3 . 5 × and 4 . 5 × more expensive than KD [1]. In this paper we introduce a method called Controlled Information Flow for Knowledge Distillation (CIFD) which is significantly cheaper to train. This brings the training cost of TA based methods closer to KD and other counterparts while obtaining state-of-the-art performance.

The proposed CIFD mechanism consists of two parts. First, we propose to process the teacher's penultimate layer embedding by a Rate-Distortion Module (RDM) to replace TAs. The RDM imposes a constraint on the amount of information through it. To replace multiple TAs of different capacities, we propose using RDMs with different levels of information constraints. Since RDMs operate on the teacher embeddings, they do not have to relearn low level features and can avoid the associated training and inference costs. As far as we know, this is the first application of Shannon's Rate-Distortion theory to aid knowledge distillation. However, as the number of RDMs increases, the student model tends to overfit to the teacher and assistants' outputs, as pointed out in [3] (also confirmed empirically here), thus reducing the generalization performance of the student model. Thus, to regularize the training of the student model in the presence of multiple RDMs, we propose the Information Bottleneck Module (IBM). The following are the contributions of this paper:

We propose the use of an RDM that takes the embeddings from a teacher model and mimics a teacher assistant. Since the RDM does not have to relearn low level features, it is two to three layers only; significantly smaller than a teacher assistant model.

We propose the use of the IBM in the student model during train-time. We find that IBM on its own provides benefits but is also a crucial regularizer as the number of RDMs increases.

On classification models, by distilling ResNet34 and ResNet50 (teachers) into ResNet18 and MobileNet-V2 (students), we achieve +1.66% (absolute) and +2.71% over KD [1], respectively.

To showcase the generality of the proposed CIFD, we apply it to distilling CLIP like models. Over three different teacher-student combinations, across five zero-shot classification and two zero-shot retrieval datasets, we find that our proposed method significantly outperforms CLIP specific distillation methods.

Teacher

Backbone

Model

Enc

Dec

RDM

RDM

Embedding

Teacher

Embedding

Input

MSE Loss

: pretrained and frozen

: trainable

Noise (

Rate Loss

Total Loss

MSE Loss

Rate Loss

2 Related Works

Knowledge Distillation: Hinton et al. introduced the concept of transfering the knowledge from a large teacher model to a smaller student model using the logit space for classification models [1]. Many works have focused on different methods for transferring information like in the output space [6], transferring information in the intermediate layers of the teacher to the student [7, 8], to note a few. Some authors have focused on efficient distillation when only a subset of the original training data of the teacher is available [9], distilling easy classes first [10], or recently by looking at feature distillation as a diffusion process [11].

Recently, many works have pointed out the gap between the teacher and student models. Some authors looked at encoding the residual error between a large teacher and student [12, 13], using an ensemble of projectors for feature distillation [6], proposing a correlation based loss function in place of the KL divergence [9], and rectifying the imbalance at the concept level [4]. Most related to our work is the work of Mirzadeh et al. [2], and Son et al. [3]. Mirzadeh et al. proposed training teacher assistant models of progressively smaller size between that of a teacher and student model and using the smallest teacher assistant for distilling knowledge to the student. Son et al. argued that such a distillation process could cause error cascading and proposed using all the assistant(s) and the teacher for distillation. Further, each assistant is trained using all larger assistants and the teacher. This leads to progressively increasing computation costs as the number of assistants increases. Unlike their works, instead of explicitly training assistant models whose size is larger than the student, we propose a mechanism to mimic intermediate teaching assistants by training a small RDM module which is significantly smaller than even the student model. Further, unlike [3] where TAs need to be trained sequentially in the order of decreasing size, our RDMs are trained independently and in parallel. Another important work that is related to the RDM is the paraphraser network proposed by [14]. However we propose a more principled (from Shannon's insight on Rate-Distortion theory) loss function to train the RDMs where the information is compressed at different levels to mimic teacher assistants of different capacities.

The proposed IBM also has parallels in works like [6, 15]. The former looks at introducing an ensemble of projector networks that map the student embedding to the teacher embedding. Unlike the proposed IBM, which has a rate constrained loss function, the projector networks are trained for reconstruction error only. The concept behind IBM, the Information Bottleneck Principle (IBP), is also related to Masked-Image-Modeling [16] and Masked Generative Distillation [15]. The connections between IBP and improved generalization [17] support the success behind MIM and our proposed IBM. While our IBM loss function attempts to directly upper-bound each of the terms of the IBP objective, MIM upper bounds by dropping negative terms from the IBP objective. On the otherhand,

Teacher

Backbone

Model

Enc

Dec

Noise (

RDM

Student

Backbone

Model

Enc

Dec

Noise during training only

IBM

Student Model

MSE Loss 1

MSE Loss

Total Loss

Student

Embedding

RDM

Embedding

Teacher

Embedding

Input

: pretrained and frozen

: trainable

Rate Loss

MSE Loss 1

MSE Loss

Rate Loss

the MGD objective is an upperbound only under certain modeling assumptions. We discuss these connections in detail in Section 3.2 and Appendix C.

CLIP Distillation: Distillation of CLIP like models is a relatively new direction. One of the earliest works was to use CLIP to distill task specific knowledge into task specific student models [18]. In the task agnostic distillation of CLIP, TinyClip proposed a knowledge distillation and a pruning method for CLIP [19]. Yang et al. studied various losses for distillation including feature distillation and relational losses [20]. Relational losses attempt to maintain the same relative distances between embeddings in the teacher and student model [21, 22]. Another noteworthy paper is MobileCLIP which used an ensemble of teachers and a data refinement technique to obtain powerful small CLIP models [23]. However, the ideas proposed in these works are CLIP specific, i.e., they exploit the interactions between different modalities to compute the distillation losses. Unlike these works, our proposed idea is general, i.e., not CLIP specific.

Rate Distortion and Information Bottleneck via Neural Networks: Neural networks have been used to learn encoders and decoders for compression using Rate-Distortion theory [24, 25, 26] and communication over noisy channels [27]. On the other hand, Information Bottleneck has been used to improve neural networks in areas like improving generalization [28, 29], Out-of-Distribution detection [30, 31] and Out-of-Distribution generalization [32, 33].

3 Controlled Information Flow for KD

3.1 Controlling the information from the teacher using Rate-Distortion Theory

Shannon proposed Rate-Distortion Theory as a principled way to compress a signal. Given an input X , the goal of compression is to find a mapping from X to its compressed version ˆ X such that ˆ X has minimal information about X but at the same time the distortion does not exceed D 0 . We can write this as an optimization problem of the form

where D ( · , · ) is some distortion measure and I ( · ; · ) denotes the mutual information between two random variables [34]. We can convert this to an unconstrained optimization objective of the form

where R determines the trade-off between information rate and distortion. A larger R corresponds to more emphasis on minimizing the distortion at the cost of a higher information rate and viceversa. This idea is best understood in the context of lossy compression where we wish to compress as much as possible while allowing tolerable distortion. However, rate-distortion theory is not only applicable to compression, but also to problems like Joint Source-Channel Coding where the compressed representation is subject to noise [34]. This case is represented in Figure 1(a). The encoded representation of the input is denoted as Y and its noisy version as ˆ Y . The independent noise added to encoded representation is denoted as Z .

However, even though Shannon's theory tells us what optimization problem to solve, it does not tell us how to solve it. When the encoders and decoders in Figure 1(a) are neural networks this problem is compounded because the objective cannot be computed ( I ( X ; ˆ X ) is intractable). Instead, we can use variational approximations to compute an upper bound on the objective, which in turn will allow us to perform gradient descent to learn the encoder and the decoder. Let us denote q ( ˆ Y ) as an approximation of true but unknown distribution of ˆ Y , p ( ˆ Y ) . Then, we can compute an upper-bound on I ( X ; ˆ X ) as

where H ( · ) denotes the entropy, and H q ( ˆ Y ) denotes the cross entropy computed using the distribution q ( ˆ Y ) . The first inequality follows from the Data Processing Inequality [34] and the second follows because cross-entropy is always greater than entropy [34]. Finally, note that H ( ˆ Y | Y ) is constant because the noise is independent of the encoder and decoder parameters.

The key here is to choose the approximating distribution q . Popular choices include the Gaussian distribution (e.g., Variational Autoencoders [35]), learning the distribution [36], or non-parametric approximations [24], we choose the latter. All these methods yield a mechanism where we can

compute q ( ˆ Y ) . Now, for simplicity let us assume that distortion measure D is the L 2 norm, then we can learn the parameters of the encoder ( Θ e ) and the decoder ( Θ d ) by putting (3) into (2) as

Figure 1(a) shows how the RDMs are trained. Since the RDMs process extracted teacher embeddings, they do not have to relearn low level feature extractors from the raw input, thus making them significantly smaller than teacher assistants, which in turn makes them computationally cheaper.

In the case where we are distilling a classification model, an additional linear layer is used to convert the reconstructed embeddings to logits. Let V represent the true classification label, ˆ V T represent the teacher's output predictive distribution on the class labels, ˆ V RDM be the same but as predicted by the RDMoutput. Then, the loss function used to train the RDM is

Here, L CE ( · , · ) is the cross-entropy loss, KL ( ·||· ) is the Kullback-Leibler divergence, and λ KL is weighting factor for the KL loss. For simplicity of writing we have dropped the presence of the temperature τ from the KL divergence term, however, it is assumed to be present. Figure 7 shows the illustration of how the RDM is trained for classification.

Connections between Teacher Assistants and RDMs: TAs limit the amount of information extracted from the input by using limited modeling capacity, i.e., smaller the model poorer the TA performance on the downstream task. On the other hand, we limit the amount of information extracted from the teacher embedding by passing it through a rate constrained communication channel. If we place a higher constraint on the information through the channel, our reconstructed embeddings will have more distortion compared to the teacher's and will perform poorly on the downstream task, just like the embeddings from a TA with a small model capacity. Thus, by choosing different values of R in (4) we can mimic different TAs with different modeling capacities.

3.2 Controlling the information to the student using Information Bottleneck

In the previous section, we introduced RDMs to control the flow of information from the teacher model and mimic teacher assistants. However, when the teacher and multiple RDMs are providing feedback, the feedback can cause the student model to overfit and lead to poor performance. We can overcome this by constraining the information from the student model exposed to the feedback, i.e., provide partial feedback. Similar to the teacher model, we can accomplish this using another rate-constrained channel, but at the student. Further, this rate-constrained model is present only when there is feedback to the student, i.e., only during training. However, unlike the case of RDMs, we are not interested in trying to reconstruct the input to the rate constrained channel. Instead we wish to reconstruct the teacher or RDM embeddings. In Information Theory, the Rate-Distortion problem deals with compressing a random variable. However, when we want the compressed representation to be informative about another variable, it is called the Information Bottleneck problem. Thus, we call our proposed rate constrained module in the student model as Information Bottleneck Module (IBM).

Tishby et al. introduced the Information Bottleneck Principle (IBP) as a generalization to the RateDistortion problem [37]. Given an input X S and some random variable of interest U , the goal in IB is to find a representation ˆ U that removes as much information about input X S while retaining as much information about U . This is formulated as

where, λ I is the Lagrange multiplier.

Figure 1 shows the IB Module in the student model. X S represents the input, W the encoded representation, ˆ W the noisy encoded representation, Z S the noise added during training only, and ˆ U represents the output of the IBM decoder. When working with neural network based encoders and decoders, both the mutual information terms in (6) are intractable. To overcome this setback, we use variational approximations, resulting in an upper bound on the IB objective (6) [28].

where D represents a suitable distortion metric, r ( ˆ W ) is the approximation of the true distribution p ( ˆ W ) . In the IBM, U is set to be the teacher's (or RDM's) embedding. So, at the output the IBM is attempting to reconstruct the teacher's (or RDM's) embedding. In case of a dimensionality mismatch, a projection layer is used. In practice, we do not use a dedicated encoder for IBM and let the student backbone model itself function as the encoder, i.e. W = X S . The decoder is a simple network of at most one to two linear layers. The IBM is not trained separately, instead it is trained along with the student model.

Connections to Masked-Image-Modeling: In this discussion, we compare IBMs with MaskedImage-Modeling [16] and its extension into distillation Masked-Generate-Modeling [15]. In MIM, an input image ( T ) is masked ( T M ). The masked image is fed through an encoder followed by an MIM encoder. The encoder provides an embedding of the image and the MIM encoder uses it to predict the tokens corresponding to the masked out part. The system is then trained to maximize the likelihood of the actual values of the masked out tokens. In Masked-Generative-Distillation, the masking is done after passing through the student backbone. Following that a generator attempts to predict the teacher's embedding (with a slight abuse of notation U ). Let us denote the embedding from the student before the masking as ˆ U -M . Even though both MIM and MGD attempt to maximize the log p ( U | ˆ U ) , we can easily show that this is equivalent to maximizing I ( U ; ˆ U ) (Lemma 1 in Appendix C). Following this we can write the following theorem to connect the MIM and MGD objectives to IBP.

Theorem 1 (Informal) . The objective function of MIM is an upper bound on the objective func-

Tokenizer

258

356

156

752

190

255

428

645

597

168

21

28

465

356

35

23

Encoder

(e.g. BEiT)

MIM Encoder

356

428

465

21

Masked

Tokens

(𝑈)

Predicted Masked Token

Distribution

.

𝑈

Maximize

Likelihood

Input

(𝑇)

Masked

Input

(𝑇

!

)

(a) Masked Image Modeling

Teacher

Student

model

backbone

Generated feature

!

𝑈

Maximize

Likelihood

Input

(𝑇)

Teacher feature

(𝑈)

Mask

generator

Random

mask

Generator

tion from IBP. The objective function of MGD is an upper bound on the objective function from IBP if ˆ U -M is a discrete random variable and the mapping from T to ˆ U -M is deterministic.

Amore detailed explanation along with the formal statement and its proof are provided in Appendix C. The loss function of our IBM is directly written as an upper bound on the IBP objective. Unlike MIM or MGD, we do not drop the second term and instead use a non-parametric upper-bound to approximate it. This empirically should ensure a tighter approximation to the IBP objective than the other two. IBM works better because it forces the student to focus on those features necessary to predict the teacher embedding (first term) and remove information not useful in the prediction (second term), whereas in MIM and MGD the removal of information is either implicit or not present respectively. There have recently been works studying how IBP helps reduce generalization errors [17] which provides support that IBM in the student model should help reduce generalization errors. Our ablation studies in Section 4.3 also show similar results.

3.3 Controlled Information Flow for KD

With all the components in place, we can now derive the final loss function. Using the methodology described in Section 3.1, we assume that N RDMs corresponding to different rate-constraints have been trained. To re-iterate, let X be the input datapoint. Let us denote the teacher model's embedding as X ; ˆ X n , Z n are the embedding and noise in the n th RDM respectively, and ˆ U corresponds to the student model's embedding. Also for sake of illustration, let us assume that the loss to match embeddings is the L 2 loss. Then, we can write the loss function for training the student model as

where λ n are weighting coefficients.

In the case of classification, let us denote the output predictive distribution of the teacher as ˆ V T , ˆ V S for the student, ˆ V n for the n th RDM, and the true label as V . Then, we can write the loss function for training the student model as

3.3.1. CIFD for CLIP style pretraining

CLIP (Contrastive Language-Image Pretraining) is a class of foundational models that are capable of embedding inputs from distinct modalities into a shared embedding space [38, 39]. In CLIP a modality specific encoder processes the input from a specific modality and embeds it into a shared embedding space. CLIP like models have shown tremendous performance in zero-shot classification, object-detection, and retrieval [38, 39]. Further, the trained encoders have also proved instrumental in powering Large Multimodal Models (LMMs) [40, 41, 42] and generative models [43, 44]. So distillation of these models has far reaching applications especially in on-device generative AI.

For simplicity, let us consider two modalities Image ( I ) and Language ( L ). Consider a batch of B image-text pairs { ( I (1) , L (1) ) , . . . , ( I ( B ) , L ( B ) ) } . Let ˆ U ( b ) I denote the L 2 -normalized embedding of the b -th image obtained from the image encoder and ˆ U ( b ) L denote the same for the b -th text obtained from the language encoder (after IBM as in Figure 2(b)). Then we can write the contrastive loss from image to language embeddings as

where ⟨· , ·⟩ represents the inner-product between the two vectors. Using this we can write the loss to train CLIP-like models as

The idea in contrastive loss is that embeddings of a paired image and text must be close to each other (high inner-product) when compared to embeddings of an image and/or text of non-pairs. Going forward we use CLIP to denote any CLIP like model.

Since the CLIP teacher has modality specific encoders, each encoder has its own set of RDMs. Each RDMis trained using (4). Let us denote the embedding for b -th image and text from the teacher by X ( b ) I and X ( b ) L , respectively. We can define the n -th RDM embeddings as X ( b ) n, I and X ( b ) n, L . Similarly, since the student has modality specific encoders, it has modality specific IBMs. Let us define ˆ W I , ˆ W L (similar to ˆ W in Figure 1(a)) for the image and language encoder IBMs respectively. The output of the IBM decoder for the b -th image and text is denoted as ˆ U ( b ) I and ˆ U ( b ) L , respectively. We can now write the modality specific CIFD loss for CLIP distillation as

Now we can write the final loss to perform distillation of CLIP using CIFD as

where λ CL is weighting factor.

4 Experiments

Experimental setup: Our experimental results are split into two sections, one dealing with supervised classification on the CIFAR-100 [45] and Imagenet (IN) [46] datasets, and another with CLIP like

models trained on Conceptual Captions 12M dataset [47]. For the evaluation of the latter models, we conduct zero-shot classification on ImageNet, Imagenet-V2 (IN-V2) [48], Imagenet-A (IN-A) [49], Imagenet-R (IN-R) [50], and Object Net (ObjNet) [51], and we do zero-shot retrieval on COCO [52] and FlickR30k test set [53]. Architecture and training details in Appendix A. Unless otherwise mentioned results from other works are obtained from those papers.

4.1 Supervised Training

In Table 1 we first look at experiments on CIFAR-100 using a simple CNN model to specifically compare against the works of [2, 3] which only study these models. We see that using CIFD with just one RDM boosts performance over KD by +1.41%. However, when using 3 RDMs, we see a difference of +1.92% (absolute) over KD, and a boost of +0.63% over DGKD. DGKD uses three teaching assistants of size eight, six, and four layer CNNs. Since RDMs are less computationally intensive to train than teaching assistants, we trained two more and used them to perform distillation. We see that the resulting student model is +2.49% over KD, and +1.2% over DGKD. We present the more comprehensive results over CIFAR-100 in Table 2. We find that our proposed CIFD performs as good as existing methods.

However, the interest of this paper is more on large scale datasets. In Table 3 we look at the performance of our proposed CIFD methods in eq. (9) for distilling knowledge from a ResNet-34 to a ResNet-18 model and a ResNet-50 to a MobileNet-V1 model over the 1.28 million ImageNet dataset, respectively. We compare against multiple prior works listed in the table. We find that CIFD achieves State-of-The-Art (SoTA) performance, +1.66% (absolute) improvement over KD [1], +0.86% (absolute) over TAKD [2], and +0.5% (absolute) over DGKD [3] on ResNet-18. Looking at the case of ResNet-50 to MobileNet-V1, we find that the proposed method performs slightly above the work of [6] in top-1 accuracy, and achieves the SoTA performance in top-5 accuracy.

4.2 Image Language Pretraining

Finally, to establish the importance of the proposed method in the large dataset regime, we study knowledge distillation for foundational CLIP like models trained over the 12 million image-text pair dataset, CC-12M [47]. In Table 4, we look at the zero-shot classification and zero-shot retrieval results over multiple datasets for CLIP like models. We tested the distillation mechanism over different students and found that our distilled models beat the distillation mechanisms that are specialized to CLIP [19, 20]. To ensure a fair comparison, for TinyCLIP [19], we used only their knowledge distillation related innovation, not pruning (details in Appendix D.4). For ViT-B-16 models,

despite multiple attempts of [19], the training diverged. Looking at the results, in zero-shot classification, our distilled models show superior zero-shot classification over multiple datasets across three different teacher-student combinations. We see as high as 5.3% improvement over the nearest prior work over IN-R dataset on ViT-S-16 (Table 14). Similar trends hold over zero-shot image-to-text and text-to-image retrieval over both the COCO [52] (Table 15) and Flickr30k dataset [53], where our proposed models outperform existing distillation methods by as much as 8% (absolute) for I ✮ T@5 on the COCO dataset on ViT-S-16. We see largest gains in performance when the ratio between teacher to student size is large.

4.3 Ablation studies

Do RDMs mimic Teacher Assistants?

In Figure 5 we show the RDMs used to obtain the results in Table 1. As the information through the RDM is constrained, the RDM's output classification performance also falls. This is because as the information rate is constrained, and the RDM is forced to drop some features that are useful for classification. The corresponding TAs used by [2] in Table 1, have accuracies of 54 . 72% , 54 . 84% , 50 . 32% for model sizes of eight, six, and four layer CNNs resepctively. This trend closely matches the RDM trend in Figure 5. Thus, by varying R we can mimic teaching assistants of different modeling capacities, i.e., lower R we mimic a smaller TA, higher R we mimic a larger TA.

How does number of RDMs affect performance? Does IBM help? In Table 6, we study the effect of the number of RDMs and the effect of IBM in conjunction with the

10

0

10

1

10

2

R

49

50

51

52

53

Acc. (%) on CIFAR-100

RDMs. We find that increasing the number of RDMs without IBM initially improves the performance, but with five RDMs the performance on CIFAR100 drops. We hypothesized that this is due to overfitting, which DGKD proposed to overcome by KD dropout [3]. In KD dropout, the gradients from the teacher and teaching assistants are randomly cut off (Fig. 3 in [3]). Despite incorporating it, the performance without IBM does not show improvement. However, with IBM we see the expected benefits. Further, we see that IBM on its own also improves the performance of KD [1]. This further corroborated on Imagenet as shown in Table 6. The RDMs used in all the experiments correspond to different values of R . This further shows that having RDMs at different information rate R is key to improving performance.

In Table 5, we study the affect of number of RDMs on distilling CLIP like models. To reduce compute, we use a smaller teacher of size ViT-B-16 trained by [58]. We see that increasing number of RDMs leads to improved performance. We also provide results from [20] for comaprison.

Analyzing CIFD gains with large teacher student capacity gap: Here,

we study the performance of CIFD when the teacher student gap is increased. In Table 7 we see that as the size of the teacher (and correspondingly its performance) increases, the student performance also increases. In fact, the increase is montonic w.r.t. the teacher size. We also compare with the performance of DistKD [5] which also studied a similar premise. We find that not only does our method outperform DistKD, that unlike DistKD the proposed CIFD does not show drop in performance when the teacher size is increased. This indicates the robustness of our proposed method. The parameter ratio of teacher to student ranges from 1 . 86 to 5 . 12 . We also study CLIP like models where the maximum parameter ratio is a larger 6 . 9 in Appendix B.4. We see that for 3 out of the 5 zero-shot classification datasets CIFD yields larger improvements over the next best competitor of ClipKD [20] when the capacity gap is larger. In zero-shot image-text retrieval, CIFD almost always yields larger improvements over the ClipKD when the capacity gap is larger. This indicates that CIFD excels when the teacher-student capacity gap is large.

Training cost analysis: Although, prior works on Teacher Assistants like [2, 3] showed promising results, training Teacher Assistants led to prohibitive increase in complexity as seen in Figure 1(b) where TA based methods are 3 . 5 × and 4 . 5 × more expensive than [1] when distilling from ResNet34 to ResNet18 on Imagenet. We computed these numbers based on

the number of Multiply and Accumulations (MACs) for every forward pass, more details in Appendix D.2. On the otherhand, our method is only 1 . 08 × more expensive than [1, 4, 57, 6, 5], thus bringing training cost of TA based methods close to other current innovations. For the CIFAR-100 experiments in Table 1, TAKD [2] is 2 × more expensive and DGKD [3] is 5 × more expensive than proposed.

5 Conclusion

In this paper we present a novel knowledge distillation framework called the Controlled Information Flow (CIFD). CIFD consists of two main components. The first is a lightweight Rate Distortion Module (RDM) that replaces the expensive Teacher Assistants by using the teacher model's embedding and a noisy communication channel. Importantly, since the RDM uses the teacher model's input embeddings, it does not have to relearn low level feature extractors, thus making the model significantly smaller than a TA. Second, we propose an Information Bottleneck Module to prevent the student model from overfitting in the presence of the teacher and multiple RDMs. The resulting framework shows impressive performance on Imagenet and significantly outperforms CLIP specific distillation methods on CLIP models. Finally, we corrobrated that an increased number of RDMs with diverse Rs is a key factor for better distillation, entailing only a small increase in computation.

Limitations and impact: An interesting direction is to study alternative algorithmic formulations for using the information from RDMs, i.e., sequentially enabling and disabling which RDM give feedback at what points of training. Regarding societal impact, the student model depends on the teacher model to transfer concepts. Unfortunately, this also means biases present in the teacher due to its training data are also transferred to the student.

References

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, 'Distilling the knowledge in a neural network,' arXiv preprint arXiv:1503.02531 , 2015.

Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh, 'Improved knowledge distillation via teacher assistant,' in Proceedings of the AAAI conference on artificial intelligence , 2020, vol. 34, pp. 5191-5198.

Wonchul Son, Jaemin Na, Junyong Choi, and Wonjun Hwang, 'Densely guided knowledge distillation using multiple teacher assistants,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 9395-9404.

Yulei Niu, Long Chen, Chang Zhou, and Hanwang Zhang, 'Respecting transfer gap in knowledge distillation,' Advances in Neural Information Processing Systems , vol. 35, pp. 2193321947, 2022.

Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu, 'Knowledge distillation from a stronger teacher,' Advances in Neural Information Processing Systems , vol. 35, pp. 3371633727, 2022.

Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang, 'Improved feature distillation via projector ensemble,' Advances in Neural Information Processing Systems , vol. 35, pp. 12084-12095, 2022.

Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi, 'A comprehensive overhaul of feature distillation,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 1921-1930.

Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai, 'Variational information distillation for knowledge transfer,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 9163-9171.

Ruifei He, Shuyang Sun, Jihan Yang, Song Bai, and Xiaojuan Qi, 'Knowledge distillation as efficient pre-training: Faster convergence, higher data-efficiency, and better transferability,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 9161-9171.

Yichen Zhu, Ning Liu, Zhiyuan Xu, Xin Liu, Weibin Meng, Louis Wang, Zhicai Ou, and Jian Tang, 'Teach less, learn more: On the undistillable classes in knowledge distillation,' Advances in Neural Information Processing Systems , vol. 35, pp. 32011-32024, 2022.

Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu, 'Knowledge diffusion for distillation,' Advances in Neural Information Processing Systems , 2023.

Xuewei Li, Songyuan Li, Bourahla Omar, Fei Wu, and Xi Li, 'Reskd: Residual-guided knowledge distillation,' IEEE Transactions on Image Processing , vol. 30, pp. 4735-4746, 2021.

Mengya Gao, Yujun Shen, Quanquan Li, and Chen Change Loy, 'Residual knowledge distillation,' arXiv preprint arXiv:2002.09168 , 2020.

Jangho Kim, SeongUk Park, and Nojun Kwak, 'Paraphrasing complex network: Network compression via factor transfer,' Advances in neural information processing systems , vol. 31, 2018.

Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan, 'Masked generative distillation,' in European Conference on Computer Vision . Springer, 2022, pp. 53-69.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, 'Beit: Bert pre-training of image transformers,' in International Conference on Learning Representations , 2021.

Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang, 'How does information bottleneck help deep learning?,' in International Conference on Machine Learning . PMLR, 2023, pp. 16049-16096.

Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, and Lu Yuan, 'Clip-td: Clip targeted distillation for vision-language tasks,' arXiv preprint arXiv:2201.05729 , 2022.

Kan Wu, Houwen Peng, Zhenghong Zhou, Bin Xiao, Mengchen Liu, Lu Yuan, Hong Xuan, Michael Valenzuela, Xi Stephen Chen, Xinggang Wang, et al., 'Tinyclip: Clip distillation via affinity mimicking and weight inheritance,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 21970-21980.

Chuanguang Yang, Zhulin An, Libo Huang, Junyu Bi, Xinqiang Yu, Han Yang, and Yongjun Xu, 'Clip-kd: An empirical study of distilling clip models,' arXiv preprint arXiv:2307.12732 , 2023.

Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash, 'Compress: Selfsupervised learning by compressing representations,' Advances in Neural Information Processing Systems , vol. 33, pp. 12980-12992, 2020.

Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu, 'Seed: Self-supervised distillation for visual representation,' in International Conference on Learning Representations , 2020.

Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel, 'Mobileclip: Fast image-text models through multi-modal reinforced training,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 15963-15974.

Johannes Ballé, Valero Laparra, and Eero P Simoncelli, 'End-to-end optimized image compression,' in 5th International Conference on Learning Representations, ICLR 2017 , 2017.

Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston, 'Variational image compression with a scale hyperprior,' in International Conference on Learning Representations , 2018.

David Minnen, Johannes Ballé, and George D Toderici, 'Joint autoregressive and hierarchical priors for learned image compression,' Advances in neural information processing systems , vol. 31, 2018.

Yashas Malur Saidutta, Afshin Abdi, and Faramarz Fekri, 'Joint source-channel coding over additive noise analog channels using mixture of variational autoencoders,' IEEE Journal on Selected Areas in Communications , vol. 39, no. 7, pp. 2000-2013, 2021.

Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy, 'Deep variational information bottleneck,' in International Conference on Learning Representations , 2016.

Junho Kim, Byung-Kwan Lee, and Yong Man Ro, 'Distilling robust and non-robust features in adversarial examples by information bottleneck,' Advances in Neural Information Processing Systems , vol. 34, pp. 17148-17159, 2021.

Alexander A Alemi, Ian Fischer, and Joshua V Dillon, 'Uncertainty in the variational information bottleneck,' arXiv preprint arXiv:1807.00906 , 2018.

Ian Fischer, 'The conditional entropy bottleneck,' Entropy , vol. 22, no. 9, 2020.

Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish, 'Invariance principle meets information bottleneck for out-of-distribution generalization,' Advances in Neural Information Processing Systems , vol. 34, pp. 3438-3450, 2021.

Bin Deng and Kui Jia, 'Counterfactual supervision-based information bottleneck for out-ofdistribution generalization,' Entropy , vol. 25, no. 2, pp. 193, 2023.

Thomas M Cover, Elements of information theory , John Wiley & Sons, 1999.

Diederik P Kingma and Max Welling, 'Auto-encoding variational bayes,' arXiv preprint arXiv:1312.6114 , 2013.

Jakub Tomczak and Max Welling, 'Vae with a vampprior,' in International Conference on Artificial Intelligence and Statistics . PMLR, 2018, pp. 1214-1223.

Naftali Tishby, Fernando C Pereira, and William Bialek, 'The information bottleneck method,' arXiv preprint physics/0004057 , 2000.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al., 'Learning transferable visual models from natural language supervision,' in International conference on machine learning . PMLR, 2021, pp. 8748-8763.

Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra, 'Imagebind: One embedding space to bind them all,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 15180-15190.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, 'Visual instruction tuning,' arXiv preprint arXiv:2304.08485 , 2023.

Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen, 'Multimodal-gpt: A vision and language model for dialogue with humans,' arXiv preprint arXiv:2305.04790 , 2023.

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, 'Minigpt-4: Enhancing vision-language understanding with advanced large language models,' arXiv preprint arXiv:2304.10592 , 2023.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, 'Hierarchical text-conditional image generation with clip latents,' arXiv preprint arXiv:2204.06125 , vol. 1, no. 2, pp. 3, 2022.

Tenglong Ao, Zeyi Zhang, and Libin Liu, 'Gesturediffuclip: Gesture diffusion model with clip latents,' arXiv preprint arXiv:2303.14613 , 2023.

A Krizhevsky, 'Learning multiple layers of features from tiny images,' Master's thesis, University of Toronto, 2009.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, 'Imagenet: A largescale hierarchical image database,' in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248-255.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut, 'Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,' in CVPR , 2021.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar, 'Do imagenet classifiers generalize to imagenet?,' in International Conference on Machine Learning , 2019, pp. 5389-5400.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song, 'Natural adversarial examples,' arXiv preprint arXiv:1907.07174 , 2019.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer, 'The many faces of robustness: A critical analysis of out-of-distribution generalization,' arXiv preprint arXiv:2006.16241 , 2020.

Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz, 'Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models,' Advances in neural information processing systems , vol. 32, 2019.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick, 'Microsoft coco: Common objects in context,' in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Springer, 2014, pp. 740-755.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik, 'Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,' in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2641-2649.

Nikos Komodakis and Sergey Zagoruyko, 'Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer,' in ICLR , 2017.

Yonglong Tian, Dilip Krishnan, and Phillip Isola, 'Contrastive representation distillation,' in International Conference on Learning Representations , 2019.

Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang, 'Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective,' arXiv preprint arXiv:2102.00650 , 2021.

Zhihao Chi, Tu Zheng, Hengjia Li, Zheng Yang, Boxi Wu, Binbin Lin, and Deng Cai, 'Normkd: Normalized logits for knowledge distillation,' arXiv preprint arXiv:2308.00520 , 2023.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt, 'Openclip,' 2021.

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, 'Optuna: A next-generation hyperparameter optimization framework,' in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining , 2019, pp. 2623-2631.

Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-Tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar, 'A system for massively parallel hyperparameter tuning,' Proceedings of Machine Learning and Systems , vol. 2, pp. 230-246, 2020.

Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica, 'Tune: A research platform for distributed model selection and training,' arXiv preprint arXiv:1807.05118 , 2018.

Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry, 'ffcv,' https://github.com/libffcv/ffcv/ , 2022, commit 4dd291a.

Frederick Tung and Greg Mori, 'Similarity-preserving knowledge distillation,' in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 1365-1374.

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho, 'Relational knowledge distillation,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 3967-3976.

Nikolaos Passalis and Anastasios Tefas, 'Learning deep representations with probabilistic knowledge transfer,' in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 268-284.

Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi, 'Knowledge transfer via distillation of activation boundaries formed by hidden neurons,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2019, vol. 33, pp. 3779-3787.

Zehao Huang and Naiyan Wang, 'Like what you like: Knowledge distill via neuron selectivity transfer,' arXiv preprint arXiv:1707.01219 , 2017.

Xiatian Zhu, Shaogang Gong, et al., 'Knowledge distillation by on-the-fly native ensemble,' Advances in neural information processing systems , vol. 31, 2018.

Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim, 'A gift from knowledge distillation: Fast optimization, network minimization and transfer learning,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 4133-4141.

Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy, 'Knowledge distillation meets self-supervision,' in European Conference on Computer Vision . Springer, 2020, pp. 588-604.

Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang, 'Decoupled knowledge distillation,' in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , 2022, pp. 11953-11962.

Xiang Deng and Zhongfei Zhang, 'Comprehensive knowledge distillation with causal intervention,' Advances in Neural Information Processing Systems , vol. 34, pp. 22158-22170, 2021.

Ying Jin, Jiaqi Wang, and Dahua Lin, 'Multi-level logit distillation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 24276-24285.

Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang, 'Curriculum temperature for knowledge distillation,' in Proceedings of the AAAI Conference on Artificial Intelligence , 2023, vol. 37, pp. 1504-1512.

Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao, 'Logit standardization in knowledge distillation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 15731-15740.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev, 'Reproducible scaling laws

for contrastive language-image learning,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 2818-2829.

Laurens Van der Maaten and Geoffrey Hinton, 'Visualizing data using t-sne.,' Journal of machine learning research , vol. 9, no. 11, 2008.

A Implementation Details

A.1 CIFAR-100

A.1.1 Simple CNNs (studied in Table 1)

Table 8: CIFAR-100 Model architecture for simple CNNs. 'C' stands for convolution layer with kernel size 3, stride 1, and padding 1. The number following it indicates the number of filters. 'FC' stands for fully connected layer with the number following it indicating the number of hidden neurons. 'MP' stands for MaxPool. After every convolutional or fully connected layer (except the output layer), we have a ReLU activation.

Table 8 shows the CNN architectures used for the experiments. The output of FC512 from the 10-layer CNN is taken as the teacher's image embedding and passed to the RDM module. The RDM module consists of three hidden layers FC512, FC306, FC100, followed by an output layer of FC100. A bottleneck based on the design of [24] is placed after FC306. The bottleneck adds noise and computes the probability log q ( ˆ Y ) as shown in (4). During inference, i.e., when the RDM is not being trained, for stability, we quantize the representation to integers instead of adding uniform noise. Note that since the uniform noise during training is sampled from [ -0 . 5 , 0 . 5] , statistically, the two are equivalent.

The IBM module is only used in the two layer CNN, the student model. Unlike the RDM, in the IBM we do not design a dedicated encoder or decoder, we allow the preceding and succeeding layers of the model itself to act like the encoder and decoder respectively. The IBM just consists of a layer that adds uniform random noise in the range [ -0 . 5 , 0 . 5] . During inference the noise addition is disabled.

The student model is trained using the loss function (9). We used the Optuna algorithm [59] along with the Asynchronous Hyperband Scheduler [60] in the RayTune package [61] package for hyperparameter optimization. Using this package we optimized, distillation temperature ( τ ), learning rate of the optimizer (SGD), learning rate decay, momentum of the optimizer, dropout (this is the KD dropout proposed in [46]), and all the λ s ( λ KL , λ 1 , . . . , λ 5 , λ IBM ) involved in (9). The weight decay was fixed to 10 -4 . We did the same tuning for both DGKD [3] and TAKD [2]. The RDM model is trained using the loss function (5), without the reconstruction losses. We did similar hyperparamter tuning as above for TAKD [2]and DGKD [3].

A.1.2 Competitive CNNs (studied in Table 2)

In this case, the architectures are standard and we follow it from [4]. The RDM module is a fully connected network with three hidden layers with the bottleneck after the second hidden layer. We train three RDMs with R = 1 . 0 , 0 . 8 , 0 . 6 for 30 epochs.

For training student models, we train with SGD for 240 epochs. Starting learning rate is 0.05 and decayed by a factor of 0.1 at 150, 160, and 180 epochs. We set λ CE = λ KL = λ 1 = λ 2 = λ 3 = 1 , where n ∈ 1 , ..., 3 indexes the three RDMs and KD dropout was set to 0 . 25 . We set τ = 2 . 0 . The weight of the embedding loss (set to 100 . 0 ). We explored λ I ∈ { 0 . 001 , 0 . 005 , 0 . 01 } and set it to 0 . 005 .

Unlike the RDM, in the IBM we do not design a dedicated encoder or decoder, we allow the preceding and succeeding layers of the model itself to act like the encoder and decoder respectively. The IBM just consists of a layer that adds uniform random noise in the range [ -0 . 5 , 0 . 5] . During inference the noise addition is disabled.

A.2 ImageNet

For the experiments related to ImageNet we used the standard model architectures. Similar to CIFAR-100, the RDM module is a fully connected network with three hidden layers with a bottleneck after the second hidden layer. The teacher embedding is accessed after the avgpool. For RDMs for the ResNet34 model, there are three hidden layers of size 512, with activation after the first and third layers only. The embedding is linearly transformed to a logits of a 1000 class classifier to compute the other components in (9). For the ResNet50 RDM, instead of hidden neuron size of 512, we had 1024. The bottleneck follows the mechanism [24]. Instead of MSE loss, the bottleneck is trained with the smooth L1 loss in (5). We split the training data in imagenet into a training and validation set with ratio 0 . 95 : 0 . 05 . We train the RDMs for 30 epochs and monitor the classification accuracy on this validation set. The values of R are 10 4 , 5 × 10 3 , 10 3 .

For training the student model, in (9), we set, λ CE = λ KL = λ 1 = λ 2 = λ 3 = 1 , where n ∈ 1 , ..., 3 indexes the three RDMs and KD dropout was set to 0 . 25 . We set τ = 2 . 0 . Instead of MSE loss for embedding reconstruction, we use the smooth L 1 loss. We search for only two hyperparameters of λ I (set to 0 . 001 ) and the weight of the embedding loss (set to 100 . 0 ). We set the learning rate to 2 . 0 for the SGD optimizer, batch size of 1024 per GPU (4 GPUs), and weight decay to 5 × 10 -5 . We train the system for 88 epochs. For all other hyperparameters, we used the defaults from [62]. We split the training data with a ratio of 0 . 95 : 0 . 05 and used the smaller part as validation for hyperparameter tuning.

Unlike the RDM, in the IBM we do not design a dedicated encoder or decoder, we allow the preceding and succeeding layers of the model itself to act like the encoder and decoder respectively. The IBM just consists of a layer that adds uniform random noise in the range [ -0 . 5 , 0 . 5] . During inference the noise addition is disabled.

A.3 CLIP

We use the package [58] for implementation. We implemented and ran the experiments for the works of both TinyClip [19] and CLIPKD [20]. For TinyClip we only used the Knowledge Distillation part of the proposed idea and not the pruning methodology as pruning is orthogonal to Knowledge Distillation and can be combined with any of the proposed models here. We used a batch size of 1024 (256 per GPU), and a learning rate of 10 -3 . For TinyClip we got better results from using a larger batch size of 6000. The models are trained for 32 epochs, with 3 million image-text pairs per epoch. Warmup is set to around 1/6th of the training. For hyperparameter tuning we use a subset of the CC12 dataset [47]. Hyper-parameters for CLIP involve only two λ I and λ CL . We tried the λ CL suggested value from (Yang et al., 2023) of 0 . 0005 and another 0 . 005 , we chose the latter. For λ I , we tested four values and selected the best based on validation performance. All hyper-parameter tests were restricted to the RN-50 architecture. For all other architectures, the same hyper-parameter settings were used, unchanged.

The RDM training in CLIP models is done for seven epochs, each epoch consisting of 3 million image-text pairs. We use the same settings as training the student model.

B More experimental results

B.1 CIFAR-100

Table 10 compares the results on CIFAR-100 dataset with more baselines. As we can see, our proposed method achieves competitve performance on this dataset.

B.2 ImageNet

Table 11 shows more baselines for experiments on ImageNet. As we can see our proposed method outperforms all the compared methods.

B.3 Zero-shot results for CLIP distilled models

Table 14 shows the zero-shot image text classification results using CLIP like models over five datasets. Table 15 shows the zero-shot image text retrieval results over COCO and FlickR30k test sets. Across all classification and retrieval, our model consistently outperforms CLIP spcific distillation methods. This showcases the generality of the idea proposed. Further, we see the largest gains due to the proposed method when the gap between the teacher and student size is the largest.

B.4 Studying the effect of student-teacher capacity gap in CLIP distilled models

We study the effect of large teacher student capacity gaps in CLIP like models. In Table 12 and Table 13, we compare the difference when a ViT-L-14 teacher was used to train ViT-B-16 and ViT-S16 students. The parameter ratios are 2.8 and 6.9, respectively. CIFD shows greater improvement over baseline when the teacher student ratio is large for 3 out of the 5 zero-shot classification datasets and almost always for the two zero-shot retrieval datasets.

C Connections between IBM and Masked Image Modeling

Masked Image Modeling (MIM), as shown in Figure 6a, has had major success in pretraining large image models like BEiT[16]. Here, T is used to represent the input, T M is used to represent the masked input, U is the tokens of the image corresponding to the masked out parts, and ˆ U is their predicted value as predicted the MIM model. In the BEiT variant of MIM, an image is first converted into tokens using a pretrained tokenizer (like a discrete V AE). Next, a masked version of the image ( T M ) is fed into the MIM encoder which attempts to predict the tokens of the masked parts. We can write the MIM training objective as

Masked Generative Distillation (MGD), as shown in Figure 6b, is an extension of using MIM type of training for distillation. Here, T is used to represent the input, U is the teacher model embedding corresponding to the full image, and ˆ U is their predicted value as predicted the MIM model. In MIM,

the target for prediction are the tokens corresponding to the masked parts of the input. In the case of MGD, the student model is tasked to predict the teacher model's embedding of the original image. MGDhas shown promising results in distillation. The loss corresponding to the Masked Generative Distillation is written as

The loss functions rightfully appear the same. However, the process differs in two key places. First is where the masking is applied. In MIM, the masking is done on the input. In MGD, the masking is done after student backbone preprocessing. Secondly, the likelihood maximization formulation is different, i.e., in MIM, the tokens are discrete classes so a cross-entropy loss is used, whereas in MGD, the teacher embedding is continuous so MSE is used. Thus the likelihood distribution in MIM is multinomial whereas in MGD it is gaussian. Note, for MGD we are only studying the masked distillation loss proposed. The other loss function is not relevant here.

Before comparing with IBP we present a small lemma that minimizing the negative log-likelihood is the same as minimizing the negative of the Mutual Information between two variables.

Lemma 1. Minimizing the negative log-likelhood of predicting a non-learnable U from some learnable ˆ U

is equivalent to

Proof. -I ( U ; ˆ U ) = -H ( U ) + H ( U | ˆ U ) = -H ( U ) -E U, ˆ U [ log p ( U | ˆ U ) ] which follow from definitions of Mutual Information and conditional entropy [34]. Since U is not optimizable, H ( U ) is a constant w.r.t. optimization. Thus the proof follows.

Tokenizer

258

356

156

752

190

255

428

645

597

168

21

28

465

356

35

23

Encoder

(e.g. BEiT)

MIM Encoder

356

428

465

21

Masked

Tokens

(𝑈)

Predicted Masked Token

Distribution

.

𝑈

Maximize

Likelihood

Input

(𝑇)

Masked

Input

(𝑇

!

)

(a) Masked Image Modeling

Teacher

Student

model

backbone

Generated feature

!

𝑈

Maximize

Likelihood

Input

(𝑇)

Teacher feature

(𝑈)

Mask

generator

Random

mask

Generator

(b) Masked Generative Distillation

Teacher

Student

Model (SM)

backbone

Noisy feature

!

𝑈

Maximize

Likelihood

Input

(𝑇)

Teacher feature

(𝑈)

+

Noise

generator

Random

noise

Student Model

Last layer

The IBP objective function, applicable to all systems shown in Figure 6 is written as

For simplicity of exposition we split Theorem 1 into the following lemmas one each for MIM and MGDrespectively.

Lemma 2. The objective function of MIM (14) is an upper bound on the objective function from the Information Bottleneck Principle (18) .

Proof. We start with the Information Bottleneck objective

We note that ˆ U -T M -T forms a Markov Chain. Thus leveraging the Data Processing Inequality [34], we note that I ( T ; ˆ U ) ≤ I ( T ; T M ) . We note that I ( T ; T M ) is constant w.r.t. the minimization, since it does not have any parameters that are optimized by gradient descent. Hence, we can drop that term. By leveraging Lemma 1, we can see that the MIM objective (14) is an upper-bound on the (18).

It is tempting to assume that MGD is also an upper-bound similar to MIM. Unfortunately, this is not the case in general. This is because the noise is added after backbone processing and this processing is optimizable. However, under certain conditions, we show that it is a valid upper-bound.

Lemma 3. Let ˆ U -M denote the random variable prior to the masking operation in MGD (Figure 6b). Then, the objective function of MGD (15) is an upper bound on the objective function from the Information Bottleneck Principle (18) if ˆ U -M is a discrete random variable and the mapping from T to ˆ U -M is deterministic.

Proof. We again start with the Information Bottleneck objective

We note that ˆ U -ˆ U -M -T forms a Markov Chain. Thus leveraging the Data Processing Inequality [34], we note that I ( T ; ˆ U ) ≤ I ( T ; ˆ U -M ) . Unlike the MIM case, I ( T ; ˆ U -M ) is not a constant w.r.t. minimization. By definition of Mutual Information, we can write

where by slight abuse of notation H represents differential or discrete entropy based on if the random variable is continuous or discrete respectively. Since the mapping from T to ˆ U -M is deterministic, it follows that H ( ˆ U -M | T ) = 0 . If we assume that ˆ U -M is discrete, then H ( ˆ U -M ) is discrete entropy and we know that H ( ˆ U -M ) ≥ 0 [34]. Thus, by dropping that term, we get an upper bound on the IBP objective.

Remark 1. If ˆ U -M is not discrete, then H ( ˆ U -M ) < 0 is possible, i.e., differential entropy can be negative. Thus, we cannot guarantee that the MGD objective is always an upper-bound of IBP in that case.

Remark 2. The MGD upper-bound employs two relaxations that make it an upper-bound on IBP. First is the Data Processing Inequality to show that I ( T ; ˆ U ) ≤ I ( T ; ˆ U -M ) . The second (under the discrete assumption) that H ( ˆ U -M ) ≥ 0 .

The loss function of our IBM is directly written as an approximation of the IBP objective as shown in Section 3.2. Unlike MIM or MGD, we do not drop the second term and instead use a non-parametric upper-bound to approximate it. This empirically should ensure a tighter approximation to the IBP objective than the other two. IBM works better because it forces the student to focus on those features necessary to predict the teacher embedding (first term) and remove information not useful in the prediction (second term), whereas in MIM and MGD the removal of information is not present. There have recently been works studying how IBP helps reduce generalization errors [17] which provides support to the proposed idea.

Teacher

Backbone

Model

Enc

Dec

RDM

RDM

Embedding

Teacher

Embedding

Input

Noise (

Teacher

FC

RDM

FC

Teacher Probability

Distribution

RDM Probability

Distribution

: pretrained and frozen

: trainable

MSE Loss

CE Loss

Ground Truth

Label

MSE Loss

Total Loss

CE Loss

Rate Loss

Rate Loss

D More details

D.1 Illustration of training RDMs and Student Models for classification tasks

In Figure 7 we show how the RDMs are trained for classification. This is the analogue of Figure 1(a) where there is no classification involved. In this setup, the input (image as an example here) is passed through the teacher backbone model. The resulting embedding is passed through the RDM encoder, subject to some noise ( Z ) and is reconstructed by the RDM decoder. The RDM in the case of classification is trained like a multi-task learning module, i.e., it is tasked with reconstructing both the input embedding and projecting the reconstructed embedding to perform classification. The RDM is trained in a similar fashion to a student model during knowledge distillation, i.e., there is feature distillation (reconstruct the teacher embedding), logit distillation to preserve the dark knowledge, and a supervised cross entropy loss component. The resulting loss function used to train the RDM can be written as (5) which we re-iterate here.

In Figure 8 we show how the student models are trained for classification in the presence of an RDM and the IBM. For simplicity, we assume there is one RDM. The input image is passed through the teacher backbone model and the obtained embedding is passed through the RDM to get the reconstructed embedding from the RDM. The RDM then provides a predictive distribution, like a teaching assistant, and we can also get the teacher's predictive distribution. The input image is also passed through the student model and the IBM to obtain the student embedding. The student embedding is then subject to feature distillation, i.e., the loss between the student embedding and the teacher embedding, and the student embedding and the RDM embedding is computed. In the case where the student embedding's dimension does not match the teacher or the RDM embedding dimension, a small trainable projector network is used. Finally, the output predictive distribution of the student is subject to both the classification loss and the KL divergence losses w.r.t. the teacher's distribution and the RDM's distribution. So, this mechanism combines feature distillation and the output logit distillation along with the RDM. In the figure, we only show 1 RDM, however, it can easily be extend to multiple RDMs. The loss function used in this case is

Teacher

Backbone

Model

Enc

Dec

RDM

RDM

Embedding

Teacher

Embedding

Input

Noise (

RDM

FC

RDM Probability

Distribution

Student

Backbone

Model

Enc

Dec

Noise during training only

IBM

Student Model

Student

Embedding

Student

FC

Student Probability

Distribution

CE Loss

Ground Truth

Label

MSE Loss 1

MSE Loss 2

: pretrained and frozen

: trainable

MSE Loss 1

Total Loss

CE Loss

MSE Loss 2

Teacher

FC

Teacher Probability

Distribution

Rate Loss

Rate Loss

D.2 Computing training costs

Here, we explain how we compute the training cost for ResNet34 to ResNet18 distillation using CIFD. First, we compute the training costs of RDMs. The RDMs consist of a three layer fully Connected Network, with a computational complexity of 1 . 31 MMACs (Mega or 10 6 MACs) per image per forward pass. We train them for 30 epochs on 1 . 28 × 10 6 images for 30 epochs. Thus the cost of training three RDMs is 151 TMACs (Terra or 10 12 MACs). The significant chunk of computation in training RDMs is coming from running the teacher model in inference mode, which is 142 PMACs (Peta or 10 15 MACs). We can potentially extract the features from the teacher model once and reuse it for 30 epochs, in which case the training cost reduces to 4 . 7 PMACs, but we do not consider that here.

The rest of the 619 PMACs (out of the total 762 PMACs), comes from the student model training. Note that the RDM forward passes are computationally insignificant at this stage, accounting for 0 . 38 PMACs (out of the total 619 PMACs). We do similar computations for [2, 3] while adding the computation for training one Teacher Assistant based on the training settings given in those papers.

Similarly we also compute the training cost for Knowledge Distillation [1]. The training costs of other high performing works [4, 6, 57, 5] are the same as Knowledge Distillation [1].

D.3 Visualizing the Teacher and RDM embeddings

In Figure 9 we study the embeddings obtained from the teacher model and two of the RDM models trained on CIFAR-100 by plotting their tSNE [77]. The RDM in Figure 9b has a test accuracy of 52.32% which is significantly closer to the teacher's accuracy than the other RDM in Figure 9c which has a test accuracy of 48.83%. First let us look at the orange class. The orange class which forms a concentrated cluster in the teacher and the more powerful RDM's embeddings, appears more scattered in the less accurate RDM's embeddings. The less concentrated cluster is easier for the student model to learn because coming up with a good representation that is sufficiently general to cover all examples in a class but at the same time not cover samples in another class is a hard problem. Thus, the RDM effectively bridges the gap by providing easier representation that the student can learn more easily before progressing to the harder representation. Further, we also notice that orange

class 1

40

class2

class3

class 4

20

class5

class6

class7

0

class9

class10

-20

-40

-60

-40

-20

0

20

40

40

20

class1

class2

0

class3

class 4

class5

class6

-20

class7

class9

-40

class 10

-40

20

Q

20

40

(c) tSNE of the RDM model embeddings. Acc. 48.83

%.

40

20

class1

口

class2

class3

class4

class5

-20

class6

class 7

-40

class9

class 10

-40

-20

20

40

and blue clusters that are close to each other in both the teacher and the better performing RDM's embeddings are now farther apart in the poor performing RDM's embeddings.

D.4 Details on CLIP baselines

While comparing with TinyCLIP [19], we disabled pruning, as pruning is orthogonal to our proposed idea. Instead we focused only on the Knowledge Distillation part from [19]. To do that in addition to the contrastive loss (11), we added the two terms as proposed in equations (1), Section 3.1 of [19]. These equations transfer the relative embedding distances from the teacher's embedding space to the student's embedding space.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: Yes, the paper proposes a mechanism for Knowledge Distillation using Teacher Assistants, especially for large datasets. We show multiple experiments as supporting evidence for our proposed method.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss this in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: The formal statements of the propositions and their proofs appear in Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: We provide detailed figures of how different loss functions are computed and the models are trained, see Figure 2, Figure 7, and Figure 8. Along with that we provide details on the hyper-parameters for all experiments in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: As our experiments are implemented based on open-source code and publicly available datasets, we have provided the necessary details in our paper for reproducing the results on top of the public code base and database, with the associated URLs provided. We also provide detailed figures of how different loss functions are computed and the models are trained, see Figure 2, Figure 7, and Figure 8. Along with that we provide details on the hyper-parameters for all experiments in Appendix A.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details are provided in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The experiments in this paper mainly focus on large datasets like Imagenet (1 million samples) and CC12M (12 million samples) that require lots of compute. For these datasets, we follow the best practices of existing papers and report accordingly. CLIP like models take 6 A100 GPU days for training. Given that we already had to train the baseline methods too, training multiple models for all of them would be very compute intensive.

Alternatively, we focus on a large set of inference experiments to showcase our proposed method's superior performance. For example for CLIP like models, we perform zero-shot classification tests on five datasets and zero-shot retrieval on two datasets showcasing superior performance on all of them on three different teacher-student pairs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: We provide those details along with the hyperparameters in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

[Yes]

Justification: We have read and followed the ethics guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We address this in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The paper proposes a method for Knowledge Distillation and uses standard and widely accepted datasets and model architectures.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: The paper uses standard widely accepted datasets. We provide references for all such datasets and code base used where applicable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: [NA] .

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: [NA] .

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSome existing methods alleviate the capacity gap between the teacher and student by setting up Teacher Assistants (TAs), introducing a large number of additional parameters and computational costs. Based on this, this paper proposes to train multiple RDM modules and connect multiple independent classification heads to generate branches with different performance to simulate the TA model. The authors think that this hierarchical model of extracting teacher knowledge can help alleviate the capacity gap between the teacher and student.\n\nSTRENGTHS:\n* Assistants and teachers sharing shallow modules are more efficient in terms of parameter quantity compared to multiple independent Assistants models.\n* Although a large number of fully connected layers have been introduced, the proposed method hardly introduces any additional training overhead.\n\nWEAKNESSES:\n* I noticed that there is a significant difference in baseline performance between Table 1 and the original text, and Table 3 only uses a single Assistant for TAKD and DGKD, while the author's proposed method uses three RDM headers, which is not a fair comparison.\n* There are significant differences in the value of R across different datasets, R=1 for CIFAR but R=10^4 for ImageNet. This means that the selection of hyperparameters on unfamiliar datasets is challenging, and the parameter tuning process may introduce multiple computational costs, which limits the versatility of the method.\n* I noticed that the method proposed by the author introduces and trains at least $3N$ additional layers of MLP, and the forward process and loss calculation cost in distillation is also increased several times. However, the training cost is even the same as the method NormKD based solely on Logits Distillation without additional modules (Figure 1 (b)). Can you present the results with specific numerical values? What is the key to introducing so many parameters without introducing additional computational overhead?\n\nQUESTIONS:\n* The experiment was conducted when there was not much difference between the teacher and student models (common settings in current KD tasks). It cannot prove that the proposed method can alleviate capacity differences, as many methods have proven that such settings (e.g., Res34 $\\to$ Res18) do not require the additional assistant model.\n* Why use the penultimate layer feature of the teacher? Is this from theoretical analysis or empirical summary?\n* For this paper, I think it would be better to place Related Works at the front.\n\nLIMITATIONS:\nThe authors have discussed a limitation in Conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
Inspired by Shannon’s rate-distortion theory, this paper proposes two modules, namely the Rate-Distortion Module and the Information Bottleneck Module, to construct intermediate representations for knowledge distillation. Extensive experiments on various datasets demonstrate the effectiveness of this method.

STRENGTHS:
1. This paper is well-presented and easy to understand.
2. This method not only works for traditional CNN networks but also performs well on modern CLIP models.
3. Extensive experiments demonstrate the effectiveness of this method.

WEAKNESSES:
1. The author's motivation and explanation for TA distillation are not very convincing. In my view, the RDM and IBM proposed in this work can be interpreted as two adapters connected to the teacher and the student respectively for distillation, and the principle is similar to the FT method.
2. In Table 2, 9 and 10, most of the compared methods were published in 2022 or before. The authors are encouraged to compare your method with recent state-of-the-art methods such as MLKD[1], CTKD[2], and LSKD[3].
3. In Eq. 5, I am a little confused about the author's formula representation. Generally speaking, the left side of the comma is the network to be trained, and the right side is the learning target. But the author seems to have it reversed here.

References:  
[1]. Multi-Level Logit Distillation. CVPR 23.  
[2]. Curriculum Temperature for Knowledge Distillation. AAAI 23.  
[3]. Logit Standardization in Knowledge Distillation. CVPR 24.

QUESTIONS:
1. $q(\hat{Y})$ is not clearly marked in Fig. 2. Which part of the network produces it? 
2. It would be more beneficial if the author could add a figure on how to perform CIFD distillation on the CLIP model.

LIMITATIONS:
Please refer to weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new distillation method, CIFD, designed based on *Shannon’s Rate-Distortion theory* and  **Information Bottleneck Principle (IBP)*. CIFD contains Rate-Distortion Modules (RDM) for the teacher to substitute heavy Teacher Assistant (TA) and Information BottelNeck Module (IBM) for the student to mimic the features from several RDMs. Experiments demonstrate the effectiveness of the method.\n\nSTRENGTHS:\n1. The paper is organized well.\n2. The experiments on CLIPs are good, verifying the broader effectiveness of the method.\n\nWEAKNESSES:\nMy main concerns are from three aspects: **i) the story of the paper; ii) the reason why CIFD works; iii) insufficient experiments and comparisons.** Some concerns are mixed among the three aspects. And I will list them one by one.\n\n1. **Insufficient experiments on verifying the basic settings of the paper.**\nThe story starts with *\"When the teacher model is significantly larger than the student, previous works that utilize TAs induce high training costs.\"* I would believe the basic settings of this work as the teacher-student network pairs are in large parameter scale differences. From this point of view, the paper should contain more systematic experiments to verify the efficacy under this setting. Specifically, CIFD should be compared with previous methods on ImageNet with teacher-student network pairs with large different parameter scales, not just traditional ResNet-34 -> ResNet-18 and ResNet-50 -> MobileNet-V1.\n\n2. **The trade-off between the story and the empirical solutions.** In my opinion, the paper is a little bit overdecorated and overclaimed. The author proposes many concepts, such as *Shannon’s Rate-Distortion theory* and  *Information Bottleneck Principle (IBP)*, and claims **\"This is the first application of Shannon’s Rate-Distortion theory to aid knowledge distillation\"**. I don't mean that the aforementioned statement is misleading.  But, if we go deeper into the design, the reason why the method works may come from the **noise-adding and noise-removing process**. Many previous works have verified that the above process could benefit the learning process in computer vision, like MIM and diffusion models, which have been empirical solutions.   In KD, there also exist distillation methods following MIM and diffusion models, like MGD and DiffKD. From this point of view, the authors should not claim ***\"the first\"*** only, but make a deeper analysis of related methods and make detailed comparisons. ***I strongly encourage the authors to make a good balance between the story and the verified empirical solutions.*** Even though it seems not as novel as this version, it would provide the readers with more useful knowledge and insights.\n\n3. **The design may alter the network architecture of the student.** It seems that the IBM module would also be included in the validation stage. If my judgment is true, the added module (though lightweight) would also benefit the performance. Under such circumstances, the comparisons with previous methods, especially for light-weight models, are unfair.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nLimitations are included in the main paper and the Appendix", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}