Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation

Authors: yu-liang  zhan
zhong-yi  lu
hao  sun
ze-feng  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:31:54

Abstract: Increased training parameters have enabled large pre-trained models to excel in various downstream tasks. Nevertheless, the extensive computational requirements associated with these models hinder their widespread adoption within the community. We focus on Knowledge Distillation (KD), where a compact student model is trained to mimic a larger teacher model, facilitating the transfer of knowledge of large models. In contrast to much of the previous work, we scale up the parameters of the student model during training, to benefit from over-parameterization without increasing the inference latency. In particular, we propose a tensor decomposition strategy that effectively over-parameterizes the relatively small student model through an efficient and nearly lossless decomposition of its parameter matrices into higher-dimensional tensors. To ensure efficiency, we further introduce a tensor constraint loss to align the high-dimensional tensors between the student and teacher models. Comprehensive experiments validate the significant performance enhancement by our approach in various KD tasks, covering computer vision and natural language processing areas. Our code is available at https://github.com/intell-sci-comput/OPDF.

Main paper: 
 introduction: 
 Large-scale pre-trained models are gradually achieving remarkable milestones due to the exhibit of remarkable performance across various tasks [1] [2] [3] [4] [5] [6] [7] . These models leverage extensive pre-training data and parameters, enabling them to effectively encapsulate a significant breadth of world knowledge [8, 9] and exhibit strong generalization capabilities across diverse tasks [1, [10] [11] [12] [13] . Following this trajectory, the utilization of increased data and parameters has emerged as a notable trend in enhancing the performance of pre-trained models in recent years, leading to the number expansion of pre-trained model parameters from millions to billions [4, 14, 15] .Despite their impressive performance, the substantial storage demands and high computational complexity hinder the practical deployment of these models in real-world applications. Therefore, on the one hand, some studies focus on pre-training relatively smaller models (such as BERTbase-uncased [2] ) on domain-specific or task-specific corpora [16] [17] [18] . However, due to the lesser over-parameterization of small models compared to large ones, their generalization capability often falls short, resulting in suboptimal fine-tuning performance on downstream tasks. On the other hand, model compression methods, such as pruning less informative parameters [19] [20] [21] or utilizing knowledge distillation (KD) [22] to transfer knowledge from larger models (teachers) to smaller ones (students), have been proposed. KD has swiftly diversified into numerous branches, primarily falling into two categories: i.e., logits-based [22] [23] [24] [25] [26] and features-based [27] [28] [29] [30] depending on the source of student model knowledge. Nevertheless, as student models have fewer trainable parameters and limited capacity, a significant performance gap remains between student and teacher models.To address the disparity between small and large models, this study aims to over-parameterize small student models as large ones during distillation training to enhance their generalization capability. Typically, most parameters of student models are stored as matrices. Through tensor decomposition techniques [31] [32] [33] [34] (e.g., Singular Value Decomposition), each matrix can be factorized into a set of matrices, effectively increasing the total number of parameters during distillation. Moreover, after convergence, the factorized matrices can be merged to reorganize the parameter matrix of the student model. This paradigm leverages the benefits of over-parameterization during training without increasing the inference latency of student models.However, incorporating tensor decomposition into over-parameterizing student models poses two major concerns that must be addressed. First, the potential information loss caused by tensor decomposition should be minimized, as small computation errors may accumulate and propagate exponentially within the stacked layers of student models. Second, in the over-parameterized student models, there is no effective mechanism to ensure the consistency of information between student and teacher models. Therefore, it is essential to choose appropriate tensor decomposition methods and design loss functions for high-order tensors to ensure the effective transfer of information from teacher to student models.To address the above issues, we introduce the matrix product operator (MPO) [34] technique as the tensor decomposition strategy. The MPO decomposition, widely used in quantum many-body physics, efficiently factorizes any matrix with arbitrary dimensions into a set of higher-dimensional tensors, which can reconstruct the original matrix in almost lossless conditions [34] [35] [36] [37] . These advantages make MPO an ideal method for over-parameterizing student models during distillation. Based on MPO, we also devise high-order tensor alignment losses for student and teacher models to ensure the effective transfer of information in tensor representation. Therefore, in this paper, we propose a general Over-Parameterization Distillation Framework, namely OPDF, to improve the performance of knowledge distillation. Given the parameter matrices of a student model, we first over-parameterize them through MPO decomposition and then utilize high-order tensor alignment losses to ensure efficient information transfer. This framework only modifies the distillation training process, making it applicable to various student models and natural language processing (NLP) and computer vision (CV) tasks. We conduct extensive experiments in both NLP and CV domains. Experimental results demonstrate that our OPDF significantly enhances the effectiveness of model distillation, e.g., improving BERT-base KD +1.6 on average. Moreover, our approach also enables the student model to achieve performance nearly on par with the teacher model, e.g., AD-KD+Ours (83.4) v.s. BERT-base (83.4) in average metric on GLUE. 
related work: 
 Large Scale Pre-trained Models Large-scale pre-trained models have achieved remarkable success in many fields (e.g., natural language processing (NLP) [38] and computer vision (CV) [15, 39] ). Since the introduction of the Transformer architecture [40] , the pre-training and fine-tuning paradigm in NLP, exemplified by models like BERT [2] and T5 [4] , has shown outstanding performance across multiple tasks. Furthermore, the emergence of models like GPT-3 has demonstrated that increasing model size can significantly improve performance on low-resource tasks [12] . In the field of computer vision, models based on Transformers, such as ViT [7] , have also performed exceptionally well. In our research, we improve the distillation process by increasing the parameters during the training phase of the student model, without introducing additional inference latency to the student model. Knowledge Distillation Knowledge Distillation (KD) methods are commonly used to compress models by transferring knowledge from a larger teacher model to a smaller student model. Building upon the initiative work by [22] , the researchers have exploited the logits follows up with different techniques in the computer vision field, e.g., minimizing KL-Divergence (DKD [25] ) or a Pearson correlation (DIST [26] ). Logit-based methods have been also popular in NLP [41, 42] . Features-based methods have tried to align the features from intermediate layers of teacher and student models and minimize the differences [43] . After the intermediate representations have been introduced [27] , a mount of features-based KD methods have been proposed to match the features, such as LGTM [44] , DBKD [45] and AD-KD [46] . However, the capacity gap between the teacher and student models makes it difficult to imitate the hidden representations of the teacher [47] . Different from these existing KD methods, our proposed OPDF has utilized MPO decomposition to over-parameterize the student model in the training procedure to improve the student model generalization capability, which can minimize the capacity gap efficiently.Matrix Product Operators Matrix Product Operators (MPOs) [34, 48] , also known as tensor-train operators (TTOs) [33] , have been proposed for a more efficient representation of the linear structure of neural networks [49, 50] . A large number of typical applications have utilized MPO-based methods to compress linear layers [51] and convolutional kernels [52] in the parameter matrices of deep models. Furthermore, existing works have applied the MPO method for lightweight fine-tuning of ALBERT [35] , the efficient expansion for the MoE framework [36] , the over-parameterization tuning process for PLMs [37] , construct efficient PLM architecture [53, 54] and compressing datasets [55] . Unlike existing methods, our approach focuses on utilizing MPO decomposition to map parameters from low-dimensional spaces to high-dimensional spaces, to over-parameterize the student model during the distillation process, allowing the student model to benefit from more parameters and achieve better distillation results.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Section 5.3. There are inherent limits to the benefits that can be achieved through over-parameterization in knowledge distillation models. The learning rate should be chosen more carefully as the scale of over-parameterization increases.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The answer NA means that there is no societal impact of the work performed. 
preliminary: 
 Tensor Product We denote a tensor T i1,i2,...,in as an array with n indices, where {i 1 , i 2 , . . . , i n } denotes the dimensions of the n indices, respectively. In this manner, a vector (i.e., v) can be considered a 1-order tensor, while a matrix (i.e., W) can be regarded as a 2-order tensor. Consider ψ 1 , . . . , ψ p and ϕ 1 , . . . , ϕ q as the orthonormal bases of tensors T (1) and T (2) , respectively. The tensor product, denoted as ⊗, can be obtained through the contraction of T (1) and T (2) . Formally, the tensor contraction of T (1) = p i=1 a i ψ i1 and T (2) = q j=1 b j ϕ i2 is defined as follow:EQUATIONThe set ψ i1 ⊗ ϕ i2 constitutes the orthonormal basis of the resulting vector Hilbert space, with the dimensionality of this Hilbert space being the product (i.e., p × q) of T (1) and T (2) .Tensor Decomposition Tensor decomposition can be viewed as the reverse operation of the tensor product. A commonly employed approach is the singular value decomposition (SVD) algorithm. Given a tensor T ∈ R • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
method: 
 In this section, we describe our proposed over-parameterized distillation framework. We first outline our approach, then introduce the details of matrix product operator decomposition and the overparameterized student model strategy, and finally present the tensor alignment loss.Current distillation methods primarily enhance the performance of student models by introducing constraints on logits or features between the student and teacher models. In contrast to these methods, our approach not only utilizes tensor decomposition to over-parameterize the student model for performance improvement but also designs alignment loss functions for the decomposed high-order tensors to further enhance the performance of the student model. To achieve this goal, we employ a tensor decomposition method to decompose the parameter matrices of the teacher and student models into a series of high-order tensor products. These high-order tensors can be used to reconstruct the original parameter matrices while significantly increasing the number of trainable parameters in the student model. After reconstruction, the student model has the same number of parameters as the original matrix without increasing inference time and model size. Additionally, by introducing distillation loss functions to allow the student model to learn from the teacher model in tensor representation, the effectiveness of knowledge distillation is further enhanced.In our proposed over-parameterized distillation framework, we integrate a tensor decomposition strategy based on MPO into the student model to enlarge the parameter matrix (Section 4.2). Furthermore, we design a tensor alignment loss function to enhance the performance of the student model in the context of knowledge distillation (Section 4.3). An overview of our approach is depicted in Figure 1 . We also provide a detailed description of our over-parameterized distillation framework in Algorithm S.1.To leverage the advantages of over-parameterization during knowledge distillation, our method utilizes the MPO, a tensor decomposition technique that increases the number of model parameters. In this part, we initially present the specifics of the MPO method and subsequently outline its adaptation for over-parameterizing the student model.The MPO decomposition is an efficient algorithm capable of factorizing a parameter matrix W ∈ R I×J into a sequential product of multiple tensors [34] . Formally, given a matrix M ∈ R I×J , its MPO decomposition into a product of n local tensors can be represented as:EQUATIONThe tensorT (k)[d k-1 , i k , j k , d k ] is a 4th-order tensor with dimensions d k-1 × i k × j k × d k ,wheren k=1 i k = I, n k=1 j k = J, and d 0 = d n = 1.To link two sequence tensors, we have adopted the concept of a bond following the work of [48] . The bond dimension d k is defined by:EQUATIONFrom Eq. (3), we can see that d k will be large in the middle and small on both sides. Following [35] , we refer to the tensor right in the middle as central tensor, and the rest as auxiliary tensor. Over-parameterzing Student Model. Utilizing the MPO method within the framework of knowledge distillation, our objective is to extend the parameter scale of the student model, capitalizing on over-parameterization. More specifically, we can employ the MPO method to break down a portion of the parameter matrices into multiple tensors as illustrated in Eq. ( 2). Following MPO decomposition, the parameter count of the matrix W will increase based on the values of {d k } m k=1 , {i k } m k=1 , and {j k } m k=1 . The precise augmentation in parameter count, denoted as N add , can be computed as follows:EQUATIONTherefore, during the knowledge distillation procedure, we can adopt MPO on student model parameter matrices to generate their corresponding multiple tensors. In this way, we can scale up the total parameter of the number of the student model without increasing its inference time consumption. After training the over-parameterized student model to convergence, we will perform tensor contraction on these decomposed tensors, to reconstruct the parameter matrices of the student model in almost lossless conditions which is detailed in Appendix B. This new student model has the same parameter number and inference latency as the original one and has benefited from over-parameterization during training.Revisiting Prediction Match of Knowledge Distillation Traditional knowledge distillation involves two stages: fine-tuning the teacher model for a specific task, followed by training strategies to constrain the student model to closely approximate the teacher model. These processes aim to transfer the knowledge from the teacher to the student model. Recent studies have mainly focused on directly learning from the features and logits of the teacher model to transfer crucial knowledge [23, 56] .However, these methods are limited by the capacity of the student model due to the limitation of total parameters. Moreover, this distillation approach based on cross-entropy loss constraints may lead to the student model losing its ability to learn independently. We aim to design a novel model distillation framework to enable the student model not only to effectively learn the knowledge from the teacher model but also to maintain its ability to learn independently.Distillation Loss for Auxilary Tensors. To achieve the goal of "learning knowledge from the teacher model while maintaining the ability of the student model to learn independently," we introduce a high-order tensor alignment training method based on the MPO decomposition. A crucial merit of MPO decomposition is its ability to reorganize and aggregate the core information, decomposing the weight matrices into a central tensor (containing a large number of parameters and important information) and auxiliary tensors (containing fewer parameters and additional information to the central tensor) [35, 36] . Therefore, in the knowledge distillation, in addition to minimizing the cross-entropy loss concerning the ground truth, we add a loss constraint for aligning the auxiliary tensors between the student and teacher models:EQUATIONwhere the matrices A t,k and A s,k refer to the auxiliary tensor of student and teacher models with the same dimensions respectively. MSE means the mean-square error loss function. To ensure that the student model learns from the teacher while preserving its central tensor for independent learning, we minimize the mean-square error loss between the auxiliary tensors of both the student and teacher models. Since this distillation framework is based on improvements to the weight matrices, it is orthogonal to most current distillation methods. Therefore, it can further enhance the distillation effectiveness based on existing distillation methods (as thoroughly discussed in the experimental section). Hence, it can be widely applied to various knowledge distillation models. 
experiments: 
 In this section, we assess the efficacy of our approach within two renowned domains: computer vision and natural language processing. Notably, the OPDF is designed to complement existing distillation techniques. Consequently, we apply our proposed OPDF with various standard distillation methods to validate its effectiveness. In the subsequent section, we detail our experimental setup's datasets and baseline methods. We then present the primary results achieved with the OPDF and provide a thorough analysis. Furthermore, we examine the influence of the degree of over-parameterization, MPO strategy and the learning rate on the performance of OPDF. We report the memory and time cost of experiments in Appendix D.1.Datasets and Metrics For NLP tasks, we evaluate our approach on text classification tasks in GLUE benchmark [57] . The tasks encompassed in our evaluation include RTE, MRPC, STS-B, CoLA, SST-2, QNLI, QQP, and MNLI. To facilitate comparison with baselines, we employ the F1 score and accuracy as metrics for MRPC and QQP, Matthew's correlation coefficient for CoLA, and the average of Pearson and Spearman correlations for STS-B. Accuracy is used as the metric for the remaining tasks, with the result for MNLI reported as the average across the matched (MNLI-m) and mismatched (MNLI-mm) domains. Additionally, we calculate the average score across all tasks to provide a comprehensive performance measure. In the context of CV tasks, we have applied the OPDF to the distillation of Vision Transformers (ViT) for image classification [7] . This was done using the ImageNet-21k dataset [58] , ImageNet-1k, ImageNet Real [59] , and ImageNet V2 [60] datasets. For these datasets, we use accuracy as the primary evaluation metric.Baseline Methods For NLP tasks, we implement OPDF on previous KD methods: BERT-of-Theseus [56] ,LGTM [44] , DBKD [45] and AD-KD [46] . We replicated the baselines using the publicly released code to assess their performance on the test set. Additionally, LGTM was not previously evaluated across all tasks in its original publications, and we have addressed this omission using the provided code. It is important to note that DBKD is designed to estimate logits from decision distributions [45] , and therefore we do not report performance on the STS-B task. For all experiments in natural language processing, we demonstrate the effectiveness of our method during the fine-tuning stage. We implement the teacher model as the fine-tuned "BERT-base-uncased" model [2] . In the context of CV tasks, TinyViT [61] , which introduces a rapid pre-training framework, has emerged as a classical distillation method for ViT. The original paper on TinyViT discusses three versions of the model with varying parameter counts: TinyViT-5M, TinyViT-11M, and TinyViT-21M.To incorporate high-order tensor alignment loss into the distillation phase, we utilize CLIP-VIT-L/14 [7, 62] , a variant of ViT, as the teacher model in our experiments. To assess the efficacy of OPDF, we pretrain the distillation model on ImageNet-21k and evaluate its linear probe performance on ImageNet-1k, ImageNet Real, and ImageNet V2, without any fine-tuning. During the pre-training stage, we adhere to the same experimental settings as described in the original paper. Furthermore, we juxtapose our method with SVD [32] , a traditional tensor decomposition technique viable for over-parameterizing student models. Concretely, we employ SVD to substitute MPO within our framework and execute over-parameterization across all parameter matrices of the student model during knowledge distillation. Appendix D.2 shows more experimental details.We present the results on BERT in Table 1 . Firstly, it is evident that integrating KD with over-parameterization methods yields the most significant performance enhancements. Over-parameterization enhances the generalization ability of the student model. Upon comparing the two tensor decomposition techniques, we find that MPO consistently outperforms SVD. This discrepancy arises from the singular value-based SVD in a two-dimensional space, limiting its ability to substantially increase model parameters compared to MPO decomposition (e.g., 90M vs. 160M in BERT-of-Theseus). In contrast, MPO allows for arbitrary scaling by increasing the order of decomposition, rendering it more suitable for over-parameterization. Secondly, following the integration of the OPDF method, the performance of prior KD techniques (BERT-of-Theseus,LGTM, DBKD, and AD-KD) have exhibited enhancements across a majority of tasks (e.g., RTE, MRPC, CoLA, QQP), while maintaining comparability with the original method in other tasks. This highlights the versatility of OPDF, demonstrating its effectiveness across diverse models and a wide range of tasks. Finally, our findings have revealed that employing the OPDF method can even outperform the performance of the teacher model in MRPC and RTE datasets. This indicates that the process of over-parameterization endows the student model with stronger generalization capabilities, suggesting that employing over-parameterization may offer a potential solution to the bottleneck in current distillation methods where the performance of the student model fails to surpass that of the teacher model.CV Tasks All CV results of our proposed method are shown in Table 2 . We apply OPDF on three kinds of TinyVit with different total parameters. It is clear that with OPDF, the performance of TinyVit can be significantly improved. In particular, in all datasets, TinyVit applied OPDF is better than vanilla TinyVit. Moreover, TinyVit utilized OPDF with 11M parameters can achieve better performance than TinyVit with 21M parameters. It demonstrates that OPDF is an orthogonal method for various KD methods based on the Transformer whether in the CV or NLP field. Note that since we only involved the over-parameterization procedure in the training phase, the total parameter of the student model will not change in the inference phase. This merit makes the OPDF unique from the existing KD method: one would not increase inference time while enhancing model accuracy and enabling the model to acquire more knowledge from the teacher model. Moreover, we can observe that the performance of the original TinyVit, SVD over-parameterization, and OPDF over-parameterization improves as the number of parameters gradually increases. This indicates that compared to SVD, the MPO decomposition, which can decompose the parameter matrix to any size, can better enhance the expressive capacity of the student model. The impact of the over-parameterization scale on distillation effectiveness will be analyzed in detail in Section 5.3.Performance Comparison w.r.t. Parameter Increasing Rate. Our OPDF method facilitates the flexible expansion of model parameters, thereby highlighting the significance of the parameter increase rate on model performance. Consequently, we investigate the influence of this rate on model efficiency further. To underscore the general applicability of our findings, we intentionally overparameterize two models: DBKD and LGTM. We then elucidate their relationship with fine-tuning performance on MRPC tasks. All results are depicted in Figure 2(a) .It is observed that the performance of both the LGTM [44] and DBKD [45] models on the MRPC task consistently improves with an increase in parameters. This enhancement substantiates the efficacy of using the OPDF for over-parameterizing models, which in turn significantly boosts the performance of knowledge distillation models. Furthermore, after over-parameterization, the performance of the models is capable of achieving, at a minimum, the level of their original benchmarks (e.g., 83.3 for DBKD and 86.3 for LGTM). The enhancement of model performance through over-parameterization has its limitations. As demonstrated in Figure 2 (a), beyond certain thresholds of over-parameterization (e.g., 1.6× for DBKD and 2.5× for LGTM), the performance improvements of the models no longer exhibit significant gains. This observation indicates that there are inherent limits to the benefits that can be achieved through over-parameterization in knowledge distillation models. These limits are likely influenced by structural characteristics of each model and size of the initial model configuration.Hyper-parameters Tuning OPDF decomposes the original weight tensor through overparameterization, leading to the updating of more parameters. Consequently, the tensor product results in larger updates to the existing parameters in the backward phase. In Figure 2 (b), we illustrate the relationship between the performance on the MRPC task and learning rate when the parameters of the DBKD model are expanded to 1.2×, 1.4×, and 1.6× their original size.There exists an optimal learning rate for every scale of over-parameterization. Deviating from this optimal rate, whether by increasing or decreasing the learning rate, results in diminished model performance. The reduction in performance due to a lower learning rate can be attributed to the model becoming trapped in a local optimum.Additionally, we observe that peak model performance consistently increases with the scale of over-parameterization. This finding aligns with the conclusions drawn from Figure 2 (a). Moreover, as the scale of over-parameterization increases, the learning rate required to achieve optimal model performance decreases. This occurs because using the tensor product to restore the shape of the tensors to that of the original weight tensors also scales the updated values, resulting in significant changes. Consequently, an increasing learning rate leads to declining performance in the KD model, indicating that the learning rate should decrease as the over-parameterization scale increases.Finally, despite changes in the learning rate, the performance of the model with OPDF consistently remains at least as high as that of the original method. This indicates that OPDF is not sensitive to learning rate variations during the distillation stage. This resilience is due to OPDF's ability to factorize the parameter matrix in almost lossless conditions, ensuring that the decomposed matrix can match or exceed the training effectiveness of the original matrix without introducing errors.To demonstrate the robustness of our MPO methods, we applied different MPO methods to the DBKD and AD-KD model on the RTE, MRPC, STS-B, CoLA, and SST-2 task. The experimental results are presented in Table 3 . To maintain consistent over-parameterization scales, we used the same decomposition scale (L) for each KD model across the same task. We can observe that the performance of our approach consistently stabilizes around certain values, indicating that our method is not sensitive to the specific MPO techniques used. Therefore, when over-parameterizing, we should focus primarily on the decomposition scale rather than the MPO method employed.Ablation Study Our approach consists of two novel improvements: (1) the over-parameterization procedure for the student model, (2) the distillation loss for auxiliary tensors for effective training. To verify the effectiveness of each component, we conduct the ablation study on the GLUE benchmark to analyze the contribution of each part. We consider removing over-parameterization and distillation loss respectively. The ablation results of our OPDF are shown in Figure 2(c ).Firstly, it is clear that regardless of the over-parameterization method used, the area of the radar chart is greater than that of the vanilla theseus. This outcome suggests that over-parameterization can greatly improve the performance of distillation methods. Secondly, further analysis of the different over-parameterization methods reveals that MPO consistently outperforms SVD across all datasets. This improvement is attributed to MPO's ability to decompose parameter matrices into higher orders, effectively enlarging the size of the parameter matrix. Lastly, we examine the contribution of the L Aux term. The radar chart area is significantly larger when OPDF is utilized in conjunction with L Aux than with MPO alone. This indicates that L Aux effectively enhances knowledge transfer from the teacher model. The underlying reason for this phenomenon is that over-parameterized models can concentrate on learning central tensors containing critical information, while the L Aux term assists in aligning auxiliary tensors. We can see that removing any component would lead to a decrease in the model performance. It shows the effectiveness of all these components in our approach. 
conclusion: 
 In this paper, we proposed OPDF, a novel over-parameterization distillation framework designed to enhance the effectiveness of knowledge distillation. This framework employs MPO as a tensor decomposition technique to expand small models into larger ones, thereby bridging the capacity gap between the teacher and student models. Moreover, to enhance the effectiveness of knowledge distillation, our proposed OPDF framework introduces a tensor constraint loss. The OPDF framework utilizes MPO to decompose each weight matrix into a central tensor and auxiliary tensors. By aligning the auxiliary tensors, OPDF not only facilitates the transfer of crucial knowledge from the teacher model but also preserves the student model's ability to think independently. This approach provides the student model with the potential to outperform the teacher model. Our ablation studies demonstrated that all components of the OPDF contribute to enhancing the effectiveness of knowledge distillation. Experimental results across various tasks in natural language processing and computer vision domains validate the efficacy of our proposed method in improving model distillation. Although the number of parameters was increased by MPO during training, the factorized matrices can be merged to reorganize the original parameter matrix in almost lossless conditions. This means that OPDF can enhance the performance of the distillation model without increasing the inference latency. Moreover, since OPDF is based on tensor decomposition, it is orthogonal to most distillation methods.In our future work, we will investigate more efficient and effective tensor decomposition methods for student model over-parameterization. In addition, we will also apply OPDF to other important backbone models, such as in the multimodal learning domains. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Guidelines: The paper poses no such risks.• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package or dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This paper proposes a novel knowledge distillation framework for model compression field, which is helpful to reduce storage requirements and computational complexity. This method facilitates the practical deployment of models in real-world applications and supports energy conservation. We focus exclusively on over-parameterizing small student models, presenting no potential ethical risks.As introduced in [34] , the concept of a tensor is specified as: Definition1 (Tensor). Let D 1 , D 2 ..., D N ∈ N denote index upper bounds. A tensor T ∈ R D1,...,Dn of order N is an N -way array where elements T d1,d2,...,dn are indexed by d n ∈ {1, 2, ..., D n } for 1 ≤ n ≤ N(Matrix product operator). We can reshape a matrix to high order tensor, denoted as: M x×y = M i1i2...in,j1j2...jn (S.1) Here, the one-dimensional coordinate x of the input signal x with dimension N x is reshaped into a coordinate in a n-dimensional space, labeled by (i1 i 2 • • • i n ).Hence, there is a one-to-one mapping between x and (i1 i 2 • • • i n ).Similarly, the one-dimensional coordinate y of the output signal y with dimension N y is also reshaped into a coordinate in a n-dimensional space, and there is a one-to-one correspondence between y and (j 1 j 2 • • • j n ). If I k and J k are the dimensions of i k and j k , respectively, thenn k=1 I k = N x , n k=1 J k = N y . (S.2)The MPO representation of M is obtained by factorizing it into a product of n local tensorsM i1•••in,j1•••jn = T (1) [i 1 , j 1 ] • • • T (n) [i n , j n ] (S.3)where T (k) [j k , i k ] is a D k-1 × D k matrixTheorem 1. Suppose that the tensor W (k) of matrix W that is satisfyW = W (k) + E (k) , D(W (k) ) = d k ,where||E (k) || 2 F = ϵ 2 k , k = 1, ..., d -1.(S.4) Then M P O (W) with the k-th bond dimension d k upper bound of truncation error satisfy:||W -M P O (W)|| F ≤ d-1 k=1 ϵ 2 k (S.5) P roof.The proof is by induction. For n = 2 the statement follows from the properties of the SVD. Consider an arbitrary n > 2. Then the first unfolding W (1) is decomposed asW (1) = U 1 λ 1 V 1 + E (1) = U 1 B (1) + E (1) (S.6) where U 1 is of size r 1 × i 1 × j 1 and ||E (1) || 2 F = ϵ 2 1 .The matrix B 1 is naturally associated with a (n -1)-dimensional tensor B (1) with elements B (1) (α, i 2 , j 2 , ..., i n , j n ), which will be decomposed further. This means that B 1 will be approximated by some other matrix B1 . From the properties of the SVD it follows that U T 1 E (1) = 0, and thus||W -B (1) || 2 F = ||W 1 -U 1 B1 || 2 F = ||W 1 -U 1 ( B1 + B 1 -B 1 )|| 2 F = ||W 1 -U 1 B 1 || 2 F + ||U 1 ( B1 -B 1 )|| 2 F (S.7)and since U 1 has orthonormal columns,||W -B (1) || 2 F ≤ ϵ 2 1 + ||B 1 -B1 || 2 F . (S.8)and thus it is not difficult to see from the orthonormality of columns of U 1 that the distance of the k-th unfolding (k = 2, ..., d k -1) of the (d -1)-dimensional tensor B (1) to the d k -th rank matrix cannot be larger than ϵ k . Proceeding by induction, we have||B 1 -B1 || 2 F ≤ d-1 k=2 ϵ 2 k , (S.9)combine with Eq. (S.8), this complets the proof.The over-parameterized distillation framework algorithm is shown in Algorithm S.1.Algorithm S.1 Over-parameterized distillation framework.Input: The parameter matrices list of student model {Ms(k)} n k=1 , the parameter matrices list of teacher model{Mt(k)} m k=1 . 1: for k = 1 → n do 2:Select Mt(kt) which has same shape as Ms(k).Ms(k) → MPO (Ms(k)).Mt(kt) → MPO (Mt(kt)). 5: end for 6: repeat 7:Compute LAux between {MPO (Ms(k))} n k=1 and {MPO (Mt(kt))} m t k t =1 by using Eq. ( 5).Compute distill loss L distill . 9:Backward LAux and L distill . 10: until Student model converges The MPO pseudocode is shown in Algorithm S.2.Input: matrix M, the number of local tensors n Output : MPO tensor list{T (k) } n k=1 1: for k = 1 → n -1 do 2: M[I, J] -→ M[d k-1 × i k × j k , -1] 3: UλV ⊤ = SVD (M) 4: U[d k-1 × i k × j k , d k ] -→ U[d k-1 , i k , j k , d k ] 5:T (k) := U 6:M := λV ⊤ 7: end for 8:T (n) := M 9: Normalization 10: return {T (k) } n k=1The distillation cost (memory and time cost) of the original model and the model after applying OPDF are shown in Table S .1. We can observe that as the number of parameters obtained from MPO decomposition increases, both the training time and memory cost increase. However, as the dataset size increases, the ratio of additional time and memory required for training by OPDF to the original training requirements generally exhibits a decreasing trend (e.g., 0.6/0.4 for RTE vs 0.3/0.1 for MNLI in BERT-of-Theseus model). Therefore, the additional time and memory introduced by our method become less of a critical bottleneck affecting the training speed as the dataset size increases. As illustrated in Eq. ( 2), when a parameter matrix W is given, its MPO decomposition into a product of n local tensors can be represented as follows:MPO (W) = T i1,i2,i3,...,in j1,j2,j3,...,jn .(S.10)The models mentioned-Bert of theseus [56] , LGTM [44] , DBKD [45] and AD-KD [46] -are all variants of BERT, which itself is built using transformer blocks. We decompose both the feed-forward network and the multi-head attention layer within the transformer block. Moreover, the teacher model must employ a decomposition granularity that is consistent with that of the student model to ensure proper alignment of auxiliary tensors. When calculating the auxiliary loss L Aux , the alignment is typically between the n-th layer of the student model and the N-th layer of the teacher model, where N is generally an integer multiple of n. The detailed hyperparameter settings for these NLP distillation models are provided in Table S .3and S.4. In NLP tasks, our method takes half to two GPU hours on A100 GPU.Additionally, we implement OPDF on TinyViT [61] to demonstrate its applicability as an orthogonal approach across various knowledge distillation methods that utilize the transformer architecture.Unlike NLP models, we decompose the projection layer in addition to the feed-forward network and multi-head attention layer in the vision transformer block. The specific experimental parameters utilized are detailed in Table S . 5 . In CV tasks, our method takes 160.0 GPU days on A100 GPUs to pretrain TinyViT-21M. We report the performance of the model that achieves the best results on the validation set when applied to the test set. EQUATIONLGTM OPDF (Ours) T 32,1,EQUATIONEQUATIONLGTM SVD T 32,24 64,48 T OPDF (Ours) T OPDF (Ours) TLGTM SVD T 32,24 32,24T 32,24 32,24T 32,24 32,24T 32,24 32,24OPDF (Ours) T OPDF (Ours) T OPDF (Ours) T 32,1,T 32,1,24 OPDF (Ours) TLGTM SVD TOPDF (Ours) T 32,1,AD-KD SVD T OPDF (Ours) T 32,1, Justification: Detailed descriptions of all main claims can be found in Section 4. Furthermore, the main claims made in the abstract and introduction are supported by the experimental results presented in Section 5.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper presents a novel framework for enhancing knowledge distillation (KD) by leveraging over-parameterization. Using the Matrix Product Operator (MPO) as a tensor decomposition technique, the authors propose to increase the capacity of the student model during training, improving its ability to learn from the teacher model without adding inference overhead. Additionally, a loss is introduced to align the student and teacher models more effectively. Extensive experiments across various vision and NLP benchmarks showcase the utility of the proposed KD framework.

Reviewers appreciated the innovative approach, particularly the use of tensor decomposition to over-parameterize the student model. The introduction of a tensor constraint loss for better alignment between teacher and student models was also highlighted as a significant strength. The comprehensive experimental validation was seen as a key strength. However, some concerns were raised about the clarity of the paper, particularly in explaining certain concepts, and the scalability and computational cost of the proposed method, especially for larger models. Despite these concerns, I will recommended the paper for acceptance and suggest the authors incorporate the feedback.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel over-parameterization framework designed to enhance the effectiveness of knowledge distillation. This framework employs MPO as a tensor decomposition technique to expand small models into larger ones to give the student model more capacity. Moreover, to enhance the effectiveness of knowledge distillation, a tensor constraint loss is introduced to align the teacher and student model. Extensive experiments verify the superiority of the method.
strength_weakness: strength_weakness: 1. I would like to know how much additional distillation cost (memory and time cost) will be incurred by introducing such tensor decomposition technique and loss constraint for aligning the auxiliary tensors between the student and teacher models. 
2. Especially in the era of LLM, the high cost of distillation often limits its application. Methods like LoRA have been proposed to reduce the trainable model parameters. I am concerned that if this approach goes against the current mainstream research directions?
3. Can the authors provide some distillation results on larger models to further validate the applicability of the approach on large models?
questions: questions: Please refer to weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: The authors propose to start with an over-parameterised student model. This is realised using high-order tensors that can reconstruct the original parameter matrices. The idea is that this over-parameterised model will benefit more from knowledge distillation.
strength_weakness: strength_weakness: TinyViT-5M⚗ achieves 80.7% top-1 with 5.4M parameters using a just a distillation token and a logit distillation loss. Comparing this with TinyViT-5M+OPDF 80.0% top-1 and 9.9M/5.4M inference parameters, it is hard to see the benefit? if anything the additional SVD operations and reconstruction losses, make OPDF much more difficult to adopt. What is worse it that this difference is even more significant when going to the larger TinyViT models.

It is understandable that transformers have been the main focus of this work, however it would be good to see some experiments with CNNs. Matrix decomposition for the transformer linear layers is easy (with SVD), but when going to higher-order tensors it will be NP-hard. All the theory presented in this paper is for generalising to arbitrary dimensions, yet all the experiments are done on 2-dimensional weights. The paper would be a lot easier to follow without the pre-mature generalisation. i.e. showing equation (4) in 2-dimensions with SVD and I am not sure if the introduction of the MPO framework is needed. 

L177: I am not sure what you mean by "think independantly?". Besides the point that models don't think, if the student can match the teacher fully then that is perfectly fine. The only goal is preserve the performance of the teacher for a much smaller model. 

L595: 160 days to pre-train TinyViT is very significant! I understand this is because ImageNet-21K is very large, but why was this chosen benchmark chosen over, for example, a more standard KD benchmark, such as that in DeIT [1] or CRD [2]. Is 160 days an expected scale of time for the number of GPUs used here? 

[1] Training data-efficient image transformers & distillation through attention ICML 2021
[2] Contrastive Representation Distillation ICLR 2020

Small points/spelling:
Fig 1a "Batchs -> Batches"
questions: questions: What is this normalization step in figure 1 and Alg S.1 L9? and its importance? If it is just to preserve the scale and stop exploding values after reconstruction, this could be explained in the text a bit.
limitations: limitations: The authors have adequately addressed these limitations in the checklist.
paper_summary: paper_summary: This paper introduces the Over-Parameterization Distillation Framework (OPDF), which addresses performance degradation in limited-parameter student networks after knowledge distillation (KD). OPDF proposes an overparameterized student model that utilizes the tensor-decomposition technique known as matrix product operator (MPO), allowing for a significant increase in parameters during student training time without imposing additional inference time burden. Experimental validation is performed across multiple KD tasks to assess the effectiveness of the proposed technique.
strength_weakness: strength_weakness: - The paper does not provide information on the time required for the student network's overparameterization using MPO and the contraction  of decomposed matrices into the original matrix. Having this information would be important for understanding the practical implications of the technique.
- The experiments are conducted on a relatively smaller model. I am curious about the feasibility of applying this technique to LLM/VLMs with billions of parameters. A significant concern is whether their approach, which involves decomposing the teacher network, can scale effectively to such large models. A deeper discussion on this topic would provide valuable insights.
questions: questions: Please refer to Weaknesses section
limitations: limitations: NA
paper_summary: paper_summary: In this work, the authors focus on the knowledge distillation (KD) task, using tensor decomposition to enhance the performance of the student model. Leveraging the principle of overparameterization, the authors employ the Matrix Product Operator (MPO), also known as tensor train matrix, to reformulate the original weight matrix. Additionally, they propose a “distillation loss” to measure the distance between the student’s weights and the teacher’s weights. In the experiments, the proposed method is integrated with existing KD methods, and the results demonstrate a clear improvement in KD performance.
strength_weakness: strength_weakness: The clarity of the paper needs improvement. For instance, Figure 1 is not fully comprehensible as the meaning of the arrows is not clearly explained. Additionally, the experiment settings do not clearly describe how the weights are reshaped into the higher-order tensor format.
questions: questions: 1.	Since the entire paper is based on the principle of overparameterization of the student model, it would be beneficial to explain in the preliminary section why the overparameterization principle is relevant to the KD problem. This addition would help non-experts follow the main idea of the paper more smoothly.

2.	In lines 176-177, the phrase “losing its ability to think independently” is highlighted. I am confused by this statement. Could you provide more interpretation of this claim, supported with sufficient evidence?

3.	Please offer more interpretation of the central and auxiliary tensors mentioned in lines 184-185, using formulas or figures, for example. I cannot clearly understand their differences from the current non-rigorous descriptions.

4.	Why does using MPO provide better performance than SVD? Is it because the MPO forms more “linear layers” than SVD? Have you considered other tensor network formats such as tree tensor networks?

5.	How does the selection of TT-ranks (e.g., d_k in Eq. 2) affect the performance?
limitations: limitations: The paper lacks a discussion on the limitations of the proposed method. I suggest that the authors address this aspect to provide a more balanced and comprehensive evaluation of their work.

Comments: title: Thank you for your positive feedback
Thank you for your feedback
Thanks for the Rebuttal
Additional vision KD results (part 2)
Thank you for raising the score
Kindly request your feedback before the end of the discussion period
Request your feedback before the end of the discussion period
Additional vision KD results (part 1)
Thank you for raising the score
Looking forward to your feedback
Looking forward to your feedback
Looking forward to your feedback
comment: Dear Reviewer fkms,

We greatly appreciate your positive feedback. The discussion with you has been quite productive and fruitful. Thank you very much for raising the score. We will include all the new experiment results and discussions in the revised paper.

Thank you for your time and effort!

Best regards,

The Authors
These results look really great and it would be good to see them in the updated manuscript. I will happily raise my score. The authors have done a great job at addressing all my concerns and it is very clear that their proposed method is complimentary and easy to integrate into many KD pipelines.
Dear Reviewer Dajm,

Thank you for your positive feedback. We will include the new experiments and texts in our revised paper.

Thank you very much for your time and effort!

Best regards,

The Authors
Dear Authors,
Thank you for your effort toward addressing my concerns. My concerns have been addressed and therefore, I would like to maintain my original score.
The following response reports the additional KD results for more recent vision models.

**Q2.The Performance of OPDF on some architecture pairs provided in CRD and several recent CNN KD methods.**

**Reply:** Thanks for your great question. We have applied OPDF to several architecture pairs in CRD and recent KD methods to show the capability of our method in enhancing the performance of CNN distillation models. The new results are reported in **Table B**.

It can be seen that our proposed OPDF can enhance student model performance in several architecture pairs provided in the CRD benchmark using more recent KD methods. We will include these results in the revised paper. 

**Table B.** Distillation results of various CNN architectures on CIFAR-100.

| Teacher to Student  | ResNet110 to ResNet20 | ResNet110 to ResNet32 | ResNet32x4 to ResNet8x4 |
| --------- | :--------: | :--------:| :--------:|
| Teacher    | 74.31 | 74.31 | 79.42 |
| CRD        | 71.46 | 73.48 | 75.51 |
| CRD+OPDF   | **72.35** | **74.57** | **75.83** |
| NST        | 69.53 | 71.96 | 73.30 |
| NST+OPDF   | **70.98** | **72.34** | **74.00** |
| ITRD [1] | 71.99 | 74.26 | 76.19 |
| ITRD+OPDF  | **72.48** | **75.92** | **77.73** |
| KD+LSKD [2]    | 71.99 | 74.26 | 76.19 |
| KD+LSKD+OPDF  | **72.33** | **74.93** |**76.89** |


Since the discussion period is soon ending, this is the best result we could supply at this moment after nightless working on the new experiments. 

Thank you very much for your time and efforts. Your consideration of further improving the rating of our paper will be much appreciated!



**Reference:**

[1] Miles, et al. Information theoretic representation distillation. In British Machine Vision Conference (BMVC), 2022.

[2] Sun, et al. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15731–15740, 2024.
Thank you for your extensive effort in response to my questions. I am happy to raise my score to 5 BA, and I am also very keen to see what will come back with regards to using other recent CNN KD methods.
Dear XV5e:

Thank you for your positive feedback. We will include the additional experiments and texts in the revised paper.

Thank you for your time and efforts!

Best regards,

The Authors
Sorry for the delayed reply. I appreciated the detailed and clear  response from the side of authors. It answered most of my concern in the review. I would adjust the recommendation score from 4 to 5.
Dear Reviewer XV5e:

As the author-reviewer discussion period is soon ending, we would appreciate it if you could review our responses and provide your feedback at your earliest convenience. If there are any further questions or comments, we will do our best to address them before the discussion period ends.

Thank you very much for your time and efforts!

Sincerely,

The Authors
Dear Reviewer XV5e:

As the author-reviewer discussion period will end soon, we would appreciate it if you could kindly review our responses at your earliest convenience. If there are any further questions or comments, we will do our best to address them before the discussion period ends.

Thank you very much for your time and efforts!

Sincerely,

The Authors
We greatly appreciate your feedback along with two additional questions. Please see our reply to your *first question* as follows. 

**Q1. The application of linear probe performance metrics; distillation results affter fine-tuining.**

**Reply:** Thanks for your question. In fact, Table 5 in original TinyVit paper [1] has already utilized linear probe performance to assess the efficacy of TinyVit. Hence, we opted it for comparison to effectively demonstrate the validity of OPDF. 

Moreover, we have fine-tuned models previously distilled following the tasks set in TinyViT (where TinyViT-5M⚗ achieves 80.7% top-1 accuracy with 5.4M parameters), and the results are showed in **Table A** below to show the efficiency of our proposed OPDF.

The experiments results show that even when employing the same experimental setup in TinyVit, OPDF consistently yields a significant enhancement in the performance of TinyVit. We will include these results in the revised paper. 

**Table A.** Models are pretrained on ImageNet-21k and then finetuned on ImageNet-1k, Imagenet-Real and Imagenet-V2 (top-1/top-5).
| Datasets  | Imagenet-1k | Imagenet-Real | Imagenet-V2 |  # Train Params | # Inference Params |
| --------- | :--------: | :--------:| :--------:| :--------: | :-----------: |
| TinyVit-5M      | 80.7/95.6 | 87.5/97.8 | 68.3/89.7 |  5.4 | 5.4    | 
| TinyVit-5M+OPDF  |**81.8/96.9** | **87.9/98.4** | **69.5/90.4** | 9.9  |   5.4  | 
| TinyVit-11M     | 83.2/96.5 | 88.3/98.1 | 72.9/91.4 |  11  |   11   |
| TinyVit-11M+OPDF| **85.1/97.1** | **89.0/98.5** | **74.1/93.3** | 23   |  11    | 
| TinyVit-21M     | 84.8/97.3 | 88.9/98.5 | 75.1/93.5 |  21  |    21  |
| TinyVit-21M+OPDF| **86.5/97.9** | **89.7/98.9** | **76.2/94.7** | 38   |   21   |

BTW, the code of applying OPDF to other recent CNN KD methods (e.g., CRD) is currently executing. We will supplement the corresponding results as soon as the model training is completed. Hence, we will respond to your *second question* soon. Thank you for your patience.

**References:**

[1] Wu, et al. TinyViT: Fast Pretraining Distillation for Small Vision Transformers. ECCV 2022, pages 68–85.
Thank you for your positive response. We would like to clarify that during the inference with LoRA, the parameters that need to be computed are $W+\Delta W$, where $W$ represents the weights of the pre-trained model (PLM) and $\Delta W$ the additional parameters introduced by LoRA. This is why we mentioned in our reply that applying the LoRA method during inference increases the number of parameters, thereby increasing the inference time.

Lightweight fine-tuning (LoRA) and model compression (KD) represent two different tracks for deploying large PLM. While LoRA is cost-effective, it slightly increases the inference time due to the added $\Delta W$ parameters. On the other hand, KD results in a compressed model that reduces both inference time and computational overhead while increasing the training cost.

Moreover, compressed models obtained through KD may have diverse applications on the edge. For instance, they are particularly suitable for deployment in resource-constrained environments, such as mobile devices and embedded systems.

Again, thank you very much for increasing the rating of our paper!
I am confused about why the authors claimed that the inference parameters of adding LoRA will increase. What I mean is that performing KD on LLMs is expensive. Reducing the number of trainable parameters might be a way to lower the cost of performing KD.

However, since the authors have presented the time and memory cost of the proposed method and it seems to be acceptable, I would like to raise my score by 1.
Dear Reviewer XV5e,

We're keen to know if there are any remaining concerns that require attention or if there are additional discussions that should take place. Your insights are greatly appreciated as they contribute to the refinement of the paper. Looking forward to your feedback. Thank you.

Best regards,

The Authors
Thank you for your thorough response and additional experiments. Most of my concerns have been addressed and in light of the other reviewers remarks I am happy to raise my score. However, I am still unsure about 2 main parts. 

Firstly, although I do understand and appreciate that the authors only report the linear probe performance and thus a comparison to TinyViT, DeiT etc is unfair, I would like to know why this is done? The authors don't seem to do this for the NLP related tasks. Are there other KD works that train models and evaluate in this fashion? (specifically for vision-tasks). Vision KD is a lot more more mature than NLP KD, especially for feature distillation, and so I believe a comparison here is important.

Secondly, the experiments on CNN models is greatly appreciated and does strengthen this submission. It is important to show that the generalisation has practical utility rather than just for the sake of generality. The authors have addressed my concern over this point theoretically too. These CIFAR100 results are fully fine-tuned which is nice to see and partly addresses my previous concern. OFD is argueably quite an old paper. Is there any chance for a comparison to some other architecture pairs provided in the CRD [1] benchmark? and using some more recent KD methods. There are some simple methods with code here [2].

Hope the authors can address my concerns. 

[1] "Contrastive Representation Distillation" ICLR 2020
[2] https://paperswithcode.com/sota/knowledge-distillation-on-cifar-100
Dear Reviewer fkms,

We're keen to know if there are any remaining concerns that require attention or if there are additional discussions that should take place. Your insights are greatly appreciated as they contribute to the refinement of the paper. Looking forward to your feedback. Thank you.

Best regards,

The Authors
Dear Reviewer Qtmi,

We're keen to know if there are any remaining concerns that require attention or if there are additional discussions that should take place. Your insights are greatly appreciated as they contribute to the refinement of the paper. Looking forward to your feedback. Thank you.

Best regards,

The Authors