Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation
Authors: Yu-Liang Zhan
Zhong-Yi Lu
Hao Sun
Ze-Feng Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: Knowledge Distillation
Over-parameterization
Tensor Decomposition
Model Compression
Primary area: deep_learning_architectures

Abstract: Increased training parameters have enabled large pre-trained models to excel in various downstream tasks. Nevertheless, the extensive computational requirements associated with these models hinder their widespread adoption within the community. We focus on Knowledge Distillation (KD), where a compact student model is trained to mimic a larger teacher model, facilitating the transfer of knowledge of large models. In contrast to much of the previous work, we scale up the parameters of the student model during training, to benefit from over-parameterization without increasing the inference latency. In particular, we propose a tensor decomposition strategy that effectively over-parameterizes the relatively small student model through an efficient and nearly lossless decomposition of its parameter matrices into higher-dimensional tensors. To ensure efficiency, we further introduce a tensor constraint loss to align the high-dimensional tensors between the student and teacher models. Comprehensive experiments validate the significant performance enhancement by our approach in various KD tasks, covering computer vision and natural language processing areas. Our code is available at https://github.com/intell-sci-comput/OPDF.

Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation
Yu-Liang Zhan
Zhong-Yo Yi Lu School of Physics Renmin University of China zluêruc. edu.cn
Gaoling School of Artificial Intelligence Renmin University of China
zhanyul ianggrin.ggreeeeee edu. cn
Hao Sun
Ze-Feng Gao"
Gaoling School of Artificial Intelligence Renmin University of China haosun@ruc. edu. cn
School of Physics Renmin University of China zf zfgao0ruc. edu. cn
Abstract
Increased training parameters have enabled large pre-trained models to excel various downstream tasks. Nevertheless, the extensive computational require- ments associated with these models hinder their widespread adoption within the community. We focus Knowledge Distillation (KD), where compact stu- dent model is trained to mimic a larger teacher model, facilitating the transfer of knowledge of large models. In contrast to much of the previous work, we scale up the parameters of the student model during training, to benefit from over- parameterization without increasing the inference latency. In particular, propose a tensor decomposition strategy that effectively over-parameterizes the relatively small student model through an efficient and nearly lossless decomposition of its parameter matrices into higher- digees-dimeniional tensors. To ensure efficiency, we further introduce tensor constraint loss align the high-dimensional tensors between the student and teacher models. Comprehensive experiments validate the significant performance enhancement by our approach in various KD tasks, covering computer vision and natural language processing areas. Our code is available /https:////tt //github. woommintellle.ee sci cocd-ttomppppp
Introduction
Large-scale pre-trained models are gradually achieving remarkable milestones due to the exhibit of
remarkable performance across various tasks (1-7]. These models leverage extensive pre-training data
and parameters, enabling them to effectively encapsulate significant breadth of world knowledge (8]
Ω and exhibit strong generalization capabilities across diverse tasks 13. Following this
trajectory, the utilization of increased data and parameters has emerged as a notable trend in enhancing the performance of pre-trained models recent years, leading the number expansion pre-trained
model parameters from millions to billions 4 14 15
Despite their impressive performance, the substantial storage demands and high computational complexity hinder the practical deployment of these models in real-world applications. Therefore, on the hand, some studies focus on pre-training relatively smaller models (such BERT- base-uncased (2)) on domain-speciffe or task-specific corpora [16)18]. However, due to the lesser over-paramettrrizaion of small models compared to large ones, their generalization capability often
"Correpoondigg authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

falls short, resulting in suboptimal fine-tuning performance on downstream tasks. On the other hand, model compression methods, such as pruning less informative parameters [9-21 or utilizing knowledge distillation (KD) [22] to transfer knowledge from larger models (teachers) to smaller ones (students), have been proposed. KD has swiftly diversified into numerous branches, primarily falling into two categories: i.e., logits-based based (62 and features-based [27 depending on the source of student model knowledge. Nevertheless, as student models have fewer trainable parameters and limited capacity, significant performance gap remains between student and teacher models.
To address the disparity between small and large models, this study aims to over aver-paraeeterze small student models as large ones during distillation training to enhance their generalization capability. Typically, most parameters of student models are stored as matrices. Through tensor decomposition techniques [3 (e.g., Singular Value Decomposition), each matrix can be factorized into a set of matrices, effectively increasing the total number of parameters during distillation. Moreover, after convergence, the factorized matrices can be merged to reorganize the parameter matrix of the student model. This paradigm leverages the benefits of wver-parameterization during training without increasing the inference latency of student models.
However, incorporating tensor decomposition into over-parameterizing student models poses two major concerns that must be addressed. First, the potential information loss caused by tensor decomposition should be minimized, as small computation errors may accumulate and propagate exponentially within the stacked layers of student models. Second, in the over- parameterized student models, there is no effective mechanism to ensure the consistency of information between student and teacher models. Therefore, it is essential to choose appropriate tensor decomposition methods and design loss functions for high-order tensors to ensure the effective transfer of information from teacher to student models.
To address the above issues, we introduce the matrix product operator (MPO) 34 technique as the tensor decomposition strategy. The MPO decomposition, widely used in quantum many-body physics, efficiently factorizes any matrix with arbitrary dimensions into set of higher-dimensional tensors, which can reconstruct the original matrix in almost lossless conditions (74) These advantages make MPO an ideal method for over wver-parameterizing student models during distillation. Based on MPO, we also devise high-order tensor alignment losses for student and teacher models to ensure the effective transfer of information in tensor representation.
Therefore, in this paper, we propose a general Paeerearrmmttrizaatinn Distillation Framework, namely OPDF, improve the performance of knowledge distillation. Given the parameter matrices of a student model, we first over-parameterize them through MPO decomposition and then utilize high-order tensor alignment losses ensure efficient information transfer. This framework only modifies the distillation training process, making it applicable to various student models and natural language processing (NLP) and computer vision (CV) tasks. We conduct extensive experiments in both NLP and CV domains. Experimental results demonstrate that our OPDF significantly enhances the effectiveness of model distillation, e.g., improving BERT-base KD +1.6 on average. Moreover, our approach also enables the student model to achieve performance nearly on par with the teacher model, egg., AD-KS+Ours (83.4) v.s. BERT-base (83.4) in average metric on GLUEE
2 Related work
Large Scale Pre-trained Models Large-scale pre-trained models have achieved remarkable success in many fields (e.g., natural language processing (NI.P) (38 and computer vision (CV) [5] 39)
Since the introduction of the Transformer architecture [40, the pre-training and fine-tuning paradigm in NLP, exemplified by models like BERT [2] and T5 [4, has shown outstanding performance across multiple tasks. Furthermore, the emergence of models like GPT-3 has demonstrated that increasing model size can significantly improve performance vision, models based on Transformers, such as VIT 77. have also performed exceptionally well. In research, we improve the distillation process by increasing the parameters during the training phase of the student model, without introducing additional inference latency to the student model.
low-resource tasks [12.. In the field of computer
Knowledge Distillation Knowledge Distillation (KD) methods are commonly used to compress models by transferring knowledge from a larger teacher model to a smaller student model. Building upon the initiative work by [22], the researchers have exploited the logits follows up with different

---

techniques in the computer vision field, e.g., minimizing KL-Divergence (DKD correlation (DIST (6)). Logit-based methods have been also popular in NLP 41 Features- based methods have tried to align the features from intermediate layers of teacher and student models and minimize the differences [3 After the intermediate representations have been introduced [27], mount of features-basee KD methods have been proposed to match the features, such as LGTTM 44
25 or Pearson
DBKD [45] and AD-KD [46]. However, the capacity gap between the teacher and student models makes it difficult to imitate the hidden representations of the teacher 177. Different from these existing KD methods, our proposed OPDF has utilized MPO decomposition to over-parameterian the student model in the training procedure improve the student model generalization capability, which can minimize the capacity gap efficiently.
also known tensor-train operators (TTOs) [33]. have been proposed for a more efficient representation of the linear structure of neural networks 49 10] A large number of typical applications have utilized MPO-based methods to compress linear layers [11] and convolutional kernels [52] the parameter matrices of deep models. Furthermore, existing works have applied the MPO method for lightweight fine-tuning of ALBERT [35], the efficient expansion for the MoE framework [36], the wee-ppramettrrzzaion tuning
Matrix Product Operators Matrix Product Operators (MPOs) 34 48
process for PLMs [37], construct efficient PLM architecture (53 [54] and compressing datasets (55). Unlike existing methods, our approach focuses on utilizing MPO decomposition to map parameters from low-dimensional spaces to high-dimensional spaces, over- wver-parameterize the student model during the distillation process, allowing the student model to benefit from more parameters and
achieve better distillation results.
3 Preliminary
Tensor Product We denote tensor Tr, do. as an array with n indices, where {iz, ig...., }n denotes the dimensions of the n indices, respectively. In this manner, a vector (i.e., v) can be considered a 1-order tensor, while matrix (i.e., W) can be regarded as a 2-order tensor. Consider .i,,.., tp and é1...., dh as the orthonormal bases of tensors (1) and (22), respectively. The tensor product, denoted as 8, can be obtained through the contraction of (1) and (2)), Formally, the tensor contraction (1) Dp im1 a;30; and T = > b, defined as follow:
T 11)
a,b,y,, @ 0
The set ti,, @ Φi constitutes the orthonormal basis of the resulting vector Hilbert space, with the dimensionality of this Hilbert space being the product (i.e., p x 4) of ( and (2))
Tensor Decomposition Tensor decomposition can viewed as the reverse operation of the tensor product. A commonly employed approach is the singular value decomposition (SVD) algorithm. Given tensor T E xgixx.ii,, the SVD operation performed n times can decompose this tensor into n local tensors ( k=1- Conversely, the decomposed tensors can reconstruct the original tensor by sequentially applying the tensor product operator.
4 Method
In this section, we describe our proposed over wver-parameterized distillation framework. We first outline approach, then introduce the details of matrix product operator decomposition and the over- parameterized student model strategy, and finally present the tensor alignment loss.
4.1 Overview
Current distillation methods primarily enhance the performance of student models by introducing constraints on logits or features between the student and teacher models. In contrast to these methods, our approach not only utilizes tensor decomposition to wverppaaaetteriee the student model for performance improvement but also designs alignment loss functions for the decomposed high-order tensors to further enhance the performance of the student model. To achieve this goal, employ
3

---

Train Batch
C5
A
An
Pre-Trained Teacher
S Student
Over wver-parametrriee Student
Aaux
Fine-Tuned Teacher
MPO
Target
CT
A
An
U
SVD
Wix
V1
A
V
times
Select Central
Normalization
A
Ax
N+1
Figure I: The overview of over-parameter distillation framework (OPDF) for knowledge ditillation. We use MPO decomposition to realize the over-parameter procedure for the student model. The auxiliary tensors the student model are trained to imitate the auxiliary tensors of the teacher model closely. b, We present an illustrative example of MPO decomposition. A parameter matrix W1x Vxx is decomposed into central tensor and auxiliary tensors.
tensor decomposition method to decompose the parameter matrices of the teacher and student models into a series of high-order tensor products. These high-order tensors can be used to reconstruct the original parameter matrices while significantly increasing the number of trainable parameters in the student model. After reconstruction, the student model has the same number of parameters as the original matrix without increasing inference time and model size. Additionally. by introducing distillation loss functions to allow the student model to learn from the teacher model in tensor representation, the effectiveness of knowledge distillation is further enhanced.
In our proposed weer-parameterized distillation framework, we integrate tensor decomposition strategy based on MPO into the student model to enlarge the parameter matrix (Section 4.2) Further- more, we design a tensor alignment loss function to enhance the performance of the student model in the context of knowledge distillation (Section 4.3). An overview of our approach is depicted in Figure We also provide a detailed description of our over-parameterized distillation framework in Algorithm S.I
4.2 Over-paramterization Distillation Framework via MPO Decomposition
To leverage the advantages of over-pore peraaaatttttiinioonnnn during knowledge distillation, our method utilizes the MPO, tensor decomposition technique that increases the number of model parameters. In this part, we initially present the specifics of the MPO method and subsequently outline adaptation for over-parameterizing the student model.
Matrix Product Operator Decomposition The MPO decomposition is an efficient algorithm capa- ble factorizing a parameter matrix W R RIxJ into a sequential product of multiple tensors [34]. Formally, given matrix M / its MPO decomposition into product of local tensors can be represented as:
MPO (M) = II T(p 11 dx). k=
(2)
The tensor T(k) iR. jk, dz is 4th-order tensor with dimensions d:-1 x ig. X jR x de, where IT ig T, -,I 1 J, and do = d, = 1. To link two sequence tensors, we have adopted the concept of bond following the work of [48). The bond dimension de is defined by:

---

dk min
II im x Ims
II im x Im
m=k+1
From Eq. (3 we can see that da will be large in the middle and small on both sides. Following 33
we refer to the tensor right in the middle as central tensor, and the rest as auxiliary tensor. Figure l presents the illustration of MPO decomposition. You can find additional descriptions of tensors and MPO in Appendix A
Over-parameterzing Student Model. Utilizing the MPO method within the framework of knowl- edge distillation, our objective is to extend the parameter scale of the student model, capitalizing on aver-parameterization. More specifically, we can employ the MPO method to break down portion of the parameter matrices into multiple tensors as illustrated in Eq. (2) Following MPO decomposition, the parameter count of the matrix W will increase based on the values of {dg j {ix ==== and {is k==1 The precise augmentation in parameter count, denoted as Nodd, can be computed as follows:
Nadd Σ iu de
II in.
A=1
Therefore, during the knowledge distillation procedure, we can adopt MPO on student model parameter matrices to generate their corresponding multiple tensors. In this way, we can scale up the total parameter of the number of the student model without increasing its inference time consumption. After training the wer-paarameterized student model to convergence, tensor contraction on these decomposed tensors, to reconstruct the parameter matrices of the student model in almost lossless conditions which is detailed in Appendix B This new student model has the same parameter number and inference latency as the original one and has benefited from over-parameterizzaton during training.
will perform
4.3 Assisted Constraints for Knowledge Distillation
Revisiting Prediction Match of Knowledge Distillation Traditional knowledge distillation in- volves two stages: fine-tuning the teacher model for a specific task, followed by training strategies to constrain the student model to closely approximate the teacher model. These processes aim to transfer the knowledge from the teacher the student model. Recent studies have mainly focused directly learning from the features and logits of the teacher model to transfer crucial knowledge (23 56).
However, these methods are limited by the capacity of the student model due to the limitation of total parameters. Moreover, this distillation approach based on cross-entropy loss constraints may lead to the student model losing its ability to learn independently. We aim to design novel model distillation framework to enable the student model not only to effectively learn the knowledge from the teacher model but also to maintain its ability to learn independently.
Distillation Loss for Auxilary Tensors. achieve the goal of "learning knowledge from the teacher model while maintaining the ability of the student model to learn independently." we introduce a high-order tensor alignment training method based on the MPO decomposition, crucial merit of MPO decomposition is its ability to reorganize and aggregate the core information, decomposing the weight matrices into a central tensor (containing a large number parameters and important information) and auxiliary tensors (containing fewer parameters and additional information to the central tensor) 35 36). Therefore, in the knowledge distillation, in addition to minimizing the cross-entropy loss concerning the ground truth, add loss constraint for aligning the auxiliary tensors between the student and teacher models:
LAuA
MSE Ar,k)
(5)
71
where the matrices Ar, and Ag,k refer to the auxiliary tensor of student and teacher models with the same dimensions respectively. MSE means the mean-square error loss function. To ensure that student model learns from the teacher while preserving its central tensor for independent learning, we minimize the mean-square error loss between the auxiliary tensors of both the student and teacher models. Since this distillation framework is based improvements to the weight matrices, it is
5

---

orthogonal to most current distillation methods. Therefore, it can further enhance the distillation effectiveness based on existing distillation methods (as thoroughly discussed in the experimental section). Hence, it can be widely applied to various knowledge distillation models.
5 Experiments
In this section, assess the efficacy of our approach within two renowned domains: computer vision and natural language processing. Notably, the OPDF is designed to complement existing distillation techniques. Consequently, we apply our proposed OPDF with various standard distillation methods to validate its effectiveness. In the subsequent section, we detail our experimental setup's 's datasets and baseline methods. We then present the primary results achieved with the OPDF and provide thorough analysis. Furthermore, we examine the influence of the degree of over-parameeerizzation MPO strategy and the learning rate on the performance of OPDE We report the memory and time cost of experiments in Appendix D.T
5.1 Experimental Setup
Datasets and Metrics For NLP tasks, we evaluate our approach on text classification tasks in GLUE benchmark (57). The tasks encompassed in our evaluation include RTE, MRPC, STS-B. B. CoLA, SST-2, QNLI, QQP, and MNLI. To facilitate comparison with baselines, we employ the F1 score and accuracy as metries for MRPC and QQP, Matthew's correlation coefficient for CoLA, and the average of Pearson and Spearman correlations for STS-B. Accuracy is used as the metric for the remaining tasks, with the result for MNLI reported as the average across the matched (MNLI-m) and mismatched (MNLI-mm) domains. Additionally, we calculate the average score across all tasks to provide a comprehensive performance measure. In the context of CV tasks, we have applied the OPDF to the distillation of Vision Transformers (ViT) for image classification [7]. This was done using the ImageNet-22 Ik dataset [58), ImageNet- Ik, ImageNet Real [59]. and ImageNet V2 [60] datasets. For these datasets, we use accuracy as the primary evaluation metric.
Baseline Methods For NLP tasks, we implement OPDF on previous KD methods: BERT-of- Theseus [56), LGTM 44), DBKD 45 and AD-KD [46]. We replicated the baselines using the publicly released code to assess their performance on the test set. Additionally, LGTM was not previously evaluated across all tasks in its original publications, and we have addressed this omission using the provided code. It is important to note that DBKD is designed to estimate logits from do not report performance all experiments in natural language processing, we demonstrate the effectiveness of our method during the fine-tuning stage. We implement the teacher model the fine-tuned "BERT-base-uncass model (2). In the context of CV tasks, ViT VIT [6]], which introduces a rapid pre-training framework, has emerged
decision distributions [45]. and therefore
the STS-B task. For
classical distillation method for VIT. The original paper on ViT discusses three versions of the model with varying parameter counts: Tiny ViT-5M, Tiny ViT- 1 1M, and Tiny To incorporate high-order tensor alignment loss into the distillation phase, we utilize CLIP-VIT. I/14 77 62). variant of VIT, as the teacher model in our experiments. To assess the efficacy of OPDE, we pretrain the distillation model on ImageNet-21K and evaluate its linear probe performance on ImageNet- 1k, ImageNet Real, and ImageNet V2, without any fine-tuning, During the pre-training stage, we adhere to the same experimental settings as described in the original paper. Furthermore, we juxtapose our method with SVD (32), traditional tensor decomposition technique viable for over-parameterizing student models. Concretely, we employ SVD to substitute MPO within our framework and execute over-parameterizzatoon across all parameter matrices of the student model during knowledge distillation. Appendix D.2 shows more experimental details.
5.2 Main Experimental Results
NLP Tasks We present the results on BERT in Table I Firstly, it is evident that integrating KD with over-parameterization methods yields the most significant performance enhancements. Over-parameteriizaion enhances the generalization ability of the student model. Upon comparing the two tensor decomposition techniques, we find that MPO consistently outperforms SVD. This discrepancy arises from the singular value-based SVD in a two-dimensional space, limiting its ability to substantially increase model parameters compared to MPO decomposition (e.g., 90M

---

Table 1: Comparison of performance the GLUE benchmark (in percent). The terms "+SVD" and "+OPDF" represent the use of different over paraattratttinnnnonnn methods in a KD model. "# Train Params" and "# Inference Params" refer to the total number of parameters during training and inference, respectively. Numbers marked with indicate tasks not tested in the original studies; results here are reproduced from the published code. The best result for each task is highlight in bold. For all the results, we report the mean values of five runs using different random seeds.
Datasets
RTE
Ace.
Fl/Ace.
MRPC
STS-B Corr.
CoLAA Mcc.
SST-2 FI/Ace.
QNLI Acc.
QQP Fl/Acc.
MNLI Acc.
Avg.
Params (M)
# Train
# Inference
Params
(M)
BERT-base 2
70.5
86.5/8188
86.6
54.2
92.0
91.2
88.091.0
84.2
83.4
110
110
BERT-of-Thee of-Theees [56
None
65.5
85,3/79.6
86.2
39.2*
90.4
88.7
86.1/8
81.5
79.2
66
66
+SVD
65.5
85 4/80.0
86.5
43.1
90.6
88.6
86. 36.2/89.7
80.3
79.6
90
66
+OPDF (Ours)
66.2
85,9/80.5
88.6
45.2
91.3
89.0
86.8/90.2
81.4
80.5
160
66
LGTM
44
None
63.3
86.3/80.1
82.9*
33.9
91.1
89.3
88.0/91.1
82.2
78.8
67
67
+SVD
64,7
86.8/81.9
83.1
37.4
91.2
88.6
86. 86.5/89.4
79.3
78.9
91
67
+OPDF (Ours)
66.9
87.8/82.4
83.3
38.9
91.5
88.7
87.0/90.2
80.9
79.8
163
67
DBKD
45
None
61.2
83.3/75.5
25.2
88.1
86.1
85.3/88.7
76.1
74.4
53
53
+SVD
64.7
86.5/78.6
26.4
88.8
85.8
85.5/89.0
76.5
75.8
69
53
+OPDF (Ours)
69.1
88.4/83.3
27.2
89.8
86.5
86.9/90.2
77.7
77.6
83
53
AD-KD 46
None
68.8
88.7/84.3
89.3
53.1
91.5
90.8
85.9/89.5
81.7
82.4
67
67
+SVD
69.4
89.3/85.8
88.8
53.5
89.9
90.1
86.4/89.8
81.5
82.6
91
67
+OPDF (Ours)
71.7
90.3/86.8
88.9
55.0
91.3
91.1
86.8/90.0
82.1
83.4
182
vs. 160M in BERT-offTheeeus) In contrast, MPO allows for arbitrary scaling by increasing the order of decomposition, rendering it more suitable for over-parameterization.. Secondly, following the integration of the OPDF method, the performance of prior KD techniques (BERT-offTheseee LGTM, DBKD, and AD-KD) have exhibited enhancements across a majority of tasks (e.g., RTE, MRPC, CoLA, QQP), while maintaining comparability with the original method in other tasks. This highlights the versatility of OPDF, demonstrating its effectiveness across diverse models and a wide range of tasks. Finally, our findings have revealed that employing the OPDF method can even outperform the performance of the teacher model in MRPC and RTE datasets. This indicates that the process of over-pe seee-aramtterzattion endows the student model with stronger generalization capabilities, suggesting that employing over-paraaeeterizaton may offer a potential solution to the bottleneck in current distillation methods where the performance of the student model fails to surpass that of the teacher model.
CV Tasks All CV results of our proposed method are shown in Table We apply OPDF on three kinds of TinyVi Vit with different total parameters. It is clear that with OPDF, the performance of Tiny Vit be significantly improved. In particular, in all datasets, Tiny Vit applied OPDF is better than vanilla Tiny Moreover, Tiny Vit utilized OPDF with I 1M parameters can achieve better performance than Tiny Vit with 21M parameters. It demonstrates that OPDF is an orthogonal method for various KD methods based on the Transformer whether in the CV or NLP field. Note that since we only involved the over-parannn paraamtttttiiiiioonnn procedure in the training phase, the total parameter of the student model will not change in the inference phase. This merit makes the OPDF unique from the existing KD method: one would not increase inference time while enhancing model accuracy and enabling the model to acquire more knowledge from the teacher model. Moreover, we can observe that the performance of the original Tiny Vit, SVD over-parameterizzaion,, and OPDF over-parameeerizzaion improves as the number of parameters gradually increases. This indicates that compared to SVD, the MPO decomposition, which can decompose the parameter matrix to any size, can better enhance the expressive capacity of the student model. The impact of the over-parameterization scale on distillation effectiveness will be analyzed in detail in Section 5.3
5.3 Further Analysis
Performance Comparison w.r.f. Parameter Increasing Rate. Our OPDF method facilitates the flexible expansion of model parameters, thereby highlighting the significance of the parameter increase rate model performance. Consequently, we investigate the influence of this rate on model
7

---

Table 2: The linear probe performance (in percentage) of ViTy pre-trained ImageNet-21K ImageNet-Ik 1k [58]. ImageNet Real [59], and ImageNet v2 [60]. Numbers marked with indicate that these results are got by official checkpoint and released code.For all the results, report the mean values of five runs using different random seeds.
Imagenet-1k
Imagenet Real
Imagenet V2
Train Params
# Inference Params
Datasets
top-1 84,8
top-5
top-1
top-5
top-1
top-5
(M)
(M)
CLIP-VII- 62
88.9
75.1
321
321
Tiny Vit-5M
[61
None
77.4*
94.1 1M
86.1*
97.5*
66.8*
87.6*
5.4
5.4
+SVD
77.9
95.1
86.3
97.3
68.7
88.4
7.6
5.4
+OPDF (Ours)
80.0
96.7
87.4
98.1
69.4
88.9
9.9
5.4
Tiny Vit-11M
None
80.5
95.6*
87.8*
98.0
70.7*
90.4*
+SVD
82.0
96.7
88.4
97.9
71.7
91.4
17
+OPDF (Ours)
82.5
96.9
88.9
98.3
72.4
92.6
Tiny Vit-2IM
None
82.3*
96.3*
88.9
98.3*
73.0
91.9*
+SVD
82.9
8
97.8
92.4
+OPDF (Ours)
84.0
97.5
89.4
98.4
74.9
93.4
88.5
QQP
MAPC
0 87.5
06-7
87.0
84.5
QNLI
STS-8
83.0-
1.5x
2.0x
2.5x
3.0x
2e-6
le:5
6e:5
Scale Factor
Leaminng Rate
COLA
LGTM OFDF
DBKD
OFDF
+PPDF 1.2%
+PPD 1.6x
Theseus
+MPO
OPDF
1OPOF
(a) Impact scale factor
(b) Impact of learning rate
(c) Impact of OPDF components
Figure 2: The impact of over- wverpparameterization scale, learning rate, and various components of the OPDF distillation model performance is explored. Figure 2(a) demonstrates the performance of the LGTM and DBKD model on the MRPC task following the implementation of the OPDE Figure 2 presents the performance of DBKD + OPDF with different wverpparameterization scales on the MRPC task. Figure 2cc displays the performance of the theseus model across various tasks, utilizing different parametttrriitioonnn methods and integrating various components of the OPDF.
efficiency further. To underscore the general applicability of our findings, we intentionally over- parameterize two models: DBKD and LGTM. We then elucidate their relationship with fine-tuning performance on MRPC tasks. All results are depicted in Figure 2(a)
is observed that the performance of both the LGTTM consistently improves with an increase parameters. This enhancement substantiates the efficacy of using the OPDF for over-parameterizing models, which in turn significantly boosts the performance of knowledge distillation models. Furthermore, after aver-parameterization. the performance of the mod- els is capable of achieving, at a minimum, the level of their original benchmarks (e.g.. 83.3 for DBKD and 86.3 for LGTM). The enhancement of model performance through veerpaaraeterrization has its limitations. As demonstrated in Figure 2(a) beyond certain thresholds of over- over-parameter parameterization (e.g.. 1.6x for DBKD and 2.5 for LGTM), the performance improvements of the models no longer exhibit significant gains. This observation indicates that there are inherent limits to the benefits that can be achieved through over-parameetrizztion in knowledge distillation models. These limits are likely influenced by structural characteristics of each model and size of the initial model configuration.
and DBKD 45 models on the MRPC task
Styper-parameters Tuning OPDF decomposes the original weight tensor through over- parameterization, leading to the updating of more parameters. Consequently, the tensor product results in larger updates to the existing parameters in the backward phase. In Figure 2bb we illustrate

---

the relationship between the performance on the MRPC task and learning rate when the parameters of the DBKD model are expanded 1.2x, 1.4x, and 1.6x their original size.
There exists an optimal learning rate for every scale of over-parameterization. Deviating from this optimal rate, whether by increasing decreasing the learning rate, results in diminished model performance. The reduction in performance due to a lower learning rate can be attributed to the model becoming trapped in local optimum.
Additionally, we observe that peak model performance consistently increases with the scale of over over-paramenn reer-prraettriizatoon This finding aligns with the conclusions drawn from Figure 2(a) Moreover, as the scale over-vvrannn pver-paramtterizttion increases, the learning rate required to achieve optimal model performance decreases. This occurs because using the tensor product to restore the shape of the tensors to that of the original weight tensors also scales the updated values, resulting in significant changes. Consequently, an increasing learning rate leads to declining performance in the KD model, indicating that the learning rate should decrease as the over-paraman werr-praaeetervzttonn scale increases.
Finally, despite changes in the learning rate, the performance of the model with OPDF consistently remains at least as high as that the original method. This indicates that OPDF is not sensitive to learning rate variations during the distillation stage. This resilience is due to OPDF's ability to factorize the parameter matrix in almost lossless conditions, ensuring that the decomposed matrix match or exceed the training effectiveness of the original matrix without introducing errors.
Impact of MPO strategy To demonstrate the robustness of our MPO methods, we applied different MPO methods to the DBKD and AD-KD model on the RTE, MRPC, STS-B, CoLA, and SST-2 task. The experimental results presented in Table To maintain consistent over- parammtertttiinionnn scales, we used the same decomposition scale (L) for each KD model across the same task.
Table 3: Comparison of performance on the GLUE benchmark (in percent). In the tensor representa-
tion, "L." denotes the number of "1"s in the dimension list.
RTE Acc.
MRPC
STS-B
CoLA Mec.
SST-2 F1/Acc.
Experiments
F1/Ace.
Conr.
DBKD
L 4.24
4
7
4
69.1
88.4/83.3
27.2
89.8
68.0
86.3/81.0
25.2
89.0
T.2.2.
68.5
87.9/82.5
26.1
88.9
AD-KD
8
3
TeL,
1,94
71.7
90.3/86.8
88.9
55.0
91.3
716,2
70.9
89.6/86.1
88.7
54.4
89.2
71.0
89.8/86.4
88.3
54.9
90.4
We can observe that the performance of our approach consistently stabilizes around certain values, indicating that our method is not sensitive to the specific MPO techniques used. Therefore, when over-par vre-parrmettrizing, we should focus primarily on the decomposition scale rather than the MPO method employed.
Ablation Study Our approach consists of two novel improvements: (1) the aver-parameterization procedure for the student model, (2) the distillation loss for auxiliary tensors for effective training. To verify the effectiveness of each component, we conduct the ablation study on the GL.E benchmark to analyze the contribution of each part. We consider removing over- per--arameterization and distillation loss respectively. The ablation results of our OPDF are shown in Figure 2(c).
Firstly, it is clear that regardless of the over- over-parameterrizaion method used, the area of the radar chart is greater than that of the vanilla theseus. This outcome suggests that over par-aaterriizztion can greatly improve the performance of distillation methods. Secondly, further analysis of the different parameterizatiin methods reveals that MPO consistently outperforms SVD across all datasets. This improvement attributed to MPO's ability to decompose parameter matrices into higher orders, effectively enlarging the size of the parameter matrix. Lastly, we examine the contribution of the TAwr term. The radar chart area significantly larger when OPDF utilized in conjunction with L than with MPO alone. This indicates that Luw effectively enhances knowledge transfer from

---

the teacher model. The underlying reason for this phenomenon is that over parameeametee models can concentrate on learning central tensors containing critical information, while the Auur term assists in aligning auxiliary tensors. We can see that removing any component would lead to a decrease in the model performance. shows the effectiveness of all these components our approach.
Conclusion
In this paper, we proposed OPDF, a novel over-paran peeeraaramttiiziiooonn distillation framework designed to enhance the effectiveness of knowledge distillation. This framework employs MPO as tensor decomposition technique to expand small models into larger ones, thereby bridging the capacity gap between the teacher and student models. Moreover, to enhance the effectiveness of knowledge distillation, our proposed OPDF framework introduces tensor constraint loss. The OPDF framework utilizes MPO to decompose each weight matrix into a central tensor and auxiliary tensors. By aligning the auxiliary tensors, OPDF not only facilitates the transfer of crucial knowledge from the teacher model but also preserves the student model's ability to think independently. This approach provides the student model with the potential to outperform the teacher model. Our ablation studies demonstrated that all components of the OPDF contribute enhancing the effectiveness of knowledge distillation. Experimental results across various tasks in natural language processing and computer vision domains validate the efficacy of our proposed method in improving model distillation. Although the number of parameters was increased by MPO during training, the factorized matrices can be merged reorganize the original parameter matrix in almost lossless conditions. This means that OPDF can enhance the performance of the distillation model without increasing the inference latency. Moreover, since OPDF is based tensor decomposition, it is orthogonal most distillation methods.
In our future work, we will investigate more efficient and effective tensor decomposition methods for student model wver-parameterization. In addition, will also apply OPDF to other important backbone models, such as in the multimodal learning domains.
Impact statement
This paper proposes novel knowledge distillation framework for model compression field, which helpful to reduce storage requirements and computational complexity. This method facilitates the practical deployment models real-world applications and supports energy conservation. We focus exclusively over wer-paraamtterizing small student models, presenting potential ethical risks.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (No. 62476278, No. 62206299, No. 922701 18, and No. 11934020) and the Beijing Natural Science Foundation (No. 1232009). H.S would like to acknowledge the support from the Fundamental Research Funds for the Central Universities (No. 202230265).
References
[1] Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, htss/2001.08661, 2020.
[2] Jacob Devlin, Ming-Was Meig-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, BAACL-HLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics, 2019.
[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 2019.
10

---

[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-6) 2020.
[5] Yuxian Gu, Jiaxin Wen, Hao Sun, Yi Song. Pei Kc. Chujie Zheng, Zheng Zhang, Jianzhu Yao, Lei Liu, Xiaoyan Zhu, et al. Eva2. O: Investigating open-domain chinese dialogue systems with large-scale pre-training. Machine Intelligence Research, 20(2):00-219,, 2023.
[6] Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Yonghong Tian, and Wen Gao. Large-scale multi-modal pre-trained models: A comprehensive survey, Machine Intelligence Research, 20(4):447-482, 2023.
Wei, Yaowei Wang.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[8] Adam Roberts, Colin Raffel, and Noam Shazeer. How Much Knowledge Can You Pack Into the Parameters of a Language Model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, 2020.
[9] Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguisties, 8:423-438, 2020.
[10] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models. 2022.
[11] Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems, 36, 2024.
[12] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
[13] Brian Lester, Rami AI-Rfou, and Noah Constant. The Power of Scale for Gtrameter-Efficiee Prompt Tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[14] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[15] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski. Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual eegguage-imagee model. arXiv preprint arXiv:2209.06799 2022.
[16] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't 't Stop Pretraining: Adapt Language Models Domains and Tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5342-8360. 2020.
[17] Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training Tasks for Embedding bassed Large-scale Retrieval. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
11

---

[18] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and William Dolan. DIALOGPT: Large-Scafe Generative Pre-training for Conversational Response Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties: System Demonstrations, pages 270-278, 2020.
[19] Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured Pruning of Deep Convolutional Neural Networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(3):1-18, 2017.
[20] Eran Malach, Gilad Ychudai, Shai Shalev-Schwarke and Ohad Shamir. Proving the Lottery Ticket Hypothesis: Pruning is All You Need. In International Conference on Machine Learning, pages 1682-6691. PMLR, 2020.
[21] Zeru Zhang, Jiayin Jin, Zijie Zhang, Yang Zhou, Xin Zhao, Jiaxiang Ren, Ji Liu, Lingfei Wu, Ruoming Jin, and Dejing Dou. Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory. Advances in Neural Information Processing Systems, 34:30196-30210, 2021.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, March 2015.
[23] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguisties: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages Association for Computational Linguistics, 2020.
[24] Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. Improving BERT Fine-Tuming via Self-ensemble and Self- Distiiiiilla,,,,,, February 2020.
[25] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled Knowledge Distillation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 1943-1 1952, New Orleans, LA, USA, June 2022. IEEE.
[26] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge Distillation from A Stronger Teacher. In Advances in Neural Information Processing Systems, May 2022.
[27] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for Thin Deep Nets. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
[28] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation. In CVPR, 2019.
[29] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling Knowledge via Knowl- edge Review. In 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 5006-5015, Nashville, TN, USA, June 2021. IEEE.
[30] Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers, June 2021.
[31] Ledyard R Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 3 31(3):279-111, 1966.
[32] ER Henry and Hofrichter. [8] singular value decomposition: Application analysis of experimental data. Methods in enzymology, 210:129-192 1992.
[33] Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2255-3113 2011.
[34] Ze-Feng Gao, Song Cheng, Rong-Qiang He, Yuan Xie, Hui-Hai Zhao, Zhong-Yi Lu, and Tao Xiang. Compressing deep neural networks by matrix product operators. Physical Review Research, 2(2):023300, 2020.
12

---

[35]
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Zhi- Yuan Xie, Zhong- Yi Lu, and Ji- Ji-Rong Wen. Enabling lightweight fine-tuning for pre-trained language model compression based on matrix product operators. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Language Processing, ACI/IJCNLP 1021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 5388-5398. Association for Computational Linguistics, 2021.
[36] Ze-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong- Yi Lu, and Ji- Rong Wen. Parameter-effiient mixture-of- architecture for pre-trained language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3263-3273, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.
[37] Ze-Feng Gao, Kun Zhou, Peiyu Liu, Wayne Xin Zhao, and Ji-Rong Wen. Small pre-trained language models can be fine-tuned as large models via over- over-oarametenn parameterization. In Proceed- ings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). pages 3819-3834 Toronto, Canada, July 2023, Association for Computational Linguistics.
[38] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
[39] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[41] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP
2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 4163 4174. Association for Computational Linguistics, 2020.
[42] Hugo Touvron, Matthieu Cord, Matthijs Douze Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357. PMLR, 2021.
[43] Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge Distillation:
A Survey. International Journal of Computer Vision, 129(6):1789-1888 1819. June 2021.
[44] Yuxin Ren, Zihan Zhong, Xingjian Shi, Yi Zhu, Chun Yuan, and Mu Li. Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990-2006, Toronto, Canada, July 2023. Association for Computational Linguistics.
[45] Qinhong Zhou, Zonghan Yang, Peng Li, and Yang Liu. Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained I Language Models. In Anna Rogers, Jordan Boyd-Grabes and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13234-13248, Toronto, Canada, July 2023. Association for Computational Linguisties.
[46] Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, and Rui Wang. AD-KD: Attribution- Driven Knowledge Distillation for Language Model Compression. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages Canada, July 2023. Association for Computational Linguistics.
Toronto,
13

---

[47] Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is More: Task-aware Layer-wise Distillation for Language Model Compression. In International Conference Machine Learning, pages 20852-20867 PMLR, 2023.
[48] Bogdan Pirvu, Valentin Murg. J Ignacio Cirac, and Frank Verstraete. Matrix product operator representations. New Journal of Physics, 12(2):025012, 2010.
[49] Ze-Feng Gao, Xingwei Sun, Lan Gao, Junfeng Li, and Zhong-Yi Lu. Compressing Istm networks by matrix product operators. arXiv preprint arXiv:2012.11943, 2020.
[50] Xingwei Sun, Ze-Feng Gao, Zhong-Yi Yi Lu, Junfeng Li, and Yonghong Yan. A Model Compres- sion Method with Matrix Product Operators for Speech Enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 28:2837-28477, 2020.
[51] Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry P. Vetrov. Tensorizing neural networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 442-450, 2015.
[52] Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. Ultimate ten- sorization: compressing convolutional and FC layers alike. arXiv preprint 2016.
[53] Peiyu Liu, Ze-Feng Gao, Yushuo Chen, Xin Zhao, and Ji-Rong Wen. Enhancing scalability of pre-trained language models via efficient parameter sharing. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13771-13785, Singapore, December 2023. Association for Computational Linguistics.
[54] Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Yipeng Ma, Tao Wang. and Ji-Rong Wen. Unlocking data-free low-bit quantization with matrix decomposition for kv cache compression. arXiv preprint arXiv:2405. 12591, 2024.
[55] Ze- Be-Feng Gao, Peiyu Liu, Wayne Xin Zhao Yuan Xie, Ji-Rong Wen, and Zhong- Yi Lu. Compression image dataset based on multiple matrix product states. In Future of Information and Communication Conference, pages
Springer, 2024.
[56] Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. BERT-of-Theseus: Compressing BERT by Progressive Module Replacing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7859 7869, Online, November 2020. Association for Computational Linguistics.
[57] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A Multi- Task Benchmark and Analysis Platform for Natural Language Understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[58] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large- Scale Hierarchical Image Database. In 2009 IEEE conference on Computer Vision and Pattern Recognition, pages 248-255. Icee, 2009.
[59] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with ImageNet? arXiv preprint SxX1:22006.07159, 2020.
[60] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet? In International Conference on Machine Learning, pages PMLR, 2019.
[61] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. ViT: Fast Pretraining Distillation for Small Vision Transformers. European Conference on Computer Vision, pages 68-85. Springer, 2022.
14

---

[62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, pages 9748-8763 PMLR, 2021.
15

---

APPENDIX
A Tensor and Matrix Product Operators
As introduced in [3]. the concept of tensor is specified as:
Definition1
(Tensor). Let D1, D2... DN E N denote index upper bounds. A tensor T E DD..., of order N an N-way array where elements da..d.....d are indexed by d, E (1,2,.., Dr for 1 I n S N
Definition2
(Matrix product operator). We can reshape a matrix to high order tensor, denoted as:
Maxy M Mi ja...hn.
(S.1)
Here, the one-dimensional coordinate æ of the input signal x with dimension N₂ is reshaped into coordinate in a re-dimensional space, labeled by (1 i..).. Hence, there is a one-to-one mapping between and (i142 i.)) Similarly, the one-dimensional coordinate y of the output signal with dimension N₂ is also reshaped into a coordinate in n-dimensional space, and there is one-to-one correspondence between y and (j1j2 Jn). If 4 and Jj are the dimensions of iz and jk- respectively, then
II k=1
N., II Jx N,
(S.2)
The MPO representation of M is obtained by factorizing it into product of n local tensors
Mi = (1. ..- ( [in
(S.3)
where [(k) [jo, ik] is DE-1 D3 matrix with D3 the virtual basis dimension on the bond linking (k) and T (k+1) with Do Dr = 1.
B Theorem
Theorem 1. Suppose that the tensor w(k) of matrix W that satisfy
W (() ((k (W)())) = dk.
where
E I EL = 1,
(S.4)
Then M PO (W) with the k-th bond dimension dy upper bound of truncation error satisfy:
IW MPO (W)llF
Σ
(S.5)
Proof. The proof is by induction. For n = 2 the statement follows from the propertics of the SVD.
Consider an arbitrary 2. Then the first unfolding W()) is decomposed as
( = U, V, + (2) = (.) + (1)
(S.6)
where U, of size ri x i x fi and | 1/2 ef. The matrix B, naturally associated with (n 1)ddimensional tensor (3() with elements (3)) (a, i2, f2....... in In), which will be decomposed
further. This means that B, will be approximated by some other matrix B,, From the properties of the SVD it follows that U ( = 0, and thus
IW - B
IW, UBB.
=| - BB
-| U. -B.
(S.7)
16

---

and since U, has orthonormal columns,
|| - 1 112 E 5 B -
(S.8)
and thus it is not difficult to see from the orthonormality of columns of U, that the distance of the k-th unfolding (k 2, d4 - 1) of the (d - 1 ---diennsional tensor ( to the dy-th rank matrix cannot be larger than ex. Proceeding by induction, we have
B - Bi
2
(S.9)
k=
combine with Eq. (S.8). this complets the proof.
C
Algorithms
The over-parameterized distillation framework algorithm is shown in Algorithm S.1
Algorithm S.1 Over wver-parameterizee distillation framework.
Input: The parameter matrices list student model (Ma(k) 1k-1. parameter matrices list teacher model [Mc [k-1-
1: for = 1 + m do
2: Select (kk) which has same shape M,(k).
3: Ma(k) MPO (M.)).)
M,(k) - MPO (Me (k.)).
5:
end for
6:
repeat
7:
Compute LAUZ between (MPO (M,(k)) and {MPO (M)(k)))) by using Eq. 5 Compute distill loss Ldistut- Backward LAu and Ldisseit until Student model converges
9:
10:
The MPO pseudocode is shown in Algorithm S.2
Algorithm S.2 MPO decomposition for matrix.
Input: matrix M. the number of local tensors
Output .PO tensor list (T(x)
I: for = 1 -> T do
2: M J - M M(de-1 x Ex x Ju, -1]
UXV SVD (M)
4: Uld x is x je: de]
- 21 [d2 it Jo, de]
((k) = U
6:
M TV
7: end for
8: (()) M
Normalization
return (Tm))
D Addition Experiment Results
D.1 Memory and time cost
The distillation cost (memory and time cost) of the original model and the model after applying OPDF are shown in [S. We can observe that the number of parameters obtained from MPO decomposition increases, both the training time and memory cost increase. However, as the dataset size increases, the ratio of additional time and memory required for training by OPDF to the original training requirements generally exhibits a decreasing trend (e.g., 0.6/0.4 for RTE vs 0.3/0.1 for MNLI BERT-offTheseus model). Therefore, the additional time and memory introduced by our method become less of critical bottleneck affecting the training speed as the dataset size increases.
17

---

Table S.1: Training time and Memory Cost. (Train time(SS) Memory Cost(GB))
Datasets
RTE
MRPC
STS-B
CoLA
SST-2
QNLI
QQP
MNLI
BERT-of-Thescus
None
400.2/14.8
739.5/14.8
754,4/14.8
1365.4/14.8
2553.7/14.1
3514.2/14,1
7518.7/14.1
8873,9/1355
+OPDF (Ours)
625.8/20.7
1054,9/20.6
1932.0/20.5
21864.5/22.1
5041.6/27.D
10301.9/18.9
LGTM
None
1086,119) 1/9.7
5/98.5/9.6
2049. 1/9.6
3358.9/9.6
5270.9/9.6
8/42.8/9.6
30272.6/966
31554.8/9.6
+OPDF (Ours)
1976.3/19.7
2611.9/19.6
3603.6/19.7
2348,4/19.6
9983
14838.4/19.6
44058,5/14.7
47849.4/19.6
None
40.7/2.1
80.3/3,2
NA
186.473.2
1355.83.2
2149.2/3.2
7487.5/3.2
15513.6/3.2
+OPDF (Ours)
93.1/5.0
213.0/6.6
NA
373.3/6.2
2793.8/5.4
6076.5/6.6
14030.7/6.6
21273.4/5.0
AD-KD
308.5/3.8
351.3/3.8
495.6/3.8
780.3/5,9
3637.4/5.9
5832.715.9
28763.3/5.9
41898.A220 4/20.6
+OPDF (Ours)
1156.8/14.4
1391.3/14.5
1604,2/12.1
2249.8/18.1
8802.3/14.5
13551.1/14,5
63095,5/14.5
65735 8/34.
We show the time of overparameterization using MPO and the contraction of decomposed matrices into the original matrix in Table S.. as follows. It can be observed that the time required for decomposition and reconstruction is acceptable compared to the training duration.
Table S.2: The spending time (s) of decomposing and reconstructing.
Datasets
RTE
MRPC
STS-B
CoLA
SST-2
QNLI
QQP
MNLI
of
Decompose
397.4
308.5
400.2
154.6
797.7
671.6
584.7
137.5
Reconstruct
2.3
2.0
2.4
0.7
12.8
3.3
10.9
0.8
LGTM
Decompose
403.6
369.5
377.8
192.2
131.4
123.9
80.2
83.0
Reconstruct
8.6
6.8
7.2
2.6
0.9
1.0
1.5
DBKD
Decompose
117.5
189.1
NA
168.5
153.1
166.2
110.7
165.0
Reconstruct
0.9
1.0
NA
0.8
0.9
0.8
0.7
0.8
AD-KD
Decompose
291.7
313.0
232.6
235.5
119.0
148.0
148.1
171.6
Reconstruct
1.6
1.8
1.2
1.4
0.9
1.0
2.5
1.2
D.2 Experimental Details
As illustrated in Eq. (2), when a parameter matrix W is given, its MPO decomposition into a product of n local tensors can be represented as follows:
MPO (W) T d9...3......
(S.10)
Ji.aa.a.......
The models mentioned-Ber of theseus (56). LGTM 44). DBKD (45] and AD-KD 46 -are are all variants of BERT. which itself is built using transformer blocks. We decompose both the feed reed-forward network and the multi-head attention layer within the transformer block. Moreover, the teacher model must employ a decomposition granularity that consistent with that of the student model to ensure proper alignment of auxiliary tensors. When calculating the auxiliary loss LAuz. the alignment is typically between the n-th layer of the student model and the N-th layer of the teacher model, where N is generally an integer multiple of n. The detailed hyperparameter settings for these NLP distillation models are provided Table S. and S.4 NLP tasks, our method takes half to two GPU hours on A100 GPU.
Additionally, implement OPDF on ViT 61 demonstrate its applicability an orthogonal approach across various knowledge distillation methods that utilize the transformer architecture. Unlike NLP models, we decompose the projection layer in addition the fced-forward network and melti-head attention layer in the vision transformer block. The specific experimental parameters utilized are detailed in Table In CV tasks, our method takes 160.0 GPU days on 100 GPUs to pretrain ViT-21M We report the performance of the model that achieves the best results on the validation set when applied the test set.
18

---

Table S.3: The ford-forward network layer settings in NLP distilation model.
Experiments
RTE
MRPC
STS-B
CoLA
BERT-of- OEERI-of-Theseee Theseus 56
SVD
T
644.8
T
OPDF (Ours)
64,,1.1,144
1,,1,11,21
11,11
1..1.11.21
1.,11....118
32,1,11,224
1,
LGTM [44
SVD
T
T
T
64.48
OPDF (Ours)
1,,44
1,1,,,1
1...1. 1,4.8
1,,1,,1
1,,44
12,1,,11,
S 1,1,1,1,
1,,1,,,
DBKD [45
SVD
T 32,24
32,24
T 32,24
61
64.48
OPDF (Ours)
3.1.1.1..11
T M.L.ITTTII..
32.1 1.2.1.111111111 1.111 1.1.1,11,
1,,118
AD-KD (46
SVD
32,24
ToL.4
T64.48
OPDF (Ours)
B.I.I.I....IIIII
664
1.11.1.1,48
64,1148
32.1.21
Experiments
SST-2
QNLI
QQP
MNLI
BERT-of Th-Tssuss
SVD
64.45
64
21
T
684.48
OPDF (Ours)
1,,11.1,1,1,122 6...1.11..1.1..6
D.I.IIIIIII LIL112
1..1.
1,,1,,1,1.4
64,
32,1,24
LGTM
SVD
T64.48
T
T332221
32,21
OPDF (Ours)
64,1.1.1,48
32,1.1.1,24
T 1,11181,1,48
1,,144
Te 64.4.488
32,1.24
1,1,,,1,,2
1,.11.1..44
DBKD
SVD
T-2224
32.24
32,24
12.24
66.488
OPDF (Ours)
32. 1,,1,,1
32.11
12,1,,1111,224
32..1.1.
64.11111.1.11.4
I.I.I.I.....I........
64.1.1.1..1.48
64.1.1111111..
AD-KD
SVD
764.48
T64.55
T64.48
764.45
OPDF (Ours)
1.,,1.1...8
161,1.1,1,48
11,124
1.1.1.1.1.18
61.11.1111188
1,,,1,,,
19

---

Table S.4: The multi-head attention layer settings in NLP distilation model.
Experiments
RTE
STS-B BFRT-of-T The-seuss
MRPC
CoL.A
SVD
T
332.2
T
32.24
OPDF (Ours)
3,,1,1,1,24
12,11,111,24
32. 1..1.111,24
92,11,111222
32,1.1.1.24
32,1,1,11,24
32,,1,24
92,1
LGTM
SVD
T32,24
T
T32.24
32.24
32:21
OPDF (Ours)
1,,1,
12.11.111
1.21.111,24
7-32,1,1,12 1,,1,,1,,,2
1,,1,1.1,24
3..11.1111,2 1,1,1,22
3-32, 1,1,1,1,22 24 1,1,1,,1,1,24
DBKD
SVD
T:12.24
T32.24
T32.
OPDF (Ours)
1..1,111111224
32, 1.1.1.1111111111 1,124
I.I.I.I.III. 1..11.11,1.111111
32.111.1.1,24
12..1111.1.1..I11.
AD-KD
SVD
322.24
T32.24
32
12.24
T32.24
OPDF (Ours)
32
12..1.1.111111
732 1.1.24
1.111.11111
T32
Experiments
SST-2
QNLI
QQP
MNLI
BERT-of-
ofRT--o--heseus
SVD
32
T
322.24
2:32.24
OPDF (Ours)
32, 1,111,1,11111,, 1,,1,,11,,,,,,2
32,,11,11,,111
32,1.1,1,24
32, 1,,,1,1,1,2
32,1.4
32,1,21
LGTM
SVD
T32,21
322244
T 32,24
32.21
32.24
OPDF (Ours)
1,,2
32.1.111.22
32,11,12222
1,,,1,1,,,4
732, 1,,1
32.1.21
1,,
32,11,1..21
DBKD
SVD
T
T
T32,
32,24
T
OPDF (Ours)
32,1,1,1,1,1,24
$2.11.111111...1
32.1111111 L.L.IIIIII.
32,1,1,1.1124
1,2.1.1.1.1.22
T
32.1.
AD-KD
SVD
T32.21
T32.21
T32.21
32,24
332
24
OPDF (Ours)
1,,,1
12.111111
1,,24
1,,,,,
122.1.1.1,24
32.11.111111.
1,,
Table S.5: The experiment settings in CV distilation model.
Feod-forward Network
Multi-head Attention
Projection I Layer
Experiments
layer1
layer2
layer3
layerl
layer2
layer3
layer1
layer2
layer3
Tiny Vit-5M
SVD
T52.16
722.20
T40.22
T24.16
224.00
333.30
1.16.8 16.1.8
166.00
220.16
OPDF (Ours)
T
16.1.1
21.116
1668
16.1.16
20.1.6
116.1.30
20.11.16
Tiny
SVD
T16.8
166,16
12..1
1
16,18
T2.34
116,16
T
OPDF (Ours)
T92.1.1
7221.32
156.1.92
T24.1.16
1.2.1.44
188.1.8
716.18
716.1.6
T.2.1.14
Vi--2M
SVD
T
T48.22
T64.36
T52.18
Tan.32
754.32
T
T
32.11
OPDF (Ours)
24.188
24.1..16
71.11.18
7541..1
24,18
T
T36.1.2
24.116
32.118
754.132
T4.18.8
24.1.33
T2211.8
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Detailed descriptions all main claims can found in Section M the main claims made in the abstract and introduction supported by the experimental results presented Section 5
Furthermore,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section through over- parameterization in knowledge distillation models. The learning rate should be chosen more carefully the scale of parameterrriiinoonnn increases.
There are inherent limits to the benefits that can be achieved
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
See Appendix A and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The framework of OPDF is detailed in Section 4 and the experimental details are provided in Appendix D.2 D.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our code is available attpppp//gshuaa.cm////aaaaaaaaap
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.co) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc. wwt.///wwwlggggnieeeeeeeeeee/eee
weaibbmmsssoooPoo ssionoolccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For all the results, we report the mean values of five runs using different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: See Section 6
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Guidelines:
The paper poses such risks.
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com hmm/datasett// has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
25

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/cooe//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "HHpIsePQHR", "reviewer": "Reviewer_Qtmi", "summary": "This paper proposes a novel over-parameterization framework designed to enhance the effectiveness of knowledge distillation. This framework employs MPO as a tensor decomposition technique to expand small models into larger ones to give the student model more capacity. Moreover, to enhance the effectiveness of knowledge distillation, a tensor constraint loss is introduced to align the teacher and student model. Extensive experiments verify the superiority of the method.", "questions": "Please refer to weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Enhancing the capacity of the student model through tensor decomposition is novel and it does not incur additional inference overhead.\n2. The loss constraint for aligning the auxiliary tensors between the student and teacher models is also quite different from the conventional logit or feature matching, representing a new approach of matching in KD.\n3. The experiments are comprehensive; the authors test the method on many benchmarks in both CV and NLP, proving the effectiveness of the method.", "weaknesses": "1. I would like to know how much additional distillation cost (memory and time cost) will be incurred by introducing such tensor decomposition technique and loss constraint for aligning the auxiliary tensors between the student and teacher models. \n2. Especially in the era of LLM, the high cost of distillation often limits its application. Methods like LoRA have been proposed to reduce the trainable model parameters. I am concerned that if this approach goes against the current mainstream research directions?\n3. Can the authors provide some distillation results on larger models to further validate the applicability of the approach on large models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the constructive comments and suggestions, which are very helpful for improving our paper. We are also grateful that you recognized the strengths and contributions of our paper. Moreover, the following responses will be incorporated into the revised paper.\n\n\n**Q1. Additional distillation cost (memory and time) incurred by introducing OPDF.**\n\n**Reply:** Great remark! We listed the distillation cost (memory and time cost) of the original model and the model after applying OPDF in **Table 1** in the *one-page PDF rebuttal file*. We can observe that as the number of parameters obtained from MPO decomposition increases, both the training time and memory cost increase. However, as the dataset size increases, the ratio of additional time and memory required for training by OPDF  to the original training requirements generally exhibits a decreasing trend (e.g., 0.6/0.4 for RTE *vs* 0.3/0.1 for MNLI in BERT-of-Theseus model). Therefore, the additional time and memory introduced by our method become less of a critical bottleneck affecting the training speed as the dataset size increases. Hope this clarifies your question.\n\n\n**Q2. OPDF may conflict with mainstream research directions aimed at reducing trainable model parameters (e.g., LoRA).**\n\n**Reply:** Excellent comment! We would like to clarify that our approach has different goal compared with LoRA. Whereas LoRA is designed to reduce the number of parameters during the lightweight fine-tuning process, OPDF aims to enhance the capabilities of existing knowledge distillation models through an over-parameterization procedure.\n\nNonetheless, we performed tests to assess the effect of over-parameterization through MPO on model performance during the lightweight fine-tuning process. The results are presented in **Table A**. We observe that the implementation of MPO can augment the efficacy of BERT-base models without extending the inference duration or expanding the parameter volume. Conversely, the adoption of LoRA necessitates the parameter volume for inference, which consequently lengthens the inference time.\n\n**Table A.** The result of lightweight fine-tuning on GLUE by using MPO and LoRA.\n\n| Datasets  | RTE | MRPC | STS-B | CoLA | SST-2 | QNLI | QQP | MNLI | Avg.     | # Train Params | # Inference Params |\n| --------- | :--------: | :--------: | :-----------: | :---------: | :----------: | :---------: | :--------: | :---------: | :--------: | :--------------: | :------------------: |\n| BERT-base | 70.5     | 86.5     | 86.6        | 54.2      | 92.0       | 91.2      | 91.0     | 84.2      | 82.0     | 110M           | 110M               |\n| LoRA      | 71.5     | 89.8     | 88.6        | 58.3      | 91.5       | 90.3      | 91.7     | 83.3      | 83.1     | 295K           | 110M+295K          |\n| +MPO      | **72.3** | **90.0** | **89.0**    | **60.6**  | **92.5**   | **91.5**  | **92.3** | **85.1**  | **83.7** | 341M           | 110M               |\n\n\n**Q3. Distillation results on larger models.**\n\n**Reply:** Great comment! We have implemented OPDF on the GPT-2-760M, OPT-6.7B and LLAMA-7B models, with the corresponding teacher models of GPT-2-1.5B, OPT-13B and LLAMA-13B, respectively. We have reported the Rouge-L scores of these models on 5 instruction-following datasets, with the results displayed in **Table B**. We can observe that, after implementing OPDF, the efficiency of distillation on large models is improved across all datasets. This demonstrates that our method is also highly effective on larger models.\n\n**Table B.** Distillation results on larger models with OPDF.\n\n| Model     | #Params | Method  | Dolly    | SelfInst | Vicuna   | S-NI     | UnNI     | Avg.     | # Train Params | # Inference Params |\n| --------- | :-------: | :-------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------------: | :------------------: |\n| **GPT-2** | 1.5B    | Teacher | 27.6     | 14.3     | 16.3     | 27.6     | 31.8     | 23.5     | 1.5B           | 1.5B               |\n| **GPT-2** | 760M    | w/o KD  | 25.4     | 12.4     | 16.1     | 21.5     | 24.0     | 19.9     | 760M           | 760M               |\n| **GPT-2** | 760M    | KD      | 25.9     | 13.4     | 16.9     | 25.3     | 28.0     | 21.9     | 760M           | 760M               |\n| **GPT-2** | 760M    | KD+OPDF | **26.1** | **14.1** | **17.5** | **25.7** | **28.6** | **22.4** | 1.3B           | 760M               |\n| **OPT**   | 13B     | Teacher | 29.2     | 18.4     | 17.8     | 30.4     | 36.1     | 26.4     | 13B            | 13B                |\n| **OPT**   | 6.7B    | w/o KD  | 27.6     | 16.4     | 17.8     | 30.3     | 28.6     | 24.1     | 6.7B           | 6.7B               |\n| **OPT**   | 6.7B    | KD      | 28.3     | 17.0     | 17.5     | 30.7     | 26.7     | 24.0     | 6.7B           | 6.7B               |\n| **OPT**   | 6.7B    | KD+OPDF | **28.5** | **17.7** | **17.9** | **31.4** | **29.8** | **25.1** | 14B            | 6.7B               |\n| **LLaMA** | 13B     | Teacher | 29.7     | 23.4     | 19.4     | 35.8     | 38.5     | 29.4     | 13B            | 13B                |\n| **LLaMA** | 7B      | w/o KD  | 26.3     | 20.8     | 17.5     | 32.4     | 35.8     | 26.6     | 7B             | 7B                 |\n| **LLaMA** | 7B      | KD      | 27.4     | 20.2     | 18.4     | 33.7     | 37.9     | 27.5     | 7B             | 7B                 |\n| **LLaMA** | 7B      | KD+OPDF | **27.5** | **21.6** | **19.7** | **34.8** | **40.0** | **28.7** | 10B            | 7B                 |\n\n\n**Concluding remark:** We sincerely thank you for reviewing our paper and putting forward thoughtful comments/suggestions. We hope the above responses are helpful to clarify your questions. We will be happy to hear your feedback and look forward to addressing any additional questions. Your consideration of improving the rating of our paper will be much appreciated!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Peyk9LdaAL", "reviewer": "Reviewer_fkms", "summary": "The authors propose to start with an over-parameterised student model. This is realised using high-order tensors that can reconstruct the original parameter matrices. The idea is that this over-parameterised model will benefit more from knowledge distillation.", "questions": "What is this normalization step in figure 1 and Alg S.1 L9? and its importance? If it is just to preserve the scale and stop exploding values after reconstruction, this could be explained in the text a bit.", "limitations": "The authors have adequately addressed these limitations in the checklist.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The ideas is quite interesting/novel for over-parameterising the student during training but preserving the inference parameters (since the higher-order tensors can just be contracted to reconstruct the original weights).", "weaknesses": "TinyViT-5M⚗ achieves 80.7% top-1 with 5.4M parameters using a just a distillation token and a logit distillation loss. Comparing this with TinyViT-5M+OPDF 80.0% top-1 and 9.9M/5.4M inference parameters, it is hard to see the benefit? if anything the additional SVD operations and reconstruction losses, make OPDF much more difficult to adopt. What is worse it that this difference is even more significant when going to the larger TinyViT models.\n\nIt is understandable that transformers have been the main focus of this work, however it would be good to see some experiments with CNNs. Matrix decomposition for the transformer linear layers is easy (with SVD), but when going to higher-order tensors it will be NP-hard. All the theory presented in this paper is for generalising to arbitrary dimensions, yet all the experiments are done on 2-dimensional weights. The paper would be a lot easier to follow without the pre-mature generalisation. i.e. showing equation (4) in 2-dimensions with SVD and I am not sure if the introduction of the MPO framework is needed. \n\nL177: I am not sure what you mean by \"think independantly?\". Besides the point that models don't think, if the student can match the teacher fully then that is perfectly fine. The only goal is preserve the performance of the teacher for a much smaller model. \n\nL595: 160 days to pre-train TinyViT is very significant! I understand this is because ImageNet-21K is very large, but why was this chosen benchmark chosen over, for example, a more standard KD benchmark, such as that in DeIT [1] or CRD [2]. Is 160 days an expected scale of time for the number of GPUs used here? \n\n[1] Training data-efficient image transformers & distillation through attention ICML 2021\n[2] Contrastive Representation Distillation ICLR 2020\n\nSmall points/spelling:\nFig 1a \"Batchs -> Batches\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reply to Reviewer fkms\n\nWe sincerely thank you for the constructive comments and suggestions. The following responses will be incorporated into the revised paper.\n\n**Q1. Results on CV tasks are different from original TinyVit paper. Compared to the distillation results of the original method, OPDF does not offer any advantages.**\n\n**Reply:** Great comment! However, we would **clarify** that *the tasks (TinyViT-5M⚗ achieves 80.7\\% top-1 with 5.4M parameters) in the original TinyViT paper differ from those in Table 2 of our paper*. In lines 230-231 (please see our paper), we report CV task results based on the performance of models tested directly **without fine-tuning**, whereas the original TinyViT paper presents results **after fine-tuning**. \n\nTypically, the more parameters a pretrained model has, the greater the performance improvement after fine-tuning, which likely explains why larger models exhibit greater performance gaps. In our replication results, our method improved TinyVit's result from 77.4 to 80.0 under the same conditions, demonstrating OPDF's efficacy.\n\n**Q2. Using OPDF in CNN distillation models.**\n\n**Reply:** Thanks for your comment. Ref. [1] affirms MPO's ability to decompose CNNs, confirming its applicability. To show OPDF's effectiveness in CNN distillation, we conducted extra experiments using the methodology from Ref. [2]. This involved distilling a WideResNet from a larger to a smaller version, incorporating OPDF into OFD and classical knowlesge distillation (KD) frameworks. Results and parameters are detailed in **Table 2 and 3** of the *one-page PDF rebuttal file*.\n\nOur results demonstrate that OPDF enhances distillation performance in classical KD and OFD through over-parameterization, underscoring its adaptability in CNN models. This finding will be included in the revised paper. Additionally, as shown in **Table 3** of the *one-page PDF rebuttal file*, OPDF does not increase inference parameters, thus preserving inference time.\n\n**References:**\n\n[1] Z.F. Gao et al. Compressing deep neural networks by matrix product operators. Physical Review Research, 2020, 2(2): 023300.\n\n[2] Heo, B. et al. A comprehensive overhaul of feature distillation. ICCV2019.\n\n**Q3. The difference between MPO and SVD. Necessity and Effectiveness of MPO.**\n\n**Reply:** Thanks for your question. MPO and SVD are different. Since adding dimensions of eigenvalues in SVD does not increase the degrees of freedom, directly expanding the model parameter quantity through SVD improves *very little* of the model capacity. Moreover, the over-parametrization process via SVD is subjected to an upper limit on the number of parameters. Conversely, MPO uses matrices of eigenvectors without expanding eigenvalue dimensions, potentially allowing unlimited model parameter growth.\n\nFor example, for a matrix ${W} \\in \\mathcal{R}^{I\\times J}$, the parameter limit via SVD is $I^2 + J^2$. In contrast, over-parameterizing ${W}$ using MPO with $n$ tensors allows more parameters, as given by follwing equation, proving more effective than SVD.\n\n$$N = \\sum_{k=1}^{m} i_kj_kd_{k-1}d_k.$$\n\nBesides allowing the decomposition of the parameter matrix into an arbitrary number of effective parameters, OPDF demonstrates more significant performance improvements compared to SVD, as detailed in Section 5.2 of our pape. According to Tables 1 and 2 in our paper, while incorporating SVD generally enhances the knowlesge distillation (KD) model's performance across most datasets, it does not perform as well as OPDF.\n\n**Q4. The meaning of \"think independantly\".**\n\n**Reply:** Great remark! We will rephrase this statement to be \"learn independently\". In the OPDF, auxiliary tensor alignment allows the student model to learn task-relevant information. Through distillation with the central tensor, the student model not only can imitate the teacher model, but also equips itself with the capability to learn independently from the original labels, detached from the teacher model. Consequently, this endows the model with the potential to surpass the teacher model. \n\nConversely, if the parameter matrix is fully aligned, the student model merely emulates the teacher model. Experimental results validate this, with Table 1 showing that after adopting OPDF, the student model exceeds the teacher by 1.2% on the RTE dataset.\n\n**Q5. The training time for TinyViT is excessively long.**\n\n**Reply:** Excellent comment! One GPU day refers to *running a single GPU for one day*, and actual distillation time can be calculated using multi-GPU parallelism. By utilizing four parallel servers (each equipped with 8 NVIDIA A100 GPUs) for training, the actual training time needed amounts to 5 days. Initially, the TinyViT paper stated a distillation time of 140 GPU days, which increased to 160 GPU days (14% rise) after using the OPDF method, significantly enhancing effectiveness.\n\nTable 1 of our paper shows the parameter changes after parameterization, e.g., DBKD(53M) *vs.* DBKD+OPDF(83M), with performance increased from 74.4 to 77.6. It is evident that introducing additional training parameters incurs a certain training cost increase, yet it notably boosts the distillation performance.\n\n**Q6. The meaning and significance of normalization.**\n\n**Reply:** Great question! During MPO decomposition, normalization distributes information evenly across tensors. As depicted in Algorithm 1 in our paper, after $n$ SVD iterations, all eigenvalue information accumulates in the final tensor, while the preceding $n-1$ tensors are derived from unitary matrix decompositions and do not contain significant information. Normalization effectively distribute information across each tensor. \n\n**Concluding remark:** We sincerely thank you for putting forward excellent comments. We hope the above responses are helpful to clarify your questions. We look forward to addressing any additional questions. Your consideration of improving the rating of our paper will be much appreciated!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GPlXLK0smV", "reviewer": "Reviewer_Dajm", "summary": "This paper introduces the Over-Parameterization Distillation Framework (OPDF), which addresses performance degradation in limited-parameter student networks after knowledge distillation (KD). OPDF proposes an overparameterized student model that utilizes the tensor-decomposition technique known as matrix product operator (MPO), allowing for a significant increase in parameters during student training time without imposing additional inference time burden. Experimental validation is performed across multiple KD tasks to assess the effectiveness of the proposed technique.", "questions": "Please refer to Weaknesses section", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The proposed technique appears to be quite novel. The use of MPO to expand parameters in the student model during the KD process, along with the tensor alignment loss function to improve student model performance, introduces innovative approaches that could offer significant advantages, particularly on low-computational devices.\n+ The proposed methodology is easy to understand, even though it includes some abstract concepts. The authors have effectively structured their methodology by first providing a high-level overview of their technique with an illustrative figure, followed by detailed explanations of their important components. \n+ Extensive experiments are conducted across both NLP and CV tasks. Multiple knowledge distillation (KD) techniques are employed to demonstrate how their  contributions is orthogonal to existing methods.   Also, parameters introduced in training as well as inference process are clearly shown.\n+ The study extensively examines the impact of overparameterization scale, learning rate, and other components of ODPF through ablation experiments.  This study helps to justify the effectiveness of their technique.", "weaknesses": "- The paper does not provide information on the time required for the student network's overparameterization using MPO and the contraction  of decomposed matrices into the original matrix. Having this information would be important for understanding the practical implications of the technique.\n- The experiments are conducted on a relatively smaller model. I am curious about the feasibility of applying this technique to LLM/VLMs with billions of parameters. A significant concern is whether their approach, which involves decomposing the teacher network, can scale effectively to such large models. A deeper discussion on this topic would provide valuable insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the positive feedback along with constructive comments and suggestions, which are very helpful in improving our paper. We are also grateful that you recognized the strengths and contributions of our paper. Moreover, the following responses will be incorporated into the revised paper.\n\n\n**Q1. Time Consumption for decomposing and reconstructing the parameter matrix.**\n\n**Reply:** This is a great remark. We listed the time of overparameterization using MPO and the contraction of decomposed matrices into the original matrix in **Table A** as follows. It can be observed that the time required for decomposition and reconstruction is acceptable compared to the training duration (please alse see Appendix C).\n\n**Table A.** The spending time (s) of decomposition and reconstruction.\n\n| Cases                             | RTE   | MRPC  | STS-B | CoLA  | SST-2 | QNLI  | QQP   | MNLI  |\n| ------------------------------- | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| **BERT-of-Theseus** Decomposition   | 397.4 | 308.5 | 400.2 | 154.6 | 797.7 | 671.6 | 584.7 | 137.5 |\n| **BERT-of-Theseus** Reconstruction | 2.3   | 2.0   | 2.4   | 0.7   | 12.8  | 3.3   | 10.9  | 0.8   |\n| **LGTM** Decomposition              | 403.6 | 369.5 | 377.8 | 192.2 | 131.4 | 123.9 | 80.2  | 83.0  |\n| **LGTM** Reconstruction           | 8.6   | 6.8   | 7.2   | 2.6   | 1.6   | 0.9   | 1.0   | 1.5   |\n| **DBKD** Decomposition              | 117.5 | 189.1 | Na     | 168.5 | 153.1 | 166.2 | 110.7 | 165.0 |\n| **DBKD** Reconstruction            | 0.9   | 1.0   | Na     | 0.8   | 0.9   | 0.8   | 0.7   | 0.8   |\n| **AD-KD** Decomposition             | 291.7 | 313.0 | 232.6 | 235.5 | 119.0 | 148.0 | 148.1 | 171.6 |\n| **AD-KD** Reconstruction          | 1.6   | 1.8   | 1.2   | 1.4   | 0.9   | 1.0   | 2.5   | 1.2   |\n\n\n\n**Q2. Applying OPDF to model with billions of parameters.**\n\n**Reply:** This is an insightful question. We have implemented OPDF on the GPT-2-760M, OPT-6.7B, and LLAMA-7B models, with corresponding teacher models of GPT-2-1.5B, OPT-13B, and LLAMA-13B, respectively. The Rouge-L scores of these models on five instruction-following datasets are presented in **Table B** (see below). Our results indicate that OPDF significantly enhances the distillation efficiency for larger models across all datasets, demonstrating its efficacy even for models with billions of parameters.\n\n**Table B.** Distillation results on larger models with OPDF.\n\n| Model     | #Params | Method  | Dolly    | SelfInst | Vicuna   | S-NI     | UnNI     | Avg.     | # Train Params | # Inference Params |\n| --------- | :-------: | :-------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------: | :--------------: | :------------------: |\n| **GPT-2** | 1.5B    | Teacher | 27.6     | 14.3     | 16.3     | 27.6     | 31.8     | 23.5     | 1.5B           | 1.5B               |\n| **GPT-2** | 760M    | w/o KD  | 25.4     | 12.4     | 16.1     | 21.5     | 24.0     | 19.9     | 760M           | 760M               |\n| **GPT-2** | 760M    | KD      | 25.9     | 13.4     | 16.9     | 25.3     | 28.0     | 21.9     | 760M           | 760M               |\n| **GPT-2** | 760M    | KD+OPDF | **26.1** | **14.1** | **17.5** | **25.7** | **28.6** | **22.4** | 1.3B           | 760M               |\n| **OPT**   | 13B     | Teacher | 29.2     | 18.4     | 17.8     | 30.4     | 36.1     | 26.4     | 13B            | 13B                |\n| **OPT**   | 6.7B    | w/o KD  | 27.6     | 16.4     | 17.8     | 30.3     | 28.6     | 24.1     | 6.7B           | 6.7B               |\n| **OPT**   | 6.7B    | KD      | 28.3     | 17.0     | 17.5     | 30.7     | 26.7     | 24.0     | 6.7B           | 6.7B               |\n| **OPT**   | 6.7B    | KD+OPDF | **28.5** | **17.7** | **17.9** | **31.4** | **29.8** | **25.1** | 14B            | 6.7B               |\n| **LLaMA** | 13B     | Teacher | 29.7     | 23.4     | 19.4     | 35.8     | 38.5     | 29.4     | 13B            | 13B                |\n| **LLaMA** | 7B      | w/o KD  | 26.3     | 20.8     | 17.5     | 32.4     | 35.8     | 26.6     | 7B             | 7B                 |\n| **LLaMA** | 7B      | KD      | 27.4     | 20.2     | 18.4     | 33.7     | 37.9     | 27.5     | 7B             | 7B                 |\n| **LLaMA** | 7B      | KD+OPDF | **27.5** | **21.6** | **19.7** | **34.8** | **40.0** | **28.7** | 10B            | 7B                 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OJydVLm4bE", "reviewer": "Reviewer_XV5e", "summary": "In this work, the authors focus on the knowledge distillation (KD) task, using tensor decomposition to enhance the performance of the student model. Leveraging the principle of overparameterization, the authors employ the Matrix Product Operator (MPO), also known as tensor train matrix, to reformulate the original weight matrix. Additionally, they propose a “distillation loss” to measure the distance between the student’s weights and the teacher’s weights. In the experiments, the proposed method is integrated with existing KD methods, and the results demonstrate a clear improvement in KD performance.", "questions": "1.\tSince the entire paper is based on the principle of overparameterization of the student model, it would be beneficial to explain in the preliminary section why the overparameterization principle is relevant to the KD problem. This addition would help non-experts follow the main idea of the paper more smoothly.\n\n2.\tIn lines 176-177, the phrase “losing its ability to think independently” is highlighted. I am confused by this statement. Could you provide more interpretation of this claim, supported with sufficient evidence?\n\n3.\tPlease offer more interpretation of the central and auxiliary tensors mentioned in lines 184-185, using formulas or figures, for example. I cannot clearly understand their differences from the current non-rigorous descriptions.\n\n4.\tWhy does using MPO provide better performance than SVD? Is it because the MPO forms more “linear layers” than SVD? Have you considered other tensor network formats such as tree tensor networks?\n\n5.\tHow does the selection of TT-ranks (e.g., d_k in Eq. 2) affect the performance?", "limitations": "The paper lacks a discussion on the limitations of the proposed method. I suggest that the authors address this aspect to provide a more balanced and comprehensive evaluation of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces an innovative application of tensor decomposition. Typically, tensor decomposition is used for dimension reduction, adhering to the low-rank principle. However, this paper utilizes tensor decomposition in a novel manner: employing MPO to construct overparameterized learning models. While I am not entirely convinced why the overparameterized MPO is superior to traditional matrix decomposition or other tensor networks, the numerical results indicate that this approach could be a promising new avenue for using tensor networks to solve more machine learning problems.", "weaknesses": "The clarity of the paper needs improvement. For instance, Figure 1 is not fully comprehensible as the meaning of the arrows is not clearly explained. Additionally, the experiment settings do not clearly describe how the weights are reshaped into the higher-order tensor format.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the constructive comments and suggestions, which are very helpful in improving our paper. The following responses will be incorporated into the revised paper.\n\n**Q1. The impact of over-parameterization on student model performance.**\n\n**Reply:** Thank for your excellent comment. Since the performance of student models are typically limited by their number of parameters, increasing the number of model parameters can significantly enhance performance (*e.g.* TinyViT-21M (91.9) *v.s.* TinyViT-5M (87.6) in top-5 accuracy on Imagenet V2). Thus, increasing parameters is beneficial for enhancing the performance of the student model. We will include information about the impact of over-parameterization on the performance of student model in the revised version of the preliminary section.\n\n**Q2. The meaning of \"think independantly\".**\n\n**Reply:** Great remark! We will rephrase this statement to be \"learn independently\". In the OPDF, auxiliary tensor alignment allows the student model to learn task-relevant information. During the distillation process, learning through the central tensor not only enables the model to imitate the teacher model, but also equips it with the capability to learn independently from the original labels, detached from the teacher model. Consequently, this endows the model with the potential to surpass the teacher model. \n\nConversely, if the entire parameter matrix is directly aligned, then the student model can only emulate the teacher model. The effectiveness of this method is further evidenced by the experimental results. As shown in Table 1 in our paper, after incorporating OPDF, the student model outperforms the teacher model by 1.2% on the RTE dataset.\n\n**Q3. The interpretation of central and auxiliary tensors.**\n\n**Reply:** Thanks for your comment. The definition of central and auxiliary tensors is shown in lines 152-153 in our paper (please kindly refer to the Figure 1b). MPO allows for the decomposition of parameter matrices into a series of tensors:\n\n$${MPO}~(\\mathbf{M})= \\prod_{k=1}^n \\mathbf{T_{(k)}}[d_{k-1},i_k,j_k,d_k].\\tag{1}$$\n\n$$d_k = \\min\\bigg(\\prod_{m=1}^k i_m\\times j_m, \\prod_{m=k+1}^n i_m\\times j_m\\bigg).\\tag{2}$$\n\nFollowing Refs. [1,2], the tensor right in the middle is termed as central tensor, and the rest as auxiliary tensor.\n\n*References:*\n\n[1] P. Liu, et al. Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators.ACL2021\n\n[2] Z.F. Gao, et al. Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models. COLING2022.\n\n\n**Q4. Reasons why MPO outperforms SVD  and the justification for not using MPO over other tensor networks.**\n\n**Reply:** This is an insightful question. MPO generally surpasses SVD because adding eigenvalue dimensions in SVD improves very little of the model capacity, due to the fixed degrees of freedom, expanding parameters through SVD has an upper limit. In contrast, MPO employs matrices of eigenvectors without increasing eigenvalue dimensions, enabling potentially unlimited growth in model parameters.\n\nWe also have considered other tensor network format like CP decomposition (CPD) and Tucker decomposition. Generally, the algorithm capacity is larger as the number of tensors $n$ increases (denoting more tensors). When $n > 3$, MPO has smaller time complexity than Tucker decomposition. It noted that SVD can be considered as a special case of MPO when $n = 2$ and CPD is a special case of Tucker when the core tensor is the super-diagonal matrix.  The specifics are illustrated in **Table A**. We will investigate other tensor networks as part of our further work.\n\n**Table A.** Inference time complexities of different low-rank approximation methods. Here, $n$ denotes the number of the tensors, $m$ denotes $max({{{\\{i_k\\} }^n_{k=1}}})$ means the largest ${i_k}$ in input list, and $d$ denotes $max({{{ \\{d_k^{'} \\} }^n_{k=0}}})$  meaning the largest dimension $d_{k}^{'}$ in the truncated dimension list. \n\n| Category    | Method| Inference Time |\n| ----- | :-------------------: | :--------------: |\n| **Trucker** | $Trucker_{(d=1)} (CP)$ | $O(nmd^2)$     |\n| **Trucker** | $Trucker_{(d>1)}$          | $O(nmd + d^n)$  |\n| **MPO**     | $MPO_{(n=2)} (SVD)$    | $O(2md^3)$     |\n| **MPO**     | $MPO_{(n>2)}$              | $O(nmd^3)$     |\n\n\n**Q5. The impact of TT-Ranks on performance.**\n\n**Reply:** Excellent comment! Indeed, we have conducted experiments regarding the impact of $d_k$ on the model performance (please kindly refer to Appendix Table S.4). We can observe that the performance of our approach consistently stabilizes around certain values, indicating that our method is *not sensitive* to the specific MPO techniques used. Therefore, when over-parameterizing, we should focus primarily on the decomposition scale rather than the MPO method employed.\n\n**Q6. The limitations of OPDF.**\n\n**Reply:** Great remark! As discussed in Section 5.3 in our paper, while OPDF can enhance the performance of the student model through over-parameterization, there are inherent limits to these benefits. Additionally, due to over-parameterizing the KD model, OPDF results in higher memory consumption and longer training cost. The memory usage and training cost before and after using OPDF can be found in **Table 1** in the *one-page PDF rebuttal file*.\n\nWhile OPDF increases the memory usage and training time, this impact diminishes as the dataset size grows, meaning that the ratio of additional memory and training time to the original requirements decreases. We will include an enhanced discussion on the limitations of OPDF in the revised paper.\n\n**Concluding remark:** We sincerely thank you for putting forward excellent comments. We hope the above responses are helpful to clarify your questions. We look forward to addressing any additional questions. Your consideration of improving the rating of our paper will be much appreciated!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Global Response:

Dear Reviewers:

We would like to thank you for your constructive comments, which are very helpful in improving our paper. We have posted the point-to-point reply to each question/comment raised by you. And we have listed three additional tables in the *one-page PDF rebuttal file*, which contained "Training time and Memory Cost for OPDF", "Distillation results on various CNN architectures with OPDF" and " Train Params and Inference Params of CNN model". Please do feel free to let us know if you have any further questions.

We are also pleased that the reviewers recognized the novelty and versatility of our work. In particular, we thank the reviewers for recognizing the *preservation of the inference parameters* (Qtmi and fkms), *adequacy of the experiment* (Dajm), and *novelty* (XV5e) of our method.

Thank you very much.

Best regards,

The Authors of the Paper

Meta-review: The paper presents a novel framework for enhancing knowledge distillation (KD) by leveraging over-parameterization. Using the Matrix Product Operator (MPO) as a tensor decomposition technique, the authors propose to increase the capacity of the student model during training, improving its ability to learn from the teacher model without adding inference overhead. Additionally, a loss is introduced to align the student and teacher models more effectively. Extensive experiments across various vision and NLP benchmarks showcase the utility of the proposed KD framework.

Reviewers appreciated the innovative approach, particularly the use of tensor decomposition to over-parameterize the student model. The introduction of a tensor constraint loss for better alignment between teacher and student models was also highlighted as a significant strength. The comprehensive experimental validation was seen as a key strength. However, some concerns were raised about the clarity of the paper, particularly in explaining certain concepts, and the scalability and computational cost of the proposed method, especially for larger models. Despite these concerns, I will recommended the paper for acceptance and suggest the authors incorporate the feedback.
Decision: Accept (poster)