Title: Transformer Compression via Subspace Projection
Venue: NeurIPS 2023
Accepted: no
Keywords: low-rank factorization

model compression

transformer
Abstract: We propose TCSP, a novel method for compressing a transformer model by focusing on reducing the hidden size of the model. By projecting the whole transform model into a subspace, we enable matrix operations between the weight matrices in the model and features in a reduced-dimensional space, leading to significant reductions in model parameters and computing resources. To establish this subspace, we decompose the feature matrix, derived from different layers of sampled data instances, into a projection matrix. 
For evaluation, TCSP is applied to compress T5 and BERT models on the GLUE and SQuAD benchmarks. Experimental results demonstrate that TCSP achieves a compression ratio of 44\% with at most 1.6\% degradation in accuracy, surpassing or matching prior compression methods. Furthermore, TCSP exhibits compatibility with other methods targeting filter and attention head size compression.

1

Abstract

We propose TCSP, a novel method for compressing a transformer model by focusing on reducing the hidden size of the model. By projecting the whole transform model into a subspace, we enable matrix operations between the weight matrices in the model and features in a reduced-dimensional space, leading to significant reductions in model parameters and computing resources. To establish this subspace, we decompose the feature matrix, derived from different layers of sampled data instances, into a projection matrix. For evaluation, TCSP is applied to compress T5 and BERT models on the GLUE and SQuAD benchmarks. Experimental results demonstrate that TCSP achieves a compression ratio of 44% with at most 1.6% degradation in accuracy, surpassing or matching prior compression methods. Furthermore, TCSP exhibits compatibility with other methods targeting filter and

attention head size compression.

1 Introduction

The transformer model [1] is widely used in Natural Language Processing as well as other do14 mains such as Computer Vision [2, 3, 4] and Speech Recognition [5, 6, 7]. Despite its impres15 sive performance, the large size of transformer models and the high inference latency limit their 16 practical deployment. To address this challenge, model compression techniques including prun17 ing [8, 9, 10, 11, 12, 13] and low-rank decomposition [14, 15, 16, 17] have been proposed to reduce 18 model parameters and improve inference speed while ensuring that the performance of the compressed 19 model is not significantly disturbed. 20

The transformer model consists of a series of weight matrices that are determined by the hidden 21 size d , attention head size d h in the multi-head attention layers, and the number of filters d f in the 22 feed-forward network layers. Existing compression methods primarily focus on reducing the attention 23 head size d h [14, 8, 13] and the number of filters d f [8, 13], or performing matrix decomposition 24 to transform d × d matrices into two d × k matrices [14, 15, 16]. However, none of these methods 25 directly address the reduction of the hidden size d . Although CoFi [13], a pruning method, attempts 26 to compress d , it can only achieve a 1% reduction while keeping model performance. 27

This paper introduces a novel method, named T ransformer C ompression via S ubspace P rojection 28

(TCSP), for compressing transform models by reducing the hidden size. By projecting the transformer 29

model into a subspace, TCSP achieves this compression. To create this subspace, we employ low-rank 30

factorization to decompose the feature matrix derived from sampled data instances into a projection 31

matrix. Specifically, we gather multiple data instances, pass each through the transformer model to 32

obtain their respective features from different layers, and concatenate them to form a feature matrix. 33

Decomposing this matrix yields the projection matrix, which allows us to project the model into a 34

subspace, resulting in a significant reduction in model parameters. Furthermore, TCSP can be easily 35

combined with other compression methods that reduce attention head size and the number of filters, 36

ensuring compatibility. Although low-rank factorization is employed in TCSP, it is considered a 37

sub-technique to achieve the primary goal of reducing hidden size. 38

Our contributions can be summarized as follows: 39

40

41

We take a fresh perspective on model compression by reducing the hidden size of the transformer model, which has been rarely explored before.

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

We propose the TCSP technique to achieve the compression goal. TCSP decomposes the feature matrix derived from sampled data instances into a projection matrix, which is then used to project the transformer model into a subspace. In addition, TCSP is compatible with other compression methods, such as reducing the multi-head attention size and the number of filters, which could further speed up inference.

Experimental results on two widely-used benchmarks, GLUE and SQuAD, show that TCSP can compress 44% of the parameters of both T5 and BERT within 1.6% accuracy loss, surpassing or matching prior compression methods.

2 Related Work

The transformer model is a general model that has been widely used in the fields of deep learning. To improve inference speed and reduce memory overhead, different approaches have been proposed to compress the transformer. Previous research [18] categorize these approaches into five distinct categories: quantization [19, 20, 21], pruning [8, 9, 10, 11, 12, 13], knowledge distillation [22, 23], low-rank factorization [14, 15, 16, 17], and weight sharing [24, 25]. These five types of methods are orthogonal to each other. In this work, we focus on exploring low-rank factorization and pruning approaches as means to directly reduce the number of parameters in fined-tuned task-specific models. We do not delve into Knowledge distillation and weight sharing, as they involve training models from scratch. We do not cover quantization, which compresses models based on storage considerations.

Low-Rank Factorization for Transformer. In recent years, various low-rank decomposition methods have been proposed specifically for Transformers. Ma et al. [26] introduce a block term tensor decomposition approach to decompose the multi-head attention in Transformers. Noach et al. [27] propose a two-stage method for Transformer compression, where they first employ SVD to decompose Transformer's weight matrix and then fine-tune the model using knowledge distillation. However, direct SVD-based compression of the weight matrix assumes that it possesses low-rank properties, which may not always hold for Transformers. To address this issue, Chen et al. [14] propose a data-aware low-rank factorization method. This method minimizes the reconstruction error of the matrix multiplication between each weight matrix and its corresponding input feature matrix rather than directly minimizing the reconstruction error of the weight matrix itself. Hsu et al. [15] and Hua et al. [16] propose FWSVD and TFWSVD, respectively, which utilize Fisher information to measure the contribution of different parts of the weight matrix to model performance during the SVD process, achieving improved results compared to direct SVD. Alternatively, Tahaei et al. and Edalati et al. [28, 29] employ Kronecker decomposition to preserve the matrix rank and successfully compress models like BERT and GPT2.

Pruning for Transformer . Pruning is a commonly used technique for eliminating unnecessary parameters in the model. Existing pruning methods can be broadly divided into two categories: unstructured pruning and structured pruning. Unstructured pruning aims to remove unimportant scalar values in model's parameters. Various unstructured pruning approaches have been proposed for Transformer, such as magnitude-based [30], first-order [31], second-order [32], and lottery ticket hypothesis [33]. Although unstructured pruning algorithms can remove many redundant parameters while ensuring accuracy, compressed models require specific sparse data structures and hardware support to take advantage of unstructured pruning. For this reason, structure pruning approaches [8, 9, 10, 11, 12, 13] are proposed to remove weight blocks in the transformer model, including entire transformer layers, attention heads, and filters. Unlike unstructured pruning, structure pruning can accelerate inference speed and reduce memory overhead without specialized data structure and hardware.

Original

Transformer

Sampled

Dataset

Entire

Dataset

Compressed

Transformer

Weight Matrices

×𝐿

×𝐿

Compressed Matrices

Feature Matrix

Finetuned

Compressed

Transformer

(1) Generating Projection Matrix

(2) Fusing Projection and Weight Matrix

(3) Fine-tuning Weight Matrix

𝑊

෡ 𝑊

𝑊

𝑃

෡ 𝑊

SVD

Σ

𝑃

𝑋

V

𝑋

Features

…

Projection Matrix

D

88

×

3 Overview 87

3.1 Preliminary: Transformer Architecture 88

We take T5 [34] as an example of the transformer model to present the proposed TCSP, although it 89 can be easily extended to other transform models such as BERT [35] and GPT [36]. 90

91

A transformer model comprises a stack of blocks, each containing a multi-head attention (MHA)

layer, an optional cross-attentive (CA) layer, and a feed-forward network (FFN) layer. 92

Multi Head Attention (MHA) . In a transformer encoder, each MHA layer consists of H attention 93

heads that facilitate interactions between tokens, with a pre-normalization step. Other transform 94

models such as BERT adopt post-normalization after the MHA layer. 95

where norm( x ) represents the normalization function such as LayerNorm and RMSNorm, 96 W Q , W K , W V , W O ∈ R d h × d are the parameters of an MHA layer, denoting the query, key, value, 97 and output matrices, respectively. Here d denotes the hidden size, and d h = d/H denotes the 98 attention head size. 99

In a transformer decoder, a CA layer is added after the MHA layer. The CA layer closely resembles 100 the MHA, with one only distinction: the feature matrices multiplied by W Q and W K in the CA layer 101 are sourced from the output of the Transformer encoder. Further information on the CA layer can be 102 found in Appendix A. 103

104

105

106

107

Feed Forward Network (FFN) . Following each MHA layer, there is an FFN layer that takes x M as input and generates x F as output:

where σ is the activation function, and W U , W D ∈ R d f × d are the parameters of the FFN layer, which correspond to the up and down matrices, respectively. Here d f indicates the number of filters.

3.2 Basic Idea of TCSP: Subspace Projection 108

The basic idea of T ransformer C ompression via S ubspace P rojection (TCSP) is to project the 109 Transformer model into a suitable subspace via matrix fusion. To provide a clearer explanation, we 110 take the linear regression model as an example. Here we denote the original model F as: 111

where W ∈ R 1 × d , x ∈ R d . 112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

Assume that the input x of the model is distributed in a k -dimensional subspace. In other words, there exists a projection matrix P ∈ R d × k such that every x in the dataset satisfies x = P ˆ x , where ˆ x ∈ R k . Thus, we have,

where ˆ W = WP ∈ R 1 × k represents the compressed weight matrix, ˆ x ∈ R k . In this way, we project the original model from a d -dimensional space into a reduced k -dimensional subspace. We will elaborate on how to extend this method to the transformer model.

Workflow of TCSP. The proposed TCSP comprises three stages, as illustrated in Figure 1. Firstly, given the training data and the original transformer model (e.g., a fine-tuned T5), we sample a subset of the training data, feed it into the transformer model to obtain the feature matrix X , and employ SVD on X to derive the projection matrix P . Then, we project the weight matrices { W } of the original transformer to a subspace via the fusion of the model's weight matrices and the projection matrix, resulting in { ˆ W } . Finally, following prior work [14, 15, 16], we fine-tune the compressed transformer with the entire training data.

4 Methodology

4.1 Generating Projection Matrix

Motivated by the fact that features produced by the transformer model usually tend to reside in a low-dimensional subspace, our objective is to carry out the forward pass of the transformer model in the low-dimensional subspace. To achieve this, the initial step of TCSP focuses on determining the subspace where the features are located. This can be formalized as a optimization problem:

where N denotes the number of layers in the transformer, L ( i ) represents the i -th layer in the transformer model, which can correspond to different components such as the MHA or the FFN layer. L (1 ∼ i ) = L ( i ) ◦ . . . L (2) ◦ L (1) represents the composite function that is formed by sequentially applying the transformations from the first layer to the i -th layer. L (1 ∼ i ) ( x ) refers to the application of the composition function L (1 ∼ i ) on the input x , resulting in its corresponding feature. I is an identity matrix, and P is the desired projection matrix. For a given projection matrix P , P /latticetop x can be interpreted as projecting x from the original space into the subspace corresponding to the matrix P , while PP /latticetop x can be viewed as projecting P /latticetop x from the subspace back into the original space. Therefore, we can use x -PP /latticetop x to measure whether the vector x belongs to the subspace.

141

where 142

Let l m denote the sequence length of the m -th data instance, and d denote the hidden size. Conse143 quently, the shape of X can be expressed as d × ( N × ∑ M m =1 l m ) , where N is the number of layers, 144 and M is the number of the sampled data instances. 145

Eq. 6 is equivalent to the following equation, the proof of which is shown in Appendix B.

Such an optimization problem in Eq. 7 can be well solved by SVD [37]. 146

where U and V are two orthogonal matrices, and Σ is a diagonal matrix.

Then, the first k columns of the matrix U (i.e., the topk important eigenvectors of the feature matrix X ) compose the desired projection matrix P .

Appendix G.1 describes the process of generating the projection matrix P . Notably, although we sample a subset of data instances for computing feature matrix X , the size of X is still large, resulting in a significant overhead of SVD. To mitigate this, we only select a subset of columns from X to create a submatrix for SVD. Following previous work [14, 8], we approximate the SVD by sampling 2,000 tokens, which corresponds to the number of columns in matrix X . In the experiments, we examine the impact of the number of sampled tokens on the performance of TCSP.

4.2 Fusing Projection and Weight Matrix

Once the projection matrix P for the the transformer model has been obtained, we can construct an approximation of the original model within the subspace using the projection matrix. The basic idea behind this is to fuse the projection matrix with the weight matrices of the transformer.

Given an input x , we pass it through the transformer model starting from the first layer to the last N -th layer, resulting in the corresponding feature vector L (1 ∼ N ) ( x ) . According to Eq.6, we can use the projection matrix P to add dimensionality reduction and dimensionality enhancement operations between each layer, while preserving the final outcome, as shown in the following equation,

where U ( x ) = Px represents the dimensionality enhancement operation, D ( x ) = P /latticetop x represents the dimensionality reduction operation, ˆ L i = D ◦ L i ◦ U is the projected layer, and ˆ L (1 ∼ N ) = ˆ L ( N ) ◦ · · · ◦ ˆ L (2) ◦ ˆ L (1) is the projected model.

According to Section 3.1, any original layer L ( i ) ( x ) can be formulated as:

where the function 'Layer' can be instantiated by various components such as MHA, CA, and FFN. The projected layer ˆ L operates on a k -dimensional feature ˆ x as input and produces a corresponding k -dimensional feature ˆ L (ˆ x ) , i.e.

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

Compared with the d -dimensional input and output vectors of the original layer L ( x ) , the projected 171 layer ˆ L (ˆ x ) in Eq.16 reduces the dimensions of the layer's input and output to k . However, this 172 reduction in dimensionality does not alleviate the computational effort. Therefore, we propose two 173 methods to compress the parameters within the projected layer: (1) Matrix fusion, which merges 174 the multiple consecutive matrix multiplications into a single matrix. (2) Normalization function 175 reconstruction, which is used to swap the computational order of matrix operations and normalized 176 functions to increase the chance of matrix fusion. Next, we first explain the normalization function 177

reconstruction and then discuss how the matrix fusion technique can be employed to compress the 178 MHAand FFA layers. 179

Normalization Function Reconstruction . Within the projected layer ˆ L as shown in Eq.16, we 180 encounter the computation of norm( P ˆ x ) . Additionally, we know that the operation following 181 norm( P ˆ x ) is a matrix multiplication. Thus, if we can find a new matrix ˆ P along with a normalization 182 function ˆ norm that satisfies norm( P ˆ x ) = ˆ P ˆ norm(ˆ x ) , we can compress the parameters of the 183 transformer model by fusing the matrix ˆ P with the subsequent matrix. 184

The normalization function in T5 is RMSNorm [38]: 185

Therefore, we have 186

where ˆ P = √ d k γ P , ˆ norm(ˆ x ) = I ˆ x 1 √ k ‖ ˆ x ‖ (note that ‖ ˆ x ‖ = ‖ P ˆ x ‖ ).

By incorporating Eq.18 into Eq.16, we can define

where ˆ norm(ˆ x ) is abbreviated as as ˆ x n .

For detailed information on reconstructing other normalization functions such as LayerNorm [39] and BatchNorm [40], please refer to the appendix D. The post-normalization reconstruction is also explained in appendix D.

MHALayer Compression . We can compress the transformer model's parameters by fusing multiple consecutive matrix multiplications into a single matrix. For the MHA layer, we have

187

188

189

190

191

192

193

194

where ˆ W ( i ) O = W ( i ) O P, ˆ W ( i ) V = W ( i ) V ˆ P, ˆ W ( i ) K = W ( i ) K ˆ P, ˆ W ( i ) Q = W ( i ) Q ˆ P . The shape of these 195 matrices is transformed from d h × d to d h × k , resulting in a reduction of k/d of the original number 196 of parameters. We introduce the CA layer compression in Appendix E. 197

198

199

200

201

202

203

FFN Layer Compression . For the FFN layer, we have

where ˆ W D = W D P, ˆ W U = W U ˆ P , which has the same compression rate as the MHA Layer. The complete compression process is shown in Appendix G.2.

4.3 Combining TCSP with Other Compression Algorithms

TCSP compresses the hidden size d , and we can further compress the number of filters d f and attention head size d h by the prior pruning and low-rank factorization methods.

Filter Pruning . We can remove the filters in the FFN layer by pruning. Following prior work [13, 8], 204 we introduce mask variables diag( m ) into the FFN layer, where m indicates which filters in the FFN 205 layer are to be retained. 206

Then the pruning problem can be formalized as an optimization problem on the mask.

Our proposed TCSP method only involves the forward computation process of the model. However, most prior methods find the optimal mask by leveraging the derivative of the loss function with respect to the mask variable ∂ L ∂ m . As a result, TCSP cannot be directly integrated into the same framework as previous pruning algorithms. To address this, we propose TCSP-pruning, a filter pruning algorithm that exclusively relies on the forward process of the model, enabling the unifying of the pruning algorithm into the TCSP framework.

TCSP-pruning employs a greedy algorithm to remove the least important filter at each step based on the following score function:

where score( i ) represents the significance of the i -th filter. We believe that the importance of the i -th filter is influenced by the factors such as the norm of W D,i (the weight matrix associated with the i -th filter), the average activation value, and the variance of activation values. The pseudo-code of TCSP-pruning is shown in the Appendix G.3.

Head Size Compression . Based on the low-rank property of data distribution, we can further compress the head size of the MHA layer. The objective can be expressed as:

DRONE [14] gives the solution to the above two optimization problems. Assuming that the optimal solution obtained from the first optimization problem is M ∗ . Afterward, we decompose M ∗ into the product of two matrices M ∗ = U /latticetop M V M through SVD. Then, we can compress matrices W Q , W K through U M and V M ( ˆ W K = U M W K , ˆ W Q = V M W Q ). The second problem is addressed in the same way. We can also formalize Eq.7 as arg min M ‖ X -MX ‖ 2 F s.t. rank( M ) = k , but the optimal solution obtained by both formulas is the same.

5 Experiment

5.1 Experimental Setup

To evaluate our method, we first fine-tune T5 base and BERT base on the training data of each task of the GLUE[41] and SQuAD [42] benchmarks to obtain the baseline models, and then adopt the proposed TCSP or the comparison compression methods to compress these baseline models. On each task, for TCSP, we sample 2000 instances from its training data to produce the project matrix, using it to compress the baseline models and finally fine-tune the compressed models using the entire training data. For the GLUE benchmark, We report accuracy for the MNLI [43], QQP [44], QNLI [41], and SST2 [45] tasks, as well as F1 score and spearman correlation for the MRPC [46] and STSB [41] tasks. For the SQuAD benchmark, we report the F1 score. For more comprehensive information regarding the experimental setup, please refer to Appendix F.

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

5.2 Performance Evaluation

Table 1 shows the accuracy results of T5 base and BERT base with different compression rates. The proposed TCSP can reduce the hidden size d . Additionally, in Section 4.3, we introduce a filter pruning method that can reduce the number of filers d f . Moreover, the existing DRONE [14] can compress the attention head size d h . In TCSP{a%, b%}, the notation a% denotes the compress rate of hidden size, while b% denotes the compress rate of the attention head size plus the number of filters. Consequently, the overall compression rate of the model is a% + b% - a% * b%.

When employing TCSP with a compression rate of {25%, 0%} which retains 75% hidden size while preserving the original attention head size and number of filters, we observe that the compressed T5 base and BERT base models exhibit a maximum drop in accuracy of only 1.3% on the GLUE and SQuAD datasets. Building upon this, We further apply filter pruning and compress the head size, resulting in a model denoted as TCSP {25%, 25%}. Remarkably, this additional compression does not negatively impact the baseline performance, highlighting the compatibility of TCSP with the filter pruning and attention head size compression methods. Overall, we achieve a compression rate of 44% for both T5 base and BERT base models with only a 1.6% loss in accuracy. Like prior work [14, 15, 16], fine-tuning is necessary for ensuring performance. For more detailed information on model performance at various compression rates, please refer to Appendix F.5.

5.3 Comparison with Prior Methods

We conduct a comprehensive comparison of TCSP with prior methods, considering both their performance and compression time cost.

Following the setting adopted by Kwon et al. [8], we compare TCSP with prior pruning methods and low-rank factorization methods. The evaluation involves compressing models with compression rate constraints on four tasks within the GLUE benchmark: QQP, QNLI, SST-2, and MRPC. Notably, we conduct this evaluation without employing knowledge distillation. The comparison methods include Sajjad et al. [47], Kwon et al. [8], DRONE [14], FWSVD [15], and TFWSVD [16]. To ensure a fair comparison, we select BERT base as the baseline model, as it is commonly employed across all the comparison methods. To assess the impact of compression, we compare the amount of accuracy drop

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

experienced by each method since the absolute accuracy of the baseline BERT base may vary slightly across different papers. The compression rate is indicated in parentheses after each method's name, except for DRONE, as their paper does not explicitly report the compression rate.

Table 2 presents the performance of the compressed models and the difference from their corresponding baselines for all the compression methods under comparison. The performance of FWSVD and TFWSVD is marked with an asterisk since they use the F1 metric instead of accuracy specifically on the QQP dataset. However, this metric does not impact the comparison of the metric difference. Our method TCSP demonstrates comparable or superior results compared to the prior methods (A lower accuracy drop indicates better performance).

To assess the time cost, we evaluate the performance on the MNLI dataset, which is larger than other datasets in GLUE. Table 3 shows that TCSP requires only 0.16 hours for model compression and an additional 2 hours to model fine-tuning. This indicates that TCSP is significantly faster than most of the comparison methods, with the exception of the pruning method proposed in Kwon et al. [8]. However, the pruning method can be effectively combined with TCSP, allowing for compatibility and further optimization.

5.4 Ablation Study

To prevent the computation of SVD on excessively large matrices, We sample a subset of columns from the feature matrix. In other words, we randomly choose several tokens within each instance to compute the projection matrix. Consequently, we evaluate the effect of the sampled tokens on the compression performance. Table 4 presents the results of this analysis, revealing that satisfactory outcomes can be achieved when the number of tokens (1,000) is greater than the hidden size of the model (768).

Furthermore, we explore the influence of SVD. In Table 5, we replace the projection matrix generated by SVD with a randomly generated matrix and show the experimental results in Table 5. It is observed that using a random matrix for compression significantly diminishes the model's performance. Therefore, it is necessary to use SVD to compute the projection matrix.

6 Conclusion

This paper proposes TCSP, a method for compressing the transformer model by leveraging subspace 293 projection. TCSP employs SVD on the feature matrix of sampled data instances to derive a projection 294 matrix. By fusing this matrix into the transformer's weight matrix, we obtain a compressed model. 295 The model is subsequently fine-tuned using the entire dataset. TCSP is applied to both T5 base and 296 BERT base and evaluated on GLUE and SQuAD datasets. Remarkably, TCSP achieves a compression 297 ratio of 44% while incurring only a 1.6% decrease in accuracy. As TCSP primarily focuses on 298 compressing the hidden size, it can be easily combined with other methods that compress the number 299 of filters and the attention head size, making it highly compatible. 300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. 1

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 1

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021. 1

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021. 1

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems , 33:12449-12460, 2020. 1

Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pretraining for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing , 16(6):1505-1518, 2022. 1

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 29:3451-3460, 2021. 1

Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. arXiv preprint arXiv:2204.09656 , 2022. 1, 2, 4.1, 4.3, 5.3, 3, F.2, F.3, 6

Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6151-6162, Online, November 2020. Association for Computational Linguistics. 1, 2, 6

Zi Lin, Jeremiah Liu, Zi Yang, Nan Hua, and Dan Roth. Pruning redundant mappings in transformer models via spectral-normalized identity prior. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 719-730, Online, November 2020. Association for Computational Linguistics. 1, 2, 6

Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. EBERT: Efficient BERT inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 , pages 4814-4823, Online, August 2021. Association for Computational Linguistics. 1, 2, 3, 6

François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. Block pruning for faster transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 10619-10629, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. 1, 2, 3, 6

Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1513-1528, 2022. 1, 2, 4.3, 3

Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Drone: Data-aware lowrank compression for large nlp models. Advances in neural information processing systems , 34:29321-29334, 2021. 1, 2, 3.2, 4.1, 4.3, 5.2, 5.3, 6

Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations . 1, 2, 3.2, 5.2, 5.3, 6

Ting Hua, Yen-Chang Hsu, Felicity Wang, Qian Lou, Yilin Shen, and Hongxia Jin. Numerical 354 optimizations for weighted low-rank estimation on language models. In Proceedings of the 355 2022 Conference on Empirical Methods in Natural Language Processing , pages 1404-1416, 356 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. 357 1, 2, 3.2, 5.2, 5.3, 6 358

359

360

Hao Yu and Jianxin Wu. Compressing transformers: Features are low-rank, but weights are not! 2023. 1, 2

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

Canwen Xu and Julian McAuley. A survey on model compression and acceleration for pretrained language models, 2022. 2

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. 2

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations . 2

Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. I-bert: Integer-only bert quantization. In International conference on machine learning , pages 55065518. PMLR, 2021. 2

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR , abs/1910.01108, 2019. 2

Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems , 33:9782-9793, 2020. 2, 3, 6

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. 2

Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied transformers: Neural machine translation with shared encoder and decoder. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 5466-5473, 2019. 2

Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. Advances in neural information processing systems , 32, 2019. 2

Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix decomposition. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing , pages 884-889, 2020. 2

Marzieh Tahaei, Ella Charlaix, Vahid Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. KroneckerBERT: Significant compression of pre-trained language models through kronecker decomposition and knowledge distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2116-2127, Seattle, United States, July 2022. Association for Computational Linguistics. 2

Ali Edalati, Marzieh Tahaei, Ahmad Rashid, Vahid Nia, James Clark, and Mehdi Rezagholizadeh. Kronecker decomposition for gpt compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 219-226, 2022. 2

Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 , 2019. 2

Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems , 33:20378-20389, 2020. 2

Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal BERT surgeon: Scalable and accurate secondorder pruning for large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 4163-4181, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. 2

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural networks. CoRR , abs/1803.03635, 2018. 2

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR , abs/1910.10683, 2019. 3.1

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. 3.1

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. 3.1

Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some applications. IEEE Transactions on automatic control , 25(2):164-176, 1980. 4.1

Biao Zhang and Rico Sennrich. Root mean square layer normalization. CoRR , abs/1910.07467, 2019. 4.2

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. 4.2

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR , abs/1502.03167, 2015. 4.2

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. 5.1, F.1, F.2

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. 5.1, F.1, F.2

Seonhoon Kim, Inho Kang, and Nojun Kwak. Semantic sentence matching with denselyconnected recurrent and co-attentive information. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 6586-6593, 2019. 5.1

Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-perspective matching for natural language sentences. arXiv preprint arXiv:1702.03814 , 2017. 5.1

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing , pages 1631-1642, 2013. 5.1

William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005) , 2005. 5.1

Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language , 77:101429, 2023. 5.3, 6

Andrzej Ma´ ckiewicz and Waldemar Ratajczak. Principal components analysis (pca). Computers &Geosciences , 19(3):303-342, 1993. B

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. CoRR , abs/1910.03771, 2019. F.1

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. F.1, F.2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a subspace projection algorithm spanned by the features' principal components. \nThe projection's fusion with the different network layers is presented.\nA gradient-free pruning approach is further suggested based on the parameters and activation statistics.\nFinally, the proposed framework is experimented on BERT and T5 and achieves a compression ratio of 44% with at most 10 1.6% degradation.\n\nSTRENGTHS:\nThe paper is fairly written.\nThe apparent main novelties of the paper are the low-rank approximation of the features and a statically based pruning approach.\n\nWEAKNESSES:\nLow-Rank approximation:\\\nThe main weaknesses seem to arise from the comparison to prior/competing works.\nFor example, low-rank approximations of features have already been presented see for example \nhttps://cs.nju.edu.cn/wujx/paper/AAAI2023_AFM.pdf \\\nAlso, I am unsure why lines 37-38 are true: performing PCA (/Kosambi–Karhunen–Loève) is a pretty old technique for model acceleration, the subspace being defined by the principal components of the parameters or activations, this is a low-rank approximation.\\\nThus, the low-rank approximation contribution of the paper should be narrowed to the definition of the data matrix.\n\nPruning: \\\nit's unclear how these simple statistics perform compared to other pruning methods or heuristics.\n\nExperiments: \\\nthe subspace dimension as well as the compression ratio are not given which leaves the speed-up metric subjective.\nThe method performance on mid-size LLM is not very good compared to old methods. \\\nRandom projection as ablation is a very weak baseline.\n\nClarity:\\\nThe paper can be refined in terms of clarity (also typos (e.g., lines 235, 282))\n\nQUESTIONS:\n1) The novelty issues described in the Weaknesses section should be discussed/addressed.\n\n2) Comparison with recent techniques such as LORA and more compression metrics should be presented.\n\n3) An ablation with SVD(weights) should be the very least comparison.\n\n4) An ablation of the pruning should be provided\n\nLIMITATIONS:\nNo limitations discussed", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

comments: SUMMARY:
This paper presents TCSP, a model compression approach for transformers by reducing hidden size via low-rank factorization. In addition, TCSP is compatible with other compression methods such as model pruning and head size compression. Experiment results demonstrate the effectiveness of proposed method, achieving a high compression ratio while incurring rare performance drop.

STRENGTHS:
$\cdot$ This paper is well-structured and clear to understand.

$\cdot$ The algorithm is general enough, and is compatible with other compression strategies.

$\cdot$ Experiment results verify the effectiveness of the proposed method.

WEAKNESSES:
$\cdot$  The novelty of this paper is limited, the core idea resembles low-rank factorization with SVD, and the approach is more like a combination of SVD and model pruning.

$\cdot$ The author claims it is the first work to reduce the hidden size, but I doubt if the method can be successfully implemented in the industry since the lack of experimental results related to inference speed of compressed model.

QUESTIONS:
$\cdot$ What are the experiment settings in Tab. 2 and the ablation study? 

$\cdot$ There are some writing mistakes in this paper, e.g. “fien-tune” in the header of Tab. 3.

LIMITATIONS:
The author has addressed the limitations and social impacts in Appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a decomposition-based method, called Transformer Compression via Subspace Projection (TCSP), for compressing transformers. By decomposing the feature matrix extracted by some sample data, the model is projected onto a subspace to reduce the size of hidden dimensions. Experimental results on the datasets GLUE and SQuAD show that TCSP enables 44\\% parameters reduction with at most 1.6\\% accuracy loss and surpassing existing methods.\n\nSTRENGTHS:\nThis paper compresses the hidden dimension of the transformers, which is less explored.\n\nThe overall presentation of the paper is easy to understand.\n\nWEAKNESSES:\nTCSP is indeed just principal component analysis (PCA) or compressed sensing (CS), all working with the dominant subspace derived from SVD. Why another name?\n\nI have concerns about the following aspects:\n\n1. Motivation: From lines 51-59, this paper discusses the compression methods for transformers. Also, it mentions \"We do not delve into knowledge distillation and weight sharing, as they involve training models from scratch\". However, knowledge distillation (KD) includes both task-agnostic and task-specific schemes. For task-agnostic KD methods, they do not involve training models from scratch, see e.g., Wu T, Hou C, Zhao Z, et al. Weight-Inherited Distillation for Task-Agnostic BERT Compression. Meanwhile, it is a normal setting in KD to reduce the hidden size of the transformer model. Therefore, task-agnostic KD methods should be compared, too.\n\n2. Computation: As TCSP requires SVD decomposition for a large matrix, more discussion about the computing cost and scalability is needed, especially in Table 2.\n\n3. Robustness: Regarding the quality of subspace, how is the performance if we add noise or adversaries to the input data when generating the projection matrix? How do you ensure the sample data are representative? The timing overhead and complexity of the SVD to ensure a good projection subspace should be explicitly characterized and quantified.\n\nIndeed, there are recent decomposition-based compression algorithms applied to transformers which the authors may benchmark against, e.g., Ren, Y., Wang, B., Shang, L., Jiang, X., \\& Liu, Q. (2022). Exploring extreme parameter compression for pre-trained language models. arXiv preprint arXiv:2205.10036.\n\nQUESTIONS:\nSee my questions above.\n\nLIMITATIONS:\nThe models used are relatively small in size, e.g. T5-base, BERT-base. There are no experiments on the Large Language Models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach to compress the hidden size of a transformer model using subspace projection. On a high level, the paper aims at projecting the transformer model into a lower dimensional subspace using a projection matrix that is computed using a sample of the training data. This method is compared against other compression techniques using the T5 and BERT models on GLUE and SQuAD datasets and it is shown that the proposed method performs on par or better then the methods under comparison. The highlight of the experimental result is that the proposed transformer compression via subspace projection technique is able to compress models by as mush as 44% with only 1.6% degradation in performance.\n\nSTRENGTHS:\nThe paper addresses an important problem of transformer compression. In the age of ever-increasing model sizes, it is vital to develop methods that compress large models with minimal loss in performance, if any. This paper presents a simple yet effective approach to leverage linear subspace projection for compression. The paper is easy to follow, offers sufficient literature review, and presents convincing experimental results. The experimental results are particularly strong -- 44% compression with only 1.6% loss in performance.\n\nWEAKNESSES:\nSome notation is used before definition. Could include more recent literature in Related Work section -- see below.\n\nQUESTIONS:\n1. What is k in line 25? Please define it before using it.\n\n2. How are the training data sampled to estimate the projection matrix?\n\n3. How is the rank value (k) chosen? The explanation to this and 2. are delegated to the appendix but it would be beneficial if a couple of lines addressing these are presented in main text.\n\n4. Was the SVD solved in batch mode? Can this be done using incremental solvers instead? This may allow the use of a large matrix X.\n\n5. What is the k value in the experimental studies? Is it 2000? Please clarify. \n\n6. What is the average hidden layer size? What are its maximum and minimum values in the models considered?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}