Learning Parameter Sharing with Tensor Decompositions and Sparsity

Authors: Cem Üyük, Mike Lasby, Mohamed Yassin, Utku Evci, Yani Ioannou
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Model Compression, Parameter Sharing, Sparsity, Tensor Decomposition, Knowledge Distillation, Transfer Learning, Vision Transformers, Transformers

Abstract:
Large neural networks achieve remarkable performance, but their size hinders deployment on resource-constrained devices. While various compression techniques exist, parameter sharing remains relatively unexplored. This paper introduces Fine-grained Parameter Sharing (FiPS), a novel algorithm that leverages the relationship between parameter sharing, tensor decomposition, and sparsity to efficiently compress large vision transformer models. FiPS employs a shared base and sparse factors to represent shared neurons across multi-layer perception (MLP) modules. Shared parameterization is initialized via Singular Value Decomposition (SVD) and optimized by minimizing block-wise reconstruction error. Experiments demonstrate that FiPS compresses DeiT-B and Swin-L MLPs to 25-40%  of their original parameter count while maintaining accuracy within one percentage point of the original models.

Decision: reject
Reviewer scores: 5, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730307871199

thread_type:
official_review

<thread_object>
creator: reviewer_ntka

<content>
summary:
This paper introduces Sparsity-enabled Parameter Sharing (SParS), a compression algorithm for neural network parameters. This method typically focuses on compressing MLP layers in vision transformers by using shared base and sparse factors across multiple layers. Authors provide visualization and analysis on stages of the algorithm and ablate different ways of truncating MLP parameters. The authors demonstrate that their approach can compress DeiT-B and Swin-L MLPs to 25-40% of their original size while maintaining accuracy within 1% of the original models. Experiments also show that SParS (+FT) outperforms baseline methods like GFM.

strengths:
- **Novel approach**: In general, the method is new and it's a smart application of sparsity on modern neural networks
- **Clear presentation**: Authors present the motivation, methodology, experiments and analysis clearly. 
- **Empirical justification**: Authors use solid experiments to justify their choice of the design, i.e. on truncation strategy, number of grouped layers.
- **Strong results**: 25-40% parameter saving is in general a decent number in model compression related works.

weaknesses:
- **Limited comparison**: in the main results section, authors provide few comparison with other related parameter saving methods. The limited comparative analysis makes it difficult to fully assess the method's effectiveness relative to the current state-of-the-art approaches.
- **Insufficient theoretical analysis**: while the empirical results are promising, the paper lacks sufficient theoretical analysis to explain why SParS outperforms baseline methods. A more detailed discussion of the underlying mechanisms and intuitive explanations would enhance the reader's understanding and make the technical contributions more accessible. 
- **Narrow application scope**: this work mainly focuses on vision transformers, whereas parameter-efficient methods are particularly crucial for large language models. The authors should consider extending their analysis to language models or discuss potential challenges and modifications needed for such applications.

questions:
**Question**:
- Yu et al., 2023 explore weight-sharing in attention layers and find that it saves parameters in trade of small performance drop. I'm wondering if you can extend your methods to attention layer parameters. If not, what's the reason?
- Same as the weaknesses section. Why didn't you provide comparison with methods like MiniViT(Zhang et al., 2022)?


**Reference**:

Yu, Yaodong, Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. "White-box transformers via sparse rate reduction." Advances in Neural Information Processing Systems 36 (2023): 9422-9457.

Zhang, J., Peng, H., Wu, K., Liu, M., Xiao, B., Fu, J. and Yuan, L., 2022. Minivit: Compressing vision transformers with weight multiplexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12145-12154).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730402698072

thread_type:
official_review

<thread_object>
creator: reviewer_httn

<content>
summary:
This paper proposes Sparsity enabled Parameter Sharing (SParS) to compress large vision transformer models, which is an algorithm based on existing algorithms such as parameter sharing, tensor decomposition, and sparsity.

strengths:
None noted.

weaknesses:
There are several major concerns for this work.

1. Lack of novelty. All the components of SParS are existing and well-known algorithms, such as low=rank decomposition for FC layers, sparse training and pruning g (Hoefler et al., 2021,Zhu & Gupta, 2017, Evci et al., 2021). As a result, SParS is a straightforward combination of such existing algorithms which clearly lack novelty.

2. Unjustified formulation. The low-rank decomposition described in lines 219-224 is not justified. Why all the individual weights share the same $U$ and the same singular values $\Sigma$ with different $V$? What is the approximation error of such heuristic? Empirical or theoretical studies are expected to explain such low-rank decomposition on the concatenation of the weights.

3. Limited experiments. The SParS is only evaluated for DeiT-B (with 12 blocks) and Swin-L, while there is a large family of vision transformers including various versions of Swin, ViT, etc and it is not clear how SParS performs on a broad range of these vision transformers. Moreover, image classification results along cannot justify the effectiveness of the propose method, and more experiments in segmentation/detection on standard benchmarks are expected.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730425983500

thread_type:
official_review

<thread_object>
creator: reviewer_mlrv

<content>
summary:
The authors provided a simple parameter-sharing method to compress the current models, which is easy to understand. The method can be divided into two parts: 1) use a Shared Initialization to decompose the weights of MLP layers to unified U matrics and individual V matrics, realizing a low-rank reconstruction; 2) use a Local Error Minimization with a small calibrated dataset to fine-tune the new models, maintaining the original performance.

strengths:
1. The method is straightforward to understand.

2. The content is very rich, including analysis, figures, an algorithm, and experiments.

3. The analysis of Section 2 reveals the motivations of the method, which is interesting.

weaknesses:
However, some aspects obstruct understanding.

Writing:
* The layout is not easy to read and appears chaotic, especially in Section 3.

* The summary of the contributions is too casual and the high-level meanings of the modules are not well mentioned.

* Some Figures are hard to understand without reading the text in different parts, lacking sufficient explanation in the caption.

* Please check some typos, including repetitive abbreviations (FC, Lines 103 and 107) and main reference (Line 583).

Method:

*  The parameter sharing is only conducted in the MLP layers. Could the linear layers of QKV also use parameter sharing?

* Did the local error minimization constrain the performance of the new compressed models?

* Although simplicity is good, the method is too simple without sufficient verification.

Experiments:

* The method mentioned focuses on large neural networks that DeiT-B and Swin-L could be recognized as large when compared to modern large models, such as LLM.

* Although this method focuses on parameter sharing, comparing it with other optimization methods, such as Distillation, and NAS, is helpful for better understanding this method.

* Apart from the Param budget, what is the computational budget or inference time for Table 1 and Table 2?

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713244226

thread_type:
official_review

<thread_object>
creator: reviewer_gurg

<content>
summary:
The paper introduces SParS (Sparsity-enabled Parameter Sharing) to efficiently compress large vision transformer models through parameter sharing, tensor decomposition, and sparsity. SParS employs a shared base and sparse factors to represent shared neurons in multi-layer perceptrons (MLPs). Experimental results demonstrate that SParS can compress MLPs of DeiT-B and Swin-L to 25%-40% of their original parameter count while maintaining less than 1% accuracy loss.

strengths:
1. This paper is well organized and is easy to follow.

2. By decomposing the MLP params into sharing basis and sparse projection matrix, SparS shows the ability to achieve high compression rates with acceptable accuracy losses.

3. Experimental results show that SparS could outperform other vision transformer compression methods with similar compression rates.

weaknesses:
1. This paper seems to be incremental in technical contributions. The method of sharing basis matrix is similar to HydraLoRA [4] in principle.

2. The differences and innovations of the proposed SparS compared to existing data compression methods [3,4] are not addressed.

3. This paper only shows compression rate and performance on the DeiT-B and Swin-L. Evaluations on models like Swin-B or ViT-L[1,2] are  missing. This limits the potential generalization ability of this paper.

4. Experiments on the language model as in the baseline [5] are missing.

5. Yu and Wu (2023) mentioned that the activation rather than weight matrix is of low rank. I recommend to further clarify this problem considering that they focus on tensor decomposition on the weight matrix.

References

[1] Dosovitskiy A. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.

[2] Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 10012-10022.

[3] Yu H, Wu J. Compressing transformers: features are low-rank, but weights are not![C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(9): 11007-11015.

[4] Zhang J, Peng H, Wu K, et al. Minivit: Compressing vision transformers with weight multiplexing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12145-12154.

[5] Tian C, Shi Z, Guo Z, et al. HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning[J]. arXiv preprint arXiv:2404.19245, 2024.

questions:
Is the redundancy and interplay among different blocks mentioned in the article limited to MLP? Will the attention layer also have the same phenomenon?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732466678416

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta Rebuttal to All Reviewers

comment:
Dear reviewers,

We have submitted a revised version of the paper, which includes updated text, captions, and expanded set of experimental results:

- Name of SParS is updated to Fine-grained Parameter Sharing (FiPS).
- Structured sparsity performance results for DeiT-B are added (New paragraph "Structured Sparsity" in Section 5 and Table 3)
- Latency and memory profiling results for structured sparsity are added (Section 3 "Memory and Latency" Paragraph and Figure 3).

Additionally, we provide results for the SWIN Base model below to provide proof for the generalisation capability of FiPS below:
| MLP Param. Budget (%) | ImageNet Top-1 Accuracy (%)     |
|----------------------|------------------|
| 10%                  | 56.50%           |
| 25%                  | 81.67%           |
| 40%                  | 83.96%           |
| 50%                  | 84.27%           |
| Original SWIN–B (224)         | 85.2%   [2]   |

We would also like to address two points of divergence among reviewers:

	1.	The novelty of the method was seen differently by reviewers. Two did not comment on it but found the method interesting. One found it strong from a novelty point of view, whereas another thought it to be lacking. We have clarified our take on this in our specific response to each review and hope it may prompt re-evaluation.

	2.	Regarding writing, three reviewers praised the clarity and figures, while another found it weak. We have revised the paper and would like to encourage those with negative thoughts on writing to consider the updated version or help us further improve it!

Thank you for your time and consideration.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary. The paper introduces Fine-grained parameter sharing (FiPS) to efficiently compress large vision transformer models through parameter sharing, tensor decomposition, and sparsity. FiPS employs a shared base and sparse factors to represent shared neurons in multi-layer perceptrons (MLPs). Experimental results demonstrate that FiPS can compress MLPs of DeiT-B and Swin-L to 25%-40% of their original parameter count while maintaining less than 1% accuracy loss.

Strengths. 
The paper is well-written and easy to follow. 
Experimental results show that proposed method could outperform other vision transformer compression methods with similar compression rates.

Weaknesses. 
The technical contribution of the paper is limited as low-rank approximation is a standard technique that has been used for model compression and adaptation. 
Limited comparisons with existing compression methods.
Missing experiments for large models. 
The choice of fixing U for all MLP modules is not adequately justified. 

Missing. 
The paper is missing comparisons with existing compression methods.
Missing experiments for large models. 
The choice of fixing U for all MLP modules is not adequately justified. 

Reasons for decision.
Lack of comparisons with existing methods and missing experiments on large models are the most important reasons for my decision.