DPPA: Pruning Method for Large Language Model to Model Merging
Authors: Yaochen Zhu, Rui Xia, Jiajun Zhang


Reviewer paper summaries: This paper proposes a model merging method for different domains based on a two-stage pruning approach. Specifically, it consists of dynamic pruning, which dynamically adjusts the pruning rates of different layers and modules, and dynamic partition amplification, which dynamically amplifies parameters based on their varying levels of importance. The proposed method was evaluated on LLaMA-2 across three domains: mathematics, finance, and law, and compared against state-of-the-art baselines. The results demonstrate the effectiveness of the proposed method.
Summary of strengths: - The motivation of this paper is very straightforward and intuitive.
- The experiments demonstrate that the method has good performance.
Summary of weaknesses: - The paper lacks some baselines from the dynamic pruning domain, such as Movement Pruning, and Taylor Pruning.
- In Table 1, the results for "Ours" are incomplete, missing the results at 10% sparsity.

[1] Movement Pruning: Adaptive Sparsity by Fine-Tuning.

[2] Importance Estimation for Neural Network Pruning.
Comments, suggestions and typos: None
Overall assessment scores: 2.5