SHARP: Accelerating Language Model Inference by SHaring Adjacent layers with Recovery Parameters

Authors: Yiping Wang, Hanxian Huang, Yifang Chen, Jishen Zhao, Simon Shaolei Du, Yuandong Tian
Primary area: foundation or frontier models, including LLMs
Keywords: inference acceleration, weight sharing, language model, model compression

Abstract:
While Large language models (LLMs) have advanced natural language processing tasks, their growing computational and memory demands make deployment on resource-constrained devices like mobile phones increasingly challenging. In this paper, we propose SHARP (SHaring Adjacent Layers with Recovery Parameters), a novel approach to accelerate LLM inference by sharing parameters across adjacent layers, thus reducing memory load overhead, while introducing low-rank recovery parameters to maintain performance.
Inspired by observations that consecutive layers have similar outputs, SHARP employs a two-stage recovery process: Single Layer Warmup (SLW), and Supervised Fine-Tuning (SFT).
The SLW stage aligns the outputs of the shared layers using  $\mathcal{L}_2$ loss, providing a good initialization for the following SFT stage to further restore the model performance. Extensive experiments demonstrate that SHARP can recover the model's perplexity on various in-distribution tasks using no more than 50k fine-tuning data while reducing the number of stored MLP parameters by 38\% to 65\%.
We also conduct several ablation studies of SHARP and show that replacing layers towards the later parts of the model yields better performance retention, and that different recovery parameterizations perform similarly when parameter counts are matched.
Furthermore, SHARP saves 42.8\% in model storage and reduces the total inference time by 42.2\% compared to the original Llama2-7b model on mobile devices.
Our results highlight SHARP as an efficient solution for reducing inference costs in deploying LLMs without the need for pretraining-scale resources.

Decision: reject
Reviewer scores: 6, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730165669908

thread_type:
official_review

<thread_object>
creator: reviewer_ihg5

<content>
summary:
This paper explores the approach of utilizing an adjacent layer-sharing strategy to compress LLMs by sharing parameters between neighboring layers. The primary motivation behind this approach is based on the observation that the output features of adjacent layers are significantly similar, which suggests the potential for parameter sharing to achieve more efficient communication and inference. The specific method used involves sharing parameters between adjacent layers and introducing a layer parameter tuning mechanism using a LoRA module. This two-part process initially focuses on minimizing the L2 loss of output between adjacent layers followed by a fine-tuning phase. The results are compared against direct parameter sharing without fine-tuning to highlight the benefits of the introduced method.

strengths:
1. **Innovative Approach:** The methodology combines parameter sharing with fine-tuning using a LoRA module, presenting a new avenue in model compression that could potentially conserve computational resources while maintaining model accuracy.    
2. **Efficiency Gains:** The speed-up achieved through this method is notable, providing a practical solution for scenarios where faster inference is crucial, such as in mobile or edge computing environments.

weaknesses:
1. **Performance in Downstream Tasks:** Despite the improvement in speed, the performance loss in downstream tasks is still significant, raising concerns regarding the practical applicability of the SHARP method in real-world scenarios where maintaining high levels of accuracy is paramount. 

2. **Ambiguity in Communication Definition:** The article does not clearly define what is meant by "communication" in the context of parameter sharing. It is essential to clarify whether it refers to internal model communication, information flow between layers, or external communications between distributed components. 

3. **Lack of Comparative Analysis:** The authors failed to include comparisons with alternate model compression techniques, such as LLM pruning, despite they also attempt to drop the LLM parameters while retaining performance. Such a comparison could have provided a more comprehensive understanding of how this new method stacks up against traditional approaches in terms of parameter reduction, tuning overhead, and performance impact. 

Overall, while the adjacent layer-sharing strategy introduces a promising direction for LLM compression by leveraging the similarity in output features of neighboring layers, the practical limitations noted in performance impact on downstream tasks and lack of broader comparisons with established techniques present significant areas for improvement. Addressing these issues and providing a clearer exposition of the methodology and its implications could make this approach more robust and widely applicable in the field.

questions:
Please see the weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595174567

thread_type:
official_review

<thread_object>
creator: reviewer_tuxm

<content>
summary:
This paper propose a layer sharing approach with additional low-cost finetuning to regaun the performance. After sharing the weights in the adjacent layers in the pretrained model, a two-stage lora-based finetuning is proposed, with one stage minimizing single layer output difference and the second stage recovering full model performance.

strengths:
This paper provides a solid study on ways to regain model performance after layer merging. Experiments are conducted on both the ratio/location of layer merging and the ways to further finetune the model and regain performance. The paper is overall well-written and easy to follow. From the novelty perspective, layer sharing appears to be a new way of compressing pretrained large models, and the paper successfully show the runtime speed up on real devices, proving the proposed method as a promising research direction.

weaknesses:
The main weakness of this work is two-fold, one is on the performance loss of the proposed method, and second is on the generalizability of the proposed finetuning method.
1. Although the proposed method show promising recovery performance on the PPL of pretraining dataset (Tab 2 and 3), we do observe significant performance drop on downstream tasks even after finetuning. This leaves me doubt if there's potential overfitting in the finetuning process, so that the performance on seen datasets are recovered but that on unseen tasks are not. A cross validation of finetuning datasets would be helpful on analyzing this issue. Additional techniques may need to be proposed to tackle overfitting.
2. Although this paper sets its background as enabling layer sharing across adjacent layers, the majority of the method is focused on the two-stage finetuning to regain performance. Finetuning is typically applied in all model compression settings, not limited to layer sharing. The porposed 2-stage finetuning may also be useful on other model compression techniques. More clarifications on the contribution is needed, as whether the 2-stage finetuning is specifically proposed for the layer sharing task, or is it borrowed from previous techniques.

questions:
1. How would the model behave if the finetuning data and the evaluation task does not fully match? For example, will fineutning on GPT4-Alpaca only helps regain the performance on Arxiv-math?
2. Is the proposed 2-stage finetuning scheme limited to layer sharing? Or could it be used for other model compression methods?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639347895

thread_type:
official_review

<thread_object>
creator: reviewer_wpoq

<content>
summary:
This paper proposes a method to reduce model memory usage and increase inference speed by sharing parameters between adjacent layers and introducing low-rank recovery parameters to maintain performance. First, Single Layer Warmup (SLW) replaces a pre-trained layer by adding a LoRA adapter to the previous layer's weights. To warm up the LoRA adapter initially, it is trained to mimic the output of the original layer. Second, Supervised Fine-tuning (SFT) fine-tunes all LoRA adapters across layers to preserve model performance. This approach significantly improves performance compared to Direct Sharing and achieves similar performance to the original model while reducing model weights by approximately 62%.

strengths:
The advantage of this paper is that it can reduce model parameters by more than 50% while maintaining similar performance to the baseline algorithm in some tasks. Especially, while full fine-tuning requires a lot of data, the initial warm-up using the L2 norm of the output of the original layer can be done in parallel, which helps improve performance. If the proposed algorithm still performs well after applying quantization to compress the model, it could further reduce the model weights in an orthogonal way to quantization.

weaknesses:
The main drawback of this algorithm is that it is not beneficial for all tasks. For instance, in Table 6, while SHARP improves accuracy over Direct Sharing and even surpasses the original on the CommonsenseQA task, its accuracy significantly degrades compared to the original across most tasks, raising doubts about its overall utility. Of course, reducing the model size by half could lead to an accuracy drop, but I’m still curious whether the compressed model performs better than an originally smaller model. For example, while it might be possible to compress the LLaMA2-13B model down to 7B, I’m concerned that its accuracy could be lower than that of the original LLaMA2-7B model, which does not require a complex compression process.

The training process also has a complexity issue, particularly with applying SLW using the L2 norm. Since the paper highlights potential problems with using SFT alone, modifying the model structure effectively may be challenging. This concern is especially relevant for models more complex than Llama-2-7B, which was used in the study. If SLW fails, the model may not be optimized effectively. Unlike MobileLLM, which extends an existing model, this paper’s approach removes certain layers from a pre-trained model, introducing the risk that training may become unstable if SLW fails.

questions:
1. Is the accuracy of a large model reduced by SHARP higher than that of a smaller model with an equivalent number of parameters? For instance, if the LLaMA2-13B model is reduced by 7B parameters using SHARP, does this new model outperform the existing LLaMA2-7B in accuracy? If so, it would effectively demonstrate the utility of SHARP (and I'm willing to increase the score if this question is answered).

2. In Table 2, it is described that the comparison is between the model after only SLW, only SFT, and the SHARP algorithm, but there seem to be no results for when only Stage 2 was applied.

3. I am curious whether this approach yields similar results when applied to larger models and when applied to recently released models like Llama-3.2.

4. If the model is BF16, the model size would be reduced to 1/4 with 4-bit quantization. I am curious about the results of applying the idea proposed in the paper after quantization.

5. Wouldn't attaching LoRA adapters to the entire model lead to even better performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731134493785

thread_type:
official_review

<thread_object>
creator: reviewer_kgwy

<content>
summary:
This paper proposes SHARP (sharing adjacent layers with recovery parameters) to accelerate LLM inference by sharing parameters across adjacent layers to reducing memory load overhead. The model performance is maintained through low-rank recovery parameters. Specifically, it employs a two-stage recovery process: SLW and SFT. Experiments results demonstrate the effectiveness in recover perplexity with a small amount of fine-tuning date while reducing the number of MLP parameters significantly. Also the inference time reduction was achieved compared to the original model on mobile devices.

strengths:
+ This work follows a relative new methodology for efficient inference, i.e., adjacent layer-sharing strategy. While prior work focuses on training from scratch, this work focuses on deploying pretrained model in a resource-saving post-training way.

+ The proposed method is motivated by the robustness of LLM when replacing adjacent MLP layers and makes new observations in support of the layer-sharing strategy.

+ The two stages by SLW and SFT provide a good heuristics in layer-sharing. The work introduces low-rank weights to predict subsequent layers.

weaknesses:
- The experiments are not comprehensive enough to compare with state-of-the-art efficient inference methods.

- In the latency analysis, models are simplified with 4-bit quantization to fit in iPhone. However, that is not a valid implementation since direct quantization may degrade model performance. On the other hand, it shows that weight sharing is not sufficient to support efficient inference on edge devices.

questions:
Although the work improves the model performance and run time performance significantly, it only compares with the direct sharing baseline. More comprehensive comparison with state-of-the-art efficient inference methods are needed to justify the advantage of the layer sharing strategy over other categories of strategy, like those mentioned in the paper, e.g., pruning or MobileLLM. The authors could comment on the advantages of weight sharing in terms of model performance, training cost, memory resources, etc. over other state-of-the-art methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732366803290

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Restate the position of our paper, and reply to all reviewers for major concerns

comment:
We sincerely appreciate all reviewers for their insightful and constructive feedback to make our paper better. We will revise our paper according to these comments. 
First, we reclaim the position of our SHARP method before answering the main concerns to clarify our motivation. 
And then we answer some concerns mentioned by most of the reviewers. Please refer to Appendix D in updated paper for baseline details.

> **OUR POSITION**: SHARP is a *structural-pruning*-like method to directly accelerate LLM inference by sharing parameters across adjacent layers and reducing memory load overhead.

There are diverse approaches in model compression. 
Structural pruning is one of them, whose main goal is in general not to achieve the most efficient parameter reduction, but to accelerate model inference without further requirements of hardware design. 
Other methods, like quantization and sparsification, can efficiently reduce the number of parameters, but they cannot always achieve the same ratio of inference speedup as the parameter savings when deployed in general hardware.
However, these methods can achieve direct optimization and is especially helpful for edge devices (like MobileLLM). 

Therefore, due to the similarity, we compare our methods with the top structure pruning baselines (All-A2). Although they can not perform as well as quantization in recovering performance, they are still important directions of model compression, especially for mobile devices.
We will add these clarifications of our position in the revised paper.

>**All-A1**: Using 4-bit quantization may degrade model performance

In the Table 9 in the updated paper, **we show that 4-bit quantization will not degrade the model's performance significantly**. Compared to using SHARP alone, using SHARP and 4-bit quantization together just brings about 1% performance drop on average, which is acceptable. This also further supports that our method is orthogonal and compatible with quantization.

>**All-A2**: Can not compare with state-of-the-art efficient inference methods

Following the claim in ‘OUR POSITION’, we mainly compare SHARP to other top structure pruning baselines. We try two state-of-the-art structure pruning methods: LayerSharing and LLM-Pruner and one variation LayerSharing-Adapted, which utilize the same replacement strategy as SHARP, equivalent to SHARP except removing the target layers rather than predict them. More details are shown in Appendix D.1 in the updated paper. Results are shown in Table A2:

**Table A2**: Additional structure pruning baseline on in-distribution tasks. 
||Arxiv-math|DialogSum|GPT4-Alpaca|Dolly|OpenOrca|
|---------------|---------------|-------|------------|----|----|
Direct Sharing| 2171.3 | 801.7 | 20662.1 | 7221.7 | 12108.5 |
LayerPruning  (w/o finetuning) | 87.6 | 19.2 | 86.1 | 283.1| 102.2 | 
LayerPruning | 4.1 | 4.4 | 4.0 | 6.8| 5.6 
LayerPruning-Adapted (w/o finetuning) | 28.3 | 19.1 | 583.0 | 86.3 | 336.7 |
*LayerPruning-Adapted*  | *3.8* | *4.3* | *3.2* | *5.8* | *5.3* |
LLM-Pruner (w/o finetuning) | 14.7 | 6.6 | 6.8 | 13.3 | 16.1 
LLM-Pruner | 11.2 | 4.5 | 4.1 | 7.6 | 7.7
SHARP (w/o finetuning) | 4.8 | 5.3 | 4.2 | 7.2 | 8.4 |
**SHARP** | **3.2** | **3.8** | **2.8** | **4.7** | **4.3** |

**[Results]**: From Table A2, we can observe that 
1. [SHARP vs Others]: **SHARP performs consistently better than structural pruning methods.** This shows the advantage of SHARP over structural pruning methods.
2. [SHARP vs LayerPruning-Adapted]: when number of parameters are the same, **reusing the previous layers will be more efficient than pruning**. 
3. [LayerPruning-Adapted vs LayerPruning]: T_next , replaces the layers at intervals, is better than LayerPruning, which removes the consecutive layers.
4. [LLM-Pruner vs Others]: removing parameters may result in a good initial performance but be more difficult in recovering.

>**All-A3**: Still has a significant performance drop on downstream tasks

First, we admit that there are still obvious performance drops on the downstream tasks.
However, as mentioned in ‘OUR POSITION’ and All-A2, SHARP is aiming for similar improvement as structural pruning, in which field all still have the same performance degradation problem (Table A2) when the data and computation resources are limited.
Nevertheless, when given the same condition, SHARP achieves better recovery performance over structural pruning baselines consistently (Table A2), showing the advantage of our method.

On the other hand, we also do an ablation study in Figure 4 in main paper to explain the reason for the downstream performance gap in recovery, like the reasoning capability of LLM may depend on more layers and be harder to recover.

Therefore, although our method hasn’t solved the universal downstream degradation problem as in structural pruning, it has improved the current best baselines in recovering model performance and still provides useful insights for future work to solve this general problem.
</content>
</thread_object>

</thread>


Metareview:
metareview:
After the rebuttal the reviewers still have the following concerns: The paper struggles to position its proposed method of layer sharing effectively, failing to provide a fair comparison with other post-training compression techniques, such as structural pruning, which is a more established method for reducing model size and accelerating inference. The author does not convincingly demonstrate that layer sharing outperforms pruning in terms of efficiency, nor do they show that layer-sharing models are comparable to scratch-trained models. Additionally, the claim that layer sharing can be a general acceleration method is undermined by its limited advantages over pruning, as it only reduces weight storage while maintaining full computation, with potential overhead in memory and computation. The paper lacks key evaluations, such as end-to-end wall-clock time comparisons, and does not provide sufficient evidence to justify the significance of layer sharing in the broader context of efficient deep learning.

Since 3 reviewers voted to reject it and the only positive review is not strongly arguing for acceptance, the paper will be rejected.