CQIL: Inference Latency Optimization with Concurrent Computation of Quasi-Independent Layers (2024)

Abstract: The fast-growing large scale language models are delivering unprecedented performance on almost all natural language processing tasks. However, the effectiveness of large language models are reliant on an exponentially increasing number of parameters. The overwhelming computation complexity incurs a high inference latency that negatively affects user experience. Existing methods to improve inference efficiency, such as tensor parallelism and quantization, target to reduce per-layer computing latency, yet overlook the cumulative latency due to the number of layers. Recent works on reducing the cumulative latency through layer removing, however, lead to significant performance drop. Motivated by the similarity of inputs among adjacent layers, we propose to identify quasi-independent layers, which can be concurrently computed to significantly decrease inference latency. We also introduce a bypassing technique to mitigate the effect of information loss. Empirical experiments of the proposed approach on the LLaMA models confirm that Concurrent Computation of QuasiIndependent Layers (CQIL) can reduce latency by up to 48.3% on LLaMA-33B, while maintaining a close level of performance. 1

heading: 1 Introduction
text: Large Language Models (LLMs) are offering unprecedented power to deliver remarkable performance across diverse tasks of natural language processing. The exceptional performance, however, comes at the cost of increasing model size and, consequently, higher inference latency. For example, the per-token inference time for GPT-4 is approximately three times longer than that of GPT3.5, according to measurements from the OpenAI API 2. Such a high latency has a direct impact on
1Code is available at https://github.com/Photooon/CQIL 2Note that latency can vary depending on time and location; the reported times is based on observations at the authors’ location.
user experience, highlighting the urgent need to mitigate inference delays of LLMs.
LLMs typically consist of a large number of sequentially connected layers with identical structures. For better illustration, we analyze the inference latency of LLMs along two dimensions: per-layer latency and the cumulative latency due to all layers. Existing low-latency inference methods, like tensor parallelism, quantization (Dettmers et al., 2022; Xiao et al., 2023), and unstructured pruning (Han et al., 2016; Frantar and Alistarh, 2023; Sun et al., 2023) and low-rank factorization (Lan et al., 2020; Lv et al., 2023), primarily focus on minimizing per-layer latency. Meanwhile, there are works, such as structured pruning (Ma et al., 2023; Xia et al., 2023) and dynamic early existing (Xin et al., 2020), proposed to address the cumulative latency through removing and/or dynamically
7293
omitting layers. These approaches, unfortunately, often result in considerable performance degradation and potential loss of learned knowledge (Geva et al., 2021). Therefore, it’s critical to develop effective methods that can reduce the latency related to the total number of layers while preserving the model performance.
This work is inspired by the observation that adjacent layers in LLMs share significantly similar inputs. Such similarity suggests the possibility of substituting a layer’s input with that of a certain preceding layer without significantly altering its output. Such layers sharing input are designated as quasi-independent layers in this work. As a result, the computing dependency between adjacent layers can be eliminated and thus unleash the potential of parallel computation. We introduce a framework of Concurrent Computation of Quasi-Independent Layers (CQIL), to reduce LLM inference latency by parallelizing the computation across layers with similar inputs. Additionally, we develop a bypassing technique to transmit the output of attention modules among input-aligned layers, with the purpose of minimizing the information loss. Extensive experiments demonstrate reductions of inference latency by up to 48.3% on the LLaMA-33B model, with a minor impact on performance. We also discuss the implication of CQIL in the context of ensembles, which may offer deeper insights into the fundamental characteristic of LLMs.
The major contributions of this work are as follows. First, we propose CQIL, a novel approach to enhance the inference efficiency of LLMs through concurrent computation of quasi-independent layers, effectively addressing the challenge posed by the increasing number of layers. Second, our method enables the adaptation of pre-trained LLMs into ensemble-like models with minimal performance loss, which may provide deeper insights for layers’ characters in LLMs. Third, We effectively reduce the inference latency of LLaMA models, with minimal impact on the model performance.

heading: 2 Related Work
text: Efficient Inference Approaches Model compression techniques, such as pruning (Frantar and Alistarh, 2023; Sun et al., 2023; Ma et al., 2023; Xia et al., 2023), quantization (Dettmers et al., 2022; Xiao et al., 2023), low-rank factorization (Lan et al., 2020; Lv et al., 2023), and knowledge distillation (Hinton et al., 2015; Sanh et al., 2019;
Jiao et al., 2020; Sun et al., 2020), reduce inference latency by trimming parameters in the model. Methods like dynamic early exit (Xin et al., 2020) and speculative decoding (Leviathan et al., 2023; Chen et al., 2023) leverage intermediate layer output and output from smaller models to predict the final outcome ahead of time. Flash Attention (Dao et al., 2022) enhances the efficiency of attention computation by carefully orchestrating computation and memory usage. These methods are orthogonal to our approach, allowing for potential integration for a higher level of improvement. Notably, we test the combination of the pruning approach with our method, detailed in Section 5.6.
Parallelism Beyond removing the model’s parameters, latency reduction and throughput enhancement can also be achieved through parallel computation strategies. Typically, parallelism in LLM computation includes data parallelism (Rajbhandari et al., 2020), pipeline parallelism (Harlap et al., 2018; Huang et al., 2019), and tensor parallelism. Among these, only tensor parallelism addresses the inference latency by distributing layers computation across multiple GPUs. Similar to tensor parallelism, our method employs additional GPUs to decrease latency. Therefore, we conducted comparison experiments in Section 5.5. Results show that, compared with tensor parallelism, our method consistently reduces latency across various batch size, making it more amenable to the scenario of online inference. In addition, our approach is orthogonal to tensor parallelism and we could achieve further acceleration by integrating tensor parallelism with CQIL.
Transformers with Parallel Architecture Previous works have explored the acceleration of pretraining phases through the parallelization of transformer architectures. GPT-J (Wang and Komatsuzaki, 2021) and PaLM (Chowdhery et al., 2022) achieve efficiency improvements by parallelizing the attention and feedforward modules within transformer layers, allowing concurrent computation for the beginning projection in these two modules. Other research efforts (Gao et al., 2020; Wang et al., 2023) have enhanced model performance by expanding model width through parallel layers design, and still focus on the pre-training stage. Our work applies concurrent computation of layers to a pretrained LLMs, aiming at reducing latency while maintaining model performance. Our approach thus differs from existing methods that focus on
pre-training transformers with parallel architecture in terms of both motivation and methodology.

heading: 3 Preliminary
text: In this section, we present a systematic investigation that motivates the design of the proposed approach introduced in Section 4. Our discussion proceeds under the assumption of pre-layer normalization, which is adopted by most LLMs (Brown et al., 2020; Zhang et al., 2022b; Touvron et al., 2023). In following experiments, we utilize the LLaMA-1 (Touvron et al., 2023) and Sheared-LLaMA-2.7B (Xia et al., 2023) models, with the input samples randomly selected from the RedPajama (Computer, 2023) dataset. We first investigate the similarity of inputs across layers in LLMs, which suggests the quasi-independence of deeper layers and the potential for parallel computation. Second, we explore the effect of substituting a layer’s input with that of its preceding layers, which aids in identifying potential layers for parallel processing with minor performance degradation.

heading: 3.1 Similarity of Layer Input
text: For a LLM with L layers, we define the input to the layer l as xl ∈ RB,T,H , where B, T , and H denote batch size, token count, and hidden dimension size, respectively. The output from the layer l is denoted as xl+1 = xl+Fl(xl). We employ cosine similarity to quantify the similarity of inputs between layers.
Figure 1 reveals the similarity of inputs across layers in LLaMA models with different parameter sizes. The following two primary observations emerge from our experiment results. First, input similarity intensifies with increasing depth within the model, attributed to the cumulative effect of pre-layer normalization, which makes the input difference Fl−1(xl−1) between adjacent layers l − 1
and l increasingly negligible compared with the cumulative value xl = x1+F1(x1)+...+Fl−1(xl−1). Second, larger models exhibit more obvious similarity of inputs, suggesting that layers in such models are quasi-independent and parallel computation may be more readily available in such models.
While prior research (Zhang et al., 2023; Din et al., 2023) has identified output similarity across layers to facilitate inference efficiency through pruning, our approach is based on a different motivation with a more systematic perspective. Experimental results of previous works indicate that layer pruning often result in substantial performance declines. For instance, although the Sheared-LLaMA1.3B model has a similar hidden dimension size to that of the Sheared-LLaMA-2.7B, it contains eight fewer layers and consequently results in significantly performance drop on downstream tasks. Contrary to removing layers, our approach seeks to parallelize layer computation to reduce inference latency while preserving the model performance. In addition, our method is orthogonal to pruning techniques and the compatibility is further demonstrated in Section 5.6.
We then empirically assess the effect of substituting each layer’s input with that of preceding layers, facilitating the identification of layers suitable for parallel computation.

heading: 3.2 Sensitivity of Input Substitution
text: As shown in Figure 1, adjacent layers have similar inputs. It is thus appealing to identify possible layers for parallel computation, i.e., quasi-independent layers. In this section, we substitute the input of a layer with that of preceding layers to evaluate the sensitivity of each layer and thus justify the feasibility of quasi-independent layers. Specifically, the output of the model can be written as
(a) Pipeline (b) CQIL (c) CQIL with Bypassing
xL+1 = x1 +F1(x1) + ...Fl(xl) + ...FL(xL). For layer l, we replace the input of Fl with xl−k, which is the input of layer l − k, and keep other terms unchanged. This process is repeated for every layer individually, assessing their adaptability to input changes through perplexity measurements on validation set. When perplexities of the original and the input-substituted models are similar, it is feasible to concurrently compute these two layers.
As illustrated in Figure 2, our substitution trials with the LLaMA-7B model indicate that the input for a majority of the middle layers can be effectively replaced by those of their immediate predecessors while maintaining a similar performance. It is observed that both the bottom and top layers are sensitive to input changes. The bottom layers display less input similarity compared to subsequent layers. In addition, substitution in bottom layers leads to a propagation of errors from substituted inputs through subsequent layers, thereby amplifying their susceptibility to input changes. The top layers, due to their direct connection with the output distribution, also experience a notable impact on performance if their input are substituted.
The empirical findings suggest that most middle layers offer a tolerance for input substitution. In other words, it is feasible to relax the comput-
ing dependence by using the same input to quasiindependent layers, revealing the potential of parallel computation for latency reduction.

heading: 4 Methodology

heading: 4.1 Problem Statement
text: Given a LLM with pre-layer normalization, we can formulate the computation of layer l as Eq. 1 and Eq. 2. ATTN and FFN represent the attention and feedforward modules in each layer. n denotes the number of heads, and dk specifies the dimension size of each head.
xl+1 = xl + ATTNl(xl) + FFNl(xl + ATTNl(xl)) (1)
ATTN(x) = MHA(Norm(x))
MHA(x) = Concat(head1, ..., headn)WO
headi = Attention(xW Q i , xW K i , xW V i )
Attention(Q,K, V ) = softmax( QKT√
dk )V
FFN(x) = W 2f(W 1(Norm(x)) + b1) + b2 (2)
Algorithm 1: Concurrent Computation of Quasi-Independent Layers Input : input x1, model with L layers,
group size p, start layer s, end layer e and bypassing distance d.
Output :output xL 1 for l = 1 → s, step=1 do 2 al = xl + ATTNl(xl) 3 xl+1 = al + FFNl(al) 4 end 5 for l = s+ 1 → e, step=p do 6 reqs1, reqs2, attns = [], [], [] 7 for i = 0 → p− 1, step=1 do 8 reqs1.add(non_block_exec( 9 al+i,attn = ATTNl+i(xl), gpu=i))
10 end 11 for i = 0 → p− 1, step=1 do 12 reqs1[i].wait() 13 attns.add(al+i,attn) 14 bp = sum(attns[−min(d, i) + 1 :]) 15 reqs2.add(non_block_exec( 16 al+i,ffn = FFNl+i(xl + bp), gpu=i)) 17 end 18 xl+p = xl 19 for i = 0 → p− 1, step=1 do 20 reqs2[i].wait() 21 xl+p = xl+p + al+i,attn + al+i,ffn 22 end 23 end 24 for l = e+ 1 → L, step=1 do 25 al = xl + ATTNl(xl) 26 xl+1 = al + FFNl(al) 27 end
Our objective is to replace the input xl in Eq. 1 with xi, i < l. This substitution implicitly enables the parallel computation of layers l and i and thus reduces the inference latency.

heading: 4.2 Layer Partition and Parallelism
text: Considering a LLM with L layers, we partition layers into K groups G1, G2, ..., GK . Within each group, layers share the same input, allowing for concurrent computation of them. The computation for group Gk, where 1 ≤ k ≤ K, can be formalized as shown in Eq. 3. ATTN and FFN operations within each summation in this equation are executed in parallel. Specifically, when the size of each group is 1, Eq. 3 reverts to the sequential
computation in Eq. 1.
xGk+1 = xGk+ ∑
l∈Gk ATTNl(xGk)
+ ∑
l∈Gk FFNl(xGk + ATTNl(xGk))
(3)
The number of potential partition schemes can be considerable. Therefore, we employ a straightforward approach to divide layers into groups. Specifically, with a maximum group size of p, start layer s, and end layer e, we sequentially arrange layers from s to e into groups of size p, while organizing remaining layers into groups of size 1. For instance, the LLaMA-7B model with 32 layers, can be segmented into groups as {1} → {2} → ... → {8} → {9, 10} → {11, 12} → ... → {29, 30} → {31} → {32}, with p = 2, s = 9, e = 30. Such partitioning strategy helps prevent low GPU utilization and significant performance degradation from parallelizing the bottom and top layers. Algorithm 1 further expound the concurrent computation process. Hyperparameter p is determined according to the number of GPUs. The choice of hyperparameters s, e depends on balancing performance against acceleration, discussed further in Section 5.4.

heading: 4.3 Bypassing
text: The transformer layer includes an attention module and a feedforward module. Notably, parallel computation of p layers in group Gk allows concurrent accesses to attention module outputs ATTNl(xGk), l ∈ Gk. We could transfer ATTNl(xGk) to the feedforward module of layer j, j > l and j ∈ Gk, thereby minimizing information loss. We designate such a approach as bypassing by following the terminology of computer architecture. We define the bypassing distance d as the maximum of j−l. With bypassing, the formulation for Gk+1 becomes Eq. 4.
xGk+1 = xGk + ∑
l∈Gk ATTNl(xGk)
+ ∑
l∈Gk FFNl(xGk + ATTNl(xGk)
+ ∑
l′∈Gk,1≤l−l′≤d ATTNl′(xGk))
(4)
Moreover, the number of transmissions for bypassing is d(2p− d− 1)/2. To avoid communication bottlenecks, we set d = 1 across all experiments unless otherwise specified. The effect of bypassing distances are detailed in Section 5.3.

heading: 4.4 Fine-tuning
text: Without additional training, models with concurrent computation of quasi-independent layers maintain a close level of performance to the original. To achieve better performance, we fine-tune the model using LoRA(Hu et al., 2022) on the pre-training dataset. Fine-tuning with just 0.5B tokens is sufficient to almost reach the original performance, suggesting that middle layers in original LLMs may inherently function as ensembles. We discuss such implication further in Section 6. We believe that continual pre-training with more tokens will yield further performance enhancements. However, due to computational resource constraints, we leave the continual pre-training for the model with CQIL as future work.

heading: 5 Experiment

heading: 5.1 Experimental Setup
text: Model configurations and Dataset We apply CQIL on Sheared-LLaMA-1.3B, Sheared-LLaMA2.7B (Xia et al., 2023), LLaMA-7B, LLaMA-13B and LLaMA-33B (Touvron et al., 2023). Con-
current computation of layers are implemented with Pytorch distributed communication package (Paszke et al., 2019) and Huggingface Transformers library (Wolf et al., 2019). We use RedPajama (Computer, 2023), the replicated dataset of LLaMA1 models, as the training dataset, and adopt the same data mixture used for training LLaMA1. We construct a held-out validation dataset with 500 sequences of 2,048 tokens.
Baselines Our method is orthogonal to most existing works on efficient inference, making it hard to be directly compared with them. Given that both tensor parallelism and our strategy employ additional GPUs to reduce latency, we evaluate the relative efficiency of our method against tensor parallelism. The effectiveness of tensor parallelism implementations can vary significantly. Based on empirical assessments, we select the DeepSpeed inference library (Rajbhandari et al., 2020) as the benchmark for tensor parallelism. DeepSpeed inference library partitions attention and feedforward projection matrices and computes them distributively, which has proven to reduce latency in our testing environment. Furthermore, we explore combination of our method and the pruning technique, showing further potential for latency reduction.
Training We fine-tune models using LoRA on Nvidia A100 GPUs (80GB). Details of fine-tuning are demonstrated in Appendix C.
Evaluation We evaluate the downstream tasks performance with lm-evaluation-harness package (Gao et al., 2023). We evaluate 0-shot accuracy of SciQ (Welbl et al., 2017), PIQA (Bisk et al., 2020), Winogrande (Sakaguchi et al., 2020), ARC Easy, ARC Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), LogiQA (Liu et al., 2020), BoolQ (Clark et al., 2019), and LAMBADA (Paperno et al., 2016). We show accuracy of 5-shot MMLU (Hendrycks et al., 2021). For the latency, we measure the inference speed on a machine with 8 Nvidia A100 GPUs connected by NVLink and report the average latency reduction across batch sizes ranging from 1 to 256.

heading: 5.2 Downstream Tasks Performance and Latency Reduction
text: Table 1 shows that our method maintains a close level of performance to that of the original model while achieving significant latency reduction. LLaMA-13B and LLaMA-33B utilizing p = 4 outperform those with p = 2, attributed to employing a larger s which keeps more bottom layers unchanged. With p = 4, our method processes four layers in parallel, achieving further latency improvements over p = 2. Additionally, as the model size increases, our method demonstrates further latency reductions. This improvement is attributed to the increased quasi-independence of layers observed in larger models.

heading: 5.3 Effect of Bypassing
text: Table 2 elucidates the effect of bypassing distances on LLaMA-7B with p = 4, s = 14, e = 29. The results suggest that increasing bypassing distance d leads to a gradual improvement of performance, attributed to diminished information loss. Meanwhile, there’s a slight decrease in latency reduction as the communication cost rises with larger d. However, even at d = 3, the additional communication
cost associated with the bypassing method is still minor in our testing environment. We also recognize that the communication cost depends on the speed of links between GPUs. Therefore, we suggest to select d based on the inference environment.

heading: 5.4 Trade-Off between Performance and Acceleration
text: CQIL involves three hyperparameters, p, s, and e. The selection of p is based on the number of available GPUs. Choices for s and e depend on balancing performance against acceleration. As shown in Table 3, the performance on downstream tasks and latency reduction vary consistently along with s and e. A larger s and a smaller e, thereby more layers are unchanged, lead to improved performance and decrease in latency. We suggest to select s and e according to the balance between performance and latency reduction.

heading: 5.5 Comparison with Tensor Parallelism
text: Both tensor parallelism and our method utilize additional GPUs to reduce latency. Therefore, we assess the latency reduction ratio between these two approaches. Given that DeepSpeed inference framework involves optimizations beyond tensor parallelism, it is challenging to make a direct comparison of latency. Therefore, we first measure the latency of both methods using a single GPU under various batch sizes, then evaluate the relative latency reduction ratio with an increasing number of GPUs. The comparison is carried out on the LLaMA-13B model, which is sufficiently large for tensor parallelism to be applied across four GPUs. The maximum batch size of tensor parallelism is 64 due to the limit of available GPU memory.
As shown in Figure 4, tensor parallelism does not significantly accelerate the inference for small batch sizes. In contrast, CQIL consistently offers latency reductions for LLaMA-13B, making it more amenable to the scenario of online inference. Moreover, tensor parallelism is complementary to our approach, allowing for the parallel computation of bottom layers, which are sensitive to input substitution. Integrating tensor parallelism with our approach may reduce the latency further. Due to time constraints, we leave the integration of tensor parallelism with CQIL for future work.

heading: 5.6 Combined with Pruning
text: Pruning has been extensively studied as an efficient technique to accelerate model inference. Recently, Xia et al. (2023) has proposed targeted structured pruning to produce efficient LLMs. Our method is orthogonal to pruning and thus possible to be integrated with it. Table 4 presents the result of applying CQIL on the Sheared-LLaMA-2.7B model with p = 2, s = 13, e = 30, and Sheared-LLaMA1.3B with p = 2, s = 13, e = 22. Results show that our approach achieves further latency reduction for pruned models, proving its effectiveness.

heading: 6 Discussion
text: Currently, the explanation of the effectiveness of transformer based LLMs can be classified into two views, pipeline (Tenney et al., 2019; Jawahar et al., 2019; Geva et al., 2021) and ensemble (Veit et al., 2016; Greff et al., 2017; Bhojanapalli et al., 2021). Based on following observations, our research sug-
gests that LLMs works as a combination of both pipeline and ensemble mechanisms.
First, our findings reveal that the bottom layers of LLMs display distinct input/output characteristics, rendering them challenging to be parallelized. On the other hand, the middle and top layers demonstrate considerable similarity in their inputs, making it possible for straightforward conversion to parallel execution without sacrificing the model performance. This observation is compatible with previous work (Zhang et al., 2022a), proving the increased robustness in higher layers within the transformer architecture.
In other words, our contributions substantiate that LLMs employ a pipeline mechanism at lower levels and an ensemble strategy at higher levels, particularly through the successful parallelization of the middle and upper layers. This work thus provide further evidence to deepen our comprehension on the internal mechanism of LLMs.

heading: 7 Conclusion
text: In this paper, we propose an efficient concurrent computation framework on quasi-independent layers that are pervasive in LLMs with the purpose of reducing inference latency while maintaining model performance. To mitigate the potential information loss resulting from input alignment and improve the performance, we develop the bypassing technique, transmitting attention outputs among input-aligned layers. Our experimental results justify the effectiveness of CQIL method. In the future, we will apply CQIL on even larger models and implement the integration of tensor parallelism with our approach to achieve further latency reductions.
Limitations
This work has three primary limitations. First, CQIL requires additional GPUs to reduce the inference latency, limiting its application in the environment equipped with only a single GPU. Second,
not every layer in LLMs could be computed concurrently, especially for layers at the bottom and top. These layers are computed on a single GPU, leaving remaining GPUs ignored. To efficiently utilize all GPUs, we suggest to apply tensor parallelism for these layers. Due to time constraints, the integration of tensor parallelism and our approach is leaved for future work. Finally, the computing FLOPS remain unchanged. Therefore, from an energy consumption perspective, CQIL does not result in electricity savings.

heading: A Attention-FFN Parallelism
text: Given that both the attention and feedforward modules use pre-layer normalization, aligning their in-
puts is feasible. However, the computation times for these two modules usually do not match, leading to one GPU being underutilized if they are processed concurrently on separate GPUs. Therefore, attention-ffn parallelism is not employed in our study. Nevertheless, parallelism between the attention and feedforward modules holds potential for acceleration, especially when two computational resources are available but their speeds differ. For example, Song et al. (2023) proposes to utilize both CPU and GPU for LLMs inference on personal computers. The formulation of attention-ffn parallelism is shown in Eq.5.
xl+1 = xl + ATTNl(xl) + FFNl(xl) (5)
Our empirical experiments show that, the attention and feedforward modules can be computed concurrently in most layers. Specifically, we parallelize attention and feedforward modules for layers ranging from 13 to 30 in the LLaMA-7B model and achieve an average downstream task score of 63.7 (compared to the original model’s score of 65.0). Such parallelism may enhance the efficiency of existing CPU-GPU collaborative inference frameworks, paving the way for further acceleration.

heading: B Pre-training Parallel Architecture
text: Our work focuses on converting pre-trained LLMs into a model with parallel layers. An alternative strategy is to pre-train an architecture with parallel layers from scratch, potentially yielding comparable results and latency improvements. We explored this direction through experiments. We first pre-trained a 12 layers GPT-2(Radford et al., 2019) model with 100K steps on Wikipedia and BookCorpus(Zhu et al., 2015) datasets. Then the model was converted to the parallel architecture with p = 2, s = 3, e = 12 and fine-tuned with 5K steps. Meanwhile, we pre-trained the parallel architecture with 105K steps from scratch. We evaluate zero-shot perplexities on LAMBADA, PTB and WikiText103 as the performance of downstream tasks. Results in Table 5 indicate that training a parallel architecture from scratch does not achieve
the downstream task performance attained by finetuning the transferred model. This performance gap may arise from the enhanced fitting capabilities inherent to deeper models during pre-training. The finding underscores the critical role of transitioning from pre-trained LLMs to the parallel architecture.

heading: C Details of Fine-tuning
text: We fine-tune all models with context length of 2,048, weight decay of 0.1, batch size of 32, dropout probability of 0, warmup steps of 0 and steps of 8000. The learning rate is constant and Adam optimizer is used. For simplicity, lora alpha is always equal to lora rank. We fine-tune all linear matrices without biases in attention and feedforward modules. Mixed precision of bf16 and DeepSpeed(Rajbhandari et al., 2020) framework are used for fine-tuning. More details could be found in Table 6.
D The Potential Performance Degradation of Pruning
During the review phase of our manuscript, two similarity based pruning methods(Men et al., 2024; Gromov et al., 2024) were posted. The findings reported in these studies indicate that the impact on performance is negligible when pruning up to half of the layers in LLMs. Considering the shared motivation but distinct approaches of these methods, we conduct the comparative experiments to demonstrate the potential problems of pruning.
Specifically, we applied the two layer pruning methods for Qwen-7B(Bai et al., 2023). Subsequently, the pruned models were fine-tuned using LoRA with rank of 64, constant learning rate of 1e-4, batch size of 32 context length of 2,048, and steps of 8000. For CQIL, we set e = 30, d = 1
and progressively decrease the s to obtain models with different depth. The CQIL applied models are fine-tuned under identical settings. For ease of reference, we denote these two approaches as ShortGPT(Men et al., 2024) and UIDL(Gromov et al., 2024), respectively. To assess the performance of downstream tasks, the fine-tuned models are evaluated on MMLU, ARC-Challenge and HellaSwag datasets in a zero-shot setting.
The result, as illustrated in Figure 5, demonstrates that the MMLU score has minimal degradation with the pruning of fewer than 9 layers, suggesting the potential redundancy of LLMs. However, as shown in Figures 6 and 7, the performance of pruned models consistently decline on other
benchmark datasets. These observations suggest that while the MMLU dataset is substantial, it may not fully capture the comprehensive capabilities or knowledge inherent to LLMs. Consequently, we argue for the necessity of retaining LLM layers to maintain their true functional potential.

heading: E Downstream Tasks Performance of CQIL-Models without Fine-Tuning
text: Without additional training, models with CQIL still preserve a close level of performance to the original. Results without fine-tuning are shown in Table 7.

heading: F Additional Details of Input Substitution Experiments
text: To clarify the details of Figure 2, we give the numerical values of each substitution experiment in Table 8 and 9. Moreover, we conducted the sensitivity experiment on LLaMA-7B with MMLU under zero-shot setting. The results in Table 8 are similar to Figure 2, revealing that the majority of middle layers are relatively insensitive.

heading: G Additional Bypassing Experiments
text: We conducted additional experiments for LLaMA-7B with p = 2. The results in Table 10 confirm that bypassing technique improves the performance on all LLaMA-7Bs with different p values.

heading: Future Work
text: In the future, we will apply CQIL on even larger models and implement the integration of tensor parallelism with our approach to achieve further latency reductions. Limitations
This work has three primary limitations. First, CQIL requires additional GPUs to reduce the inference latency, limiting its application in the environment equipped with only a single GPU. Second,
not every layer in LLMs could be computed concurrently, especially for layers at the bottom and top. These layers are computed on a single GPU, leaving remaining GPUs ignored. To efficiently utilize all GPUs, we suggest to apply tensor parallelism for these layers. Due to time constraints, the integration of tensor parallelism and our approach is leaved for future work. Finally, the computing FLOPS remain unchanged. Therefore, from an energy consumption perspective, CQIL does not result in electricity savings