AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment

Authors: yonggan  fu
zhongzhi  yu
junwei  li
jiayi  qian
yongan  zhang
xiangchi  yuan
dachuan  shi
roman  yakunin
celine  lin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:26:00

Abstract: Motivated by the transformative capabilities of large language models (LLMs) across various natural language tasks, there has been a growing demand to deploy these models effectively across diverse real-world applications and platforms. However, the challenge of efficiently deploying LLMs has become increasingly pronounced due to the varying application-specific performance requirements and the rapid evolution of computational platforms, which feature diverse resource constraints and deployment flows. These varying requirements necessitate LLMs that can adapt their structures (depth and width) for optimal efficiency across different platforms and application specifications. To address this critical gap, we propose AmoebaLLM, a novel framework designed to enable the instant derivation of LLM subnets of arbitrary shapes, which achieve the accuracy-efficiency frontier and can be extracted immediately after a one-time fine-tuning. In this way, AmoebaLLM significantly facilitates rapid deployment tailored to various platforms and applications. Specifically, AmoebaLLM integrates three innovative components: (1) a knowledge-preserving subnet selection strategy that features a dynamic-programming approach for depth shrinking and an importance-driven method for width shrinking; (2) a shape-aware mixture of LoRAs to mitigate gradient conflicts among subnets during fine-tuning; and (3) an in-place distillation scheme with loss-magnitude balancing as the fine-tuning objective. Extensive experiments validate that AmoebaLLM not only sets new standards in LLM adaptability but also successfully delivers subnets that achieve state-of-the-art trade-offs between accuracy and efficiency.

Main paper: 
 introduction: 
 The remarkable abilities and transformative impacts of large language models (LLMs) [1, 2, 3, 4] have been paralleled by a growing interest in deploying them across a wide range of real-world applications and diverse platforms. However, given the rapid evolution of computational platforms and varying application-specific requirements, the challenge of deploying LLMs efficiently on various platforms with differing specifications has become more pronounced. This is because diverse platforms often feature different resource constraints and deployment flows, necessitating LLMs with varying structures and shapes (i.e., depth and width) to achieve maximized execution efficiency, as affirmed by our profiling in Sec. 2. Moreover, even the same platform may have varying requirements for LLMs' execution efficiency depending on factors such as on-device battery status. These varying requirements demand a flexible framework capable of adapting to both the intrinsic hardware constraints and the extrinsic demands of diverse application scenarios.Existing efficient LLM solutions [5, 6, 7, 8, 9, 10] , which primarily use model compression to bridge the gap between the resource constraints of the target device and the prohibitive complexity of LLMs, fail to fully address the above needs. This is because these solutions either focus on a single dimension of compression, resulting in limited efficiency improvements, or require a costly fine-tuning process for each target platform with its unique specifications. This strategy is particularly unscalable and inefficient for deploying widely used public LLMs like LLaMA [1, 2] , where each user must compress and fine-tune the LLMs for their specific platform and application needs.In light of this, it is highly desirable to develop a suite of LLMs designed such that compressed subnets of arbitrary shapes, which can achieve the accuracy-efficiency frontier without the necessity of individual fine-tuning, can be instantly extracted, thus allowing for immediate adaptation to the diverse needs of various platforms and applications. To achieve this, previous one-for-all training techniques [11, 12, 13, 14, 15] , which strategically sample subnets for joint training to deliver models with switchable complexity, are promising candidates. However, directly applying them to pre-trained LLMs would lead to failure due to the following challenges: (1) their adopted subnet sampling strategies, which are dedicated to models trained from scratch, are not applicable to extensively pre-trained LLMs as informative and critical components that store useful knowledge are highly likely to be skipped; (2) jointly fine-tuning different subnets on commonly adopted small-scale tuning datasets can easily cause severe gradient conflict [16, 17] for LLM weights pre-trained on a large corpus, thus leading to poor performance of all subnets.To address these challenges, we propose a framework called AmoebaLLM, which endows a given LLM with the ability to instantly derive compressed subnets of arbitrary shapes that can achieve the accuracy-efficiency frontier. We achieve this through the development of three key components of AmoebaLLM's one-for-all fine-tuning scheme: a subnet selection strategy, a trainable adapter design, and a fine-tuning objective. Specifically, we summarize our contributions as follows:• We develop a framework called AmoebaLLM, which grants a given LLM the capability to deliver subnets of arbitrary shapes that achieve state-of-the-art (SOTA) accuracy-efficiency trade-offs after a one-time fine-tuning. In this way, AmoebaLLM can greatly facilitate rapid deployment across varying platforms and applications. This is achieved by integrating the following three key components to enable one-for-all fine-tuning.• For extracting high-quality subnets with diverse shapes, we propose a knowledge-preserving subnet selection strategy that features dynamic programming (DP)-based depth shrinking and importance-driven width shrinking. This addresses the aforementioned challenge (1) by preserving the encoded factual knowledge and reasoning capabilities of pre-trained LLMs.• For the trainable adapter during one-for-all fine-tuning, we propose a shape-aware mixture of LoRAs (SMoL), which selects and combines a sparse set of LoRAs [18] using a gating function that takes the subnet shape as input. This technique addresses the aforementioned challenge (2) by mitigating gradient conflicts among subnets. More importantly, once the target subnet shape is determined based on the target platform/application at deployment time, all the selected LoRAs can be merged into the LLM weights, thus eliminating overhead.• For the fine-tuning objective, we enhance the in-place distillation strategy [11, 13, 14] by integrating a loss-magnitude balancing scheme. This scheme is based on the observation that the loss magnitudes of subnets with different shapes in pre-trained LLMs are unbalanced, leading to a bias toward specific subnets and thus poor overall performance. Our proposed technique effectively addresses this issue and improves the performance of all subnets.• Extensive experiments validate that our AmoebaLLM can deliver LLMs with instantly serviceable subnets of any shape, each performing better or on par with SOTA efficient LLM solutions. Additionally, when leveraging AmoebaLLM as a pure LLM compression framework to compress a given LLM to the target parameter, it can achieve new SOTA compression effectiveness, thanks to its subnet selection strategy. 
motivation and profiling: 
 Before introducing our framework, we first conduct a profiling of generation latency across different devices, deployment flows, and use cases to examine the demand for LLMs with adaptable structures that meet the diverse needs of various platforms and applications. Profiling setup. We study the efficiency of different LLM shapes on different devices by uniformly scaling either the depth or width of LLaMA2 7B [1] to the same model size. Here, depth is defined as the number of self-attention blocks, each including both a multi-head attention module and a feed-forward network, while width is defined as the hidden dimensions. We profile these workloads using (1) two devices, including an NVIDIA A5000 consumer-level GPU and an NVIDIA Jetson Orin NX edge GPU; and (2) three deployment flows, including TensorRT-LLM [19] , MLC-LLM [20] , and vanilla PyTorch [21] .Observation and analysis. As shown in Fig. 1 , we can observe that (1) first of all, the same workload on two GPUs with different resources exhibits large latency gaps, indicating the need for LLMs with adaptable structures to adapt to different devices when aiming to ensure a comparable latency to satisfy user needs; (2) the preference regarding LLM shapes differs across deployment flows. Specifically, under the same model size, reducing model depth and width have a comparable impact on the measured latency using TensorRT-LLM on A5000 and MLC-LLM on Orin NX, while reducing depth using PyTorch or MLC-LLM on A5000 can achieve a notably lower latency than reducing width. This implies that for emerging platforms with limited compatible deployment flows, proper customization of LLM shapes for maximized efficiency is needed.3 The Proposed AmoebaLLM Framework 
amoeballm: methodology overview: 
 To address the challenges associated with traditional one-for-all network training, as mentioned in Sec. 1, our AmoebaLLM is equipped with three components: a knowledge-preserving subnet selection strategy, an SMoL adapter, and an in-place distillation fine-tuning objective with loss-magnitude balancing. We illustrate our overall framework in Fig. 2 : given a target LLM, our AmoebaLLM endows it with the capability of instantly deriving capable subnets via a two-stage process.In the first stage, AmoebaLLM generates the subnet selection strategy. Specifically, given the target depth/width remaining ratios, this step decides which layers/neurons to maintain, respectively. To maximally preserve the knowledge and language modeling capabilities of pre-trained LLMs, we propose employing dynamic programming [22] to determine the retained layers under different remaining ratios and leverage neuron importance metrics [8] to retain important neurons in a structured manner, as detailed in Sec. 3.2. After this stage, the subnet selection strategy is determined and fixed.In the second stage, we insert our proposed SMoL adapter into the target LLM for a one-time, one-for-all fine-tuning. Specifically, SMoL is composed of a set of LoRAs [18] and employs a gating function to sparsely activate different subsets of LoRAs for different subnets, thus mitigating their gradient conflicts. This process is elaborated upon in Sec. 3.3. In addition, our fine-tuning objective enhances the sandwich sampling and in-place distillation methods described in [11, 13, 14] by adding a loss-magnitude balancing scheme. This prevents bias towards specific subnets in pre-trained LLMs, as detailed in Sec. 3.4. At deployment time, the selected LoRAs, which are determined by the extracted subnet shape favorable to the target platform, can be merged into the LLM weights.Motivation. As detailed in Sec. 5, previous one-for-all training techniques [11, 12, 13, 14, 15] often select the first layers of a model or the first channels of a layer. These techniques, intended for models trained from scratch, are unsuitable for pre-trained LLMs with rich knowledge encoded in their weights. Considering the difficulty of recovering lost knowledge through fine-tuning or our one-for-all fine-tuning on a relatively small corpus [23] , it is crucial to identify informative and critical layers/neurons that store useful knowledge during the subnet selection process instead of relying solely on fine-tuning. To this end, we propose our knowledge-preserving subnet selection strategy to select the most informative layers/neurons under a given remaining ratio, as detailed below.DP-based depth shrinking. Previous works have made diverse observations regarding the layer locations that store knowledge in different series of language models [24, 25, 26, 27, 28, 29] . As such, it is highly desirable to have a principled strategy to evaluate the joint contributions of different layer combinations in a target LLM to derive the optimal layer selection strategy for each remaining ratio. To achieve this goal, we propose a DP-based depth shrinking strategy.Problem formulation. Given a target LLM with N decoder layers, we define the layer selection strategy by a vector s ∈ {0, 1} N . Here, s[n] = 1 indicates that the n-th layer is retained; otherwise, the layer is removed. The objective is to determine the selection strategy s that achieves the optimal target metric, such as maximal accuracy or minimal perplexity (PPL), on a calibration dataset C, subject to the constraint that M layers are removed.Key hypothesis. Thanks to the residual structure [30] of common LLMs [1, 2, 3] and the observations that LLMs' knowledge is compositional across layers [24, 31, 32] , we hypothesize that the layer selection problem described above can be divided into smaller and approximately independent subproblems. Consequently, we can employ dynamic programming [22] to effectively and efficiently solve the layer selection problem.Our DP-based methodology. We define a DP As illustrated in Fig. 2 (a), similar to general DP problems [22] , D[n][m] can be derived by a recurrence relationship. Specifically, to derive eachD[n][m],we compare the metrics achieved by the following two cases: (1) removing m layers from the first n -1 layers, and (2) removing m -1 layers from the first n -1 layers and removing the n-th layer. The strategy yielding better metrics is adopted. More formally, this process can be formulated as follows:EQUATIONwhere P(n, m) is the metric obtained by removing the n-th layer on top of the best-known strategyS[n -1][m -1]for removing m -1 layers from the first n -1 layers. This is computed as follows:EQUATIONwhere remove(s, n) is a function that sets the n-th layer to 0 in a strategy s. Leveraging this recurrence relationship, after initializing the DP table with the base cases, i.e., Differences from and connections with previous methods. The most relevant direction is layer pruning for LLMs. Pioneering works along this direction either utilize single-layer importance to determine which layers to prune [9] , failing to measure the layers' joint contribution and thus aggressively losing factual knowledge as shown in Sec. 4, or rely on pre-defined rules [33] , i.e., removing consecutive layers among the last ones, which is suboptimal and may not be generalizable to future LLMs. In contrast, our DP-based strategy can measure the joint impact of different layer combinations and is principled and fully automated without relying on human priors for specific LLMs. More importantly, the two aforementioned works [9, 33] are subsets of our DP-based strategy's solution space. Thus, it can serve as a new SOTA layer pruning method, as demonstrated in Sec. 4.3, in addition to serving as a component (i.e., the subnet selection strategy) in one-for-all fine-tuning. Previous works [34, 35] have also applied dynamic programming for pruning at different granularities with varied formulations, and our work is the first to introduce this classical approach to LLMs.D[i][0] = ∞ and S[i][0] = {1} N (∀i ∈ [1, N ]),Importance-driven width shrinking. Compared to layer selection, removing neurons (and all corresponding weight connections) for width shrinking is more fine-grained and involves a much larger design space. Therefore, instead of using dynamic programming, we directly employ existing structured pruning metrics for LLMs to obtain importance scores for each neuron and select the most important ones during one-for-all training, given a width remaining ratio.To achieve this, we employ the metric in [8] due to its outstanding performance compared to other alternatives. Specifically, we adopt the same width remaining ratio for linear layers within the same block (either the self-attention block or the feed-forward network), where the importance scores of input neurons of the last linear layer in each block are used to determine the width configuration of this block. In other words, when a subset of input neurons to the last linear layer is removed, all associated neurons and weights [7] within this block will be removed in a structured manner.Importance metric. The importance score F ℓ i of the i-th input neuron in the last linear layer W ℓ of the ℓ-th block is computed asF ℓ i = E k,t,j (X ℓ k,t,i -X ℓ i ) 2 • ∥W ℓ j,i ∥ 2 2, where j is the index of the output neurons, X ℓ k,t,i is the input features of the t-th token in the k-th batch, and X ℓ i is the averaged input features across these two dimensions, both received by the i-th input neuron. To better maintain the LLMs' capability under a given remaining ratio: (1) the importance score is further normalized over all input neurons in each layer and globally sorted for non-uniform width shrinking; (2) a pre-computed bias term B ℓ is added to the output neurons to compensate for the removed input neurons, i.e.,B ℓ = W ℓ ((1 -M ℓ ) ⊙ X ℓ ), where M ℓ is the binary mask indicating whether the input neurons are retained. We refer the readers to [8] for more details.The Proposed Shape-aware Mixture of LoRAs Motivation. As demonstrated in Sec. 4.3, joint weight fine-tuning of different subnets on smallscale datasets can lead to severe gradient conflicts [16, 17] , resulting in poor performance across all subnets. One potential solution is to adopt parameter-efficient adapters like LoRA [18] . However, accumulating gradients from all subnets onto the same LoRA still suffers from gradient conflicts, making fine-tuning unstable and causing some subnets to underperform. On the other hand, tuning a separate LoRA for each subnet configuration is infeasible due to the large design space. To address this, we propose an intermediate solution to balance performance and efficiency: the SMoL adapter that features a set of LoRAs, which are sparsely activated based on the subnet shape.SMoL adapter design. Our SMoL adapter consists of a set of T LoRAs {∆W i = B i A i } T i=1 , which are sparsely activated for each subnet shape using a gating function G. Specifically, we employ a one-hot mask M to indicate the shape, i.e., the layer/width configuration, of the subnet. This mask is fed into the gating function G to calculate a score for each LoRA. Only the top k LoRAs are activated and weightedly averaged for each subnet shape during fine-tuning, thus mitigating the gradient conflicts among different subnets.Implementation. We extend the noisy top-K gating mechanism from [36] to implement our SMoL. Specifically,G(M) = Softmax(KeepTopK(H(M), k)), where KeepTopK(v, k) i is v i if v i is in the top k of v and -∞ otherwise, and H(x) i = (x • W g ) i + std • Softplus((x • W noise ) i ).Here, W g and W noise are learnable, with the latter used to control the noise std for load balancing [36] . The final composited weight W = W base + T i=1 G(M) i ∆W i , where W base is the pre-trained weight. Key difference from previous works. Previous mixture-of-LoRA designs [37, 38, 39] are inputadaptive at inference time, and thus the weights of different LoRAs cannot be merged into the original model. In contrast, our SMoL depends only on the subnet shape and is independent of model inputs. This implies that once the target subnet is extracted based on the target platform, all activated LoRAs can be merged into the model weights, thus enhancing parameter efficiency at deployment time.3.4 AmoebaLLM: The Fine-tuning Objective Motivation. Strategically sampling and jointly fine-tuning different subnets is necessary to ensure high-performance subnets within the same LLM, where the sandwich sampling and in-place distillation mechanisms [11, 13, 14] can serve as promising sampling and fine-tuning schemes, respectively. However, naively doing so can cause the larger subnets to gradually underperform during fine-tuning. We identify that this is due to notable differences in the loss magnitudes of different subnets, with smaller subnets, which diverge more from the well pre-trained full model, exhibiting much higher losses than larger subnets. To address this, we propose a simple but effective solution: equipping the in-place distillation with a loss-magnitude balancing mechanism, which we elaborate on as follows.In-place distillation with loss-magnitude balancing. During each fine-tuning iteration, we employ the sandwich sampling [11, 13, 14] to sample K subnets {T i } K i=1 with different layer/width remaining ratios, including the largest/smallest ones and K -2 random ones from our design space. Detailed layer/width configurations of sampled subsets can be obtained from the strategies derived in Sec. 3.2. We fine-tune our SMoL adapter as detailed in Sec. 3.3 by accumulating the gradients from all sampled subnets using in-place distillation, where only the loss of the largest subnet T 1 is calculated using ground truth, while those of other subnets {T i } K i=2 use distillation from the largest one [11] . To balance the loss magnitude from different subnets, we normalize all subnets' loss magnitudes to that of the largest subnet, as visualized on the rightmost side of Fig. 2 (b ). In this way, the final loss direction is jointly determined by all subnets' loss directions, falling on a unisphere without being severely impacted by their unbalanced magnitudes. We formulate the fine-tuning objective as follows:EQUATIONwhere x and y are the input and ground truth, respectively, and L CE is a cross-entropy loss function.Final subnet search after fine-tuning. We adopt a simple hierarchical search strategy to select subnets from the fine-tuned LLM to satisfy the target efficiency constraint while maximizing achievable accuracy. Specifically, we first perform a coarse grid search across uniformly spaced depth and width settings based on a small calibration set (e.g., 40 samples from the MMLU dataset) to identify subnets that meet the efficiency constraint with maximized accuracy. Next, we conduct a more fine-grained grid search within depth and width ranges surrounding the optimal subnet identified in the coarse grid search stage. More advanced search strategies, such as evolutionary search [11, 12, 13, 14] , are left for future work.4 Experimental Results 
experiment setup: 
 Baselines. Our baselines include two SOTA structured width pruning methods: LLM-Pruner [7] and FLAP [8] , and one layer pruning method Shortened LLaMA [9] . All these baselines are open-sourced and we apply their official code to different LLMs. All baselines, including FLAP which were not fine-tuned in their original paper, are fine-tuned using the settings below for a fair comparison.Fine-tuning setting. Following [7, 9] , we adopt 50K samples from Alpaca [40] for our one-for-all fine-tuning as well as for fine-tuning all baselines. For both our method and the baselines, we adopt a constant learning rate of 2e-4 with an AdamW optimizer and a LoRA rank of 64, and fine-tune for 10K iterations. It takes 40 GPU hours on an NVIDIA A5000 GPU for our one-for-all fine-tuning. Models. We apply our method to LLaMA2 7B [1] and Vicuna 7B v1.5 [41] .Evaluation. Following our baselines [7, 9, 8, 6] , we leverage lm-evaluation-harness [42] to measure the zero-shot accuracy on 7 commonsense reasoning datasets, including BoolQ [43] , PIQA [44] , HellaSwag [45] , WinoGrande [46] , ARC-easy [47] , ARC-challenge [47] , and OpenbookQA [48] .We also benchmark on the factual knowledge dataset MMLU [49] .AmoebaLLM's setting. We set the depth choices to range from 20 to 32 and the width remaining ratios as {1, 7/8, 3/4, 5/8, 1/2} by default unless specifically stated. In each iteration, we sample 4 subnets for joint training and select 2 LoRAs out of a total of 5 LoRAs in SMoL for each subnet.We benchmark our AmoebaLLM against SOTA LLM width/layer pruning methods on LLaMA2 7B/Vicuna 7B v1.5 in Tab. 1/Tab. 2, respectively. Note that the subnets produced by AmoebaLLM are instantly extracted from the same one-for-all fine-tuned LLM, where the (depth, width scale) settings for 80%/65%/50% remaining ratios, determined by the final subnet search in Sec. 3.4, are (30, 0.875)/(28, 0.75)/(22, 0.75), respectively. We also provide the per-subnet fine-tuned counterparts of our delivered subnets, denoted as AmoebaLLM † , to compare one-for-all and individual fine-tuning.Benchmark under comparable model sizes. As shown in Tab. 1 and Tab. 2, we observe that (1) the subnets instantly extracted by AmoebaLLM can achieve higher MMLU accuracy compared to all baselines, suggesting that our method better preserves the factual knowledge acquired during pretraining, as further analyzed in Sec. 4.3 and Sec. 4.4; (2) AmoebaLLM's delivered subnets, extracted from the same model, can also achieve better or comparable average commonsense reasoning accuracy compared to the strongest baselines, each trained separately; (3) AmoebaLLM † achieves the best performance across all metrics and tasks compared to the baselines, indicating that AmoebaLLM † can serve as a new SOTA LLM compression framework in addition to its one-for-all functionality, thus advancing the achievable accuracy-efficiency trade-off; (4) compared to our per-subnet finetuned variant AmoebaLLM † , the instantly delivered subnets achieve comparable performance. This demonstrates the effectiveness of our one-for-all fine-tuning scheme, as further ablated in Sec. 4.3. Benchmark accuracy-latency trade-offs on real devices. We further benchmark the achieved trade-off between average commonsense reasoning accuracy and measured latency of LLaMA2 7B using MLC-LLM and PyTorch as the deployment flows on an NVIDIA A5000 GPU, following the settings in Sec. 2. For our method, we select the subnet shape that favors the hardware characteristics based on the profiling in Sec. 2 from the one-for-all fine-tuned LLM. As shown in Fig. 3 , we observe that (1) our method consistently achieves the best tradeoff on both deployment scenarios; and (2) although FLAP achieves higher accuracy than Shortened LLaMA under comparable model sizes, its real-device speed is limited when using vanilla PyTorch, aligning with observations in [9] . In contrast, our method can instantly deliver subnets that favor the hardware/deployment flow characteristics, thus enjoying both high accuracy and real-device friendliness.We perform ablation studies to validate the effectiveness of each component of AmoebaLLM.The DP-based depth shrinking strategy. We benchmark our DP-based strategy against two existing LLM layer pruning methods, Shortened LLaMA [9] and Unreasonable [33] , on LLaMA2 7B. Specifically, we employ the three methods to select important layers using Wikitext2/MMLU with PPL/accuracy as calibration metrics, respectively. We directly report the achieved PPL/MMLU accuracy after calibration under various layer remaining ratios without fine-tuning to indicate their effectiveness in identifying important layers Observations and analysis. As shown in Tab. 3, we observe that our DP-based strategy outperforms the other two strategies on both calibration datasets and metrics, especially under small remaining ratios, e.g., a +9.4% MMLU accuracy and a -33.1 PPL over the strongest baseline when remaining 18 layers. This demonstrates the superiority of our method over the two baselines in selecting important layers that optimize the target calibration metric, thus significantly contributing to knowledge preservation.Remark. This set of experiments supports our analysis in Sec. 3.2 that the superiority of our method arises from its consideration of different layers' joint contributions, rather than focusing on singlelayer importance [9] , and its avoidance of reliance on pre-defined rules [33] , thus ensuring generality.The SMoL adapter. To assess the efficacy of our SMoL adapter, we substitute it with full model fine-tuning or the standard LoRA [18] and benchmark it against our AmoebaLLM with SMoL as well as the per-subnet fine-tuning variant. To more clearly demonstrate its efficacy, this experiment is conducted under a depth-shrinking-only setting, which only enables depth shrinking ranging from 20 to 32 layers during one-for-all fine-tuning. We select three layer configurations, covering both the largest and smallest ones, to report the performance.Observations and analysis. As illustrated in Tab. 4, we observe that (1) full model fine-tuning on a relatively small corpus results in suboptimal performance with large PPL; (2) employing the standard LoRA stabilizes the one-for-all fine-tuning compared to full model fine-tuning, albeit with notable performance reductions in larger subnets, e.g., a 5.8% MMLU accuracy drop in the largest subnet compared to per-subnet fine-tuning; (3) when equipped with our SMoL, the MMLU accuracy of all subnets is substantially enhanced, e.g., a +6.6% improvement on the largest subnet compared to the LoRA case, even exceeding that of the per-subnet counterparts. This demonstrates the essential capability of our AmoebaLLM to deliver high-quality LLM subnets across a wide range of accuracy-efficiency trade-offs.The loss-magnitude balancing scheme. As shown in Tab. 4, we further disable the loss-magnitude balancing scheme and observe a significant MMLU accuracy drop in larger subnets. This supports our analysis in Sec. 3.4 that the larger losses from smaller subnets may dominate the overall objective, thereby impairing the fine-tuning of larger subnets and underscoring the necessity of our method.We conduct an ablation study on the choice of calibration datasets for our DP-based depth shrinking introduced in Sec. 3.2. Specifically, we use accuracy on the training set of MMLU [49] and PPL on the training sets of Wikitext2 [50] /BookCorpus [51] as target metrics. We report the evaluation metrics, including MMLU test accuracy and Wikitext2 test PPL, under different layer remaining ratios both after calibration and after one-for-all fine-tuning under a depth-shrinking-only setting.Observations. As shown in Tab. 5, we observe that (1) after calibration without fine-tuning, the subnets perform well on the evaluation metric for which they were calibrated and underperform in terms of the other metric; (2) after fine-tuning, the subnets calibrated using PPL continue to perform poorly on MMLU accuracy, indicating a severe loss of factual knowledge. In contrast, the subnets calibrated using MMLU accuracy achieve notably lower PPL compared to before fine-tuning, even on par with the subnets calibrated using PPL, while still maintaining high MMLU accuracy.The key insight. This set of experiments indicates that the loss of factual knowledge during compression is hard to restore during fine-tuning, echoing the observations in [23] , while the language modeling capability is easier to recover through fine-tuning. As such, we adopted MMLU as the calibration dataset throughout the previous experiments, and we believe this insight could inspire future LLM compression frameworks and calibration metrics.One limitation of our work is that due to the limited fine-tuning data and resources, our methodology is applied to parameter-efficient fine-tuning, which mitigates gradient conflicts under small data conditions while limiting the achievable accuracy-efficiency trade-off. We anticipate that by leveraging more extensive fine-tuning data beyond our current use of Alpaca [40] and extending our design insights regarding subnet selection and gradient conflict mitigation, more aggressive accuracy-efficiency trade-offs can be achieved, which will be the focus of our future work. 
related work: 
 Large language models. Before the advent of LLMs, transformer-based language models [52, 53, 54, 55] demonstrated their ability to effectively analyze relationships among tokens in complex input sequences, facilitated by the attention mechanism [52] . These models also exhibit notable scalability [56, 57, 58] with respect to model size and the scale of pre-training datasets. This decent scalability has led to the emergence of LLMs, such as GLM [59] , OPT [60] , BLOOM [61] , the Llama family [1, 1, 2], Gemma [3] , and GPT-4 [4] , which exhibit impressive zero-shot and few-shot in-context learning capabilities. However, these LLMs often feature billions of parameters and prohibitive computation complexity, which limits their widespread use across diverse platforms.Large language model compression. To facilitate efficient deployment of LLMs in real-world applications, existing works primarily focus on compressing LLMs by extending traditional compression techniques such as knowledge distillation [62, 63] , quantization [64, 65, 66, 67, 68, 69] , system acceleration [70, 71] , and pruning [6, 5, 72] . Our work is most closely related to LLM pruning. Along this direction, early works [5, 6] employ unstructured and semi-structured pruning [73] by zeroing out connections among neurons. Despite their plausible performance, these methods require specialized support to achieve real-device speedup. To benefit commodity platforms, structured LLM pruning methods remove more coarse-grained components, such as all connections related to a single neuron [7, 8, 10] or even entire layers [9, 33] . For instance, LLM-Pruner [7] and FLAP [8] reduce LLM width by eliminating identified redundant neurons, while Sheared-LLaMA [10] learns a set of binary masks to reduce both the width and depth of LLMs. However, these methods either focus on a single dimension of compression (i.e., depth or width) with limited efficiency improvements, or they require a costly fine-tuning process for each target configuration and platform. In contrast, our AmoebaLLM can instantly extract subnets of arbitrary shapes that reach the accuracy-efficiency frontier, thus facilitating rapid deployment across devices.One-for-all networks. Slimmable networks [74, 75] are pioneering works that enable a single model to operate at varying widths. Follow-up works [11, 76, 12, 13, 14, 15] further extend this approach to train more general one-for-all networks with switchable depth and width, thus enabling tasks like neural architecture search. In particular, BigNAS [11] builds one-for-all networks using a sandwich sampling strategy that samples a random set of subnets and jointly trains them in each iteration through in-place distillation, where the largest model guides the learning of the smaller ones. This approach has been inherited by subsequent works [12, 13, 14] . Additionally, this idea has been extended to any-precision networks [77, 78, 79] that allow switchable precision at runtime. A very recent work [80] has further extended this concept to any-precision LLMs.Nevertheless, directly applying these methods to the depth and width of pre-trained LLMs would likely fail because their subnet sampling strategies often select the first layers of a model or the first channels of a layer, which are intended for models trained from scratch. This approach is unsuitable for pre-trained LLMs, as it may omit layers or neurons containing crucial knowledge. Our AmoebaLLM framework addresses these challenges by developing three key components: the subnet selection strategy, the trainable adapter design, and the fine-tuning objective. One concurrent work [81] also aims to train many-in-one LLMs that support instant subnet derivation, targeting a full-model continual training setting on 90 billion tokens. In contrast, our method targets a parameter-efficient tuning setting with only 8.5 million fine-tuning tokens. 
conclusion: 
 In this work, we present a framework called AmoebaLLM that grants a given LLM the capability to instantly deliver subnets of arbitrary shapes, which achieve the accuracy-efficiency frontier and can be extracted immediately after a one-time fine-tuning. This is achieved by the development of three dedicated components that enable AmoebaLLM's one-for-all fine-tuning scheme: a knowledgepreserving subnet selection strategy, an SMoL adapter, and an in-place distillation objective with loss-magnitude balancing Justification: We provided this information in Sec. 5.1 of our paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This work targets the acceleration of large language models to facilitate its widespread use and thus does not suffer from obvious negative societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We built on top of public models and datasets and do not suffer from high risks of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, we have added the reference for all used data.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Justification: Yes, it conforms with NeurIPS Code of Ethnics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: After the rebuttal, the ratings were: borderline accept, borderline accept, accept, and borderline accept. While none of the reviewers rejected the paper, only one expressed strong support. However, the reviewers did acknowledge the practicality of the problem and the novelty of the proposed method.

Reviews: review: paper_summary: paper_summary: To address the problems of diverse resource constraints and deployment flows while using LLM for multiple real-world applications, this paper proposes an AmoebaLLM, featuring a knowledge-preserving subnet selection strategy, a shape-aware mixture of LoRAs and a distillation scheme with loss-magnitude balancing.
strength_weakness: strength_weakness: Overall, the motivation and ideas of this paper are commendable. However, I have a few concerns:
1. DP-based depth shrinking aims to select suitable layers that can achieve the optimal target metric. Does this mean that the proposed method needs to evaluate the performance for each selection strategy? If so, how long is required for each evaluation? What is the relationship between the number of layers and the overall search time?
2. Another concern is about the optimal subnet. I wonder if the optimal subnet identified in the first phase is truly “optimal.” I believe that the selected subnet is the optimal choice for the initial parameters. However, the proposed method consists of two stages. After the fine-tuning stage, some subnets that were not optimal in the first phase might achieve the best target metrics. I understand that searching subnets with fine-tuning is very expensive, but have the authors considered this issue? Is this correct?
3. I am particularly interested in the training efficiency comparisons, as the authors mentioned that existing methods are not efficient. However, the experimental section does not show such comparisons. Could the authors provide relevant comparisons in Table 1 and Table 2?
questions: questions: Please see the weaknesses part.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposed a new framework, named AmoebaLLM, that adapts any LLMs to achieve optimal efficiency across different platforms and applications. In specific, the framework contains two stages. The first stage denoted as a knowledge-preserving stage, creates a subnet of the LLM by dynamic programming given an arbitrary depth or width. This second stage denoted as the one-for-all finetune stage is to fine-tune the obtained subnet to achieve the optimal performance on the given application (dataset). To finetune the subnet, the paper proposed shape-aware LoRAs to adapt the knowledge and a loss-magnitude balancing to ensure efficient distillation in the second stage. All combined, the method achieves SOTA performance among all the pruning methods and obtains the best accuracy in the same latency in different platforms.
strength_weakness: strength_weakness: 1. About latency in section 5.2:The work claimed that it achieve the accuracy-efficiency frontier. In terms of accuracy, it is convincing as shown in Table 1. However, for efficiency, it seems only Fig.3 is related to this topic yet only shows a trade-off between latency and accuracy. It is not sufficient to evaluate the efficiency purely based on latency. 

2. In my understanding, efficiency should also consider the resources consumed in the pruning method. First, the DP algorithm itself has a high computational complexity. Second, the proposed method is somewhat limited to its one-for-all finetune, which is a training process related to a specific dataset, and a manual selection of the subnet, the method is not perfectly efficient. In this case, the author may consider adjusting their claim in efficiency.

3. The selection of the subnet seems to be related to the profiling of the generation latency. It is purely empirical and may be error-prone when it is generalized to other platforms.
questions: questions: 1. What is the definition of the latency?

2. Is it possible to provide the details of selecting the subnet shape that favors the hardware characteristics?

3. Which LLM is used across all the experiments?
limitations: limitations: No negative societal impact is found.
paper_summary: paper_summary: This paper proposes AmoebaLLM: a one-for-all fine-tuning and compression framework for delivering pruned and accurate subnets from a pre-trained LLM at various pruning (both depth and width) ratios without the need to fine-tune individual subnets. AmoebaLLM consists of two components: 1) a dynamic programming-based approach to layer selection for selecting subnets of certain depth that takes into account various layer interactions, as opposed to greedy-layer based approaches; 2) after the subnets are defined, a one-for-all fine-tuning strategy is proposed based on gating multiple LoRA heads that are subnet-specific. The training strategy uses a hybrid loss where the largest subnet (always sampled) computes its loss with the ground truth, and all subsequently sampled subnets compute a distilled loss from the largest subnet. The final loss is computed by weighing each subnet's loss appropriately. To demonstrate the efficacy of their method, the authors perform experiments on both LLama 2 7B and Vicuna 7b v1.5 and compare against prior LLM pruning techniques.
strength_weakness: strength_weakness: - The claims that full model fine-tuning suffers from severe gradient conflicts appear rather weak. In line (233) the authors state "As analyzed in Sec. 2 and demonstrated in Sec. 5.3 ... ", whereas Section 2 simply conjectures that this is an issue as it uses phrases like" .. would likely fail .." and "as it may omit layers". I would suggest citing the results from Section 5.3 directly to motivate this issue. As for the results themselves in Table 3, the performance of the full model (second row) does not seem consistent. While it performs poorly on Wikitext2, the MMLU performance remains comparable with the SMoL baseline, and seems to improve when pruning more (24 to 20) as opposed to all other methods. It is not clear how these results lead to the conclusion that there is a "gradient conflict". How would the same experiments look when using the Vicuna model? and how would the other evaluation metrics look?
- Using dynamic programming in the context of neural network pruning is not new (https://arxiv.org/pdf/2308.10438, https://ieeexplore.ieee.org/document/9116287 to name a few). The authors should cite and discuss these works, even if they do not tackle LLMs specifically. 
- In line 199, the authors claim that the DP complexity is O(MN). This is a bit misleading, as it ignores the complexity of computing P(n,m) which I argue is also dependent on M and N. Furthermore, the actual run-time of the DP step is not discussed, is it too negligible? 
- In Table 1, comparing with AmoebaLLM$^\dagger$ is not a fair as it is trained for longer (on top of the training performed in regular AmoebaLLM). How will the other methods compare with the same amount of training?
- It would improve reading clarity if the authors explicitly mention how many layers do the models being studied contain, as simply mentioning how many layers are being pruned in depth shrinkage is hard to gauge without that context. This should also be mentioned in the Tables.
questions: questions: - AmoebaLLM produces different subnets, some may have the same parameter count due to different width/depth shrinkage ratios, but it is not clear how a model is selected for a certain experiment (e.g. Fig 3). Is it simply enumerating all the subnets and picking the fastest  one? Are the profiling results obtained from Section 3 being used for this, or is that purely for motivating the problem?
- What ranks are used for LoRA?
limitations: limitations: The authors have addressed the limitations of their proposed work.
paper_summary: paper_summary: The authors proposed a novel pipeline for once-for-all training multiple subnets in a supernet LLM under different resource constraints. The  entire pipeline consists of a knowledge preserving subnet selection utilizing DP to sample depth and width and a new LoRA to resolve the gradient conflicts during training multiple subnets jointly. The experimental results on two widely used decoder-only architectures show the efficacy of the pipeline.
strength_weakness: strength_weakness: - Some technical details are not clear to me. For example, how are the depth and width defined in decoder models? Is a self-attention block a layer? Or does the authors consider FFN and MHSA as two individual layers? In terms of the width, it is clear for FFN but is that the number of heads in MHSA?
- What metric is used as the importance in subnets sampling? I assume that that importance is not derived from gradients, since DP is done layer-by-layer in forward manner. If so, will that metric work better than gradient-based metrics?
- The authors introduce a new LoRA in the paper. Does the authors freeze the backbone LLM and only tune the adapters? If not, how does the authors avoid the co-adaptation of backbone and adapters? Since the original LoRA is proposed to preserve the learned knowledge and fine-tuning only low-rank adaptation for a small amount of new data. 
- Also are the adapters only added in MHSA? If so, can the authors provide some theoretical or experimental results to show why FFN layers do not suffer from gradient conflicts?
questions: questions: See above
limitations: limitations: Yes

Comments: title: Author Response - Part 2
Author Response - Part 2
comment: Thank you for taking the time to review our rebuttal responses and for providing positive feedback! We are encouraged to hear that our rebuttal has addressed most of your concerns. Following your suggestion, we will include the additional data and clarifications from this rebuttal in our final manuscript.
Thank you for the detailed response. The authors have addressed most of my concerns/questions. The additional data/clarifications provided in this rebuttal should be included in the final manuscript as the authors mentioned. I will increase my score (6->7).
Thank you for taking the time to review our rebuttal responses and provide feedback! Any further suggestions or comments you have are welcome!
Thank you for your reply. I will maintain my score.
Thank you for taking the time to review our rebuttal responses and for providing positive feedback! We are encouraged to hear that our rebuttal has addressed most of your concerns.

Following your suggestion, we will include (1) the real-device efficiency table and (2) the complexity and runtime of our DP algorithm, both in our first response, in the final version of our manuscript to strengthen the coherence of our work.
Thank you for your detailed response. My concerns have largely been addressed, and after considering the feedback from other reviewers, I am inclined to support the acceptance of this paper. I have a few additional suggestions:

1. The table provided in the first response should be included in the main paper. It would strengthen the argument regarding the “Frontier of accuracy and efficiency,”, as particularly the method demonstrates state-of-the-art performance across various efficiency metrics. 
2. It would be beneficial to include a discussion of the complexity of the DP algorithm and its corresponding running time in the main paper. This addition could alleviate any concerns readers might have about the
Dear Reviewer,

We sincerely appreciate the time you dedicated to providing valuable feedback on our paper. In this author response, we have addressed all of your initial concerns. If you have any further questions or concerns, we are happy to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!

Best,

The Authors of Paper #13852
Dear Reviewer,

We sincerely appreciate the time you dedicated to providing valuable feedback on our paper. In this author response, we have addressed all of your initial concerns. If you have any further questions or concerns, we are happy to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!

Best,

The Authors of Paper #13852
Dear Reviewer,

We sincerely appreciate the time you dedicated to providing valuable feedback on our paper. In this author response, we have addressed all of your initial concerns. If you have any further questions or concerns, we are happy to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!

Best,

The Authors of Paper #13852
Dear Reviewer,

We sincerely appreciate the time you dedicated to providing valuable feedback on our paper. In this author response, we have addressed all of your initial concerns. If you have any further questions or concerns, we are happy to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!

Best,

The Authors of Paper #13852
**5. Benchmark with baselines with the same amount of training time**

We first clarify that in Table 1, we follow a standard criterion in the literature, i.e., ensuring the same number of fine-tuning tokens across both the baselines and our method.

Following your suggestion, we further train the baselines with the same amount of training time as our method, which corresponds to 20k training iterations on Alpaca-gpt4. As shown in the table below, we report the MMLU accuracy as well as the average accuracy across 7 tasks, following the task list of Table 1. We observe that (1) the baselines fine-tuned with 20k iterations generally maintain comparable performance with those trained with 10k iterations, potentially because 10k iterations is sufficient for fine-tuning on Alpaca-gpt4; and (2) our method still outperforms all baselines across various tasks.

| **Remaining  Ratio** | **Method** | **Training Iterations** | **Training Time** | **MMLU (%)** | **Average Acc (%)** |
|:---:|:---:|:---:|:---:|:---:|:---:|
| 80% | FLAP | 10k | 15h | 40.21 | 60.98 |
|  |  | 20k | 30h | 38.62 | 61.40 |
|  | Shortened LLaMA | 10k | 15h | 26.45 | 58.72 |
|  |  | 20k | 30h | 26.55 | 59.63 |
|  | Ours  | 10k | 30h | **40.70** | **62.29** |
| 65% | FLAP | 10k | 15h | 33.28 | 56.12 |
|  |  | 20k | 30h | 35.4 | 55.93 |
|  | Shortened LLaMA | 10k | 15h | 24.89 | 52.57 |
|  |  | 20k | 30h | 24.70 | 53.95 |
|  | Ours  | 10k | 30h | **36.00** | **56.96** |
| 50% | FLAP | 10k | 15h | 27.67 | 51.12 |
|  |  | 20k | 30h | 27.55 | 51.32 |
|  | Shortened LLaMA | 10k | 15h | 24.76 | 47.35 |
|  |  | 20k | 30h | 25.10 | 49.22 |
|  | Ours  | 10k | 30h | **30.60** | **52.19** |


**6. The number of remained depth and width in Table 1**

Thank you for the suggestion! The (depth, width scale) for AmoebaLLM with 80%/65%/50% remaining ratios in Table 1 are (30, 0.875)/(28, 0.75)/(22, 0.75), respectively. We will follow your suggestion to add this information to Table 1 to improve readability.

---
**7. The detailed subnet selection strategy**

The profiling results in Section 3 are purely for motivating the problem. Here is our current subnet selection strategy: we adopt a hierarchical search strategy to deliver subnets from our design space that satisfy the target efficiency constraint, e.g., 50% weight remaining ratios in Table 1 of our manuscript, while maximizing the achievable accuracy. Specifically, we first perform a coarse grid search across uniformly spaced depth and width settings based on a small calibration set, e.g., 20 samples from the MMLU dataset, to identify the subnets that satisfy the given efficiency constraint with maximized accuracy. Next, we perform a more fine-grained grid search within depth/width ranges surrounding the optimal subnet identified in the coarse grid search stage. This process typically evaluates 40 promising subnets and takes no more than 10 minutes on an NVIDIA A5000 GPU.

We empirically find that the above strategy works well at the scale of our target problem and can already deliver high-quality subnets that outperform previous compression methods. We also note that more complex subnet selection strategies, such as the evolutionary search adopted by [11]-[15] cited in our manuscript, can also be employed, which will be our future work.

We will clarify this in the final version.

---
**8. The rank of LoRA**

We follow QLoRA [1] and adopt a rank of 64. We will add this information to the final version.

[1] “QLoRA: Efficient Finetuning of Quantized LLMs”, T. Dettmers et al., NeurIPS’23.
**5. Detailed subnet selection strategy**

Thank you for pointing this out! Currently, we adopt a hierarchical search strategy to deliver subnets from our design space that satisfy the target efficiency constraint, e.g., 50% weight remaining ratios in Table 1 of our manuscript, while maximizing the achievable accuracy. Specifically, we first perform a coarse grid search across uniformly spaced depth and width settings based on a small calibration set, e.g., 20 samples from the MMLU dataset, to identify the subnets that satisfy the given efficiency constraint with maximized accuracy. Next, we perform a more fine-grained grid search within depth/width ranges surrounding the optimal subnet identified in the coarse grid search stage. This process typically evaluates 40 promising subnets and takes no more than 10 minutes on an NVIDIA A5000 GPU.

We empirically find that the above strategy works well at the scale of our target problem and can already deliver high-quality subnets that outperform previous compression methods. We also note that more complex subnet selection strategies, such as the evolutionary search adopted by [11]-[15] cited in our manuscript, can also be employed, which will be our future work.

We will follow your suggestion and our promise in the abstract to open source all source code and the delivered subnets upon acceptance.

---
**6. Which LLMs are used**

As mentioned in Section 5.1 of our manuscript, we employ LLaMA 2 7B and Vicuna 7B v1.5.