DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models

Authors: shangqian  gao
chi-heng  lin
samsung  research
yen-chang  hsu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 23:18:56

Abstract: Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, including language modeling, understanding, and generation. However, the increased memory and computational costs associated with these models pose significant challenges for deployment on resource-limited devices. Structural pruning has emerged as a promising solution to reduce the costs of LLMs without requiring post-processing steps. Prior structural pruning methods either follow the dependence of structures at the cost of limiting flexibility, or introduce non-trivial additional parameters by incorporating different projection matrices. In this work, we propose a novel approach that relaxes the constraint imposed by regular structural pruning methods and eliminates the structural dependence along the embedding dimension. Our dimension-independent structural pruning method offers several benefits. Firstly, our method enables different blocks to utilize different subsets of the feature maps. Secondly, by removing structural dependence, we facilitate each block to possess varying widths along its input and output dimensions, thereby significantly enhancing the flexibility of structural pruning. We evaluate our method on various LLMs, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Experimental results demonstrate that our approach outperforms other state-of-the-art methods, showing for the first time that structural pruning can achieve an accuracy similar to semi-structural pruning.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have revolutionized the field of natural language processing by leveraging deep learning techniques to process and generate human-like text. Compared to smaller models, LLMs exhibit unique characteristics and demonstrate remarkable abilities in tackling a wide range of complex tasks [40] . Despite their impressive capabilities, the vast number of parameters in LLMs often hinders their deployment on resource-constrained devices, such as mobile phones. Consequently, there is significant interest in reducing the computational and memory requirements of LLMs.Existing compression techniques for large language models (LLMs) include weight sparsification [9] , structural pruning [30] , and quantization [10] . In this work, we focus on structural pruning and ˚Part of this project was completed at Samsung Research America. Correspondence to sgao@cs.fsu.edu Figure 1 : We use an MLP layer as an example. Left: Regular pruning methods have to follow structural dependence thus their flexibility is limited. Right: Our dimension-independent structural pruning method breaks the structural dependence via index operations and thus largely improves the flexibility for pruning. address the limitations of previous methods in this category. Structural pruning [30] is a generalpurpose compression solution that maintains LLM performance across various tasks, facilitates deployment on devices, and is computationally efficient. However, existing methods may restrict pruning flexibility or add significant overhead to the compressed model. For instance, LLM-Pruner [30] follows structural dependence during pruning, requiring different layers to use the same subset of feature maps, which limits pruning flexibility. SliceGPT [2] alleviates this issue by applying orthogonal projections for each layer but introduces a non-trivial number of additional parameters (e.g., 5% to 13% of the parameters of the original model for LLaMA-2 7B). Our approach aims to overcome these drawbacks and offer a better performance-cost trade-off for structural pruning.We aim to increase the flexibility of current structural pruning methods and consequently improve performance. Our method provides different sub-spaces or subsets of features to different layers, but unlike SliceGPT, it doesn't introduce additional parameters. To achieve this, we break the structural dependence of regular structural pruning methods, allowing different layers to have different subsets of features along the embedding dimension and an example is given in Fig. 1 . After pruning, we employ index selection and index addition operations to sample subsets of features from the residual connection and add them back after the computation of each layer. Furthermore, our method introduces an additional level of flexibility by learning different widths for each layer. Our approach significantly improves the flexibility of structural pruning without adding additional parameters.Extensive experimental results show that our method can outperform state-of-the-art structural pruning methods for LLMs while still maintaining low computational costs. Our method does not require recovery fine-tuning to obtain such performance. In addition, our method does not update the remained model weights during pruning which is a distinct departure from several other methods, such as SparseGPT [9] and LLM Surgeon [37] . Our contributions are as follows:• We break the structural dependence of regular structural pruning methods, significantly increasing the flexibility of structural pruning. This allows different layers to select their own subset of features from the embedding dimension. Importantly, our method achieves this without introducing additional parameters, unlike SliceGPT. • We propose to learn the widths of each layer using gradient-based optimization methods.A hypernetwork generates the column or row selection matrices, while the width of each layer is controlled globally. This approach allows for fine-grained control over the pruning process and enhances the adaptability of our method to various models and tasks. • Our method demonstrates superior performance compared to state-of-the-art structural pruning techniques for LLMs across a range of models, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2. Notably, the resulting model from our method is a sub-network that exists within the original model, indicating the effectiveness of our method in discovering strong sub-networks. 
related works: 
 Magnitude-based pruning is the most straightforward approach to reduce model size, where weights with the smallest magnitude are pruned. Han et al. [14] employ this strategy for pruning with L 1 or L 2 norm of weights. Filter pruning [24] extends this setting to structures of the model instead of performing weight-level sparsification. Although magnitude-based pruning methods are very efficient, they result in significant performance drops for LLMs, even for weight pruning [9] . Another line of research, Optimal Brain Damage [23] and Optimal Brain Surgeon [15] , utilize second-order information to remove connections. These methods require calculating the inverse of the Hessian matrix, which is computationally intensive for modern neural network architectures like Convolutional Neural Networks (CNNs) [22, 16] , Transformers [38] , or Large Language Models (LLMs) [35] . To reduce the cost of computing the Hessian inverse matrix, Optimal Brain Surgeon can be applied in a layer-wise fashion [7, 8] , making the computation tractable. However, further scaling up these methods for LLMs remains challenging. Recent methods like SparseGPT [9] or GPTQ [10] aim to minimize the squared error before and after pruning or quantization of a given layer. In this setting, the Hessian inverse matrix becomes easy to compute, as it is simply the multiplication between the feature map and its transpose for a given layer. GPTQ and SparseGPT then quantize or sparsify model weights in a column-by-column manner, and the unpruned or unquantized weights are updated to compensate for the error of pruning and quantization. Wanda [34] further avoids computing the inverse of the Hessian matrix by only considering the diagonal of the Hessian matrix. While SparseGPT and Wanda achieve good results, unstructured sparsity is known to be harder to achieve actual speedup. They also applied their methods on semi-structured settings [31] , but the performance becomes much worse.Several researches [28, 19, 44, 13, 42, 12] apply learnable parameters for specific structures when pruning vision or language models. However, many of these methods cannot be scaled up to LLMs since they need to learn weights and structures together. In contrast, our method explores sub-networks within the original model without updating model weights. Additionally, our method mainly explores the regime of pruning without recovery fine-tuning, which is rarely presented in previous methods with learnable parameters on structures. Our method is also related to the unconstrained channel pruning for CNNs [39] . However, our method explores this idea from the perspective of breaking structural dependence and scales it to much larger models than [39] . Moreover, our method thoroughly explores the global allocation of parameters, where [39] fails to do.Recently, several works have been proposed to reduce the size of LLMs. LLM-Pruner [30] aims to remove connected structures using importance calculated from Taylor expansions. SliceGPT [2] offers more flexibility than regular pruning by projecting the feature maps to different spaces but introduces extra parameters in the residuals. LLM Surgeon [37] periodically updates model weights and structures, resulting in a higher cost than LLM-Pruner and SliceGPT. Our proposed DISP-LLM breaks the structural dependence relied on by LLM-Pruner, without additional transformation matrices in the residual connections like SliceGPT. Furthermore, in contrast to LLM Surgeon, which requires extensive computational resources, our method is significantly more efficient. 
notations: 
 To better understand our paper, we first define some notations. We use d to denote the model dimension or embedding dimension of LLMs. X P ℜ bˆnˆd is used to represent feature maps, and b is the mini-batch size, n is the number of tokens. W P ℜ d1ˆd2 is the model weights of size d 1 ˆd2 . Let S denote a pseudo-index selection matrix of size d ˆd, which is a diagonal matrix filled with 0 or 1 and the positions of the ones indicate the selected index. We further use Ŝ of size d ˆdsmall to represent the actual selection matrix by removing d ´dsmall columns with all zeros from S. For any matrix A, nnz(A) represents the number of nonzero entries of A.The core idea of SliceGPT [2] is to achieve computational invariance within the transformer architecture. It demonstrates that orthogonal projections can be applied to the output of each block and subsequently undone in the next block. This transformation is computed using Principal Component Analysis (PCA), allowing the feature maps between blocks to be projected into their principal components. A significant advantage of this approach is that it projects the feature maps of different blocks into distinct spaces, thereby introducing an additional degree of freedom for compression. This flexibility is not captured by regular structural pruning methods like LLM-Pruner [30] , which rely on structural dependence.After slicing (pruning), the feature map and weight matrix of lth layer of SliceGPT become:EQUATION)where Ŝ is a d ˆdsmall selection matrix, X l is the output of the lth block, andQ l contains eigenvectors of C l : C l " ÿ i X J l,i X l,iand X l,i is the i-th column of X l (corresponding to the ith sequence in the calibration dataset). From Eq. 1, we can see that SliceGPT uses the same selection matrix Ŝ for all layers, but the feature map X l is firstly projected by Q l , and the pruning for different layers is along with different directions. One crucial drawback of SliceGPT also comes from the projection matrix Q l , since the residual connection must be multiplied by the linear transformation Q J l Q l`1 (shown in Fig. 7 left in the Appendix). These additional operations bring a non-trivial amount of additional parameters. For a model that has L blocks, with the model dimension d and the remaining percentage of parameters p P r0, 1s, it brings approximately Ld 2 p 2 additional parameters to the model (more than 10% of model parameters in some cases, and more details are given in Fig 10 in the Appendix).SliceGPT offers significant flexibility, but achieving similar flexibility with regular structural pruning methods without adding extra parameters is challenging. This section explains the reasons behind this difficulty. To simplify our reasoning, we replace Ŝ with its pseudo selection matrix S.Assume we follow the basic setting of dependence-based structural pruning but allow each layer the flexibility to have its own selection matrix, S l , along the embedding dimension. Under this assumption, due to structural dependence, all layers will share the same width of nnzpS 0 S 1 ¨¨¨S L q.In order to prune different positions for different layers, we need to add a transformation matrix to align the width of layers l and l `1. Intuitively, if we have S l and S l`1 , we can then insert S J l S l`1 in the residual connection to align consecutive layers.Algorithm 1: Block inference after pruning. Input: Feature map of the previous block X in . Preserved indices sets Ind 1 , Ind 2 , Ind 3 , Ind 5 . 1. Xin " LayerNormpX in r:, Ind 1 sq.Ź Index Selection for Attention 2.X att " MultiHeadp Xin ŜJ 1 W q , Xin ŜJ 1 W k , Xin ŜJ 1 W v qW o Ŝ2 . 3. X res " Index_AddpX in , X attn , Ind 2 q.Ź Index Addition with the input 4. Xres " LayerNormpX res r:, Ind 3 sq.Ź Index selection for MLP 5. X mlp " pσp Xres ŜJ 3 W 1 Ŝ4 q d p Xres ŜJ 3 W 2 Ŝ4 qq Ŝ4 J W 3 Ŝ5 . 6. X out " Index_AddpX res , X mlp , Ind 5 q Ź Index Addition with the residual Return X out for the next block.With this setup, we can use X l S l to select subsets of features for different layers, mimicking Q l S for SliceGPT. Although it seems promising, this formulation has issues with layer widths, as detailed in Proposition 1.Proposition 1 (Decreasing feature dimensions for deeper layers). Let the pseudo-selection matrices in layers l and l `1 be S l and S l`1 , respectively. The number of nonzero entries in the residual adapter satisfies nnzpS J l S l`1 q ď mintnnzpS l q, nnzpS l`1 qu. For compression strategies that remove dependent structures for layer l `1 following S J l S l`1 , this implies that the dimension in layer l `1 is less than or equal to that in layer l, with equality holding when the feature indices selected in layer l `1 are contained within those in layer l or vice versa.Remark. The proof of Proposition. 1 is straightforward and it is given in the Appendix A.1. From Proposition 1, we observe that if we naively apply S l for different layers, the model width will progressively decrease as we go deeper into the network. It also fails to provide different sets of features for different layers; instead, it merely passes a subset of features from the previous layer to the next. To avoid this restriction, all blocks must share the same width and the same pruned columns or rows. And we then fall back to the regime of previous structural pruning methods such as LLM-Pruner [30] , Shared LLaMA [43] , etc. Proposition 1 highlights two significant obstacles. First, dependence-based structural pruning methods result in a uniform width along the embedding dimension. Second, inserting selection matrices in the residual connections causes the embedding dimension to decrease with depth. These challenges are unavoidable due to the residual connections linking structures across layers. To enhance flexibility along the embedding dimension, bypassing the residual connections is crucial.4 Dimension-Independent Large Language Model 4.1 Break the Structural dependence Section 3.3 demonstrates that the residual connection is the primary barrier preventing pruning methods from achieving better flexibility. To avoid modifying the residual connection, we relocate the selection matrices inside the residual connection. This approach allows us to successfully create different subsets from the feature maps for different layers.Based on this idea, we propose a solution that involves pruning different positions in consecutive blocks and selecting or merging feature maps from or back to the residual connection. This approach breaks the structural dependence inherent in previous pruning methods. Formally, given a transformer block, we apply the following operations:EQUATIONMLPpXqEQUATIONwhere S 1 , . . . , S 5 are pseudo selection matrices of size d ˆd, and d denotes element-wise multiplication. Eq. 3 gives an example operation for gated MLP modules used in LLaMA [35] . For Phi models [1] or OPT [46] , the MLP operation is defined as MLPpXq " σpXS J 3 W 1 S 4 qS 4 J W 3 S 5 . Given the operations defined in Eq. 2 and Eq. 3, we successfully remove the constraint in Proposition 1. The input and output of both the Attention layer and the MLP layer can be selected differently from the original feature maps for different layers, mimicking the function of Q l in SliceGPT. Additionally, our method eliminates the need for extra parameters in Q J l Q l`1 as it does not alter the residual connection. We also enhance flexibility by pruning the middle dimension of the MLP layer. Additionally, this flexibility can be further improved by allowing the query, key, and value weight matrices to use different selection matrices. Our current form is kept for two reasons: (1) SliceGPT uses one Q l per layer, and we followed this design for a fair comparison, and (2) adding separate selection matrices would increase indexing operations, potentially slowing down the inference. Once we have the final selection matrices S 1 , . . . , S 5 , the pruned model will use a combination of index selection and index addition for inference as shown in Algorithm 1, where Ind i is a set containing all indices equal to one in the diagonal of S i :Ind i " tj | if s irjs " 1u, s i " diagpS i q.The same color is used to mark the index set Ind i and its corresponding selection matrix Ŝi . Index_AddpA, B, Indq adds matrices A and B along the last dimension on selected positions from Ind, then returns A after index addition. With index selection and addition, the block dimension can be freely changed. Index selection and addition introduce some overhead, but as demonstrated in the experiment section, we still observe improvements in throughput. 
learning the width for dimension-independent llms: 
 Building on the dimension-independent setting introduced in Section 4.1, our approach offers much greater flexibility in selecting sub-networks from the original dense model compared to the constrained settings in LLM-Pruner [30] . The next challenge is determining the width of each layer. Given the large search space of our dimension-independent structural pruning and the computationally intensive nature of LLMs, it is impractical to use reinforcement learning [17] or evolutionary search-based algorithms [27] . Therefore, we adopt gradient-based methods to address this challenge. Given the diagonal vector s i P t0, 1u d from S i , the Straight-Through (ST) gradient estimator [3] is used to estimate the gradients with respect to learnable continuous latent parameters. More specifically, we use the recently proposed gradient estimator ReinMax [26] to estimate the gradients through the binary operation. A detailed explanation of ReinMax for the binary case is provided in Appendix A.2.Given the large search space of our method, we find that only using element-wise learnable parameters is insufficient. To address this issue, a hypernetwork is introduced to generate latent parameters for ReinMax, as detailed below:EQUATIONwhere Θ represents the parameters of the hypernetwork and s contains s i from all blocks. The hypernetwork is composed of GRU [5] and fully connected layers, where the GRU captures blockwise relationships and the fully connected layers capture relationships between different dimensions.With the hypernetwork and ReinMax, we can effectively learn the width of each block. The details of the hypernetwork are provided in Appendix A.3.With the methods described above, we can formulate dimension-independent structural pruning as an optimization problem, with regularization to control the number of remaining parameters. We insert s back into S as defined in section 4.1 for forward and backward calculations. The overall objective function is listed below:EQUATIONRpT psq, pT total q " logpmaxpT psq, pT total q{ minpT psq, pT total qq,where L is the language modeling loss function of next word prediction, X represents the input tokens, W is the collection of model weights, s is defined in Eq. 4, and R is a parameter regularization loss function defined in Eq. 6. Here, T psq denotes the number of parameters controlled by the current structure s, T total is the total number of parameters of the model, and p P p0, 1s is a user-defined parameter to control how many parameters should be preserved within the model. With the objective function in Eq. 5, the structures for dimension-independent pruning can be efficiently optimized. Moreover, the overhead of our method is minimal and comparable to parameter-efficient fine-tuning methods like LoRA [18] , as it does not require storing gradients or the first and second-order momentum of model weights for the Adam optimizer [21] . 
settings: 
 Models. We evaluate our DISP-LLM method using several LLMs with transformer blocks. Specifically, we choose the following models: OPT [46] : OPT-125M, OPT-1.3B, OPT-2.7B, OPT-6.7B; Phi-1.5 [25] and Phi-2 [20] ; LLaMA 7B [35] ; LLaMA-2 [36] : LLaMA-2 7B and LLaMA-2 13B.Implementations. We implemented our method using Pytorch [32] and Hugging Face transformer library [41] . We freeze the model weights W when training the hypernetwork. We use the AdamW [29] optimizer to optimize the hypernetwork. The hypernetwork is trained for 10,000 iterations for all models. For all experiments, we set λ in Obj. 5 to 6. Depending on the size of the base model, we use 1 to 4 NVIDIA A100 GPUs to train the hypernetwork. More implementation details can be found in the Appendix A.4.Datasets. Following previous papers [2, 30] , we use WikiText-2 and Alpaca datasets to train the hypernetwork. Following SliceGPT [2] , we evaluate our method and other methods on five wellknown zero-shot tasks: PIQA [4] ; WinoGrande [33] ; HellaSwag [45] ; ARC-e and ARC-c [6] . We use llm-eval-harness [11] to evaluate the compressed models.Baselines. We compare our DISP-LLM across baselines from structural pruning like LLM-Pruner [30] , SliceGPT [2] and LLMSurgeon [37] . We also include semi-structure pruning baselines like SparseGPT [9] and Wanda [34] .In Table 1 , we report the perplexity of pruned OPT and LLaMA-2 models. Our DISP-LLM, which does not update weights, consistently outperforms more complex pruning methods such as K-OBD and LLM Surgeon, which involve weight updates, across all pruning ratios and models. The performance gap is even larger when compared to SliceGPT. The advantage is particularly clear in better-trained models like LLaMA-2 7B and 13B. For instance, our method surpasses LLM Surgeon by margins of 5.54 and 2.22 when pruning 50% of parameters of LLaMA-2 7B and 13B, respectively. Against K-OBD, our performance advantage extends to 36.80 and 9.49 under the same setting. For consistency, we let the pruning ratio of SliceGPT equal the slicing ratio. However, the actual pruning ratio for SliceGPT is much lower than the slicing ratio. More details are given in Appendix A.5.In Table 2 , we report the perplexity of pruned LLaMA and LLaMA-2 models and we compare our method with semi-structure pruning methods. From the table, we can see that our method outperforms both SparseGPT and Wanda on LLaMA 13B and LLaMA-2 7B/13B models. Our method performs on par with SparseGPT and Wanda with the LLaMA 7B model, and our DISP-LLM is a little bit worse than SparseGPT and is similar to Wanda. We are the first to show that structural pruning methods can have a better or similar performance than semi-structural pruning methods.In Tab. 3, we present the zero-shot performance of the pruned model. For the LLaMA 7B model, we compare our method against LLM-Pruner with and without recovery fine-tuning. Our method consistently outperforms LLM-Pruner without fine-tuning, and the gap ranges from 2.63 to 8.88 across different pruning rates for average task performance. After fine-tuning, the performance of LLM-Pruner is largely boosted, however, our method is still able to outperform it demonstrating the existence of strong sub-networks within the original model. For the LLaMA-2 7B model, we compare our method against SliceGPT, K-OBD, and LLM Surgeon. With weight updates, LLM Surgeon performs well with a lower pruning ratio like 30%. At this pruning ratio, our method performs similarly to LLM Surgeon, and our method outperforms other comparison baselines. When increasing the pruning ratio to 50%, the advantage of our method becomes obvious, and the gap between our method and LLM Surgeon is 5.37 for average task performance. We further compare our method with SliceGPT on Phi-1.5 and Phi-2, and our method consistently achieves better performance.Ablation Study. We visualize the results of ablation studies in Figs. 5a, 5b, 5c, 5d with Phi-1.5 model. The setting "DISP-LLM w/o HN" refers to using element-wise gates for learning subnetworks. The setting "Constrained LLM w HN" refers to pruning the embedding dimension following the structural dependence like LLM-Pruner. From Figs. 5a, 5b, we can see that using the hypernetwork largely accelerates the learning process for DISP-LLM, which is also verified in Figs. 5c, 5d . From Figs. 5c, 5d , we also see that our DISP-LLM always outperforms constrained structural pruning, demonstrating the value of added flexibility by breaking the dependence. To further study the impact of the HyperNetwork architecture, we provide more results in Tab. 4. "w/o HN" is equivalent to "DISP-LLM w/o HN". The setting "w/o Bi-GRU" simply removes GRU and adds a fixed input (initialized in the same way as see Appendix A.3 for more details) for each linear layer. These results indicate that both GRU and linear layers within the HyperNetwork affect the final performance. One explanation is that linear layers connect different dimensions of the model, accelerating learning, while GRU layers capture inter-layer relationships, further reducing the difficulty of learning sub-network structures. Therefore, both GRU and linear layers are essential to the HyperNetwork.Different Pruning Ratios, Costs and Throughput. In Figs. 5e, 5f, we show the language modeling loss L and regularization loss R in Obj 5 given different pruning ratios p with Phi-1.5 model. We can see that our method consistently minimizes the language modeling loss across different p. In addition, our method quickly pushes the regularization loss R to near 0 values within 200 iterations. In Fig. 5g , the pruned model from LLaMA-13B improves the throughput of the dense model by 1.08ˆto 1.50ˆ.In Fig. 5h , we compare the costs of our method against LLM Surgeon. Our method is 27.39ˆand 14.76ˆcheaper compared to LLM Surgeon with LLaMA-2 7B and LLaMA-2 13B models.Every Embedding Dimension is Important. In Fig. 4 , we visualize the pruning decisions along the embedding dimension and depth for the LLaMA-2 7B model, we can see that all embedding dimensions have been used across different layers. This becomes more obvious in the second right figure of Fig. 4 , where we sum all pruning decisions along the depth dimension, and we can see that every embedding dimension is kept around 80% given all layers. We further visualize the model width after pruning for the LLaMA-2 7B model in Fig. 6 , where we can see that several layers are more severely pruned than other layers.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Due to the company policy, the code will only be released after going through the internal review process.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we proposed dimension-independent structural pruning for Large Language Models. By breaking the structural dependence imposed by previous compression methods, our method creates sub-networks with a lot more flexibility than regular structural pruning methods and also avoids the overhead brought by SliceGPT. The flexibility of our method is reflected in two perspectives. Firstly, our method provides different subsets of the feature maps for different layers. Secondly, our method freely selects the width for each layer without considering architecture dependence. With dramatically increased flexibility, our method outperforms other structural pruning and semi-structural pruning methods given similar pruning ratios. The novel design of the unconstrained pruning space along with strong empirical performance opens new possibilities for structural pruning for LLMs. Output: x: binary vector. 1. π 0 " sigmoidpx `cq, 2. B " sample_binarypπ 0 q, 3. π 1 " B`sigmoidppx`cq{τ q 2 , 4. π 1 " sigmoidpstop_gradientplnpπ 1 q ´px `cqq `px `cqq, 5. π 2 " 2π 1 ´1 2 π 0 , 6. x " π 2 ´stop_gradientpπ 2 q `B Return x.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The details are provided in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow the settings of previous methods where the result of a single run is provided.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Provided in the Appendix. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: adapter satisfies nnzpS J l S l`1 q ď mintnnzpS l q, nnzpS l`1 qu. For compression strategies that remove dependent structures for layer l `1 following S J l S l`1 , this implies that the dimension in layer l `1 is less than or equal to that in layer l, with equality holding when the feature indices selected in layer l `1 are contained within those in layer l or vice versa.Proof. Consider the pseudo-selection matrices S l and S l`1 , both of size d ˆd, and both diagonal matrices. The number of nonzero entries in S l and S l`1 are given by nnzpS l q " k l and nnzpS l`1 q " k l`1 , respectively.The product S J l S l`1 is also a diagonal matrix of size d ˆd. Each diagonal entry pi, iq in S J l S l`1 is the product of the i-th diagonal entry of S l and the i-th diagonal entry of S l`1 . For an entry pi, iq to be nonzero, both S l pi, iq and S l`1 pi, iq must be nonzero.Thus, the number of nonzero entries in S J l S l`1 , nnzpS J l S l`1 q, is the number of indices i where both S l pi, iq and S l`1 pi, iq are nonzero. This count cannot exceed the smaller of the total number of nonzero entries in S l and S l`1 . Hence, nnzpS J l S l`1 q ď mintnnzpS l q, nnzpS l`1 qu.This implies that the effective feature dimension will be smaller or equal to the previous layer. Equality holds if and only if the set of indices corresponding to nonzero entries in S l`1 is a subset of those in S l , or vice versa. This concludes the proof.Figure 10 : Expected compression rate vs. actual compression rate of our method and Slice-GPT on the LLaMA-7B model.In Proposition 1, "remove dependent structures for layer l `1 following S J l S l`1 " means that the actual selection matrix for layer l `1 becomes S 1 l`1 " S J l S l`1 , and the structure dependence is cut off by the next residual connection. The pruning for layer l `1 will based on S 1 l`1 instead of S l`1 . Although this setting partially breaks the structural dependence, it has the limitation that the embedding dimensions will be reduced when going deeper.In this section, we provide details for ReinMax when handling binary variables. The ReinMax in our work can be written as shown in Algorithm. 2. We add a constant bias c to x so that we can control binary vectors to have all one value at the beginning when learning the sub-network architecture for DISP-LLMs. Through all experiments, we set c to 3.0 and τ to 1.0. As we discussed in the paper, the Hypernetwork is composed of linear layers and Bi-GRUs, and now we present the architecture of the HN in Tab. 5. z is initially sampled from a normal distribution, and it is then fixed during training. Outputs s l are continuous values and it is then fed to ReinMax to produce the binary vector: s " ReinMaxpsq, where s is the collection of s l from all layers. N l is the original model width, and it equals the embedding dimension for S 1 , S 2 , S 3 and S 5 . We train the hypernetwork for different models, we always set the mini-batchsize to 1 on each GPU. For OPT 6.7B, LLaMA 7B, and LLaMA-2 7B models, we use 2 NVIDIA A100 GPUs, and for LLaMA 13B and LLaMA-2 13B models, we use 4 NVIDIA A100 GPUs. For all the rest models, we use 1 NVIDIA A100 GPU. We set p " t0.5, 0.4, 0.3, 0.2, 0.1u when the pruning ratios equals to t50%, 40%, 30%, 20%, 10%u. For the Alpaca dataset2 , we use the 'text' column within the dataset which combines the columns of 'instruction' and 'output'. When training the hypernetwork, we again minimize the language modeling loss on the Alpaca dataset instead of applying the training process of instruction fine-tuning.Details of Eq. 6. The parameter regularization loss function in Eq. 6 is defined as follows:Rpx, yq " log ˆmaxpx, yq minpx, yq˙" $ ' & ' % log ´x y ¯if x ą y, 0 if x " y, log `y x ˘if x ă y .Since y is fixed, when x ą y, Eq. 6 will decrease x, making it closer to y. Conversely, when x ă y, Eq. 6 will increase x, also making it closer to y. Thus, the parameter regularization loss always tries to push the current sub-network to achieve the pre-defined parameter budget. Ablation Study Settings. In the ablation study 5.4, we removed the hyperntwork, and we revise Eq. 4:s " ReinMaxpΘq,where Θ now has the same size of s, and the parametrization space becomes much smaller. For the constrained setting used in the ablation study 5.4, we simply let S 1 " S 2 " S 3 " S 5 . In section 5.4, we calculate the costs of our method and LLM-Surgeon, and the price comes from the official website of Lambda Cloud3 . We also list the detailed time costs of our method in Tab. 6. In Fig. 5b , 5f and also in Fig. 11 , we normalized the parameter regularization loss R with its maximum value to make plots more consistent. SliceGPT compression rates: In Fig. 10 , we show the expected compression rate and the actual compression rate for our method and SliceGPT given the LLaMA-2 7B model. It can be seen that SliceGPT adds 5% to 13% parameters of the original model across different pruning rates, which is non-trivial for most LLMs. Notably, SliceGPT with 10% slicing actually adds 3% more parameters to the original model.LLaMA-2 13B Results. In Tab. 7, we show the results of the LLaMA-2 13B model given different pruning rates. From the table, we can see that our method consistently outperforms LLM Surgeon under different pruning rates. The advantage of our method becomes more obvious compared to other methods like K-OBD and SliceGPT.Phi-2 Results. In Tab. 8, we present a more comprehensive comparison of our method compared to SliceGPT. Our method outperforms SliceGPT by 5.28 to 7.01 giving SliceGPT with or without fine-tuning on the WikiText dataset. These observations again demonstrate the effectiveness of our method, and our method outperforms methods with recovery fine-tuning in several settings.LLaMA-2 3B Results. In Table 9 , we present a comparison of our method against LLM Pruner and magnitude pruning. At a pruning ratio of 20%, our method surpasses the performance of LLM Pruner both with and without fine-tuning. Remarkably, even when the pruning ratio is increased to 50%, our method continues to outperform the LLM Pruner Channel and the Magnitude pruning baselines at a 20% pruning rate. These results further illustrate our method's ability to identify strong sub-networks within the original dense model.Architecture of the pruned LLaMA-2 13B. In Fig. 8 and Fig. 9 , we visualize the pruned architecture of the LLaMA-2 13B model pruned with the WikiText dataset. We have similar observations as in section. 5.4. In Fig. 9 , we can see that middle to late layers have large pruning rates, especially for the attention layer. In Fig. 8 and Fig. 12 , we can also see that the preserved rates for different dimensions are similar, and all embedding dimensions are effectively utilized. More interestingly, similar pruning rates across different dimensions are achieved without adding any regularization or constraints.Training loss for LLaMA-2 7B. In Fig. 11 , we further visualize the training dynamics of the LLaMA-2 7B model on WikiText and Alpaca datasets, respectively. The regularization loss with the Alpaca dataset decreases a little bit faster than the WikiText dataset, probably because the loss value on the Alpaca dataset is smaller. From Fig. 11c , we can also see that the language modeling loss continues to decrease when training longer, especially when the pruning ratio is higher.Lastly, we evaluate our method on the Phi-1.5 model with 5 runs, and we report our result with mean and standard deviation in Tab. 10. We also measure the throughput of our method on the LLaMA-2 13B model, and the result is shown in Tab. 11 Impact of λ on Phi-1.5 when pruning 50% of parameters. We show the result in Tab. 12. 'NC' means the loss R does not converge, and it is much larger than zero, thus it can not prune the model to the target budget. From the table, we can see that the PPL of the model becomes quite stable if it is larger or equal to 6. If λ is not large enough, it takes longer to push the loss R to reach near 0 values and thus leaves less time for the model to explore different configurations of subnetworks. On the other hand, if λ is large enough, the loss R will reach zero in several hundred iterations and leave enough time to find the desirable subnetwork. Due to this reason, our method is quite stable across larger values of λ as shown in the table. More results on pruning ratio vs. performance trade-offs. We provide the trade-off between the pruning ratio and performance for the Phi-2 and LLaMA-2 7B model below in Tab. 13 and Tab. 14.LoRA fine-tuning [18] of the compressed LLaMA-2 7B model. We follow similar settings of the SliceGPT and the model is fine-tuned on Alpaca. The result is shown in Table . 15. We can see that our method can be further boosted by using parameter-efficient fine-tuning techniques. Since the performance without fine-tuning is already good enough, we prefer not to involve this additional process in our method to save time and computational costs.We show the generated text given DISP-LLM and SliceGPT in Tab. 16 . The examples are obtained based on removing 20% of the model weights with DISP-LLM and 20% slicing of SliceGPT ( 10% compression rate). Both models are compressed from LLaMA-2 7B and they are not finetuned. From these two examples, we can see that SliceGPT only generates a small part of meaningful content and then starts repeating itself. On the other hand, our method tends to generate more relevant content, and the self-repeating behavior is much less obvious. These observations comply with the performance gap between DISP-LLM and SliceGPT.Our method explores how to break the structural dependency of pruning for LLMs. Although our method achieves competitive performance, there are still some limitations that are not solved in our current version. The throughput improvements of our method are not consistent across different models, which is probably because the current implementation of index add or index select operations with PyTorch is not efficient enough. An alternative implementation of our method is to select weight matrices instead of feature maps. In this approach, we perform matrix multiplication with the full feature map and fill the weight matrices with zeros. If we have a custom matrix multiplication implementation that ignores zero rows or columns, it may achieve further speed-up. However, this approach primarily focuses on the engineering perspective and is beyond the scope of this work. We leave the exploration of this alternative implementation to future research.Our dimension-independent structural pruning method significantly reduces the computational and memory requirements of LLMs, enabling their deployment on resource-limited devices and lowering energy consumption, thus contributing to environmental sustainability. By facilitating the use of efficient LLMs on affordable hardware, our approach democratizes access to advanced AI technologies. However, it is crucial to address potential ethical concerns when reducing the size of LLMs, ensuring privacy and fairness in AI applications with compressed LLMs. Overall, our method fosters a more accessible and sustainable use of AI.Table 16 : Generation samples of LLaMA-2 7B with 20% compression rate SliceGPT AI is a very important part of the game. The game is played in a 3D environment, the player controls a character that can move freely around the environment. The player can interact with the environment by picking up items, using them to solve puzzles. The player can also interact with other characters in the game. The player can also interact with the environment by picking up items and using them to solve puzzles. The player can also interact with other characters in the game. The player can also interact with the environment by picking up items and using them to solve puzzles. The player can also interact with other characters in the game. The player can also interact with the environment by picking up items and using them to solve puzzles. The player can also interact with other characters in the game. The player can also interact with the environment by picking up items and using them to solve puzzles. The player can also interact with The room-temperature superconductor is a material that exhibits superconductivity at room temperature.(100 °C). The first room temperature superconductor was discovered in 1970 by Peter C. Edwards and his colleagues at the University of Michigan. The material was a mixture of bismuth and tin, and was named " bismuth-tin."The discovery of bismuth tin was followed by the discovery of other room temperature superconductors, including bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin, bismuth tin DISP-LLM AI is a branch of computer science that seeks to create intelligent machines that can think and learn like humans.The field was developed in the 1950s and 1960s by researchers who focused on making machines that could learn and solve problems like people do. The field was commercially successful in the 1980s and 1990s with the development of the first generation of intelligent agents. The term " artificial intelligence " was coined by John McCarthy in 1956. He was inspired by the idea of creating a machine that could learn and solve problems like humans do.The room-temperature superconductor is a class of superconductors that exhibit zero resistance at room temperature. ## History The room-temperature superconductor was discovered in 1986 by the Japanese scientist K. Masamichi Aoki and his colleagues at the University of Tokyo. The discovery was made possible by the use of a new technique called " zero temperature transport measurement " ( ZTM ), which allowed them to measure the resistance of the superconductor at temperatures as low as 0.05 K. The discovery was made possible by the use of a new technique called " zero temperature transport measurement " ( ZTM ), which allowed them to measure the resistance of the superconductor at temperatures as low as 0.05 K. ## Discovery The discovery of the room-temperature superconductor was made possible by the use of a new technique called " zero temperature transport measurement " ( ZTM ),The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:• You should answer [Yes] , [No] , or [NA] .• [NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.• Please provide a short (1-2 sentence) justification right after your answer (even for NA).The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.https://huggingface.co/datasets/tatsu-lab/alpacahttps://lambdalabs.com/service/gpu-cloud#pricing 


Decision: This paper introduces an innovative method of structural pruning that does not rely on structural dependence among embedding dimensions. The dimension independency nature of this method can significantly increase the flexibility of structural pruning. Experimental results demonstrate its effectives on various open-source LLMs by improving upon state-of-the-art methods. Overall, this is a solid work, and its contribution is appreciated by reviewers. The rebuttal addressed the reviewer concern by clarifying that this approach requires lightweight training and hyper-parameter selection. The consensus among reviewers emerged through rebuttal and discussion. Authors are encouraged to include these additional results and discussions into the final version.

Reviews: review: paper_summary: paper_summary: Structural pruning is a method to prune the weights of large language models (LLMs) while keeping their original performance as much as possible it can. However, structural pruning has limitations caused by depending on the structure, like the residual connection of LLMs. In this work, the authors propose a new pruning method, dimension-independent structural pruning for LLMs (DISP-LLM), that can ignore structural dependence in pruning. This characteristic of DISP-LLM makes it possible to prune their weights more flexibly and provides new pruning ways using different feature maps between blocks and varying pruning dimensions of the input and output of the block. However, just pruning the dimensions and ignoring residual connections causes inconsistency between the input and output of a block. To solve this problem, the authors apply transformation matrices to each residual connection to absorb these discrepancies. Furthermore, pruned indices are optimized through training. Experimental results on the language modeling dataset WikiText-2 and the zero-shot task datasets PIQA, WinoGrande, HellaSwag, ARC-e, and ARC-c show that DISP-LLM tends to outperform conventional approaches in pruning LLMs, OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.
strength_weakness: strength_weakness: - Even though DISP-LLM does not update the original weights of LLMs, training is required.
- DISP-LLM requires hyper-tuning for $\lambda$.
questions: questions: - How did you decide to set $\lambda$ to 6?
- How are the performances sensible to the changes of $\lambda$?
- As you state in the paper, DISP-LLM is as efficient as LoRA. If so, can applying LoRA weights only to the kept dimensions be a baseline?
limitations: limitations: DISP-LLM requires training and hyper-parameter tuning for $\lambda$ that controls mixing ratios of losses.
paper_summary: paper_summary: This paper proposes a novel structural pruning method for LLMs, primarily based on SliceGPT. It characterised in (i) removing structural dependence by facilitating each block to possess varying widths along its input and output dimensions (ii) no need for introducing addition paramaters like sliceGPT.  The empirical studies show that it demonstrates superior performance compared to state-of-the-art structural pruning techniques for LLMs across a range of models, including OPT, LLaMA, LLaMA-2, Phi-1.5, and Phi-2.
strength_weakness: strength_weakness: 1.  More comprehensive literature review about model compressing, like sparse mechanism.
2. Any case study for qualitative analysis to the pruned model? It is possible to conduct some interpretability analysis showing where the difference between sliceGPT and the proposed method along with a specific input example.
questions: questions: 1.  When training with the pruning objectives all the layers are fine-tuned or only selected layers?
2. Any insights from Figure 6, different model width as layer changes. If the observations aligned with existing research results?
limitations: limitations: Pay attention to the generated sequence after model prune because the prediction changes are unpredictable.
paper_summary: paper_summary: DISP-LLM introduces dimension-independent structural pruning for LLMs, breaking structural dependencies to allow flexible pruning across layers and dimensions. It uses a hypernetwork for pruning decisions and claims superior performance over existing methods on various LLMs without weight updates or additional parameters.
strength_weakness: strength_weakness: - Limited novelty beyond combining existing concepts
- Inconsistent performance gains across models/tasks
- Insufficient ablation studies and hyperparameter sensitivity analysis
- Lack of detailed computational efficiency comparisons
- Inadequate error analysis and failure case exploration
- Overstated claims based on limited experiments
- Insufficient discussion of pruning ratio vs. performance trade-offs
questions: questions: - How sensitive is DISP-LLM to hypernetwork architecture and hyperparameters?
- Can you provide detailed computational cost comparisons with baselines?
- How does it perform on fine-tuning tasks?
- Have you explored progressive pruning approaches?
limitations: limitations: Yes
paper_summary: paper_summary: This paper concerns about the dimension dependence of hidden states across different layers. It have been argued that previous studies with dimension dependence would be inefficient due to additional parameters or architectural constraints. This work breaks the dimension dependence by introducing indexing operations to previous methods. Although simple, the proposed method is shown to be more effective than baselines due to permitted asymmetry across layers.
strength_weakness: strength_weakness: 1. Although the proposed method allows asymmetric dimensions in module weights across layers, it also requires the dimensions of hidden states from different layers maintain the same as that of the non-pruned model. This would restrict the use in scenarios where the dimension of the output hidden states is critically important (e.g., embedding for retrieval).
2. The introduced indexing operation could be very time-consuming. Unexpectedly, the latency introduced by the indexing may outweight the latency reduced by pruning. And it is not discussed whether the pruned model is faster than the non-pruned one. I would recommend including such experiments in proper positions.
questions: questions: N/A
limitations: limitations: Yes

Comments: title: Regarding LoRA
comment: Dear Reviewer BGu5,

Thanks again for helping review this paper! Since we are approaching the end of the author-reviewer discussion period, would you please check this author response regarding your concerns? We really appreciate it!

Best, AC
Dear Reviewer AWnP,

Thanks again for helping review this paper! Since we are approaching the end of the author-reviewer discussion period, would you please check this author response regarding your concerns? We really appreciate it!

Best, AC
We want to thank Reivewer FhgQ for giving us the chance for further clarification. We list the comparison of our method, quantization, and QLoRA in the table below:

 Methods | W Update? | Reduce Weight Precision | Reduce #Parameters | Goal
---|---|---|---|---
 DISP-LLM | No | No | Yes | Compression
 Quantization | Depends | Yes | No | Compression
 QLoRA | Yes | Yes | No | Fine-tuning

As a pruning method, DISP-LLM is orthogonal to Quantization or QLoRA methods as shown in the table, meaning it can be combined with quantization to further reduce model size. Firstly, we can combine our method and Quantization by applying DISP-LLM on top of a quantized model to reduce the training costs as stated in the response to **W1**. Additionally, once a model has been compressed using DISP-LLM, QLoRA can be employed to fine-tune the model with minimal resources. Besides the aforementioned contents, we believe that the search space in the dimensional independent setting offers new insights for designing a compact large language model that is not captured by QLoRA or quantization.
Thank you for answering my questions! The response has resolved some of my concerns. Regarding the use of LoRA, one of the frequently used approaches is QLoRA [1], which uses LoRA on quantized frozen weights. Even though QLoRA does not prune the model weight, it accomplishes producing lightweight models with fine-tuned information. Therefore, considering the quantization in the comparison is along with the realistic situation.
As seen in the general response, the author seems to notice the necessity of investigating the use of quantization in the proposed method. Currently, this direction has not been decided in the paper. However, if you can clarify the difference between the proposed method, quantization, and QLoRA in terms of their benefits and still claim the superiority of your approach to other methods, I can raise my score.

[1] Dettmers, Tim, et al. "Qlora: Efficient finetuning of quantized llms." Advances in Neural Information Processing Systems 36 (2024).
We would like to thank Reviewer 6CTn for providing the reference, which is an excellent addition to explaining the pruned structure. It is possible that the retained capacity is related to the monosemanticity of the corresponding layer. We will add the related discussion to our final version to provide further insights regarding the pruned structure.
Thank you for your clarification and the case study you provided—it was very helpful! 
I find the observations in Figure 6 particularly interesting, especially how both the earlier and later layers are changing significantly. A similar study referenced in [1] also presents insights on layer-wise monosemanticity, which I hope might offer some additional inspiration.

Encourage or Inhibit Monosemanticity? Revisit Monosemanticity from a Feature Decorrelation Perspective