LEMON: Reviving Stronger and Smaller LMs from Larger LMs with Linear Parameter Fusion (2024)

Abstract: In the new era of language models, small models (with billions of parameter sizes) are receiving increasing attention due to their flexibility and cost-effectiveness in deployment. However, limited by the model size, the performance of small models trained from scratch may often be unsatisfactory. Learning a stronger and smaller model with the help of larger models is an intuitive idea. Inspired by the observing modular structures in preliminary analysis, we propose LEMON to learn competent initial points for smaller models by fusing parameters from larger models, thereby laying a solid foundation for subsequent training. Specifically, the parameter fusion process involves two operators for layer and dimension, respectively, and we also introduce controllable receptive fields to model the prior parameter characteristics. In this way, the larger model could be transformed into any specific smaller scale and architecture. Starting from LLaMA 2-7B, we revive two stronger and smaller models with 1.3B and 2.7B. Experimental results demonstrate that the fusion-based method exhibits flexibility and outperforms a series of competitive baselines in terms of both effectiveness and efficiency.

heading: 1 Introduction
text: Transformer-based language models (LMs) shine in various natural language tasks due to their powerful understanding and generation capabilities (Anthropic, 2023; OpenAI, 2023; Touvron et al., 2023). Considering that the training and deployment of large-scale LMs require a large amount of computing resources, small LMs are more cost-effective in actual production environments (Gunasekar et al., 2023; Li et al., 2023). However, in the pre-training stage, it has been proven that there is a certain scaling law between model size and performance (Kaplan et al., 2020; Hoffmann et al., 2022). How to
†Corresponding author. ‡ Project lead.
break through this limitation to train a stronger and smaller model with the help of large models has recently attracted the attention of researchers (Ma et al., 2023; Xia et al., 2023; Xu et al., 2023).
Inspired by the idea of function-reserving transformation in efficient training (Chen et al., 2015), there are some model compression efforts that utilize layer dropping, structured pruning, or weight selection to preserve the parameter functionality of large models as much as possible, so as to accelerate pre-training and improve the final performance of small models (Zhang and He, 2020a; Xia et al., 2022; Xu et al., 2023). However, since there are tens or even hundreds of billions of parameters in modern LMs, searching for the essential parameter combinations requires significant computing resources, while pruning inappropriate parameters inevitably leads to performance degradation in downstream tasks (Frantar and Alistarh, 2023). In this case, how to construct an appropriate initial point with capabilities close to large LMs at the lowest possible cost remains an open problem.
Given the limited understanding of parameter characteristics and roles in modern large LMs, we perform an empirical analysis from the perspectives
8005
of representation similarity, and observe that there are some modular structures with high internal similarity. In particular, there is a high similarity in the parameters of adjacent layers, while the similarity of distant layers to a lesser extent, which coincides with the sparsity found in previous studies (Qin et al., 2022; Zhang et al., 2023). Nonetheless, for a parameter module, each part has its own unique role, and thus no model parameter should be arbitrarily discarded. Based on the observation, we believe that the functionality of a module should be reconstructed by selectively fusing corresponding parameters during the initial point construction phase, as illustrated in Figure 1.
In this paper, we propose LEMON to construct compact initial points for small models through linear fusion of large model parameters. Specifically, we first identify the importance of parameters in larger models with fusion operators, and then compress them into new parameters for the smaller model, maintaining the same functionality of related parameters before and after transformation. To support mapping a larger model into any specific architecture, we decompose the fusion process into layer and dimension operations. The layer operator establishes layer-to-layer correspondence between larger and smaller models, while the dimension operator aims to learn the fusion of different dimensions within each parameter matrix. Recalling the modular structures among layers, we further factorize the fusion operators into Kronecker products (Schacke, 2004) with controllable receptive fields, instead of directly learning it on a huge number of parameters. Eventually, the fusion operators are initiated by the prior features of modular parameters and optimized on unlabeled data. Based on well-learned operators, it is convenient to efficiently achieve compact initial points for smaller models and facilitate subsequent training.
To verify the effectiveness, we conduct experiments on the LLaMA 2-7B model (Touvron et al., 2023) and compress it into two smaller models, 1.3B and 2.7B, respectively. Within only 0.26 billion tokens, it is sufficient to learn competent initial points for smaller models, and with an additional 50 billion tokens of post-training, LEMON outperforms a series of powerful baselines on 11 representative downstream tasks. Meanwhile, compared to traditional models trained from scratch (Biderman et al., 2023; TogetherAI, 2023; Xinyang Geng, 2023), LEMON only requires 5.26% of pre-training
token budget. When achieving comparable performance to recent pruning-based method (Xia et al., 2022), the learning process of our fusion-based method saves 62.5% of training budget. Overall, LEMON exhibits considerable flexibility with controllable layer and dimension operation, making it a unified view of existing model compression techniques, which explore a new way for model compression using in-model parameters fusion.

heading: 2 Parameters Characteristics in Large Language Models
text: To figure out the inherent characteristics of the parameters in the large language model, we calculate the layer-by-layer parameter similarity of individual components with Centered Kernel Alignment (stated in Appendix A.6). From Figure 2 (a) (here we provide the parameter similarity of multi-head attention in LLaMA 2-7b, see Appendix A.5 for the similarity of feed forward layer), one can observe that the parameters of large LMs tend to form cluster-like modular during pre-training, in which the parameter within the cluster, especially in neighboring layers, shows a high degree of similarity, while there are significant differences in the layers outside the cluster. It is intuitive to speculate that these modules naturally distinguish the functional areas of large models, allowing for the processing of language features at different levels. Specifically, the cluster-like similarity structure primarily emerges in the middle to deeper layers, which indicates a shift from explicit textual information processing in shallower layers to more abstract and progressive feature processing in deeper layers. On the contrary, the start and final layers of these models show distinct characteristics, highlighting their unique functions.
This enlightens that, on the one hand, the observation of similar parameters shows that there is a
large amount of redundancy in the pre-trained large model, which opens up the exploration of merging modules with high parameter similarity to reduce the model size. On the other hand, the highest parameter similarity is still less than 80% (see Figure 2 (b)), indicating that each part in the module has its unique function behavior and should not be directly discarded. Further motivated by findings that deep neural networks’ parameter optima can be connected via linear low-loss paths (Garipov et al., 2018; Qin et al., 2022), is imperative to explore the linear fusion of parameter modules in large models to achieve model compression.

heading: 3 Method
text: Building a stronger small model typically involves two steps. The first step is to construct a compact initial point from a larger model while preserving its functional behavior, and the second step is to perform (restorative) pre-training to further improve performance. The second step is technically no different from training from scratch, we focus on the first step in this paper.
Formally, we define a model as a set of layer weights ΘL,D = [W1,W2, · · · ,WL], where Wl ∈ RD×D ′ denotes the weight of layer l with row dimension D and column dimension D ′ . In the initialization stage, our goal is to transfer knowledge from a larger model Θ(large)L1,D1 to a smaller one Θ (small) L2,D2
, where total layers L1 > L2, hidden dimension D1 > D2, by learning a fusion operation that minimizes the functional difference between original and fused parameters,
argmin M
Ex∼D L ( x;Θ(small) −Θ(large) )
subject to Θ(small) =M(Θ(large)) (1)
where M is the fusion operator, D is the training data set, and L is the loss function. However, there are a total of O (L1D1D1L2D2D1) mapping relations between the parameters of two models, simply optimizing the above loss function is infeasible due to the high computational complexity. In the subsequent sections, we decompose the fusion operator into layer operator and dimension operator, to achieve an optimal balance between computational efficiency and model performance.

heading: 3.1 Layer Operators: Vertical Fusion with Layer-by-Layer Mapping
text: Based on the observing inherent modularity and redundancy observed in large models, it is very
intuitive to learn layer-by-layer mapping from large models to small models, in order to achieve similar functional behaviors of corresponding clusters with fewer layers.
Following this line of thinking, we introduce Monarch matrix (Dao et al., 2022) to reduce the computational cost of matrix multiplication. By decompose the fusion operator M into layer operator Llayer and dimension operator Rdim. i.e. M = LlayerRdim. The layer operator Llayer = γ⊗ I, where γ is the layer-by-layer mapping matrix, I is a highly sparse block diagonal matrix. The parameter module in the smaller model is a linear sum of the parameter modules of multiple layers in the larger model: Wsmalli = ∑L1 j=1 γi,j(W large j ), where γi,j represents the mapping relationship from the j-th layer of the original model to the i-th layer of the target model.
M =     γ1,1 · · · γ1,L1 ... . . . ...
γL2,1 · · · γL2,L1
 ⊗ I  
︸ ︷︷ ︸ Layer Operators
  R1dim . . .
RL1dim
 
︸ ︷︷ ︸ Dimension Operators
(2)
where Rldim ∈ RD2×D2 is the dimension operator of l-th layer. Llayer performs linear mapping of parameter modules layer by layer, enabling precise and efficient structural transformations while preserving layer functionality. This approach simplifies parameter learning and mapping, especially in large-scale models with extensive parameters1.

heading: 3.2 Dimension Operator: Horizontal Fusion with Controllable Receptive Field
text: In this section, we create Rdim, a set of dimension operators for compressing parameters into desired shapes, by splitting R into B and A, which independently manage row and column transformations. Given a source matrix W ∈ RD1×D ′ 1 and a target matrix W′ ∈ RD2×D ′ 2 , we utilize the Kronecker product: since vec(CAB) =( B⊤ ⊗ C ) vec(A) (Schacke, 2004), operator Rl = Al ⊗ Bl , where A ∈ RD ′ 1×D ′ 2 and B ∈ RD1×D2 , achieving W′ = B⊤l WAl. Due to the considerable size of A and B in large models, we further refine them into sparse and lowrank matrices A∗ and B∗ to minimize memory and computational demands. We illustrate this with B∗ acting on W, defining the parameter modularization’s receptive fields θ within W as r, thus simpli-
1Reduce to O(D1D1D2D2 + L1L2)
fying the formulation to B∗ = ∑D2/r
i=1 ∑D1/r j=1 bi,j
and W = ∑D1/r
i=1 θi,j , the formula as:
B∗W =     b1,1 · · · b1,D1/r ... . . . ...
bD2/r,1 · · ·bD2/r,D1/r
 ⊗ I  
︸ ︷︷ ︸ Row mapping matrix
  θ1
. . . θD1/r
 
︸ ︷︷ ︸ Parameter matrix
(3)
The row scaling operator B is provided with a receptive range. Each element b in B maps the parameters within its receptive range in the original matrix as a whole. By providing different receptive ranges to different parameter modules, we not only ensure the integrity of the modular parameter structure but also ensure the diversity in the mapping of parameter matrices with different depths and functionalities. Moreover, this approach optimises the operator complexity in a controlled way2. Finally, we have RΘ = [ B∗1W1A ∗⊤ 1 · · · B∗L1WL1A ∗⊤ L1 ] .
In Figure 3, we intuitively describe the workflow of build a compact initial point by fusing parameters from larger models, which adeptly preserves essential information embedded in the model parameters while effectively reducing the model size. Subsequently, we perform routine pre-training to optimize the goal of language modeling, the initial point will help us break through the scaling law, and quickly surpass the model performance training from scratch.

heading: 3.3 Implementation in Compressing LLaMA
text: Our approach is theoretically applicable to transforming parameters across various neural network designs, focusing on preserving the knowledge within large language models parameters.
2Reduce to O((D1D2 +D′1D′2)/r2 + L1L2).
Using the LLaMA model as a case study, we implement our compression technique on all parameters. By compressing the embedding layer and output heads using B*emb and sharing parameters across the compression operator’s hidden size dimension, this method significantly reduces the number of parameters required for learning, enhancing efficiency. In the multi-head attention and feed forward components, due to distinct parameter roles, we tailor operators A*attn and A*ffn for each unique module, ensuring effective compression without loss of function.
We quantify the discrepancy between source (ps) and target (pt) model distributions using KL divergence Lkl = DKL(ps∥pt) and guide mapping learning with language model loss. The total loss merges these aspects:
Lfinal = λLlm + (1− λ)Lkl (4)
Due to the flexibility of the LEMON, we can represent the results of other model compression methods (see Sec 3.4) and base our training on these. In implementation, to conserve resources, we experimented with various training-free initialization methods: such as directly truncating the model, extracting similar layers based on modular features, and deletion results based on pruning algorithms, among others. The pseudocode for a single forward shown in algorithmic 1 in Appendix A.

heading: 3.4 The Connection Between LEMON and Existing Methods
text: Layer Dropping Methods (Zhang and He, 2020b; Sajjad et al., 2023) remove layers to reduce the model’s depth. The selection of subset layers L2 to keep (L2 ⊆ L1, L1 denotes the set of total layers) indicates the depth reduction process as:
W initi =
L1∑
j=1
γi,jW large j , γi,j = { 1 if j ∈ L2, 0 otherwise. (5)
Structured Pruning Methods (Jiang et al., 2023; Ma et al., 2023; Xia et al., 2023) prune both rows and columns of the matrix and removes layers to achieve a target model size. It shares the layer reduction formula with layer dropping. For row and column pruning, represented by D2 ⊆ D1 and D′2 ⊆ D ′ 1 respectively, the pruning process is:
W initi,m =
D1∑
j=1
D ′ 1∑
n=1
ai,jbm,nW large j ,
ai,j = { 1 if j ∈ D2, 0 otherwise. , bm,n = { 1 if n ∈ D′,2 0 otherwise. (6)
Parameter Selection Methods (Xu et al., 2023; Kim et al.) for initializing models with a subset of original parameters, including regular, random, and search-based selection, mirror the structured pruning’s approach to matrix dimensionality reduction. These methods apply similar principles to selectively maintain or discard matrix dimensions and layers.

heading: 4 Experiments

heading: 4.1 Setup
text: Data To learn LEMON operators and continue pre-training, we use the RedPajama (TogetherAI, 2023) dataset, mirroring the training data in LLaMA across seven domains (CommonCrawl, C4, Github, Wikipedia, Books, ArXiv, StackExchange). The dataset includes a validation set of 2 million tokens, a training set of 4 billion tokens, and a continued pre-training set of 50 billion tokens.
Training Our setup uses Sheared-LLaMA (Xia et al., 2023) code base on Composer package (Team, 2021), tested on 8 NVIDIA A100 GPUs (80GB). The models were trained at the sequence length of 4096 using a batch size of 32 during the pruning phase and continued pre-training phase.
Following the setup in Sheared-LLaMA (Xia et al., 2023), baseline models were trained for 6400 steps (22 hours) with dynamic batch gradient loading (DoReMi (Xie et al., 2023)) during the pruning phase. In contrast, LEMON were trained for 1200 steps (8 hours) without DoReMi. For continued pre-training, we employed 50 billion tokens with DoReMi with 256 batch size (387 hours). Learning rates were set to 1e-4 for language modeling loss, and for layer and dimension operators, 5e-4 and 5e-5, respectively. For more information, please refer to Appendix A.3.
Evaluation Following the multiple assessment metrics reported in the baseline studies (Xia et al., 2023), we utilized lm-evaluation-harness (Gao et al., 2023) to evaluate our models. In common sense and reading comprehension evaluation, we report 0-shot accuracy of SciQ (Welbl et al., 2017), PIQA (Bisk et al., 2020), WinoGrande (Abbreviated as Wino.) (Sakaguchi et al., 2020), ARC Easy (Clark et al., 2018a), HellaSwag (Zellers et al., 2019), 25-shot accuracy of ARC Challenge (Clark et al., 2018b). In world knowledge evaluation, we report accuracy of 32-shot NQ (Kwiatkowski et al., 2019) and 5-shot MMLU (Hendrycks et al., 2021). In Continued QA and text understanding evaluation, we report accuracy of 0-shot LogiQA (Liu et al., 2020), 32-shot BoolQ (Clark et al., 2019) and 0-shot LAMBADA (Paperno et al., 2016).

heading: 4.2 Baseline
text: Structured Pruning Methods construct initial points by searching and eliminating non-essential neurons, followed by continued pre-training. We designate Sheared-LLaMA (Xia et al., 2023) as our primary baseline. Sheared-LLaMA provided a robust training framework, dynamic batch gradient pre-training methods, and an evaluation framework, achieving competitive outcomes.
Pre-training from Scratch Baseline Models train from scratch without utilizing any pre-existing model parameters. We choose similar-scale opensource LLMs includes Pythia-1.4B & -2.8B (Biderman et al., 2023), which use different training data than RedPajama. We choose INCITE-Base3B (TogetherAI, 2023) as baseline which use the same training data.

heading: 4.3 Result
text: Foundational Capabilities. Table 2 shows the foundational capacity performance of the model
after initialization with LEMON operators then continued pre-training with 50B tokens. We also evaluate capabilities of similarly sized models and models built from pruning methods. Experiments show that our approach achieves even better performance with very little training resource overhead (i.e. 50B tokens of training data) compared to models built from scratch. LEMON-1.3B outperforms the Pythia-1.4B models, which were initially pretrained with 300B tokens. LEMON2.7B also outperforms 300B tokens pretrained Pythia-1.4B models and 300B tokens pretrained INCITE-Base-3B. Compared to prune-based initialization methods, we obtain better initial points retaining the knowledge of larger model, which leads to substantial savings in pre-training. Our LEMON-2.7B, LEMON-1.3B outperforming Sheared-LLaMA-2.7B and Sheared-LLaMA-1.3B respectively.
Better Initialization. We check whether LEMON produces a better initialization than existing language models of the same size. We continue to pre-train the LEMON-2.7B model on the original RedPajama data, comparing it to the INCITE-Base-3B adn Sheared-LLaMA-2.7B
model. Figure 4 shows the INCITE-Base-3B model has much higher initial accuracy, but its performance stagnates throughout the continued pre-training. Sheared-LLaMA and LEMON-2.7B has lower initial accuracy but rapidly improves and eventually outperforms the INCITE-Base-3B model. LEMON-2.7B has higher initial accuracy and final performance than Sheared-LLaMA-2.7B. The parameter-fused model outperforms the pruned model and is more suitable for initialising a strong model for further pre-training.
Fast Convergence. We report the Eval PPL during training process for LEMON and ShearedLLaMA in the same configuration. As shown in the Figure 4 (Middle), although the initial point is not sufficiently satisfactory, LEMON can converge quickly within 2,000 steps, using only 33% of Sheared-LLaMA training steps. LEMON reaches a PPL of 7.47 at 2,000 steps, which is below the 7.49 PPL of Sheared-LLaMA at 4,800 steps. The PPL of LEMON achieves parity with Sheared-LLaMA at 500 steps and surpass the Sheared-LLaMA by 3.5 point at 1000 steps, demonstrating the ability to effectively retain the information in parameters. We can continue to allocate the LEMON training

heading: Method SciQA PiQA Wino. ARC-E LogiQA AVG
text: budget to reduce the initial PPL, which involves a trade-off in computational resource allocation between compression and pre-training.

heading: 4.4 Analysis
text: In this section, we evaluate our layer fusion operators by reducing LLaMA2-7B’s depth from 30 to 24/16 layers (L.) and evaluate dimension operators through compress parameter matrix to 62.5% origin dimensions. We compare this with the following methods: last layers cutting (Cut), random layer removal (Random), layer parameter averaging (Avg), and Sheared-LLaMA (Shear). For weight averaging, we cluster similar layers, averag-
ing within clusters for new parameters. We choose the optimal result from Sheared-LLaMA as our baseline. The LEMON-Cut and LEMON-Prune operator initialized from the Cut and Shear, then were trained for 1200 steps. The results are show in Table 3.
Layer Fusion As shown in 3, even in the dimension of layer fusion alone, our method shows great advantages. New model parameters after fusion improve 6.1% and 7.5% absolute accuracy compared to the initial point in the scenarios with 8-layer deletion and 16-layer deletion, and 3.4% and 7.9% compared to pruning approach. The effectiveness of LEMON improves as the magnitude of model compression increases, suggesting LEMON help to preserve the Large Model performance at largescale parameter discarding.
Dimension Fusion We report the accuracy of LEMON dimension compression for different configurations of Multi-Head Attention (MHA) and Feedforward Neural Network (FFN) in the Table 4 and Figure 4 (Right). Under the simple Cut initialization method, our operator achieves ShearedLLaMA performance. Even at high dimension compression settings (4096 to 2560), the fused model performance still rise dramatically compared to the initial point. This indicates the strong adaptability and generalisation of our method.
Receptive Fields Table 4 and Figure 4 (Right) shows the performance of dimension compression operators under different receptive fields settings (MHA-FFN). (r1-r2) represents the MHA operator receptive fields of r1 and the FFN operator receptive fields of r2. In the experiment, Attention modules show robustness to larger receptive fields (e.g.128), highlighting heads’ modular parameter structure. Conversely, hidden and inter-
mediate sizes are more sensitive, where finer settings yield greater performance boosts. However, the increased computational overhead from finer receptive fields suggests a balance is necessary. Optimal receptive fields settings should consider the model’s parameter modularity, waiting further systematic investigation.
Operators Initialization Cut and Prune represent the operators’ initialization based on direct cutting model and pruning algorithm results. As Shown in Table 3, the starting point of operators affects the performance after LEMON fusion. Although the performance is lower after cutting model, through learning, LEMON enhancing the average accuracy by 4.8% when retaining 24 layers and by 11.3% when retaining 16 layers. These improvements surpass the pruning search algorithm by 2.3% and 12.9% under the same experimental settings. The experiments also demonstrate that LEMON operator can be combined with existing compression methods, improving upon the pruning methods by 0.7% and 10.2%. Experiments with 7B and 3B models indicate that providing LEMON with better initial points helps enhance model’s performance ceiling. This is a valuable area for future research. For more information about compressing 3B model, please refer to Appendix A.4.
Explainability Analysis We visualized the layer fusion operators of the LLaMA model in Figure 5 and Appendix A.7. Through learning, the dimension operators autonomously form and aggregate clusters of similar layers within the model, reflecting observed parameter module similarity. The operator weights decrease with distance, even turning negative, indicating the operators deem distant parameter modules irrelevant or even detrimental to the current module. After fusion, the parameter distribution within the LEMON model still shows consistency with the original model (discussed in
Appendix A.5, Figure 6). This indicates that new fused parameters reconstructed the functions of multiple parameters from the original model.

heading: 5 Related Work
text: Efficient pre-training approaches In recent years, the ability of incremental training to accelerate large-scale model training by studying how to obtain the optimal initialization point for training has thus attracted much attention (Xie et al., 2017; Wu et al., 2019). Asymptotic training (Zhang and He, 2020a) is used to learn larger-scale models by training small transformers with a small number of layers, and then gradually expanding them by stacking layers. Net2Net (Chen et al., 2015) uses function-holding transformations to expand the width by duplicating neurons, and uses a unitary layer implementation to expand the depth. LiGO (Wang et al., 2023) proposes a learnable expansion method that can be used at the initial initialization point of a transformer. learned expansion methods that can expand models of arbitrary structure at initialization. LEMON is inspired by these methods, but we investigate how to learn to map the parameter matrix from large to small without losing the ability of the larger model itself.
Model Compression Our approach is dedicated to obtaining a high-performance lightweight language model, which is the same goal as the task of model compression. Quantization (Gray and Neuhoff, 1998) reduces the numerical accuracy of model weights and activations, and speeds up training and inference, but results in a loss of model accuracy and the inability to freely build targetspecific models. CRash (Zhang et al., 2023) and LayerDrop (Zhang and He, 2020b; Sajjad et al., 2023) methods discard ineffective layers during training, which do not allow for target-specific structuring and come with a large performance loss. Pruning (Wang et al., 2020) minimizes the impact on performance by cutting out redundant neurons that over-parameterize the model. In the LLM era, this leads to a significant reduction in neuron redundancy as models move from task-specific to generalized (Frantar and Alistarh, 2023). Pruning LLM leads to performance degradation at larger pruning magnitudes. LLMsheairng (Xia et al., 2023) uses the results of pruning as initialization for continuous pre-training of the model to recover performance, but this approach requires more data and computational overhead. We avoid the information
loss caused, by learning the parameter fusion matrix of the model to reach a specific structure, thus obtaining better initialization points and reducing the overhead of continuous pre-training.

heading: 6 Conclusion
text: In this paper, we propose a new paradigm for building smaller LMs based on larger LMs. By learning a parameter fusion operator with controllable receptive fields from a larger model to smaller models, we can obtain a good starting point to facilitate subsequent training. This fusion mapping operator consists of a layer operator and a dimension operator. Experimental results demonstrate that our method can compress larger LMs into smaller LMs of arbitrary architectures and better preserve the knowledge in large models. By comprising the requirements of pre-training data, we demonstrate the effectiveness of the method against several baseline approaches in terms of training speedup and computational expenditure savings. The paradigm has greater research value and has the potential to reach smaller performance losses and lower computational effort in the future.

heading: Limitation
text: There are limitations to our approach: First, we have only explored the use of linear methods for parameter fusion in our model. In the future, nonlinear methods deserve more exploration as they have the potential to better link different parameters and reach optimality. Second, limited by computational resources, we only experiment on 7B-scale and 3B-scale models. However, our method is scalable (discussed in Appendix A.4) and can be extended to models of arbitrary size in future work. Third, although we minimize the computational complexity of the fusion operator, it still requires a lot of memory and computation for optimization. Finally, we believe that the fusion of parameters within models is a direction that has not been fully explored. How to solve the information loss caused by parameter fusion is a problem we will try to solve in the future.

heading: Ethical Consideration
text: In our study, we leverage open and online accessible data and techniques, mitigating privacy concerns. Our method emphasizes enhancing model parameter efficiency and reducing size to create powerful, compact, and openly accessible models,
thereby promoting the open dissemination and democratization of NLP technologies. By employing pre-training strategies, we aim to mitigate biases through extensive training and large corpus sizes, contributing to an ethical AI development that prioritizes openness, efficiency, and bias reduction. Our work is committed to advancing accessible and efficient NLP technologies, fostering a more inclusive and automated future for AI.

heading: Acknowledgments
text: We thank the anonymous reviewers for their insightful comments and constructive suggestions. We would like to thank Yinqi Yang, Jiawei Sheng, Xinhua Zhang, Shicheng Wang, Chuanyu Tang and members of the IIE KDsec group for their valuable feedback and discussions. We are very grateful to Mengzhou Xia for providing the concise and effective ShearingLLaMA experimental code and for her assistance during the reproduction process. Work done during Yilong Chen’s internship in Baidu Inc. This research is supported by the National Key Research and Development Program of China (Grant No.2021YFB3100600) and the Youth Innovation Promotion Association of CAS (Grant No.2021153).

heading: A Appendix

heading: A.1 LEMON Operator
text: Our final operator is as follows:
M =     γ1,1 · · · γ1,L1 ... . . . ...
γL2,1 · · ·γL2,L1
 ⊗ I  
︸ ︷︷ ︸ LLayer    A∗1 ⊗B∗1
. . . A∗L1 ⊗B∗L1
   
︸ ︷︷ ︸ RParam
(7)
We freeze the parameters of LLaMA model and optimise only Lemon Fusion operators. In the LLaMA-7B model, our single forward operation is summarised in Algorithm 1.
Algorithm 1 Learning operators in LLaMA 1: Model parameters θ, Hidden dimensions D1, D2, Number
of layers L1 and L2, Intermediate sizes H1, H2, Embedding sizes E. 2: W (emb) ∈ RD1×E , WQl ,WK ,W Vl ,WOl ∈ RD1×D1 , W
(up) l ,W (gate) l ∈ RH1×D1 ,W (down) l ∈ RD1×H1 ,
W (ln1) l ,W (ln2) l ∈ RD1 , ∀l ∈ [L1], W (out) ∈ RE×D1
3: initialization operator γ,A∗, B∗ 4: Step 1: Learning Depth Mapping 5: for l = 1, · · · , L2 do 6: for Θ = Q,K, V,O, up, gate, down, ln1, ln2 do 7: W ′Θl ← ∑L1 j=1 γ Θ l,jW Θ j 8: end for 9: end for
10: Update θ, γ with L(θ, γ) 11: Step 2: Learning Width Mapping 12: W ′(emb) ← B∗(emb)W ∗(emb) 13: for l = 1, · · · , L1 do 14: for Θ = Q,K, V, up, gate do 15: W ′Θl ← A∗(Θ)WΘl B∗(emb)⊤ 16: end for 17: for Θ = O, down do 18: W ′Θl ← B∗(emb)WΘl A∗(Θ)⊤ 19: end for 20: for Θ = ln1, ln2 do 21: W ′(ln2)l ← B∗(emb)W (ln2) l 22: end for 23: end for 24: W ′(out) ←W (out)B(emb)⊤ 25: Update θ,A∗, B∗ with L(θ,A∗, B∗) 26: Step 3: Continuous-pretrain model with parameters θ.

heading: A.2 Model Configurations
text: For a fairer comparison, we directly used the model architecture of baseline works (Xia et al., 2023) to construct LEMON-2.7B & -1.3B. Beyond the initial, in order to perform ablation experiments to verify the performance of the layer fusion operator and the dimension fusion operator, we constructed
LEMON-Vertical, LEMON-Horizontal -37.5% and -50% . Table 5 report the parameter configurations adopted for the different scale models in our experiments.

heading: A.3 Training Details
text: The hyperparameters used in our experiments are presented in Table 6. We employ fully sharded data parallel to efficiently train our models in parallel. A cosine learning rate scheduler is used, with the learning rate decaying to a minimum of 10% of the peak value. Preliminary experiments were conducted to determine the optimal peak learning rate for learning the fusion operators and Lagrange multipliers.

heading: A.4 Pre-experiments in 2.7B Model and LEMON Scalability
text: Considering the experimental cost and compression efficiency, we conduct pre-experiments on ShearedLLaMA 2.7B to explore the optimal compression operators and hyperparameter comparisons, which are later applied to the 7B-scale LLaMA model.
As shown in Table 7, LEMON demonstrated the same excellent performance on both the compressed 3B model and the compressed 7B model. The LEMON fusion operator with the clipping and pruning algorithms as initial points improves performance by 3.5% and 4.2%, respectively, when compressing the 2.7B model.This is consistent with our observation that the LEMON fusion operator with clipping as an initial point improves performance by 4.8% on the 7b scale. On more layers of compression, our approach achieves equally impressive performance at 2.7B and 7B. This suggests that our method has good scale scalability and can be applied to larger models with good performance.
It is worth noting that our approach is validated at different model scales with appropriate hyperparameters and applied at larger scales. For example, in both 2.7B and 7B model compression we adopt a layer fusion operator learning rate of 5e-4, and a dimension fusion operator learning rate of 5e-5, as shown in Table 7. In the pre-experiments, we also found that layer fusion (deleting layers) inevitably brings more information loss, so we perform dimension fusion first and then layer fusion.

heading: A.5 Parameter Modularity in LLaMA2 Model and LEMON Model
text: Here we show the modularity phenomenon observed in the llama model at all scales. For details,
see Figure 7, 8. We visualized the similarity of the Attention query matrices between layers in a 24-layer model compressed using LEMON layer fusion operators in Figure 6. We compared this with the original, uncompressed LLaMA2-7B model. Although the fused model has not yet undergone pre-training, the parameter distribution within the model still shows consistency with the original model. This indicates that LEMON successfully reduced the size of similar parameter clusters and
that the new fused parameters successfully reconstructed the functions of multiple parameters from the original model.

heading: A.6 Centered Kernel Alignment (CKA)
text: Centered Kernel Alignment (CKA) is a statistical measure used to quantify the similarity between two sets of data representations. Unlike traditional correlation measures, CKA is designed to be invariant to orthogonal transformations and scaling of
the data. To calculate the similarity between two sets of representations using CKA, we employ a kernel function to map the original data into a higherdimensional space, where the alignment of their central tendencies can be more easily measured. The CKA value ranges from 0 to 1, where 0 indicates no similarity and 1 indicates identical representations.
The mathematical formulation of CKA, when using a linear kernel, is given by the following equation:
CKA(X,Y ) = ∥XTY ∥2F√
∥XTX∥2F · ∥Y TY ∥2F
Here, X and Y are matrices whose columns are the vectors of the representations to be compared, ∥ · ∥F denotes the Frobenius norm, and XT and Y T are the transposes of X and Y , respectively.

heading: A.7 Operator Visualisation
text: In order to better observe the parameter distributions of the operators and investigate the explainable principles, we visualised the post-training distributions of the operators initialised based on Cut’s method in Figure 10.
Figure 9: FFN Weights Similarity of different scale LLaMA2 Models
Figure 10: Visualisation of operator parameters under Cut initial points

heading: Future Work
text: Our work is committed to advancing accessible and efficient NLP technologies, fostering a more inclusive and automated future for AI