Venue: NAACL 2025

# MoDification: Mixture of Depths Made Easy

 Chen Zhang\({}^{\spadesuit}\), Meizhi Zhong\({}^{\spadesuit}\), Qimeng Wang\({}^{\spadesuit}\), Xuantao Lu\({}^{\spadesuit}\), Zheyu Ye\({}^{\spadesuit}\), Chengqiang Lu\({}^{\spadesuit}\), Yan Gao\({}^{\spadesuit}\), Yao Hu\({}^{\spadesuit}\), Kehai Chen\({}^{\spadesuit}\), Min Zhang\({}^{\spadesuit}\), Dawei Song\({}^{\spadesuit}\).

\({}^{\spadesuit}\)Beijing Institute of Technology

\({}^{\spadesuit}\)Harbin Institute of Technology, Shenzhen

\({}^{\spadesuit}\)Xiaohongshu chenzhang9702@outlook.com

###### Abstract

Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to \(\sim\)1.2\(\times\) speedup in latency and \(\sim\)1.8\(\times\) reduction in memory compared to original LLMs especially in long-context applications.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 5137–5149

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Long-context efficiency is turning to be one of the core concerns in large language model (LLM) serving [6, 11, 12, 1]. Typically, a long context can incur dramatically huge latency and memory overhead either at prefilling or decoding stage [23].

To address this consumption, strategies like speculative decoding [13, 14, 15] and key-value (KV) cache compression [16, 17, 18] have already been utilized to enhance the latency and memory, respectively. However, dedicating efforts to separate components is not always ideal. As a consequence, mixture of depths [15] has recently been introduced to simultaneously consider both two facets. Conceptually, the key feature of MoD is enabling LLMs to conditionally eliminate computations of layers over certain tokens. And distinguished from early exiting (EE, Chen et al., 2024), MoD skips over rather than exiting from intermediate layers so that performance is better preserved.

Frustratingly, MoD can hardly guarantee both efficiency and effectiveness without extremely costly training. To be more specific, current advances of MoD are only observed in cases where training is started from scratch, predominately narrowing the impact of MoD.

The sub-optimality is mainly due to the improper use of top-k operator. The top-k operator is basically employed at each layer to determine which tokens should be skipped in the computation of the layer. Unfortunately, for efficiency, the top-k operator is computationally expensive [23] and forces a constant number of tokens retained, possibly restricting the efficiency improve

Figure 1: The sub-optimality of MoD due to the use of top-k operator. For efficiency, the efficiency improvement is restricted because top-k operator is not cheap and unimportant computations are preserved by retaining a constant number of tokens at every layer. For effectiveness, the effectiveness is undesired because dominated computations are saved by skipping a constant number of tokens at every layer.



ment brought by computation saving at one unimportant layer, as shown in Figure 1 (left). For effectiveness, it deviates from the distribution of layer significance Men et al. (2024), potentially resulting in undesired computation saving at one dominated layer, as shown in Figure 1 (right). These disadvantages largely impede the applications of MoD.

In this paper, in order to broaden the scope of MoD, we target at converting existing LLM checkpoints to MoD ones with our designed MoDiffication.1 The designs of MoDiffication follow two guidelines: 1) alleviating the sub-optimality of MoD, and 2) minimizing the training of MoD. Under the guidelines, we firstly use threshold-p operator as an drop-in replacement of the top-k operator. On one hand, threshold-p operator is not much computationally expensive and allows any number of retained tokens; on the other hand, it can make the layer itself determine the priority of tokens through adaptive gates across layers. We secondly find around 10B tokens with decent diversity are fairly enough in the conversion process.

Footnote 1: MoDiffication is a re-inverted compound representing the process of adapting a LLM to a MoD one, following the naming convention of MoEfication Zhang et al. (2022).

We conduct experiments on model scales ranging from 3B to 70B. The efficiency and effectiveness of MoDiffication are verified on such a wide array of scales. With acceptable performance decline, MoDiffication can achieve up to \({\sim}1.2\times\) speedup in latency and \({\sim}1.8\times\) reduction in memory compared to original LLMs especially in long-context applications. On the contrary, MoD can unexpectedly lead to slow-down in latency under the same experimental settings. Further ablations and discussions validate the trustworthiness of the design choices in MoDiffication.

## 2 Background

Long-context EfficiencyOver the years, the super expressiveness of large language models (LLMs) has been widely witnessed thanks to the stable transformer architecture Vaswani et al. (2017) and the massive pretraining Brown et al. (2020). Concretely, two modules are involved in a transformer layer, namely, self-attention Attention module and multi-layer perceptron (MLP) module. Essentially, layer norm (Norm) and residual connection are also armed around. Given the hidden state of the \(i\)-th token \(\mathbf{x}_{i}\), the forward process in one transformer layer can usually be depicted as below:

\[\begin{split}&\mathbf{y}_{i}=\text{Attention}(\text{Norm}( \mathbf{x}_{i}))+\mathbf{x}_{i},\\ &\mathbf{z}_{i}=\text{MLP}(\text{Norm}(\mathbf{y}_{i}))+\mathbf{ y}_{i},\end{split} \tag{1}\]

While the transformer architecture admits the scaling of pretraining, a critical bottleneck during inference is also revealed by the transformer. In other words, the transformer architecture mainly grants parallelism during training, yet only generates tokens one followed by another and requires KV caches adequately stored during inference. This bottleneck may easily slow down the expected speed and shoot over the provided hardware, especially for long-context circumstances Xiao et al. (2024b).

To this demand, many approaches have been proposed to either counter the latency or the memory increment Wan et al. (2024). Among these approaches, speculative decoding Chen et al. (2023) is a representative cluster in reducing the latency and KV cache compression Xiao et al. (2024b) is another symbolic group in mitigating the memory. However, for an integrated inference system, it is might be somehow redundant to conquer the latency and the memory separately. Consequently, later solutions to some extent shift the focus to conditional computation Bengio et al. (2015), in which early exiting EE, Chen et al. (2024) and mixture of depths MoD, Raposo et al. (2024) are of central roles.

Mixture of DepthsIn a nutshell, MoD attempts to eliminate unnecessary layer computations over tokens via a token-level gate (Gate) together with a top-k operator (Top-k). In this way, the forward process within one transformer layer is changed as below:

\[g_{i} =\text{Gate}(\mathbf{x}_{i})=\text{sigmoid}(\mathbf{W}\mathbf{ x}_{i}), \tag{2}\] \[f_{i}=\text{Top-k}(g_{i},k),\] \[\mathbf{y}_{i} =\begin{cases}\text{Attention}(\text{Norm}(\mathbf{x}_{i}))+ \mathbf{x}_{i},&f_{i}=1,\\ \mathbf{x}_{i},&f_{i}=0,\end{cases}\] \[\mathbf{z}_{i} =\begin{cases}g_{i}\cdot\text{MLP}(\text{Norm}(\mathbf{y}_{i}))+ \mathbf{y}_{i},&f_{i}=1,\\ \mathbf{y}_{i},&f_{i}=0,\end{cases}\]

where \(g_{i}\in[0,1]\) is a sigmoid-normalized score after a linear weight \(\mathbf{W}\), and \(f_{i}\) is an indicator of whether the \(i\)-th token is exactly ranked among the Top-k tokens regarding its score yielded by the Gate. In addition, MoD is usually applied in an 

interleaved manner, that is, only one out of two adjacent layers is equipped with MoD.

It is acknowledged that EE is a fine alternative of MoD. Instead of skipping over intermediate layers, EE aggressively exits from intermediate layers and directly ignores later layers, as in Figure 2. Because of this, it is claimed that MoD is more flexible than EE does, and can offer better performance as well.

Despite the superiority of MoD, it does not come without pain. Although up-to-date progress Raposo et al. (2024) has indeed supported that MoD delivers great success, the training is always launched from scratch with an enormous amount of data. We argue training MoD with an appropriate quantity of data and from publicly released checkpoints should likewise be a pivotal track that is worthy of exploration. And through easing the training, MoD might fully shed its light on related areas.

In our pilot study, we have uncovered sub-optimality pertained to both efficiency and effectiveness of MoD in above lite training setting. Primarily, the sub-optimality is concerned with the improper use of the top-k operator. From the efficiency perspective, the top-k operator is computationally expensive Wang et al. (2021) and forces a constant number of tokens retained, and can limit computation saving thus efficiency improvement for unimportant layers. From the effectiveness perspective, the top-k operator suffers a divergence from the true distribution of layer significance Men et al. (2024), and can undesirably result in excessive computation saving for dominated layers. The evidence can be referred to in Figure 1.

## 3 MoDification

In this work, we put forward such a method, named MoDification, that is aimed at converting existing LLM checkpoints to MoD ones with slight training compute.

To circumvent the sub-optimality of original MoD, two designs are engaged, in which one is associated with architecture and the other is connected with data. The first one is to use threshold-p operator in place of the top-k operator, as shown in Figure 2. threshold-p operator is not much computationally expensive and allows any number of retained tokens. We then instantly discover this threshold-p replacement correlates MoDification to mixture of experts (MoE) Shazeer et al. (2017); Fedus et al. (2022) and we can empower MoDification with cutting-edge techniques that have been widely used. The second one is to enhance the data diversity and constrain the training to the scale of 10B tokens. We luckily unveil this data scale is fairly enough in practice.

ArchitectureAfter applying threshold-p operator to Equation 2, we make a further refinement to the formulation. The formula can now be briefly formed as below:

\[g_{i}=\text{Gate}(\mathbf{x}_{i}),\quad f_{i}=\text{Threshold- p}(g_{i},p), \tag{3}\] \[\mathbf{y}_{i}=\begin{cases}g_{i}\cdot\text{Attention}(\text{ Norm}(\mathbf{x}_{i}))+\mathbf{x}_{i},&f_{i}=1,\\ \mathbf{x}_{i},&f_{i}=0,\end{cases}\] \[\mathbf{z}_{i}=\begin{cases}g_{i}\cdot\text{MLP}(\text{Norm}( \mathbf{y}_{i}))+\mathbf{y}_{i},&f_{i}=1,\\ \mathbf{y}_{i},&f_{i}=0,\end{cases}\]

where the refinement is imposed as also multiplying the gate value \(g_{i}\) to the Attention module. Plus to this, MoDification also follows the interleaved fashion as MoD does.

In fact, the conditional provided by \(f_{i}\) can be interpreted from an MoE view as below:

\[\mathbf{z}_{i}=\begin{cases}g_{i}\cdot\text{MLP}(\text{Norm}(\mathbf{y}_{i})) +\mathbf{y}_{i},&f_{i}=1,\\ (1-g_{i})\cdot\text{NoOp}(\mathbf{y}_{i})+\mathbf{y}_{i},&f_{i}=0,\end{cases} \tag{4}\]

where NoOp generally means no operation is carried out and always leads to zero output.

Thereby, MoDification is fundamentally a two-expert MoE Fedus et al. (2022) where one of the two experts is NoOp. In this sense, the \(p\) value of the threshold-p operator should theoretically be 0.5, which is henceforth leveraged unless otherwise specified. Moreover, to promote the sparsity of MoDification, we are inspired by the expert load balancing technique Fedus et al. (2022) in MoE and suggest a layer load reducing objective as below:

\[F_{j}=\mathbb{E}_{\mathbf{x}_{i}}[f_{i}],\quad G_{j}=\mathbb{ E}_{\mathbf{x}_{i}}[g_{i}], \tag{5}\] \[\mathcal{R}=\alpha\cdot\sum_{j=1}^{L}F_{j}\cdot G_{j},\]

where \(\alpha\) is a coefficient that should be manually tuned, and subscript \(j\) denotes the \(j\)-th layer among overall \(L\) layers. While \(F_{j}\) depicts the fraction of tokens dispatched to the layer, \(G_{j}\) describes the fraction of the routing probability to the layer. And reducing \(R\) principally triggers more tokens be assigned to the NoOp and thus skipped by the layer. This reducing objective, together with the language 

modeling objective \(\mathcal{L}\) that drives necessarily layer load preserving, makes MoDification a reasonable game where LLMs learn to strike a good balance between efficiency and effectiveness.

DataWith the aim of reducing training time, we try to incorporate diverse domains of data to compensate limited numbers of data points. The data mixture can be found in Table 1.

The proposed data mixture generally covers domains like webs, wikis, books, codes, maths, etc. In doing so, we expect MoDification could be trained to approximate original mixtures of any existing LLM checkpoints.

## 4 Experiments

SetupWe benchmark MoDification mainly against MoD, and also compare these two to original LLM. We conduct evaluation upon MiniMA-2-3B Zhang et al. (2023) and LLaMA-2-{7,13,70}B Touvron et al. (2023) to examine the scalability of MoDification. For efficiency measures, we consider average latency in second and memory in gigabyte across a wide array of prefilling and decoding lengths ranging from 64 to 2,048 as key metrics. For effectiveness measures, we test above methods on commonly cared datasets including MMLU Hendrycks et al. (2021), CEval Huang et al. (2023), DROP Dua et al. (2019), BBH Suzgun et al. (2023), HumanEval Chen et al. (2021), and GSM8k Cobbe et al. (2021). The core metrics reported on these datasets are either accuracy, exact matching score, or pass@1 according to the designs.

ImplementationWe train MoDification and MoD with an overall batch size of 1,024 and a sequence length of 4,096. This leads to 4M tokens per optimization step. The learning rate is 3e-5, and weight decay is 1e-1. The learning rate is scheduled with a linear warm-up and a cosine decay, where the warm-up stage takes 1% optimization steps and the decay stage takes all the left optimization steps. The gradients will be properly clipped to keep the norm under 5e-1. The training precision is bfloat16. DeepSpeed Rasley et al. (2020), FlashAttention Dao et al. (2022), and gradient checkpointing Chen et al. (2024) are necessarily enabled to make the training possible with 16 Nvidia A100 GPUs.

The \(k\) for the top-\(\mathbf{k}\) operator is 512 by default, and the \(p\) for the threshold-\(\mathbf{p}\) operator is 0.5 for MiniMA-2-3B as mentioned earlier but 0.55 for LLaMA-2-{7,13,70} B since slightly larger threshold is demanded for larger LLMs in our trials. The best coefficient \(\alpha\) for layer load reducing objective is 0.01 in our developments.

ResultsFrom the main comparison results between MoD and MoDification in Table 2, we unearth that MoDification is a way better choice than MoD does. For efficiency, MoDification can reduce both the latency and memory consumption while

\begin{table}
\begin{tabular}{l r r} \hline \hline
**Data** & **Tokens** & **Proportion** \\ \hline CCI (2023) & 1.5B & 14.3\% \\ Wikipedia (2019) & 0.2B & 1.9\% \\ BookCorpus (2015) & 0.1B & 1.0\% \\ Cosmopedia (2024) & 1.0B & 9.5\% \\ FineWeb-Edu (2024) & 4.5B & 42.9\% \\ RedPajama-Stack (2023) & 0.5B & 4.8\% \\ RedPajama-GitHub (2023) & 1.3B & 12.3\% \\ Proof-File (2023) & 1.3B & 12.3\% \\ FLAN (2024) & 0.1B & 1.0\% \\ \hline Total & 10.5B & 100.0\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: The data mixture used to conduct MoDification.

Figure 2: The comparison among transformer, early exiting (EE), mixture of depths (MoD), and our MoDification.

 

MoD can unexpectedly increase the latency. For effectiveness, MoDiffication also enjoys better results and and preserves most performance from original considered LLMs. The comparison showcases the superiority of MoDiffication over MoD. And the realized efficiency merits seem to be positively correlated to the model scales in practice.

One may think that MoD is not properly configured in training and thus shows very unstable and almost meaningless results. To demonstrate that the training of MoD does not suffer from fluctuations such like loss spikes, we hereby display the training losses as in Figure 3. From the loss curves, we conclude that both MoD and MoDiffication are rigorously optimized. Besides, one may argue that the sparsity of MoD influenced by \(k\) does not align with the sparsity of MoDiffication influenced by \(p\), thus exhibiting unreasonable results. To show that the sparsity of MoD is not the key factor, we confirm in Table 3 that MoD is still hard to optimize even when the number of activated tokens in MoD is smaller than that in MoDiffication.

To further enhance the plausibility of the comparison, we also compare MoDiffication to an alternative efficient method named ShortGPT [20] that is designed to directly drop unimportant layers, therefore decreasing latency and memory, as in Figure 4. The considered ShortGPT baseline is configured as preserving 22 out of 32 layers (i.e., number of activated tokens is 90,112). The major conceptual difference between MoDiffication and ShortGPT lies in that the layer skipping of ShortGPT happens at task level. We observe that ShortGPT may lead to more apparent efficiency gains, nevertheless, it results in unacceptable performance degradation. This phenomenon implies that MoDification has considerable advantages over other efficient designs.

Figure 3: The training losses. The steady loss curves indicate that both MoD and MoDiffication are rigorously optimized.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
* MoD & 16.2 & **6.7** & 25.8 & 25.7 & 14.0 & 27.7 & 0.0 & 2.2 \\
- MoDifcation & **10.9** & **6.7** & **30.0** & **32.2** & **20.5** & **30.8** & **9.8** & **5.7** \\ \hline LLaMA-2-7B & 14.5 & 16.2 & 45.0 & 33.1 & 33.1 & 32.1 & 13.4 & 12.1 \\
- MoD & 21.6 & 14.9 & 25.5 & 26.0 & 1.8 & 7.0 & 0.0 & 0.0 \\
- MoDifcation & **13.5** & **14.7** & **36.4** & **28.6** & **25.8** & **30.0** & **8.9** & **10.3** \\ \hline LLaMA-2-13B & 19.4 & 30.3 & 54.4 & 41.2 & 43.4 & 38.1 & 14.0 & 24.1 \\
- MoD & 30.9 & 28.2 & 25.7 & 24.6 & 1.5 & 7.8 & 0.0 & 0.0 \\
- MoDifcation & **18.1** & **27.9** & **44.4** & **37.3** & **34.2** & **34.1** & **11.0** & **21.2** \\ \hline LLaMA-2-7OB & 201\({}^{\dagger}\) & 72\({}^{\ddagger}\) & 68.6 & 53.9 & 63.3 & 51.6 & 28.7 & 53.4 \\
- MoD & 276\({}^{\dagger}\) & 71\({}^{\dagger}\) & 41.2 & 30.1 & 2.5 & 12.2 & 0.0 & 0.0 \\
- MoDifcation & 178\({}^{\dagger}\) & 71\({}^{\ddagger}\) & **54.6** & **46.9** & **43.6** & **38.3** & **15.2** & **38.0** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The main comparison results concerning both efficiency and effectiveness. Better results among the ones reported from MoD and MoDifcation are **boldfaced** at varying blocks. Results enabled with INT8 quantization [20] are marked with \({}^{\dagger}\).
 

Ablation StudiesTo get an in-depth understanding of the inner working mechanisms of MoDification, we perform a series of ablation studies on the threshold value \(p\), layer loading reducing coefficient \(\alpha\), etc.

Threshold Value \(p\).We show the impact of the threshold value \(p\) in Figure 5, from which we inspect that while the memory consumption is always reduced across all threshold values, the latency and the performance can vary a lot from one threshold to another. Disappointingly, though smaller threshold values can cut down memory consumption, they can cause increased latency as a result of limited computation saving. For MiniMA-2-3B in the plot, the most suitable \(p\) value is 0.5 regarding the trade-off between efficiency and effectiveness. However, as aforementioned, larger models such as LLaMA-2-70B urge larger \(p\) values like 0.55 due to potentially larger structural sparsity.

Layer Load Reducing Coefficient \(\alpha\).In Table 4, we know that the layer load reducing coefficient \(\alpha\) impacts both efficiency and effectiveness. Particularly, the coefficient trades effectiveness for efficiency. For instance, as \(\alpha\) increases itself by an order of magnitude, the latency diminishes, so is the performance. In contrast, without the layer load reducing objective (i.e., when the coefficient is 0), MoDification tends to save very few computations. In this case, the latency is even enlarged. In summary, we assert that \(\alpha\) would be better set to 0.01 to reach equilibrium.

Interleaved v.s. Full StrategyIn our design, we follow the de facto strategy of original MoD and use MoDification in an interleaved style where MoDification is applied at every other layer. To confirm the interleaved strategy is the best choice, we compare the interleave startegy to a tactic that applies MoDification to top half layers and another that applies MoDification to full layers, as in Table 5. We note that the interleaved strategy is better than the half strategy in efficiency and is better than the full strategy in effectiveness. The reasons sit behind may be that the half strategy is inclined to conservatively skip too few computations and the full strategy is likely to skip overly many computations and affect the performance. To sum up, the interleaved strategy is currently the recommended option.

Shared v.s. Separate GateWe also include a refinement in MoDification where a shared gate value is multiplied to both the Attention and MLP modules if they are not skipped. Hereby, we would like to test whether two separate gate values respectively for the Attention and MLP modules are

Figure 4: The comparison results concerning other efficient designs. ShortGPT is more efficient yet less effective.

Figure 5: The impact of threshold value \(p\).

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Method**} & **Latency** & **Memory** & **MSMk** \\  & Second & GiB & 8-shot CoT EM \\ \hline MiniMA-2-3B & 11.0 & 7.4 & 14.6 \\ - \(\alpha=0\) & 15.7 & 6.7 & 8.3 \\ - \(\alpha=0.001\) & 14.4 & 6.6 & 7.0 \\ - \(\alpha=0.01\) & 10.9 & 6.7 & 5.7 \\ - \(\alpha=0.1\) & 8.2 & 6.4 & 1.7 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The impact of layer load reducing coefficient \(\alpha\).

 

better or not. We detect from Table 6 that the performance would not be boosted by the separate gates but the latency would be elevated by the separate gates since additional gating is introduced. So we believe the shared gate is a fair pick.

Execution VisualizationFor a more intuitive comparison between MoD and MoDiffication, we attempt to clearly illustrate why MoDiffication is more promising than MoD via visualizations of executed layers over input tokens correspondingly for MoD and MoDiffication.

In Figure 6, we unleash that MoD saves computations with the sub-optimality due to the use of top-k operator. The forced constraint that a constant number of tokens should be retained may: 1) limit the computation saving (efficiency) when computations of more tokens should be skipped, 2) affect the computation preserving (effectiveness) when computations of fewer tokens should be skipped. Conversely, MoDiffication permits more flexible computation savings, leading to a better balance between efficiency and effectiveness.

Latency ProfilingWe previously only provide latency results in average values, and here we would like to share more latency results across all prefilling lengths and accordingly decoding lengths to offer detailed insights on how MoDiffication overwhelms MoD.

We should tell from Figure 7 that MoDiffication surpasses MoD at arbitrarily any prefilling length or decoding length. And the micro gains add up to a macro gain in average. Besides, MoDiffication regularly overtakes LLaMA-2 in efficiency. We also find the latency is more sensitive to the decoding length increment than the prefilling length increment. Explicitly, the latency approximately doubles when the decoding length doubles but only increases a bit when the prefilling length doubles. We conjecture the distinction is resulted from that prefilling is parallel while decoding is sequential. And this hints that future explorations should be emphasized on decoding-time efficiency. Trickily, we also uncover that the latency delta along the increment of prefilling length gradually becomes larger. This pinpoints that prefilling-time efficiency would possibly be a concern when the prefilling length is immense, say more than 10k. This naturally drives us to explore the potential of MoDiffication in long-context applications later in Section 5.

Latency DecompositionMore precisely, we investigate the latency through a nano view. To put it differently, we decompose the latency to three parts: the one consumed by Attention and MLP, by Gate, and by Top-k or Threshold-p. Furthermore, the Threshold-p part could be broken down to: Branch phase where some tokens are selected and the others are skipped, Merge phase where both selected and skipped tokens are merged after. The resultant chart is positioned in Figure 8.

It is manifested that MoDiffication not only yields smaller latency from the Thrd-p, but also yields smaller latency from the Attention and MLP. This denotes that MoDiffication benefits from both the transition from the Top-k to the Third-p and the removal of hard constraint that at least k tokens should be retained.

It is questionable why MoDiffication can have

\begin{table}
\begin{tabular}{l c c c} \hline \hline \multirow{2}{*}{**Method**} & **Latency** & **Memory** & **BBH** \\  & Second & GiB & 3-shot Acc \\ \hline Shared & 10.9 & 6.7 & 30.8 \\ Separate & 12.0 & 6.7 & 30.6 \\ \hline \hline \end{tabular}
\end{table}
Table 6: The impact of shared gate.

Figure 6: The visualization of executed computations of layers over tokens. MoD can save computations to a certain degree. MoDiffication can save computations in a more flexible way.

 

smaller latency. Intuitively, the computationally expensive top-k operator only dominates prefilling stage and is computationally comparable to threshold-p operator at decoding stage, while prefilling latency is not the main concern in comparison to cumulatively decoding latency as in Figure 7. However, the above conjecture largely neglects the superiority of the non-uniformity yielded by threshold-p operator as in Figure 6. To counter this, we showcase exemplary latency results across prefilling and decoding stages in Table 7.

For Gate latency, MoDiffication is nothing different from MoD, and decoding one is smaller than prefilling one as decoding only processes one token at one time. For Top-k / Third-p latency, MoDiffication is significantly faster than MoD at prefilling stage due to the transition from top-k operator to threshold-p operator. Further, MoDiffication is comparable with MoD at decoding stage.

For Attention and MLP latency, MoDiffication would skip most tokens at one unimportant layer, significantly reducing latency compared to constantly retain tokens from MoD; MoDiffication would retain most tokens at one important layer, marginally yielding latency increment compared to MoD. The true reason sits behind may be that decoding is memory-bounded instead of compute-bounded (Yuan et al., 2024), so the latency will increase significantly when the number of retained tokens excels a minimal number, then increase slowly until the number of retained tokens goes beyond an acceptable range regarding memory bandwidth.

## 5 Long-context Applications

While we have fully justified the usefulness of MoDiffication in comparably short-context scenarios, we are obliged to substantiate the potential of MoDiffication in long-context applications. We take two long-context applications as testbeds, i.e., longer generation that requires long decoding and long-context retrieval that requires long prefilling. As a prerequisite, we extend the context capabil

\begin{table}
\begin{tabular}{l l c c c} \hline \hline \multirow{2}{*}{**Method**} & \multirow{2}{*}{**Stage**} & **Attention** & \multirow{2}{*}{**Gate**} & **Top-k** \\  & & **\& MLP** & **/ Thrd-p** \\ \hline MoD & prefill & 29.9 & 3.1 & 9.8 \\ MoDiffication & prefill & 24.3 & 3.1 & 4.1 \\ \hline MoD & decode & 20.9 & 2.7 & 4.9 \\ MoDiffication & decode & 18.8 & 2.7 & 3.9 \\ \hline \hline \end{tabular}
\end{table}
Table 7: The exemplary latency results across prefilling and decoding stages. Prefilling latency and per-token decoding latency are reported for prefilling and decoding stages, respectively, in milliseconds and upon MiniMA-2-3B.

Figure 8: The nano view towards the latency.

Figure 7: The detailed latency results across different prefilling lengths and decoding lengths. MoDiffication overwhelms both MoD and LLaMA-2 at any prefilling or decoding length.

 

ity of MoDification-7B from 4k to 32k. We adopt NTK-aware interpolation Roziere et al. (2023) and conduct post training on PG19 dataset Rae et al. (2019).

In long-text generation scenario where prompts sampled from Proof-Pile Azerbayev et al. (2023), MoDification-7B can achieve up to \(\sim\)1.2\(\times\) speedup in latency and \(\sim\)1.8\(\times\) reduction in memory in comparison to LLaMA-2-7B. And through long-context extension, MoDification is compatible with long-context modeling and consistently behaves in low perplexity without compromise on efficiency.

In long-context retrieval scenario where questions are gathered from Qasper Dasigi et al. (2021), MoDification-7B-32k yields 10.3 on Qasper, which is competitive with 10.2 from LLaMA-2-7B-32k. In contrast, StreamingLLM-7B Xiao et al. (2024b) which significantly prunes KV caches yields a dramatically degraded number of 6.9.

## 6 Related Work

Efficient Language ModelsImproving efficiency of language models is a long-standing and challenging task Wan et al. (2024). Besides system-level efficiency optimizations, algorithm-level efficiency optimizations mainly lie in model compression and efficient architecture. Model compression approaches Bucila et al. (2006) aim to reduce the model size directly. These compression approaches can be grouped into quantization Dettmers et al. (2022), pruning Ma et al. (2023), and distillation Yang et al. (2022); Zhang et al. (2024b, a, 2023b). Efficient architectures Child et al. (2019); Shazeer (2019) instead aim to reduce the quadratic time complexity of attention to quasi-linear one. Mixture of depths Raposo et al. (2024) stands at the intersection of model compression and efficient architecture. On one hand, mixture of depths can skip layers, which can be viewed as a type of pruning. On the other hand, mixture of depths can drop tokens, which can be viewed as a sort of complexity reduction in time.

Conditional ComputationA third perspective towards improving efficiency of language models would be conditional computation Bengio et al. (2015). It aims to relieve unnecessary computations by conditionals. This dynamic property can to the maximum extend preserve effectiveness while improving efficiency. Typical work in this area would be mixture of experts Shazeer et al. (2017); Fedus et al. (2022), early exiting Chen et al. (2024), and mixture of depths. And mixture of depths is recently viewed as one of the best choices among available conditional computation approaches.

## 7 Conclusions

It is no doubt that MoD is an ideal choice to lift the efficiency of LLMs. In this paper, we would like to address the concern that MoD comes with a must that costly training (from scratch) should be executed. Via both architecture and data designs, we make it possible to convert existing LLMs to MoD ones with appropriate data. The designed MoDification successfully outweighs MoD in both efficiency and effectiveness.

### Limitations

Our work is limited in the following two aspects: 1) we do not apply MoDification to recently released LLMs such like LLaMA-3 Dubey et al. (2024) or Owen-2 Yang et al. (2024), 2) we do not apply MoDiffication to extremely lengthy texts, say more than 100k Zhang et al. (2024c), and 3) we have to admit the uniform architecture of MoD might be more efficient on special hardware platforms.

## Acknowledgments

This work is funded in part by the Natural Science Foundation of China (grant no: 62376027).

## References

* Azerbayev et al. (2017) Zhangir Azerbaijan, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck.

Figure 9: The application to long-text generation. MoDification-7B can achieve up to \(\sim\)1.2\(\times\) speedup in latency and \(\sim\)1.8\(\times\) reduction in memory in in comparison to LLaMA-2-7B. And MoDification is compatible with long-context modeling.

 

2023. Llemma: An open language model for mathematics. _Preprint,_ arXiv:2310.10631.
* BAAI (2023) BAAI. 2023. Cci, chinese corpora internet.
* Bai et al. [2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Bingyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianjang Zhu. 2023. Qwen technical report. _CoRR_, abs/2309.16609.
* Bellal et al. [2024] Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. Cosmopedia.
* Bengio et al. [2015] Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. 2015. Conditional computation in neural networks for faster models. _CoRR_, abs/1511.06297.
* Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Bucila et al. [2006] Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In _Proceedings of the Twelfth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006_, pages 535-541. ACM.
* Chen et al. [2023] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiaux, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. _CoRR_, abs/2302.01318.
* Chen et al. [2022] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Hedy Khaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chandratzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebigen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchit Balaji, Santanu Jain,William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojcicch Zaremba. 2021. Evaluating large language models trained on code. _CoRR_, abs/2107.03374.
* Chen et al. [2024] Yanxi Chen, Xuchen Pan, Yuliang Li, Bolin Ding, and Jingren Zhou. 2024. EE-LLM: large-scale training and inference of early-exit large language models with 3d parallelism. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_. OpenReview.net.
* Child et al. [2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. _CoRR_, abs/1904.10509.
* Chung et al. [2024] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Axanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaura Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2024. Scaling instruction-finetuned language models. _J. Mach. Learn. Res._, 25:70:1-70:53.
* Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _CoRR_, abs/2110.14168.
* Computer [2023]Together Computer. 2023. Redpajama: An open source recipe to reproduce llama training dataset.
* December 9, 2022_.
* Dasigi et al. [2021]Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. _arXiv preprint arXiv:2105.03011_.
* Dettmers et al. [2022] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multiplication for transformers at scale. _CoRR_, abs/2208.07339 

Jaco Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. External Links: Link Cited by: SS5.
* D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner (2019)DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 2368-2378. External Links: Link Cited by: SS5.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahale, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Gravankumar, A. Korenev, F. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Brion, B. Tang, B. Chern, C. Caucheteux, C. Nayak, Chloe Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. Canton Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlbBadawy, E. Lobanova, E. Dinan, E. Michael Smith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. Lewis Anderson, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Yourron, I. Zarov, I. Djarra, I. Darria, I. Darrata, I. M. Klourmann, I. Misra, I. Evtimov, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Button, J. Spissak, J. Park, J. Rocca, J. Johnston, J. Saxe, J. Jia, Kalyan Vasuden Alwala, K. Upasani, K. Plwaiak, K. Li, K. Heafield, K. Stone, and et al. (2024)The llama 3 herd of models. CoRRabs/2407.21783. External Links: Link Cited by: SS5.
* W. Fedus, B. Zoph, and Noam Shazeer (2022)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res.23, pp. 1-120:39. External Links: ISSN 1574-147X, Document, Link Cited by: SS5.
* D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021)Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, External Links: Link Cited by: SS5.
* 16, 2023, X. Bengio, R. Fergus, J. Krausek, O. Karayatskaya, S. Ren, and J. Leskovec (Eds.), pp. 1-10. External Links: Link Cited by: SS5.
* Y. Leviathan, M. Kalman, and Y. Matias (2023)Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, Vol. 202 of Proceedings of Machine Learning Research, pp. 19274-19286. External Links: Link Cited by: SS5.
* Y. Li, Y. Huang, Bowen Yang, B. Venkitesh, A. Locatelli, H. Ye, T. Cai, P. Lewis, and D. Chen (2024)Snapkv: LLM knows what you are looking for before generation. CoRRabs/2404.14469. External Links: Link Cited by: SS5.
* Z. Liu, C. Zhang, and D. Song (2024)How speculative can speculative decoding be?. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 8265-8275. External Links: Link Cited by: SS5.
* A. Lozhkov, L. Ben Allal, L. von Werra, and T. Wolf (2024)Fineweb-edu.. External Links: Link Cited by: SS5.
* 16, 2023, X. Meng, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen (Eds.), pp. 2403. External Links: Link Cited by: SS5.
* X. Meng, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen (2024)Shortgpt: layers in large language models are more redundant than you expect. CoRRabs/2403.03853. External Links: Link Cited by: SS5.
* J. W. Rae, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap (2019)Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507. External Links: Link Cited by: SS5.
* D. Raposo, S. Ritter, Blake A. Richards, T. P. Lillicrap, P. Conway Humphreys, and A. Santoro (2024)Mixture-of-depths: dynamically allocating compute in transformer-based language models. CoRRabs/2404.02258. External Links: Link Cited by: SS5.
* J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He (2020)DeepSpeed: system optimizations enable training deep learning models with 

over 100 billion parameters. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_, pages 3505-3506. ACM.
* Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafori, Wenhan Xiong, Alexandre Defossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code Llama: Open foundation models for code. Preprint, arXiv:2308.12950. ArXiv: 2308.12950.
* Shazeer (2019) Noam Shazeer. 2019. Fast transformer decoding: One write-head is all you need. CoRRabs/1911.02150.
* Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziazar, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 13003-13051. Association for Computational Linguistics.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izcaard, Xavier Martinet, Marie-Anne Lacchaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lampte. 2023a. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian CantonFerrer, Moya Chen, Guillem Cucurull, David Esioub, Jude Fernandes, Jeremy Fu, Wenyi Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harsthorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcia Kardas, Viktor Kerkez, Madian Khabasa, Isabel Kloumann, Artem Korenev, Punit Singh Kouva, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Dinanaiskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaloykup, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Runeta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaqqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iilyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Rambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. CoRRabs/2307.09288.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Wan et al. (2024) Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosnaraf Chowdhury, and Mi Zhang. 2024. Efficient large language models: A survey. Trans. Mach. Learn. Res., 2024.
* March 3, 2021_, pages 97-110. IEEE.
* Xiao et al. (2024a) Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxiain Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. 2024a. Duoattention: Efficient long-context llm inference with retrieval and streaming heads. Preprint, arXiv:2410.10819.
* Xiao et al. (2024b) Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024b. Efficient streaming language models with attention sinks. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Yang et al. (2020) An Yang, Baosong Yang, Bingyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haorran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jianxin Jiang, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juanying Lin, Kai Dai, Zengming Lu, Kejin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruiz Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenhin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Youiqo Liu, Keyei Cui, Zhenrun Zhang, Zhifang Guo, and Zhihao Fan. 2024. Qwen2 technical report. CoRR, abs/2407.10671.
* Yang et al. (2022) Yi Yang, Chen Zhang, and Dawei Song. 2022. Sparse teachers can be dense with knowledge. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 3904-3915. Association for Computational Linguistics 

Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. 2024. LLM inference unveiled: Survey and roofline model insights. _CoRR_, abs/2402.16363.
* Zhang et al. (2023a) Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. 2023a. Towards the law of capacity gap in distilling language models. _CoRR_, abs/2311.07052.
* Zhang et al. (2024a) Chen Zhang, Yang Yang, Qiuchi Li, Jingang Wang, and Dawei Song. 2024a. Task-agnostic distillation of encoder-decoder language models. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy_, pages 15629-15639. ELRA and ICCL.
* Zhang et al. (2023b) Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, and Dawei Song. 2023b. Lifting the curse of capacity gap in distilling language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 4535-4553. Association for Computational Linguistics.
* Zhang et al. (2024b) Chen Zhang, Yang Yang, Qifan Wang, Jiahao Liu, Jingang Wang, Wei Wu, and Dawei Song. 2024b. Minimal distillation schedule for extreme language model compression. In _Findings of the Association for Computational Linguistics: EACL 2024, St. Julian's, Malta, March 17-22, 2024_, pages 1378-1394. Association for Computational Linguistics.
* Zhang et al. (2024c) Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024c. Infbybench: Extending long context evaluation beyond 100k tokens. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024_, pages 15262-15277. Association for Computational Linguistics.
* Zhang et al. (2022) Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. Moefication: Transformer feed-forward layers are mixtures of experts. In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 877-890. Association for Computational Linguistics.
* Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_, pages 19-27. IEEE Computer Society