Efficient Large Language Models: A Survey
Authors: Zhongwei Wan
Xin Wang
Che Liu
Samiul Alam
Yu Zheng
Jiachen Liu
Zhongnan Qu
Shen Yan
Yi Zhu
Quanlu Zhang
Mosharaf Chowdhury
Mi Zhang
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding and language generation, and thus have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey. We will actively maintain the repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.

Published in Transactions on Machine Learning Research (May/202)
Efficient Large Language Models: A Survey
Zhongwei Wan"
wwn.512@osu. edu
Xin Wang"
wang. 15980.55.00 eda
Che Liu'
che.liu21 @imperialmaaccccmmmmmm ac. uk
Samiul Alam"
alam. alam.1000ou.. edu
Yu Zheng
zhengy300msuu eda
Jiachen Liu
amberljc@umich. edu
Zhongnan Qu
znguBamuzon.com
Shen Yan
shenyan@gooole.cco
Yi Zhu
yp yi@bboon.aai
Quanlu Zhang"*
gusha@microosott.com
Mosharaf Chowdhury
mosharaf@umich. edu
Mi Zhang"
mizhungg @osu. edu
The Ohio State University
Imperial College London Michigan State University
University of
Michigan
Amazon AWS AT
Google Research
*Microsoft Research Asia tBoson AT
Reviewed on OpenReview: http:://pppaaeeeeeeeeeeoooooottt
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities important tasks such natural language understanding and language generation, and thus have the potential to make substantial impact
society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in taxon- omy consisting three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at https://github.c..
We will actively maintain the repository and incorporate new research it emerges. We hope survey serve as valuable resource to help rescarchers and practitioners gain systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.
LloT-MLSys-Lab/ DaatrssL////ffitttttt/eee
11 The work done outside Amazon.

---

Published in Transactions on Machine Learning Research (May/2024)
1 Introduction
Large Language Models (LLMs) are a type of advanced AI models designed to understand and generate human languages. Recently, we have witnessed surge in LLMs including those developed by Open AI (GPT-4 (Achiam et al., 2023) and GPT-3 (Brown et al., 2020)): Meta (LLaMA-3 (Meta, 2024), LLaMA- 2 (Touvron et al., 2023b), LLaMA-1 (Touvron et al., 2023a)), and Google (Gemini (Team & Google, 2023),
PaLM-2 (Anil et al., 2023), PaLM (Chowdhery et al., 2022), GLaM (Du et al., 2022)) as well as many other models such as BLOOM (Scao et al., 2023), PanGu-E (Ren et al., 2023b), and GLM (Zeng et al., 2023).
These models have demonstrated remarkable performance across a variety of tasks such as natural language understanding (NLU), language generation, complex reasoning (Yang et al., 2024),, and Nomain-specith tasks related to biomedicine (He et al., 2023; Wan al., 2023; 2022), law (Eliot, 2021) and code generation (Wei et al., 2022b; Chen et al., 2021b). Such performance breakthroughs can be attributed to their massive scales in model sizes and volumes of training data, as they contain billions or even trillions of parameters while being trained on a gigantic amount of data from diverse sources.
Although LLMs are leading the next wave of Al revolution, their remarkable capabilities come at substantial resource demands (Achiam et al., 2023; Du et al., 2022; Chowdhery et al., 2022; Ren et al., 2023b). Figure! 1 illustrates the relationship between model performance and model training time in terms of GPU hours for LLaMA series, where the size of cach circle is proportional to the number of model parameters. As shown, although larger models are able to achieve better performance, the amounts of GPU hours used for training them grow exponentially as model sizes scale up. In addition to training, inference also contributes quite significantly to the operational cost of LLMs. Figure 2 depicts the relationship between model performance and inference throughput. Similarly, scaling up the model size enables better performance but comes at the cost of lower inference throughput (higher inference latency), presenting challenges for these models expanding their reach to a broader customer base and diverse applications in cost-effective way.
The high resource demands of LLMs highlight the strong need to develop techniques to enhance the efficiency of LLMss As shown in Figure 2, compared to L.aMA-1-33R,, Mistral-7B (Jiang et al., 2023a), which uses grouped-query attention and sliding window attention to speed up inference, achieves comparable perfor- mance and much higher throughput. This superiority highlights the fcasibility and significance of designing efficiency techniques for LLMs.
The overarching goal of this survey is to provide a holistic view of the technological advances efficient LLMs. As rated in Figure 3, we organize the literature in laxonomy consisting of three main categories, covering efficient LLMs topics from model-centric, data-centric, and framework-centrric perspective, respectively. These three categories cover distinct yet interconneeted research topics, collectively providing a systematic and comprehensive review of efficient LLMs research. Specifically,
Model-Centric Methods: Model-centric methods focus both algorithm-level and system- level efficient techniques where the model itself is the focal point. With billions or even trillions of parameters, LLMs exhibit distinct characteristics (Wei et al., 2022a) compared to smaller-scale models, necessitating the development of new techniques to enhance their efficiency. In $2, we survey efficient techniques that cover research directions related to model compression, efficient pre-training, efficient fine-tuning, efficient inference, and efficient architecture design.
Data-Centres Methods: In the realm of LLMs, the importance of data is as crucial as that of the model itself. Data-centric methods focus on the role of the quality and structure of data in enhancing the efficiency of LLMs. In $3, we survey efficient techniques that cover research directions related to data selection and prompt engincering.
The advent of LLMs necessitates the development of specialized frameworks to efficiently handle their training, fine-tuning, inference, and serving. While mainstream AI frame- works such as TensorFlow and PyTorch provide the foundations, they lack built-in support for spe- cific optimizations and features crucial for LLMs. In $4, we survey existing frameworks specifically designed for efficient. LLMs, covering their unique features, underlying libraries, and specializations.
LLM Frameworks:
2

---

In addition the survey, we have established GitHub repository where we compile the papers featured We will actively maintain it Published in Transactions on Machine Learning Research (May/2024)
1.75
Number of model parameters
Training Time (Million GPU hours)
1.50
5B
10B
25B
50B
75B
LLaMA-2-70B
1.25
LLaMA-1 65B
LLaMA-2-34B
0.75
LLaMA-1-33B
0.50
LLaMA-2-13B
LLaMA-2-7B
0.25
LLaMA-1-138
LLaMA-1-7B
62
74
Performance (Commonsense Reasoning Score)
Figure 1: Illustaaionn of model performance and model training time in GPU hours of LLAMA models at dif- ferent scales. The reported performance is the average score of several commonsense reasoning benchmarks. The training time is based on Nvidia A100 80GB GPU. The size of each circle corresponds to the number of model parameters. The original data can be found in Touvron et al. (2023a;b).
Memory
5GB 10GB 50GB
80GB
Cerebras-
GPT-1.3B
Throughput (tokens/s)
Cerebras-
MPT-7B
60
GPT-2.7B
OPT-2.7B
OPT-6.7B
50
Cerebras-
XGLM-7.5B
GPT-6.7B
LLaMA-1-7B LLaMA-2-7B
40
Pythia-6.9B
Mistral-7B
XGLM-4.5B
OPT-13B
LLaMA-1-13B
30
LLaMA-2-13B
CodeGen-NL NL-6B
CodeGen-NL-16B
GPT-NeoX-20B
20
OPT-30B
LLaMA-1-33B
30
35
40
45
50
55
FaggingFace Open LLM Leaderboard Score (%)
Figure 2: Performance score inference throughput for various LLMs. The throughputs measured Nvidia A100 S0GB GPU with 16-bit floating point quantization. The size of cach circle corresponds to the memory footprint (in Gigabyles) of each model when rumning with a batch size of 1, prompt size of 256, and generating 1000 tokens. The original data can be found Tlyas Moutawwakil (2023).
in this survey at https://github.co wttpaai.co////ooo....... AIoT- Sabfffaaa/ffiiiinttttt. and incorporate new research it emerges.
3

---

Published in Transactions on Machine Learning Research (May/2024)
Wrighe-fhily Quantiration
Powt- Training Quantbarion
Quantitation
Weigle-Aelivttio Co-Qunsiaation
Quastisation-Ane Truining
Struetured Praning
Model Compreston ($2.1)
Parameter Proning
Unstructured Pruning
Low-Rau Bank Approcimation
White- Box KD
Knowledge Distillation
Black-Box KD
Mixed Precision Traiiinng
Scaling Models
Elficient Pre- Training ($2.2)
Initiallention Techniques
Triining Optimieerss
System-LLve Pre-Training
Ellcicscy Optimiestion
Low-Rank Adaptation
Frrameter-Ellkiis
Adaptreshaed Tining
Rilictent Piee-Tiniing ($2.3)
Fin-- Tuning
Prefix Tuning
Codel-Coosate (87)
Memoaty-ailliien Fin Fino-Tunin Tuning
Prompt Tuning
AlgorithmmI.ovl Inference Acoelection
Speonlative Decoding
KV-Coche Optimisatiion
Efficinn Infervmen (52.4)
System-Leeel Inferenee Accseeraion
Sharing haord Attention
Kreselization or Bank
Elliciet Alicution
Fixed Patters Strategies
Learnable Pattern Strategles
Buriitur-AAsisted Assised Aftccttion
MoE-based LLMs
Edflcceut LLMs Methods
Reperts (Mal)
Algortthm-Lewe Optimization
Edficient Architecture ($2.5)
System- Level- MoE Optimiaation
Extragulaion and Interpolation
Recarrcut Structure
Long Context LLMss
Seguectttion and Sliding Window
Momory- -tatrimal Angmentation
Thussennct-Atsraale
State Space Modals
Architectures
Other Sequentiaa Medels
Tats Selection Efficient Pre- Pre-Training
Dats Selection [511]
Data Sclection for Edfeiiett Fine- Tuning
Deuonstration Sclecton
Demonstratio Organisation
Data-Centric
Demonstration Ordering
Prre-Shor Pronsptiing
Instraction Generaton
Prompt Enginecting ($1.3)
Pronpt Compression
Tomplate Formalting
Multi-teo Rossoning
Promut Gemeration
DerpSpodd, Xlegatnn Coloeal-AL, Nanetnn, MagsBilrkke Fatrienle, Pas, Componr, Openl&M. L&M Yourdry T. TGI, R. LTM, Sot,
Prameworks (8/1
Figure 3: Taxonomy of efficient large language models (LLMs) literature.
Although there are few surveys LLMs (Zhao et al., 2023a; Chang et al., 2024; Wang et al., 20231; Kaddour et al., 2023), this survey provides focused review and discussion on the litcrature related to the efficiency aspect of LLMs. There are also surveys on efficient. Transformers (Tay et al., 2022) and their training methods (Zhuang et al., 2023). In contrast, this survey specificlly focuses on efficiency techniques designed for models of more than billions of parameters. We hope this survey together with the GitHub repository can help researchers and practitioners navigate through the literature and serve as a catalyst for inspiring further research on efficient. L.MMs.
4

---

Published in Transactions on Machine Learning Research (May/2024)
LIMCis80) (Detimmen es al, 2022), GPTQ (Pouriar et al, 2000)
ORG (Prantar & Alletah, 10009), QuIP (Chen al,, al,99013)
Weight-Onll Quantiaaioon
AWG (Lin
(Lam sl, 2023).
SyQR (Dettmerrs et al, 2004), QlarQuant (Kim et al. 20234)
Post: Training Quantization
Zeeaggone (Noo al, (002t)) Zeroquent-PP (Wo et aly s0000,
2neQuant-V2 (Noo 20290), SmmotQuun (Nioo al, 2028).
Veight-AAttvation Co- Quantization
OtVV (Cho es ol, 2029), RPTQ (Youn ol,, 20200).
Quantistionn
Alnadion et al (1005), Outller Suppresdort (Wke et nl, 2013))
(Lin 3021g]
Quastization- Awaae Troining
QuarCPT (Iho es al. 2023), LLM-QAT [Lia et al, 20233), BRNee (Wang al, 20236)
Model Compresson
LL3C
LL3C-Prucct Prunct (Ma al., 20215 Stoared LLMMA (Xin al, 2023), LoBAPruns (2haug et al,, 2003)),
Structured Pruning
LoItAStoor (Chom 20230, Deja
al, 2020)
Purumeter Pruming
Unstructaned Pruning
SpanGPP (Prantar & Albeach, 2025) Wands (Sin in ad, 2024), Sino al. (2001)
Low-Rank Approsimation
TRerGGP (Kn et al, 2003s,, LeSpara (li it al, 9003g). FWSVD (Hou at al, 2002), ASVD (Viam et al, 20220), SVD-LAM (Wbig se al., (02
Baby LLoMA (Timirysoo Tistet, 2029), MuiLLM (Ou al, 2024). KPTD (Paduandhlu al., 3n23),
Whitee Box KD
H&D (Liing dl, 2003), 1SLD (K0m 2020)), MELMA (2hang et a0sd)), GKD (Agarnal 2000)
Knowledge Distillation
MetaCC (Min ol, 20220), Multitekk ICT Olting et ol, a0zz), Li et al. (2024b), Like (lling ol, 20200),
0800 (Cher et ab, soagg, Pa es (1015))) Deeliing Stepehy-Step (Hoei 2008,
Black-Box KD
Fins-ome-Ca CoT
2023g, SOCRATIC (Staiilu 2029), SCOTT (Wams 2023.).
SCoID (Ll et al, 20200) Prng et al, (20200), Zeplex (Tunstal et al, 2023)
Figure 4: Summary of model compression techniques for LLMs.
2 Model-Centric Methods
2.1 Model Compression
Model compression enhances efficiency by reducing the sizes and the amount of arithmetic operations of LLMs. Unlike conventional model compression techniques, most LLM compression approaches are designed under the post-training setting to avoid resource-intensiio retraining. As summarized in Figure 4, model compression techniques for LLMs can be grouped into four categories: quantization, parameter pruming, low-rank approximation, and knowledge distillation. These four categories are orthogonal to each other, and compress LLMs from different perspectives.
2.1.1 Quantization
Quantization compresses LLMs by converting model weights and/or activations of high-proeision data types X such as 32-bit floating point into low-precision data types X such as 8-bit integer (Dettmers et al., 2023) as:
aassmax (x4)
x L Round
absmax (XH) X x²)
Round (KC x")
(1)
where Round denotes mapping a floating point number into an approximate integer; absmaxx denotes the absolute maximum of the input elements; and I. denotes the quantization constant. Quantization techniques for LLMs can be classified into post-training quantization (PTQ) and quantization-aware training (QAT). Compared to other LLM compression methods such as parameter pruning and low-rank approximation, quantization methods have been shown to achieve superior compression-accuracy trade-offs (Li et al., 2024c).
Post- Dost-Training Quantization (PTQ). PTQ quantizes LLMs after the model has been trained. To com- pensate for the accuracy drop, PTQ uses small calibration datasct to update the quantized weights and/or activations. PTQ in general can be grouped into two categories: weight-only quantization, and weight- activation co-quantization.
Weight- Only Quantization focuses on quantizing model weights only. For example, Dettmers al. (2022) introduce the first multi-billion-se 8-bit integers (or INT8) weight quantization method named L.IM.nt8(( that significantly reduces memory usage during inference while being able
5

---

Published in Transactions on Machine Learning Research (May/2024)
Precision Weight with Diferent Values
Pol--rreisoon Activation with Difierntt Values
Zero Weight Activation Value
Low- PreePPrei Weight Different Values
Low- Precision Activation with Difisrent
PTQ
(optional)
Calibration
Calibration
Data
Data
update
update
update
Training Data
Structured
Unstructured
update
(a) Quantization
(b) Parameter Pruning
Training Data
Training
Teacher Model (Transparent)
T
Student Model
Teacher Model (Hidden)
Decompose
X
Box KD
Box KD
(c) Low-Rank Approximation
(d) Knowledge Distillation
Figure 5: Illustrations of model compression techniques for LLMs.
to maintain the performance of the full-precision model. Frantar et al. (2023) push one step further and propose GPTQ, a post-training weight quantization method that compresses LLM weights 3 4 bits instead of bits. GPTQ employs layer-wise quantization with Optimal Brain Quanti- zation (OBQ) (Frantar & Alistarh, 2022) to update weights with inverse Hessian information. This technique enables quantizing GPT models with 175 billion parameters in roughly four GPU hours with minimal accuracy drop compared to the original model. Furthermore, driven by the insights that quantization can be more effective when model weights and proxy Hessian matrices are incoher- ent, Chee et al. (2023) propose QulP, a post-training quantization method that applies incoherence processing to quantize LLMs to 2 bits per weight. As another line of research under weight-only quantization, Lin et al. (2023) observe that there exists small subset of model weights, character- ized by larger activation magnitudes, known salient weights, play a crucial role in determining the quantization loss. Based on this observation, they propose approach named activation-awarr weight quantization (AWQ) to quantize LLMs while preserving the salient weights in high pre- cision, demonstrating superior performance over GPTQ. Similarly, Lee et al. (2023) observe that activation outliers amplify weight quantization loss. They propose outlier-aware weight quantiza- tion (OWQ) to identify those vulnerable weights with activation outliers and allocate high-precision to them. Dettmers al. (2024) introduce Sparse- Quanse--uattizedd Representation (SpQR) separate outlier weights that prone to large quantization errors. These outlier weights preserved higher precision, while the remaining weights are compressed to 3-4 bits. Additionally, they intro- duce a decoding scheme tailored for the SpQR format that enhances the elficiency of inference on a token-by-tee by-token basis. Lastly, Kim et al. (2023d) aim to address the issue of outliers that distort the distribution of quantized weights, and propose FineQuant that employs an empirically crafted, heuristic- based approach to allocate varying levels gramularity to different weight matrices.

---

Published in Transactions on Machine Learning Research (May/200) /2044
Weight-Activation Co- Co--antttzation differs from weight-only quantization in the sense that it quantizes both model weights and activations. For instance, Yao et al. (2022b) propose ZeroQuant, which combines group-wise quantization for model weights and token-wise quantization for activa- tions. However, ZeroQuant falls short in maintaining accuracy for models with more than 175 billion parameters. To address this issue, Yao et al. (2023d) and Wu el al. (2023b) propose ZeroQuant-FP and ZeroQuant-V2 respectively, both of which utilize low-rank matrices recover the accuracy drop. A key challenge of weight-activation co-quantization is that due to the existence of outliers, activations are more difficult to quantize than model weights (Bondarenko et al., 2021). To address this challenge, Xiao et al. (2023) propose SmoothQuant which introduces a per-channel scaling trans- formation that migrates the quantization difficulty from activations to weights to achieve lossless quantization of weights and activations to 8 bits for LLMs up to 530 billion parameters. Guo et al. (2023) pinpoint that outliers are critical in weight-activation co co-quantization but their nearby normal values are not. Given that, they propose OliVe, which pruncs normal values adjacent to the outliers so that the outliers can be encoded with higher precision. Yuan et al. (2023a) iden- tify the challenge of quantizing activations when different channels have disparate ranges. They propose RPTQ, which groups channels in activations that have similar value ranges and applies uniform quantization parameters to the values in each group. Ahmadian et al. (2023) demonstrate that it is possible to suppress large activation outliers at scales as large as 52B. Given the right optimization choices during pre traiiinnn they can quantize models ranging in size from 410M to 52B with minimal accuracy degradation. Wei et al. (2023) observe that the activation outliers in LLMs are asymmetric and tend to cluster in particular channels. Based on this observation, they propose Outlier Suppression+,, which introduces operations that shift and scale channels individually to neutralize asymmetric outliers. Lastly, Liu et al. (2024c) propose QLLM, an adaptive channel reassembly method that tackles activation outliers and utilizes calibration data to offset the in- formation loss incurred from quantization. Experimental result shows that QLLM achieves better compression performance than SmoothQuant and Outlier Supprcssion+ on LLaMA model family.
Quantization-Aware Training (QAT). Different from PTQ, QAT quantizes LLMs during the training process, allowing LLMs to learn quantization- -iaiiizttion-rieedlly representations. Since QAT requires training using the complete training dataset, it is much more expensive and time consuming than PTQ. Tao et al. (2022) propose QuantGPT, which combines contrastive distillation from a full-precision teacher model and logit distillation to a quantized student model during autoregressive pretraining. Quant achieves 14.4x and 13.4x compression rates on GPT-2 and BART with comparable performance with the full-precision models. LLM-QAT (Liu et al., 2023d) uses data generated by LLMs itself to distill knowledge with the objective of quantizing student model. Specifically, LLM-QAT retains the original output distribution and is capable of quantizing a model irrespective of its initial training data. Besides quantizing weights and activations, LLM- QAT also quantizes the key-value cache, a crucial step for enhancing throughput and accommodating long sequence dependencics in LLMs. Experimental results show that LLM-QAT achieves better performance over training-free methods especially in low-bit seltings. Lastly, BitNet (Wang el al., 2023b) pioneers QAT for 1-bit LLMs. It proposes to use low-precision binary weights and quantized activations while keeping optimizer states and gradients high-precision during training. Experimental results show that compared to FP16 Transformer baselines, BitNet is able to achieve competitive performance while substantially reducing memory footprint and energy consumption.
2.1.2 Parameter Pruning
Parameter pruning compresses LLMs by removing redundant less important model weights. Parameter pruning methods for LLMs can be categorized into structured pruning and unstructured pruning.
Structured Pruning. Structured pruning focuses on pruning structured patterns such as groups of consec- utive parameters or hierarchical structures such as rows, columns, or sub-blocks of the LLM weight matrices. For instance, LLM-Pruner (Ma et al., 2023) introduces a task-agnostic structured pruning strategy that se- lectively climinates non-essential interconnected structures using gradient information. L.M-Pruner utilizes a small amount of data to obtain the weight, parameter, and group importance of the coupled structure for LLaMA (Touvron et al., 2023a), and uses LoRA (Hu et al., 2022) to recover accuracy after pruning,
7

---

Published in Transactions on Machine Learning Research (May/2024)
showing competitive zero-shot performance. Sheared LLaMA (Xia et al., 2023), on the other hand, proposes two techniques to improve the performance of L.M-Prunner The first technique, named targeted struc- tured pruning, prunes a larger model to a designated target shape by eliminating layers, heads, intermediate and hidden dimensions in an end-to-end manner. The second technique, named dynamic batch loading, dynamically configures the composition of sampled data in each training batch based on losses in various domains. Through these two techniques, Sheared LLaMA is able to prune LLaMA2-7B down to 1.3B pa- rameters, achieving superior compression ratio compared to LLM-Pruner. LoRAPrune (Zhang et al., 2023c) introduces LoRA-based pruning criterion using LoRA's weights and gradients for importance estimation. By employing an iterative structure pruning process to eliminate excess channels and heads, LoRAPrune achieves better efficiency over LLM-Pruner at 50% compression rate. Lastly, given the input, Deja. Vu (Liu et al., 2023() predicts a small set of attention heads and MLP parameters, referred to as contextual sparsity, yields approximately the same output as the dense model. By exploiting such contextual sparsity, Deja Vu is able to achieve much lower latency compared to PasterTraasformer without accuracy drop.
Unstructured Pruning. Unstructured pruning, on the other hand, focuses on pruning model weights indi- vidually. Compared to structured pruning, unstructured pruning has much more pruning flexibility and thus enjoys a lower accuracy drop. However, unstructured pruning incurs irregular sparsification, which in general makes the resulting pruned models difficult to be deployed on hardware except specific types of hardware such as Nvidia Ampere GPUs (Busato & Pool, 2020). For instance, Frantar & Alistarh (2023) introduce SparseGPT,, an one-shot LLM unstructured pruning approach that does not require retraining. SparseGITT formulates pruning as a sparse regression problem and solves it by utilizing an approximate solver based the inversion of the Hessian matrix. In doing so, SparseGPT reaches about 60% unstructured sparsity models such as OPT-135B while experiencing only slight performance drop. Sun et al. (2024) propose Wanda, which prunes weights based on the product values of weight maguiludes and their respective input activations. Compared to SparseGPT, Wanda neither relies on second-order information nor necessitates weight update, and is able to achieve competitive performance. Shao et al. (2024) improve the performance of SparseGPT in another way. Specifically, instead of performing the unstructured pruning with a unified ratio for every layer, they propose to utilize Hessian sensitivity-aware mixed sparsity pruning to achieve a minimum of 50% sparsity in LLMs without retraining. This method adaptively assigns sparsity based on scnsitivity to minimize the error induced by pruning while preserving the overall level of sparsity.
2.1.3 Low-Rank Approximation
Low-rank approximation compressos LLMs by approximating the LLM weight matrix Wmx* with smaller low-rank matrices U and V such that W ~ UVT, where U E Rmxr, V E Raxr, and r is typically much smaller than m, n. In doing so, low-rank approximation reduces the number of parameters and enhances efficiency. For example, Xu et al. (2023a) introduce TensorGPT which compresses the embodding layers of LLMs using Tensor- Train Decomposition (TTD). It transforms and breaks down each token embedding and creates an efficient embedding format named Matrix Product State (MPS) that can be efficiently computed in a distributed manner. LoSparse (Li et al., 2023g) improves the performance of TensorGPP by compressing the coherent and expressive components within neurons through low-rank approximation while eliminating the incoherent and non-expressive elements through pruning. As another line of research, FWSVD (Hsu et al., 2022) compresses the weight matrix of an LLM instead of the token embedding matrix via low- rank approximation. Specifically, instead of using vanilla singular value decomposition (SVD), FWSVD proposes a weighted SVD approach which uses Fisher information to weigh the importance of the weights for compression. While FWSVD demonstrates competitive compression results under low compression ratios, it requires calculating the gradients based on the training dataset of the target task to estimate the importance scores, which is task-specific and demands significant computation resources. In contrast, ASVD (Yuan et al., 2023b) proposes training-free SVD-based approach. It scales the weight matrix based on the activation distribution that enhances the decomposition accuracy and efficiency for model compression. However, neither FWSVD nor ASVD directly correlate singular values with compression loss. Consequently, truncating the smaller singular values might result in increased compression loss. SVD-LLM (Wang et al., 2024c) addresses this drawback by incorporating a truncation-aware data whitcning stratogy that establishes a direct mapping between singular values and compression loss. Experimental results demonstrate the superiority of SVD-LLM over FWSVD and ASVD in terms of compression performance and speed.
8

---

Published in Transactions on Machine Learning Research (May/2024)
2.1.4 Knowledge Distillation
Knowledge Distillation (KD) compresses L.MSs by transferring knowledge from large teacher L.M.M to smaller student LLM. Though effective, compared to other LLM compression methods, knowledge distillation methods ineur a resource-demanding distillation process. In general, KD for LLMs can be categorized inlo white-box KD methods and black-box box KD methods.
White-Bot Knowledge Distillation. White-box KD refers to KD techniques where the paramoters or logils of the teacher LLM are used in the distillation process (Gou el al., 2021). For example, as a pioneering effort in this direction, Baby LLaMA (Timiryasor & Tastet, 2023) trains an ensemble of GPT-2 and collection of smaller LLaMA models using the BabyLM dataset of 10M words. This ensemble is then distilled into compact LLaMA model with 58 million parameters, which outperforms both its original teacher models as well as comparable model that was trained without the use of distillation. Gu et al. (2024) observe that conventional KD objectives, such as Kullback-Leible divergence (KLD), may not be suited for open text generation tasks due to their complex output spaces compared to classification tasks. To address this issue; they propose MiniL.M which minimizes reverse KL.D using the gradient of the objective function through policy gradient techniques (Sutton et al., 1999). Experimental results show that MiniLLM achieves better accuracy than conventional KD or directly fine-tuning student models. KPTD (Padmanabhan et al., 2023) demonstrates that white-box KD can transfer and disseminate knowledge from entity definitions into the parameters of a pre-trained language model. Specifically, KPTD creates a transfer set by prompting the language model to generate text based on the definition of the entity. The model parameters are then updated to align the distribution of the student model with that of the teacher model. TED (Liang et al., 2023) introduces a technique for layer-specific task distillation. It uses specially designed filters to align the internal states of both student and teacher models in cach layer. These filters extract relevant knowledge from the internal states that is beneficial for the specific task. TED shows considerable and steady gains in performance on both continual pre-training and fine finiitning. TSLD (Kim et al., 2023b) leverages token-level distillation to enhance QAT. It addresses the limitations of layer-to-layer KD in token prediction recovery by reforming intermediate representation and has successfully applied QAT to LLMs. MiniMA (Zhang et al., 2023a) proposes a viewport towards the capacity gap in distilling LLMss, converting it into a principle through analysis and introducing a 3B Language Model that sets a new benchmark for compute-performance pareto frontier. Experimental results show that MiniMA achieves the best accuracy compared to other 3B distilled LLMs. Lastly, Generalized knowledge distillation (GKD) (Agarwal et al., 2024) addresses the issue of distribution mismatch by drawing output sequences from the student model during training. GKD can be applied in combination with other distillation methods to improve their compression performance.
Black- Box Knowledge Distillation. Different from white-box KD, in black-box KD, only the outputs generated from the teacher LLM are used in the distillation process. Inspired by ICT (Chen et al., 2022c) and MetaICL (Min et al., 2022a), where the language model is meta-trained under a wide range of tasks using in-context learning objectives and then fine-tuned for unseen tasks through in-context learning, Multitask- ICT (Huang et al., 2022) introduces a concept known as in-context learning distillation to transfer the few-shot learning capabilities from the teacher model to the student model. Experimental results show that under Mullitask-ICT, in-context learning objectives achieve the best performance when combined with language modeling objectives. Similarly, Li et al. (2024b) introduce a hybrid prompting technique that employs multi-task learning along with explanations generated by GPT-3 text-davinci-002 version (OpenAI, 2023). This method is used to distill explanations into smaller models, achieving consistent and significant improvements over strong single-task fine-tuning benchmarks in different scenarios. Experiments on multiple reasoning tasks show that this method even perform better than finetuning or prompting a 60x larger GPT-3 (175B) model by up to 9.5% in accuracy. Lion (Jiang et al., 2023b) introduces an adversarial distillation architecture aimed at enhancing the efficiency of knowledge transfer by incrementally improving the skill level of the student model. Specifically, it prompts LLMs to recognize challenging instructions and creates new complex instructions for the student model, thereby establishing a three-phase adversarial cycle involving imitation, discrimination, and generation. Experimental results show that Lion-13B not only achieves comparable open-ended generation capabilitics to ChatGPT but surpassos conventional instruction- tuned models. DISCO (Chen et al., 2023j) prompts general LLM to produce phrasal perturbations. These generated perturbation are then filtered by specialized teacher model to distill high-quality counterfactual
9

---

Published in Transactions on Machine Learning Research (May 2024)
Table Pre-training costs of representative LLMs.
Model
Parameter Size
Scale
GPUs Cost
Training Time
GPT-3 (Brown al, 2020)
175B
300B tokens
GPT-NeoX-20B (Black al,, 2022)
20B
825GB corpus
96 A100-40G
OPT (Zhang et al, 2022a)
175B
180B takens
992 A100-80G
BLOOM (Seao et al. 2023)
176B
306B tokens
381 A100-S0G
103 days
GLM (Zeng al., 2023)
130B
400B takens
786 A100-40G
days
LLaMA (Touvron el al., 2023a)
65B
1.4T lokens
2048 A100 80G
21 days
LIaMA-2 (Touvron et al., 2025b)
70B
21 tokens
A100-80G
71 680 GPU days
Gopher (Rae et al., 2022)
280B
300R tokens
1024 A100
18.4 days
LaMDA (Thoppilan et ol., 2022)
137B
768B takens
1024 TPU-33
57.7 days
GLaM (Du et al, 2022)
1200B
280B tokens
1024 TPU-v
374 hours
PanGu-o [Zeng et al., 2021)
13B
1.1TB corpus
2048 Ascend 910
PanGu-E (Ren et al., 2023b)
1085B
329B tokens
512 Ascend 910
100 days
PaLM (Chowdhery et al., 2022)
540B
780B tokens
6144 TPU-v1
PaLM-2 (Anil al., 2023)
3.6T tokens
TPUv1
WeLM (Su al,, 2023a)
10B
300B tokens
128 A100-10G
21 days
Flau-PaLM (Chung et al,, 2022)
510B
512 TPU-v1
37 hours
AlexaTM (Soltan et al., 2022)
20B
1.3 tokens
128 A100
120 days
Codegeex (Zheng al., 2023)
850 tukens
1536 Ascend 910
days
MPT-7B (Team, 2023)
7B 1T tokens
data into smaller student models, allowing the smaller models to learn causal representations more reliably. As another line of research, some studies have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks step by step, with the aim to transfer such ability from large models into smaller ones through black-box KD. For example, to enhance the CoT math reasoning capabilities of smaller models, Fu el al. (2023b) propose a method for instruct-Luning a student model (5lanT55 by distilling the reasoning pathways found in the GSM8K dataset from teacher model (GPT- 3.5 code-davinci-002 (Chen et al., 2021b)). Fine-tuning and distilling smaller models require substantial amounts of training data to match the performance of the large model. To address this issue, Hsich et al. (2023) propose Distilling Step-by-Step. a technique that uses CoT prompting to extract L.M.M rationales for extra guidance in training smaller models within multi-task setting. Experimental results show that Distilling Step-by-Step achieves better performance with much fewer labeled or unlabeled training examples compared to both fine-tuning and standard distillation. zero-shot CoT prompting techniques (Kojima et al., 2022) to create rationales from LLMs. These rationales are then used to fine-tune smaller student models. It also introduces diverse reasoning, a method that employs stochastic sampling to generate a variety of reasoning solutions from teacher models, which serves to enrich the training data for the student models. SOCRATIC CoT (Shridhar et al., 2023) breaks down the original problem into a scries of smaller sub-problems and utilizes this dccomposition to direct the intermediate steps of reasoning. It is used to train pair of smaller, distilled models: one specializes in dissecting the problem and the other focuses be an effective alternative to CoT, enabling a much smaller model (GPT-2 large) to outperform a 10x larger model (GPT-3 6B). SCOTT (Wang et al., 2023c) uses rationales generated by LLMs to train a student model under a counterfactual reasoning framework. It ensures that the student model does not overlook the provided rationales, thereby preventing it from making inconsistent predictions, Experimental results show that SCOTT can generate CoT rationales that are more faithful than original CoT prompting. Li et al. (2023b) present method called symbolic CoT distillation (SCoTD) that draws CoT rationales from LLM using unlabcled data instances. A smaller model is then trained to predict both the sampled rationalcs and the associated labels. Lastly, Peng et al. (2023a) utilize GPT-4 as leacher model to generate English and Chinese instruction-basee datasets to refine student LLMs. They show that the 52K data points generated by GPT-4 are able to improve zero-shot performance compared to instruction-followie data generated from previous state-of oftte-tf- the-art models.
Tine-tune-Cofe (Ho et al., 2023) utilizes existing
solving these sub- nub-problems. SOCRATIC COT is shown to
10

---

Published in Transactions on Machine Learning Research (May/2024)
AMP (Alicilaniiiis ei 2018; Flosbook Al Rosnach (FAIR), 2022; Rao el 2012))
Precision Training
Drain Floating Potur (BPLNNII)) (Kolamkar et ol, 2019; Durgess et 2019)
Etficieut Pre- Drainiigg
Prograsiio Slarking (Gong al, 2019), MSLT (Yang al, 2000), ComponiiSrrw (Gi et 2021), beet (Chon 20220) Krowledg Inhorizance (Qin et nl, 2022), Staged Thaining 25hmn et al, 2022), L&GO (Wang es al,, 20031), Mongo (Pai et al, 20253, Yao sl. (2012)), Geowth Sienlogy (Li e ad, 20226)
Scaling Models
Kimar (20171, Pocup (2hong et al, 2019), ZerO (Zhoo eo al, 2000), Skiphit (De Smoth, 2020), Rollon (Hartiiscime et sl, 2021), T-Fixap (H augg st sh, 2020), DhspNol (Wiugg et al,, 202)
Init Initialization Techiiques
Training Optimizers
Lion (Closs sl., 2023)), Saphis (Iin et all, 2026a)
System-Lett Level Pre- Trainninn
ZeRO (Baghhandart ol, 2020), PSDP (Zhao es al, 2020e), Zeht Offead (Ria it al,, 2021), Zett Infinity (Tajhhondan er al,, 20213 Zeras (You es al. 2023), Pensous (Chnnn ot ol., 2029;
Efficiency Optimisation
Figure 6: Summary of efficient pre-training techniques for LLMs.
Training Data
New Layer
PrecisProi Weight/ Gradient
Old Layer
Low-Precision Weight/ Activation Gradient
Forward
Forward
Update
Convert
Backward
Progressive Update
(a) Mixed Precision Training
(b) Scaling Models
Initial Model
Weight:
Training
Efficient Optimizer
Activation:
Training Data
Gradient:
(c) Initialization Techniques
(d) Training Optimizers
Figure 7: Illustrations of efficient pre-training techniques for LLMs.
2.2 Efficient Pre- Tre-Training
As shown in Table 1, pre-training LLMs incurs significant costs. Efficient pre-training techniques focus on roducing the costs of the LLM pre-training process in terms of compute resources, training time, memory and energy consumption. As summarized in Figure 6, enhancing the efficiency of pre-training can be achieved through different and complementary techniques, including mixed precision acceleration, scaling models, initialization techniques, training optimizers, and system-level pre-training efficiency optimization.
Mixed Precision Training. Mixed precision training enhances pre-training efficiency by using low- precision models for forward and backward propagation and then converting the calculated low-precision gradients to high-precision ones for updating the original high-precision weights. For example, Micikevicius al. (2018) propose Automatic Mixed Precision (AMP) to keep master copy of weights in full-precision (FP32) for updates, whereas weights, activations, and gradients are stored in FP16 for arithmetic operations. Notably, the improved version of AMP (Facebook AI Research (FAIR), 2023) has climinated the copy of FP32 weights, but the optimizer (AdamW)) still uses FP32 internally. Meanwhile, Rae et al. (2022) demon- strate that FP16 AMP results in accuracy loss due to the restricted numerical range. To address this issue, Brain Floating Point (BFLOAT16), which has a greater dynamic range i.c., number of exponent bits than FP16, was proposed (Kalamkar al., 2019; Burgess et al., 2019) to achieve better training performance.
11

---

Published in Transactions on Machine Learning Research (May/200) /2044)
Scaling Models. Techniques based scaling models accelerate pre-training convergence and reduce train- ing costs by leveraging the weights of smaller model to upscale to larger one. For example, Gong et al. (2019) introduce a technique named progressive stacking to transfer knowledge from a simpler model to a more complex one to enhance model training efficiency. Mcanwhile, Yang et al. (2020) observe that as the depth of the model increases through progressive stacking, the training speed however decreases. To address this issue, they propose multi-stage layer training (MSLT), which only updates the output and newly intro- duced top encoder layers while keeping the previously trained layers unchanged. Once all the layers have been trained, MSLT fine-tunes the entire model by updating each layer with 20% of the total steps, making it more time-efficient than the traditional progressive stacking approach. Similarly, Gu et al. (2021) intro- duce CompoundGrow, which begins with training a small model and then incrementally expands it using a mix of model growth techniques, including increasing input length, model breadth and depth, leading lo an acceleration in the pre-training process in wall-clock time compared to progressive stacking. Chen et al. (2022b) propose bert2BERT, which applics funetion-preservee initialization (AKI) to transfer the knowledge of a smaller pre-trained model to a large model to improve the pre-training efficiency of the large model. Specifically, FPI enforces the initialized larger model to closely mirror the behavior of the smaller model, laying strong basis for later optimization; and AKI promotes faster convergence by replicating weights from higher layers. Experimental results show that bertBBERT is able to save a significant amount of training cost over MSLT. Qin et al. (2022) propose Knowledge Inheri- tance which employs knowledge distillation as an auxiliary supervision during pre-training. This facilitates training a larger model from a smaller teacher model, thereby enhancing both the pre-training speed and the generalization ability. Shen et al. (2022) introduce Staged Training that begins with a small model and progressively increases its depth and breadth through growth operator. By starting each stage with the results from the previous one, it effectively reuses computation, leading to more efficient training process compared to previous techniques like CompoundGrow and progressive stacking. Wang et al. (2023d) propose Linear Growth Operator (LiGO) that linearly maps the parameters of a smaller model to initiate a larger one. By using a composition of width-and depth-growth operators further enhanced with Kronecker factorization to capture architectural knowledge, LiGO outperforms ber(2BERT which saves about 30% computation costs. Pan et al. (2023) introduce technique named Mango which establishes linear relationship between each weight of the target model and all weights of the pretrained model to boost acceleration capabilities. It also employs multi-linear operators to decrease computational and spatial complexity during pre-training, achieving 59.9% acceleration ratio compared to Chen et al. (2022b) and LiGO. Drawing from these scaling techniques and the progressive pre-training (Yao et al., 2024), recent LLMs like FLMM101B (Li et al., 2023e) introduce a growth strategy to cut LLM training costs by expanding model structures offline and resuming from the previous stage's smaller model checkpoint.
initialization (FPI) and advanced knowledge
Initialization Techniques. Initialization plays a key role in enhancing the efficiency of LLM pre-training because good initialization can accelerate the convergence of the model. Most LLMs employ initialization techniques that were adopted in training smaller-scale models. For example, initialization method introduced by Kumar (2017) balances input and output variances. Fixup (Zhang et al., 2019) and ZerO (Zhao et al., 2022) set the backbone to zero, preserving signal identity. Skiplnit (De & Smith, 2020) substitutes batch normalization with a zero-value multiplier. ReZero (Bachlechner et al., 2021) adds zero-valued parameters to maintain identity which leads to faster convergence. T-Fixup (Huang et al., 2020) follows Fixup to adopt rescaling schemes for the initialization of the residual blocks of Transformer models. DeepNet (Wang et al., 2024a) adjusts the residual connection in deep Transformers using Post-LN-init, ensuring stable inputs to layer normalization and mitigating gradient vanishing for stable optimization.
Training Optimizers. Popular LLMs such as GPT-3 (Brown et al., 2020), OPT (Zhang et al., 2022a), BLOOM (Scao et al., 2023), and Chinchilla (Hoffmann et al., 2022) are predominately pre-trained using Adam (Kingma & Ba, 2017) or AdamW (Loshchilov & Hutter, 2019) as optimizers. However, both Adam and Adamn are memory hungry and computationally expensive. Some studies (Chen et al., 2023g; Liu et al., 2024a) propose new optimizers to accelerate LLM pre-training. Specifically, Chen et al. (2023g) propose to leverage search techniques to traverse large and sparse program space to discover optimizers for model training, The discovered optimizer, named Lion (EvoLved Sign Momentum), is more memory-efficient than Adam as it only keeps track of the momentum. Liu et al. (2024a), on the other hand, propose Sophia as a lightweight second-order optimizer that outpaces Adam with doubling the pre-training speed. Sophia
12

---

Published in Transactions on Machine Learning Research (May/2024)
LoRA (Hu al, 2022), LoRA-FA (Zinnng al,, 20256)),
Localaa (Mlang et al, 2020), LongLoRA (Chen et al., 2023),
Low-Rank Adaptation
Multé Head Routing (Caotis et al., 2023), AdaLoRA (Zhang et al., 20234).
DyLoRA (Vollpour et al. 2025), CRPT (Zhao et al, 20231),
Tied (Reudochintala et al, 2020)
L.M-Arapters (Ill et al,, 2023h), Compacter (Karimi Mahabadi al. 2021).
[IA) (Lin et al,, 2022s), Mela- Adapters (Bamed et al., 2022), AdaMix (Wang ot al,, 2022e), OpenDelta (lil ot al,, 2023s),
Adapter-based Tuning
Sarameter-Efflcien Fino-Tuning
SpansAdapter (He es sl, 20226)
Effiiient Fine- Fine-Tunin Tuning
Prefis-Tuaing (LL Linng, 2021), MaMMA-Aaatee (Zhong al,, 2024)
Prefix Tuning
Hyper (Phang el sd. 2023)
Prompt Toning (Lester al, 2021), P-Tiiin (Lii al,, 20231),,
P-Tming (Lin nl., 2022g), Tam al. (2025), MP (Sun nl,, 2023e),
Prompl Turing
(Ga al,, 2022%), Maltitask Prompt Tuning (Wing et al, 2022)),
Xu al. (20238)
QloRA (Deitmers al., 2023), QA-LoRA (Xu el al,, 20245), Loaq (1i al, 20246), PEQA (Iiim et atl, 2025s), Selective Fine- Taning (Smoonli 2023), LOMO (Lx et ol,, 2025), MeZ0 (Malledi al,, 2023), (2023g)
Memory-Efflciet Fino-Tuning
Figure 8: Summary of efficient fine-tuning methods for LLMs.
calculates the moving average of gradients and the estimated Hessian, dividing the former by the latter and applying clement-wise clipping- It effectively moderates update sizes, addressos non-convexity and rapid hessian changes, enhancing both memory utilization and efficiency.
System-Level Tre-Training Efficiency Optimization. Due to high demand memory and compute resources, LLMs are usually pre-trained across multiple compute nodes in distributed manner. Therefore, most system-level optimization Lechniques are designed in the setting of large-scale distributed training. For instance, Zero Redundancy Data Parallelism (ZeRO) (Rajbhandari et al., 2020) provides three stages of optimization to partition various training states across different devices. Specifically, ZeRO-1 only partitions the optimizer states, whereas ZeRO-2 partitions both the optimizer states and the gradients. ZeRO-3 further partitions the model parameters across devices compared with ZeRO-1 and ZeRO-2. Although runtime memory is further reduced through ZeRO-3, there is about 50% increase in communication volume. Therefore, it is recommended to use ZeRO-3 within a node to minimize the communication time while using ZeRO-1 and ZeRO-2 across nodes. Fully Sharded Data Parallel (FSDP) (Zhao et al., 2023c) shares a similar idea for optimization, and designs hybrid sharding strategy to allow users to define which nodes or processes to partition the gradients, model parameters, and optimizer states across different nodes. In the case when the weight memory exceeds the aggregated memory that can be provided by all of the compute nodes, ZeRO-Offlodd (Ren et al., 2021) enables offladding any stage of ZeRO to CPU memory, whereas ZeRO-Infinity (Rajbhandari et al., 2021) provides mechanism to offload to NVMc drives in addition to CPU memory. However, it is quile difficult to maintain performance using these two alternalives, as the dala movement between CPU and GPU is slow. Lastly, training LLMs on numerous GPUs consumes a massive amount of energy, Zeus (You et al., 2023) and Perseus (Chung et al., 2023) are proposed to optimize enorgy consumption by finding the best GPU-level configurations based on the unique LLM characteristics. The evaluation shows that Perseus reduces energy consumption of large model training by up to 30%.
2.3 Efficient JuninTuning
Efficient fine-tuning techniques focus on reducing the costs of the LLM fino-tuning process. As summarizod in Figure 8, efficient fine-tuning techniques can be grouped into parametterofficient fino-tuning (PEFT) and memory-efficient fine-tuning (MEFT).
13

---

Published in Transactions on Machine Learning Research (May/2024)
tuned
tuned (added)
Large Activation
Small Activation
Fine-Tuning
Fine-Tuning
Data
Data
reduce
LLMs
forward
fixed
fixed
backward
(a) Low-Rank Adaptation tuned
(b) Adapter-based Tuning
update
tuned
LLMs
reduce
Fine-Tuning
Fine-Tuning
Small
Data
Data
Gradient
Large Gradient
(e) Memory- Efficient Fine-nuning
Embedding fixed
fixed
(c) Prefix Tuning
(d) Prompt Tuning
Figure 9: Illustrations of paramoter-cfficient fino-tuning (a)-(d) and memory-offcicnt finc-tuning (c).
2.3.1 Parameter-Efficiens Fine-Tume Tuning
Parameter-efficien fine-tuning (PEFT) adapts an LLM to downstream tasks by freezing the whole LLM backbone and only updating a small set of newly added extra parameters. In general, PEFT methods can grouped into four categorics: low-rank adaptation, adapter-based tuning, prefix tuning, and prompt tuning.
Low-Rank Adaptation. Low-rank adaptation (ToRA) (Hu et al., 2022) is widely used PEFT approach for LLMs. The hypothesis is that the change in weights during model adaptation has a low "intrinsic rank". Hence, LoRA introduces two trainable low-rank matrices A E Rxx and B E R and adjusts the weight matrix by W < W+AW= W A B. As such, only the small matrices A and B are updated during fine- tuning, while the original large weight matrix remains frozen, making the fine-tuning process more efficient. To enhance the efficiency of LoRA, LoRA-FA (Zhang et al., 2023b) keeps the projection-down weights of A fixed while only updating the projection-up weights of B in each LoRA adapter so that the weight modifications during fine-tuning are confined to low-rank space, thereby eliminating the need to store the full-rank input activations. It achieves comparable accuracy related full parameter fine-tuning and LoRA. Building top of LoRA, LoraHub (Huang el al., 2023) explores the composability of LoRA for the purpose of generalizing across different tasks. It combines LoRA modules that have been trained on various tasks with the goal of attaining good performance on tasks that have not been seen before. LongLoRA (Chen et al., 2023i), on the other hand, exlends LoRA to the long-contexi fine-tuning scenario. introduces shift short attention (S²-Attn), which effectively facilitates context expansion, showing that LoRA effective for long context when utilizing trainable embedding and normalization. Multi-Head Routing (MHR) (Caccia et al., 2023) extends LoRA to ffisture-of-Experti (MoE) architectures. It outperforms Polytropon (Ponti et al., 2023) when operating with similar parameter allocation. Notably, it achieves competitive performance while focusing on fine-tuning the routing function alone, without making adjustments to the adapters, demonstrating remarkable parameter efficiency. Zhang et al. (2023d) observe that many PEFT techniques neglect the differing significance of varions weight parameters. To address this, they propose AdaLoRA which employs singular value decomposition to parameterize incremental updates and adaptively distributes the parameter budget based on the importance score of each weight matrix. The rank in LoRA is static and cannot be adaptively adjusted during fine-tuning. Valipour et al. (2023) propose DyLoRA to introduce dynamic low-rank adaptation method that trains LoRA blocks across multiple ranks rather than just one by organizing the representations learned by the adapter module based on their ranks. Different from the above- mentioned methods that apply LoRA-based methods to full-size LLMs, CEPT (Zhao el al., 2023b) introduces a framework that utilizes compressed LLMs. Specifically, it assesses how prevalent LLM compression methods affect PEFT performance and subscquently implements strategics for knowledge retention and recovery counteract the loss of knowledge induced by compression. Lastly, Tied-LoRA (Renduchintala al., 2023) uses weight tying and selective training to further increase parameter efficiency of LoRA.
14

---

Published in Transactions on Machine Learning Research [May/2244]
Adapter-based Tuning. Adapters are bottlencck-like trainable modules integrated into LLMs, which first down-project the input feature vector followed by non-linear layer and then up-project back to the original size (Houlsby et al., 2019). Adapter-based tuming includes both series adapters and parallel adapters. In series adapters, each LLM layer has two adapter modules added after its attention and foed-forward modules; whereas parallel adapters position two adapter modules alongside the allention and feed-forward modules within each layer of the LLM. In particular, Hu et al. (2023b) propose LLM-Adapters, which integrates scries or parallel adapters into LLMs for fine-tuning on different tasks. Karimi Mahabadi et al. (2021) propose Compacter, which unifies adapters, low-rank techniques, and the latest typer-comple multiplication layers to achieve balanced trade-off between the amount of trainable parameters and task performance compared to original Adapter method (Houlsby et al., 2019). Furthermore, (IA)² (Liu et al., 2022a) introduces a technique that scales aclivations using learned vectors. It outperforms Adapter (Houlsby et al., 2019) and Compacter on few-shot setting with better accuracy and computational efficiency. Following meta- learning principles, Meta-Adapters (Bansal et al., 2022) designs a resourco-efficient fine-tuning tochnique for the few-shot scenario where it incorporates adapter layers that have been meta-learned into a pre- trained model, transforming the fixed pre-trained model into an efficient few-shot learning framework. Meta- Adapters outperforms Adapter (Houlsby et al., 2019) at few-shot fine-tuning with less parameters to fine- tune. AdaMix (Wang et al., 2022c) takes inspiration from sparsely-activated mixture-of-experts (MoE) models (Zuo et al., 2022) and proposes a mixture of adaptation modules to learn multiple views of the given task. Compared to Adapter, it demonstrates better results on both natural language understanding and generation tasks with less learnable parameters. Lastly, OpenDelta (Hu et al., 2023a) is an open-source software library that offers a versatile and plug-and-play framework for implementing a range of adapter- based techniques, and is designed to be compatible with various LLMs architectures.
Prefix Tuning. Prefix-Tunie Tuning (Li & Liang, 2021) adds series of trainable vectors, known as prefix tokens, to each layer in an LLM. These prefix tokens are tailored to specific tasks and can be treated as virtual word embeddings. Building on top of Prefix-Tuning, LLMMA-Adapter (Zhang et al., 2024) incorporates a set of trainable adaptation embeddings and attaches them to the word embeddings in the upper layers of the LLMs. A vero-initialized attention scheme with zero gating is also introduced. It dynamically incorporates new guiding signals into LLaMA-1 while retaining its pre-trained knowledge. Different from conventional prefix tuning, HyperTuning (Phang et al., 2023) employs a hyper-model to produce task-specific parameters such as soft prefixes for a downstream model, showing improved performance through initialization from hypermodel--enerater parameters for subsequent fine-tuning.
Prompt Tuning. Different from prefix tuning, prompt tuning incorporates trainable prompt tokens only at the input layer. These tokens can be inserted either as a prefix or anywhere within the input tokens. Prompt Tuning (Lester et al., 2021) keeps the entire pre-trained model fixed while adding an extra k trainable tokens at the beginning of the input text for each downstream fask. It outperforms few-shot prompts and narrows the performance gap compared to full-model fine-tuning. P-Tuming (Liu et al., 2023b) utilizes a small number of parameters as prompts, which are processed by prompt encoder before being used as input for pre-trained LLMs. Instcad of scarching for discrete prompts, P-Tuning fine-tuncs these prompts through gradient descent and improves performance on a wide range of natural language understanding tasks compared to Prompt Tuning. Liu et al. (2022c) observe that earlier versions of prefix tuning struggle with complex sequence labeling tasks. To address this, they propose P-Tuning v2, which borrows the idcas from prefix tuning by introducing continuous prompts at each layer of the pre-trained model. This modification has proven effective in boosting performance across various parameter sizes for tasks related to natural language understanding. Tam et al. (2023) introduce efficient prompt tuning for text retrieval, updating just 0.1% of parameters and outperforming traditional full-parameter update methods in diverse domains. Sun et al. (2023a) claim that prompt. tuning tends to struggle in few-shot learning scenarios, and thus propose MP2 that pre-trains a collection of modular prompts using multitask learning. These prompts are then selectively triggered and assembled by a trainable routing mechanism for specific tasks. As a result, MP² can quickly adapt to downstream tasks by learning how to merge and reuse pretrained modular prompts. Different. from MP², PPT (Gu et al., 2022b) attributes the performance degradation of prompt tuning in few-shot learning to the poor initialization of soft prompt, and thus proposes to add the soft prompt into the pre-training stage for a better initialization. Lastly, Multitask Prompt Tuning (Wang et al., 2023j) extends Prompt Tuming and harnesses the knowledge of the various tasks through the use of prompt vectors
15

---

Published in Transactions on Machine Learning Research (May/202) /224)
in multitask learning settings. Specifically, it initially learns single, transferable prompt by extracting knowledge from various task-specific source prompis, and then applies multiplicative low-rank updates to this prompt to effectively tailor it for each downstream task. By doing this, Multitask Prompt Tuning is able to attain performance levels that are competitive compared to full-model fine-tuning methods.
2.3.2 Memory-Efficient Fine- Tine-Tuning
Different from PEFT methods which focus parameter efficiency, MEFT methods focus on memory savings during the LLMs fine-tuning process. For instance, Dettmers et al. (2023) propose QLoRA which first quantizes the model into a 4-bit NormalFloat. data lype, and then fine-tunes this quantized model with added low-rank adapter (LoRA) weights (Hu et al., 2022). In doing so, QLoRA reduces memory usage during fine-tuning without performance degradation compared to standard full-model fine-tuning. QA- LoRA (Xu et al., 2024b) improves QLoRA by introducing group-wise operators that improve quantization flexibility (each group is quantized separately) while reducing adaptation parameters (each group utilizes shared adaptation parameters). Similarly, LoftQ (Li et al., 2024d) combines model quantization with singular value decomposition (SVD) to approximate the original high-precision pre-trained weights. As a result, it offers a favorable initialization point for subsequent LoRA fine-tuning, leading to enhancements over QLoRA on both natural language understanding and generation tasks. PEQA (Kim et al., 2023a) introduces a two- stage approach to quantization-aware fine-tuning. In the first stage, the parameter matrix for each fully connected layer is quantized into a matrix of low-bit integers along with a scalar vector. In the second stage, the low-bit matrix remains unchanged, while fine-tuning is focused solely on the scalar vector for each specific downstream task. Employing this two-stage approach, PEQA not only minimizes memory usage during fine-tuning but also speeds up inference time by maintaining weights in low-bit quantized form, showing better perplexity than GPTQ (Frantar et al., 2023) with LoRA. Different from above-mentioned MEFT methods that combinc LoRA with quantization to reduce fine-tuning momory footprints, as another line of research, some studies propose MEFT methods based on gradient optimization. Specifically, Simoulin et al. (2023) propose Selective Fine-Tuning which minimizes memory usage by specifically preserving a subset of intermediate activations from the forward pass for which the calculated gradients are nonzero. Notably, this approach delivers performance equivalent to full-model fine-tuning while using just up to one-third of the GPU memory required otherwise. Lv et al. (2023) introduce T.OMO, which minimizes memory consumption during fine-tuning by combining gradient calculation and parameter updating into a single step. As such, LOMO eliminates all components of the optimizer state, lowering the memory requirements for gradient tensors to O(1). Lastly, MeZO (Malladi et al., 2023) improves the zeroth-order method (Spall, 1992) for gradient estimation using only two forward passes. This enables efficient fine-tuning of LLMs with memory requirements similar to inference and supports both full- uill-parameter and PEFT methods like LoRA (Hu et al., 2022) and prefix tuning (Li & Liang, 2021), enabling MeZO to train a 30-billion parameter model on single A100 80GB GPU,
2.4 Efficient Inference
Efficient inference techniques focus on reducing the costs of the LLMs inference process. As summarized in Figure 10, efficient inference techniques can be grouped into techniques at algorithm level and system level.
Algorithm-Level Inference Efficiency Optimization. Techniques that enhance LLM inference effi- ciency at the algorithm level include speculative decoding and KV-cache optimization.
Speculative Decoding. Speculative decoding (i.e., speculative sampling) (Leviathan et al., 2023) is decoding strategy for autoregressive language models that speeds up the sampling process by computing tokens using smaller draft model in parallel to create speculative prefixes for the large target model. Chen et al. (2023a) focus on the distributed serving sctting for LLMs and propose to run faster autoregressive model K times and then evaluate the preliminary output with the large target model. A tailored rejection sampling strategy is employed to approve a selection of the draft tokens a left-to-right order, thereby recapturing the distribution of the large target model during the procedure. Staged Speculative (Spector & Re, 2023) transforms the speculative batch into tree structure representing potential token sequences. This restructuring expedites the
16

---

Published in Transactions on Machine Learning Research (May/2024)
Speculative Decoding (Levicthon al., 2020), Chen al. (20236),
Staged Speculative (Spector 2025), BLLD (Kim et al., 2023c).
Speculative Decoding
SpecInfer (Miso al., 2024), LLMA (Yang et al., 20236),
Efficlent Inference
Algorichm-Level Inference
Modusa (Cui sl. 2024), Swotilli (2023), PaSS (Mones al,, 2023)
Efficiency Optimization
KIVI (Zirui Liu es al. 2023). KVQuaut (Hooper et al,, 2024),
KV-Cache Optimization
Howyy-Hitter Oraele (120) (Zhing al, 20230; Scissoriamdss (Lin 2023e). StremmingLLM (Xino al., 2024)
Systen-Lovel
(Shong al, 2023), Pope (2023), 8 (Jin al, 2023), Orea or al, 2022), vLLLM (Kuoo al, 2023),
Eficiency Optimization
DeepSperd Infereuce (Aminahadd et al, 2022), Flach Decoding (Dao et al., 2023), FlaehDenndnng (Hong et al,, 2023)
Figure 10: Summary of elficient inference Lechniques for LLMs.
old key, value
LLMs
LLMs
Small LMs
Check & Regenerate
new key. value
new query
Generate
cache
[Token]
Large language model
has witnessed a huge advance
(a) Speculative Decoding
(b) KV-Cache Optimization
Figure 11: naliioations of algorithm-level elficiency optimization Lechniques for LLM inference.
generation of larger and improved speculative batches. It also introduces an additional phase for speculative decoding of the initial model, thereby enhancing overall performance, showing 1.36x over standard speculative decoding. BiLD (Kim et al., 2023c) optimizes speculative decoding through two innovative techniques: the fallback policy that permits the smaller draft model to waive control to the larger target model when it lacks sufficient confidence; and the rollback policy that enables the target model to revisit and rectify any inaccurate predictions made by the smaller draft model. SpecInfer (Miao et al., 2024) extends speculative decoding and speeds up inference by employing speculative inference techniques and token tree validation. Its core idea involves merging a range of small speculative models that have been fine-tuned collectively to collaboratively forecast. the output of the large target model, which is then used to validate all the predictions. Different from speculative dccoding that neods to introduce an additional efficient drafter model to generate a draft for checking, LLMA (Yang et al., 2023b) chooses a text segment from closely related reference and duplicates its tokens into the decoder. Tt then concurrently assesses the suitability of these tokens as the decoding output within single decoding step. This approach results in speed increase of more than two times while maintaining the same generated results as traditional greedy decoding. Similarly, instead of using a separate draft. model to sequentially generate candidate output, Medusa (Cai et al., 2024) proposes to freeze the LLM backbone, fino-tune additional heads, and use a tree-based attention mechanism to process predictions in parallel to speed up the decoding process. Lastly, Santilli et al. (2023) propose parallel decoding including the Jacobi and Gauss- fixed-point iteration methods for speculative decoding. Among these methods, Jacobi decoding extended into Lookahead decoding (Fu et al., 2023c) to further enhance the efficiency.
Gauss-Seidel Seidel
KV-Cache Optimization. During inference, LLMs need to store the Key- Value (KV) pairs of the past tokens into the cache for future token generation. The size of KV cache needed enlarges mas- sively with the increase of generated token length, resulting in considerable memory consumption and long inference latency. Therefore, reducing the size of KV cache is key to enhancing inference efficiency. Existing KV-cache optimization techniques can in general be grouped into two categories. The first category is to compress the KV cache. For example, Zirui Liu et al. (2023) propose KTVI, a tuning-free free 2bit KV cache quantization algorithm which quantizes the key cache per-channel and the value cache per-token, achieving 2.6x less peak memory usage during inference. Similarly, Hooper
17

---

Published in Transactions on Machine Learning Research (May/202) /204)
et al. (2024) conduct empirical study on the impact of per-channel quantization and other types of quantization such as quantization before rotary positional embedding. Based on their findings, they propose KVQuant which combines these quantization methods to quantize the KV cache of LLaMA to 3-bit. The second category of KV-Cache optimization techniques is to evict some KVs from the cache. For instance, Zhang et al. (2023f) propose Heavy-Hitter Oracle (H₂O), a KV cache eviction strategy that formulates the KV cache eviction as dynamic submodular problem and dynami- cally retains a balance between recent and performance-criticaal tokens, improving the throughput for LLMs inference. Similarly, Liu et al. (2023e) propose hypothesis named the persistence of importance, suggesting that only tokens that were crucial at an earlier phase will have a significant impact on subsequent stages. Based on this hypothesis, they design Seissorhands which significantly reduces the KV cache without compromising model quality. Lastly, StreamingLLM (Xiao et al., 2024) incorporates window attention, where only the most recent KVs are cached into a fixed-size sliding window, into their algorithm design. Through evicting the outdated KVs, StreamingI.IM ensures constant memory usage and decoding speed after the cache is initially filled.
System-Level Inference Efficiency Optimization. The efficiency of LLM inference can also be op- timized at the system level under specific hardware architecture. For example, FlexGen (Sheng et al., 2023) is high-throughput inference engine that enables the execution of LLMs on GPUs with limited memory. It uses linear programming based search approach to coordinate various hardware, combining the memory and computation from GPU, CPU, and disk. Furthermore, FlexGen quantizes the weights and atlention cache to 4 bits, which increases the inference speed of OPT-175B (Zhang et al., 2022a) on a single 16GB GPU. Pope et al. (2023) develop a simple analytical framework to partition a model in order to scale Transformer inference basod on the application requirements. By combining it with scheduling and memory optimizations, they are able to achieve better efficiency on PaLM (Chowdhery et al., 2022) in com- parison to PnsterTransformer (NVIDIA, 2023a). Orca (Yu et al., 2022) employs iteration-level scheduling to serve batched sequences with variable output sequence length. When a sequence in a batch is com- pleted, it is returned to the user so that a new sequence can be served immediately. As a result, Orca improves GPU utilization compared to static batching, showing 36.9x throughput improvement under the same level of latency compared to PrsterTransformer. $3 (Jin et al., 2023) creates a system that is aware of the output sequence beforehand. It can anticipate the length of the sequence and arrange generation requests accordingly, optimizing the utilization of device resources and increasing the rate of production, showing higher throughput than Orca with the same number of GPUs. However, both Orca and $ lead to memory fragmentation due to their inaccurate memory provisioning for cach request. vLLM (Kwon et al., 2023) addresses the memory efficiency problem with PagedAAtenntin,, which enables the storage of contin- nous keys and values in non-contiguous memory space. Specifically, PagedAttention divides the KV cache of each sequence into blocks, each containing the keys and values for a fixed number of tokens. During atlention computation, PagedAttention kernel manages these blocks efficiently by maintaining a block table to reduce memory fragmentation. Specifically, the contiguous logical blocks of sequence are mapped to non-contiguous physical blocks via the table and the table automatically allocates a new physical block for every newly generated token. This reduces the amount of memory wasted when generating new tokens, thus improving its efficiency, showing that Paged Attention improves the throughput of popular LLMs by 2-4x with the same level of latency compared to Faster Transformer (NVIDIA, 2023a) and Orca (Yu et al., 2022). On the other hand, infinitely optimizing server-side aggregated metrics does not necessarily lead to good user experience or Quality of Experience (QoE) especially under high server load. Andes (Liu et al., 2024b) first defines QoE for the LLM--bsod text streaming services and proposes a QoF-aware serving systems to optimize QoE by prioritizing requests based on their resource demand and service acquired. DeepSpeed- Inference (Aminabadi et al., 2022) is a multi-GPU inference approach designed to enhance the efficiency of both dense and sparse Transformer models when they are contained within the collective GPU memory. Furthermore, it provides a mixed inference technique that utilizes CPU and NVMc memory, in addition to GPU memory and computation, enabling high-throughhut inference even for models that are too large to fit in the combined GPU memory. This approach demonstrates lower latency than Transforreerr under the same throughput. Flash-Decoding (Dao et al., 2023) boosts the speed of long-context inference by breaking down keys/values into smaller pieces, computing attention on these pieces in parallel, and then combining them to generate the final output. Tt outperforms Faster Transformer and Flash Attention (Dao
18

---

Published in Transactions on Machine Learning Research /224)
Shuring--asoe lased Allcntion
X0QA (Slanon,, 2019), CQA [Niasic th., 3001)
Sumforaer (Allhetti st, 2025), FlaIKA (Goptta sl,, 2325), Seatterlimin (Chen et sl., 2101a).
Kerneiaation Low-Runk
LRT (Wasttu es 2050), Pafoeaer (Chorunareki 2121), RPA (Pess al,
Lazar Trundoemrr (Kathanoporlos sh, 20000) Linferner (Warg 23010
Paglandint (2025)
Zisher 2000), Peolingfome (Phang 2021),
Kificimet Attention
Fized Puttern Stratogiis
Longionee (Balag in sl, 2010), Blockvie Traakomar (Gun sl,, 20001, Spune Triaiintr (Chilé al, 2019).
Lichtiis Mbcetion (Qia
Uyper Arteerion (flan sl, (.11)) Refoener (Iitare 901,, Spene Stakhorr Atreation (Tay el, 2000)
Laartable Pattern Strategien
Chustend Atsention (frra sl,, 2120)) ChrateeFiener [Wang al, 200tthy Boetiig Treniimtte (lloe et 2012)
Fludsttttntion (The 1001)
1001)
Göhand (lapiilin en sh, 2021), Switck Trarafoeerr (Fidts et they 2002)), Artonan al (20229, BASE Layer (Lonis
MoE-laned LLMs
Mictral 4xTB al. 2023c)
Repert Charze (2hon al,, 2022) StaliadeE: (Dua al., 2002), X-ME (Cts et 2012)
Difieieot Architecture Design)
Experts (Mol)
Algoritha--Low Lavel Mol Optinstzacion
Hoiingg Molt (Chen 20237) Flo- Moll (Shen al, 1020))
(fle al, 2011), FasterMal es eh, 2000-), DeggSpeed-Mod Moll URojtheaded ol, 5007),
Syetem- Leowl Molt Opeimization
TA-Mall (Cher st al, 20175) EdgeMMil eVi al, 2000) Theel (Ihonn et all 2013))
SmsrrhlS (Zhaz 201), Monalladde (Cale
ALaBs (Pross 0023) APOS (Sun e. th, 2000k), CLEX (Choca ul, 2024x).
Positional Extropuaation and Interpoiation
30000), NTK Ietepolation (bleet, 2000), YalTY Interpobeion (hrng en al., 2021), FIRE (Io al, (311))) PoS8 (2he et al, 2011
RoPE-P
Inaaieetee-L (Dai sl, 2009) Xendenux (WV et ol, 20222), zo-ooue OMlaciio et al. 2022).
Recurvent Structure
[Euistus et al, 5121,, Boomenl Trusformu (Ealddiss al., 2025), Retealive Netwokk (San al, 20256)
Long Contect LiMbs
Mistrll (lisgg in al,, 2125k), StraminuLLL (Xiso et al, 2004), PCW (Ratner it al. 2021),
S-guentation Sikling Window
Lougls
3000ul, SLED RAPTOR (Berila sl., 2024)
2009), Maalltikee (Clas 2009)),
Mlenecenn Transforive (We al, attz)e, Landrrack Attentuve Dilittodumn & Jagat, 2012).
Memory- Retrical Asgmennttion
Lenghers (Wing sl., 2325e), Undnafremrr (Bertads sh., 2001)
Fictzed Trandocuer (Doukoreki sl,, 2323), Xi (8006a)
Space
stitaa), Miggrnd Spoce (Clepta al. 2012), (Pts REE3)).
State Space Models
Gated Stato Space (biein et al. 2123) Blrk State Trmndiemer PPllant et si., 2000)) Nanita 2009),
Tounformne--lteeaaiie Architecture
Other Sequeutial Models
RWKY (Pong et al, 20031) Hyosa (Puli et 3029), MERABTT (YU el., 2089)
Figure 12: Summary of efficient architecture designs for LLMs.
et al., 2022) in decoding speed for very large sequences. Flash PlashDeeoding+++ (Hong el al., 2023) supports mainstream language models and hardware backends through asynchronous softmax, double buffering for flat GEMM optimization, and hcuristic dataflow, resulting in up to 4.86x and 2.18x acceleration on Nvidia and AMD GPUs respectively compared to HuggingFace implementations, showing higher speedup compared to Plash-Deooding under the same throughput.
2.5 Efficient Architecture Design
Efficient architecture design for LLMs refers to the strategic optimization of model architocture and compu- tational processes to enhance performance and scalability while minimizing resource consumption. Figure 12 provides summary of existing efforts on designing efficient architectures for LLMs.
2.5.1 Efficient Attention
The quadratic time and space complexity of attention modules considcrably slows down the pro-training, inference, and fine-tuning of LLMs (Duman Keles et al., 2023). Many techniques have been proposed to make attention lightweight for more efficient execution. These techniques can be generally categorized sharing-based attention, kernelization or low-rank, fixed pattern strategies, learnable pattern strategios, and hardware-asssiteed attention.
Sharing-based Attention. bharing-based attention accelerates attention computation during inference through KV heads sharing. For example, LLaMA-2 (Touvron al., 2023b) optimizes the autoregres- sive decoding process by using multi-query attention (MQA) (Shazeer, 2019) and grouped-query attention (GQA) (Ainslie et al., 2023). In traditional multi-head attention (MHA), each head has distinct linear
19

---

Published in Transactions on Machine Learning Research (May/2024)
Outer Loop
Multi-Head
Sharing-based
Key
d:dxN
Value
I 000000
Copy Block SRAM
Outer Loop
Nxdxd
W:NXd
Query
OKL:NX
(a) Sharing-based
Attention
Inner Loop
Compute Block
Copy
Outer Loop
on SRAM
Inner Loop
Copy
Original Weight
Value Index
Original Feature
Learned Pattern:
W1
X
Output HBM
(b) Kernlization
(c) Fixed Pattern Strategies
(d) Learnable Pattern Strategies
Inner Loop FlashAttention
Low-Rank
Figure 13: Illustrations of attention optimizations.
Figure 14: Design of FlashAttention (Dao al., 2022).
transformations for input matrix queries (Q), keys (K) and values (V), allowing for diverse representations and attention mechanisms across different subspaces. In MQA, all heads share a single set of key and value weights across all query heads. Thus, MQA speeds up the inference but could compromise output quality, To address this drawback, GQA interpolates MQA and MHA by employing one key and value heads for cach group of query heads to enhance inference quality.
Kernelization or Low-Rank. Kernelization or low-rank techniques adopted by models such Sun- former (Alberti et al., 2023), FluRKA (Gupta et al., 2023), Scatterbrain (Chen et al., 2021a), Low-Rank Transformer (LRT) (Winata et al., 2020), Performer (Choromanski et al., 2021), Random Feature Attention (RFA) (Peng et al., 2021), Linear Transformer (Katharopoulos et al., 2020), and Linformer (Wang et al., 2020) enhance the efficiency by utilizing low-rank representations of the self-attention matrix or by adopting attention kernelization techniques. Specifically, low-rank methods focus on compacting the dimensions of at- tention keys and values. For example, Linformer (Wang et al., 2020) proposes to segment scaled dot-product attention into smaller units via linear projection. Kernelization, variant of low-rank techniques, focuses approximating the attention matrix (Choromanski et al., 2020). For example, Performer (Choromanski et al., 2021) condenses softmax attention- kernels using positive orthogonal random features, outperforming Reformer and Linformer on long protein sequence benchmark. Sumformer (Alberti et al., 2023) approximates the equivariant sequence-lo-sequeei function, offering a universal solution for Linformer and Performer.
Fixed Pattern Strategies. Fixed pattern strategies adopted by models such as (Pagliardini et al., 2023), Big Bird (Zaheer et al., 2020), Poolingformer (Zhang et al., 2021), Longformer (Beltagy et al., 2020), Block- wise Transformer (Qiu et al,, 2020), and Sparse Transformer (Child et al., 2019) improve efficiency by sparsifying the attention matrix. This is achieved by confining the attention scope to predetermined pat- terns, such as local windows or fixed-stride block patterns. For instance, the attention mechanism adopted by Longformer (Beltagy et al., 2020), designed as an alternative to conventional self-attention, merges local windowed attention with globally oriented attention tailored to specific tasks. Pagliardini al. (2023) ex- pand Fltenttno (Dao et al., 2022) to support a broad spectrum of attention sparsity pattcrns, including key-query dropping and hashing- based attention techniques, achieving a multi-fold runtime speedup on top of FlashAttention on long text benchmark.
Learnable Pattern Strategies. Different from fixed pattern strategies, lcarnable pattern strategies adopted by models such as Hyper (Han el al., 2024), Reformer (Kilaev el al., 2020), Sparse Sinkhorn Attention (Tay et al., 2020), Clustered Attention (Vyas et al., 2020), ClusterFormer (Wang et al., 2022b), and Routing Transformer (Roy al., 2021) improve efficioncy by learning token relevance and subsequently grouping tokens into buckets or clusters. As an example, Hyper Attention (Han et al., 2024) proposes a parameterization for spectral approximation and employs two key metrios: the maximal column norm in the
20

---

Published in Transactions on Machine Learning Research (May/2024)
Experts
Experts
LLMs
lltss short. can do it.
Input
Output
Input
Router Network 2
good generation
Router Network
iits long. I can't do it.
sentence
sentence
Bad Generation
sentence
sentence
long But can do
(1) Extrapolation Interpolation
sentence
good generalion
Memory- Retifesal
sentence
sentence
Augmemation
H4 recument structure
sentence
(a) Mixture of Experts (MoE)
(b) Long Context LLMs
Figure 15: Illustaaioons of Mixture of Experts (MoE) and long context LLMs.
normalized allention matrix and the row norm ratio the unnormalized matrix after large entry removal. It also utilizes the learnable sort locality-sensithe hashing (sortLSH) technique and fast matrix multiplica- tion via row norm sampling. The experiment results show that HyperAttention enhances both inference and training speeds for LLMs with only minimal performance degradation, giving significant speed improvements compared to FlashAttentin (Dao et al., 2022) on long contexts.
Hardware-Assister Attention. Hardware-assisted attention focuses on developing hardware-specific tech- niques to enhance attention efficiency. For example, Fltenttt (Dao et al., 2022) reduces the number of memory access between GPU high-bandwidth memory (HBM) and GPU on-chip SRAM when calculating the attention module LLMs. Instead of transmitting the values and results between HBM and SRAM multiple times is done in the standard allention mechanism, Altenttn combines all the attention operations into one kernel and tiles the weight matrices into smaller blocks to better fit the small SRAM shown in Figure 14. As result, only one communccttin is required to process cach attention block, which significantly enhances the efficiency for processing the entire attention block. vAttention (Prabhu et al., 2024) is proposed to store KV cache in contiguous virtual memory without committing physical memory ahead-of-time. avoids the software complexity to store KV cache by leveraging CUDA support of low-level virtual memory APIs.
2.5.2 Mixture of Experts (MoE)
Mixture of Experts (MoE) represents a sparse hethodology utilized prominently in large-scale models like LLMs. operates on the principle of segmenting a designated task into several sub-tasks, and then de- veloping numerous smaller, specialized models, dubbed experts, with each honing in distinct sub-task. Subsequently, these experts collaborate to deliver consolidated output. For pre-traning or fine-tuning, MoE requires developers to manage huge number of parameters efficiently, enhancing the model's capacity and potentially its performance while keeping the computational and memory requirements relatively manage- able. For inference, MoE decreases the inference time by not engaging all experts simultancously, but rather activating only a select few. Additionally, MoF. capable of minimizing communication between devices model-distributed sccnarios by allocating cach expert to an individual accelerator; communication only necessary between the accelerators that host the router and the relevant expert model (Kaddour et al., 2023).
MoE-based LLMs. Several MoE-based LLMs have been proposed. For example, GShard (Lepikhin et al., 2021) is MoE-based LLM that offers refined method to articulate a variety of parallel computation frameworks with minor modifications to the existing model code. It also amplifies a multilingual neural machine translation Transformer model with Sparsely-Gated MoE beyond 600 billion parameters through automatic sharding. Switch Transformer (Fedus et al., 2022) brings forth a switch routing algorithm and crafts intuitively enhanced models, lowering communication and computational expenditures. encompasses up to one trillion parameters, dividing tasks among up to 2,048 experts, thereby illustrating the scalability
21

---

Published in Transactions on Machine Learning Research (May/202) /204)
and efficacy of the MoE framework. Artetxe al. (2022) scale sparse language models to 1.1T parameters, discerning superior performance up to this scale in language modeling, zero-shot and few-shot learning in comparison to dense models. This suggests that sparse MoE models are a computationally efficient substitute for traditionally employed dense architectures. Its largest MoE model outperforms its dense counterpart where the latter requires twice as nuch computation. BASE Layer (Lewis el al., 2021) defines token-to-expert allocation as a linear assignment problem, allowing an optimal assignment where each expert acquires an equal number of tokens, achieving lower validation perplexity during training relative to Switch Transformer. PanGu-E (Ren et al., 2023b) is MoE-based LLM with 1.085T parameters, transitioned from the dense Transformer model to sparse one with Random Routed Experts (RRE), and effectively trains the model over 329B tokens utilizing Expert Computation and Storage Separation (ECSS). It outperforms dense model like ERNIE 3.0 Titan Wang el al. (2021) on zero-shot test of Chinese downstream task. Lastly, Mixtral 8x7B (Jiang et al., 2023a) is a MoE with 46.7B total parameters. By leveraging the advantage of MoE architocture, Mixtral 8x7B outperforms LLaMA-2 70B on most benchmarks such as MMLU, MBPP, and GSM-8K with 6x laster inference by only using 12.9B parameters of the model per token for inference.
Algorithm-Level MoE Optimization. The efficiency of MoE-based LLMs can be improved at the al- gorithm level. Expert Choice (Zhou et al., 2022) allows experts to pick the top-k tokens instead of having tokens choose the top-k experts, implying that each token can be directed to a variable number of experts while each expert maintains a fixed bucket size. This method demonstrates higher performance in the GLUE and SuperGLUE benchmarks, and outperforms T5 dense model in seven out of the 11 tasks. StableMoE (Dai et al., 2022) identifies the issue of altering target experts for identical input during training and ad- dresses this by creating two training phases. Initially, it cultivates balanced routing strategy, which is then distilled into a decoupled lightweight router. In the following phase, this distilled router is used for a fixed token-to-expert assignment, ensuring a stable routing strategy. StableMoE shows better results than Switch Transformer and BASE Layer with lower validation perplexity on language modeling. X-MoE (Chi et al., 2022) notes that earlier routing mechanisms foster token clustering around expert centroids, indicating a tendency toward representation collapse. It proposes to estimate the routing scores between tokens and experts on a low-dimensional hyper-sphere, showing improvements over Switch Transformer on multilingual multi-task benchmark. Lifelong-Mol MoE (Chen et al., 2023f) observes that MoE increases the capacity of the model to adapt to different corpus distributions in online data streams without extra computational cost, simply by incorporating additional expert layers and suitable expert regularization. This facilitates contin- uous pre-training of a MoE-based LLM on sequential data distributions without losing previous knowledge. It outperforms other MoE models such as GShard on natural language generation and understanding tasks. Lastly, Shen et al. (2024) observe that compared to dense models, MoE gains more from instruction tuning. Based on this observation, they propose Flan-MoF,, which combines MoF. and instruction tuning to en- large language models without increasing demands in memory and compute resources while showing better zero-shot and few-shot performance compared to FLAN-T5 dense model.
System-Level MoE Optimization. The efficiency of MoF-based LLMs can also be improved at the sys- tem level. For example, FastMoE (He et al., 2021) is distributed MoE training system built on PyTorch, compatible with common accelerators. This system offers a hierarchical interface that allows both flexi- ble model design and easy adaptation to various applications, such as Transformer-Xi and Megatron-LM. FasterMoF. (He et al., 2022a) trics to address the challenges of dynamic load imbalance, inefficient syn- chronous execution mode, and congested all-to-all communication during MoE training. It first introduces performance model that predicts latency and analyzes end-to-end performance through rooffine-like methodology. Utilizing this model, it presents a dynamic shadowing technique for load balancing, concur- rent fine- ane-grained schedule for operations, and a strategy to alleviate network congestion by adjusting expert selection for model training. It outperforms FastMoF, and achieves 1.37 17.87x speedup compared to methods including ZeRO, GShard, and BASE Layer. Lina (Li et al., 2023a) also improves training efficiency and inference time by optimizing communication and load balancing in distributed MoE. Lina first prioritizes all-to-all over allreduce using tensor partitioning and pipelining to improve its bandwidth in training, and then dynamically balances the workload with token-level expert selection pattern in inference. DeepSpeed- Vyramid-Residud MoE (PR-MoE) architecture to enhance
MoE (Rajbhandari et al., 2022) has designed both the training and the inference efficiency of the MoE model parameter. PR-MoE is dense-MoE hybrid that employs residual connections to optimally utilize experts, managing to reduce the parameter size by
22

---

Published in Transactions on Machine Learning Research [May/2044)
up to 3x without sacrificing quality compute requirements. It serves massive MoE models with up to 4.5x faster and 9x cheaper inference compared to quality-equivalent dense models. DeepSpeed-MoF also proposes a highly optimized MoE inference system which enables efficient scaling of inference workloads on hundreds of GPUs, providing up to 7.3x reduction in inference latency and cost when compared with existing MoE inference solutions. TA-MoE (Chen el al., 2022a) highlights that, current MoF dispatch patterns do not fully leverage the underlying heterogeneous network environment and thus introduces a topology-aware routing strategy for large-scale MoE training that dynamically modifies the MoE dispatch pattern based on the network topology, making it outperform FastMoE, FasterMoE, and DeepSpeed-ModE EdgeMoE (Yi et al., 2023) presents an on-device inference engine tailored for MoF-based L.MMs. It optimizes memory and computation for inference by distributing the model across different storage levels. Specifically, non-expert model weights are stored directly on the edge device, while expert weights are kept externally and only loaded into the device's memory when necessary. Tutel (Hwang et al., 2023) proposes adaptive parallelism and pipelining fcatures to adapt to the dynamic workload. It employs a consistent layout for MoE param- eters and input data, supporting switchable parallelism and dynamic pipelining without any mathematical inconsistencies or tensor migration costs, thus enabling free run-time optimization, achieving up to 5.75x speedup for a single MoE' layer. SmartMoE' (Zhai et al., 2023) focuses on the automatic parallelization for MoE distributed training. In the offline stage, SmartMoE constructs a search space of hybrid parallelism strategies. In the online stage, it incorporates light-weight algorithms to identify the optimal parallel strat- egy. It achieves up to 1.88x speedup in end-to-end training over FasterMoE on a distributed training setting, Lastly, MegaBlocks (Gale et al., 2023) transforms MoE-oriented computation with block-sparse operations and creates block-sparse GPU kernels to optimize MoE computation on hardware. This leads to training time up to 40% shorter compared to Tutel and 2.4x shorter than dense models trained with Megatron-L.M.
2.5.3 Long Context LLMs
In many real-world applications, such as multi-turn conversations and meeting summarization, existing LLMs are often required to comprehend or generale context. sequences that are much longer than what they have been pre-trained with and may result in a degradation in accuracy due to the poor memorization for the long context. One direct way to address this issue is to fine-tune LLMs with similar long-sequence data, which, however, is time consuming and cnaputation-intenivee To fill this gap, new methods have been developed to enable LLMs to adapt to longer context lengths in a more efficient way. These methods can be in general grouped into four categorics: extrapolation and interpolation, recurrent structure, segmentation and sliding window, and memory-relrievall augmentation.
Positional Extrapolation and Interpolation. Standard positional encoding methods such as abso- lute positional embeddings (APE) (Vaswani et al., 2017), learned positional embeddings (LPE) (Wang et al., 2022a), relative positional embeddings (RPE) (Shaw el. al., 2018), and rolary position embeddings (RoPE) (Su et al., 2023b) have advanced the integration of positional information in LLMs. For example, LPE has been used by GPT-3 and OPT, RPE was used by Gopher (Rac et al., 2022) and Chinchilla (Hoff- mann et al., 2022), whereas RoPE was used by LLaMA-1 and GLM-130B. However, it is still challenging to train LLMs on sequences with a limited maximum length while still ensuring them to generalize well on sig- nificantly longer sequences during inference. Given that, techniques based on positional extrapolation (Press et al., 2022; Sun et al., 2023c; Chen et al., 2024a) and positional interpolation (Chen et al., 2023d; Peng et al., 2024; Li et al., 2024a) have been proposed.
Positional extrapolation strategies extend the encoding of positional information beyond what the model has explicitly learned during training. For example, ALiBi (Press et al., 2022) applies attention with linear biases to attain extrapolation for sequences that exceed the maximum length seen during training. By ap- plying negatively biased attention scores with a lincarly diminishing penalty based on the distance between the pertinent key and query, it facilitates efficient length extrapolation. Different from ALiBi, xPOS (Sun et al., 2023c) characterizes attention resolution as a marker for extrapolation and utilizes a relative posi- tion embedding to enhance attention resolution, thereby improving length extrapolation. However, these techniques have not been implemented in some of the recent LLMs such as GPT-4, LLaMA, and LLaMA- 2. CLEX (Chen et al., 2024a) proposes to generalize position embedding scaling with ordinary differential
23

---

Published in Transactions on Machine Learning Research (May/200 /204)
equations to model continuous dynamics over length scaling factors. In doing so, CLEX gets rid of the limitations of existing positional extrapolation scaling methods to enable long-sequence generation.
Positional interpolation strategies, on the other hand, reduce the scale of input position indices and extend the context window sizes, allowing LLMs to maintain their performance over longer text sequences. For example, Chen et al. (2023d) observe that extending beyond the trained context length could impair the self-attention mechanism. They propose RoPE-PI to reduce the position indices through linear interpolation, aligning the maximum position index with the prior context window limit encountered during pre-training. NTK interpolation (bloc97, 2023) modifies the base of the RoPE, effectively changing the rotational velocity of cach RoPE dimension. YaRN interpolation (Peng et al., 2024) uses ramp function to blend linear and NTK interpolation in varying proportions across dimensions and incorporates a temperature factor to counteract distribution shifts in the attention matrix caused by long inputs. Experimental results on long- text modeling shows that YaRN outperforms existing RoPE interpolation methods including RoPE-PI and NTK. FIRE (Li et al., 2024a) proposes a functional relative position encoding using learnable mapping of input positions to biases and progressive interpolation, ensuring bounded input for encoding functions across all sequence lengths to enable length generalization. It demonstrates competitive results compared to ALiBi, RoPE, and RoPE-PI on long text benchmarks. Lastly, PoSE (Zhu et al., 2024) proposes positional skip-wisc training that simulates long inputs using a fixed context window and designs distinct skipping bias terms to manipulate the position indices of each chunk. This strategy reduces memory and time consumption compared to full-length fine-tuning.
Recurrent Structure. LLMs' ability to manage long sequences can also be enhanced through recurrence structure. For example, Transformer-XL (Dai et al., 2019) presents a segment-level recurrence mechanism and utilizes enhanced relative positional encoding to capture long-term dependencies and address the long- context fragmentation issue. Memformer (Wu et al., 2022a) leverages an external dynamic memory for encoding and retrieving past information, achieving linear time and constant memory space complexity for long sequences. It also proposes Memory Replay Back-Propagation (MRBP) to facilitate long-range back-propagation through time with siguificantly lower memory requirements, achieving better results than Transformer-Xi on language modeling and image generation benchmarks. co-former (Martins al., 2022) presents Transformer model augmented with unbounded long-term memory (LTM). It employs continuous space attention framework to balance the quantity of information units accommodated in memory against the granularity of their representations, showing better results than Transformer-Xi on long text sorting and modeling. Recurrent Memory Transformer (RMT) (Bulatov et al., 2022) uses a recurrence mechanism to retain information from the past segment level by incorporating special memory tokens into the input or output sequence, and demonstrates superior performance compared to Transformer-XT, in long context modeling. Block-Reeurrent Transformer (BRT) (Hutchins et al., 2022) utilizes self-attention and cross- attention to execute a recurrent function across a broad set of state vectors and tokens so as to model long sequences through parallel computation. Lastly, Retentive Network (Sun et al., 2023b) introduces multi- scale retention mechanism as an alternative to multi-head attention. By leveraging parallel and chunk-wise recurrent representations, it enables effective scaling, achieves training parallclization and constant inference cost, and offers linear long-sequence memory complexity compared to other Transformer models.
Segmentation and Sliding Window. Segmentation and sliding window techniques tackle the issue of long-context processing by dividing the input data into smaller segments, or applying a moving window slide through the long sequence. For instance, Mistral (Jiang et al., 2023a) uses sliding window attention to handle sequences of arbitrary length with reduced inference cost. StreamingLLM (Xiao et al., 2024) identifies an attention sink phenomenon, noting that retaining the Kalue of initial tokens significantly restores the performance of window attention. Based on this observation, it suggests an efficient frame- work via merging window context and the first token, allowing LLMs trained with a finite length attention window, but have the ability to generalize to infinite sequence lengths without any fine-tuning. Parallel Context Windows (PCW) (Ratner et al., 2023) segments a long context into chunks, limiting the attention mechanism to function only within each window, and then re-deploys the positional embeddings across these windows. LongNet (Ding et al., 2023a) proposes dilated attention, which exponentially expands the attentive field as the distance increases, enabling the handling of sequence lengths of more than one billion tokens. SLED (Ivgi et al., 2023) handles long sequences by partitioning the long text into small chunks and leveraging
24

---

Published in Transactions on Machine Learning Research (May/202) (204)
pretrained short-text language models for encoding and decoding. Different from the methods mentioned above, MemWalker (Chen et al., 2023c) transforms lengthy texts into segmented summaries within tree structure, leveraging LLMs as an interactive entity for guided reading through iterative prompts. It out- performs traditional methods based on extended context, recurrence, and retricval. Similar to MemWalkor, RAPTOR (Sarthi et al., 2024) utilizes lext chunks to construct a recursive tree to enable information in- tegration from extensive documents across various abstraction levels during inference, achieving superior performance in multi-step reasoning.
Memory-Retrieval Augmentation. Lastly, several studies tackle the inference of extremely long text by employing memory-rotrieval augmentation strategics. A notable example is the Memorizing Transformer (Wu et al., 2022c), which extends the atlention context size by utilizing k-nearest--eighhbor (KNN) lookup to fetch previously similar context embeddings. Additionally, Landmark Attention (Mohtashami & Jaggi, 2023) em- ploys a landmark token to represent each block of input and trains the attention mechanism to utilize it for choosing relevant blocks. This allows the direct retrieval of blocks through the attention mechanism while maintaining the random access flexibility of the previous context, demonstrating comparable perplexity Transformer-XL while reducing FLOPs for long-conters modeling. LongMem (Wang et al., 2023e) proposes decoupled network architecture with the original backbone LLM as a memory encoder and an adaptive residual side network as a memory retriever and reader, efficiently caching and updating long-term past contexts to prevent knowledge staleness. It outperforms Memorizing Transformer on long text modeling and natural language understanding tasks. Unlimiformer (Bertsch et al., 2023) enhances the KNN-augmented Transformer by outputting attention dot-product scores as KNN distances to enable indexing of virtually unlimited input sequences. Experimental results show that Unlimiformer outperforms Memorizing Trans- former on long document summarization benchmarks. Tworkowski et al. (2023) observe that the ratio of relevant keys to irrelevant ones diminishes as context length increases. Based on this observation, they propose Focused Transformer (FoT), contrastive learning- based technique to refine the structure of the Key-Value space. Unlike Memorizing Transformer and Transformer-XT, FoT does not require training on long sequences and shows better performance on both long context and short context tasks. Lastly, Xu et al. (2024a) discover that an LLM with a 4K context window, when augmented with simple retrieval during generation, can match the performance of a fine-tuned LLM with a 16K context window using positional interpolation (Chen et al., 2023d) on long context tasks while requiring significantly less computation.
2.5.4 Transformer-Aterrnae Architectures
While Transformer-based architectures are now at the forefront of LLMs, some studies propose new archi- tectures to supplant Transformer- based architectures.
State Space Models. A promising approach that aims to substitute the attention mechanism is state space models (SSMs). SSM is formulated as z'(1)= = Ax((I= + Bu(t), v(1) = Cx(() + Du((), which maps a single- dimension input signal u(t) to an N-dimension latent state sx()) before projecting to a single-dimension output signal y(t,, where A, B, C, D are parameters learned by gradient descent (Gu et al., 2022a). Compared to attention that has quadratic complexity, SSMs provide near-linear computational complexity related to the length of the sequence. Given such advantage, a series of techniques have been proposed to improve SSMs. For example, the Structured State Space (S4) sequence model (Gu et al., 2022a) refines SSMs by conditioning matrix A with a low-rank correction. This enables stable diagonalization and simplifies the SSM to the well-studied computation of a Cauchy kernel. Diagonal State Space (DSS) (Gupta et al., 2022) improves SSMs by proposing fully diagonal parameterization of state spaces instead of diagonal plus low rank structure. To bridge the gap between SSMs and attention while adapting modern hardware, H3 (Hungry Hungry Hippo) (Fu et al., 2023a) stacks two SSMs to interact with their output and input projection, allowing it to log tokens and facilitate wequence-wide comparisons. Mehta et al. (2023) introduce more efficient layer called Gated State Space (GSS), which has been empirically shown to be 2 to 3 times faster than DSS while maintaining the perplexity on multiple language modeling benchmarks. Block-State Transformer (BST) (Pilault et al., 2023) designs a hybrid layer that combines an SSM sublayers for extended range contextualization with a Block Transformer sublayer for short-term sequence representation. Gu & Dao (2023) propose Mamba to enhance SSMs by designing a selection mechanism to climinate irrelevant data and develop hardware-aware parallel algorithm for recurrent operation, achieving 5x throughput than
25

---

Published in Transactions on Machine Learning Research (May/2024)
Data Selection
Data Selection for Edfciient Pre- Training
SSPT (Glass et al, 2020), Yao et al. (2022s), DSIR (Xie et al., 20231b), DoRtMi (Nie es al., 2023a)
(2023), Insoraction Mining (Cho 2023), DStapley (Schoch all, 2023), Tuming (Chen 20251), Alpa
Data Selection for Efficient Fine-Tuning
2024b), LIMA (Zhon al,, 2023a)
Figure 16: Summary of data selection techniques for LLMs.
Training Data
Fine-Tuning Data
Fine- Tuning
Select
Training
Select
subset
subset
LLMs
(a) Data Selection for Efficient Pre- Traininn
(b) Data Selection for Efficient Fine- Tine-Tuning
Figure 17: Illustrations of data selection techniques for LLMs.
Transformers. Ren et al. (2023a) extend SSMs and propose a general modular activation mechanism named Sparse Modular Activation 1,M), which unifies MoE, adaptive computation, dynamic routing and sparse attention, and further applies SMA to develop a novel architecture, SeqBoat, to achieve state-of-the-art quality-effciency trade-off.
Other Sequential Models. Some other architectures have been proposed to replace the Transformer layer. For instance, Receptance Weighted Key Value (RWKV) model (Peng et al., 2023b) combines the advantages of recurring neural networks (RNN) and Transformers. Such combination utilizes the effective parallelizable training feature of Transformers coupled with the efficient inference ability of RNNs, thereby effectively tackling the challenges associated with long sequence processing, outperforming Transformer- based models such as BLOOM and OPT. Poli et. al. (2023) propose Hyena, sub-quadratic alternative the attention mechanism mitigate the quadratic cost in long sequences. This operator includes two efficient sub-quadratic primitives: an implicit long convolution and multiplicative element-wise gating of the input. Hyena facilitates the development of larger, more efficient convolutional language models for long sequences and perforrs RWKV and GPT-Neo on SuperGI.UR. tasks (Wang et al., 2019). Lastly, MEGABYTT (YU et al., 2023) breaks down long sequences into fixed-sized patches akin to tokens, comprising a patch embedder for encoding, a global module acting as a large autoregressive Transformer for patch representations, and a local module for predicting bytes within a patch.
3 Data-Centric Methods
3.1 Data Selection
Data selection is a fundamental technique for enhaneing efficiency. As summarized in Figure 16, in the context of LLMs, data selection techniques have been primarily used for enhancing the efficiency of pre- training and fine-tuning.
3.1.1 Data Selection for Efficient Pre- Pre-Training
Data selection enhances LLMs pre-training efficiency by strategically selecting informative and diverse data samples during training. For example, SSPT (Glass et al., 2020) is pre-training task based on the principles of reading comprehension. It involves selecting answers from contextually relevant text passages, which has shown notable improvements in performance across various Machine Reading Comprchension benchmarks. Yao et al. (2022a) propose meta-learning- based method for selecting linguistically informative sentences which significantly elevates the quality of machine-generated translations. Xie et. al. (2023b) propose DSTR,
26

---

Published in Transactions on Machine Learning Research (May/202) /224)
data selection method based on importance resampling for both general-purposs and specialized LLMs. It calculates how important different pieces of data are within a simpler set of features and chooses data based on these importance calculations. Experimental results demonstrate that DSIR achieves similar performance to expert curation across eight different target distributions. In the context of pro-training general-domain models, DSIR outperforms random selection and heuristic filtering baselines by 2 2.5% on the GLUE bench- mark. Different from DSTR, Xie al. (2023a) design DoReMi to address the distribution shift between pre-training and downstream tasks, which is also a critical problem for training data selection.
3.1.2 Data Selection for Efficient Fine- Tuning
Data selection can also boost LLM fine-tuning efficiency given that only curated subset of examples is employed to refine the model. For example, Ivison et al. (2023) propose to use few unlabeled data samples to retrieve similar labeled ones from a larger multitask dataset, improving task-specific model training. This method outperforms standard multitask data sampling for fine-tuning and enhances few-shot fine-tuning, yielding an 2-23% relative improvement. With the success of instruction luning, many studies start focusing on the selection of high-quality instruction data to fine-tune LLMs. For example, Instruction Mining (Cao tasks. It
et al., 2023) presents a linear evaluation method to assess data quality in instruction-follveee highlights the importance of high-qualily data, showing that models trained with Instruction Mining-curated datasets outperform those trained generic datasets in 42.5% of the considered cases. TS- IS-DShapley (Schoch et al., 2023) is introduced to address the computational challenges of applying Shapley-based based data valuation to fine-tuning LLMs. It employs an efficient sampling- baased method that aggregates Shapley values computed from subsets to evaluate the entire training set. Low Training Data Instruction Tuning (LTD Instruction Tuning) (Chen et al., 2023b) challenges the need for large datasets in fine-tuning, showing that less than 0.5% of the original dataset is able to effectively train task-specific models without compromising performance. This approach enables more resource-efficiest practices in data-scarce environments, combining selective data strategies with tailored training protocols for optimal data efficiency. AlpaGasus (Chen et al., 2024b) is model fine-tuned on mere 9K high-quality data samples, which are meticulously filtered from a larger dataset of 52K. It outperforms the original model trained on the full dataset and reduces the fine-tuning time by 5.7x, demonstrating the power of high-quaiity data in instruction- ceerrucctooo------ee tine-tuning Lastly, LIMA (Zhou et al., 2023a) fine-tuncs LLMs with a small, selected set of examples, showing strong performance and challenging the need for exlensive luning. It generalizes well to new lasks, matching or exceeding GPT-4 43% of the considered cases.
3.2 Prompt Engineering
Prompt engincering (Liu et al., 2023a) focuses designing effective inputs (i.c., prompts) guide LLMs in generating desired outputs. It enhances inference efficiency by tailoring the input prompts queries to better suit the capabilities of a specific language model. When used for some simple tasks, such as semantic classification, prompt engineering can even substitute fine-tuning to achieve high accuracy (Liu et al., 2022a). As summarized in Figure 18, prompt engineering techniques can in general be grouped into few-shot prompting, prompt compression, and prompt generation.
3.2.1 Few-Shot Prompting
Few-shot prompting aims to provide LLM with a limited set of examples, referred to as demonstrations, to steer its understanding to a task it is required to execute (Wei et al., 2022a). These demonstrations are selected from the training corpus based on their similarity to the test example, and the LLM is expected to use the knowledge gained from these similar demonstrations to make the correct prediction (Dong et al., 2023), Few-shot prompting provides an efficient mechanism to use LLM by guiding the LLM to perform wide variety of tasks without the need for additional training or fine-tuning. Furthermore, an effective few-shot prompting approach can make the created prompt concise enough to allow LLMs to quickly adjust to the task in high accuracy with only slight increase of extra context, thus significantly improving inference efficiency. As illustrated in Figure 19, few-shot prompting techniques can in general be grouped into demonstration organization and template formatting.
27

---

Published in Transactions on Machine Learning Research (May/2024)
RATE (Lii thy 200203, Yotck (SU es dh, 2000), Waur is al, 120210. Mia (anz2)), MDL (Wa al,, 2020k), Zanug ot al, (202200, EPR [Ruliss al, 20033, CDR al. 20250, Wang
2000,,
Demorstration Selection
Demoustraton Organization
(2125)
Demonstratio Ordeving
Lu (2002)
Few-Shoo Prompting
Inserction Endartion: (Bluzarid 2021) Astomarte Pronup Engineer (2hers al,, 20159), Snsrrur (Wing al, 20126) (PRD (Vang al,
Instraction Geneaati
ThClee (Chen al, 20851)
Prompt Engineering
Tamplate Formal
Cheineoo-Tooogg (Wid et si, 50221), Aato (2long di, 3001s3 Self-bok (Phins al, 2023),
Balat (Nio et ol. 20250), Mona Pronzping (2lmn sh, 2001)).
Sulti-Step Roasoning
Thoulf (Yio sh, 2000k), CUT-SC (Wong 50031),
Grapl of Thoughts (Bois di sh,, 2004), Cuttmaaive CUT (Clin et 2023).
Xol (Uing 20216), Rackeso-d--Taaaaa (Nin al, 2024)
Goriing (Mi et sky 2000), AstsCompneoss (Chemllerr st sl,, 2500), PCRL. (Jing & Kin, 2015),
Pronugd Compresaton
ICAE (fl eh, Nogget (Qjn ei, 5083l), Meng (flvng 5031)
Prompe Gemeration
Astolrrampt (Siin sl, 2100)), Terglll (2hing al, altte), PrmpeGen (2hang Bi, 20022))
Figure 18: Summary of prompt engineering techniques for LLMs.
It nice restaurant. The positive.
Training Data
is nice restsurant. review positive. dining room dirly. The review is negative, This t-shirt looks cool. The review is positive.
Input
This tastes good. The review
select
This t-shirt looks cool. The review positive.
embedding
Output
dining dirty. The review negative.
This tood lastes good.
positive
Demonstration Selection
Demonstration Ordering
(a) Demonstration Organization
generation
Is the review positive or negative?
Is the review positive or negative?
It is a nioe restaurant. The sentence contains positive word "nice". The review is positive.
embedding
restaurant. review positive.
The dining room is dirty. The sentence contains a negative word "dirty". The review is negative.
The dining dirty. The is negative. This t-shir looks cool. The review is positive.
This t-shirt looks cool. The sentence contains positive word "cool". The review is positive.
LLMs
Multi-Step Reasoning
Instruction Generation
(b) Template Formatting
Figure 19: Illustaaioons of few-shot prompting techniques for LLMs.
Demonstration Organization. Demonstration organization refers to organizing the demonstrations in an appropriate way so as to form suitable prompt for inference. Demonstration organization has a significant impact on the inference efficiency since improper organization may result in the processing of a considcrable amount of unnecessary information, leading to significant slowdown. The optimization of demonstration organization comes from its two main steps: demonstration selection and demonstration ordering.
Demonstration Selection. Demonstration selection aims to choose the good examples for few- shot prompting (Dong et al., 2023). In order to generate a satisfactory result, a good selection of demonstrations may only require a few nunber of demonstrations to be used for the prompt, thus making the prompt concise and straightforward for a more efficient inference. Existing demonstration selection tochniques can be grouped into unsupervised methods (Liu et al., 2022b; SU et al., 2023; Wang et al., 2023f; Qin al., 2023a; Min et al., 2022b; Li & Qiu, 2023; Wu et al., 2023c; Zhang et al., 2022b) and supervised methods (Rubin al., 2022; Li et al., 2023f; Wang al., 2024b; Luo
28

---

Published in Transactions on Machine Learning Research (May/2024) (204))
et al., 2023). Unsupervised methods aim to select the nearest examples from the training set using a predefined similarity function, such as 1.2 distance, cosine distance, and minimum description length (MDL) (Wu et al., 2023c). For example, KATE (Liu et al., 2022b) is an unsupervised selection method that directly uses the nearest neighbors of a given test sample as the corresponding demonstrations. VoteK (SU el al., 2023) is an improved version of KATE to resolve its limitation that requires a large set of examples to achieve good performance. Unlike KATE, VotcK increases the diversity of the demonstrations by penalizing examples similar to those already selected. In contrast, supervised methods require training domain-specific retriever from the training set and using it for demonstration selection. For example, EPR (Rubin et al., 2022) is trained to select demonstrations from a small set of candidates initialized by the unsupervised retriever such as BM25 from the training corpus. UDR (Li et al., 2023() further enhances EPR by adopting a unified demonstration retriever to unify the demonstration selection across different tasks. Compared to unsupervised mothods, supervised methods often lead a more satisfying generation result but require frequent adjustment of the retriever for handling the out-of-domain data, making them relatively less elficient for inference.
Demonstration Ordering. After selecting representative samples from the training set, the next step is ordering these samples in the prompt. The order of the demonstrations also has a significant impact on the performance of the model. Therefore, selecting the right order of demonstrations can help the model quickly reach a good generation quality with fewer samples, thus improving the inference efficiency. To date, only a few studies have delved into this area. For example, Liu et al. (2022b) suggest arranging demonstrations based on their distance from the input, placing the closest demonstration furthest to the right. Lu et al. (2022) proposc to develop both global and local entropy metries and use the entropy metrics to set up the demonstration order.
Template Formatting, Template formatting aims to design a suitable template to form the prompt. A good template typically compiles all the information needed by LLMs into a brief statement, making the prompt and the entire input context as succinct as possible, thus leading to a higher inference efficiency. Template formatting can be divided into two parts: instruction generation and multi-stop reasoning.
Instruction Generation. The instruction of the template refers to short description of the task. By adding instructions to the prompt, LLMs can quickly understand the context and the task they are currently performing, and thus may require fewer demonstrations to create desirable prompt. The performance of a given task is highly affected by the quality of the instructions. The instructions vary not only between different datasets for the same task but also between different models. Unlike demonstrations that are usually included in traditional datasets, the generation of instructions is heavily dependent on human ellorts. To enhance the efficiency of instruction generation, automatic instruction generation techniques have been proposed. For example, Instruction Induction (Honovich et al., 2023) and Automatic Prompt Engincer (Zhou et al., 2023c) demonstrate that LLMs can generate task instructions. Wang el al. (2023h) propose Self-Instruct, an approach that allows LLMs to align with self- teff-generated instructions, highlighting their inherent adaptability. Experimental results show that it achieves an 33% improvement over the original model when applied on vanilla GPT3. Yang et al. (2023a) also discover that LLMs can be treated as an optimizer to iteratively generate better instructions for the target L.LM and have applied this technique to various LLMs. Experiments demonstrate that the best prompts optimized by this method outperform human- designed prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks. Chen et al. (2023h) develop TeGit for training language models as task designers, which can automatically generate inputs and outputs together with high-quality instructions to better filter the noise based on a given human-written text for fine-tuning LLMs. Despite the promise of automatic instruction generation methods, their complexity is still a major bottleneck for their real-world adoption.
Multi-Step Reasoning. Multi-step reasoning (Huang & Chang, 2023) refers to techniques that guide the LLMs to produce sequence of intermodiate steps before outputting the final answer. Compared to fine-tuming, conducting specific task reasoning directly through this way is a more effi- cient. approach. At the same time, its accuracy improvement is often not as good fine-tuning. One
29

---

Published in Transactions on Machine Learning Research (May/2024)
Demonstrations
Input
Input
Test
Test
oo
Compress
LMs
Generate
LMs
A'
B
Test
A'
Test
(a) Prompt Compression
(b) Prompt Generation
Figure 20: Illustrations of prompt compression (a) and prompt generation (b) for LLMs.
of the widely used techniques in multi-step reasoning is Chain-of-Thought (CoT) prompting (Wei et al., 2022b), which adds a series of reasoning steps into the prompt to make it more informative and comprehensive. Despite its advantages, it still difficult for CoT to ensure the accuracy of every intermediate step (Dong et al., 2023). A number of techniques have been proposed to address this issuc. For example, Auto-CoT (Zhang al., 2023g) proposes to generate the CoT step by step from LLMs. Self-Ask (Press et al., 2023) incorporates the self-generated questions of each step into COT. ReAct (Yao et al., 2023b) performs dynamic reasoning to create, maintain, and adjust high-level plans for acting, while interacting with external environments to incorporate additional information into reasoning. Least-to-Most Prompting (Zhou et al., 2023b) is new milestone in COT. It breaks down the complex question into smaller ones and answers them iteratively within the context of former questions and answers. Experimental results show that Least-to-Mont Most Prompting can boost the accuracy of GPT-3 code-davinci-D0) model with CoT prompting to 99.7% by using just 14 exemplars. Tree-of- Thooghht (ToT) (Yao et al., 2023a) expends CoT include exploration over coherent units of text and deliberates decision-making processes. It outperforms GPT-4 with CoT prompting in the Game of 24 benchmark. CoT-SC (Wang et al., 2023h) introduces novel decoding approach called self-consistency to replace the greedy decoding in CoT prompting. It starts by sampling various reasoning paths instead of just the greedy one and then determines the most consistent answer by considering all the sampled paths. Graph of Thoughts (GoT) (Besta el al., 2024) represents information produced by an LLM as generic graph, with "LLM thoughts" as vertices and edges indicating dependencies between these vertices. Experimental results show that GoT increases the quality of sorting by 62% over ToT while reducing the cost by over 31%. Contrastive CoT (Chia et al., 2023) proposes to enhance language model reasoning by providing both valid and invalid reasoning demonstrations. XoT (Ding et al., 2023b) utilizes pretrained rein- forcement learning and Monte Carlo Tree Search (MCTS) to integrate external domain knowledge into the thought processes of LLMs, thereby boosting their ability to efficiently generalize to new, unseen problems. Lastly, Skeleton of Thought (SoT) (Ning al., 2024) proposes a method that first prompts the LLM to organize the output and then parallelizes the generation of different segments, Through splitting the serial generation into two different steps, it makes the generation workload more paralleiizable Therefore, it improves hardware utilization and provides speedups across LLM, revealing the possibility of exploiting data-level organization to enhance efficiency.
3.2.2 Prompt Compression
Prompt compression (Figure 20(a)) accelerates the processing of LLM inputs through either condensing lengthy prompt inputs or learning compact prompt representations. Mu et al. (2023) propose to train LLMs distill prompts into more concise set tokens, referred to as gist tokens. These gist tokens encapsulate the knowledge of the original prompt and can be stored for future use. In doing so, is able compress prompts by up to 26x, leading to reduction in floating-point operations per second (FLOPs) by up to
30

---

Published in Transactions on Machine Learning Research (May/202) 2224)
40%. Chevalier et al. (2023) propose AntoCompressors to condense long textual contexts into compact vectors, known as summary vectors, which can then be used as soft prompis for the language model. These summary vectors extend the model's context window, allowing it to handle longer documents with much less computational cost. In particular, Cotoeompesssors can utilize long contexts to improve perplexity of both fine-tuned OPT and LLaMA-2 on sequences of up to 30,720 tokens. Jung & Kim (2023) propose Prompt Compression with Reinforcement Learning (PCRL) that employs a policy network to directly edit prompts, aiming to reduce token count while preserving performance. It achieves an average reduction of 24.6% in token count across various instruction prompts. Ge et al. (2024) propose In-context Autoencoder (ICAE), which consists of a learnable encoder and fixed decoder. The encoder compresses a long context into a limited number of memory slots, which the target language model can then condition on. With such design, ICAE is able to obtain 4x context compression. Nugget 2D (Qin et al., 2023b) represents the historical context as compact "nuggets" that are trained to enable reconstruction. Furthermore, it has the flexibility to be initialized using readily available models like LLaMA. Eperimental results show that Nugget 2D compresses context al a 20x compression ratio with BLEU score of 98% for reconstruction, achieving nearly lossless encoding. Lastly, LongLLMLingua (Jiang et al., 2024) introduces a prompt compression technique containing question-aware coarse-to- to-fine compression, document reordering, dynamic compression ratios, and post-compression sub-sequence recovery to enhance LLMs' key information perception. Experimental results show that LongLLMLingun achieves 17.1% better performance over the original prompt with 4x fewer tokens as input to GPT-3.5-Turte
3.2.3 Prompt Generation
Prompt generation (Figure 20(b)) enhances elficiency by automatioally creating effective prompts that guide the model in generating specific and relevant responses. For instance, AutoPrompt (Shin et al., 2020) proposes an automated method to generate prompts for a diverse set of tasks based on a gradient-guided search. It underscores the significance of humtn-writtenn lext in refining the quality and authenticity of data, emphasizing its pivotal role optimizing LLM performance. Experimental results demonstrate that AutoPrompt outperforms manually created prompts on the LAMA benchmark in eliciting more precise factual knowledge from LLM. TempLM (Zhang et al., 2023e) proposes to combine generative and template- based methodologies to distill LLMs into template-based generators, offering a harmonized solution for data-to-text tasks. TempLM not only reduces the unfaithfulness rate of a fine-tuned BART model from 83% to 0%, but also substantially improves upon human-wrillen ones in BERTScore. Lastly, PromptGen (Zhang et al., 2022c) considers dynamic prompt generation for knowledge probing based on pre-trained LLMs. It antomatically generates prompts conditional on the input sentence and outperforms AutoPrompt on the LAMA benchmark.
4
LLM Frameworks
LLM frameworks can be in general grouped based on whether they support the tasks of training, fine- tuning, and inference. Specifically, frameworks that support training and/or fine-tuning aim to provide scalable, efficient, and flexible ructure that improves compulation efficiency, reduces memory footprint, optimizes communication efficiency, and ensures reliability of the training/ tuinng process. Frameworks that support inference focus on optimizing inference throughput and reducing memory footprint and latency. These frameworks offer a variely of deployment features to serve LLM requests. Table 2 provides a summary of existing LLM frameworks along with their key features.
DeepSpeed. Developed by Microsoft, DeepSpoed (Rasley et al., 2020) is an integrated framework for both training and serving LLMs. It has been used to train large models like Eegatron-Turrnn NLG 530B (Smith et al., 2022) (in a joint effort with Nvidia Megatron framework) and BLOOM (Scao et al., 2023). Within Inference is the foundational library. A pivotal feature of CeanSSeee-IIfffeenne is
this framework, DeepSpeed-IIferrnncee (Rajbhandari et al., 2020; 2021), an optimization technique created to address GPU memory constraints for large model inference. ZeRO-Inference distributes model states across multiple GPUs and CPUs, providing an approach to managing the memory constraints of GPUs
31

---

Published in Transactions on Machine Learning Research MMyy22224
Table Comparison of LLM frameworks.
Framework
Training
Fine-Tuning
Inference
Key Features
DeepSpeed
3D Parallelism with ZeRt (Rasley et al., 2020), ZeRO-2 (Rajbhandari et al., 2020), ZeRO In- finity (Rajbhandari et al., 2021) and ZeRO- Offload (Ren et al., 2021), Expert Parallelism (Ra- jbhandari et al., 2022), FlashAttention (Dao et al., 2022), PagedAttention (Kwon et al., 2023), Dy- namic SplitFuse (Holmes et al., 2024), Continuous Batching (Yu et al., 2022), ZeroQuant (Wu et al., 2023b; Yao al., 2023d), INT4 Quantization (Wu et al., 2023a), XTC (Ternary quantization) (Wi al., 2022b), RLHF (Yao et al., 2023c), Kernel Optimizations, Diverse Hardware Support.
Megatron
3D Parallelism (Shoeybi et al., 2020; Narayanan et al., 2021), Sequence Paralellism (Korthikanti et al., 2023; Li et al., 2023d), Expert Par- allelism (Singh et al., 2023), FasterTrras- former (NVIDIA, 2023a), FlashAttention (Dao et al., 2022), Selective Activation Recomputa- tion (Korthikanti et al., 2023).
Colossal-AT
3D Parallelism (Xu You, 2023; Wang et al., 2023a; Bian et al., 2021), Sequence Parallelism (Li et al., 2023d), ZeRO Optimizer (Zhao et al., 2022), Auto-Parallelise (Liu et al., 2023c), Het- erogeneous Memory Management (Fang et al., 2023), Expert Parallelism (Singh et al,, 2023; Xue et al., 2023), PagedAttention (Kwon et al., 2023), FlashAttention-2 (Dao, 2024), Quan- tization (GPTQ (Frantar et al., 2023) and SmoothQuant (Xiao al., 2023)), RLHF (Grif fith et al., 2013; Singh et al., 2023).
Nanotron
3D Parallelism (Narayanan et al., 2021; Huang et al., 2019), Expert Parallelism (Singh et al., 2023), ZeRO Optimizer (Zhao el al., 2022), SSM (Gu & Dao, 2023) Support, Spectral 'Transfee Parametrization (Yang et al., 2022), DoReMi (Xie et al., 2023a).
MegaBlocks
FSDP (Zhao et al., 2023c), dropless-MoF: (Gale al., 2023), Integration with Megatron and vLLM.
FairScale
FSDP (Zhao et al., 2023c), Pipeline Paral- lelism (Huang et al., 2019), AdaScale opti- mizer (Johnson al., 2020).
Pax
Data Parallelism, Tensor Parallelism (Shoeybi al., 2020).
Composer
FSDP (Zhao al., 2023c), Elastic Sharded Check- pointing.
Continued on next page
32

---

Published in Transactions on Machine Learning Research (May/2024)
Table Comparison LLM frameworks. (Contimued)
OpenLLM
FSDP (Zhao et al., 2023c), Quantization (GPTQ (Frantar et al., 2023), AWQ (Lin et al., 2023), SqueezeLLM (Kim et al., 2024), SpQR (Dettmers et al., 2024), LLM.nn88 (Dettmers et al., 2022)), LangChain, Transformers Agents, Prometheuus Metrics.
LLM
FSDP (Zhao et al., 2023c), Continuons Batch- ing (Yu et al., 2022).
Foundry
vLLM
Data Parallelism, Tensor Parallelism (Shoeybi al., 2020), PagedAttention (Kwon et al., 2023), Continuous Batching (Yu et al., 2022), Quanti- zation (GPTQ (Frantar et al., 2023), AWQ (Lin et al., 2023), SqueezeLLM (Kim et al., 2024)); Multi-LoRa (Wang et al., 2023g).
TensorRT-
3D Parallelism, Paged (Kwon et al., 2023), Continuous Batching (Yu et al., 2022), Quantization (GPTQ (Frantar et al., 2023), AWQ (Lin al., 2023), SmoothQuant (Xiao et al., 2023)).
LLM
TGI
Data Parallelism, Tensor Parallelism (Shoeybi et al., 2020), PagedAttention (Kwon et al., 2023), Continuous Batching (Yu et al., 2022), Quantization (BitsAndBytes Dettmers (2023), GPTQ (Frantar al., 2023)).
RayLLM
Data Parallelism, Tensor Parallelism (Shoeybi et al., 2020), Continuous Batching (Yu et al., 2022), Quantization (GPTQ (Frantar et al., 2023), AWQ (Lin et al., 2023), SqueezeLLM (Kim et al., 2024)). ) Prometheus Metrics.
MLC LLM
TVM-based Compiler Accderation (Feng el al., 2023; Chen et al., 2018); Continuous Batching ( (Yu et al., 2022), Quantized Model Support.
Sax
Data Parallelism, Tensor Parallelism (Shoeybi et al., 2020), Serves Pax, JAX, and PyTorch mod- els, Slice Serving, Prometheus Metrics.
Mosec
Data Parallelism, Continuous Batching (Yu et al., 2022), Metrics.
Task Coordinator, Prometheus
Another important feature of DeepSpeed--nferennce is its deep fusion mechanism, which allows for the fu- sion of operations without the necessity for global synchronization by tiling computations across iteration space dimensions (Ren ct al., 2021; Tang et al., 2021; Li et al., 2022; Lu et al., 2023). Building on this, the DeepSpeed Model Implementations for Inference (DeepSpeed MII) module introduces Dynamic Split- Fuse (Holmes et al., 2024), which leverages continuous batching (Yu et al., 2022) and non-contiguous KV caches to enable increased occupancy and higher responsivity for LLM serving. It also supports scaling beyond tensor, data, and pipeline parallelism (3D Parallelism) with Zero Infinity (Rajbhandari et al., 2021) and efficient post-training quantization Int8 with ZeroQuant (Yao el. al., 2022b), Tnt4 (W4A4) with Wu et al. (2023a) and ternary quantization with XTC (Wu al., 2022b). Furthermore, the introduction of DeepSpeed-Chan (Yao et al., 2023c) adds chat support the framework. This module focuses on train- ing chatbot models across different scales, integrating techniques like Reinforcement Learning from Human Feodback (RLHF) (Griffith et al., 2013) with the DeepSpeed training system. Notably, its integration of the
33

---

Published in Transactions on Machine Learning Research (May/200) (202))
ZeRO-Offlor optimizer (Ren et al., 2021) facilitates training on both CPUs and GPUs, irrespective of their memory capacities.
Megatron. Megatron (Shoeybi et al., 2020) is Nvidia's efforts to streamline training and serving of LLMs such as GPT (Radford et al., 2019) and T5 (Raffel et al., 2020). It is the underlying framework used for Nvidia Megatron models (Shoeybi et al., 2020; Narayanan et al., 2021; Korthikanti et al., 2023). Central to Megatron's design is the strategic decomposition of the model's tensor operations, distributed across multiple GPUs, to optimize both processing speed and memory utilization, thus enhancing training throughput without compromising model quality (Shoeybi et al., 2020). In conjunction with 3D Parallelism, it also implements sequence parallelism and selcetive activation rocomputation (Korthikanti et al., 2023), which enhances training efficiency. Megatron also uses PesterTraassormes (NVIDIA, 2023a) for optimizing the inference process for large Transformer models and handling varying precision modes like FP16 and INT8, catering to diverse operational needs.
Colossal-AI. Colossal-A -AI (Li et al., 2023c) is framework mainly designed for large-scale distributed train- ing (Wang et al., 2023a). Colossal-AI unifies a wide range of parallelism techniques (Xu & You, 2023; Wang et al., 2023a; Bian et al., 2021) including sequence parallelism (Li et al., 2023d), auto-parallelism (Liu et al., 2023c), and Zero Redundancy Optimizer (Zhao et al., 2022). It also implements heterogeneous memory management (Fang et al., 2023) through a streamlined API. This integrated approach mitigates the steep learning curve often associated with orchestrating large-scale training in distributed environments. In ad- dition, the framework integrates several other features like quantization (Frantar et al., 2023; Xiao et al., 2023), RLHF (Griffith et al., 2013), OpenMoE, and mixed-precision training.
Nanotron. Nanotron (HuggingFace, 2023a), introduced by Huggingface, is an LLM training framework with primary focus on providing functionality with minimal overhead. As such, the library only subjectively incorporates the best optimizations required for modern LLM training requirements. Some highlights are its implementation of tensor, data, and pipeline parallelism with mueforraard-one-bekwaaa pipeline engine, ZeRO-1 optimizer (Zhao et al., 2022), FP32 gradient accumulation, parameter
and spectral n uTransfer parametrization (Yang et al., 2022) for scaling up neural networks. Nanotron also incorporates DoReMi (Xie et al., 2023a) to further speed up training.
MegaBlocks. Developed by Databricks, MegaBlocks (Gale et al., 2023) is an LLM training framework for training cestuure-of-xxeert (MoF) models. The core of MegaBlocks is dropless-MoF, a reformulation of MoE in terms of block-sparse operations, that avoids token dropping without sacrificing hardware efficiency. This design simplifies and accelerates training as it does not require the capacity factor as a hyper-parameter.
FairScale. Developed by Meta, FairScale (FairScale authors, 2021) is an extension library to PyTorch, dedicated to high-performance and large-scale training. As a highlight, FairScale uses Fully Sharded Data Parallel (FSDP) (Zhao et al., 2023c) the preferred method for scaling the training operations of large neural networks. It uses AdaScale optimizer (Johnson et al., 2020) as its distributed optimizer.
Pax. Developed by Google, Pax (Google, 2023a) is a JAX-based distributed training framework. Pax has been used to train PaLM-2 (Anil et al., 2023) and Bard (Hsiao et al., 2023). It targets scalability and has reference examples for large model training, including across modalities (e.g., text, vision, speech). It supports data and tensor parallelism, and is heavily integrated with JAX and uses many librarics in the JAX ecosystem. Several key components Pax contains include SeqIO to handle sequential data processing, Oplax for optimization, Fiddle for configuration, Orbax for checkpointing, PyGLove for automatic differentiation. and Flax for creating high-performancee neural networks.
Composer. Composer (MosaicML, 2023a) is an LLM framework designed by Mosaic ML. It has been used to train Mosaic ML's MPT 7B and MPT 30B models and Replit's Code V-1.5 3B. The framework is built on top of PyTorch and provides a collection of acceleration methods that users can incorporate into their training loops or use with the Composer trainer. Composer supports FSDP for efficient parallelism, elastic shared checkpointing for robust. intermittent. training, and dataset streaming implementation allowing the download of datascts from cloud blob storage on the fly during training. Composer also provides functional API for integrating methods directly into ils training loops, as well a Trainer API which automatically implements a PyTorch- baased training loop, reducing the workload for ML developers.
34

---

Published in Transactions on Machine Learning Research (May/202) /204)
OpenLLM. OpenLLM (Pham et al., 2023) was made by BentoML to serve and fine-tune LLMs within production environments. Anchored within the BentoMT. ecosystem, OpenI.LM makes it easy to self-host LLMs and integrate them with other cloud services. OpenLLM emphasizes on flexibility, SOTA LLM support, and strcamlined APIs for self-hosting. Recognizing the diverse needs of production environments, OpenLLM supports the automatic generation of docker images. It also supports serving models as serverless endpoints using BentoML's 's cloud platform. OpenILM further integrates advanced quantization techniques (GPTQ (Frantar et al., 2023), AWQ (Lin et al., 2023), SqueezeLLM (Kim et al., 2024), SpQR (Dettmers et al., 2024), LLM.int8 (Dettmers et al, 2022)) for efficient inference. OpenLLM's design also incorporates robust monitoring with Prometheus metrics and logging tools, ensuring that operational insights are readily available for performance tuning and troubleshooting.
LLM Foundry. LLM Foundry (MosaicML, 2023b) is a library developed by MosaicML for fine-tuning, evaluating, and serving LLMs. It supports distributed inference via FSDP and continuous batching (Yu et al., 2022) for efficient serving. It also has cloud intogration with the MosaicML platform.
vLLM. vLLM (Kwon et al., 2023) is an open-source library for LLM inference and serving. It adopts a different design in how KV cache is stored in memory. Central to this design is PageddAtenntin,, a mechanism that segments the attention key and value (KV) cache for a set number of tokens. Unlike contiguous space storage, Attenionsnnson's blocks for the KV cache are stored flexibly, similar to the virtual memory management in operating systems. This facilitates memory sharing at a block level across various sequences tied to the same request or even different requests, thus enhancing memory management efficiency in handling attention mechanisms. Hence, vLLM can reduce the total memory usage when using complex decoding techniques such parallel sampling and beam search the memory blocks be shared across different candidate samples. It also allows on-demand buffer allocation, while also eliminating external fragmentation as the blocks are uniformly sized. Furthermore, vLLM incorporates safeguards that prevent GPU memory overflow due to the increasing size of KV cache by evicting and recovering blocks as needed via swapping and rocomputation. vLLM also supports Multi-LoRA (Wang et al., 2023g), continuous batching (Yu et al., 2022) and quantization (GPTQ (Frantar el al., 2023), AWQ (Lin el al., 2023) and SqueezeLLM (Kim el al., 2024)). Lastly, it implements efficient kernels for both Nvidia and AMD GPUs.
TensorRT-LLM, TensorRT-LLM (NVIDIA, 2023b) is a streamlined library to serve LLMs. Built on top of the TensorRT engine, TensorRT-LLM integrates optimized kernels from TraserTrannormmer (Timonin el al., 2022) and employs tensor parallelism, facilitating efficient inference at scale across multiple GPUs and servers without necessitating developer intervention or model changes. It also integrates seamlessly with the Nvidia Triton Inference Server for serving LLMs. Additionally, it offers features like tensor parallelism, continuous batching (Yu et al., 2022), and quantization modes and techniques.
(Kwon et al., 2023) as well as supports a wide range of
(TGI). Text- Textt-Generatioon-Ifeeeeee (TGI) (HuggingFace, 2023b) is high- performance LLM serving library developed by Huggingface and is used to power Hugging Chat. TGT supports a large varicty of LLMs, and offers wide range of features like tensor parallelism, continuous batching (Yu et al., 2022), elficient attention mechanisms like FlashAutentin (Dao et al., 2022) and Page- dAttention (Kwon et al., 2023), and quantization (BitsAndBytes (Dettmers, 2023), GPTQ (Frantar et al., 2023)) support.
Textt-eeneraiionIIfeeee
LLM. RayLLM (Ray Project, 2023) is an LLM serving framework as part of the Ray ecosystem (Moritz et al., 2018). Built with Ray Serve and the Ray library developed by AnyScale, RayLLM eases LLM serving in multi-GPU and multi-node systems. At the core of RayLLM is the leveraging of Ray's inherent distributed computing capabilities. RayLLM integrates Ray's distributed task scheduling and execution mechanisms, ensuring that LLM tasks are efficiently distributed across available resources. RayLLM also simplifies adding new LLMs for custom use cases, and offers auto-scaling support and high-performance features like continuous batching (Yu et al., 2022), quantization (GPTQ (Frantar et al., 2023), AWQ (Lin et al., 2023), SqueezeLILM (Kim et al., 2024)), and monitoring via Prometheus metrics. It also comes with advanced monitoring support well which includes CLI and a web frontend that can be used to compare and rank models and get cost and latency estimates.
35

---

Published in Transactions on Machine Learning Research (May/200) /2244)
MLC LLM. MLC LLM (Machine Learning Compilation for Large Language Models) (MLC-LLM, 2023) allows individuals to develop, optimize, and deploy LLMs on wide range of platforms such as mobile phones and web browsers. Central to MLC LLM is its focus on machine learning compilation techniques. MLC- LLM compiles LLMs and deploys them in a process that is inherently tailored to the specific capabilities and constraints of each platform and hardware (Chen et al., 2018; Shao et al., 2022; Feng et al., 2023). This platform-native approach ensures that LLMs are not only efficient but also highly optimized for the platforms in which they operate.
Sax. Sax (Google, 2023b) is a platform designed by Google for serving Pax, JAX, and PyTorch models for inference tasks. It supports distributed inference with tensor and data parallclism. It also integrates casily with Google Cloud and cloud monitoring with Prometheus metrics.
Mosec. Mosec (Yang et al., 2021) is a serving framework built to streamline the serving of machine learning models into backend services and microscrvices. Moscc's key features include continuous batching (Yu et al., 2022), pipelined stages for handling mixed workloads, and essential cloud features like model warmup, graceful shutdown, and Prometheus monitoring metries, making it easily manageable by Kubernetes other container systems.
5 Concluding Remarks
In this survey, we provide a systematic review of efficient LLMs, an important area of research aimed at democratizing LLMs. We start with motivating the necessity for efficient. LLMs. Guided by a taxonomy, we review algorithm-level and system-level efficient techniques for LLMs from model-centric and data-centric perspectives respectively. Furthermore, we review LLM frameworks with specific optimizations and features crucial for efficient LLMs, We believe that efficiency will play an increasingly important role in LLMs and LLMs-oriented systems. We hope this survey could enable rescarchers and practitioners to quickly get started in this field and act as a calalyst to inspire new research on efficient LLMs.
6 Acknowledgement
We would like to thank the action editor Greg Durrett and anonymous reviewers of Learning Research for their helpful and constructive comments.
Transactions on Machine
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Bal- aji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna- Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosic Camp- bell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cumminns, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Jo- hannes Heidocko, Chris Hesso, Alan Hickey, Wade Hickey, Peter Hocschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael
36

---

Published in Transactions on Machine Learning Research (May/200 /204)
Lampe, Ikai Lan, Toddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Modina, Aalok Mehta, Jacob Menick, Luke Melx, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichhro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila. Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Adilya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarclli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Ben- jamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, 1lya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergyya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023, arXie preprint arXiv:2009. 08774. URL http: /ttp///axvvv/////////eee/eaaavv/ org/ abs abs/2303 08774.
Rishabh Agarwal, Nino Vicillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garca, Matthicu Gcist, and Olivier Bachem. Generalized knowledge distillation for aulo-regressive language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: http://pppeeeeeeeeeeeeee net/forum?? d=3zKtaqxLhw.
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Zhen Stephen Gou, Phil Blunsom, Ahmet Ustün, and Sara Hooker. Intriguing properties of quantization at scale. In Thirty-seventh Conference on Newral Information Processing Systems, 2023. URL https://openreviie httt:/orrmmm.ddd/ee33777777/
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023. URL htttps:/accanntholog. watpheooooo.oog//2331 org/2023. emnlp-main. 298.
Silas Alberti, Niclas Dern, Laura Thesing, and Gitta Kutyniok. Sunformer: Universal approximation for efficient transformers. In Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML), volume 221, 2023. URL https: sttps://proceedingg mlr. wtwpsppwssseeeeeeeeee alber
ti23a.htm
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, and Yuxiong He. Deepspced-inferenee En- abling efficient inference of transformer models at unprecedented scale. In Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, Dallas, Texas, 2022. URL https://dl. acm.://ddlccc...ee//mmmmmmmm org/ /o// abb//bbaaaaaa 10. 5555/3571885 11711711111.8111166.6611
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafcy, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Jun- whan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Choou, Aakanksha Chowdhery, Clé- ment. Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer,
37

---

Published in Transactions on Machine Learning Research (May/202) 2204)
Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lu- cas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea. Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sncha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lec, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023, Xix preprint ar 23v: 10403. URL attp://arxiv.... org/ abs/aabs.....
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasumuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficient large scale language modeling with mixtures of experts. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Pro- ceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11699-11732, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 2tt.:/333/ww22222mmmm/l../ 804. URL
on Uncertainty in Artificial Intelligence, volume 161, 2021. URL org/2022. emnlp-man malppmmain.ee 804.
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Henry Mao, Gary Cottrell, and Julian McAuley. Rezero is all you need: fast convergence al large depth. In Proceedings of the Thirly-Sevendh Conference
mlr. p.r.sress/vi
61/bachleecher21a.hbrr
Trapit Bansal, Salaheddin Alzubi, Tong Wang, Yoo- Lee, and Andrew McCallum. Meta-adapters: Parameter ellicient few-shot fine-tuning through meta-learning, In International Conference on Aulomated Machine Learning, Baltimore, US, 2022. URL https://openrevi wwt.wwwwwwwwweee.........e. paad/foouummdd-BCK--PPrr
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long- don-meeemmnnt transformer, 2020, arXiv preprint ar Xiv:2000 051500. URL http://arxiv org/abs//0// /at.//0004.0.......
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley, Unlimiformer: Long-range trans- formers with unlimited length input. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: : /pwp/reveeeee.e.//////e not/fooruridd-1JWJW..CC
Macicj Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38, 2024. URL https: //ojs.......///// aaai melai.org org/ hetg/index.,pp http:///pp////atttctc///////////
Zhengda Bian, Qifan Xu, Boxiang Wang, and Yang You. Maximizing parallelism in distributed training for huge neural networks, 2021, ar arxiv preprint ar arXiv:2105. 14450. URL http: oww//ogg///////o http:::/aaa..v.orrr/aa////// 14450.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usven Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NNeX-200B: An open-source autoregressive language model. In Proceedings of BigScience Episode #5 - Workshop on Challenges g Perspectives in Creating Large Language Models, Dublin, 2022. URL https://aclamtholooyy. big.geiiiiiccinc-1....
bloc97. Ntk-aware scaled rope allows Ilaaa models to have extended (8k+) context size without any fine- tuning and minimal perplexity degradation. https: / httpwwtwwwddttttt wttm:////ooallaaa//oooatoo... j5/ntkaware,
saaeaarrrrssaeeeeeeeeeeeeaeeeeee scaled acaleddrope______e_eeeeeeeeeeeee patperlllllaaaaaaeaaaaaeeeeeeeee allows_ 11ama llamaamodellla models tt_ ra_eav/ 2023.
38

---

Published in Transactions on Machine Learning Research May/2024)
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. Tn Proceedings of the 2021 Conference on Empirioal Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, 2021. URL httpp:///cccaii wwc.og..org//ae emnlp-main. 627.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, 2020. URL ottps://proceedings. neereidinnnnnnnipss... neurips. wwtppsp.ccppppppppppppppppp 2f64a-Paper. .df..
Aydar Bulatov, Yuri Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In Advances in Neural Information Processing Systems, 2022. URL http://pwweiai.eeee///////aaaaaa
Neil Burgess, Jelena Milanovic, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. Bfloat16 processing for neural networks. In IEEE Symposium Computer Arithmetic, Kyoto, 2019. URL https: //ieeexplore leee. org/ //////mmt///7779900
Federico Busato and Jeff Pool. Exploiting nvidia ampere structured sparsity with cusparselt. https: wwwpttiinnngaeeeeeeeeeeeeeeeeeee www.ssuturrrrpprsstttteeeeeeeeee cusptthsssssptttttcee 2020.
//developer httpvelloer..vidia.... mvidia.
Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Percira, Nicolas Le Roux, and Alessandro Sordoni. Multi- head adapler routing for cross-task generalization. In Thirlg-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreria http:/p/wrrreeeeeeeeeeooooa/aaa/
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple 1lm inference acceleration framework with multiple decoding heads, 2024, arXiv preprial arXiv:2401.10774 URL http: http///xxvvvvoo/// wrggaa////01...... 10774.
Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: When data mining meets large language model finetuning, 2023, arXiv preprint 2xXh::2377.06200 URL .06290.
org/ has///b///// /2307
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15, 2024. URL https: //to:.///ooioo/ htt.///oooorr1111111////44111112 1145/364188.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QulP: 2-bit quantization of large language models with guarantees. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL https:://ppereeee
net. nett//orum//dexxx9g555
Boidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank atlention. Advances Newral Information Processing Systems, 34, 2021a. URL https://openreview httppppwwrrrreeeeeeeeoerrrrrreee
Chang Chen, Min Li, Zhihua Wu, Dianhai Yu, and Chao Yang. TA-moe: Topology-aware large scale mixtuure-f-exxpert training. In Advances in Neural Information Processing Systems, 2022a. URL https: / //openreview.c
Charlic Chen, Sebastian Borgcaud, Geoffrey Irving. Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023a, arXiv preprint 01318.
arXin:2302.01318 URL wwtpp//wwwwww///////....oo
39

---

Published in Transactions on Machine Learning Research (May/202) /224)
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Läu, and Qun Liu. bert2BERT: Towards reusable pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, 2022b. URL https: / ////acclaattt aclanthology org/2022. acl-long... 151.
Guanzheng Chen, Xin Li. Zaiqiao Meng, Shangsong Liang, and Lidong Bing. CLEX: Continuous length extrapolation for large language models. In The Twelfth International Conference on Learning Represen- tations, 2024a. URL https://openree wwtpeeeewwweeeetoorneeeeeeeeeeee
Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. Maybe only 0.5% data is neoded: preliminary exploration of low training data instruction tuning, 2023b, arXiv preprial Niw:2305.09246. URL http://arxiv http://aariiv.orrrr//////// org/ 09246..
Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. Walking down the memory maze: Beyond context limit through interactive reading, 2023c, orXiv preprint orXiv:231000502, URL 1lbs/2311 22100. 12211..000000000 05029.
http://arxiv org/aaa abs/
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vi- jay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024b. URL
https://openreviae wttpeerreiie..eeettouuu//// seet/forumnid=Faaxxee
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Olivcira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Fe- lipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantamu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Exaluating large langnage models trained on code, 2021b, Xixi preprint vxXi:21107.3337, URL http://arxix> org/abs/2107. 033744
Shouyuan Chen, Sherman Wong, Liangiaa Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023d, arXiv preprint arXiv:2306.1588 URL http::/aatt //arxi
v.org/abs/2306. 15595.
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwci Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. TVM: An automated End-to-End optimizing compiler for deep learning. In USENIX Symposium on Operaling Systems Design and Implementation (OSDI), Carlsbad, CA, 2018. URL https: / /wwp://wwwww/o// wwwwix.eeii.c.m wwseeni..og//oniiiee httg//onfeeenncosdiiii /presentation/ chen.
Tianyi Chen, Tianyu Ding, Badal Yadav, Πya Zharkov, and Luming Liang, Lorashear: Efficient large language model structured pruming and knowledge recovery, 2023e, arXiv preprinl arXio:2310. 18356. URL http:///aaat /arxiv.ivvaa org/ott/ abs///////0////// 18356.
Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Life- long language pretraining with distributtion-speciaiiiee experts. In Proceedings of the 40th International Conference Machine Learning, Honolulu, Hawaii, USA, 2023f. URL https://dl. acm org/doi/10.55 001058884083668822
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsich, Yifeng Lu, and Quoe V Le. Symbolic discovery of optimization algorithms. In Thirty-seventh Conference on Neural Information Processing Systems, 2023g. URL https: //pper///ppnrrvier ftt.ffrrruu2ddeee222e222...2
40

---

Published in Transactions on Machine Learning Research (May/202) /224)
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Meta-learning via language model in- context tuning. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), Dublin, Ireland, 2022c. URL
org/2022.cco/ acl longlong--- 53.
Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi. Tegit: Generating high-quality instruction-tuming data with text-grounded task design, 2023h, arXiv preprint WxXi:2200000444 URL http://arxiv.orgg http://axxvv...//abba www./rgg//aa///////ee////// 2209. 05447.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-coutext large language models, 2023i, arXiv preprint orXiv:2309.1900 12307. URL
http://arxiv.. ooroor///oo /a////bb/2209/ 12307.
Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. DISCO: Distilling counterfactuals with large language models. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023j. URL https://aclan: hology - www///23......... acl- aall-long.33 302.
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023. URL
org/2023. eenll-mailli 232.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture experts. In Advances in Neural Information Processing Systems, 2022. URL https: /tppr////ep/eeeee../mmo
Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing. Contrastive chain-of- thought prompting, 2023, arXiv preprint vxXi:2331.0092) URL attp://arxiv httg/bbs223111
Rewon Child, Scott Gray, Alec Radford, and Πya Sutskever. Generating long sequences with sparse trans- formers, 2019, arXiv preprint Xiv.v:1904. 10509. URL http://arxiv. org/ orgg/bs////// aas/a/94..
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, and Adrian Weller. Masked language modeling for proteins via linearly scalable long-context transformers, 2020, arXiv preprint or orXiv:2006. 03555. URL http://arxiv.. org/ aa//abb/20000 22006.03555..
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers, 2021. URL https://opent:
wiew.net/fore httee/forummiddaaaeuuuaat
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gauray Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thamunalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Morcira, Rewon Child, Oleksandr Polozov, Katherine Leo, Zongwei Zhou, Xuczhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eek, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022, arXiv preprint ar Xiv:2200 02311. URL http:///rxivv org/ abs/ 2aabb/2204 023111.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuczhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha
41

---

Published in Transactions on Machine Learning Research (May/200 (024)
Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Fd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling cnerrutiionfinnturen language models, 2022, arXiv preprint arxi::2210.111 11416. URL http://arxixi org/aaa abs/ 2210. 11416.
Won Chung, Yile Gu, Insu Jang, Luoxi Meng, Nikhil Bansal, and Mosharaf Chowdhury. Perseus: Removing energy bloat from large model training, 2023, arXiv preprint arxiv:2312.000 06902.. URL http:
//arxiv. .rg. www.abs//aa.oooooooooooooee htt.////......m 06902.
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. StableMoE: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), Dublin, Ireland, 2022. URL gy gy.org/2022 acl-long. 489.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer- XL: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Associalion for Compulational Långuislics, Florence, Italy, 2019. URL https: atttss//accaattiie aww.aacooaooooooy.oooooomooooooa #8/P19-1225 1285.
Tri Dao. Flashaltention-2: Faster attention with belter parallelism and work partitioning. In The Twelfth International Conference Learning Representations, 2024. URL https://openrree
net/forum"
d=m2n2Xyh9Ec.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35, 2022. URL
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference. 2023.
https://pytords wwtg//logfllash-eeeeee
Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020. URL attpp://dl. acm. org/
Tim Detlmers. Bitsandbyles.
2023.
Tim Dettmers, Mike Lewis, Youncs Belkada, and Luke Zettlemoyer. GPT3.int8(): 8-bit matrix multiplication for transformers al scale. In Advonces in Neural Information Processing Systems, 2022. URL https: / //openreview.o
wwt. saarrvviewnne/ffoummt did=ddiig@@BBxaaa
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Elficient finetuning of quantized LLMs. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL
https://openrevier http:r/wwwww.eee.fooooooooooooe
Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: A sparse-quantized representation for near-lossless LLM weight compression. In The Twelfth International Conference on Learning Representa- tions, 2024. URL http:::://p//evvieeee/ wtt.rriwww...eeeeeeeeeeeeeee mett//orumiidd-uu25aahss
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens, 2023a, arXiv preprint arXiv:2307.02,4 024486. URL attp://arxila: http://axvvvv..o/aaaaa wabs////0 02486.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation, 2023b, Xr preprint arXw:2331.04255, URL http:///xxvvv.o/aaabbb
04254.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023, arXiv preprint 2xXiw:2311002253 URL http: //arxiv. wttprvvv.rr//bb/21..022222.
42

---

Published in Transactions on Machine Learning Research (May/202) (024)
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellsserry Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Proceedings of the 39th International Conference html.
on Machine Learning, 2022. URL https://proceedlngs mlr. pwwps/reeeee.///
Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegdc. On the computational com- plexity of self-attention. In Proceedings of The 34th International Conference on Algorillonic Learning Theory, 2023. URL https://proceediin mlr, press/v201 /dtmmankeleeeaaae/ kel.eeea..aaaeee.eeeeeee html.
Lance Eliot. Generative pre-trained transformers (gpt-3) pertain to ai in the law, 2021. URL http:
//dx.doi.ooo. wrg/10. 200....19///// htt///srnn333 3974887.
Facebook AT Research (FAIR). fairseq: Fp16 optimizer - line 468. httpp://ggtaab..co/cccabbkreer b/fatreee////oo/ss/s/////aeee//p http://iii.pppoooooiiieeeee 2023.
FairScale authors. Fairscale: A general purpose modular pytorch library for high performance and large scale training. attps://giithi httpllccooofcaaaaae catn//accboorreeearccaaaaaaaaal 2021.
Jiarui Fang, Zilin Zhu, Shenggui Li, Hui Su, Yang Yu, Jie Zhou, and Yang You. Parallel training of pre-trained models via chunk-based dynamic memory management. IEEE Transactions on Parallel and Distribuled Systems, 34(1):304-313 2023. doi: 10.100//TDSS22222221111811 URL sttps://eeeepploee.tt ieee. org/ document/9940588
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23, 2022. URL https://dl. cm. omm.or///o//b//
5555555555535/ 3586709.
Siyuan Feng, Bohan Hou, Hongyi Jin, Wuwei Lin, Junru Shao, Ruihang Lai, Zihao Ye, Lianmin Zheng, Cody Hao Yu, Yong Yu, and Tianqi Chen. Tensorir: An abstraction for automatic tensorized program optimization. In ACM International Conference on Archittectural Support for Programming Languages and Operating Systems, 2023. URL bctpa://doi..rgge 10. 1ttp//10..111///3/ 11t.1115///6633311 05515755566333363333333
Elias Frantar and Dan Alistarh. Optimal brain compression: A Iramework for accurate post-training quan- tization and pruning. In Advances Neural Information Processing Systems, New Orleans, Louisiana, 2022. URL stttp:///pennrrriee.neet wew.e//oormiiiiese//ooooo..
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned one- shot. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https: //proceedings. wwwiiinnnmmm......mm /rress/v202/fraatr/// html.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for gen- erative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https: https:///peeneeeee http:////eeeeeeeeeeeeeeeee wwwpeeeiiwiinneeeeeeeeeeeeeeeeee www..neetttruumi..dcccbi....
Daniel Y Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference Learning Representations, 2023a. URL http:///opppreeeeeeeeeeeee/t/tee aet//forum11d--Cozyyovrr
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. Tn Proceedings of the 40th International Conference on Machine Learning, Honolulu, Hawaii, 2023b. URL httpss:/prroeeddiin mmr. ppwps///02/se/// https///222///..oeeee/// html.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Breaking the sequential dependency of Ilm inference using lookahead decoding, 2023c. URL https: //wmsssm.ss./////m/////es wwwpblogg//23-/1111/ooo 111--111---ooaaaadddeeeeee
43

---

http://..orrg/.11///11111//001/0 Published in Transactions on Machine Learning Research (May/200 (024)
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture--f-experts.. Proceedings of Machine Learning and Systems, 5, 2023. URL https://proceeddig 8 s.mlsys. .rgg gaaggppppr_pp/ 2023.html.
Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. In The Twelfth International Conference on Learving Represen- tations, 2024. URL
Michael Glass, Alioo Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, GP Shrivalsa Bhargav, Dinesh Garg, and Avi Sil. Span selection pre-training for question answering. In Proceedings of the 58th Annual Meeting of the Association for Compulational Linguistics, 2020. URL 0.acl -main. 247.
org/202
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of BERT by progressively stacking. In Proceedings of the 36th International Conference Machine Learning. volume 97, Long Beach, California, 2019. URL eppaxml. GitHub repository.
mlr press/v97 0tt97//gogg9..attee html.
Google. Pax: A jax--baedd machine learning framework for large scale models. e/paxm1, 2023a. URL wtt. coom/googee/saxml
Google. Sax. htt.:h/c////gtttu.ooo/////g/////
2023b.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129, 2021. URL
1453-z.
Shane Griffith, Kanshik Subramanian, Jonathan Scholz, Charles Isbell, and Andrea Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In Advonces in Newrol Information Processing Systems, volume 26, 2013. URL https / http://ooeeennn..neurrreeeeeeeee pssppssssaaepeesllleeese//e//eee Papere
Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces, 2023, arXiv preprint 0tt::2312.0052. URL http://arxiv, http://aaaa....oor/b////11100000
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022a. URL tttps://openreviii
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive BERT training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 2021. URL https: /paps://aaaaataaa aalaatholooooyyooo wwachhooogy.oooooooooooo org/2021 naacl-main 406.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. PPT: Pre-trained prompt tuning for few-shot learning, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), Dublin, Ireland, 2022b. URL https / /ttp://aaaaaatn /acaaaaaaaooooyyyyyooooo org/ 22t//2222.aa acl-oog... 576.
Yuxian Gu, Li Dong, Furu Wei, and Minlic Huang. MinILLM: Knowledge distillation of large language
models. In The Twelfth International Conference Learning Representations, 2024. URL https: plopenreview.netee wwt. htt///ormmmmdshhohfttt
Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. Olive: Accelerating large language models via. hardware-friendly outlier-victim pair quantization. In Proceedings of the 50th Annual International Symposium Computer Architecture, 2023. URL https: hopps://doo /ttp://oiiioo org/o0........... 1et..11153597777 14583579777.338800888
Ahan Gupta, Yueming Yuan, Yanqi Zhou, and Charith Mendis. Flurka: Fast fused low-rank & kernel attention, 2023, Xiu preprint ar 20v6 15799. URL http:://arrd..orr/aa/22300 15799.
44

---

Published in Transactions on Machine Learning Research (May/2024)
Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective structured state spaces. In Advances in Newral Information Processing Systems, 2022. URL https://openreviev..rr net/f
Drun?id=RjSojetsSH
Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David Woodruff, and Amir Zandieh. Hyperat- tention: Long-context attention in near-linear time. In The Twelfth International Conference on Learning Representations, 2024. URL https: sttps::/openrevien. htt.://mooooorr111000000111111
Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang, Fastmoe: A fast mixture-of expert training system, 2021, Xixiv preprint ar Xiv:221199. 13262. URL http:://axvvv.or//aaaeee 62.
132
Jiaao Hc, Jidong Zhai, Tiago Antunes, Haojic Wang, Fuwen Luo, Shangfeng Shi, and Qin Li. Fastermoc: modeling and optimizing training of large-scale dynamic pre-trained models. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2022a. URL https:
//dl.acm org//oi/////oooooooooooo 1145/3503221 3508418.
Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. A Survey of Large Language Models for Hcalthcare: from Data, Technology, and Applications to Accountability and Ethics, 2023, Xiv preprinl urXiv:2310.0569) URL httt:://axxi..ooo wwwpiiv.org/bbb////1/ 056944.
Shwai He, Liang Ding, Daize Dong, Miao Zhang, and Dacheng Tao. Sparseadapter: An easy approach for improving the parameter-effieiency of adapters. In Findings of FMNLP, 2022b. URL https: /aap:///cclatt logy. org/ 20222022.0002222 findingsnnniii meadiing-emml..... 160.
Namgyu Ho, Laura Schmid, and Yo-Yo Yun. Large language models are reasoning teachers. In Proceedings of the 61st Araual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023. URL https: http://ccaaathohooeoo/o
httgg/223..aaa acl-long........... 880.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hemnigan, Fric Noland, Katherine Millican, George van den Dricssche, Bogdan Damoc, Aurelia Guy, Simon Osindcro, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, 2022. URL https: htttps://operreeee http:////ppeeeeeeeeeet.ee
Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, and Yuxiong Hc. Decpspeed- fastgen: High-throughput text generation for 1lms via mii and seepspeedlinffreneee 2024, arxi preprinl arXiv:2401.08677 URL http:///rxii org/ abs////1.///0//5
Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. Flashdecoding+++ Faster large language model inference on gpus, 2023, arxiv preprint
arX1v:2311.01982. URL attp://arxiv ceaxxiv.orgge org/ abt///b1.///1.2... 01282.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), Toronto, Canada, 2023. URL gy. 1y..oo//023.ac1-ooon..11 108.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length 1lm inference with kv cache quantization, 2024, arXiv preprint arXiv:2401.1807 URL http://arxix: 2rg/abs/2401.1800
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-officient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97, 2019. URL https: / //proceedings... mlr wtt....ree//v////illll.
html.
45

---

Published in Transactions on Machine Learning Research (May/200 (224)
Sissie Hsiao, Yury Pinsky, and Sundar Pichai. Bard: Google's generative language model. https://blog.s oogle e/ httpe//prouccs////oeee heapttssseeeccc//baaeeeee htwparchhbbrdduudatees/eees 2023.
Cheng- Yu Hsich, Chun-Liang Li, Chih-kuan Ych, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Kr- ishna, Chen-Yu Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, 2023. URL https: htt///aaaaattt /aclaaaaaaaooooyyoooooooo org/2023..........o findings ac1.507.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model com- pression with weighted low-rank factorization. In International Conference on Learning Represenlations, 2022. URL
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: / http://ppppreeee..eeeee/ /ttp///weeeeeeeeeeooottnetaaaa.a mwt./foruur..eemeaaee.r....
Shengding Hu, Ning Ding, Weilin Zhao, Xingtai Lv, Zhen Zhang, Zhiyuan Liu, and Maosong Sun. OpenDelta: A plug-and-ple library for parameter-efficien adaptation of pre-trained models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume System Demonstrations), Toronto, Canada, 2023a. URL https://achee hatp::///acllanhoooggyy org/ ptrgg2223..2 acl- mecl-demo... 26.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ec-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LLM-adaplers: An adapter family for parameler-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023b. URL httpp://acceeeee /aclaaaacaooooyy...a org/
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross- task generalization via dynamic lora composition, 2023, arXiv preprint arXiv:2307.1328 13269. URL http: //arxiv.org/// absrg///b//337...mm 13269.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, In Findings of the Associalion for Compulational Linguislics: ACL 2023, Toronto, Canada, 2023. URL https://acll bttts://aclanthologe htthhooogy.rrg/223333 findings-acl,
Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Maksims Volkovs. Improving transformer optimization through better initialization. In Proceedings of the 37th International Conference on Machine Learning, volume 119, 2020. URL
mlr. pptpss/v19//uaanngeeeeeeeee
html.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJJoong Lee, Jiquan Ngiam, Quoe V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019.
Yukum Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. In-context learning distillation: Transferring few-shot learning ability of pre-trained language models, 2022, arXiv preprint arXiv2212.10077. URL.
http:://axxiv..rg/abbba 10670.
HuggingFace. text-generatioo-ninfeencce https://githin 2024-05-10.
2023a. Accessed:
HuggingFace. cext-generatiioninneeene
ce, 2023b. Accessod: 2024-05-10.
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Beockerecurrett transformers. In Advances in Neural Information Processing Systems, 2022. URI. https://
Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafacl Salas, Jithin Jose, Prabhat Ram, Joe Chau, Peng Cheng, Fan Yang, Mao Yang, and Yongqiang Xiong. Tutel: Adaptive mixturee-ffexperts scale. Proceedings of Machine Learning and Systems, 5, 2023.
46

---

Published in Transactions on Machine Learning Research (May/202)
Régis Pierrard Ilas Moutawwakil. LIm-perf leaderboard. 2023.
Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short-text models. Transactions of the Association for Computational Linguistics, 11, 2023. URL rg/2023. tac2-1..c1-1.1.
Hamish Ivison, Noah A. Smith, Hannanch Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors. In Findings of the Association for Computational Linguistics, Toronto, Canada, 2023. URL attps::/aclanthologg. org/ 2trg/2223.22 1.8.ffnddin findings-acl..577 .766.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023a, arXiv preprint (rXi:2310006622) URL http: attp://arxiv. org/ org/abs/23 abs/23 10.06825.
Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin- Yew Lin, Yuqing Yang, and Lili Qiu. Long.IMMingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Poundation Models, 2024. URL http///nrrreeeettt/forrrrr?? wawprruniddyyyffRrmmmm
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprietary large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023b. URL https: http:///aaaaanooogorrrrr.. atgg/023.emmlpmmail..... momlp-main.180
Yunho Jin, Chun-Feng Wu, David Brooks, and GeoY Wei. $s Increasing GPU utilization during generative inference for higher throughput. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL https:///ont /tttts:///penrevievw http:p///ppaarrereeeeeeee//tta/p
Tyler Johnson, Pulkit Agrawal, Haijie Gu, and Carlos Guestrin. AdaScale SGD: A user-friendly algorithm for distributed training. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4911-4920. PMLR, 13 18 Jul 2020. URL sstps://proceediie
mlr, ppwss//119/////eeee. johsson2aa..........ooo html.
Hoyoun Jung and Kyung-Joong Kim. Discrete prompt compression with reinforcement learning, 2023, arXiv preprint XrX::22308. 08758. URL tttp://arxiv. org/ www.ooo////////eeeeooo httpg//as/330..////
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models, 2023, arXiv preprint arXiv:230711016, URL http: //arxiv. haax/i..rg//ao abs/2307 10169.
Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, and Pradoep Dubey. A study of bfloat16 for deep learning training, 2019, arXiv preprinl ar Xvv: 1 12322. URL http::/ //tpxvv///iv////////a org/abs////a 12322.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hyper- complex adapter layers. In Advances in Neural Information Processing Systems, volume 34, New Orleans, Louisiana, 2021. URL https: http::::////cceeies//// www.gg...uris... wwt../ip..cp.pppprepepepe http:///.p.pppp.eeeeee///p/////p P.F0FFFbcc000333000-=770000---- pdf.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autorcgressive transformers with lincar attention. In Proceedings of the 37th International Conference on Machine Learning, volume 119, 2020. URL https://proceeding..tt www.s/reessiiiieee http11/////taoooooo020000 html.
47

---

Published in Transactions on Machine Learning Research (May/200) (024)
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https://openre view.net/feeeee 2ett/fooummidd2jUUUUEEE
Minsoo Kim, Sihwa Lee, Janghwan Lee, Sukjin Hong, Du-Seong Chang, Wonyong Sung, and Jungwook Choi. Token-scaled logit distillation for ternary weight generative language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL d=FUnEkOkodU.
wwwperrrriee..nt/forrnn" net/forum?
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Thirby-seventh Conference on Neural Information Processing Systems, 2023c. URL https://openrevier
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization, 2024, arXiv preprint arXiv:2306. 07629. URL http:://axixii wwwrxii.oog//bbeee org/abs/2300 0tt///3300000770
Young Jin Kim, Rawn Henry, Raffy Fahim, and Hany Hassan Awadalla. Finequant: Unlocking efficiency with ine-grained weight-only quantization for Ilms, 2023d, arXio preprint 2tt.:2308.0972555 URL http:
//arxiv. org/ http:/ab//300.00050
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017, arXiv preprint arXiv:1412. 6980. CRL http://arxiv org/ htts/1442.698...
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL https://openrevies wttp://www....toooooo/oorrgtaaa.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022. URL
htapp:///peenreriee.nete/// wtt.://eee....urrie///////3933
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. In Proceedings of Machine Learning and Systems, volume 5, 2023. URL httpss//prooceedingg mlsys mlyyy220 html.
iipape_filest
Siddharth Krishna Kumar. On weight initialization in deep neural networks, 2017, arXiv preprinl (XXiv:104408863. URL http://arxiv.. http://axxvvvv.g//.//./ orgg/bs//704.000
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, 2023. URL https://doi. org.///00.orr./////// 10. 1111/116000666661333333
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Lessons learned from ac- tivation outliers for weight quantization in large language models, 2023, arXiv preprint (ttt:::33600291 URL atttp://axivv org/abs// abs/aa/....../a/a/////////.aaaae 222//2220622222...2222222222
Dmitry Lepikhin, HyoukJoong Lec, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021. URL https:
//openreview.net..
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parametor-officient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021. URL https: https://aclanttooogy..... wwwp:/thwwwooooooo///...2ee./ emnlp-main 243.
48

---

Published in Transactions on Machine Learning Research (May/2024)
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, 2023. URL
https://dl. acmm::////lccc...../ommmmm acm. org//oi///..ooooooooo /ttp///00.5555331848888 36199203.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE Layers: Simplifying training of large, sparse models. In Proceedings of the 38th International Conference on Machine Learning, mlr. (eeess/vi///ee///
volume 139, 2021. URL https::/proceeeeee
html.
Conglong Li, Ammar Ahmad Awan, Hanlin Tang, Samyam Rajbhandari, and Yuxiong He. 1-bit lamb: Communication efficient. large-scale large-batch training with lamb's convergence speed. In TEEE In- ternational Conference on High Performance Computing, Data, and Analytics (HiPC), 2022. URL https://ieexplore.iies org/
Jiamin Li, Yimin Jiang, Yibo Zhu, Cong Wang, and Hong Xu. Accelerating distributed MoE training and inference with lina. In USENIX Annual Technical Conference (USENIX ATC), Boston, MA, 2023a. URL
http://eeetttttnniiiiieeee atttps://wer usenix www.ni.wwneeeneeeeeee.eeeeeoeeee wttg//oofeeeeee/attttee htt.apaww.....aaaa..a.aaaaaaaaaa
Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai- Wei Chang, and Yejin Choi. Symbolic chain- of-thought distillation: Small models can also "think" step-by-step. In Proceedings of the 61st Annual Meeling of the Association for Compulalional Linguistics (Volume 1: Long Papers), Toronlo, Canada, 2023b. URL https: //////aaaaaaaloooao/aaaa org/2023.acl-oong. 150.
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. In The Twelfth International Conference on Learning Representations, 2024a. URL h/tpss:///penreeeee
Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You. Colossal-ai: A unified deep learning system for large-scale parallel training. In Proceedings of the 52nd International Conference on Parallel Processing, 2023c. URL https://doi. org/ 10. 1145/3605 573. 3605613.
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You. Sequence parallelism: Long sequence training from system perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023d. URL httpss://cclss hology httg//223.aace acl 1onl-long... 134.
Shiyang Li, Jianshu Chen, yelong shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. Explanations from large language models make small reasoners better. In Workshop on Sustainable AI, 2024b. URL https: https://openreview.eettt www..eeooourdddrrrt 8ZUcfL9r.
Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models, 2024c, arXiv preprint arXiv:2402. 18158. URL http:// arxiv. wwreeii..or//aee. abs/2402. 18158.
Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun, and Yequan Wang. Flm-101b: An open I1m and how to train it with $100k budget, 2023e, arXiv preprint ar XrXi::2309.0388 URL http://arxiv.. www./rr//////oooooeee 03852.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anreual Meeling of the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Popers), 2021. URL https://acla nthology. wtgp//21. 1o11-lon.3300 353.
Xiaonan Li and Xipeng Qiu. Finding support examples for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 2023. URL https: pttes://accaantooloo. 202/202 3. f. findings--- emnlp.41. 411.
49

---

Published in Transactions on Machine Learning Research (May/200 (024))
Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023f. URL https: http:////ccaaatoooooooo org/2023......... 2tt22223..111111 acl-long 256.
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tho Zhao. LoSparse: Structured compression of large language models based on low-rank and sparse approximation. In Pro- aeedings of the 40th International Conference on Machine Learning, volume 202, pp. (1936-20350, 23-29 Jul 2023g. URL httpp::/pprcceednng. www.iing.......p..... wtt....pree//////12222222 htm.......m............ html.
Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao Loftq: LoRA-fine-tunin taenntunnnnnnann aware quantization for large language models. In The Twelfth International Conference on Learning Representations, 2024d. URL
net/ wttp://ww.nee/tomri--e----- /ft.://id--LPPPPP.....
Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less more: Task- aware laser-wise distillation for language model compression. In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023. URL https://proceedings mlr preas/v202/1ia ng23j. html.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration, 2023, orXiv preprinl orXiv:2306.0099 URL http://arxiv.. wwwpriiiii..///bb. arggabs/2306..... 200.b///00000088.. 00978.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot paramotor-cfficiet fine-tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems, volume 35, 2022a. URL https: htw.rroceediinnnnnnirss......... Paper- Conere--oo-aaee--CCCCeeeeeeeee Cowferenceeeeccccccomeoeoe e
Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Conference Learning Representations, 2024a. URL
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-contest examples for GPT-3? In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Edtrccion and Integration for Deep Learning Architectures, 2022b. URL logy. Logy.org//i.... www.orgg//2222 2022. dew.lie1..1......11
Jiachen Liu, Zhiyu Wu, Jae-Won Chung, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. Andes: Defining and enhancing ofxeitt- in Ilmebased text streaming services, 2024b, arXiv preprint 11Xiv32404.66283 URL /ttp://arxiv... wwwiiii..rg//aaa httg//bs/24000 16283.
Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang, QLLM: Accurate and efficient low- bitwidth quantization for large language models. In The Tweifth International Conference on Learning Representations, 2024c. URL https : httppp:://///eeeieeeeeee wwwprrewwwwe.eeeeeeeeeeeeeeeee
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pro-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55, 2023a. URL https://doi, org/10//// 10. 1145/3560815.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Dublin, Ireland, 2022c. URL https:/ https////cllathhlggooo org/ 2022 acl- whorshhrr....
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too, 2023b, arxiv preprint arXiv:2108.1033 10385. URL http///riiv..org//////////// 10385.
50

---

Published in Transactions on Machine Learning Research (May/2024) (024)
Yuliang Liu, Shenggui Li, Jiarui Fang, Yanjun Shao, Boyuan Yao, and Yang You. Colossal-auto: Uni- fied automation of parallelization and activation checkpoint for large-scale models, 2023c, orXiv preprinl arXiv:2302.0300 025999 URL http://arxiv.org wwggaas//3302.0222 02599.
Zechum Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models, 2023d, aixiv preprint Xiv:20305 1888.. URL http:://rxix http://aaxxvv..o/abbaas///a 178 88.
Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava. Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time. In Thirty-seventh Conference on Neural Information Processing Systems, 2023e. URL
not//orrumiidd/Zfg6iiaa
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Deja vu: Contextual sparsity for efficient LLMs inference time. In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023f. URL https: sttpss///proceedings mlr wwwpps/eee//iiiieeeee h2u///22222//23mmmmmmm/ html.
Zirui Liu, Guanchu Wang, Shaochen Zhong, Zhaozhuo Xu, Daochen Zha, Ruixiang Tang, Zhimeng Jiang, Kaixiong Zhou, Vipin Chaudhary, Shuai Xu, and Xia Hu. Winner-take-all column row sampling for memory efficient adaptation of language model. In Shirth-seventh Conference Neural Information Processing Systems, 2023g. URL https://openreviae net/forum?:d-Squuiyy
Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Represendations, 2019. URL
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeling of the Associalion for Compulational Linguislics (Volume 1: Long Papers), Dublin, Ireland, 2022.
URL stttps//aclaattooll
wrg//022. aww.:on...l-loo....... 556.
Yucheng Lu, Conglong Li, Minjia. Zhang, Christopher De Sa, and Yuxiong He. Maximizing communication efficiency for large-scale training via 0/1 adam. In The Eleventh International Conference on Learning wwt.://ooounnidddeeeeeeen
Representations; 2023. URL
Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mchran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Zhao. Dr.ICL: Demonstrationn-elriieee in-context learning. In Ro-FoMMo.Bobustuess of Few- shot and Zero-shot Learning in Large Foundation Models, 2023. URL httpp::////ereeee..eee//e Fid-WDNb6L5xjI.
Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine- tuning for large language models with limited resources, 2023, arXiv preprint arXiv:2306. 09782. URL http:///rrs //tpiv///xxvv./// www.ii.wg//aaaaeeeeeeooooowoeeee abaaaaaa/0000 22w../2366..0 09782.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large languago models. seirth- Conference on Neural Information Processing Systems, 2023. URL https: //openreview www./eeotriiddaeeeee. net//toruumice--JJJjjEEEE
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Thirty- seventh Conference on Neural mett/foruum.tt-oocaarrna
Information Processing Systems, 2023. URL
Pedro Henrique Martins, Zita Marinho, and Andre Martins. so-forme: Infinite memory transformer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Treland, 2022. URL attpss//aclanntholoo.
51

---

Published in Transactions on Machine Learning Research May/2024)
Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. Long range language modeling via
gated state spaces. In The Fleventh International Conference on Learning Representations, 2023. URL
https://openreview httpe://pppareeeeee.ttffffrrrr/r mett/forumiid-SMYYIYOO
Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024. URL https:
//ai weai.meta.cmm cowmbboogmsttaaaaantttt
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification, 2024, arXiv preprint or arXin:2305.0078 URL http: //aaxiv.orgg/bb/226..00/ 09781.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. Tn International Conference Learning Represenlations, 2018. URL http::://pperreviex. ?id=rigs9JgRZ.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetaICT: Learning to learn context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguislies: Human Language Technologies, Seattle, United States, 2022a. URL https: h/aaa//aaallaaogggo/ooo org/2022..2. maacc-main....
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Arteixe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 2022b. URL
MLC-LLM, MLC-LLM, 2023,
cow.mll..com/mmcc.... aacllc-ai/aee ai/mlc-11m
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for transformers, 2023, arviv preprint arXiv:2305. 16300. URL http://arxiv. h05//bb/23000 16300.
Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling, 2023, arXiv
preprint ar Xrx::22311. 133881 URL http: http://arxvvv../ www..osg/////.eeeooooo...os..s// http//11....aba... 13581.
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ton Stoica. Ray: A distributed framework for emerging AT applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Carlsbad, CA, 2018. URL https://wwwtt http///ww.....i.cc////// tz.
MosaicMT. Composer. httpp:///gtaab.cccc/o/aaaaaaaaaa 2023a. GitHub repository.
MosaicMI. Llm foundry. http:///hhhh..ccccocc//aaaaaa/aa
2023b. GitHub repository.
Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens. In Thirty- seventh Conference on Neural Information Processing Systems, 2023. URL 2runP1d-2DtX013315
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa. Patwary, Vijay Kor- thikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-hn Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021. URL /ttps:://di..... org/ 10. 11.....//35111111111111 3476209.
Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton-of-thought: Prompting LLMs for efficient parallel generation. In The Twelfth International Conference on Learning Representations, 2024. URL httpps://operreviev..
52

---

Published in Transactions on Machine Learning Research (May/202))
NVIDIA. Fastertransformer: High performance transformer kernels. httpss///gtaat...oo/////////// Transformer, 2023a. GitHub repository.
NVIDIA. Tensorrt-llm.
2023b. Accessed: 2024-05-10.
OpenAL. Gpt base model. https://platfors hpt.:/aaa..aa//c////oooeeeo..... base, 2023.
Shankar Padmanabhan, Yasumasa Onoe, Michael JQ Zhang, Greg Durrett, and Eunsol Choi. Propagating knowledge updates to LMs through distillation. In Thirty-seventh Conference Newral Information Processing Systems, 2023. URL https://openrevier
Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, and François Fleuret. Fast attention over long sequences with dynamic sparse flash attention. In Thirty-seventh Conference on Neurol Information Processing Systems, 2023. URL https: / http////ppeeeeeeeeeeeee/ee aet//forum/iddJJINiukk////
Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, and Qun Liu. Rousing pretrained models by multi-linear operators for efficient training. In shirth-seventth Conference on Neural Information Processing Systems, 2023. URL https / //tp///eeiee/eeee//////o net/fooruriddRRgXX.rr..
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023a, arXiv preprint orx::22004.000 03277. URL attp://arxiv wwexxiv.org//bbaee hts///bs//2a/ 03277.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Loon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Tin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanislaw Woćniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer cra. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 2023b. URL https http:///aaaaaahaaaeeeee hwwpolllooyyooggg////. org/2023. findings-emnlp... .366.
Bowen Peng, JefTrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension large language models. In The Twelfth International Conference on Learning Representations, 2024.
URL attps://openreet
net/ www.tod
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature allention. In International Conference on Learning Representations, 2021. URL
Aaron Pham, Chaoyu Yang, Sean Sheng, Shenyang Zhao, Sauyon Lee, Bo Jiang, Fog Dong, Xipeng Guan, and Frost Ming, OpenLLM: Operating LLMs in production, 2023, LM.
Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. Hypertuning: toward adapting large language mod- cls without bropaprop... In Proceedings of the 40th International Conference on Machine Learning, 2023. URL https: acm
36195566.
Jonathan Pilault, Mahan Fathi, Orhan Firat, Christopher Pal, Pierre-Lue Bacon, and Ross Goroshin. Block- state transformers. In Thirth-seventhh Conference on Neural Information Processing Systems, New Orleans, Louisiana, 2023. URL https: h /owpreeiieeeeeeeeeeeeeee http:///wwreeeeeeeeeoerrrreeeeea
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Re. Hyena hicrarchy: Towards larger convolutional language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023. URL https: html.
//proceedings mlr wtt...prrees////2o11111//
Edoardo Maria Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy. Combining parameter-efficient modules for task-level generalisation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Dubrovnik, Croatia, 2023. URL https://aclantholoog .org/2023. eacl-main main.49.
53

---

Published in Transactions on Machine Learning Research (May/202)) /224)
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff' Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023.
Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, and Ashish Panwar. vatten- tion: Dynamic memory management for serving Ilss without pagedattention, 2024, arXio preprinl www.ooggg///......//// 04437.
arXiv:2405.0444 URL
Ofrr Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with lincar biascs cnables input length extrapolation. In International Conference on Learning Representations, 2022. URL https:
//openreview.. nwt/ disammmdddddaPPoo
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguislies: EMNLP, Singapore, 2023. URL https: http://aaaaantoogg....... 20tp/aoooooorg//2223...... f findings- emmpp.mmmlpp...mm 378.
Chengwei Qin, Aston Zhang, Anirudh Dagar, and Wenming Ye. In-context learning with iterative demon- stration selection, 2023a, arXiv preprint arXiv:2310.000 09881. URL /ttt::///xxiv..// org.abs////..ooo..oooo.......s /2300. 098811.
Guanghui Qin, Corby Rosset, Ethan C. Chau, Nikhil Rao, and Benjamin Van Durme. Nugget 2d: Dynamic contextual compression for scaling decoder-only language models, 2023b, arXiv preprint ar Xrx:223300. 0.02499. URL attp://arxiv org/abs//////eeaoooeogggeeeeeeae abs/ 2220022200.........
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Knowledge inheritance for pre-trained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, 2022. URL http::://acaaatholoogo org/2022.n. na 288.
Zhen Qin, Weigao Sun, Dong Li, Xuyang Shen, Weixuan Sun, and Yiran Zhong. Lightning attention- 2: A free lunch for handling unlimited sequence lengths in large language models, 2024, aixie preprint arXiv:2401.0400 URL http:/ http:://axxvvv. org/ httgg/bbs/24011. 04658.
Jiezhong Qiu, Hao Ma, Omer Levy, Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self-atiention for long document understanding. In Findings of the Association for Computational Linguistics: EMNLP, 2020. URL https: sttpa://aclanthologg wwwp://0...oooo/. findings-ennnll..
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAT blog, 2019. URL https: openai. www bettrr models, are_ unsuperviseedd seiaaeddddmiiiiieeeeeeeeeeeeeeee www.mmllitttta. salltitsk___ai wwwttittsk_eernerrreee learners.
age-models/langee lew.aeeeeelaagggggmdeeeeeees
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Fliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganíni, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean daan-sae Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johuson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Ge- offrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2022, arXiv preprint ar xiv:221122 12. 11446. URL http://arxiv. org/ barg/abs// abs 2212. 11446.
54

---

Published in Transactions on Machine Learning Research (May/2022 (024)
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21, 2020. URL sttps://dl acm. org/ org/doi/snn 555511..555//4557
16. 3455856.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong Hc. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2020. URL attps://di /dll. acm, orgo htt//oo//1..../// 5555/3333555/33370 .3433727.
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 2021. URL https://doi. 0rg/10.111 5/3458817 3476205.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong He. DeepSpeed MoE: Advancing of inference and train- ing to power next-generation Al scale. In Proceedings of the 39th International Conference on Machine Learning, volume 162, 2022. URL https://proceediie mml. press watess/vv2r/rraaaaaaaaae 22aa. html.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, 2020. URL https://doi. org/10. 10. 1145/3394488 3341//3994466337733...
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: 352.
Long Papers), Toronto, Canada, 2023. URL
Ray Project, RayLLM - LLMs on Ray, GitHub repository, 2023, https://githiei http:///gtthuu..o.aaaaaaaaaaaaaa www.c//aa-rojjjcc//eeeee
-11m, Accessed on: 2023-10-02.
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. ZeRO-Ofload: Democratizing Billion-Scale model training. In USENIX Annual Technical Conference (USENIX ATC, 2021. URL https://www wwsiii/ usenix org/ conference/ateeeeeooccccoccc saeaaion/rrenjize
Liliang Ren, Yang Liu, Shuohang Wang, Yichong Xu, Chenguang Zhu, and ChengXiang Zhai. Sparse modular activation for efficient sequence modeling. In Thirlg-seventh Conference on Neural Information
Processing Systems, 2023a. URL
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang. Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, Andrey Bout, Irina Piontkovskaya, Jiansheng Wei, Xin Jiang, Teng Su, Qun Liu,, and Jun Yao. Pangu-L:: Towards trillion parameter language model with sparse heterogeneous computing, 2023b, arXiv preprind arXiv:2303. 10845. URL http://arxtt org/abs/2303 .10845.
Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev. Tied-lora: Enhacing parameter efficiency of lora with weight tying, 2023, arxiv preprint or S5w:2311009574 URL http:://arxxw..rg/aab////2 09578.
Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9, 2021. URL tac1-1.4.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant, Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, 2022. URL https://aclanthol .org/2022 2022. maacl-main.199 191.
55

---

Published in Transactions on Machine Learning Research (May/202) 2204)
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. Accelerating transformer inference for translation via parallel decoding. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long .rrg/2023 20003aacl-oomm.. acl-long. 689.
Papers), Toronto, Canada, 2023. URL
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldic, and Christopher D Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval. In The Twelith International Conference on Learning Representations. 2024. URL
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muen- nighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan- Major, Iz Beltagy, Huu Nguyen, Lucile Saulmier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Lau- rengon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Tfeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Saško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Thoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddecn Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaich Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Tagar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Fliza Szezechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrík Strobelt, Jason Alan Frics, Jos Rozen, Lco Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Mattco Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng- Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarich, Adam Roberts, Hyung Won Chung, Jac- sung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Taure Ligozat, Arjun Subramo- nian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Takta- sheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, -hn-Cshitt Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Ma- rine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada. Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdennk Kasner, Alice Rueda, Amanda Pes- tana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldrcaj. Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini,, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kicla, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sodenko, Isar Ne- jadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri,
56

---

Published in Transactions on Machine Learning Research (May/202) (024)
Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasimus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Vigu- ier, Thanh Lc, Tobi Oyebade, Tricu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Pscallada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel Loón Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyased- din Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàriiss, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihalj- cic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasckhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee aroonsio,,i,r, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojeiech Kusa, Yanis Labrak, Yash Shailesh Ba- jaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. Bloom: A 176b-parameter open-access multilingual language model, 2023, arXiv preprint ar arxiv:2211.051 05100. URL http: / /ttx/vriivvva./// www.iv..rg//bbbb http//bbb/221.11../ 05100.
Stephanie Schoch, Ritwick Mishra, and Yangfeng Ji. Data selection for fine-tuning large language mod- els using transferred shapley values. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguistics (Volume 4: Student Research Workshop), Toronto, Canada, 2023. URL https://aclantholoog org/2023. acl-srw.37
Hang Shao, Bei Liu, and Yanmin Qian. One-shot sensitivity-aware mixed sparsity pruning for large language models, 2024, Xiv preprint arXi::2910.09499, URL http: //tpxvaaa/ivv//aa///a/ org/ abs/2310 09499.
Junru Shao, Xiyou Zhou, Siyuan Feng, Bohan Hou, Ruihang Lai, Hongyi Jin, Wuwei Lin, Masahiro Masuda, Cody Hao Yu, and Tianqi Chen. Tensor program optimization with probabilistic programs. In Advances in Neural Information Processing Systems, volume 35, 2022. URL https: //tpc//////cec/sss.e/ee neur cc/p SperX5Ffilest
Paper-Conference patp://raaaaaeeaeaaaaaaaaaee-eee Conf pdf.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations, In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Popers), New Orleans, Louisiana, 2018. URL http:///ccaanhoooo..orrg.o.
Noam Shazeer. Fast transformer decoding: One write-head is all you need, 2019, arXiv preprint arXiv:1911.021500 URL http://arxiv.ot wwwp:///gg.bbbb....o....
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Ix Beltagy. Staged training for transformer language models. In Proceedings of the 39th International Conference on Machine Learning, volume 162, 2022. URL https: //pppe/////ccceinggs mlr. mar.pee press/v162 shen22f html.
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent Y Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. Mixture-of-eeperts meets instruction tuning: A winning combination for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL htttpp:///peereveee/ net net//oormmidddml.jii....
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie, Beidi Chen, Clark W. Barrett, Joseph Gonzalez, Percy Liang, Christopher Ré, Ioan Cristian Stoica, and Ce Zhang. High-throughhut generative inference of large language models with a single gpu. In International Conference on Machine Learning, 2023. URL https: //dl. acm. www..oorg//o/////eeo// 5555/36184 08. 361966..
57

---

Published in Transactions on Machine Learning Research May/2024)
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompis. Tn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 2020. URL https:
//aclanthology. org/2020. emnlp-main. 346.
Mohammad Shoeybi, Mostola Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism, 2020, arXiv preprint (tte:1999.080553 URL http://arxiv, http:///rgggb////gg//////e// 0aba//99..00055
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilitios into smaller language models. In Pindings of the Associalion for Compulational Linguislics: ACL 2023, Toronto, Canada, 2023. URL https: http://laaaattlooyoooooo wtgg2023.findinge-ea..... findings-acl.e444
Antoine Simoulin, Namyong Park, Xiaoyi Liu, and Grey Yang. Memory-effiiiet selective fine-tuning. In Workshop on Efficient Systems for Foundation Models, 2023. URL
net/forum7
d=zaNbLceVHm.
Siddharth Singh, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, Yuxiong He, and Abhinav Bhatele. A hybrid tensor-expert-data parallelism approach to optimize mixture-of-experts training. In ICS 2023, 2023.
Shaden Smith, Mostofa Patwary Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yaz- dani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong Hc, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, a large- scale generative language model, 2022, arXiv preprint vrXiv:2201.11990 URL ptttp://axiivv org/ abs/ 2201. 11990.
Salch Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv Verma, Gokhan Tur, and Prem Natarajan. Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model, 2022, arXiv preprint arXiv:2208.01. 01448. URL http://ar
xiv. xiv.org/abs. wtgg/b//////////////e///g/e////e 01448.
J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Tronsoctions on Automatic Control, 37, 1992. URL attps://ieeexploree ieee. org/ document/11 19632.
Benjamin Spector and Chris Re. Accelerating Ilm inference with staged speculative decoding, 2023, arv preprint Xrx::22308. 04623. URL http://arxiv. org/ ha//abb//300/ 04623.
Hongjin SU, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. Selective annotation makes language models better few-shot learners. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviie wtt.://oooo/tddagggniieeee
Hui Su, Xiao Zhou, Houjin Yu, Xiaoyu Shen, Yuwen Chen, Zilin Zhu, Yang Yu, and Jie Zhou. Welm: A well-read pre-trained language model for chinese, 2023a, arXiv preprint Xrxi:22209.100 10372. URL http:
//arxiv. arrg/bbs22099 10372.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023b, XrXiv preprint or arXim:2144009664 URL http://arxi v. v.org/abs//114.. 2984.
Mingjic Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openreview net/forrmmha--xxxPusse http://ppmmeeieeeeeeeeeouuut//ea
58

---

Published in Transactions on Machine Learning Research May/2024)
Tianxiang Sun, Zhengfu Hc, Qin Zhu, Xipeng Qiu, and Xuanjing Huang. Multitask pre-training of modular prompt for Chinese few-shot learning. In Proceedings of the 61st. Annuul Meeting of the Associalion for Computational Linguistics ( (Volume 1: Long Papers), Toronto, Canada, 2023a. URL https://aclann logy.org 2xrg/2023 acl-long. 625.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023b, orXiv preprinl 1.Xiv:2307.08622 URL http:://arss http:://axxvv..rre org//b//2307 08621.
Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length--oxtrapolatbbl transformer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023c. URL https://aclanthenn wttp:///attnooooooo///002.....1 acl- monl-long.o. 816.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for rein- forcement learning with function approximation. Tn Advances in Neural Information Processing Systems, volume 12, 1999. URL httpe://prooeedingg.neee. neurips....uur........ wwtpseee.eppp.pc.pppp.r.. feaper_fflese files wriles/papee paaaperrfileespppee//9999// 500bed88@00a000543000 Paper. pdf.
Weng Tam, Xiao Liu, Kaixuan Ji, Lilong Xue, Jiahua Liu, Tao Li, Yuxiao Dong, and Jie Tang. Parameter- efficient prompt luning makes generalized and calibrated neural text retrievers. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2023, Singapore, 2023. URL http://aclantoooogy rg/2023. findings-enni emnlp.ennm.....
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, and Yuxiong He. 1-bit adam: Communication efficient large-scale training with adam's 's convergence speed. In Proceedings of the 38th International Conference on Machine Learning, volume 139, 2021. URL https: / http///pooeeeen...oee mlr. haasss/vv39//tngee
html.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. In Proceedings of the 60th Annual Meeting of the Association for Computational Lánguistics ( (Volume 1: Long Papers), Dublin, Ireland, 2022. URL
https: https////cllathogggaoc wrgg/org//2222 orgg2022.acl11ommmm acl acl-long 331.
Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn altention. In Proceed- ings of the 37th International Conference on Machine Learning, 2020. URL https://ddi acm. org/doi/a
111111055555535549333 3525813.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55, 2022. URL attps://doi http:://bb...rrg/// 1145/3530811
org/10.
Gemini Team and Google. Gemini: A family of highly capable multimodal models. https:// 2023.
ogleapis. comm /ttp://pmiiiddiiiaaeeeaeaaeaaeee
The MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable 1lms. https://wwwmmosaicmlll www.b/og//ppto//// 2023.
www.iossiiicoooooo cml
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tzeg-Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Chin Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier- Hellstern, Mcredith Ringel Morris, Tulsce Doshi, Renclito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaisc Aguera-Areas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022, arXiv preprint ar Xrx::2201108223 08239. URL http:///rxiza org/ argg/bbs/2001.000 082399999....0......2//2/2
59

---

Published in Transactions on Machine Learning Research (May/200 (024)
Iaar Timiryasov and Jean-Loup Tastet. Baby Ilama: knowledge distillation from an ensemble of teachers trained on small dataset with no performance penalty. Tn Proceedings of the BabyI.M Challenge al the 27th Conference on Computational Natural Language Learning, Singapore, 2023. URL https://aaaa www.gg..og//2..... concloonlllaoooo balll-bablm.... 24.
Denis Timonin, Bo Yang Hsueh, and Vinh Nguyen. Accelerated inference for large transformer models using nvidia triton inference server, 2022. URL https://developpet wwwp://www.....aaa..aoooooooo//a http:///////o//cc/////aaaaaaa infer mace-for-large paarge-transto oww.://eeeeooooooooooeooooooooor atdtraatgooodttttttttttaroorrrr - triito-iitoono inf erence-server.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Bapliste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/ /tt.//222......1. 2023a. URL https: weappp..mmaticcchii...ee
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Fer- rer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Tsabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojuic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b, arXiv preprint Xrx::23007.0000 092888. URL aatt:://aree aarxiv.iivvv////i///aa///aaaavva wttpa/iwwww..aaaaa./////raaaaaae htt////ab////...e// http//22700000220 09288.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023, arXiv preprint arXiv:2310.169) URL http://arxiv. http:///xxvvvvoo/a/////e httggaab//3310.. 169444........
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Miloš. Focused transformer: Contrastive training for context scaling. In Thirty-seventh Conference Newral Information Processing Systems, 2023. URL https://openreviee http:////ooooooiiooooooooooo//a
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-effiiient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguústics, Dubrovnik, Croatia, 2023. URL https: / ht////acaaaatltoogoe/ org/2023. eacl-maing 239.
Ashish Vaswani, Noam Shazoer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Autention is all you need. In Advances in Neural Informalion Processing Systems, volume 30, 2017. URL https://proceeding. mnw.reeeeddinnssuuirpppee. cttpppprccppppp.ppppppp http:///pppppe///1//////////aaa/ #7beee pdf.
Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Tn Advances in Neural Information Processing Systems, volume 33, 2020. URL https:// neur Paper. .af...pd........pppp.....p.....p
Zhongwei Wan, Yichun Yin, Wei Zhang, Jiaxin Shi, Lifeng Shang, Guangyong Chen, Xin Jiang, and Qun Liu. G-MAP: General memory-augmented pre-trained language model for domain tasks. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, Abu Dhabi, United Arab Emirates, 2022. URL https://aclanthology orgg 2022. emnlp-main. maa.pmmma........... 441.
org/2022.
60

---

Published in Transactions on Machine Learning Research (May/2024)
Zhongwei Wan, Che Liu, Mi Zhang, Jie Fu, Benyou Wang, Sibo Cheng, Lei Ma, César Sallodrán-Casaas and Rossella Arcucci. Med-unic: Unifying cross-lingual medical vision-language pre-training by diminishing bias. In Advances in Neural Information Processing Systems, volume 36, 2023. URL https: /pproeeddi ngs. neurips. ctt.pp..cc/pppprr htt.://pp/p.e/111aaaooaaaoo111a. Paw--ea---pprr-----eee rence.pde pdf.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: stickier benchmark for general-purpose language understanding systems. In Proceedings of the 33rd International Conference on Newral Information Processing Systems, 2019. URL https://d/i acm wwtg/oo//////ooooooooo/oo 11/1///.5555534425///// 3454581.
Boxiang Wang, Qifan Xu, Zhengda Bian, and Yang You. Tesseract: Parallelize the tensor parallelism efficiently. In Proceedings of the 51st International Conference on Parallel Processing, 2023a. URL https: //doi. org/o....11. 1145/3545008 3545087.
Guoxin Wang, Yijuan Lu, Lei Cui, Tengchao Lv, Dinei Florencio, and Cha Zhang. A simple yet effective learnable positional encoding method for improving document transformer model. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, 2022a. URL htttss//aclaatthlogg/ rg/2022. findings bacl.42.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijic Wang, Lingxiao Ma, Fan Yang. Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit. transformers for large language models, 2023b, arXiv preprint ar trXie:2310.1 11453. URL http://arxiv. www///g///////eeee//.//eeee/ee// aas.///0.../1////11
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Dcepnet: Scaling transformers to 1,000 layers. IEEE Transactions on Pallern Analysis and Machine Intelligence, 2024a. URL https://ieeexplon
ieee. org/ htg///ccument/04922333
Liang Wang, Nan Yang, and Furu Wei. Learning to retrieve in-context examples for large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian's, Malta, 2024b. URL https: https: /aclantholl wwcellloooogooogooooooooo 202 4. 4.eacl-long.111 105.
Ningning Wang, Guobing Gan, Peng Zhang, Shuai Zhang, Junqiu Wei, Qun Liu, and Xin Jiang. Clus- terFormer: Neural clustering attention for efficient and effective transformer. In Proceedings of the 60th Annual Meeling of the Association for Compulational Linguistics (Volume 1: Long Papers), Dublin, Ire- land, 2022b. URL https: // /ttp:////aaaaalaaaaaaaaa org/2022 acl -ong.. 170..
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. SCOTT: Self-consistent chain-of-thought distillation. In Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Popers), Toronto, Canada, 2023c. URL https: https://aclantholll org 12023.ac1-1om 304.
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. In The Eleventh International Comference on Learning Representations, 2023d. URL https:://ppnreview
Shuohuan Wang, Yu Sun, Yang Xiang, Zhihua Wu, Siyu Ding, Weibao Gong, Shikun Feng, Junyuan Shang, Yanbin Zhao, Chao Pang, Jiaxiang Liu, Xuyi Chen, Yuxiang Lu, Weixin Liu, Xi Wang, Yangfan Bai, Qiuliang Chen, Li Zhao, Shiyong Li, Peng Sun, Dianhai Yu, Yanjun Ma, Hao Tian, Hua Wu, Tian Wu, Wei Zeng, Ge Li, Wen Gao, and Haifeng Wang. Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation, 2021, Xrxiu preprint ar 11x::2112.12711. URL http:/ /ttp:://aaae //wpvv//./vvv//////////aaeaaaav orggobss// abs/2111..2/111111
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020, ar Xio preprint arXiv:2006.00 0400060..4000 768. URL http://arxiv http:p///axxx...r///////...... wttp://s/////.oo//////// 04768.
61

---

Published in Transactions on Machine Learning Research (May/202) (224)
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. In Thirty-seeenth Conference on Newral Information Processing Systems, 2023e. URL https://openreviier http:://ppeeerreeeee////orrrraop
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-aware singular value decom- position for large language model compression, 2024c, orXiv preprint arXin:2403.0737. URL http:
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Thirty- seventh Conference on Neural Information Processing Systems, 2023f. URL orum?ide Bouniid-BGkkweett.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. AdaMix: Mixture-of for parameter-effiientt model tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 2022c. URL sttpps:/aclaatthttii www.aatoooooooooo//222 wwwp/022..2222.. wwt2222..mmmmmmmmmeemm commlp-main.... mtpmmmaii..8................33.. 388.
Yiming Wang, Yu Lin, Xiaodong Zeng, and Guannan Zhang. Multilora: Democratizing lora for better multi-task learning, 2023g, arXiv preprint ar arXiu:2311.11501 URL
11501.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeling of the Association for Compulational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023h. URL
org/ 22///2223..2 11233.ccl-10 acl-long. 754.
Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey, 2023i, arXiv preprint 1XXi:2307712066 URL http://arxiv.ott wttg/aa/////////////////////// 12w223777.666.....
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multitask prompt tuning enables parameter-offiiient transfer learning. In The Eleventh International Conference Learning Represenlations, 2023j. URL
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Scbastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Fxl H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions Machine Learning Research, 2022a. URL https://openreviee: net//forumid=yyaASSuea
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Newral Information Processing Systems, 2022b. URL https://openreet http:://ppppmeeeeeeeeeeeorordddo http//orrmmiidd__VVeee...-. _VjQ1Mes_____
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, Singapore, 2023. URL
202 emnlp-mmin..01.
Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, and Pascale Fung. Lightweight and efficient end-to-end speech recognition using low-rank transformer. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020. URL ptttp::/iieexplores ieepp..rreeeeeee....e.e.eeeeeeee org/ docume nt/9053878.
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A memory-angmented transformer for sequence modeling. In Findings of the Association for Computational Linguislies: AACI-IJCNLP 2022, 2022a. URL httppss//aclatholooyy wwwpoooooyyyyogg///////// or2/2022. findings-aade .9.
Xiaoxia Wu, Zhewei Yao, Minjia Zhang, Conglong Li, and Yuxiong He. Extreme compression for pre-trained transformers made simple and efficient. In NeurIPS 2022, 2022b.
62

---

Published in Transactions on Machine Learning Research (May/202) (024)
Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, and Yuxiong He. Understanding int4 quanti- zation for language models: latency speedup, composability, and failure cases. In Proceedings of the 40th International Conference on Machine Learning, 2023a. CRL attps://dl acm /cc..orr//o//1..555/3311111 08. 3619970.
Xiaoxia Wu, Zhewei Yao; and Yuxiong He. Zeroquant-fp: A leap forward in Ilms post-training w4a8 quan- tization using floating-poli point formats, 2023b, arXiv preprint arXit:230.00978: URL http://arxiv.org/ abs/2307.097)
Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022c. URL http:://ppaareteesene///oo Pid=frjbxzRenf.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Lingpeng Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. In Proceedings of the 61st Arnual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023c. URL https: hatp:s///alaathologyy org/2023. acl-long. 79.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In Workshop on Advancing Newral Nelwork Training: Compulational Efficiency, Scalability, and Resource Optimization, 2023. URL https: https:://peeneie http:///pppaeeeeeeeeeeeterrrrrra wew./e/ortoto/fd 77hjBNfS.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202, 2023. URL https://proceediigg mlr pwwppp/e02/es////eeeeee o23c.html
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficcintt streaming language models with attention sinks. In The Twelfth International Conference Learning Representations, 2024.
URL https:///opt htttpp://openreveer www/eeetttooeeee www.orddddTTsSsaaa
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe V Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Thirtg-scventh Conference on Neural Information Processing Systems, 2023a. URL https: //openreview net//forum/id--Xuuyyeeeeee
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Thirly-seventh Conference on Neural Information Processing Systems, 2023b.
URL https://openreviie: http:///wwrreeeeeeeeeeerrrrriiie pess:foorumidd=uuSSDrr
Mingxue Xu, Yao Lei Xu, and Danilo P. Mandic. Tensorgpt: Efficient compression of the embedding layer in IIms based on the tensor-train decomposition, 2023a, arXiv preprint wxXiv:2307.00522 URL http:/ // arxiv. wwwiiv.og//aiie abs 20t./2207.0000 00526.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https: //openreview.net weet//ooummid==x5nffii
Qifan Xu and Yang You. An efficient 2d method for training super-large deep learning models. In 2023 IEEE International Parallel and Distributed Processing Symposium. (IPDPS), pp. 222 222, 2023. doi: /ttp://IPPSS999992223333333
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, XIAOPENG ZHANG, and Qi Tian. QA-loRA: Quantization-aware low-rank adaptation of large language models. In The Twelfth International Conference on Learning Representations, 2024b. URL https: /ttp::////eeree...a
63

---

Published in Transactions on Machine Learning Research (May/202) (024)
Zhaozhuo Xu, Zirui Liu, Beidi Chen, Yuxin Tang, Jue Wang, Kaixiong Zhou, Xia Hu, and Anshumali Shri- vastava. Compress, then prompt: Improving accuracy-efficiency trade-off of 1lm inference with transferable prompt, 2023b, arXiv preprint (oXi::2305.11880 URL http:///axiiv.ogabbbboo
1111//16.......6.....1 11186.
Fuzhao Xue, Zian Zhong, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoc: Open mixture-of-exerts language models, 2023. URL bettps://githee cww. Swweeeuaaa//ppeaaaae
Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru Hc, and Jingqiao Zhang. Progressively stacking 2.0: A multi-stage layerwise training method for bert training speedup, 2020, arXiv preprint
arXiv:2011.13633 URL http: htpp:///axxiii wwrriiiv.org/aaii org/abs/2011 13635.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers, 2023a, arXiv preprint arXiv:2309.0344 URL awwivv.xvvvvvv/////////////aavv/ org/ abs/ 2309. 03409.
Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, David Farhi, Jakub Pachocki, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyper- parameter transfer. In NeurIPS 2021, March 2022. URL http://wwwwwcoooooooooo
wwtp://st...ooooeeeoeeeeeeeeeeee ch/publication.l weaetiiiontnnnnnnnnneeee waannnglargg-ane laaga--neraaaaa wewral-eetwarkkeeaeeee-a- http:://wwwetettaoooeeererrrrrer
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. ACM Transactions on Knowledge Discovery from Data, 2024. URL https://doi.
1145/3649508
Keming Yang, Zichen Liu, and Philip Cheng, MOSEC: Model Serving made Efficient in the Cloud, 2021, https://githul htt.btttttttt..//gg www/mosecorg/ooeee.cccceeeeeeeee
Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models, 2023b, arXiv preprint arXiv:2304.0444 URL http://arxizi www.aa......//////////ee 04487.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Tzhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventth Conference on Neural Information Processing Systems, 2023a. URL https://openrerie
met/forum?
d=5Xc1ecx0th.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023b. URL https://openreviers net/forumiid-ipp vluYUL-X.
Xingcheng Yao, Yanan Zheng, Xiaocong Yang, and Zhilin Yang. NLP from scratch without large-scale pretraining: A simple and efficient framework. In Proceedings of the 39th International Conference on Machine Learning, volume 162, 2022a. URL https: http///peeeenn....//////// mlr mmr.press//1162/ya222... html.
Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. Masked structural growth for 2x faster language model pre-training, In The Twelfth International Conference on Learning Representations, 2024. URL
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems, volume 35, 2022b. URL https::/proceeeding: neur murips... wttpppcccppp....pppppppppppp pdf.
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Loon Song, and Yuxiong He. Deepspeed-cher Easy, fast and affordable rlhf training of chatgpt-like models at all scales, 2023c, arXiv preprint ar Xiv:2308..000 01320. URL http: arxiv.vvvvv..//a///aaa/aaaaaavaa org/ httg//bb/2330.... 01320.
64

---

Published in Transactions on Machine Learning Research (May/202) /224))
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post- training quantization in llms from comprehensive study to low rank compensation, 2023d, arXiv preprinl arXiv:2303.0880 URL http://arxiv httpaaawwii....iaaaa///e httg//bb/2303.... 08302.
Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Fast on- device inference of moe-based large language models, 2023, arXiv preprinl 1xXi::2308.14322 URL http:
//arxiv.org/ wtt.a///rrggbb///3////// abs//308. 14352.
Jie You, Jae-Won Chung, and Mosharaf Chowdhury. Zeus: Understanding and optimizing GPU energy consumption of DNN training. In 20th USENIX Symposium Networked Systems Design and Imple- mentation (NSDI), Boston, MA, 2023. URL https:://wwt /twp.////////u.iii..mmmm usenix. org/ ion/you.
Gyeong-In Yu, Joo Seong Jeong, Geon- Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: distributed scrving system for Transformer-Baut generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI), Carlsbad, CA, 2022. CRL https: //tp://wwww.... usenix oates/rr/cccoo oog/con ference/ osdoosddii 2tte2//eeeeeeeeeeeerrrrr.
LILI YU, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: Predicting milte sequences with multiscale transformers. In Thirty-sevnnth Con- ference on Neural Information Processing Systems, 2023. URL attps://openrevieet net/forum?id= m02V9Xpz.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models, 2023a, arXiv preprinl ar Xxxx:23044000000 URL http:
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models, 2023b, arXio preprint 05821.
arXiv:2312 05821. URL http:///xxvv...oo/..
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: transformers for longer sequences. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020. URL pettps:/ddi acm org//oi/aalii doi//bs//////////////// 10. 5555/ 34917777744497744
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023. URL https:///pppp https////ppreeree.etttee
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing Jiang, Han Zhang, Lingfeng Deng, Yehong Zhang, Zhe Lin, Chao Zhang, Shaojie Zhang, Mingyue Guo, Shanzhi Gu, Gaojun Fan, Yaowei Wang, Xuefeng Jin, Qun Liu, and Yonghong Tian. Pangu-o: Large-scale autoregressive pretrained chinese language models with auto-parallel computation, 2021, arXie preprint arXiv:210,113369 URL sttt:///rxiie org/ abs/2104. 12369.
Mingshu Zhai, Jiaao He, Zixuan Ma, Zan Zong, Runqing Zhang, and Jidong Zhai. SmartMoE: Efficiently Activated models through combining offline and online parallelization In 2023 USENIX Annual Technical Conference (USENIX ATC), Boston, MA, 2023. URL https: : //www. http://wwwwwwww.t///////// usenix. org/ con ference/atc22 hatpnnceaatc////pee arrenee/At223/preeenttiiii// http://eeeeetttttttaaaeaea
training Sparsely-Activated
Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. Towards the law of capacity gap in distilling language models, 2023a, arXiv preprint ar Xiv:2331 1.07052. URL http: wtwpr/iiv..rgg/b///11113 07052.
65

---

Published in Transactions on Machine Learning Research (May/202) (024)
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. Pool- ingformer: Long document modeling with pooling attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139, 2021. URL https: / http:://ppoceeee http///eeeeeng..... mlr mlrr.press/v33//mm ng21h.htm html.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In International Conference on Learning Representations, 2019. URL https://openrevi
ew http:/forrmmdddi1s1aea
Longleng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, 2023b, orXiv preprint arXi::238.. 033003 URL http: //arxiv. cerzxiv..og/aaa /twg/b///308..... 03303.
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Pruning meets low-rank pframeter-efficiett fne-tuning, 2023c, arXiv preprint arXiw:2305.88403. URL http://arxiv. wwwpii.oorg/bbba orgg/bs///000 18403.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao Adaptive budget allocation for parameter--fficiet fine-tuning. In The Eleventh International Conference on Learning Representations, 2023d. URL https://openreriew net/ffouumid-lg@2uur.
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. I.AMMA-ddapee Efficient fine-tuning of large language models with zero-initialized attention. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevies
orum?id-dan XAHN2W.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022a, arXiv preprinl or Xrx: URL setp://arxxr org.a////g////// 05. 01068.
Tianyi Zhang, Mina Lee, Xiang Lisa Li, Ende Shen, and Tatsunori Hashimoto. TempLM: Distilling language models into template-based generators. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, 2023e. URL https: hatps://aclaattata anthoolyyttygyyy wttptthhhooggy.oo/222333233 findings-acl. 124.
Yiming Zhang, Shi Feng, and Chenhao Tan. Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Melhods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 2022b. URL attps://aclanthee
org/2022 momlp-main... 622.
Yue Zhang, Hongliang Fei, Dingeheng Li, and Ping Li. PromptGen: Automatically generate prompts using generative models. In Findings of the Association for Compulational Linguistics: NAACL 2022, Seattle, United States, 2022c. URL https: //tp///ccaantoooo..... org/ 2022.
naac1.3.
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Re, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Workshop on Efficient Systems for Foundation Models, 2023f. URL https://openreviess wwt.:/wwwwweeeeeeeeeeeeeeeeeeeee mett//orumid-cccieee...
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, 2023g. URL https://openrevee/ http://wwww....teeta/t///t/////.
Jiawei Zhao, Florian Tobias Schacfer, and Anima Anandkumar. ZerO Initialization: Initializing neural networks with only zeros and ones. Transactions on Machine Learning Reseorch, 2022. URL https: / //openreview wtwpre/wwwwweeeeeununeeree
66

---

Published in Transactions on Machine Learning Research (May/202) /224)
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A Survey of Large Language Models, 2023a, arXiv preprint arXiv:2303. 18223. URL http: /ttp:///axxiv../ http//aaivv....raaaa... org/abs/233 .12223.
Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, and Maosong Sun. CPET: Effective parameter-effiient tuning for compressed large language models, 2023b, arXiv preprint arXie:2307.07705 URL httt:://ta/iiii www.wwrrggas......e.. /20.//237.... 2w1.233077007 07705.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. Proceedings of the VLDB Endowment, 16, 2023c. URL attpss//doi..cm org./.....g...... 147 78/3611540 3611569.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023. URL
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Thirby-seventh Conference on Neural Information Processing Systems, 2023a.
URL https: /ttpp:://opnnaeee opwnreviereeeeeeeee/eeeee net/f www.oordduunnnnne....c
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, and Ed H. Chi. Teast-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023b. wttt//orumidd-iin W99999..f...
URL //tpp:/ss///eereveerr
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoe V Le, and James Laudon. Mfxture-ofexxerts with expert choice routing. In Advances in Neural Information Processing Systems, volume 35, 2022. URL Paper- Conference, pddf
neurips....... cttpssccpppppraaepeeeeeepee fil
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2023c. URL httppp//022211111000ccceeee11110
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient. con- text window extension of LLMs via positional skip-wise training. In The Twelfth International Conference on Learning Representations, 2024. URL https://opennreview net/ nett/forommidd332grua00
Bohan Zhuang, Jing Liu, Zizhong Pan, Haoyu Hc, Yuctian Weng, and Chunhua Shen. A survey on efficient training of transformers, 2023, arXie preprinl Xrxi:23002.01 107. URL http://arxiv... 1107.
http:://axxivoor/bbbbaa wwg./bo/////////e /2302.0
Zirui Lin, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi Plug-and-play 2bit kv cache quantization with streaming asymmetric quantization. Arziu, 2023. doi: 10.13140/RGC 2... URL https://rgdoi. 1e.../.....././....... 13140/RG..222 2. 28267.3728.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations, 2022. URL htttps///operrviees wttp://www......ommmrr///////...
67


Reviews:
[{"review_id": "EFQ4qBxzSK", "reviewer": "Reviewer_28rG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a very timely survey in that the importance of efficient LLMs keeps increasing as the size of LLMs and their computational cost become bigger and bigger. It includes comprehensive lists of different directions and corresponding representative papers.\n\nEfficiency is often a broad term since it could be one of model size, training/inference time/memory, and many others. Some methods only target a single metric, and others can benefit multiple aspects. Therefore, clearly demonstrating what efficiency criteria each approach improves will be helpful, especially for people who have their own specific optimization requirements to meet the computational budget.\n\nGiven that there have been many studies on efficient language models since a few decades ago (e.g., efficient transformers by Tay et al. is a good survey on BERT-era) and this paper is a survey on efficient *large* language models, it would be great to have a clear distinction on that. Of course, it is difficult to say how large a language model should be to be called an LLM. Also, many previous approaches for small language models could also apply to LLMs without or with a slight modification. Nevertheless, it would be necessary to make this survey different from the existing survey by narrowing down its scope.\n\nHaving an unbiased position in any particular direction looks objective, but giving some insights on the pros and cons (or limitations) of each direction/method will be advantageous. \nMoreover, the paper should give a deeper view of the relationship between different approaches. For example, some are seamlessly orthogonal, but some are not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper gives as a comprehensive literature review delving into the various methodologies employed to enhance the efficiency of Large Language Models (LLMs). Its primary objective is to serve as a valuable repository of knowledge for both researchers and practitioners, providing a comprehensive understanding of the latest advancements and potentially inspiring innovative contributions. The techniques discussed in this paper are systematically categorized into three primary focus areas: model-centric, data-centric, and framework-centric.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address concerns in the above section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "N1dSAaAxHX", "reviewer": "Reviewer_npES", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n\n1. Efficiency is one of the most important topics in LLMs, and there are many research papers out every week focusing on different types of efficiencies. This survey is very comprehensive to put various efficiencies together. It is a nice read to understand the big picture of these research works.  \n2. Inside each of the sub-topics such as model compression and efficient fine-tuning, the authors did a good job to further construct the taxonomy, and each subsection in this paper is like a nice separate survey on its own.\n\n### Weaknesses\n\n1. Currently the survey reads more like a bland description of each research work inside each subtopic. However, I think it is important to summarize and compare different lines of research that aim for the same goal to understand the connections among different research lines as well as their pros/cons. For example, in model compression, there are different types of approaches such as quantization, pruning, and knowledge distillation. What are the pros/cons of each of these methods? What is the current preferred practice? What is the SOTA for model compression? Why are some of the lines more popular than the others? What is recommended to compress a model and why? Discussing these questions is important for the readers to understand the different tradeoffs and properties from these methods.  \n2. Another example aligning with the first point is on parameter-efficient fine-tuning. It is not wise to group LoRA separately from the adapters, when LoRA is more like a special case of adapters and LoRA is also often referred to as LoRA adapters. Similar to before, among these many different PEFT methods, what are their connections? Why does LoRA stand out as the most popular one? What are their pros and cons? For example, [1] presents a unified view to connect these different methods – I would like to see this kind of discussion for each subsection of the paper to raise more interesting insights and comparisons, rather than only blandly describing a lot of papers. \n\n[1] He et al. Towards a Unified View of Parameter-Efficient Transfer Learning. ICLR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a very comprehensive survey on the efficiency of LLMs. Different from existing efficient LLMs surveys that typically focus on one specific aspect (e.g. inference efficiency), the concept of efficiency here is broad, covering model-centric efficiency such as model compression, efficient pretraining, efficient fine-tuning, efficient inference, and efficient architecture, data-centric efficiency that includes data selection and prompt engineering, and frameworks that compares different open-source libraries. Each of these subcategories are further classified with many details. I appreciate that the authors constructed such a taxonomy and put different types of efficiency into a single survey.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned in the weaknesses above, I think more discussions on the connections and pros/cons of different research lines inside each subtopic should be presented to provide more insights and make the paper coherent. This will offer a clearer picture on what is going on in this regime as well. Currently the paper is merely describing each paper under the constructed taxonomy, with few discussions and insights.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ea4N9MT3jd", "reviewer": "Reviewer_h5tv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe authors claim that they will actively maintain the repo of the paper list in each section; that will be fantastic. \n\nI have worked on many of the fields in the paper, but not all. I have learned a lot from this survey, and I believe it will be beneficial for the community.\n\nThe figures throughout the survey paper (that describe the difference among different groups of methods (e.g., Figure 5, Figure 7, Figure 9, and many others) are very helpful.\n\n\n--- \n\nWeaknesses and other comments \n\nOne complaint I have is that many sections (especially the ones I’m not familiar with) read like a list of paper summaries. There is no way for me to tell which method I should be using at the end. Is one method better than the other? Which methods are mostly used in today’s products (and most useful for industry practitioners)? Which methods are most promising for researchers to dig into? On the other hand, which methods are already abandoned or outdated? It’s unclear what my takeaway is after reading some of the sections. \n- There are many examples throughout the paper. One example is quantization on Page 5. Many methods are enumerated. Which one should practitioners and researchers use? Which ones are already out-of-date? \n- Another example is in Section 2.5 (efficient architecture design); in the paragraph “feature information reduction,” I’m wondering if anyone is still using Funnel transformer or Nystromformer – which ones are outdated, and which ones are still promising today (so that we should keep investigating)?\n- Table 2 in Page 30 (comparing different LLM frameworks), on the other hand, has a clear takeaway and I really appreciate the table \n\n(Personal preference:) I would appreciate it more if the authors highlight the mostly used and most promising approaches in each section. The details should be at least a paragraph and hopefully contain an equation or a figure when necessary.  \n- For example, FlashAttention and FlashAttention-2 are widely used, but the description isn’t detailed enough especially for people who don't know much about hardware / FlashAttention. Hopefully there could be a figure and a discussion on backward pass and how it saves time and memory. \n\nFor quantization (Section 2.1), I would appreciate a discussion on the tradeoff between compression rate and quality. \n\nIn the table detailing pretraining costs (Section 2.1), I’m a bit confused why Llama-2 is in a different unit compared to others.  \n\nIn efficient fine-tuning (e.g., Section 2.3.1), the line of work on hypertuning (https://arxiv.org/abs/2211.12485) could be relevant. \n\nIn efficient architecture (Section 2.5), some of the other subsections also contribute to building today’s long context LLMs. In the long context LLM section, perhaps it’s prudent to mention that some other sections’ approaches are also relevant (e.g., FlashAttention). \n\nIn long-context LLM (Section 2.5.3), I wonder if Memwalker (https://arxiv.org/abs/2310.05029), RAPTOR (https://arxiv.org/abs/2401.18059) are relevant, as well as other recent methods that traverse a memory tree / memory document. More recently (I understand this just came out), gist memory: https://arxiv.org/pdf/2402.09727.pdf. \n\nI wonder if it’s worth referring to other survey papers in Section 3.2.1 (related to prompting).\n\nWhen I was clicking into different papers in the bibliography, I found that the format isn’t consistent. Some papers use initials for author names, but most papers use full names. Some papers have capitalized author names. Different arXiv papers’ citation formats are also very different. Many papers only include the author names and titles without arXiv / conference proceeding reference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a survey paper on efficient LLMs. The overall structure of the survey is shown in Figure 3 – the survey is into three major sections: model-centric methods (model compression, efficient pre-training, efficient fine-tuning, efficient inference, efficient architecture), data-centric methods (data selection and prompt engineering), and frameworks (e.g., DeepSpeed, Megatron, etc.)", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address every point in the \"weaknesses and comments\" section above if the authors think they make sense.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers and I uniformly found this survey to be informative and timely. It serves as a great collection of papers, and if the repository is continually updated as promised, it will be great a resource that I imagine myself pointing students to in the future.

The main weakness of this paper is the lack of a stronger position on the pros and cons of different methods. The update of the paper does a good job of comparing approaches within a single section, particularly by reporting results to compare multiple approaches. However, there is less insight or guidance on which of the many research areas in the overall taxonomy are most worth pursuing, or which have yielded the strongest results thus far. There is a brief discussion added about quantization, for example, but it mostly says that there has been a lot of work, yet tradeoffs remain.

I see this as somewhat difficult to add within the current structure of the survey. The authors are encouraged to take a step in the direction of adding comparison where possible (reviewer npES particularly thought that such comparisons would be valuable), but I believe the paper is above the bar for survey papers as is.
Decision: Accept (poster)