Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Authors: Muru Zhang
Mayank Mishra
Zhongzhu Zhou
William Brandon
Jue WANG
Yoon Kim
Jonathan Ragan-Kelley
Shuaiwen Leon Song
Ben Athiwaratkun
Tri Dao
Venue: ICML 2025 poster (2025)
Keywords: Language Model
Inference
Distributed Inference
Architecture
Efficiency
Parallelism
Primary area: deep_learning->large_language_models
TL;DR: Architecture modification to allow full overlapping of communication within Tensor Parallelism. 29% inference speed up on 70B size when applied to Transformer

Abstract: Large language model inference is both memory-intensive and time-consuming, often requiring distributed algorithms to efficiently scale. Various model parallelism strategies are used in multi-gpu training and inference to partition computation across multiple devices, reducing memory load and computation time. However, using model parallelism necessitates communication of information between GPUs, which has been a major bottleneck and limits the gains obtained by scaling up the number of devices. We introduce Ladder Residual, a simple architectural modification applicable to all residual-based models that enables straightforward overlapping that effectively hides the latency of communication. **Our insight is that in addition to systems optimization, one can also redesign the model architecture to decouple communication from computation.** While Ladder Residual can allow communication-computation decoupling in conventional parallelism patterns, we focus on Tensor Parallelism in this paper, which is particularly bottlenecked by its heavy communication. For a Transformer model with 70B parameters, applying Ladder Residual to all its layers can achieve 29% end-to-end wall clock speed up at inference time with TP sharding over 8 devices. We refer the resulting Transformer model as the Ladder Transformer. We train a 1B and 3B Ladder Transformer from scratch and observe comparable performance to a standard dense transformer baseline. We also show that it is possible to convert parts of the Llama-3.1 8B model to our Ladder Residual architecture with minimal accuracy degradation by only retraining for 3B tokens.

Ladder-Residual: Parallelism-Awar Architecture for Accelerating Large Model Inference with Communication Overlapping
Muru Zhang 12 Mayank Mishra * Zhongzhu Zhou 14 William Brandon
Jue Wang' Yoon Kim
Jonathan Ragan-Kelley
Shuaiwen Leon Song Ben Athiwaratkun
Tri Dao 16
Abstract
1. Introduction
Large language model inference is both memory- intensive and consumnnn,,nn often requiring dis- tributed algorithms to efficiently scale. Various model parallelism strategies are used partition computation across multiple devices, reducing memory load and computation time. However, using model parallelism requires communication of information between GPUs, which limits the gains obtained by scaling up the number of de- vices. We introduce Ladder Residual, a simple ar- chitectural modification applicable to all residual- based models that enables straightforward over- lapping to hide the latency of communication. Our insight is that in addition to system op- timizations, the model architecture can also be redesigned to decouple communication from computation. While Ladder Residual can allow communication compptttiiioooottiitinoooo decoupling in con- ventional parallelism patterns, focus Ten- sor Parallelism in this paper, which is particularly bottlenecked by its heavy communication. For a Transformer model with 70B parameters, apply- ing Ladder Residual to all its layers can achieve 29% end-to-end wall clock speedup inference time with sharding over 8 devices. train a 1.2B and 3.5B Ladder Residual based Transformer models from scratch and observe comparable per- formance to standard dense transformer baseline. We also show that it is possible to convert parts of the 1 8B model to our Ladder Residual architecture with minimal accuracy degradation by only retraining for 3B tokens.
With the rapid scaling of Large Language Models (LLMs) (Smith et al., 2022; Workshop al., 2023; Brown, 2020), the compute and memory requirements for training and inference have grown significantly. Tensor parallelism (TP) (Shoeybi et al., 2020) is a widely used model paral- lelism technique that partitions the weights and intermediate activations across multiple GPUs. In contrast pipeline parallelism (Narayanan et al., 2021) and data parallelism (Li et al., 2020), which rely processing independent batches of user requests on each device, TP enables multiple devices to cooperate to process a single request, therefore in theory allowing infinite scaling given a sufficient number of proces- sors. However, TP requires synchronizing the partitioned intermediate activations across the GPUs. This synchroniza- tion blocking A.l R.lReduce operation on the activations across the GPUs and is therefore bottlenecked by the net- work communication latency. Even for GPUs connected via fast interconnects (like NVLink (NVIDIA Corporation, 2024)), the communication costs can account for 38% of the latency at inference time when running 70B transformer with batch size and TP world size of 8.
Past works have attempted overlap the communication latency of TP by overlapping computation and commu- nication. Chang et al. (2024) write fused kernels for All followed by matmul and matmul followed by ReduceScatter. They break down matmuls into tiles hide the latency of communicating a tile with the com- putation of subsequent tiles. Jangda al. (2022) propose CoCoNet, a domain-specifee language to express distributed machine learning workloads. They propose generate ef- ficient GPU kernels for computation and communication using a custom compiler for the DSL. This approach has limited applicability with existing frameworks like PyTorch (Paszke et al., 2019) and JAX (Frostig et al., 2018) since the user needs be well-acquainted with the DSL to gen- erate efficient GPU kernels. Moreover, with the breakneck pace of accelerator and interconnect changes, these low- level systems optimizations require rewrite for every new generation of hardware. However, there is fundamental limit to how much communication latency can be reduced
Together AI University of Southern California 'MIT- IBM Watson Lab "University of Sydney "Massachusetts In- stitute of Technology 'Princeton University. Correspondence to: Muru Zhang <muruzhan edus, Mayank Mishra <mayank31 11998 gmail.com>, Tri Dao <tri@tridaome>>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
1s - hofre) 177
Table I. Inference speedup from applying Ladder Residual on a Transformer model. The test setup is 1024 prompt length. 512 gen- erated tokens, batch size 4, and TP world size of for IB, 3B, 8B, 34B, 70B, 176B model and TP world size of 16 for 405B model (using two nodes). The models are designed according to Llama model families (and 2loom-176B) The speedup value is calculated by comparing Ladder Transformer with Standard Transformer's in- ference throughput in tokens per second. We measure the speedup both with and without P2P communication (modern server nodes usually support P2P communication). Note that disabling P2P communication significantly increases the A1 Reduce latency.
MUP
MLP
omendor
(,05))
Atta ston
Ay(t)) 4-7
ky(x)
MLP
12. 6y50) 1.n
1 hyfr)) 1
Aneetion
Agft.)
Model size
P2P disabled
P2P enabled
Figure/ 1. Illustration of a standard Transformer block (left) and Ladder Residual block (right). The blue edge denotes the residual connection. In Ladder Residual, the residual connection remains the same while each module h, takes the stale input 72-2.
IB
1.39x
1.56x
3B
1.50x
1.57x
8B
1.40x
1.46x
34B
1.47x
1.44x
70B
1.59x
1.29x
176B
1.54x
1.35x
405B
1.57x
1.31x
with these approaches which do not change the underlying model architecture. Instead of pure hardware optimizations (e.g., larger NVLink domain connecting 36 or 72 NVIDIA Blackwell GPUs) or pure low-level software optimization (e.g., rewriting all matmuls to overlap with communication), we explore model architectural changes that would enable a reduction in communication latency while maintaining accuracy. This makes our approach quite simple to apply in practice using high level machine learning framework like PyTorch (Paszke et al., 2019) or JAX (Frostig et al., 2018) without writing any lewelevel device code.
running a 70B Transformer with Ladder Residual can be around 30% faster than the standard Transformer. In Ta- ble 1, we provide the inference speedup on Transformers of different sizes. The proposed Ladder Residual method can also be used to accelerate other forms of parallelism, although we focus on Tensor Parallelism in this paper as it is particularly bottlenecked by the heavy communication. Our method obtains 5-7% training speedup when training an 8B model with 8k context length on 64 H100s with 3D paral- lelism across the GPUs (Tensor Parallel. Sequence Parallel and Fully Sharded Data Parallel (FSDP) (Zhao et al., 2023; Rajbhandari et al., 2020)), but we decide to focus the inference speed ups since training with pure FSDP is usually faster because weights can be pre-fetched and gradient syn- chronization using ReduceScat ter can be overlapped in FSDP making communications in FSDP non- blockiigg
In current LLMs, communication is blocking because there is a sequential structure between communication and compu- tation in existing model designs: we wait for communication in order to prepare the correct input for the next computation. In the prevalent residual- based architectures, the computa- tion flow can be written as 1 = h4+ (x+) + Iy, where 2; is the residual stream after layer and ha+1 is the compu- tation at layer i + 1. Notice that the communication of i; needs to be done before executing h,+1 if h1 is partitioned across devices. Liu et al. (2023b) found that activation changes slowly in Transformer, as the norm of each update (x:) is small compared to the residual. Based on this observation, we hypothesize that maintaining the regular residual connection is enough to restrict the representation shift, and we can feed each module a "stale" input to create overlapping opportunities.
Because of the widespread use of Transformer (Vaswani, 2017) based Language models, we focus on applying Ladder Residual on Transformer models in this paper, and we call the resulting model as Ladder Transformer. However, it should be noted that Ladder Residual is compatible with any residual based model architecture. We conduct experiments under two scenarios to verify if we can maintain the same performance as standard Transformer:
Pretraining from scratch: We train a 1.2B and a 3.5B parameter Ladder Transformer model with 100B to- kens on the FeneWeb-edu dataset (Lozhkov et al., 2024) and compared it with the standard transformer of the same size trained on the same amount of tokens. We find that the Ladder Transformer matches the perfor- mance of the standard Transformer model.
We propose Ladder Residual, simple change where we reroute the residual stream after module i (instead of module i) as input to module i+1: Z441 = h4+1 (x;-1)+ +2; With this design, the computation of h4+1 is decoupled from the communication of z, enabling straightforward overlap- ping to hide the latency of communication. Figure/ shows how Ladder Residual can be applied to the Transformer architecture. At inference time with TP world size of 8,
Post-training adaptation: We take the pretrained
2

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Llama-3.1-3 3.1---..11111 1-8B-instruct model (Dubey al., 2024) and apply Ladder Residual the upper half layers. We then fine-tune it on 3B tokens to adapt to the represen- tation shift. With this relatively light retraining, we can obtain a hybrid Ladder Llama that is on par with the original Llama model across a variety of multiple- choice and generative tasks.
where the denotes partial-sum that requires an Al Reduce replicate full output across all the GPUs.
A Transformer with N layers needs to perform the A 1llReuuce 2N times and this can account for 38% of the inference latency for a 70B model using TP world size of 8, even with NVLink interconnect. If P2P communi- cation is disabled, AllReduce latency can account for over 50% of the end-to-end latency. Modern nodes connect GPUs via NVLink but usually have more than 8 GPUs per node, due to limited PCIe lanes, power density and cooling constraints in datacenters. There is a steep falloff in communication bandwidth and sharp increase in latency when the communication happens across nodes either over InfiniBand or Ethernet thus making scaling TP practically infeasible outside a node.
In our paper, we propose change the model architec- ture to accelerate model parallelism without touching low-level kernels, making it easily deployable on any hardware. We show that such architecture modification per- forms on-par with the standard transformer. As model size grows, multi-gpu or even cross-node serving will become more and more important, and our research provides a fresh perspective on designing a model architecture with paral- lelism optimizations in mind. Such design can be applied to any architecture that suffers from blocking communi- cation, although in this paper we conduct experiments on transformer-based language models due to their widespread use and popularity.
Algorithm 1 Ladder Transformer Layer with Tensor Paral- lelism. Note that the AsyncAl www.cclleeeuuuco Reduce (ARR) returns a handle which is passed to the next layer.
I: function LAYERressidual, attn out, mippoott, attn_work, mlp_work)
2:
attn.work.wait/ residual +- residual + attn_out
2. Tensor Parallelism Background
3:
4:
Tensor parallelism (TP) (Shoeybi et al., 2020) is widely used technique in distributed training/inference. It partitions weights and activations across devices and performs partial computations on each device. Consider a sequence of 2 linear layers with weight matrices A and B and input acti- vation X that is running on GPUs (TP world size of 2), we split A along the output dimension into [A1, A2], and
outn + AttenttionNormmpesscuull
attn.out + Attention(attr.out) attn attn.work +- AAR(attn.out)
8:
9:
mip work.wait ()
10:
residual + residual
residual + mlp.out
11:
B1
12:
mlp out + Mr.NNor((resi.dua))
split B along the input dimension into
Then the out-
B2
13:
mlp.out MLP(mlp out)
put of the sequence of the 2 linear layers can be computed
14:
mlp.out, mlp_work +- AAR(mlp.out)
as (X = (X B1 + B2 and we effectively partition the computation on the two devices. The final sum- mation requires an All Aelued operation to aggregate the partial sums on each device, which introduces communica- tion overhead. The A1 RRducce overhead increases with increasing message size and increasing number of devices participating in the Reluce. A transformer layer con- sists of an attention block and an MLP block: both can be considered as a sequence of two matrix multiplications and therefore fit into the tensor parallelism paradigm described above. Thus each transformer layer contains 2 A.l Reduce operations: one for attention and another for MLP. Denoting the input to the ith block as Ti-1, the transformer can be viewed as the following sequential structure:
15:
16:
return residual, attn out, mlp out, attn_work, mlp.work
17:
18:
end function
3. Ladder Transformer
In this section, we introduce the Ladder Residual archi- tecture when applied to Transformer and benchmark its efficiency under various model sizes and generation setups.
3.1. Architecture description
In Equation 1, the Al AllReduce operation is blocking the next block from execution since 14+1 requires x; as the input. Ladder Residual mitigates this problem by routing the -;- to block h4+ effectively making the input of h441 independent of the output of the AllReduce, therefore allowing overlapping A Reducce ce with h441-
z 1x:-11
2; Al (11Reducees;)) + :;-1
(1)
4/41 (z;)
Z1t1 = AllReduce(((t.) +
3

---

eventually Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
batch
4,
standard
350
transformer P2P-1
standard transformer P2P=0
standard transformer P2P=0
100
transformer P2P=1
trensformer P2P=1
ladder transforner P2P=0
ladder transformer P2P=0
300
parallel
perallel attn P2P=0
parallel attn F2P=0
upper
upper bound
takang/ae
80
takenstes
250
200
60
150
40
batch size
batch size
batch 16, 708 model
batch size 64, 708 model
1100
standard transtomer
standard transtormer
standard transtorerr P2P-0
2400
standard transformer P2P-0
ladder transformer
ladder transformer P2P-1
1000
ladder transformer P2P-0
ladder transformer P2P-0
perallel attn P2P=1
parallel P2P-1
perallel attn P2P=0
parallel attn P2P=0
upper bound P2P=1
2000
upper bound P2P=1
tokors/ac:
tolterster:
1800
600
1400
500
batch size
batch size
Figure 2. Improvement in end-to-end inference throughput achieved by cetpuuniicatioefffccieeeee architectures relative to a standard transformer, benehmarked Llama-3 70B. Standard refers to the regular Llama-3, and Ladder is Llama-3 with our Ladder Residual architecture. Ladder Residual architecture can achieve up to 29% greater throughput than the standard Transformer. With slower communication (P2P disabled or P2P=0), we observe speedups up to 60%. All experiments were conducted on a generation task with 1024 prompt tokens and 512 completion tokens. Missing data points indicate CUDA OOM.
Specifically, change the computation flow of Equation into:
To implement the Ladder Transformer, following the de- scription of Equation the Attention's output. This returns a handle that can be used to synchronize the output to ensure that the AsyncAl Reduce has finished. It should be noted that NCCL collectives in PyTorch always run on a different CUDA stream than the default compute stream used by PyTorch thus making them asynchronous. As soon the AsyncAllReduce for Attention is called, synchro- nize the previous layer's MLP's output by calling wait on the previous layer's 's MLP's All allReduce handle and sub- sequently the CPU launches the kernels for MLPNorm and then MLP on the default compute stream calling the AsyncA1 Reduce for MI.P. The handles for these NCCL operations are then passed onto the next layer
call AsyncA1 1Reduce for
4; (x(2122)
2; Al AllReduce(1;) + x;-1
#|41 het1 (1
4i+ Al (AL1Reduce((111) +
Note that the residual stream of each block still takes the output from the previous block as usual, this ensures block can still process information from all previous 2 blocks.
3.2. Inference Implementation
Ladder Residual Implementation: We present the Lad- der Transformer's layer's pseudo-code in Algorithm

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Table Detailed breakdown for prefill latency, decode latency and generated token/sec improvement (%) for 70B model. The speedup
(%) is calculated by using latency of optimized model divided by original model. All the experiments are based on batch size 1. TP world
size of GPUs.
Model
Prefill Latency Improvementt (%)
Decode Latency Improvement (%)
Token/sec Improvement (%)
UgperBound-LLamaa- (P2P=1)
30.54
30,00
42.90
Parallel- Llama 70B (P2P=1)
5.42
18.04
21.75
Lalder-LJama-708 (P2P=1)
5.78
23.71
30.79
UpperBBound Laauund---mmma Lama- 70B (P2P=0)
35.88
52.71
110.7
Jama-el-L (P2P=0)
14.92
28.73
40.07
Ladder-Llama-700 (P2P=0)
6,94
37.71
59.87
which uses them for synchronization when needed.
are performed in parallel. While not proposed for accel- erating Tensor Parallelism, we realize this architecture can effectively half of the communication, with the extra benefit of being able to fuse attention and mlp together, therefore we consider it as an alternative to Ladder Residual.
Alignment with Real-World Scenarios: To evaluate the practical benefits of Ladder Residual, we integrated this mechanism into standard Llama-like Transformer. Build- ing upon 9pt-fast (PyTorch Labs, 2024), we partition the weights of the attention and feedforward modules for ten- sor parallelism to optimize inference speed. We use CUDA graphs (Coleman, 2020) via PyTorch compile (with "reduce- overhead" mode) to generate static computation graphs for both the prefill and decode phase to reduce CPU kernel launch overheads which can be big bottleneck especially during the decode phase of Transformer inference. PyTorch compile also additionally helps in accelcrating inference and reducing the memory footprint for inference by emit- ting more efficient kernels.
Ladder Residual: The architectural optimization we propose to overlap computation with communication required for Tensor Parallelism.
Communication Free Upper Bound: An upper bound that removes all communication operations in the model represent the theoretical maximum speedup achievable.
3.3. Faster Inference with Ladder Residual
Note that different Transformer-basedd models usually have slightly different designs. We choose to use Llama-3's model architecture since it's one of the most widely used models. The benchmarking results here are a good repre- sentative of variety of different design choices since the communication patterns mostly the same across various transformer variants.
In this section, we benchmark Ladder Residual under var- ious scenarios and show that across various model sizes, batch sizes, and TP world sizes, Ladder Transformer can obtain considerable speedup over the standard Transformer.
We also benchmark under the case where P2P communi- cation is disabled (for testing in scenario with increased AllReduce latency), and show that our method can obtain more than 50% speedup. Finally, we consider cross-node Tensor Parallelism, which can be necessary for serving large models like Llama 3. 405B, and show that Ladder Residual can improve inference throughput by more than 30%.
To simulate various inference scenarios. we select multiple experimental configurations. The prompt length and gen- cration length is fixed to 1024 and 512 respectively, while we vary the tensor parallel world sizes among 1, 2, and 8, and batch sizes among 1, 4, 16 and 64 understand perfor- mance under different generation settings. All benchmarks are done on NVIDIA H100 GPUs.
3. SETUP
We benchmark several algorithmic variants to evaluate their performance in large-scale language model inference. The candidates include:
To evaluate the impact of hardware communication capabili- ties, we attempt to simulate the performance in the presence of slow interconnects. We set NCCL_ P2C _DD to disable the point-to-point communication. This significantly slows down the NCCL communications and allows us to assess the performance of difference algorithms in varying communication environments. We observe a significant slowdown (for 8 GPUs) for AllReduce when disabling P2P communication. In our experiments and figures, we refer to this setting as P2P=0 or P2P disabled. Unless other- wise stated, P2P is always enabled (or P2P=1).
Standard Transformer: The standard transformer im- plementation.
Parallel Attention and MLP: Following the PaI.M par- allelization strategy (Chowdhery et al., 2022: Wang & Komatsuzaki, 2021), we fuse the weights of the query, key, value, gate, and up projections into a single matrix. The outputs are then split, and the attention and swiglu
5

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
P2P enabled (TP 16)
Latency vs. Throughout per GPU Models Paaree Ogtimal only)
transformer
paralici
upper bound
2.2
colenosec specoup
2.0
Denna 703
Dense 708 Paresa Fronter
1.8
Ladder
Upper Francier
1.6
Upper
1.4
Labency (mes]
1.2
Figure 4. Pareto frontier of completion latency vs aggregate throughput per GPU different 70B-scale model architectures in batched inference setting. For each architecture, over both batch size and TP world size to find the Pareto-optimal configurations. Using less TP size results in higher throughput while using a higher TP size optimizes the latency. both have its use-case and found that ladder architecture achieves Pareto improvements over the standard transformer architecture and the parallel transformer. All experiments measure end-to-en time on generation task with 1024 prompt tokens and 512 completion tokens per sequence.
1.0
16
batch size
sweep
P2P disabled (TP 16)
2.4
perallel
upper bound
2.2
tolararsec soossup
1.8
1.6
the Ladder Residual architecture significantly outperforms the standard transformer architecture, and that these im- provements larger when communication is slower. The throughput gains from adopting the Ladder Residual archi- tecture increase as the TP degree increases, reflecting the greater proportion of run time spent in communication rel- ative to compute as we partition the computation across a larger number of devices. Lastly, as shown in Table the amount of improvement decrease as we increase from 8B to 70B when running with P2P communication, the compu- tation scales faster than communication. However, the trend is reversed when running without P2P communication, due to the much higher cost of communication in that scenario.
1.4
1.2
1.0
16
batch size
Figure 3. End-to-end inference throughput improvement Llama- 3-405B on a generation task with 1024 prompt tokens and 512 completion tokens. Here each with 8 GPUs, connected with InfiniBand. Due to the high cost of noos communication, Ladder Residual architecture is able to achieve more than 30% improvement across various batch sizes with P2P enabled around 50% with P2P communication disabled.
TP size 16 across two nodes
Finally, we consider serving model that is too large to be loaded on one node. Cross-node TP communication is very expensive, however the common practice of using intra-node TP with cross-node Pipeline Parallelism (PP) is dependent batch size to reduce gpu idle time (for example, with batch size = 1, half of the GPUs will be idle at any given time). With the speedup from Ladder Residual, cross-node TP can be a viable option. We benchmarked 405B under such setting in Figure 3, found that even for nodes with fast InfiniBand interconnect. Ladder Residual can achieve more than 30% throughput improvement across batch sizes.
3.3.2. BENCHMARKING
We characterize the inference efficiency improvements en- abled by Ladder Residual in three different ways.
First, we measure the best latency achievable (batch size 1, TP degree 8) using both the Ladder Residual architecture and a standard transformer baseline. We report end-to-end latency broken down by inference phase (prefill vs decode) regime, both with and
in Table 2. In this without NVLink, Ladder Residual outperforms the parallel attn-mlp alternative in both prefill and decode latency.
4. Experiments and Results
Second, we measure the throughput across different TP world size and batch sizes for 70B model in Figure 2. In these throughput-orienter experiments, we again find that
We empirically verify our assumption that applying Ladder Residual does not hurt the performance. We show that Ladder Residual can be either used when training from
6

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
scratch, or be applied pre-trained model with hybrid adaptation, and in both cases, the performance is par with the original architecture.
4.2. Post-training adaptation
We investigate the feasibility of directly applying Ladder Residual an existing pre-trained model. We applied Lad- der Residual to the upper half of the Llama-3.1 8B Instruct to keep the performance since we found that touching the lower layers can destroy knowledge that is hard to recover without large-scale retraining. We evaluate the adapted models on benchmarks across a range of domains: ac- curacy on MMLU (5-shots) (Hendrycks et al., 2021) and ARC-Challenge (ARC-C, 25-shots) (Clark al., 2018), normalized accuracy on OpenBookQA (OBQA) (Mihaylov et al., 2018), Hella (HS, 10-shots) (Zellers et al., 2019), and TruthfulQA (TQ, mc (Lin et al., 2022). exact-match accuracy on GSM8K(GSM, 8-shots) (Cobbe et al., 2021), pass@) on HumanEval-+(HE)) (Chen et al., 2021), aggre- gated accuracy on IFEval (Zhou et al., 2023), and length controlled win rate (Dubois et al., 2024) against gpt4-turbo AlpacaEval (Li et al., 2023c). The evaluation of Hu- manEval+ is conducted with EvalPlus (Liu et al., 2023a), Al- pacaEval is done with the AlpacaEval2 library, and the rest of the evaluations are conducted with the LM-Evaluation- Harness library (Gao et al., 2024).
4.1. Training From Scratch
We train 1.2B and 3.5B Ladder Transformer from scratch and compare its performance with an equally sized standard Transformer model. All our models are trained on 100B tokens of Web-eed dataset (HugginggaceeFW 2024) using the StarCoder tokenizer (Li et al., 2023b). We also compare our model with the Parallel Attention/MLP archi- tecture (Chowdhery et al.. 2022: Wang & Komatsuzaki, 2021) which parallelizes the computation of the attention and the MLP module. This effectivvly reduces the commu- nication cost by 50% for the tensor parallel A1 RelRed in both the forward and backward computation.
4.1.1. EXPERIMENTAL DETAILS
We use DDP (Distributed Data Parallel) (Li et al., 2020) to train the 1.2B and HSDP (Hybrid Sharded Data Par- allel) (Zhao al., 2023; Rajbhandari et al., 2020) to train the 3.5B models. For HSDP, we shard the model within node (equipped with 8x H100 GPUs) and repli- cate the model outside the node. We use mixed precision training (Micikevicius et al., 2018) in BF16 (Kalamkar et al., 2019) with gradient accumulation and gradient in FP32 for training sta- bility. We train all our models with 2048 context length with a batch size of 4M tokens in a batch. The models are trained with cosine scheduler with warmup of 8B tokens to peak learning rate of 3x x 10 4. The learning rate is then decayed over 92B tokens to 3 x 10-5.
1.2.1. EXPERIMENTAL DETAILS
We convert state-of-the-art open-source model, Llama- into a hybrid Ladder Residual structure, by applying Ladder Residual to the upper half of the model (layers 16-32 for LLaMA- 3.11 B-BB -B-ttruuutt.. We call this vari- ant Hlybrid-Laddeer-ee 8B- 161. in Table 4. We also experiment with more aggressive adaptation where we applied Ladder Residual to the layers 12-32 and we call this experiment Hybrid-Ladder-SBB -B-20.. We conduct supervised fine-tuning (SFT) for the resulting model on the 7M subset and the Gen subset of the Infinity-Instruct dataset', which contains 3B tokens. We train for 2 epochs with Adam optimizer with a batch size of 32. We use 5 x 10-6 learning rate with 200 steps of linear warmup, followed by cosine annealing to the end. We use Axolot for our SFT experiments and Open LM Engine for our pretraining experiments.
3.1
Aal.eeduce/eduueeecaee
We use Eleuther LM eval harness (Gao et al., 2024) to evaluate models on ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017) and Winogrande (Trinh & Le, 2018). We also evaluate perplexity on Wikitext (Merity et al., 2017).
4.1. RESULTS
As shown in Table 4, after adaptation, there is a huge perfor- mance drop mainly on generative tasks as the computation flow is messed up. But after light retraining, the hybrid Lad- der Llama is able to reach the same level of performance with the original Llama. By applying Ladder Residual on the last 16 layers, we can obtain 21 end-to-end wall clock speed up for inference with TP world size of 8 and batch
The full results can be found at Table 3. We find that at the 1.2B model scale, Ladder Transformer achieves perfor- mance similar to Standard Transformer while beating the Parallel Transformer. At 3.5B parameter scale, we find that Standard Transformer is better than Ladder Transformer model with 3.2% lower perplexity and 1.2 points of abso- lute difference in accuracy. The Parallel Transformer has almost the same performance as Ladder Transformer at the 3.5B scale.
ruct
hntopa://github.cm httph//w...oomooooooaaaa---eeeee
axoloti
wegp:tthb..cmmooee-t--eeegee Im-engine
7

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Table 3. Performance of three architectures under two sizes, trained Weh-Weh-edu for 10DB tokens.
Model
ARC-C
ARC-E
HellaSwag
PIQA
SciQ
Winogrande
Averae
Wikitext PPI.
Stundund-Transsinnn Transformer- 1.2B
34.22
70.33
41.10
71.49
87.30
55.41
59.98
18.54
Parallel. Transformer- 1,2B
30.46
67.97
40.35
71.16
87.40
55.17
58,75
18.95
1.2B
31.31
67.76
41.18
71.49
86.60
55.17
58.92
18.42
Stundard Transformer- 3.5B
38,99
74.12
46.48
74.59
92.00
58.48
64,11
14.48
Parallel- Transformer-3.58
38.48
73.02
45.55
73.67
90.00
57,46
63.03
14.96
Ladder-Transfommsssee Tranffrrmc-33588
36.77
72.43
45.66
73.72
89,90
58.96
62,91
14.90
Table All models are either LLama-3.1 models or are adapted from L.ama-3.1 8B Instruct in this table, Performance comparison across various benchmarks. Zeroshot denotes directly applying Ladder Residual without any retraining. nL denotes that layers of the Llama-3.1 1.1-a-3.1-B--- ----SB--ntttuctt are adapted with Ladder Residual.
Model
MMLUU
ARC-C
OBQA
TQ
Average
68.14
60.32
43.00
80.04
36.84
84.99
60.40
52.57
18.69
56.11
Hybrid Ladder 8B zeroshot
63.19
56.57
42.60
77.70
35.50
10.54
30.50
46.25
11.99
41.65
Hybrid-Ladder- 1efder----16---erranne
65.93
59.13
42.20
78.86
39.66
80.29
59.10
59.02
21.95
56.24
62.31
59.90
42.60
77.49
36.72
76.19
48.80
59.05
21.72
53.86
size of 1. Our results demonstrate the potential of Ladder Residual being drop-in adaptation technique to make the model faster without sacrificing performance. We addition- ally experiment with applying Ladder Residual to the last 20 layers of Llama and found that it leads to a slight drop in performance. There is a chance that with longer adaptation, or smarter adaptation techniques like distillation or iterative training, we can obtain Ladder-Llane with more layers adapted. We leave the further exploration to future work.
curacy across bechmarks and wikitext perplexity while still achieving much higher inference throughput.
6. Related Work
Communication overlapping in parallelism Overlap- ping communication has been a widely explored area in prior works in order to achieve higher performance for distributed training. For Tensor Parallelism, prior works (Jangda et al., 2022; Wang et al., 2022; NVIDIA, 2023) decompose the communication into more fine-grained operations in order to find computations with no depen- deney to overlap. Our work doesn't 't rely on such decompo- sitions and therefore doesn't 't require Sequence Parallelism to handle the partitioned activations before all all-gather. In FSDP (FairScale authors, 2021), the all-gather commu- nication is usually prefetched to be overlapped with the communication. Pipeline Parallelism (NVIDIA, 2023; Li et al., 2023a; Lamy-Poirier, 2023) on the other hand, chunks the data into mini-batches which creates more opportunity for overlapping. Compared to these other parallelism ap- proaches, TP has the advantage to be independent of the batch size or sequence length, and can partition the compu- tation much as possible given enough GPUs in theory.
5. Discussion
5.1. Compatibility with other Parallelism Techniques
It should be noted that Ladder Residual is fully compati- ble with other model parallelism techniques like Pipeline Parallelism (PP). The A1 Reduce communication for TP is still asynchronous except at the PP boundary where the All AllReduce needs to be waited upon to complete. Once the AI Reduee result is available, we can for- ward 3 tensors (residual) current current.mlp.out mlp-output and current.attention putt to the next pipeline stage. It should be noted that this is still pretty cheap since generally the P2P communication during inference latency bound and can be implemented easily using the batch i ssen i irec API in PyTorch. Our approach also seemlessly works with Distributed Data Parallel (DDP) and Fully Sharded Data Parallel (FSDP).
Efffciency-awand architecture improvements Prior works have explored various alternative designs for Trans- former, for example parallel attention and mip (Chowdh- ery et al., 2022; Wang & Komatsuzaki, 2021), linear atten- tion (Katharopoulos et al., 2020), Grouped Query Atten- tion (Ainslie et al., 2023), Cross-Layer Attention (Brandon et al., 2024) to improve the training and inference efficiency. Some of these variants are more widely adopted than others, due to the degree of impact they have on performance and efficiency. Past works have also considered adapting an existing checkpoint to these efficient variants. Ainslie et al.
5.2. Comparison to a 30% larger Ladder Transformer
Since the 70B Ladder Tranformer model achieves 30% higher tokens/sec compared to the 70B standard transformer as shown in Table 2, we compare the standard transformer to a 30% larger ladder transformer in Table 5. We find that the 30% larger ladder transformer is better than the standard transformer both the 1.2B and 3.5B scale on average ac-
8

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Table Performance of the Standard Transformer compared a 30% larger I. Ladder Transformer at 1.2B and 3.5B scale, trained on FineWeb for 100B tokens.
Model
ARC-C
ARC-B
HellaSwag
PIQA
SciQ
Winogrande
Average
Wikitext PPL
Tokens/see
Standard- Tranaaadd-oosfooe-----. 1.2B
34,22
70.33
41.10
71.49
87.30
55,41
59,98
18.54
1008.22
Laddey- Transforeer- 1.55
33.96
70.16
42.58
71.98
87,90
55.41
60.33
17.47
1277.66
Standard Transformen-..3 3.5B
38.99
74.12
46.48
74.59
92.00
58.48
64.11
14.48
949.6
Ladder: Transforer--4.5
40.96
75.00
46.81
73.99
90.80
57.70
64.21
14.05
1217.71
(2023) extracted grouped-query attention from a multi-head attention model, and Wang al. (2024) converted Llama model to a Mamba model by retraining on 50B tokens to close the performance gap. Comparatively, our adaptation is much lighter (3B tokens), showing that the representa- tion shift introduced by Ladder Residual is easier to recover. Wang et al. (2024) considered converting a Llama model to Mamba model and used distillation retrain the con- verted model. Such training paradigms that specifically tune the model to align with the original model could further improve the Ladder Residual based models.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Brandon, W. Mishra, M.. Nrusimha, A., Panda, R., and Kelly, J. R. Reducing transformer key-value cache size with cross-layer attention, 2024. URL https:
watrxx..org/ab///200 12981.
Brown, T. B. Language models are few-shot learners. arXiv preprint vrXiv:2005.14165 2020.
7. Conclusion
Chang, L.-W., Bao, W., Hou, Q. Jiang, C., Zheng, N., Zhong, Y., Zhang, X., Song, 7.,, Jiang, 7.,, Lin, H., Jin, X., and Liu, X. Flux: Fast software-
We introduce Ladder Residual, architectural modifications that allow overlapping communication with computation- for model parallclism. We show that when running Ten- sor Parallelism, Ladder Residual can achieve great speed up across various model sizes, batch sizes, and number of GPUs. When applying Ladder Residual to Llama-3.1 8B Instruct, we only need lightweight retraining to reach the same level of performance as the original model while be- ing 21% faster, showing its potential to be plug-in for any pretrained Transformer. We also trained 1.2B and 3.5B Ladder Transformer from scratch, and found that they are comparable to the standard Transformer of the same size while achieving over 55% speedup. Given that such a sim- ple architectural change can obviate the need for expensive interconnects while maintaining model quality, we hope that our method will inspire even closer co-design between model architecture and inference systems.
communica- tion overlap on gpus through kernel fusion, 2024. URL https: / arxiv. org/ abs/ 2lbp://446006000 06858.
Chen, M., Tworek, J.. Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E.,, Herbert- Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J.. Babuschkin, I., Balaji, S., Jain, S., Saun- ders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., Mc- Grew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code, 2021. URL https: / www.aaaivvvvvo..../ org. abs/ 2 0 0307003744
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S. Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson,
References
Ainslie, J., Lee-Thorp, J., de Jong, M.. Zemlyanskiy, Y.,
Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head check- points, 2023. URL https:/ //a aarxiv. org/ /bs/ 2305. 13245.
K., Fedus, L.,, Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov. A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S.,
9

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Pellat, M., Lewkowyez, A., Moreira, E., Child, R., Polo- Zov, O., Lee, K., Zhou, Z.,, Wang. X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstem,, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling language modeling with pathways, 2022. URI. https: / /atxiviivvv./////a///e.//aaaavv. httpaatwwi.aaaaaaaaa///tp/ptoott abs///bs///// 2204 02311.
R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabas- appa. S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang, S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot. S., Gururangan, S., Borodinsky, S., Her- man, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov. T., Xiao, T., Karn, U.. Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, Z. D., Yan, Z., Chen, Z., Papakipos, Z.., Singh, A., Grattafiori, A., Jain, A.,, Kelsey, A.. Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabricl, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C.,, Burton, C., Mejia, C., Wang. C.,, Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C.,, Feichtenhofer, C., Civin, D., Beaty, D.,. Kreymer, D., Li, D.. Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D.,, Wang. D., Le. D.,, Holland, D.,, Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute E. Dunbar, E., Smothers, E., Sun, F. Kreuk, F., Tian, F., Ozgenel, F., Caggioni. F., Guzmán, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern G., Thattai, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Shojanazeri, H. Zou, H., Wang. H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Damlaj, I., Molybog, I. Tufanov, I., Veliche, I.-E., Gat, I., Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J.. Zhen, J., Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang, J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J., Wang. J., Wu, K., U, K. H. Saxena, K., Prasad, K., Khandelwal, K., Zand, K., Matosich, K., Veeraragha- van, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, I., Garg, I.,, A, Silva, L., Bell, L., Zhang, L., Guo, L., Yu, L., Moshkovich, L., Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Tsimpoukelli, M.. Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Ke- neally, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Maccy, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari,
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: /ttp:///aavvvv/iaa org/ abs/ 1803. 05457.
Cobbe, K. Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems, 2021. URL
hts/abs/2111 14168.
Coleman, C. Cuda graphs: Flexible, efficient parallelism in cuda, 2020. URL htttt:://d//steppstt nvidia.
com/blog Baaloo/cuda-ggaappi
Accessed: 2024-09-29.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B.. Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C. Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D.. Garcia-Olano, D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E.,, Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F,, Zhang, F., Syn- naeve, G., Lee, G., Anderson, G. I.., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H., Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I. Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J.. Wang, J., Yu, J., Bitton, J.. Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K. Iyer, K., Malik, K., Chiu, K., Bhalla, K., Rantala- Yeary, L., van der Maaten, L., Chen, L., Tan, L., Jenkins, L., Martin, L.,, Madaan, I.., Malo, L.,, Blecher, L.., Landzaat, L.. de Oliveira, L., Muzzi, M., Pasupuleti, M., Singh, M., Paluri. M., Kardas, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan M., Goyal, N., Torabi, N., Bashlykov, N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R.,
Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel,
10

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P.,
HuggingFaceFF. fineweb-edu (revision 22b0aca), 2024.
URL https: ///ggg/ ng acce. co/datasetss HuggingFaceFW/ wttpeee//ww/eeeeebeeeeeeeeeee fineweb- edu.
Jangda, A., Huang, J., Liu, G., Sabet, A. H. N., Maleki, S.,
Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj, P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li. R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu, Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay. S., Feng, S., Lin. S., Zha, S. C., Shankar, S., Zhang, S., Zhang. S., Wang. S., Agarwal, S., Sajuyigbe, S., Chintala, S., Max, S., Chen, S., Kehoe, S., Satterfield, S., Govindaprasad, S., Gupta, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Albiero, V., Ionescu, V., Poe- naru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang. W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang. X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y.. Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., Vito,o, Z., Rosnbrick, Z., Wen, Z., Yang. Z., and Zhao, Z. The llama 3 herd of models, 2024. URL. https: /arxivvv.vv. org/ abs 2407 21783.
Miao, Y., Musuvathi, M., Mytkowicz, T., and Sarikivi, O. Breaking the computation and communication abstraction barrier in distributed machine learning workloads, 2022. URL https: / arxiv, ierivivorgee org/ /asg///b///a 2tabs/2105 2105. 05720.
Kalamkar, D., Mudigere, D., Mellempudi, N., Das, D.,
Banerjee, K., Avancha, S., Vooturi, D. T., Jammala- madaka, N., Huang, J., Yuen, H., et al. A study of bfloat16 for deep learning training. arXiv preprint 12322, 2019.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are mns: Fast autoregressive transformers with linear attention, 2020. URL https: //rxivv /rg/abs//22 2000 16236.
Lamy-Poirie J. Breadth- pipeline parallelism, 2023. URL
/2111 05953.
Li, F., Zhao, S., Qing, Y., Chen, X., Guan, X., Wang, S., Zhang, G., and Cui, H. Fold3d: Rethinking and paral- lelizing computational and communicational tasks in the training of large dnn models. IEEE Transactions on Par- allel and Distributed Systems, 34(5): 1449, 2023a. doi:
Dubois, Y. Galambosi, B., Liang. P., and Hashimoto, B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint urXiv:2404.04477 2024.
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.0616/ 2023b.
FairScale authors. Fairscale: A general purpose modular pytorch library for high performance and large scale training. https: https://githhut /tgpttg.t.t//ggg/gge/gggg...e..t com/ Baaebookreseerrc/ffairrre
Li, S., Zhao Y., Varma, R., Salpekar, O., Noordhuis, P., Li, T., Paszke, A., Smith, J., Vaughan, B., Damania, P., et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv: 15704, 2020.
2021.
Frostig, R., Johnson, M. J., and Leary, C. Compiling ma- chine learning programs via ligh-level tracing. Systems for Machine Learning, 4(9), 2018.
Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-follovinn models. https: /ggthtu.t.u...gg a alpaca_ eval, 5 2023c.
Gao, L., Tow, Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff. N., Ociepa, C., Phang, J., Reynolds, I.., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL. https: //tp://///oddooo// org/records/ 12608602.
Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 3214 3252, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10.18653//1/2022..acll 229. URL https: pectps:/acclae http:///aaaaaaalca wrg. 2022.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021. URL https: //arxiv htsg//bb/2000 03300.
acl-long. .229.

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Liu, J., Xia, C. Wang, Y., and Zhang, L. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty- seventh Conference on Neural Information Processing Systems, 2023a. URL https: /ttp:///ppeeeeeeeeeeeeeee net./ forum?id=1qvz61
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model par- allelism, 2020. URL https: htap:::/arxxv.orggp.. /bbs/ 1909. 08053.
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., and Chen, B. Deja Contextual sparsity for efficient Ilms at inference time, 2023b. URL https: www.iv.xvvv.../.../..a abs/2310. 17157.
Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhan- dari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas,
Lozhkov, A., Ben Allal, L., von Werra, L., and Wolf, T. Fineweb-edu, May 2024. URL
G., Korthikanti, V., Zhang, E., Child, R., Aminabadi, R. Y., Bernauer, J., Song, X., Shocybi, M., He, Y., Hous- ton, M., Tiwary, S., and Catanzaro, B. Using deep- speed and megatron to train megatron turrionnttuinngggg nlg 530b, large-scale generative language model, 2022. URL 11990.
https: http:///tggggggc.cco///ooo http://nggaac..o//aaaaaeeaaaeeee co/datasets/ BhoginngfaceF///Eneeeeeee
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https:// openreview net fortt//orum?aa Bd=ridd=Bjjooe 72udxe.
Trinh, T. H. and Le, Q. simple method for common- sense reasoning. arXiv preprint arXiv:1806.0255 2018.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., and Wu, H. Mixed precision training. In International Conference on Learning Representations, forum?
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
2018. URL httpp::t/tpttereeeeeeeee http///pperrreeeeeeeeeeeee id-r1gs9JRRZ.
Wang, B. and Komatsuzaki, A. Gpt-j-bb: A 6 billion param- eter autoregressive language model, 2021.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering, 2018. Utthttps:// arxiv. org/ abs / 809. 02789.
Wang. J., Paliotta, D., May, A., Rush, A. M., and Dao, T. The mamba in the Ilama: Distilling and accelerating hybrid models, 2024. URL abs 15.377.......
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V. A., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., Phanishayee, A., and Zaharia, M. Efficient large-scale language model train- ing on gpu clusters using megatron-Im, 2021. URL https://arxiv. wtrpaiiio./.aaaoaaaooooooaoooloe httpxivv.rrg/bb//1//000 /tw.////.... 04473.
Wang. S., Wei, J., Sabne, A., Davis, A., Ilbeyi, B., Hecht- man, B., Chen, D., Murthy, K. S., Maggioni, M., Zhang, Q., Kumar, S., Guo, T., Xu, Y., and Zhou, Z. Over- lap communication with dependent computation via de- composition in large deep learning models. In Pro- ceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, ASPLOS 2023, pp. 93-106, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450399159, doi: 10. 01553367755533567559 URL 10. 10.1145/356798 3567959.
NVIDIA. Transformerengine, 2023. URL https://
github. httpa///////////////aaffooeeeeee
NVIDIA Corporaton Nvlink, 2024. httpst //ww.
nvidia.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Welbi, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derczynski, L., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/1//W17-4413.
Py Torch Labs. fast 2024. URL https: //githb// com/ htm/pytorcch.. labs gpt- fast. Accessed: 2024- 09-29.
URL htt.hhs:/acclentholloseeooo www.l/gy..rgg/11-4///
12

---

Ladder-Residual: Parallelism-Aware Architecture for Accelerating Large Model Inference with Communication Overlapping
Protasov, V., Mikhailov, V., Pruksachatkun, Y., Belinkov, Workshop, B., Scao, T. L., Fan, A., Akiki, C., Pavlick,
E., Ilié, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., Biderman, S., Webson, A., Ammanamanchi, P. S., Wang, T., Sagot, B., Muennighoff, N., del Moral, A. V., Ruwase, O., Baw- den, R., Bekman, S., McMillan-Major, A., Beltagy, I., Nguyen, H., Saulnier, L., Tan, S., Suarez, P. O., Sanh, V., Laurençon, H. Jernite, Y. Launay, J., Mitchell, M., Raffel, C., Gokaslan, A., Simhi, A., Soroa, A., Aji, A. F., Alfassy, A., Rogers, A., Nitzav, A. K., Xu, C., Mou, C., Emezue, C.,, Klamm, C.,, Leong, C., van Strien, D., Adelani, D. I. Radev, D., Ponferrada, E. G., Lev- kovizh, E... Kim, E., Natan, E. B., Toni, F. D., Dupont, G., Kruszewski, G., Pistilli, G., Elsahar, H., Benyam- ina, H., Tran, H., Yu, I., Abdulmumin, I.. Johnson, I., Gonzalez-Dios, I., de Rosa, J., Chim, J., Dodge, J., Zhu, J., Chang. J., Frohberg, J., Tobing, J., Bhattacharjee, J. Almubarak, K., Chen, K., Lo, K., Werra, L. V., Weber, L., Phan, L., allal, L. B., Tanguy, L., Dey, M., Muñoz, M. R., Masoud, M., Grandury, M., Saško, M., Huang, M., Coavoux, M., Singh, M., Jiang. M. T.-J., Vu, M. C., Jauhar, M. A., Ghaleb, M., Subramani, N., Kassner, N., Khamis, N., Nguyen, O., Espejel, O., de Gibert, O., Ville- gas, P., Henderson, P. Colombo, P., Amuok, P., Lhoest, Q. Harliman, R., Bommasani, R., López, R. L.,, Ribeiro, R., Osei, S., Pyysalo, S., Nagel, S., Bose, S., Muhammad, S. H., Sharma, S., Longpre, S., Nikpoor, S., Silberberg, S., Pai, S., Zink, S., Torrent, T. T., Schick, T., Thrush, T., Danchev, V., Nikoulina, V., Laippala, V., Lepercq, V., Prabhu, V., Alyafcai, Z., Talat, Z., Raja, A., Heinzer- ling, B., Si, C., Taşar, D. F., Salesky, E., Mielke, S. J., Lee, W. Y. Sharma, A., Santilli, A., Chaffin, A., Stiegler, A., Datta, D., Szczechla, E., Chhablani. G., Wang, H., Pandey, H., Strobelt,, H., Fries, J. A., Rozen, J., Gao, I.., Sutawika, L., Bari, M. S., Al-shaibani, M. S., Manica, M., Nayak, N., Teehan, R., Albanie, S., Shen, S., Ben-David, S., Bach, S. H., Kim, T., Bers, T., Fevry, T., Neeraj, T., Thakker, U., Raunak, V., Tang, X., Yong, Z.-X., Sun, Z., Brody, S., Uri, Y., Tojarieh, H., Roberts, A., Chung, H. W., Tae, J., Phang, J., Press, O., Li, C., Narayanan, D., Bourfoune, H., Casper. J., Rasley, J., Ryabinin, M., Mishra, M., Zhang, M., Shoeybi, M., Peyrounette, M., Patry, N., Tazi, N., Sanseviero, O., von Platen, P., Cor- nette, P., Lavallée, P. F., Lacroix, R., Rajbhandari, S., Gandhi, S. Smith, S., Requena, S., Patil, S., Dettmers, T., Baruwa, A., Singh, A., Cheveleva, A.. Ligozat, A.LL., Subramonian, A., Névéol, A., Lovering, C., Garrette, D., Tunuguntla, D., Reiter, E., Taktasheva, E., Voloshina, E., Bogdanov, E., Winata, G. I.. Schoelkopf, H., Kalo, J.-C., Novikova, J., Forde, J. Z., Clive, J., Kasai, J., Kawamura, K. Hazan, L., Carpuat, M., Clinciu, M., Kim, N., Cheng, N., Serikov, O., Antverg, O., van der Wal, O., Zhang, R., Zhang, R., Gehrmann, S., Mirkin, S., Pais, S., Shav- rina, T., Scialom, T., Yun, T., Limisiewicz, T., Rieser, V.,
Y., Bamberger, Z., Kasner, Z., Rueda, A., Pestana, A., Feizpour, A., Khan, A., Faranak, A., Santos, A., Hevia, A., Unldreaj. A., Aghagol, A., Abdollahi, A., Tammour, A., HajiHosseini, A., Behroozi, B., Ajihade, B., Saxena, B., Ferrandis, C. M., McDuff, D., Contractor, D., Lansky, D., David, D., Kiela, D., Nguyen, D. A., Tan, E., Bay- lor, E., Ozoani, E., Mirza. F., Ononiwu, F., Rezanejad. H., Jones, H., Bhattacharya, I., Solaiman, I., Sedenko, I., Nejadgholi, I., Passmore, J., Seltzer, J., Sanz, J. B., Dutra, I.., Samagaio, M., Elbadri, M., Mieskes, M., Gerchick, M., Akinlolu, M., McKenna, M., Qiu, M., Ghauri, M., Burynok, M., Abrar, N., Rajani, N., Elkott, N., Fahmy, N., Samuel, O., An, R., Kromann, R., Hao, R., Alizadeh, S., Shubber, S., Wang, S., Roy, S., Viguier, S., Le, T., Oyebade, T., Le, T., Yang, Y., Nguyen, Z., Kashyap. A. R., Palasciano, A., Callahan, A., Shukla, A., Miranda- Escalada, A., Singh, A., Beilharz, B., Wang, B., Brito, C., Zhou, C.. Jain, C., Xu, C., Fourrier, C., Periñán, D. L., Molano, D., Yu, D., Manjavacas, E., Barth, F., Fuhrimann, F., Altay, G., Bayrak, G. Burns, G., Vrabee, H. U., Bello, I., Dash, I., Kang, J., Giorgi. J., Golde, J., Posada, J. D., Sivaraman, K. R., Bulchandani, L.,, Liu, L., Shinzato, L., de Bykhovetz. M. H., Takeuchi, M., Pàmies, M., Castillo, M. A., Nezhurina, M., Sänger, M., Samwald, M., Cullan, M. Weinberg, M., Wolf, M. D., Mihaljeic, M., Liu, M., Freidank, M., Kang, M., Seelam, N., Dahlberg, N., Broad, N. M., Muellnnr, N., Fung. P., Haller, P., Chandrasekhar, R., Eisenberg, R., Martin, R., Canalli, R., Su, R., Su, R., Cahyawijaya, S., Garda, S., Deshmukh, S. S., Mishra, S., Kiblawi, S., Ott, S., Sang-aroonsirg S., Kumar, S., Schweter, S., Bharati, S., Laud, T., Gigant, T., Kainuma, T., Kusa, W., Labrak, Y., Bajaj. Y. S., Venkatraman, Y., Xu, Y., Xu, Y., Xu, Y., Tan, Z., Xie, 7.,, Ye, 7., Bras, M.. Belkada, Y., and Wolf, T. Bloom: A 776b-parameter open-access multilingual language model, 2023. URL https: wtt.aawwwwwwr.a///a....1112 05100....
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. URL https: http//awwwvvggg.aa..aa//.a
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C,, Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. Pytorch fsdp: experiences scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. instruction-follwinng evaluation for large language models, 2023.
13

---

Ladder-Residual: Parallelism-Awar Architecture for Accelerating Large Model Inference with Communication Overlapping
domer peet
Figure Traces generated by PyTorch Profiler. shown the plot for Standard transformer the NCCL operations block the com- putation whereas Ladder Transfomer the NCCL operations can be overlapped with the computation.
A. Appendix
A.1. PyTorch Profiler Trace
Here we provide the trace generated by the PyTorch Profiler* in Figure 5.
htttss///pptarcchooog/uuuoottttt recippppseeeeee//ee/eeeee recipes/profilerrreeii... html
14


Reviews:
[{"review_id": "3m8gg36oSw", "reviewer": "Reviewer_Mhub", "summary": "This paper proposes Ladder Residual, an alternative to the transformer architecture that breaks the communication-computation dependency in conventional parallelism patterns, in order to accelerate the inference, at the cost of accuracy degradation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and the feedback! We want to clarify that in a lossy efficient method (where a more efficient architecture is proposed to approximate the original one), trading accuracy for efficiency is common and we provide a good trade-off. It’s difficult to have one-size-fits-all as some users might prefer higher efficiency while others prefer higher accuracy.\n\nHowever, to make things more clear, we provide additional results that achieve **both higher accuracy and higher throughput (token-per-seconds)** below. This was accomplished by training a slightly larger ladder model for the train-from-scratch experiment and employing an improved post-training pipeline for the post-training-adaptation experiment.\n\n\n> For Table 3, Ladder-Transformer has a gap of 1 to the Standard-Transformer.\n\nIn our train-from-scratch experiments, we previously reported that the Ladder-Transformer slightly underperformed compared to the Standard-Transformer of the same size. As shown in Table 1 from the paper, Ladder-residual can offer at least 30% speedup for any model size. To further explore the accuracy-throughput trade-off, we increased the parameter count of the Ladder-Transformer by 30%. This allows us to answer the question: _For two models with similar throughput, how does accuracy compare?_\n\nImportantly, increasing the Ladder-Transformer’s size by 30% leads to less than a 30% decrease in throughput. As a result, the scaled-up Ladder models still achieve higher tokens-per-second (TPS) than their smaller Standard counterparts. We report both accuracy and TPS below following the format of Table 3 in the paper. (Note: the “1B” and “3B” models in the paper correspond to 1.2B and 3.5B parameters respectively; here, we write out the exact values for clarity.)\n\n\n| Model | ARC-C | ARC-E | HellaSwag| PIQA | SciQ | WinoGram | Average | Wikitext PPL | TPS |\n| ----------- | ----------- | ----------- | ----------- |  ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| Standard-Transformer-1.2B | 34.22 | 70.33 | 41.10 | 71.49 | 87.30 | 55.41 | 59.98 | 18.54 | 1008.29 |\n| Ladder-Transformer-1.55B | 33.96 | 70.16 | 42.58 | 71.98 | 87.90 | 55.41 | 60.33 | 17.47 | 1277.66 |\n| Standard-Transformer-3.5B | 38.99 | 74.12 | 46.48 | 74.59 | 92.00 | 58.48 | 64.11 | 14.48 | 949.6 |\n| Ladder-Transformer-4.5B | 40.96 | 75.00 | 46.81 | 73.99 | 90.80 | 57.70 | 64.21 | 14.05 | 1217.71 |\n\n**As shown above, the Ladder-Transformer achieves higher average accuracy, lower perplexity, and significantly higher TPS at both the 1.2B and 3.5B sizes.**\n\n\n>  For Table 4, although the average score of Hybrid-Ladder-8B-16L-retrained is comparable to Llama-3.1-8B-Instruct, in most of the evaluation categories (6 out of 9) the hybrid ladder is worse than the baseline.\n\nWe retrain the hybrid-ladder-8B-16L model with the same data (3B tokens, all open-source datasets), but this time with logit distillation (KL divergence loss with the logits from the original Llama-3.1-8B-Instruct). Following the format of Table 3 in our paper:\n\n| Model | MMLU | ARC-C | OBQA | HS | TQ | GSM | HE+ | IE | AE | Average |\n| ----------- | ----------- | ----------- | ----------- |  ----------- | ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n| Llama-3.1-8B-Instruct | 68.14 | 60.32 | 43.00 | 80.04 | 36.84 | 84.99 | 60.40 | 52.57 | 18.69 | 56.11 |\n| Hybrid-Ladder-8B-16L-retrained | 67.33 | 59.98 | 79.05 | 45.00 | 37.58 | 86.81 | 60.51 | 59.76 | 22.43 | 57.61 |\n\nNow we are better on 5 out of 9 tasks while being 1.5 points higher for the average score, and all tasks we are worse at are within a very small margin (< 1 point). This confidently demonstrates Hybrid-Ladder-8B-16L-retrained can be a drop-in replacement to Llama-3.1-8B-Instruct, achieving 23% speedup with no accuracy lost.\n\nWe also thank the reviewer for the feedback on clarity and the formatting issue, we will incorporate these in the next version.", "claims_and_evidence": "I have some doubts about the claim \"We also show that it is possible to convert parts of the Llama-3.1 8B model to our Ladder Residual architecture with minimal accuracy degradation by only retraining for 3B tokens\"\n\nAccording to the experiment results in Table 3 and 4, it seems that the proposed architecture has significant degradation in accuracy compared to the baseline. For Table 3, Ladder-Transformer has a gap of 1 to the Standard-Transformer. For Table 4, although the average score of Hybrid-Ladder-8B-16L-retrained is comparable to Llama-3.1-8B-Instruct, in most of the evaluation categories (6 out of 9) the hybrid ladder is worse than the baseline. Thus, it is difficult to justify that such degradation of accuracy is acceptable.", "methods_and_evaluation_criteria": "The method makes sense. The evaluation could be more comprehensive, such as adding more models (there are a lot of open-sourced pretrained models could be used for post-training adaptation)", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "For the experiments, I hardly find it convincing that the evaluation could justify the degradation of accuracy of the proposed architecture. May be adding more experiments (on different models, etc.) could make the results more convincing.", "relation_to_broader_scientific_literature": "There is nothing related to the broader scientific literature.", "essential_references_not_discussed": "The references look good to me.", "other_strengths_and_weaknesses": "Minor issue:\n\n1. The abbreviation \"AAR\" in Algorithm 1 is not defined or explained. I guess AAR means AsyncAllReduce, but the paper shouldn't let the readers to guess what an abbreviation means. Please use \"... Note that the AsyncAllReduce (AAR) returns a handle ...\" in the algorithm caption, so that it could be more friendly to the readers.\n\n2. For some unknown reason, there is a \"‘\" in page 1 on the left-hand side of \"Abstract\" (check the 1st column, between line 010-011). Please double check the latex source file.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Is it possible to replace the transformer block with ladder transformer in ViT? If so, is there any corresponding experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZjtlVXhJ8b", "reviewer": "Reviewer_9rvM", "summary": "This paper proposes Ladder-Residual, which modifies the residual connection in Transformers such that the i-th block reads from the (i−2)-th block's result instead of the (i−1)-th. This interleaved schedule allows direct overlap between the Transformer block's computation and the subsequent all-reduce communication step, thus significantly speeding up tensor parallelism across multiple devices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback and the questions. Below we discuss how our paper is novel, why it’s compatible with other parallelism, address the concern on scaling, and provide analysis on the change of activations.\n\nWe also thank the reviewer for the suggestions on the presentation. We will incorporate them in the next draft.\n_________________\n**Novelty over parallelizing MHA and MLP blocks**\n\nFirst, Ladder-residual is a fundamentally different architecture from parallelizing MHA and MLP blocks, with distinct motivations. Ladder-residual is designed to decouple computation and communication to enable their overlap. In contrast, the goal of parallelizing MHA and MLP blocks is to combine computation for speedup. Due to the reduced depth, parallelizing MHA and MLP blocks also saves communication but is not its original focus. The speedup of Ladder-Residual comes from overlapping all (except last) communication where parallelizing MHA and MLP focuses on accelerated matmul and as a byproduct, it cuts 50% of the communication. \n_________________\n**Combining with other communication optimization methods**\n\nYes. A key advantage of Ladder-Residual is that it is hardware-agnostic and does not rely on custom kernels, making it fully compatible with other low-level optimization methods. However, since Ladder-residual hides latency of all communication except the last one, further applying these techniques can only accelerate the final communication which won’t achieve much gains.\n_________________\n**Comparison with other frameworks**\n\nWe focus on inference speedups in this paper and gpt-fast and vLLM are highly optimized for inference. We choose gpt-fast to demonstrate the inference speedups but it should be noted that we still achieve training speedups (around 5-8%) when combining TP+FSDP.\n_________________\n**Compatibility with other parallelism**\n\nIt is possible to use Pipeline Parallel (PP) with the Ladder architecture: just before the pipeline boundary, we wait for the async AllReduces to complete, and send 3 tensors to the next pipeline stage: residual, current_mlp_output tensor and current_attention_output. It should be noted that this is still pretty cheap since generally the P2P communication during inference is latency bound and can be implemented easily using the batch_isend_irecv API (https://pytorch.org/docs/stable/distributed.html#torch.distributed.batch_isend_irecv).\n```\ndef forward(\n   self,\n   previous_attention_out: Tensor,\n   previous_mlp_out: Tensor,\n   residual: Tensor,\n   attention_handle,\n   mlp_handle,\n) -> Tensor:\n   attention_handle.wait()\n   residual = residual + previous_attention_out\n\n   current_attention_out = self.attention(self.attention_norm(residual))\n   current_attention_out = all_reduce(current_attention_out, async_op=True)\n\n   mlp_handle.wait()\n   residual = residual + previous_mlp_out\n\n   current_mlp_out = self.feed_forward(self.ffn_norm(residual))\n   current_mlp_out, mlp_handle = all_reduce(current_mlp_out, async_op=True)\n\n   if is_last_layer_on_pp_stage:\n       attention_handle.wait()\n       mlp_handle.wait()\n\n   return current_attention_out, current_mlp_out, residual, attention_handle, mlp_handle\n```\nFor Data Parallelism or FSDP, the cross-layer dependency doesn’t pose any complications. Therefore Ladder-residual can be seamlessly incorporated in a multi-dimensional parallelism training paradigm. Our training framework already supports TP+FSDP for ladder residual model training and can be easily extended to support PP as illustrated by the above code snippet.\n_________________\n **Performance and scaling**\n\nIn our response to Reviewer Mhub, we provide experiments where we increase the number of parameters of the Ladder-Transformer by 30%. We show that at both 1B and 3B scale, the 30% larger ladder-Transformer has both higher accuracy and higher TPS than the standard-Transformer. This confirms our architecture truly pushes the Pareto frontier.\n\nIt’s difficult to conclude the scaling trend from just two sizes and in the future we hope to run models of more sizes to study this more carefully.\n_________________\n**Activation difference analysis**\n\nOur motivation is that, given in a trained standard Transformer, the activation differences between modules are small, replacing with the Ladder architecture won’t lead to large degradation. We don’t care too much if this property is still held during training, and it’s difficult to have a theoretical guarantee due to unpredictable training dynamics.\n\nTo further verify this intuition, we analyze block similarity (between consecutive attention and MLP modules) in both original Llama-3.1-8B-Instruct and Hybrid-Ladder-8B-16L-retrained. The visualization can be found at [This Anonymous GitHub link](https://anonymous.4open.science/r/ICML25_rebuttal-F932/README.md). Overall there is a slight decrease in activation similarity after adapting to Ladder-residual but the similarity remains high overall (above 0.9 at most of the layers).", "claims_and_evidence": "I think the Ladder-Residual approach is conceptually straightforward and effective, enabling communication-computation overlap without altering underlying systems or kernels. However, a few points need additional clarification:\n- \"Liu et al. (2023b) found that activation changes slowly in Transformer, as the norm of each update $h_{i+1}(x_i)$ is small compared to the residual.\" It conducts the experiments under an inference setting. Is this still true in the training? Do you have any theoretical guarantees that this property still holds during training, i.e., whether activation differences remain bounded across consecutive blocks.\n- Deja Vu's Appendix C.3 already discusses the possibility of parallelizing MHA and MLP blocks. Can you explain the novelty of Ladder-Residual over it?\n- Can Ladder Residual be combined with existing communication optimization methods, such as [A] and [B] that decompose the computation into small chunks to enable better compute-communication overlapping? You mentioned that your work does not rely on decomposition, but can such chunking approaches be combined with Ladder-Residual to further boost performance?", "methods_and_evaluation_criteria": "- It would be valuable to demonstrate Ladder-Residual using widely adopted frameworks such as Megatron-LM or DeepSpeed. For instance, instead of purely self-comparison in Table 2, showing improvements relative to standard baselines in these frameworks would help the community judge the method's portability and real-world impact.", "theoretical_claims": "- Is there a formal proof or theoretical justification indicating that altering the residual connection in this way preserves the Transformer’s representational power or performance?", "experimental_designs_or_analyses": "- While Ladder-Residual aids tensor parallelism, cross-layer dependencies can complicate pipeline parallelism. Section 3.3.2 focuses on multi-node experiments with TP=16, yet it remains unclear how Ladder-Residual would perform compared to standard 3D parallelism (e.g., Megatron-LM) or ZeRO-3 (DeepSpeed). Clarification of its interplay with pipeline parallelism is necessary.\n- Results on 3B models appear worse than those on 1B models, raising questions about scalability. Can you provide additional experiments or explanations for how Ladder-Residual scales with model size?", "relation_to_broader_scientific_literature": "The proposed approach is simple but potentially impactful, requiring no special modifications to kernels or communication libraries. This simplicity may facilitate broad adoption across diverse hardware environments.", "essential_references_not_discussed": "* [A] Wang, Shibo, et al. \"Overlap communication with dependent computation via decomposition in large deep learning models.\" Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1. 2022. (Already in the reference)\n* [B] Chen, Chang, et al. \"Centauri: Enabling efficient scheduling for communication-computation overlap in large model training via communication partitioning.\" Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 2024.", "other_strengths_and_weaknesses": "See the above sections.", "other_comments_or_suggestions": "See the above sections.", "questions_for_authors": "See the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3FckL9r7gz", "reviewer": "Reviewer_Wjzo", "summary": "This paper proposes a communication-friendly Transformer layout, Ladder Residual, to accelerate tensor-parallel training. Ladder Residual  enables straightforward computation-communication overlapping compared with vanilla sequential or parallel layout. It achieves 29% end-to-end speedup in TP8 training. Ladder Residual outperforms Parallel Transformer and shows a narrow gap than standard sequential Transformers. As a candidate of hardware-friendly layout, Ladder Residual has potential on replacing standard Transformer in LLM pre-training.\n\n## Update after rebuttal\nMy score keeps as weakly accept. I think a TP-friendly architecture is an interesting idea and the proposed method actually makes accelerations. However, the biggest disagreement between my and authors is the actual usefulness of Ladder Residual. Since the architecture's performance is not on par with baseline, and the large-TP training is not often used in actual LLM training, Ladder Residual is not a general-promoted architecture.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback on our paper! Below we provide a few clarifications on how our experiments demonstrate practical gain from ladder-residual and its compatibility with other parallelism.\n\n> In the paper, some experiments are conducted under TP16, where inter-node communication cost is stronger on 8-gpu per node clusters.\n\nWe agree with the reviewer that tensor parallelism is not commonly conducted across nodes due to the heavy communication. Our TP16 experiment is intended to demonstrate the potential of Ladder-Residual in scenarios where cross-node model parallelism becomes necessary. Generally, larger models are served with TP+PP (for example DeepSeek 671B model). However, it should be noted that with newer generation GPUs (B200 NVL72) and the recently announced (Rubin NVL576), it will become very feasible to run these models with purely TP since these systems expand the NVLink (high bandwidth) domain from 8 GPUs to 72 and 576 GPUs respectively.\n\nWe also want to note that Ladder-residual is compatible with multi-dimensional parallelism, please see our response to Reviewer 9rvM for further details. Therefore, even if we want to combine TP with other parallelism, ladder-residual can still be applied to accelerate the TP part. Furthermore, despite we didn't explore it in this work, we believe ladder-residual can accelerate other heavy-communication parallelism like EP as well.\n\n> It's possible that the advantage of this paper only happens in large TP settings, which may be not often used.\n\nTable 1 and Figure 2 investigates the benefit of ladder-residual when intra-node TP (2, 4 and 8 GPUs) is used, which is the most common and flexible inference-time parallelism approach. In this regime, ladder-residual consistently show a large speedup against the standard Transformer and beats the parallel attn-mlp alternative until the batch size is very large. Figure 4 also shows that under the intra-node TP setting, Ladder-residual can reliably push the Pareto-frontier between latency and throughput.", "claims_and_evidence": "There are two main claims in this paper: end-to-end latency and overall performance. These two experiments are conducted under the same environment and hyper-parameters. The evidence is solid.", "methods_and_evaluation_criteria": "The proposed method is simple and direct, focusing on the macro layout of Transformer models. The evaluation criteria is widely-accepted in the community.", "theoretical_claims": "No Theoretical claims.", "experimental_designs_or_analyses": "The experimental designs are sound to me. However, I have concerns on the experiment settings. Since large TP brings heavy communication latency, the proposed method will show a stronger advantages. In the paper, some experiments are conducted under TP16, where inter-node communication cost is stronger on 8-gpu per node clusters. \n\nLarge TP size is not commonly used due to the heavy communication latency. For example, in Deepseek V3, EP and PP is activated instead of large TP to avoid heavy communication. (Of course Deepseek V3 is published within 3 months and should not be considered.) But the other option of LLM pre-training always exists. Admittedly, Ladder Residual should also work under EP setting. I think it's better to conduct more experiment on these more practical settings.", "relation_to_broader_scientific_literature": "The proposed method is an extension for hardware-friendly layout, after standard sequential layout, and parallel layout. Since there are not many closely related works, this work will be a strong candidate among these macro layouts. This work is orthogonal to other efficient architectures, such as Linear Attention, Group Query Attention.", "essential_references_not_discussed": "Since there are not many references on the connection efficiency, the most essential references are all discussed. However, if the authors think it is essential to discuss other strands of efficient architectures, there will be more references. For example, if this paper discusses [1],   a broader discussion of different linear-time models may be essential.\n\n[1] The mamba in the llama: Distilling and accelerating hybrid models, 2024.", "other_strengths_and_weaknesses": "Strength:\n1. It is a very simple and elegant design. I think it will be broadly discussed in the future and a strong candidate for communication-friendly architecture.\n\nWeakness:\n1. It's possible that the advantage of this paper only happens in large TP settings, which may be not often used.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "1. Following my comments on Experimental Designs Or Analyses, is Ladder-Residual useful under different settings? \n\n2. Since the performance is not comparable with sequential layouts, could you evaluate performance with a efficiency-aware metric? For example, the performance under same training time instead of same training tokens.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an architecture modification that is claimed to allow overlapping of communication and computation. The main idea is to decouple the computation of a layer and the residual sum, which enables overlapping one communication step with the next computation stage. The architecture is defined by introducing  $y_i = x_{i-1}$ and $X_i=(x_i,y_i)$, then the activations at depth $i+1$ are given by

\\[
X_{i+1} = H_i(X_i) \quad \text{where } \quad H_i(x_i, y_i) = (x_i + h(y_i), x_i).
\\]

From this formulation, it becomes more obvious how successive computations can be overlapped. I think the connection with TP/AAR (all-reduce) could be slightly toned down — this is more relevant for the ML systems community and may not be the primary focus of ICML.

That said, the paper demonstrates that the method performs well compared to standard Transformers. It smoothly utilizes hardware acceleration both at inference and training time and can be easily incorporated into a standard Transformer. While the method is simple and doesn’t rely on any significant implementation or architectural tricks, it still achieves strong results.

The reviewers find this method to be of interest. Nevertheless, Reviewer Wjzo noted that the authors could have done a better job citing and comparing their method with existing work, and I agree with this assessment. Relevant prior work should be properly incorporated and discussed, a point that was only lightly addressed in the rebuttal. The reviewer also noted a small discrepancy in results, where the proposed method sometimes underperforms, which could be better discussed.

Regarding this last point: given the simplicity of the trick, the fact that no specific hyper-parameter adaptation were used, and the likelihood that current training methods are stuck in local minima of hyper-parameter configurations, I don’t blame the authors. This game of tuning is the one everybody is playing and is probably improductive for research.

Overall, I believe this is a worthwhile contribution to ICML.
Decision: Accept (poster)