SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models
Authors: Han-Byul Kim
Duc N.M Hoang
Arnav Kundu
Mohammad Samragh
Minsik Cho
Venue: ICML 2025 poster (2025)
Keywords: sync point drop
tensor parallelism
distributed inference
model optimization
Primary area: optimization

Abstract: With the rapid expansion in the scale of large language models (LLMs), enabling efficient distributed inference across multiple computing units has become increasingly critical. However, communication overheads from popular distributed inference techniques such as Tensor Parallelism pose a significant challenge to achieve scalability and low latency. Therefore, we introduce a novel optimization technique, Sync-Point Drop (SPD), to reduce communication overheads in tensor parallelism by selectively dropping synchronization on attention outputs. In detail, we first propose a block design that allows execution to proceed without communication through SPD. Second, we apply different SPD strategies to attention blocks based on their sensitivity to the model accuracy. The proposed methods effectively alleviate communication bottlenecks while minimizing accuracy degradation during LLM inference, offering a scalable solution for diverse distributed environments: SPD offered about 20\% overall inference latency reduction with < 1\% accuracy regression for LLaMA2-70B inference over 8 GPUs.

Large Language Models (LLMs) (Gunter al., 2024; Brown et al., 2020: Bubeck et al., 2023; Touvron et al., 2023a;b; Zhang et al., 2022; Penedo et al., 2023; Jiang et al., 2023) have revolutionized the field of natural language pro- cessing (NLP), driving significant advancements in a wide range of applications. Their ability to understand and gen- erate human-like like text has opened new possibilities for both research and practical uses. However, as these models grow in size and complexity, optimizing their performance be- SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models
Han-Byul Kim
Duc Hoang'
Arnav Kundu Mohammad Samragh Minsik Cho
Abstract
comes crucial challenge, particularly in terms of latency.
With the rapid expansion in the scale of large language models (LLMs), enabling efficient dis- tributed inference across multiple computing units has become increasingly critical. However, com- munication overheads from popular distributed inference techniques such as Tensor Parallelism pose significant challenge to achieve scalability and low latency. Therefore, we introduce a novel optimization technique, Sync-Point Drop (SPD), to reduce communication overheads tensor par- allelism by selectively dropping synchronization attention outputs. In detail, first propose block design that allows execution to proceed without communication through SPD. Second, we apply different SPD strategies to attention blocks based on their sensitivity to the model accuracy. The proposed methods effectively alleviate com- munication bottlenecks while minimizing accu- racy degradation during LLM inference, offering scalable solution for diverse distributed environ- ments: SPD offered about 20% overall inference latency reduction with < 1% accuracy regression for LLaMA2-700 inference over 8 GPUs.
A proven approach to achieving low latency is to run LLM inference in distributed computing environments, notably using Tensor Parallelism (TP) (Shoeybi et al., 2019). By sharding tensor operations into separate tracks or blocks and processing them simultaneously on parallel devices, TP significantly reduces inference time while maintaining accuracy.
However, to maintain mathematical parity with single- device inference, TP requires collective communica- tion-often referred to as model. These sync-points serve as communication barriers across all parallel devices synchronize hidden represen- tation tensors, as shown in Figure 1a. Because of this com- municative nature, the overhead of sync-points depends on hardware constraints-such as the interconnects between de- vices and the network connections between nodes-which can become a bottleneck during execution. As LLMs grow in size, one must use more compute devices, which in turn increases the number of sync-points and further worsens inference latency. Therefore, in any distributed system, opti- mizing sync-points would greatly improve overall system performance.
the
To tackle this important issue, we propose Sync-Point Drop (SPD) a simple yet novel optimization technique with broad applications for LLM systems. Unlike existing works which tried to improve the communication process itself (NVIDIA, 2019; Jeaugey, 2019; Cheng et al., 2023) on system-level. SPD directly removes syne-point in the self-attention output (as in Figure 1b) within the target budget. To enable SPD directly on decoder block, we first introduce a block design for SPD that minimizes negative effects resulting from re- duced communication (see Figure 3). Second, we apply SPD strategies differently to each blocks based on commu- nication sensitivity, which we defined as the relative impact downstream performance when all communications dropped up to that point (see Figure 4). Our experimental re- sults show effective possibiliiyy of latency improvement with minimizing the accuracy degradation throughout diverse sizes of models. In summary, our contributions are:
1. Introduction
Apple. Correspondence Han-Byul Kim <han- byul@apple.com
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We propose novel block designs for SPD that minimize

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
LayonHom
LayerNom
Activvation
LayerNom
Salf Attornion
LayerNorm
Actiraaion
Lineer
Linear
Liner
GPU 1
Alkreduuee
Allredica
GPU
Al-redkuze
www co GPU
LayenHorm
Layertiom
Activation
Layerlorm
Salf Altuntion
Layerliom
Actriation
Atternton
Liner
Linear
(a) With fully synchronous lensor parallelism.
(b)
With the elimination of the syne-point in the attention output.
Figure I. Tensor parallelism applicd on transformer decoder block (in 2-GPUs distributed inference case).
information loss from lack of communication.
model. One of the distributed deployment techniques, ten- sor parallelism (Shoeybi et al., 2019), enables fast serving of a model by parallel execution of block into multiple devices. However, this technique requires large commu- nication overheads between devices to keep the numeric precision of execution flow. Considering the communica- tion bottleneck of tensor parallelism, existing works also focus on improving the communication operation itself sys- tematically, including ring-topology all-reduce (NVIDIA, 2019) and tree-topolggy all-reduce (Jeaugey, 2019). Specif- ically for large models, ATP (Cheng et al., 2023) improves training efficiency by dynamically choosing the parallel strategy.
We identify the sensitivity of each block within the model and classify them into three distinct categories, allowing for the application of tailored optimization strategies to each group based on their characteristics.
Empirical results on various datasets and models show that proposed SPD with optimization strategies can offer better accuracy/latency trade-off by enabling a scalable solution for distributed environments and mini- mizing quality loss for overall communication budgets.
2. Related Works
In this paper, we leverage optimization benefits in model- level from the system perspective (enabling SPD in the sys- tem while minimizing accuracy degradation in the model).
The inefficiency of LLMs, which emerged with significant impact on computation and memory, has led to a large de- mand for optimization techniques. Model-level optimization has gained attention for opportunities within the vast redun- dancies of LLMs. Quantization (Frantar et al., 2023; Xiao al., 2023; Lin et al., 2024; Shao et al., 2024; Chee al., 2023; Ashkboos et al., 2024) reduces the precision of model parameters, allowing for faster operations with minimal im- pact on performance. In particular, (Agrawal et al., 2024; Dong et al., 2024) are intended to optimize communication performance with low-bit expressions. Pruning (Frantar & Alistarh, 2023: Sun al., 2024; Liu et al., 2023; Xia et al., 2024) eliminates less critical parameters or neurons from the model, thereby reducing its size and computational com- plexity. In the aggressive scale, block skipping (Xia et al., 2024: Song et al., 2024), which involves bypassing certain blocks, further enhances efficiency by eliminating the oper- ations of a block. These approaches focus on compressive and computational effects which makes the model suitable for real-time and resource-constraaiee environments.
3. Preliminary: Tensor Parallelism in LLMs
Tensor Parallelism (TP) (Shoeybi et al., 2019) is a system- atic computing technique on distributed environment used to accelerate large-scale language models. This is real- ized by partitioning individual weight tensors of model across multiple devices. Instead of replicating the entire model across GPUs (as in data parallelism), TP divides each block's ' computation across multiple devices (as in Fig- ure la), enabling the model to handle larger tensors that would otherwise exceed the memory capacity of a single GPU. This approach significantly improves the scalability and efficiency in both training and inference, particularly in LLMs. However, the realization of effective TP requires collective communication (all-reduce in Figure 1) between devices to synchronize and exchange partial computations. The communication latency is typically decided by network bandwidth between devices. The lower the bandwidth, the more the parallel system gets bottleneck originated from the syne-point. To resolve the bottleneck of distributed in- ference, our proposed optimization technique, SPD, simply eliminates the communication within each decoder block (as in Figure lb).
Following the underlying mechanism of model deployment, beyond model-level optimization, system-level optimiza- tions (Shoeybi et al., 2019; Huang et al., 2019; Zhao et al., 2023; Aminabadi et al., 2022; Kwon et al., 2023) are ex- plored. Different from model-level approach, system-level optimization does not change any numerical values of
2

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
560
200
190
158
8.0%
25.0%
50.0%
70.0%
108.0%
50%
25.5
58.0%
73.3%
103.8%
00%
25.h:
55.0%
78.5%
100.0%
035
25.8%
50.2%
70.0
100.5
Percerageed
Meok cumber blockka
Pacerlage block lotdl number Meddo
Perordage SPD black number Mocce
Percerage of number
(a) 1-nodex8 G-UUs, HBW
(b) I-node 8-GP LBW
(c) 2- XPss, HBW
(d) 2-nodex4-GPUs, LBW
Figure 2. Data transfer latency of LLaMA2- 70B distributed inference with SPD on different system settings of NVIDIA A A100-80G GPU node. 'HBW" represents high bandwidth setting and 'LBW" represents low bandwidth setting for device interconnect. Input consist of batch size of and sequence length of 128 is used.
Figure 2 shows the data transfer latency of all-reduce in- curred by GPU kernel. The metrics are measured on LLaMA2-70B distributed inference in different system set- tings with diverse levels of SPD across the model blocks. The measurement is conducted under both high bandwidth (HBW; 300GB/s interconnect) and low bandwidth (LBW; 10GB/s interconnect) device interconnect (Detailed settings about connection bandwidth are explained in Section 5.1). Applying SPD 100%, which halves the number of sync points in the entire model, significantly reduces data trans- fer latency (over 46% reduction in all system settings), re- sulting in substantial overall model latency improvement across diverse system configurations (as in Figure 7c). This highlights the importance of addressing communication bot- tlenecks for efficient distributed inference. However, reduc- ing syne points to minimize latency may lead to disrupted numerical parity, which does not always guarantee non- degraded accuracy. To address this, we propose novel block design and techniques combined with SPD that alleviate communication bottlenecks while minimizing quality loss across various SPD budget cases, providing a scalable solu- tion for distributed inference systems.
complexity of such changes increases if the output projec- tion layers in self attention include a bias.
4.1.1. BLOCK WITHOUT BIAS IN LINEAR LAYER
Figure 3a shows the SPD block design used without bias in linear layer. The most essential objective of block design is constructing the combination of connections which gives least numerical difference between TP and SPD.
MLP input The sync-point enables each parallel device to capture the attention output from all the other devices. How- ever, when the outputs from other devices are unavailable by climination of sync-point, the only information the device (i) can utilize is its own attention output (Y;), Therefore, to minimize the numerical difference compared to the case of all information available, residual connection (X) and attention output of own device (Y;) are added and fed into MLP input (X + Y2).
MLP output When the sync-point exists after attention output, the MLP input is utilized as residual connection added to the MLP output. However, dropping the sync-point yields incomplete MLP input (X +Y) with lack of attention outputs from other devices. The desired block output is a combination of block input (X), attention output from all devices E Yi), and MLP output from all devices E; Z,). Therefore, we disassemble the original residual connection to block input residual (X) and attention output residual from a device (Yz). Then, Y; forms a new type of residual connection which is added before the sync operation. X is added on the same point as the original connection, after the syne operation which finally leads to a complete form of output (X Σ. Yi + Σ, ZT).
4. Sync-Point Drop
As an efficient method to improve distributed inference performance. Sync-Point/ Drop (SPD) selectively removes the all-reduce communication operation after self atten- tion output, as illustrated in Figure lb. In this section, we discuss how to maintain high model quality with reduced communication overhead. First, we introduce a novel block structure design that serves as the foundation block for the non-communicating structure with minimal quality degra- dation. Second, we propose strategy of applying SPD in block-wise manner which achieves lower latency with minimal accuracy loss.
4.1.2. BLOCK WITH BIAS IN LINEAR LAYER
In TP, each of the linear layers in self attention part of a block is parallelized in a different manner. The linear layers before self attention operation (query, key, and value projection) are divided in column-wise manner which enables the bias divided along the same dimension. However, the linear layer after self attention operation (output projection) is parallelized in an orthogonal way, row-wise manner. The bias, vector along the column dimension, therefore, can not be divided in the direction of the row. This requires a
4.1. Block Design
If the synchronization of the self attention output is skipped, the attention output will diverge across multiple devices. Therefore, SPD requires two architecture changes in the transformer block, the MLP input and output in way that information loss from SPD can be minimized. Also, the
3

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
LayerNoms
Salf Attirnton oon
Layerrorm
Activation
LayerNorm
X
W
Liner
Linear
Attontion
Linear
Linear
GPU
GPU 1
GPU 2
GPU 2
Layeroom
Salf Attontion
Linear
Linear
Activation
Lincar
Layeroom
Solf Attordion
Linear
Leyerhoom
Linear
Activation
Linwer
N
Wi
(a) Block design without bias in linear layer.
(b) Block design with bias linear layer.
Figure 3. Decoder block structure with syne-point drop (in 2-GPUs distributed inference case). 'W,' and "8' represent weight and bias of linear layer on each device (i). 'X', 'Y', 'Z,' and 'P,' denotes hidden representation of each device (x) on the figure.
new mechanism of the bias application on MLP input and output as shown in Figure 3b.
better balance between model performance and quality on deployment.
MLP input Different from the case with bias (Section 4.1.1), the indecomposable bias term (b) is included after weight multiplication. Following the essential objective, the least error in the MLP inPinput compared to the result of TP, we use the partial weight multiplication result with the addition of bias (Y; = P. + b) and input residual connection (X) as MLP input (X + P. + b).
4.2.1. BLOCK- WISE SYNC SENSITIVITY IDENTIFICATION
To identify the sensitivity of block SPD, we utilize the perplexity metric by measuring the relative impact of a block to performance (the difference from application between TP block and SPD block in Figure 4) as sensitivity mea- surement. For example, when we measure the sensitivity of i-th block to SPD, we apply SPD to all blocks starting from the {i + 1}-th block to the final block and measure the per- plexity, while leaving the i-th block unchanged (TP block). We then measure the perplexity by additionally modifying the system setting of i-th block to SPD. The difference in perplexity before and after applying SPD to i-th block is used as a measure of sensitivity. In this measurement, we use calibration data obtained by sampling small portion of the large training dataset. By progressive replacement of TP block to SPD block and measurement of quality degra- dation as relative perplexity difference, we can compare the sensitivity between blocks in the entire model and classify the blocks into three sensitivity categories (ISB, SB, and ESB).
ML.P output Following Section 1, the original residual connection is disassembled block input residual (X) and attention output residual (Y₁) from device. Due to the exis- tence of bias, we further disassemble Y; to the result of the partial weight multiplication (P:) and the bias (b). To make the bias not affected by communication, we place the bias residual add after the sync operation while adding the par- tial weight multiplication result before the syne operation. Finally, device, this makes the bias residual be added MLP output while the parallelized weight multi- plication results form complete state through collective communication (X + Σ, P. + b+ Σ, Z.).
4.2. Sync-Point Drop based on Block-wiss Sensitivity
While the lack of communication incurs numerical disparity across all parallel devices, such disparity in different blocks will impact the model accuracy differently. In this section, we introduce a multi-tirre block-wise approach to mini- mize the overall accuracy loss based on the SPD-specific per-layer sensitivity.
Algorithm shows the overall process applying SPD in a multi-ticred block-wise approach with measured sync sen- sitivity. Based on the sync sensitivity value of blocks (S), rank the blocks in an ascending order (B). According the predetermined ranking of the sensitivity, SPD is applied within the target number of blocks to optimize (Nspa). In the sequence, the processing of a block is classified into three sensitivity categories based on predetined threshold criteria (7) and T2). This classification allows us to apply separate approaches aimed at minimizing quality degrada- tion according the identified groups. In the following sections, we introduce the individual strategies applied to three categories of the blocks.
First, in Section 4.2.1, categorize transformer blocks based on their sensitivity to SPD: in-sensitive blocks (ISB),
sensitive blocks (SB), and extremely sensitive blocks (ESB). Based on the classification result, before applying SPD,
we perform individual preprocessing steps (Section 4.2.2,
4.2.3, and 4.2.4). This specific strategy allows to min- imize accuracy degradation from SPD thereby enabling

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
Model
System
Algorithm 1 Sync-point drop based on sensitivity
decoder
1: SPD: SYNC-POINT DROP
block
blook
2: B2B: BLOCK-TO- BLOOKKKKK DISTILLATION
decoder block
TP block
TP
3: HG: ATTENTION HEAD GROUPING INITIALIZATION
bleck
4: Block <- list of all decoder blocks model
5: S list of sensitivity measurement
decoder
TP brock
TP
block
block
6: B < block index list in ascending order of S
7: Napd < Target budget: the number of blocks to SPD
decoder block
TP block
SPD
8:
t sensitivity thresholds
9: for i = 0 to Napd do
decoder
SPD
10: if S[B[]] I T1 then
> Categorize as ISB: Section 4.2.2
block
11:
[Bock[[[i]] +- (PD(Block)
porploiity
porpioaity
12:
else if S|B[ii] S 72 then
b Categorize as SB: Section 4.2.3
Mossure dfference
13:
Block [B[]] SPD (BDDB(Boock]
14:
else
> Categorize ESB: Section 4.2.4
15:
Block [[i]] < (BD (B)]))
Figure 4. Syne sensitivity identification of a decoder block (measuring the sensitivity of i-th block).
16:
end if
17:
end for
4.2.2. IN-SENSITIVE BLOCKS: ZERO-SHOT DROPPING
originally use the same model parameters. However, to ob- tain the special weights aware of eliminated communication, parameters for SPD are newly initialized from the original and used separately.
ISBs show minimal quality degradation with SPD. There- fore, within the targeted budget of communication opti- mization (Nap,), we drop the syne-point of these blocks, prioritized over other types of blocks, in zero-shot manner. Note that zero-shot dropping can give significant amount of benefit with sensitivity identification. As shown in Sec- tion 5, in every model, zero-shot dropping can obtain at least 44% of blocks as SPD blocks with little sacrifice of accuracy.
argmin MSE(SPD(Ospt: 2:), T))), Papa
(1)
4.2.4. EXTREMELY SENSITIVE SLOCKS: SPD AWARE ATTENTION HEAD GROUPING INITIALIZATION
4.2.3. SENSITIVE BLOCKS: SPD AWARE DISTILLATION
Beyond the recovery of block-to-block distillation on SBs, a few blocks show sharp quality degradation. We define these blocks ESBs and introduce novel SPD aware initializa- tion before conducting sync-points are removed, the model partitions located on each device are isolated from each other, preventing mutual access. This makes a decoder block as if it is a combination of parallel and independent mini decoder blocks. In this cir- cumstance, self attention fragment cannot access any MIP partitions in other parallel devices and also MLP partitions are unable to access self attention output in other parallel de- vices, resulting in inevitable information loss. To ensure that these parallel architectures operate as close as the original structure, is important to make attention heads evenly dis- tributed based on functionality following the sparse nature of head activated differently (Liu et al., 2023) and redun- dancy of head showing similar behaviors (Agarwal et al., 2024) on in-context. To reflect these in-context properties to out-context as much as possible, we utilize calibration data and obtain attention score (α) as a metric of the head functionality.
SBs exhibit larger effects on quality degradation compared to ISBs. To further achieve the optimization objectives and recover the associated performance degradation in SB, we obtain SPD aware parameters by adopting block-to-blode distillation. Block-to-block distillation is a low-cost fine- tuning method that involves training only the specific SB with SPD setting. We set the teacher block as TP block and the student block as the SPD block. For the data used in tuning, utilize calibration data used in the sensitivity identification step in Section 4.2.1. This data passes through consecutive TP blocks of the model by the block in which distillation will be conducted. Then, we utilize the hidden representation of the block's 's input where distillation will be performed. Following the fine-tuning objective Equation we forward the hidden representation (x) to each teacher and student block and apply outputs to mean squared error (MSE) loss. Note that the parameter of SPD block (Pspa) is initialized from the parameter of TP block (f). Since SPD and TP are execution methods within the system, they
boocktt-bblock distillation. As the
Head scattering In the self attention, the set of the query
5

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
(Q), key (K) and value (V) associated with each head can be defined A = ; K1, Vi >; <Q, K2, V,>..... ....,.. QN, KN, Vv >} where N is the number of heads. The goal of head scattering is finding the set of heads show- ing the even distribution of attention score (o(Q,, K:)) across the parallel devices. By defining a set of heads to be placed in a device as A; where A; C A and n (A,)) - N e/number_off -devicss the objective of head scattering is defined in Equation 2. We achieve the objective of finding an even distribution based on head functionality by max- imizing the sum of distances on the clustering algorithm which originally utilized the opposite metric. For the dis- tance, attention scores of each sequence as a high dimension vector are utilized with euclidean distance (d).
LayerMor
Layertorm
Liveanu
Actination
Linaaun
GPU 1
LayerNom
Lineshurm
Activation
Linestharr
GPU 2
Self Attention
Linser
Al- reduuc
LayerNorm
Acthvation
Lineersup
GPU 3
Self Attention
Linear
LayeHoom
LayerNam
Lineanua
Actisation
Lineenun
GPU4 4
Sull Attention
m(A.) m(A.) d(o(QAL, KALD), k=tt1
Σ
Σ
1K Kz. /g,.~0o,, Ko
K,, V,>,<s. V3>},
argmax
A;
(2)
KL Kz. V3>. Ky, Vy>), K2 K.. v,>.< Kg. V3y}}}
aQQAN:: KAzw)): (
MC (< MMMPM.LPAAAA. MLP;>, <A MLP,> Ag, MLP+>}
where CCA
Figure 5. SPD block in case having 8- 8-heads on 4- GPG parallel with given head subset (A:) and matching combination (MC
MLP matching After getting the scattered clusters of atten- tion heads, matching A, with proper MLP partition (sharded MLP operation in parallel device) should be conducted to search for complete parallel independent architecture that operate close to the original structure. We found that the norm of MIP output before adding residual connection is well-fit indicator. The MLP output norm is small compared to the residual connection norm (Liu et al., 2023). A large block output norm implies that the block contributes well when combined with the residual. Therefore, we compare the norm of all the matching combinations and pick the best maximum case the matching result. By defining the MLP partition of a device as MLP, and matching combination as MC and its universal set (all combinations) as M MCll, the objective of MLP matching is defined as Equation 3.
in-sensitive
sensitive
extremely sensitive
LLaMA2-7b 8-GPUs
LLaMA2-70 4-GPUs
LLaMA2-13b 8-GPUs
LLaMA2-13b 4-GUUS
LLaMA2-70b 8-GPUs
OPT-6.78 8-GPUs
OPT-6.7B 4-GPUs
OPT-13B 8-GPUs
OPT-13B 4-GPUs
OPT-30B 8-GPUs
OPT-66B 8-GPUs
0%
20%
40%
60%
80%
100%
Figure 6. Block-wise syne sensitivity identification result for LLaMA2 and OPT models over 8-GPUs and 4-GPUs.
MC Σ MLP.
argmax MC
(A.)).
(3)
reside on the same parallel device. As result, group of scattered heads subset and MLP partition is assigned to a single device, working as SPD aware initialization. Apply- ing block-to- block distillation after head grouping further enhances accuracy recovery in the ESBs.
where MC E MC.I..
After determining the optimal A, and MC, the hidden rep- resentation of each head should be physically located on the device designated by MC. Figure 5 illustrates the ex- ample of SPD with best A, and MC. To align the assign- ment with the static behavior of the system in SPD, we reorder the columns of the query, key, and value linear layer weights (Wq. WK. Wy) based on their head-specific partitions (WQn. W WK.. WVE, where h denotes the head in- dex). Similarly, the row order of output linear layer weight (Wo) based on head partitions (Wor) is reordered. This reordering ensures that the hidden representations are dis- tributed in the order of MC, allowing the heads in A, to
5. Experiments
5.1. Setup
Models We conduct experiments on LLaMA2 (7B, 13B, and 70B) (Touvron et al., 2023b) and OPT (6.7B, 13B, 30B, and 66B) (Zhang et al., 2022). We apply 8-GPUs and 4-GPUs distributed inference for all the models except LLaMA2- 70B, OPT-30B, and 66B which apply 8-GPUs setting only.
6

---

SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models
25-020
BIDANG
SPD
25+020+11G
20:82B
87%
30%
13%
15.5%
200%
113%
755%
15
18605
12%
20%
HBW
Trodd
F-noda LBW
x NEW
1-noda
Syeedhp
Spreddip
23%
100
20%
HBW
L-W
22node
2-rode LSW
x2:000 NOW
12-00da LOW
Spectup
37%
10%
17%
Parceringe
(a) LLaMA2-7B 8-GPUs distributed
(b) LLaMA2-13B 8-GPUs distributed
(c) LLaMA2-70B 8-GPUs distributed
29-B28
28-828-H0
75-82B
25-82B +HG
No SPD
70%
60%
25.0%
111%
13%
750%
$625
11.%%
10000%
25.5%
33.0%
111%
110
10.0
10.0%
17.5%
100 PS
1-nodd HBW
sinode LBW
B #:node
Spreedapp
25.0%
750%
10005
25.0%
50.0%
00.%
919%
19.9%
17.5%
100.0%
Peroentage
(d) LLaMA2-7B 4-GPUs distributed
(c) LLaMA2-155 13B 4-GPUs distributed
Figure 7. LLaMA2 distributed inference average accuracy Speedup is normalized based on latency of 0% (No SPD state which consists of TP blocks entire model) in each distributed inference setting. 'ZS' represents applying zero-shot dropping all blocks. 'ZS+B2B' represents applying shot dropping ISBs and block- to-block distillation to the other remaining SBs and ESBs. "H++BB+HG'' is applying zero-shot dropping on ISBs and block-to-block distillation SBs and block-to-block distillation with head grouping initialization the other remaining ESBs. 'HBW' represents high GPU interconnect bandwidth of 300GB/s setting and 'LBW" represents low GPU interconnect bandwidth of 10GB/s setting.
shot tasks (top) and normalized speedup of time first token (bottom).
Calibration data From WikiText2 (Merity et al., 2016)
each 1-epoch utilizing whole 128-samples of calibration data.
training dataset, randomly selected 128- 128-samples consisting of tokens with a sequence length of 2048 are used by fol- lowing existing work (Shao et al., 2024). Each sample of calibration data is utilized as mini batch for distillation.
Environment setting The accuracy and latency of all our experiments are measured on nodes with Nvidia A A100-80G GPU node under high (300GB/s) and low (10GB/s) band- width GPU interconnect setups following (Cho al., 2024). The low bandwidth interconnect is established by turning off the high-speed CUDA-direct link (NVIDIA, 2019). For 2-node of 8-GPUs distributed cases, each node consists of 4-GPUs with 50GB/s interconnect between nodes.
Evaluation data We evaluate the accuracy of optimiza- tion method zero-shot tasks (ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019). LAMBADA (Paperno et al., 2016), PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), and WinoGrande (Sakaguchi et al., 2020)) by aver- aging all the results and MMLU tasks (Hendrycks et al., 2021).
5.2. Sensitivity Identification
Hyper-parameter setting For all models except larger mod- els (LLaMA2-70B, OPT-30B, and OPT-66B), we Ti 0.05 and as 10. For larger models, we use Ti as 0.02 and as 10. In block-to-block distillation on SBs and ISBs, the learning rate is used as 5 x 10-5 for LLaMA2 and x 10-6 for OPT. 10-epochs distillation conducted with
Figure 6 shows the block-wise sync sensitivity identification result of the blocks in LLaMA2 and OPT models. For all models, the percentage of ISBs (yellow bar) indicates that the same amount of blocks can used as SPD with an ignorable accuracy drop (less than about 1% on zero-shot tasks). This can be achieved in the shro-shot manner (de-

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
25
Perceriage BPD block to total number blodka
Percentage SPD blak number blocka
Percentage SPD block total number blocks
(a) OPT-6.7B 8-GPUs distributed
(b) OPT-13B 8-GPUs distributed
(c) OPT-30B 8-GPUs distributed
25+020
BID-II
No SPD
10%
k70
69
53.5%
$3.5%
1000%
3505
36.5%
65.7%
100%
18.35
500%
90.0%
100
150%
50.5
62.5%
15.%
112%
9%.%
55.0%
1000%
Percentage GPD black number
Parcentage SPD book number
(d) OPT-6.7B 4-GPUs distributed
(e) OPT-13B 4-GPUs distributed
(f) OPT-668 8-GPUs distributed
Figure 8. OPT distributed inference average accuracy zero-shot tasks (Notations are same in Figure 7),
tailed results described in Section 5.3). The percentage of ISBs increases when the model size gets larger (75% in LLaMA2-70B 8-GPUs and 84% in OPT-66B 8-GPUs). Overall, LLaMA2 models show higher sensitivity compared to OPT models. LLaMAA-7B 8-GPUs model is available with zero-shot drop of 44% while entire OPT models are available with dropping 70% of blocks. ESBs are shown only in smaller models (LLaMA2-7B, 13B, and OPT-6.7B)
has connection between the nodes, both HBW and LBW interconnect setups show significant amount of latency im- provement. For all models with 2-node system, SPD shows over 10% speedup on both HBW and LBW of SPD per- centage over 70%. In LLaMA2-13B (Figure 7b) and 70B (Figure 7c), speedup over 20% can be achieved on the ex- treme level SPD (nearly 100%) with LBW setting.
Figure 8 shows the SPD results of OPT models zero-shot tasks. OPT models show less drop compared to LLaMA2 models possibly due to high redundancy (Liu et al., 2023; Agarwal et al., 2024). Models except 6.7B show a maximum 1.3% degradation regardless of the sensitivity of the block. Therefore, results in OPT with 7S+B2B show small im- provements since they already have less drop only with ZS. However, in OPT-6. 6. (Figure 8a and 8d), when the drop occurs in ZS by increasing the percentage of SPD block, ZS+B2B and BS+B2B+HG give recovered accuracy (+2.8% in 8-GPUs of Figure 8a and +2% in 4-GPUs of Figure 8d on 100% SPD).
with one two blocks.
5.3. Sensitivity based Sync-Point Drop
Figure 7 shows the SPD results of LLaMA2 models on zero- shot tasks. After the amount of target SPD blocks exceeds in-sensitive boundary, zero-shot dropping (ZS) shows large accuracy drop (over 1%) in all models and system settings. Block-to-block distillation with ZS (ZS+B2B) successfully recovers large amount of accuracy degradation in SB region, especially giving larger amount on smaller models (+28% % on 13B 8-GPUs of Figure 7b and +20% on 7B 4-GPUs of Figure 7d on 100% SPD). Furthermore, smaller models hav- ing ESBs show further accuracy recovery from B2B (+3% on 7B 8-GPUs of Figure 7a and +2% on 13B 4-GPUs of Figure 7e on 100% SPD) with adding head grouping initial- ization (ZS+B2B+HG). Similar tendencies are appeared on MMLU results in Appendix A.
Overall the proposed SPD effectively alleviates sync-point bottleneck while minimizing accuracy degradation. This shows that SPD gives both moderate optimization with no performance degradation and the better trade-off between larger optimization and performance leading to scalable solution.
SPD brings benefits of latency improvement from the elimi- nation of syne point while the multi-tiered block-wise ap- proach recovers accuracy degradation. In 1-node cases, overall, the lower the device interconnect bandwidth, the larger the speedup SPD achieves. For LLaMA2-70B with LBW in Figure 7c, 70% SPD offers about 19.7% speedup while only sacrificing 0. 0.94% accuracy. Note that this result from simple zero-shot dropping from ISBs. In 2-node cases (Figure 7a, 7b and 7c) where the distributed system
6. Conclusion
In this paper, we present Sync-Point Drop (SPD), a novel optimization technique that improves the latency of LLMs distributed inference systems. By adopting new block design and separated approaches based on block-wisely identified sensitivity for lack of sync-point, SPD enables efficient deployment across multiple computing units with
8

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
little compromising model performance. Our experiments show that SPD successfully alleviate communication over- head in tensor parallelism with minimum quality loss in all budgets, which enable scalable solution for distributed inference systems.
Language models are few-shot learners. arXiv preprint 2005::0005//1163, 2020.
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E.,, Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lund- berg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of artificial general intelligence: Early ex- periments with gpt-4. arXiv preprint arXiv:2303. 127112 2023.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chee, J., Cai, Y., Kuleshov, V., and Sa, C. M. D. Quip: 2-bit quantization of large language models with guaran- tees. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Acknowledgments
Cheng, S., Liu, Z., Du, J., and You, Y. Atp: Adaptive tensor parallelism for foundation models. arXiv preprint WrXiv:230108865, 2023.
We thank Rasoul Shafipour, Raviteja Vemulapalli, Chen- fan Sun, Iman Mirzadeh, Keivan Alizadeh Vahid, Devang Naik, Kulin Seth, and Brendan Langoulant for valuable discussions and comments.
Cho, M., Rastegari, M., and Naik, D. Kv-runahead: Scalable causal I1m inference by parallel key-value cache gener- ation. International Conference on Machine Learning (ICML), 2024.
References
Agarwal, S., Acun, B., Hosmer, B., Elhoushi, M., Lee, Y., Venkataraman, S., Papailiopoulos, D., and Wu, C.-J. Chai: Clustered head attention for efficient Ilm inference. International Conference on Machine Learning (ICML), 2024.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C.,, and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv::802 08880000557. 2018.
Dong, H., Johnson, T., Cho, M., and Soroush, E. Towards low-bit communication for tensor parallel 11m inference. arXiv preprint S.ti:2411.0099444 2024.
Agrawal, A., Hedlund, M., and Hechtman, B. exmy: A data type and technique for arbitrary bit precision quantization. arXiv preprint arXiv:2405. 13338, 2024.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. Interna- tional Conference on Machine Learning (ICML), 2023.
Aminabadi, R. Y., Rajbhandari, S., Zhang, M., Awan, A. A., Li, C., Li, D., Zheng, E., Rasley, J., Smith, S., Ruwase, O., and He, Y. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale. International Conference on High Performance Comput- ing, Networking, Storage and Analysis, 2022.
Frantar, E., Ashkboos, S., Hoefler, T,, and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. International Conference on Learning Representations (ICLR), 2023.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Jaggi, M.. Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.004566, 2024.
Gunter, T., Wang, Z., Wang, C., Pang. R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C.-C., Qiu, D., Gopinath, D., Yap, D. A., Yin, D., Nan, F., Weers, F,, Yin, G., Huang, H., Wang, J., Lu, J., Peebles, J., Ye, K., Lee, M., Du N., Chen, Q., Keunebrock, Q. Wiseman, S., Evans, S., Lei, T., Rathod, V., Kong, X., Du, X., Li, Y., Wang, Y., Gao, Y., Ahmed, Z., Xu, Z., Lu, Z., Rashid, A., Jose, A. M., Doane, A., Bencomo, A., Vanderby, A., Hansen, A., Jain, A., Anupama, A. M., Kamal, A., Wu, B., Brum, C., Maalouf, C.,, Erdenebileg, C., Dulhanty, C., Moritz, D., Kang, D., Jimenez, E.,, Ladd, F.,, Shi, F., Bai, F,, Chu, F., Hohman, F., Kotek, H., Coleman, H. G., Li, J., Bigham, J., Cao, J., Lai, J., Cheung, J., Shan, J., Zhou, J., Li, Qin, J., Singh, K., Vega, K., Zou, K., Heckman, L.., Gardiner, I.., Bowler, M., Cordell, M.,
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural lan- guage. AAAI conference on artificial intelligence (AAAI), 2020.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J.. Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M.. Gray. S., Chess, B., Clark, J., Berner, C., McCan- dlish, S., Radford, A., Sutskever, I., and Amodei, D.

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
Cao, M., Hay, N., Shahdadpuri, N., Godwin, O., Dighe, P., Rachapudi, P., Tantawi, R., Frigg, R., Davarnia, S., Shah, S., Guha, S., Sirovica, S., Ma, S., Ma, S., Wang, S., Kim, S., Jayaram, S., Shankar, V., Paidi, V., Kumar, V., Wang, X., Zheng, X., Cheng, W., Shrager, Y. Ye, Y., Tanaka, Y., Guo, Y., Meng, Y., Luo, Z. T., Ouyang, Z. Aygar, A., Wan, A., Walkingshaw, A., Narayanan, A., Lin, A., Farooq, A., Ramerth, B., Reed, C., Bartels, C., Chaney, C., Riazati, D., Yang. E. L.., Feldman, E., Hochstrasser, G., Seguin, G., Belousova, I., Pelemans, J., Yang, K. Vahid, K. A., Cao, I.., Najibi, M., Zuliani, M., Horton, M., Cho, M., Bhendawade, N., Dong, P., Maj, P., Agrawal, P., Shan, Q. Fu, Q., Poston, R., Xu, S., Liu, S., Rao, S., Heeramun, T., Merth, T., Rayala, U., Cui, V., Sridhar, V. R., Zhang. W., Zhang, W., Wu, W., Zhou, X., Liu, X., Zhao, Y., Xia, Y., Ren, Z., and Ren, Z. Apple intelligence foundation language models. arXiv preprint arXiv:2407.21075. 2024.
inference time. International Conference on Machine
Learning (ICMI.), 2023.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint WMXi::1609007433 2016.
NVIDIA. Nvidia collective communications library (ncel), http::/ddeeeooprr.vviiaa.ccooo/ 2019.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. Association for Com- putationaa Linguistics (ACL), 2016.
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cap- pelli, A., Alobeidli, H., Pannier, B., Almazrouci, E., and Launay, J. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.00 I 16, 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. International Conference on Learning Representations (ICLR), 2021.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. AAAI conference on artificial intelligence (AAAI), 2020.
Huang, Y. Cheng, Y., Bapna, A., Firat, O., Chen, M. X., Chen, D., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., and Chen, Efficient training of giant neural networks us- ing pipeline parallelism. Neural Information Processing Systems (NIPS), 2019.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang. K., Gao. P., Qiao, Y., and Luo, P. Omniquant: Omnidirectionally calibrated quantization for large lan- guage models. International Conference on Learning Representations (ICLR), 2024.
Z. Gpipe:
Jeaugey, S. Massively scale your deep learning training with http:///wwwo.iiiiiiiooooo/oammia
neel 2.4. heeepleemiing--rainnnn--eo------ 2019.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv: 190.000990 2019.
Jiang A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., Lavaud, L. R., Lachaux, M.-A,, Stock, P., Scao, T. L., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., and Kim, J.-J. Sleb: Streamlining Imms through redundancy verifica- tion and elimination of transformer blocks. International Conference on Machine Learning (ICML), 2024.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. Symposium on Operating Systems Principles (SOSP). 2023.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. International Conference on Learning Representations (ICLR), 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302 /391 2023a.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation weight quantization for on-device Ilm compression and acceleration. P Proceedings of Machine Learning and Systems (MLSys), 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D.,, Fernandes, J., Fu, J., Fu, W.,
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., and Chen, B. Deja vu: Contextual sparsity for efficient lims at
10

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez,
V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, M., Subramanian, R., Tan, X. E., Tang. B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092880 2023b.
WelbI, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. Proceedings of the 3rd Workshop Noisy User-generated Texi (WNUT), 2017.
Xia, M., Gao, T., Zeng, Z., and Chen, Sheared llama: Accelerating language model pre-training via structured pruning. International Conference Learning Repre- sentations (ICLR), 2024.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. International Conference on Machine Learning (ICML), 2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? Association for Computational Linguistics (ACL), 2019.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M.,
Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mi- haylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
Zhao, Y. Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L.., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauhan, G., Hao, Y., Mathews, A., and Li, S. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proceedings of the VLDB Endowment, 2023.

---

SPD: Syne-Poin Drop for Efficient Tensor Parallelism of Large Language Models
A. Sensitivity based sync-point drop
25-818
25.525+HG
-o SPD
2S-020
254525HH1
-N SPD
25-54B -.-M SPD
40%
68%
756
67%
17%
67%
376
209
20.%
13.0%
60.0%
10.0%
40.0%
69.9%
99.%%
108
25%
18.0%
85.0%
68.50
10.0%
10.0%
50.5%
00.0%
100
37%
280%
500%
15.0%
11.5%
20.0%
08.95
09
08.
189.95
Parcentage block number
Parcentage
blocks
Parcentage STSP
number
(a) LLaMA2-13B 8-GPUs distributed
(b) LLaMA2- 10BMMA---BBBBB 4-GPUs distributed
(c) LLaMA2-7000 A0MB 8-GPUs distributed
Figure LLaMA2 distributed inference accuracy on MMLU tasks (Notations are same Figure 7),
B. Ablation study
B.1. Effects of design choice block design
Table I. SPD MLP output design choice WikiTex12 perplexity on block (SPD only on block the model).
Attention output residual add (Y,)
PPL (4)
Bias residual add (b)
PPI. (1)
LLaMA2-7B no SPD
5.47
OPT-6.7B SPD
10.86
Before MIP all-reduce
10.65
Before MLP all-reduce
332.60
After MLP all-reduce
177.69
After MLP all-reduce
13.07
(a) Without bias linear layer
(b) With bias in lincar layer
Section 4.1 shows that the tensor parallelism block system not compatible with lack of communication and this makes several design choices block structure. Table 1a and 1b show quality degradation per design choice MLP output. Whether the targeted residual connections each table use collective communication not will be determined by the addition point (before and after MIP all-reduce). The results show that using collective communication on attention output residual (Table la) and not using it on bias (Table 1b) are the proper choice of residual addition point design selections as in Figure which minimizes negative effect from SPD.
C. Discussion
C.1. Sensitivity identification
In Section 1,2.1, when measuring the sensitivity of block SPD, we apply SPD consecutive blocks starting from the selected block and extending toward the last block. This approach allows us to evaluate the worst-case impact of SPD on the block's output while ensuring that its input activations remain numerically identical to the original, unaffected by prior SPD modifications. By analyzing the perplexity difference between a given block and the next block, we can isolate the effect of SPD at that specific layer, excluding any influence from subsequent layers. This method provides fast yet precise and independent measurement of each block's sensitivity.
C.2. Compatibility with other parallelism
SPD can be integrated with diverse settings to suit specific application contexts. this section, we describe how SPD can be combined with such settings.
Data parallelism Data parallelism can be achieved with SPD by replicating an SPD model across distributed GPU environments. For example, in 8-GPUs configuration, 4-GPUs SPD model can be instantiated as single replica, which is then duplicated across the remaining 4-GPUs. Consequently, two replicas operate in parallel, forming data parallelism setup with SPD.
Pipeline parallelism Pipeline parallelism can be incorporated into SPD by applying it at the level of each parallelized decoder block, which the attention and forwwrd network (FFN) components are already distributed across multiple GPUs. For instance, in an SPD configuration where a single decoder block is partitioned across 4-GPUs, pipeline parallelism
12

---

SPD: Syne-Point Drop for Efficient Tensor Parallelism of Large Language Models
be realized by further dividing the entire distributed blocks into two sequential groups. This results in 4x2-GPUs system, where each of the four SPD branches is executed in two-stage pipeline across pair of GPUs, thereby enabling pipeline parallelism within SPD blocks.
Hybrid parallelism A appeline-paralleinn model can be instantiated as replica to enable data parallelism. By replicating this aopeline-parallelinn model, data parallelism can be applied on top of the pipeline setup. This approach ultimately facilitates the implementation of combined parallelism in conjunction with SPD.
C.3. Impact of SPD on tensor parallelism compared to pipeline parallelism
In this work, we focus on optimizing tensor parallelism (TP) by addressing the communication bottlenecks that frequently arise in distributed inference systems. Although TP is widely adopted in practice due to its engineering simplicity and high compute utilization, pipeline parallelism (PP) is often considered promising alternative, primarily due to its lower communication overhead. Nonetheless, TP remains the preferred strategy in many real-world distributed inference deployments, even under constrained interconnect bandwidth, due its ability more effectively utilize available compute resources.
While the relative reduction in communication volume achieved by SPD may appear modest (maximum 50% since SPD only targets the sync-point after self attention output), its impact on end-to-end latency is substantial. This because the reduction communication bottlenecks allows the remaining computational tasks, which are less affected by the system's bandwidth, proceed more efficiently, ultimately improving the end-to-end performance. Therefore, TP optimized by SPD more effective in deployment scenarios where compute resources need to be fully utilized across GPUs, as seen in prefill stage of large language model (Figure 7).
13


Reviews:
[{"review_id": "q5CWzbGn8w", "reviewer": "Reviewer_e4AF", "summary": "This paper introduces SPD aiming at reducing communication overhead in tensor parallelism for LLMs. SPD selectively removes synchronization points in attention outputs to mitigate latency bottlenecks during distributed inference. The authors propose a block-based SPD approach and classify transformer blocks into three sensitivity categories (insensitive, sensitive, and extremely sensitive) to balance performance trade-offs. Their empirical results on LLaMA2 and OPT models demonstrate up to 20% inference speedup with less than 1% accuracy regression.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful advices to our work. Here are the replies to your concerns.\n\n**Q1. The evaluation is limited to LLaMA2 and OPT models. Testing on more recent models could validate its generalizability.**\n\nDue to space limit, we couldn’t share our new results here. Please refer to the displayed result of Qwen2.5-7B on [Reviewer USNW - Q2](https://openreview.net/forum?id=23zxLtvder¬eId=ZTV6Aoih0B).\n\nIn short, we’ve observed comparable effectiveness of SPD on Qwen2.5-7B, further validating its generalizability to recent models. Thank you for your understanding.\n\n**Q2. The evaluation benchmarks are logprob-based tasks. It would be appreciated if more generation based or reasoning tasks will be included.**\n\nAgain, due to space limit, we would appreciate if you can check out the displayed result of Longbench-Multinews (summarization) benchmark on Llama-2-7b-chat on [Reviewer USNW - Q4](https://openreview.net/forum?id=23zxLtvder¬eId=ZTV6Aoih0B).\n\nIn short, we’ve obtained positive results on generation tasks with SPD, showing performance comparable to the logprob-based benchmarks. \n\n**Q3. Would SPD work on MoE architectures or hybrid parallelism setups?**\n\nThank you for the chance to elaborate how SPD works with MoE and hybrid parallelisms. The blocks and algorithms of SPD can be applied to any model or system that tensor parallelism (TP) can be applied.\n\n* Applying on MoE architectures\n    * Token Routing (Top-k Experts Selection)\n        * In SPD, the routing network (softmax over token scores) can be replicated across GPUs following TP. The overhead with the replicated routers is negligible since they have much fewer parameters and require much less computation than the expert network. The router in each GPU assigns each token to the top-k selected experts having same decision per devices. GPUs begin expert computations immediately with local routing decisions.\n    * Expert Feedforward Layers (Sparse Computation)\n        * Each expert is a fully connected feedforward network (FFN). The selected experts process only their assigned tokens, with evenly distributed to all GPUs. After computation, the expert outputs are processed through next FFN layer, then requiring final all-reduce communication between GPUs.\n    * Residual connection and bias in SPD block\n        * As in dense models, skip connections and bias connections can be applied using the same structure presented in Figure 3.\n* Applying on hybrid parallelism setup\n    * Data parallelism\n        * Copying a SPD model replica in GPU setting can make data parallelism work with SPD. For example, when we have 8-GPUs, we can set 4-GPUs distributed SPD model as one replica and set the copied replica to the other 4-GPUs so that 2 replicas are running as data parallelism setup.\n    * Pipeline parallelism\n        * Pipelining each parallelized execution of SPD (a fraction of attention and FFN of one block in a single GPU branch) can make pipeline parallelism work with SPD. For example, we have 4-GPUs distributed SPD block. In this case, we can divide a fraction of attention operation and FFN operation of one GPU to a pair of GPUs. As a result, 4 x 2 - GPUs server can work for pipeline parallelism setup and SPD eliminates all-reduce operation of former GPU in each 4-GPU-pairs.\n    * Combined data and pipeline parallelism\n        * One pipeline parallelized model can be set as a replica for data parallelism and the copied replica can make data parallelism setup with pipeline parallelized setup. This can finally make hybrid parallelism working with SPD ({TP+SPD} + data parallelism + pipeline parallelism)\n\n**Q4. For recent hardware e.g. H200/H100, the communication speed is actually very fast. The gain of removing an AR may be incremental.**\n\nThank you for your comments. We first like to share how H100 is compared with A100 on computation and communication.[6]\n\n* H100 offers 3.17× higher computation performance (on FP16 TFLOPS)\n* H100 offers 1.5× faster NVLink interconnect for device-to-device communication\n\nHence, we can easily say that overall inference latency will go down thanks to stronger HW capability. However, we’d like to highlight that since compute has grown even faster, communication becomes an even larger bottleneck than before. In the Table below, we projected how compute vs. communication ratio changes for Llama2-7B 8-GPUs distributed inference when we move from A100 to H100, and it clearly shows that now the communication takes about 1.8x larger portion of the overall latency (i.e. from 15% to 27%), thus making SPD would be even more important. We hope this address your concern around how SPD will help with the recent hardware advancements.\n\nTable: H100 projected computation/communication ratio compared to A100 on HBW (300GB/s) interconnect (Llama2-7B 8-GPUs)\n|  | Computation (FP16) | Communication |\n| --- | :---: | :---: |\n| A100 | 85% | 15% |\n| H100 (projected) | 72.8% | 27.2% |\n\n\\\nReference: Please refer to the references of Reviewer USNW.", "claims_and_evidence": "1. SPD reduces inference latency by selectively dropping synchronization in tensor parallel models.\n\n    Supported: Experimental results on LLaMA2 and OPT models show significant speedup (up to 20%) with low accuracy degradation.\n\n2. The block-wise sensitivity identification approach effectively balances performance and accuracy.\n\n    Supported: Results indicate that zero-shot SPD can be applied to ~44-84% of blocks with minimal impact on accuracy.", "methods_and_evaluation_criteria": "The paper employs:\n\n1. Distributed inference on LLaMA2 (7B, 13B, 70B) and OPT (6.7B, 13B, 30B, 66B) across 4-GPU and 8-GPU setups.\n2. Zero-shot accuracy tasks (ARC, HellaSwag, LAMBADA, etc.) and MMLU benchmarks to measure accuracy degradation.\n3. Latency measurements across different interconnect bandwidths (300GB/s vs. 10GB/s).\n\nPotential concerns:\n\n1. The evaluation is limited to LLaMA2 and OPT models. Testing on more recent models (e.g., Llama 3.3, Qwen 2.5, DeepSeek V3/R1) could validate its generalizability.\n2. The evaluation benchmarks are logprob-based tasks. This do not aligns with the goal of reducing decoding latency in this paper. It would be appreciated if more generation based or reasoning tasks will be included.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiments effectively validate SPD’s performance:\n1. Multiple LLM scales (7B-70B) and architectures (LLaMA2, OPT).\n2. Both high-bandwidth and low-bandwidth scenarios.\n\nHowever, there are also some concerns:\n1. The evaluation is limited to LLaMA2 and OPT, which are relatively old.\n2. The evaluation benchmarks are logprob-based and relatively simple. It would be appreciated if more generation based or reasoning tasks could be included.", "relation_to_broader_scientific_literature": "The paper builds on prior work in Tensor Parallelism, where SPD modifies this by selectively removing synchronization. There are other LLM Optimization, e.g. Quantization, Pruning, while SPD is an orthogonal approach that focuses on communication efficiency.", "essential_references_not_discussed": "There are no specific references currently on my mind that have not been discussed.", "other_strengths_and_weaknesses": "Pros:\n1. SPD provides a simple yet effective method by removing sync point to reduce the inference latency.\n2. The latency vs. accuracy trade-offs are well-analyzed.\n\nCons:\n1. The evaluated models are relatively outdated and weak. It would be great to evaluate on more recent LLMs.\n2. The evaluation benchmarks are logprob-based and relatively simple. It would be appreciated if more generation based or reasoning tasks could be included.", "other_comments_or_suggestions": "1. Would SPD work on MoE architectures or hybrid parallelism setups?\n2. For recent hardware e.g. H200/H100, the communication speed is actually very fast. The gain of removing an AR may be incremental.", "questions_for_authors": "Please check \"Other Comments Or Suggestions\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A2sE2ejJiO", "reviewer": "Reviewer_LBEy", "summary": "This paper proposes a novel method to drop one of the two all-reduce communications in tensor parallelism (TP) to mitigate the communication of TP in LLM inference. After dropping the first all-reduce communication, the sync point between the attention and mlp is removed, raising concerns on the model accuracy. To reduce the negative impact on accuracy of the model, the paper proposed several methods:\n- Modifications to model architecture, especially residual links to retain most accurate semantics\n- Classify the model blocks (transformer layers) into insensitive, sensitive and extremely sensitive blocks by the impact it would have if the sync point in this layer is dropped.\n- For sensitive and extremely sensitive layers, the authors proposed several methods including distillation and to recover the performance loss\nThe experiments results shows 10%-20% performance improvements with variant performance loss. In the best case where the communication cost is high due to low communication bandwidth, the method can achieve 20% acceleration with less than 1% accuracy loss.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. This paper claims to improve the scalability of TP, but no direct scaling evaluation is shown.**\n\nThank you for the comment. While we haven’t directly measured the scalability of TP with SPD, we can intuitively understand that SPD would lead to better scaling efficiency as GPU counts increase since SPD surgically targets the communication overhead (which is a key barrier to high quality scalability). The question is how much better and it also depends on many complex factors including the compute/communication ratio as discussed with [Reviewer e4AF - Q4](https://openreview.net/forum?id=23zxLtvder¬eId=q5CWzbGn8w).\nWe couldn’t directly measure the scalability due to the time and resource limit during this rebuttal period, but plan to add detailed scalability measurements in the final draft.\n\n**Q2. The method might have different performance on different models.**\n\nThank you for the important comments. For more data point, we experimented with Qwen2.5-7B [Reviewer USNW - Q2](https://openreview.net/forum?id=23zxLtvder¬eId=ZTV6Aoih0B) and Longbench-Multinews (summarization) on [Reviewer USNW - Q4](https://openreview.net/forum?id=23zxLtvder¬eId=ZTV6Aoih0B). In short, our new results indicate SPD shows the comparable effectives as it showed on the paper.\n\n**Q3. Lack of impact: A safe threshold for a negligible accuracy loss is 50% SPD results in 25% reduction of the overall TP volume.**\n\nThank you for the chance to discuss the impacts. Although a 25% reduction may seem modest at first, achieving this reduction with no additional cost is significant. Most model-level optimizations, such as quantization or sparsity, require preprocessing steps or specialized hardware to realize performance gains. In contrast, SPD selectively removes sync point for blocks by simply not performing all-reduce, effectively removing 25% with negligible accuracy loss. Furthermore, in larger model (Llama2-70B), 70% of SPD results in 35% reduction. These results demonstrate that SPD offers a very low cost approach to improve efficiency in large-scale distributed inference. Also, note that SPD provides a scalable communication that adapts to different budget constraints while minimizing accuracy loss. \n\n**Q4. What is the definition of latency in the evaluations?**\n\nThank you for the chance for clarification. The latency reported is end-to-end model latency when generating one token on prompting (time to first token) with batch size 1 and prompt length 128 on A100 by following the metrics utilized in existing optimizations [2,3] . On Figure 2, we will add the definition of latency. On Figure 7, to provide the intuitive comparison of reduced latency, we displayed the speedup with improved latency of SPD from latency without SPD. For the decoding latency, since the operations are bottlenecked to memory, the communication optimization including SPD and existing works [4,5] does not give great improvement compared to prefill stage. We will add more details to the final version.\n\n**Q5-1. 10GB bandwidth is too low.**\n\nSince the GPU only supports interconnect with NVLink (300GB/s) and without NVLink (10GB/s), there are no more selections that we can test with.\n\n**Q5-2. For high-bandwidth (HBW) scenarios, the latency improvement is around 5% without accuracy loss. The performance gain in practical case seem to be moderate.**\n\nGeneral-purpose infrastructure, such as GPU servers, is designed for high performance across all aspects, featuring hardware support that minimizes communication overhead. As a result, with HBW, the latency improvement from SPD may appear moderate. However, in specialized infrastructures, particularly inference-focused hardware, cost constraints often limit HBW interconnect support compared to GPUs. In such environments, where communication bandwidth is a more significant bottleneck, SPD can play a crucial role in reducing sync overhead and improving overall efficiency. This makes SPD particularly valuable for inference infrastructure with limited interconnect performance.\n\n**Q5-3. If the paper wants to claim TP becomes practical in this scenarios, it should compare with other parallelisms.**\n\nThank you for the comments. We design SPD for TP since TP is the most widely used techniques in distributed inference. Therefore, we focus our evaluation on TP to specifically address its communication bottleneck. Also, TP would work with other parallelisms in a hybrid mode (please see [Reviewer e4AF - Q3](https://openreview.net/forum?id=23zxLtvder¬eId=q5CWzbGn8w)). We plan to explore the suggestion as a future direction.\n\n**Q6. Why the SPD blocks have to be consecutive blocks from the last block?**\n\nPlease refer to [Reviewer J4A1 - Q2](https://openreview.net/forum?id=23zxLtvder¬eId=ekQlcDOwVR). In short, measuring SPD sensitivity with consecutive blocks enables isolating a block’s impact by analyzing perplexity difference. We will add more details to the final version.\n\n\\\nReference: Please refer to the references of Reviewer USNW.", "claims_and_evidence": "One of the key insight of this paper lies in the finding that most (>50%) layers can drop the TP sync point without significant latency, which is well supported by the experiments, where <= 50% zero shot drop usually have negligible accuracy loss.\n\nHowever, the performance gain is a bit overclaimed if conditioned on negligible accuracy loss. It is misleading to claim 20% acceleration. According to Figure 7 (a), (b) (c), which is the most common setting to use TP on 8 GPUs on llama2 models, to keep <1% performance achieves approximately 5% latency gain if NVLINK is used. There is a 20% latency improvement under the condition of running 70B model on 8 GPUs with only a bandwith of only 10GB, which is not a common setting for running TP. Notice that SPD cuts the communication volume by at most 50%.\n\nThis paper claims to improve the scalability of TP, but no direct scaling evaluation is shown. For example, how does the improved latency changes if the number of GPUs scales out?\n\nMoreover, there is a divergence between the performance of OPT and llama 2 models, suggesting the method might have different performance in terms of accuracy loss on different models.", "methods_and_evaluation_criteria": "The SPD method and the accuracy recovery methods for SB and ESB blocks are well designed. Their model accuracy is verified by evaluations properly.\nThis paper compares the latency with baseline methods. It is unclear what this latency exactly means. e.g. is it e2e latency of a single request? or an iteration in LLM inference?", "theoretical_claims": "The methods involved in model accuracy recovery are well supported by empirical evaluations.", "experimental_designs_or_analyses": "10 GB bandwidth is too low for an efficient deployment of TP. If the paper wants to claim TP becomes practical in this scenarios, it should compare with other model parallelism schemes applicable in model inference. At least, the paper should reveal more common metrics of a model inference system like throughput, time between tokens (TBT), etc so readers can get an idea whether the baseline is a legit baseline.\n\nOverall, for HBW senarios, the latency improvement is around 5% if the accuracy loss is negligible. The performance gain in practical cases seem to be moderate.", "relation_to_broader_scientific_literature": "The method introduced in this paper would accelerate LLM serving systems.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: An effective method showing dropping sync points in TP might be beneficial with negligible accuracy loss\nOther Weakness: \n- Lack of impact: From the evaluations, a safe threshold for guaranteeing a negligible accuracy loss is around cutting 50% sync points, which results in 25% reduction of the overall TP communication volume.", "other_comments_or_suggestions": "The header of each page is still 'Submission and Formatting Instructions'.", "questions_for_authors": "- What is the definition of latency in the evaluations?\n- Why the SPD blocks have to be consecutive blocks from the last block?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTV6Aoih0B", "reviewer": "Reviewer_USNW", "summary": "This paper introduces SPD, a novel optimization to reduce communication overheads in tensor parallelism b selectively dropping synchronization on attention outputs. SPD categorizes attention blocks into 3 types based on their sensitivity to the model accuracy, and applies different block design. SDP effectively reduce the communication while minimizing accuracy degradation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your support on our work:\n\n**Q1. Insufficient details about how to set hyper-parameters**\n\nThank you for the comment. While we will add more details on hyper-parameters to the final draft, we like to point out one observation we made during our study about τ1 and τ2. We observed that there are distinct points where the sensitivity metric (perplexity difference with consecutive SPD blocks) sharply increases as SPD is applied to more sensitive blocks. These points served as natural candidates for setting τ1 (over 0.05 perplexity difference) and τ2 (over 10 perplexity difference), allowing us to categorize blocks into insensitive, sensitive, and extremely sensitive groups. This approach strikes a balance between maintaining model accuracy and achieving communication reduction, as described in Section 4.2. The results in Section 5 show that the thresholds are robust across different configurations, minimizing accuracy degradation while maximizing the benefits of SPD. We will add this details in the final version of paper.\n\n**Q2. It remains unclear how well the method generalizes to modern LLM architectures featuring GQA and MoE.**\n\nWe are happy to share more experimental results around the modern LLM architectures. In general, the blocks and algorithms of SPD can be applied to any model or system, as long as tensor parallelism (TP) can be applied.\n\n**1. GQA:**\n\nThe each head of key and value can be firstly divided across GPUs and the matching queries can be followed as in TP. We provide more results of Qwen2.5-7B on zero-shot tasks (same as the presented results of Figure 7) in the Table below. The results show that SPD can be successfully applied to LLM architectures with GQA.\n\nTable: Qwen2.5-7B 4-GPUs distributed inference average accuracy on zero-shot tasks (ARC, HellaSwag, LAMBADA, PIQA, SciQ, WinoGrande). Notations are same as in Figure 7 (‘ZS’: applying zero-shot dropping. ‘ZS+B2B’: applying ZS on ISBs and block-to-block distillation to the other remaining SBs. ‘ZS+B2B+HG’: applying ZS on ISBs, B2B on SBs and B2B with head grouping initialization to the other remaining ESBs.)\n| SPD drop percentage | 0% | 25% | 50% | 64.3% | 75.0% | 85.7% | 92.9% | 96.4% | 100% |\n| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| ZS | 75.3 | 73.8 | 72.0 | 71.6 | 71.5 | 68.9 | 65.8 | 60.3 | 38.9 |\n| ZS+B2B | 75.3 |  | 72.8 | 72.6 | 72.7 | 70.4 | 70.5 | 64.6 | 61.2 |\n| ZS+B2B+HG | 75.3 |  |  |  |  |  |  |  | 61.6|\n\n**2. MoE:**\n\nDue to the space limit, we would appreciate if the reviewer can check out our response to [Reviewer e4AF - Q3](https://openreview.net/forum?id=23zxLtvder¬eId=q5CWzbGn8w).\n\n**Q3. Is it possible to integrate this approach with sparsification techniques such as sparse attention?**\n\nThank you for the question, and the our answer is yes.\nSPD is an orthogonal approach focusing on communication efficiency to other LLM optimizations, e.g. sparse attention, quantization, pruning. Sparse attention can be easily integrated into distributed attention weights distributed in each devices. The distributed attention weights are derived from query and key which are came from partial linear Q, K projection layers divided along the head dimension. Therefore, diverse sparse attention techniques working within each head dimension of partial attention weights are easily applicable with SPD settings.\n\n**Q4. How does the method affect accuracy on tasks that require handling long context lengths? + Q5. These zero-shot tasks evaluated typically generate a single token. What is the impact on generation accuracy when the model is required to produce a longer sequence?**\n\nThank you for raising important question and we conducted extra experiments to find answer to the question.\nThe Table below shows Longbench-Multinews (summarization) benchmark [1] on Llama-2-7b-chat. Similar with Figure 7 and 8 of the paper, in generation task, SPD effectively alleviate communication bottlenecks while minimizing accuracy degradation during LLM inference.\n\nTable: Longbench-Multinews Rouge-L score on Llama-2-7b-chat 8-GPUs distributed\n| SPD drop percentage | 0% | 25% | 50% | 62.5% | 75.0% | 84.4% | 93.8% | 96.9% | 100% |\n| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| ZS | 25.8 | 26.0 | 25.6 | 25.4 | 25.2 | 18.3 | 7.1 | 4.2 | 1.2 |\n| ZS+B2B | 25.8 |  |  | 25.4 | 25.4 | 25.1 | 24.9 | 24.3 | 20.1 |\n| ZS+B2B+HG | 25.8 |  |  |  |  |  |  | 24.7 | 22.7 |\n\n[1] https://github.com/THUDM/LongBench/tree/main/LongBench#summarization\n\n[2] Liu et al, Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time, ICML 2023\n\n[3] Agarwal et al, CHAI: Clustered Head Attention for Efficient LLM Inference, ICML 2024\n\n[4] Agrawal et al, exmy: A data type and technique for arbitrary bit precision quantization, 2024\n\n[5] Dong et al, Towards Low-bit Communication for Tensor Parallel LLM Inference, NeurIPS 2024\n\n[6] Nvidia, https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/", "claims_and_evidence": "Claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Methods and evaluation make sense.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experimental design is sound.", "relation_to_broader_scientific_literature": "This paper helps improve the LLM inference efficiency when utilizing tensor parallelism.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n1. The paper presents a novel solution to alleviate the communication bottleneck for LLM inference with tensor parallelism.\n2. SDP achieves good speedup without accuracy degradation.\n\nWeaknesses\n1. Insufficient details about how to set hyper-parameters $\\tau_1$ and $\\tau_2$\n2. It remains unclear how well the method generalizes to modern LLM architectures featuring GQA and MoE.", "other_comments_or_suggestions": "Thanks for submitting to ICML '25!", "questions_for_authors": "1. How does the method affect accuracy on tasks that require handling long context lengths?\n2. Is it possible to integrate this approach with sparsification techniques such as sparse attention?\n3. These zero-shot tasks evaluated typically generate a single token. What is the impact on generation accuracy when the model is required to produce a longer sequence?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ekQlcDOwVR", "reviewer": "Reviewer_J4A1", "summary": "Sync-Point Drop (SPD) is an inference-time technique for reducing the latency of distributed inference with tensor parallelism. The paper suggests removing all-reduce operation between attention and feed-forward layers of transformer blocks. At the same time, some of the residual connections within the transformer block also need to be added/modified. They refer to this compound modification as SPD. The authors show that it is possible to apply it without significantly losing the accuracy of the model. \n\nThe authors start sync-point dropping from the later layers of the model. As they proceed with each block, they measure the perplexity of the resulting configuration. They compute differences between consecutive perplexity evaluations and take those as the sensitivity values of each individual block. They sort the blocks by sensitivity and assign them to three tears: insensitive, sensitive, and especially sensitive. They recognize that applying SPD as-is (zero-shot) to sensitive blocks degrades the performance, so they come up with two additional techniques based on attention head reordering (HG) and distillation (B2B) explicitly directed at them.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your support of our work, and we provide detailed responses to your comments as follows:\n\n**Q1. The paper would benefit from quantifying the potential speed-up in set-ups that do not suffer communication bottlenecks.**\n\nThanks for the comment. We plan to add more details to the final draft.\n\nSPD targets the communication bottlenecks incurred by all-reduce operation after self attention. In Figure 7, we utilize 100% percentage of SPD block to total number of blocks as maximum speed-up case. In this case, all communications after self attention are eliminated while all communications at the end of the block (after feed forward network) are remaining. However, using the case of removing all the communications in model which doesn’t suffer from communication bottleneck is impossible since the proposed block structure and sensitivity measuring of SPD cannot be exploited in this case. In conclusion, instead of measuring latency of all communications removed cases which is not realizable, we used latency from removing all communications after self attention which is the best case in realizable solution.\n\n**Q2. The way sensitivity is measured looks arbitrary. Why don’t I measure sensitivity by applying SPD to individual blocks or applying SPD to the blocks at the beginning of the model?**\n\nThank you for the question. When measuring the sensitivity of a block to SPD, we apply SPD to consecutive blocks starting from the selected block and extending toward the last block. This approach allows us to evaluate the worst-case impact of SPD on the block’s output while ensuring that its input activations remain numerically identical to the original, unaffected by prior SPD modifications. By analyzing the perplexity difference between a given block and the next block, we can isolate the effect of SPD at that specific layer, excluding any influence from subsequent layers. This method provides a fast yet precise and independent measurement of each block's sensitivity.\nThe sensitivity analysis reveals that the impact of SPD varies significantly across different layers. For instance, in Qwen2.5-7B ([Reviewer USNW - Q2](https://openreview.net/forum?id=23zxLtvder¬eId=ZTV6Aoih0B)), the sensitivity ranking of blocks from least to most sensitive is observed as (26-24-5-25-22-17-11-20-21-18-12-10-7-14-16-15-4-23-19-8-2-13-9-3-6-1-27-0). This highlights that some layers are highly resilient to SPD, whereas others are more sensitive, underscoring the importance of a structured approach in determining which blocks to apply SPD while minimizing its impact on model accuracy.\n\n**Q3. The authors suggest B2B distillation. HG includes modifynig the architecture. The motivation for me to use them is unclear.**\n\nThank you for the comment. While B2B and HG may not provide a perfect accuracy without loss, they offer valuable options for users seeking more aggressive optimizations. Additionally, HG does not modify the model architecture but rather permutes weights before distillation, consequently having same shapes of parameters. This simple adjustment helps improve accuracy after distillation in a distributed setting, making it a useful technique for enhancing performance without structural changes to the model.\n\n**Q4. Figure 1.b is misleading. You are suggesting architecture in Figure 2.**\n\nThank you for the comment. Figure 1.b is a conceptual illustration showing the removal of sync operations, while Figure 2 presents the proposed block structure designed to achieve this without accuracy loss. To clarify the distinction, we will revise the title of Figure 1.b to \"With elimination of sync-point in attention output\", emphasizing that it does not represent SPD itself but rather illustrates the conceptual effect of removing sync points.\n\n**Q5.  How does SPD interplay with other latency reduction techniques (quantization, pruning, block skipping)? Do I actually need SPD if I already have all of them?**\n\nSPD is an orthogonal approach focusing on communication efficiency to other LLM optimizations targeting computation efficiency, e.g. efficient attention, quantization, pruning. This means that while computation has improved, communication is not optimized, making synchronization a larger bottleneck than before. This leads to a greater proportion of time spent in end-to-end model execution. In this circumstance, SPD can addresses this issue by reducing the number of sync points rather than just making them faster.\n\n**Q5.a. How much does sync-point dropping affect the communication-computation gap? One would expect that the first all-reduce in the block is a major bottleneck.**\n\nSince SPD is optimizing communication channel, after SPD, the percentage of communication time channel is reduced. However, the difference between first all-reduce (after self attention) and second all-reduce (after feed-forward layer) are negligible since they use same tensor size for communication and they cannot be overlapped in both cases.", "claims_and_evidence": "The paper presents well-supported claims related to Sync-Point Drop (SPD). The experimental results would benefit from including margin of error for measurements, especially for the measurements of latency. \n\nThe paper would benefit from quantifying the potential speed-up in set-ups that do not suffer communication bottlenecks. Detailed profiling of an inference step (before and after modifying the block) is missing.", "methods_and_evaluation_criteria": "Measuring the model performance in terms of downstream task accuracy and latency ms is a reasonable approach.\n\nAn alternative to measuring latency could be a characterization of just how much communication-computation gap this approach can close in realistic scenarios.\n\nThe way sensitivity is measured looks arbitrary. Why don’t I measure sensitivity by applying SPD to individual blocks instead? Why don’t I start applying SPD to the blocks at the beginning of the model instead of applying it at the end?", "theoretical_claims": "The paper is empirical.", "experimental_designs_or_analyses": "The experimental setup in Section 5 is elaborated clearly. There are no obvious errors in the experiments or their analysis. \n\nIt remains unclear how much overlap between communication and computation the authors were able to achieve in their experimental setup.\n\nThe analysis would benefit from a more obvious illustration of the trade-off between latency and accuracy of the model.", "relation_to_broader_scientific_literature": "The Section 2 of the paper gives a good overview of the research on improving the latency of comms-bound inference systems. Quantization, pruning, and block-skipping reduce the communication load. \n\nThe paper only touches upon improving a single type of model parallelism, namely, the tensor parallelism. Other important types, the pipeline parallelism and the expert parallelism, are left aside.\n\nThe paper does not reflect on the other common types of bottlenecks that could impact latency: memory and computation bottlenecks.", "essential_references_not_discussed": "The Section 2 of the paper gives a good overview of the research on improving the latency of comms-bound inference systems.", "other_strengths_and_weaknesses": "The key advantage of the proposed approach is the simplicity of zero-shot SPD and its apparent effectiveness.\n\nThe authors suggest B2B which includes distillation. HG includes modifynig the architecture. Both of these techniques are much more complex than zero-shot SPD and don’t seem to recover performance to a significant degree. The motivation for me to use them is unclear.", "other_comments_or_suggestions": "Figure 1.b is misleading. You are suggesting architecture in Figure 2; reconciling it with Figure 1.b takes time.", "questions_for_authors": "The top concern that is not addressed by the paper:\n1. How does SPD interplay and compare with other latency reduction techniques (quantization, pruning, block skipping)? Do I actually need SPD if I already have all of them?\n\n1.a How much does sync-point dropping affect the communication-computation gap (the difference between the percentage of time comm channels are fully utilized and the percentage of time comp kernels are fully utilized)? One would expect that the first all-reduce in the block is a major bottleneck since most of the computation is actually happening in the feed-forward layer.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Sync-Point Drop (SPD), a novel technique to mitigate communication bottlenecks in distributed LLM inference by selectively skipping synchronization steps in attention blocks during tensor parallelism. SPD’s block design enables communication-free execution while balancing accuracy and efficiency, achieving a 20% reduction in inference latency with <1% accuracy loss for LLaMA2-70B on 8 GPUs.

During rebuttal, the authors effectively addressed reviewer concerns, including SPD’s applicability to modern accelerators, improved user-perceived latency in bandwidth-constrained scenarios, and its advantages over pipeline parallelism in some scenarios. These clarifications led to unanimous positive ratings from reviewers. The AC also concurs with the recommendation to accept this paper.
Decision: Accept (poster)