DynaPipe: Dynamic Layer Redistribution for Efficient Serving of LLMs with Pipeline Parallelism
Authors: HongXin Xu
Tianyu Guo
Xianwei Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model
Pipeline Parallelism
KV Cache
Sample
Primary area: infrastructure
TL;DR: DynaPipe dynamically redistributes layers and uses asynchronous coordination to balance computation during LLM inference, significantly reducing latency and outperforming existing pipeline parallelism systems.

Abstract: To accelerate large language model (LLM) inference, pipeline parallelism partitions model layers into sequential stages, each assigned to a different device for concurrent execution. However, this method often suffers from pipeline bubbles caused by imbalanced computation in the tail stage. While upstream stages focus solely on layer-forward operations, the final stage must also handle post-processing tasks like sampling, introducing significant latency. This uneven workload leads to pipeline misalignment, forcing upstream stages to idle and degrading overall performance. Existing frameworks typically distribute layers evenly across stages without accounting for computational load differences. To address this, we propose DynaPipe, a dynamic layer redistribution scheme that adaptively balances computation by predicting execution latency in real time. Moreover, we introduce an asynchronous key-value (KV) cache migration coordinator to enable
non-blocking layer redistribution during inference. Experiments on representative LLMs demonstrate that DynaPipe reduces average end-to-end request latency by 8% to 49% across diverse workloads, outperforming state-of-the-art pipeline parallelism systems.

DynaPipe: Dynamic Layer Redistribution for Efficient Serving of LLMs with Pipeline Parallelism
Hongxin Xu Tianyu Guo"l, Xianwei Zhangi
1CSE, Sun Yat-Sen University
(xuhx56,guote guoty9u wmall...aaa1111......... edu. en, zhangxu770mail. sys.:...mmmmal.u........ sysu. edu, cn
Abstract
To accelerate large language model (LLM) inference, pipeline parallelism parti- tions model layers into sequential stages, each assigned a different device for concurrent execution. However, this method often suffers from pipeline bubbles caused by imbalanced computation the tail stage. While upstream stages focus solely on layer-forwart operations, the final stage must also handle additional post-processing tasks like sampling, which introduces significant latency. This discrepancy in workload leads to pipeline misalignment, forcing upstream stages to idle and degrading overall performance. Existing frameworks typically distribute layers evenly across stages without accounting for computational load differences. To address this, we propose DynaPipe, dynamic layer redistribution scheme that adaptively balances computation by predicting execution latency in real time. Moreover, we introduce an asynchronous key-value (KV) cache migration coordi- nator to enable non-blocking layer redistribution during inference. Experiments on representative LLMs demonstrate that DynaPipe reduces average end-to-end request latency by 8% to 41% across diverse workloads, outperforming state-of- the-art pipeline parallelism systems. Our implementation publicly available at https: / thub. wwt.c//////ooo/. wwwpxhxwcccccccc/ttt whx102xx220 wttp//022//D//aapaa/ DDynaPipe.
Introduction
The rise of large language models (LLMs) [1, 3] has become cornerstone of modern artificial intelligence, empowering a wide range of application fields, from conversational agents [4, 5] and intelligent coding assistants [6. 7] impressive capabilities, deploying LLM at scale presents significant challenges, primarily due to their immense computational and memory demands. As model sizes grow from billions hundreds of billions of parameters, inference costs rise exponentially, rendering single-device execution impractical. This has spurred extensive researches and engincering efforts aimed developing efficient distributed inference strategies [11, 12, 13, 14, 15].
aaugmeel- generation systems [8, 9, 10]. Despite their
Among these strategies, Tensor Parallelism (TP) [12, 16, 17] and Pipeline Parallelism (PP) [18, 19, 20] have emerged as two dominant methods, now widely adopted in industrial- LLM serving systems [21, 22]. TP partitions the computation within each layer across multiple devices, but its scalability can be often restricted by substantial inter-device communication overhead, particularly in bandwidth- constrained scenarios. In contrast, PP divides the model layers into sequential stages, each assigned to different device. Thus, PP processes multiple throughput by reducing communication overhead and enabling better overlap of computations across stages.
batches in a pipelined manner, improving
"Equal contribution
'Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Layer
Sample
Redistributed layer
Inter-batch dependence
Layer adjustment
S0: 15L
Even Layer Assignment
S1: 15L
S2: 15L
$3: 15L
S0: 15L
Uneven Layer Assignment
$1: 16L
$2: 16L
S3: 13L
Figure 1: Illustration of pipeline bubbles during autoregressive decoding. Top: Conventional even layer allocation leads to idle upstream stages due to tail-stage sampling delays. Bottom: Uneven layer assignments to reduce idle time and improve hardware efficiency. "SO:15L. means stage consists of 15 layers.
While PP has proven effective in owwbbadwwithh environment, it introduces critical computational overhead that has not been adequately addressed: the additional latency introduced by the sampling' operation at the final stage the pipeline. As shown in Figure traditional pipeline systems typically distribute model layers evenly across stages but fail to account for the workload imbalance caused by sampling computation. In autoregressive generation, each decoding step strietly depends on the previous one, meaning the next step can only begin after the current token is generated. This dependency forces upstream stages to wait for the output from the final stage, resulting in significant pipeline bubbles that reduce hardware utilization. As sampling becomes more time-consuming, it bottlenecks the entire pipeline's 's progress. The inherent stage dependency exacerbates this issue, as each stage must wait for the previous one to complete. Consequently, the sampling latency propagates through all stages, amplifying the bubble effect and degrading overall execution efficiency.
To address these challenges, we propose DynaPipe, a solution that dynamically reallocates model layers across pipeline stages during runtime optimize workload balance. As illustrated in the bottom of Figure 1, DynaPipo redistributes some layers from the final stage earlier pipeline stages, thereby reducing computational imbalance and significantly minimizing pipeline bubbles. To achieve this, DynaPipe continuously monitors the runtime status of each stage and dynamically adjusts the suyertt--ttage mapping balance the forward computation and sampling load. Moreover, DynaPipe avoids pipeline stalls during redistribution by introducing asynchronous KV cache migration mechanism, which enables seamless and non-blocking layer redistribution across devices.
In summary, the contributions of this paper are as follows:
We identify critical pipeline efficiency bottleneck in LLM inference that has been largely overlooked. The sampling operation at the last stage creates severe pipeline bubbles and significantly reduces GPU utilization.
We propose DynaPipe, dynamic layer redistribution scheme that adaptively optimizes layer allocation across pipeline stages at runtime to balance forward computation and sampling workloads. By introducing an asynchronous KV cache migration mechanism, DynaPipe enables non-blocking pipeline adjustments for seamless execution.
Our evaluation of DynaPipe across diverse data conditions demonstrates its consistent superiority over state-of oftte-ff- frameworks, reducing average end-to-end latency by 8-41% while enhancing service-level objective (SLO) attainment.
For convenience, collectively refer the calculation of logits and the sampling operation as "sample".

---

2 Background and Motivation
2.1 Transformer-based LLM Inference
LLMs based on the Transformer architecture [23] perform autoregressive generation by stacking multiple decoder layers. Each decoder layer primarily comprises self-attention mechanism and feed-forward network (FFN). During inference, each input token is first converted into a continuous vector through an embedding layer and then passed sequentially through the decoder layers. Within each layer, the model computes the query (Q), key (K), and value (V) vectors. The attention mechanism calculates the similarity between Q and K via dot product, applies softmax function to obtain normalized weights, and uses them to compute a weighted sum over V. thereby capturing dependencies between tokens different positions. The FFN then applies fully connected layers with non-linear activation to enhance the model's 's representational capacity. The final hidden state is then projected logits, representing probability distribution over the vocabulary. The next token sampled based on this distribution using strategies such as greedy decoding [24] or beam search [25].
To improve inference efficiency, modern LLM incorporates KV cache mechanism. During autore- gressive generation, a new Q attends to past K and V at each step. Since K and V remain unchanged, caching them avoids redundant KV computation and improves inference efficiency. Let n denote the number of tokens to be computed at the current step, L the total sequence length, and d the dimension of the hidden states. The total computational complexity of the attention mechanism is (nd + nd). which includes the cost of computing Q. K, V and the attention operations. The complexity of the FFN is (nd2). By reusing cached K and V, the KV cache significantly reduces the time and memory overhead of attention, especially in long- uongsequencce generation tasks, thereby improving inference speed and overall efficiency.
2.2 Parallelism in LLM Inference
To achieve efficient distributed inference for LLM, modern systems typically employ two mainstream parallelization strategies: TP and PP. TP divides the large matrix operations within model layer and distributes them across multiple devices for parallel computation, overcoming the memory limitations of single GPU. However, this fine-grained parallel strategy requires frequent exchange of intermediate activations during the model's forward pass, leading to significant cross-device communication overhead.
Alternatively, PP employs a coarse-grained model partitioning strategy, which divides the model into sequential stages of consecutive layers, with each stage being assigned to different devices for execution. During inference, input data is split into multiple micro- batches, which are fed into the pipeline in time-sequential manner and processed through each stage. Once completing computation on a micro-batch, a pipeline stage asynchronously transmits the intermediate outputs to the next stage, and immediately proceeds process the next batro-batch. As batcro-batches flow concurrentyy across different stages, the system forms an efficient pipeline scheduling mechanism, significantly improving computational resource utilization and overall throughput. Since PP only requires data exchange between adjacent stages, substantially reduce communication bandwidth pressure. As result, PP has become a core parallelization technique widely adopted in current industrial-grade LLM inference systems.
2.3 Challenges in PP
Inspired by TD-Pipe [26] and gLLM [18), we find that although PP provides notable efficiency benefits for LLM inference, its performance remains susceptible to inter-stage bubbles. These bubbles typically arise from imbalanced computational workloads across different pipeline stages, leading to inefficiencies and idle time. Specifically, while some stages are still processing time-consuming tasks, others remain idle, waiting for results. Through detailed performance analysis, we have identified critical but previously overlooked bottleneck caused by the sampling latency at the end of each model forward.
While most pipeline stages perform uniform forward computations, the final stage necessitates to handle additional token sampling operation, which adds extra computational load. This imbalance makes the final stage a potential performance bottleneck, as the additional sampling workload disrupts the even distribution of computational tasks across stages. Although static layer redistribution
3

---

0.6
Moun:
Simppe Time (mns)
Layer 1ime (mi)
0.4
30
60
80
189
120
260
400
400
900
Sample Tokens
Sample Layer Time
(c) Foeward Tokens
Figure 2: The overhead analysis of sampling and forwarding.
can partially mitigate the sampling overhead, the dynamic nature of both sampling and forward computation times makes a single static strategy unable to maintain balanced workloads under dynamic workload conditions. The fundamental reason is that sampling overhead mainly depends on the number of tokens to be sampled, which is determined by the number of decode requests, since prefill tokens do not require sampling.
Figure 2 presents profiling results that analyze this overhead in detail. As shown in Figure 2(a), sampling latency grows approximately linearly with the number of sampled tokens, indicating that increased decode requests directly lead higher sampling overhead. Meanwhile, Figure ((c) illustrates the relationship between the forward computation time of single layer* and the total number tokens, showing that forward computation time is also dynamic and positively correlated with the combined number of prefill and decode tokens in the batch. Consequently, the ratio of sampling time to single-layer forward time is not constant, but varies with the composition of tokens in each batch, reflecting the dynamic proportion of decode to prefill tokens. As shown in Figure 2(b), this ratio approximately follows a normal distribution. Statistical results further indicate that, on average, the sampling overhead is about 3.09x the latency of a single layer's ' forward computation. This finding highlights that the sampling operation is not a negligible terminal step but rather a non-trivial performance factor that introduces substantial latency and significantly impacts the overall efficiency of the pipeline.
In summary, current PP-based inference systems lack the ability to dynamically adapt to the fluctuating workload introduced by sampling. Consequently, sampling-induced bottlenecks at the tail stage can propagate upstream, negatively impacting overall system performance. To address this issue, a dynamic workload balancing mechanism is needed that reallocates computational workloads to -inpllieg induced bubbles and improve overall efficiency of the pipeline.
reduce sampling-
3 Design
DynaPipe is collaborative optimization framework designed to reduce pipeline bubbles caused by tail-stage sampling. As illustrated Figure 3, DynaPipe consists of three core components: an execution time predictor, bubble-aware scheduler, and a migration coordinator. The predictor employs a lightweight predictive model to estimate the per-layer forward computation time and the sampling latency. Using these estimates, the scheduler determines whether dynamic layer redistribution is needed to balance the workload across pipeline stages. When migration is triggered, the system asynchronously transfers KV cache across devices ensuring that the pipeline execution continues without interruption. By tightly coordinating these components, DynaPipe dynamically adapts to runtime workload variation, effectively minimizing pipeline inefficiencies and improving hardware utilization.
3.1 Execution Time Predictor
As described in Section 2.3, the latency of forward and sampling computations in LLM inference varies significantly at runtime. To develop an effective layer redistribution strategy, accurate estimation of latency each stage is essential. To this end, we construct two independent latency prediction models. The first model estimates the forward computation time of a single Transformer layer based on the number of tokens processed per step and the sequence length The second model estimates
"With identical architecture and batch, pay-layer forward time is uniform and serves as standard measure for sampling cost, guiding the subsequent determination of the number of layers to reallocate.

---

$3.1 Execution Time Predictor
$3.2 Bubble-Aware Scheduler
15
window
16
Predicted Time
15
threshold
16
mm3
15
Batch size
15
12
Number of token
$3.3 Migration Coordinator
adjust to uneven
adjust to even
Figure 3: Overall architecture and workflow of DynaPipe. "S/T" means source/target stage.
the sampling overhead, which increases approximately linearly with the number of sampled tokens, as shown in Figure 2(a). Together, these models form our execution time predictor, capable of dynamically estimating per-layer inference latency and sampling cost, enabling an accurate and adaptive pipeline layer redistribution strategy.
To construct latency prediction model with theoretical grounding, the analysis starts with the computational characteristies of a single Transformer layer. As discussed in Section 2.1, the compu- tation of a single layer primarily consists of the QKV projection, self-attention mechanism, and the feed-forward network (FFN). Since the computational complexity of the QKV projection and FFN is the same, they be combined, simplifying the overall computational complexity of the layer as O(nd² + nLd).
For practical deployment, we adopt scheduling strategy known as chunked prefill[19], which merges prefill and decode requests within the same computation chunk. The prefill phase is compute- intensive, as it processes the full input sequence in chunk. In contrast, the decoding step generates one token at a time, relies heavily on KV cache access, and thus constrained by memory bandwidth. Despite their different computational characteristics, both phases are executed together in unified forward computation. This unified computational flow motivates a joint modeling approach to capture the latency characteristics of both phases. Notably, in the decode phase, represents the number of tokens to be computed per forward pass, which is typically 1.
We model the total execution latency of a single Transformer layer over all requests within a chunk as:
Thyer Σ (oin + L2n, + +
Here, d1, thy and are parameters fitted from profiling data. N denotes the number of requests contained in the chunk. This formulation accounts for request-level computational costs, providing a practical estimate of total layer latency under chunked execution.
In addition, to account for the sampling overhead in the tail stage of decoding, we introduce separate prediction model:
sample = O: Neteodle + /
where Natexdt denotes the number of decode requests in the chunk, and a, B are fitted coefficients representing the growth rate and fixed overhead of sampling, respectively.
5

---

3.2 Bubble-Aware Scheduler
Building on the latency estimates provided by the execution time predictor, pipeline stages can be assessed for potential imbalances during LLM inference. To address these imbalances, we design bubble-aware scheduler that dynamically adjusts layer allocation across stages. The scheduler aims to minimize pipeline bubbles resulting from uneven execution times, thereby improving overall pipeline efficiency and hardware utilization.
In particular, as the number of decode requests increases and more tokens need to be sampled, the sampling overhead in the final pipeline stage becomes non-negligible. To ensure that this stage maintains the same computation time as other stages and further to achieve pipeline alignment, we reduce the number of layers assigned to this final stage. Let & denotes the number of layers removed from this stage and m denotes the number of stages received these layers. These removed layers are evenly redistributed to the upstream pipeline stages maintain workload balance.
To quantify the misalignment introduced by this adjustment, we define the following metric:
k: TTayer m min(k, stages
A T sample
k
Thayer
This metric measures the difference in computation time between the final stage and the preceding stages after layer redistribution. Ideally, when A = O and m = stgges - 1, all pipeline stages aligned in execution time, leading to improved pipeline efficiency and resource utilization.
To further enhance the stability of the scheduling strategy and avoid the overhead caused by frequent redistribution, introduce stability window threshold mechanism. A configuration is activated only when it appears consistently across all positions within a sliding window, effectively filtering out short-term fluctuations. This mechanism enhances scheduling robustness while ensuring smooth and efficient scheduling during inference.
3.3 Migration Coordinator
During the initialization phase, the system pre-allocates memory and loads the weights for additional layers that may be reassigned. This enables pipeline stages to quickly access these preloaded parameters during redistribution. However, weight preloading alone does not complete the migration process, as it also requires the transfer of the corresponding KV cache. To address this, we designed and implemented migration coordinator to manage and transfer KV cache during layer redistribution, thereby enabling efficient collaboration across different pipeline stages.
As shown in Figure 3, when the scheduler triggers layer redistribution, the migration coordinator minimizes disruption to ongoing inference, enabling smooth transition and preserving overall system performance. The core idea behind this design is to leverage pipeline parallelism and maximize the overlap between computation and communication during the migration process. Specifically, when the layer configuration of a stage changes, the system compares the current and adjusted layer assignments to identify which layers need to be migrated or received, thereby determining the precise migration scope. During migration, once the source stage completes computation for the layers to be migrated, it asynchronously transmits the corresponding KV cache to the target stage. Meanwhile, the target stage asynchronously receives the KV cache for the layers it is about to take over and continues executing forward computations for the unaffected layers. The target stage only waits for the KV cache when it reaches the newly assigned layers, and begins computation as soon as it receives the cache for those layers. Since these KV cache are typically sent advance by the source stage, the target stage can complete reception promptly, avoiding prolonged stalls. This design achieves parallel overlap of computation and communication, avoiding prolonged pipeline stalls caused by migration operations and ensuring quick switch in layer redistribution.
4 Evaluation
4.1 Experimental Setup
Implementation. We implement DynaPipe based on gLLM [18], compact and efficient LLM inference framework that outperforms existing production-level inference frameworks pipeline parallelism. The pipeline workers use ZeroMQ [27] for efficient inter-process communication.

---

gLLM
VLLM
SGLang
DynaPipe
AdEx
ShareGPt
E2II (U-
SLO AIt (%)
Att.
6.701
0.9%
12
18
24
30
36
12
18
24
30
12
24
12
18
24
30
Request Rate (ragglss)
Request Rate (requis)
Request Rate (requh)
Request Rate (roguin)
Azare Cone
X222 (5
SL. Att- (10)
100-40
3.5%
PART (1-
SLO Att. (%
10th 1dth
8.0c
0.30
12
(roqu/s)
Request Rate (roquis)
Request (roqul))
Request
(rogs1))
Qwem2.5-148
Qwea2.5-322
Figure 4: The average latency and SLO attainment rate of different LLM serving systems. DynaPipe achieves the lowest average latency and the highest SLO compliance across all workloads.
ensuring smooth synchronization between stages, with each worker responsible for specific tasks. KV cache migration is performed using NCCL [28] for nigh-perfommnne cross-GPU communication. optimizing synchronization and data transfer in multi-GPU environments. All experiments are conducted on system with four NVIDIA A A100-PCle-40G [29] GPUs connected via PCIe. For DynaPipe, we set the window threshold size 25 to strike balance between performance and overhead.
Datasets and models. We use the ShareGPT [30] and Azure-Conv [31] datasets evaluate the system's performance. The ShareGPT dataset, which contains real-world conversational data between users and ChatGPT [3], widely used for evaluating dialogue systems. In contrast, the Azure-Come dataset, sourced from Azure's production environment for dialogue inference services, typically shows longer input lengths than output lengths. Table summarizes the average input and output lengths for both datasets. We perform experiments using two different model variants: Qwen2.5-14B and Qwen2.5-322 [1].
Table 1: Input/ Output length.
Table 2: SLO requirements.
Dataset
#Input
#Output
Dataset
Model
TIFT (s)
TPOT (ms)
P50
221
157
14B
100
ShareGPT
ShareGPT
P90
627
382
32B
4
250
P50
514
192
14B
1
100
Azure-Conv
Azure-Conv
P90
1008
412
32B
150
Baselines. We compare DynaPipe with several ofate-of-the-ar LLM inference frameworks, includ- ing vLLM (v0.8.5 V1) [21] and gLLM [18], both using pipeline operations, well as SGLang (v0.4.3.post2) [22], which employs tensor parallelism. All these systems use the chunk prefill technique [19], with the maximum chunk size set 2048, serving the baseline for performance evaluation.
Metrics. We evaluate system performance using two metrics: the average end-to-end latency per request (E2EL) and the SLO compliance rate. E2EL refers to the total time from when a request is issued to when the response generation is completed. The SLO compliance rate measures the system's stability under service-level constraints: a request considered compliant if both TTFT and TPOT are below the thresholds listed in Table 2. TTFT is the time from request submission to the generation of the first token; TPOT the average time to generate each output token. The SLO compliance rate is defined the proportion of requests that meet both latency constraints.
4.2 Overall Performance
We conduct comprehensive evaluation of DynaPipe across multiple datasets and model scales. Experimental results demonstrate that DynaPipe consistently outperforms mainstream inference frameworks under various workloads. As illustrated in Figure the performance gap is particularly
7

---

notable when compared with vLLM and SGLang, both of which fall significantly behind gLLM due architectural limitations. Specifically, SGLang relies solely on tensor parallelism, which incurs substantial communication overhead under high-concurrency conditions, quickly becoming performance bottleneck. vLLM, while adopting pipeline parallelism to improve throughput, uses fixed-size chunks that lead to pronounced inter-batch pipeline bubbles, resulting in suboptimal hardware utilization. gLLM mitigates these inter-batch bubbles via more adaptive scheduling but overlooks the non-trivial overhead introduced by sampling, leading to inter-stage bubbles that further constrain end-to-end performance.
In contrast, DynaPipe redistributes layers to enable sampling to overlap with the forward compu- tation of extra layers other stages. This design effectively fills idle phase in the pipeline and alleviates structural pipeline bubbles, yielding a substantial performance boost. As shown in Figure 4, DynaPipe consistently achieves superior performance under varying request loads, with especially pronounced gains under high-throughput scenarios. This improvement stems from earlier completion of the prefill phase at higher loads, resulting in an increased proportion of decode requests per batch. Consequenly, the sampling overhead becomes more significant bottleneck, which DynaPipe is well-positioned to mitigate.
Furthermore, experiments across different model sizes show that DynaPipe consistently reduces end-to-end latency. highlighting its effectiveness in mitigating sampling overhead. On the ShareGPT dataset, DynaPipe achieves up to a 40% reduction in E2EL, while on Azure-Cony, -onv, the reduction reaches 34%. The relatively smaller gain on Azure-Conv is attributed its higher input-output length ratio, which leads to longer per-layer execution time in the prefill phase and reduces the relative impact of sampling. Nevertheless, DynaPipe still delivers substantial improvements in this scenario, highlighting its versatility and resilience under varying workload conditions and data distributions.
In the SLO attainment analysis, DynaPipe demonstrates significant advantages. Under the 90% SLO attainment target, SGLang and vLLM struggle to meet the requirements at high request rates. In contrast, DynaPips ip maintains higher SLO compliance under high request rates. DynaPipe can also sustain up to 19% higher request rates than gLLM under 90% SLO attatinment.
4.3 Performance under Different Output-Input Length Ratios
The acceleration effect of DynaPipe is closely related to the output-input length (O:I) ratio. We thus test various static redistribution strategies on a synthetic dataset, with the input length fixed at 512 and the output length varying to form different O:I ratios. As shown in Figure 5, during the prefill phase (output length 1), sample overhead is minimal. In this case, static average distribution achieves the lowest latency, with DynaPipe performing similarly. In contrast, other static redistribution strategies are unnecessary at this case and instead introduce inappropriate layer assignments, leading to inter-stage load imbalance and degraded performance. As the output length increases, the decoding phase becomes significantly longer, leading to a gradual rise in sampling overhead. In this scenario, static load-balancee strategies fail to address the resulting inter-stage imbalance, ultimately causing decline in hardware utilization. On the other hand, the performance of other static layer redistribution strategies begins improve, mainly because adding additional layers helps offset the sample overhead, balancing the pipeline. However, the effectiveness of different static layer redistribution strategies varies under different input and output length ratios. The greater the proportion of sample overhead, the more layers are allocated, which is more beneficial. Nevertheless, DynaPipe consistently outperforms other methods with the lowest latency, and its adaptive capabilities make it especially effective in real-world scenarios where input and output lengths fluctuate.
1.2
Norm. EZEL.
Static (l=:8)
Scattc (1a1)a-
- Static (l (k-2,m=21 2) - DynaPipe
Stattc (k-3.m-3)
Norm. para.
14-
Narmalaed K2EL. Adjumtuut Count
Stude Rederbuuie (Bor))
1.1
1.00
0.0
0.125
0.25 O:l Ratio
0.375
8.5
Window Threihald
Figure 5: Normalized E2EL under different O:1 ratios with various adjustment strategies.
Figure 6: The effect of adjustment window threshold on E2EL and adjustment frequency.
8

---

4.4 Performance under Different Window Thresholds
Figure 6 normalizes the E2EL using the best static layer redistribution strategy on the ShareGPT dataset as the baseline. illustrating the impact of the adjustment window threshold on the E2EL and adjustment count for the 32B model. can be observed that when the adjustment window threshold is too small, the scheduler frequently triggers re-adjustments, leading to significant overhead, with the end-to-end latency increasing by 35% compared to the best static redistribution strategy. the other hand, if the window threshold is too large, the scheduler fails to adjust in time to address the overhead caused by sample operations, resulting an increase end-to-end latency. Setting the window size to 25 strikes good balance between these factors. As shown in the figure, on the ShareGPT dataset, only 4 adjustments were made, but the end-to-end latency is comparable to the best static layer redistribution strategy.
4.5 Performance under Multi-Node Setting
20
gLLM
E2EL (s)
0.76x
DynaPipe
E2EL (8)
10
0.89
0.91x
0.77x
0.95x
0.93x
0.91x
0.82x
0.90x
0.85x
12
30
8
12
Request Rate (reqs/s)
Request Rate (reqs/s)
(a) ShareGPT
(b) Azure-Conv
Figure 7: The latency comparison between DynaPipe and gLLM on Qwen2.5-14B (4 nodes).
To emulate realistic cross-node deployment scenario, configure NCCL by setting NCCL_SHM S____SMMDDSSABBEEEII PCCC__IPPPP..SSAALEEI11 and NCCL_ DISAALL==1, NCCI_II_DDSABLLE-T thereby dis- abling shared memory, peer-to-peer, and InfiniBand communication backends. This setup forces all inter-GPU communication to through the TCP stack, effectively simulating network latency and bandwidth characteristics distributed environments. All experiments are conducted on the Qwen- 14B model with four NVIDIA A100 GPUs. As shown in Figure 7, DynaPipe achieves significant performance improvements compared gLLM. Although cross-node communication introduces additional KV Cache migration overhead, which slightly narrows the optimization margin compared with the single-nod scenario, the overall performance remains at a high level. This demonstrates that the core mechanism of DynaPipe maintains strong effectiveness and scalability in cross-node settings.
4.6 Effect of Execution Time Predictor
Layar
Layer
Sample (A)
Sample (8)
Predict. (ms)
4
Predict. (mx)
5.0
2
2.5
0.0
2 Actual Time (ms) (b) Qwen2.5-32B
5
Actual Time (ms) (a) Qwen2.5-14B
Figure 8: Actual vs. predicted execution time distribution. "A/S" means Azure- Cone////an/Sr.....
We construct a synthetic dataset by randomly sampling input and output sequence lengths and measured the offline execution times of the 14B and 32B models fit the corresponding execution time predictors. In Figure 8, we evaluate the accuracy of the predictors for both layer execution time (Layer) and sampling execution time (Sample). The results show that the predicted values closely align with the y=x line, indicating a high consistency between the predicted and actual execution times. The average relative error for layer execution time 4.95%, while the error for sampling execution time is only 0.31%. This demonstrates that the predictors fitted for the same LLM maintain

---

high accuracy across different dataset scales, validating the generalization ability of our model. Furthermore, the predictor takes only 0.5 microseconds per prediction, with negligible overhead.
5 Related Work
LLM inference optimization. Recent system-level research has increasingly concentrated on enhancing the memory efficiency and throughput performance of LLM inference. Representative efforts include vLLM [21], which alleviates GPU memory fragmentation through the PagedAttention mechanism, and SGLang [22], which improves execution efficiency by exploiting structured execution and KV cache reuse. In addition, several studies [32, 33] employ performance modeling to guide scheduling strategies, thereby decoupling the prefill and decode phases to achieve higher GPU utilization. Within distributed inference settings, Llumnix [34] and Seesaw [35] further address load balancing through KV cache migration and adaptive re-sharding. Orthogonal these works, DynaPipe focuses on a previously underemphasized issue of stage imbalance in pipeline parallelism, which adaptively mitigate stage imbalance runtime and significantly improve overall inference efficiency.
Pipeline Parallelism. Extensive pipeline parallelism research in LLM training has primarily focused on communication latency optimization [36]. memory balancing [37, 38, 39], pipeline bubble mit- igation [38, 40, 41, 42], and activation checkpointing [37, 43]. Some studies have also identified inter-stage imbalance issues, with approaches such as Skywork-MoF: [42] and Megatron-L.M [44] employing static layer redistribution alleviate this problem. However, such static strategies often prove less effective under the dynamic workload conditions that are typical in model serving. In addition, [41] proposes Vocabulary Parallelism method designed for training scenarios to mitigate bubbles caused by sampling overhead during training. Nevertheless, applying this method directly to inference requires stricter synchronization, which significantly increases system design complexity and synchronization overhead, making it less suitable for efficient serving deployment. In LLM serving, several studies [19, 18, 26] have explored pipeline parallelism by tuming micro-batch size and balancing workload to reduce pipeline bubbles. However, these approaches have not fully addressed the inter-stage imbalance caused by uneven sampling overhead. DynaPipe complements these efforts by dynamically adjusting layer distribution based on real-time workload conditions, triggering redistribution only when significant changes occur. This design maintains relatively low communication overhead and significantly improves the overall efficiency of inference serving.
6 Limitation and Future Work
This work employs deliberate design trade-offs to improve overall system performance. Although the optimized runtime KV cache migration still incurs some communication overhead, it only results in minor local performance fluctuations, while overall system performance remains improved. In addition, to support potential future layer reallocations, extra GPU memory must be reserved for storing incoming weights, which reduces memory utilization efficiency. To address these issues, future work may incorporate compression techniques to reduce transfer overhead and leverage offloading mechanisms minimize memory waste, thereby improving overall system performance. Furthermore, integrating vocabulary parallelism [41] into the inference pipeline is also a promising direction for future exploration.
7 Conclusion
In this paper, we investigate a significant but insufficiently explored aspect: the pipeline inter-stage bubble problem caused by sampling operations. We propose DynaPipe, novel runtime dynamic layer redistribution scheme to address this challenge effectively. By dynamically adjusting the computational load across layers, DynaPipe achieves balanced task distribution among pipeline stages, effectively aligning the pipeline and mitigating inter-stage imbalance. Compared to of the-art pipeline inference frameworks, DynaPipe delivers significant performance improvements.
10

---

Acknowledgements
We sincerely thank the anonymous reviewers for their valuable comments and suggestions, which have greatly helped improve this work. This research is supported by the National Key R&D Program of China (Grant No. 2033YF3000202), the NSPC grants (#62472462, #62461 146204), and is sponsored by CCF-Tencent Rhino-Bird Open Research Fund (CCF-Tencent RAGR20240102).
References
[1] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. D. Liu, F. Huang. H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang. Yang. J. Zhou, Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Su, Zhang, Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu, "Qwen2.5 technical report," CoRR. vol. abs/2412.15115, 2024.
[2] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang. A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev,
A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Rozière, B. Biron,
B. Tang, B. Chern, C. Caucheteux, C. Nayak. C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, M. Kloumann, I. Misra, I. Evtimov, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, Lee, Fu, J. Chi, Huang, Liu, J. Wang. J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, and et al., "The Ilama herd of models," CoRR, vol. abs/2407.22188. 2024.
E. Lakomkin, E. AlBadawy,
[3] OpenAI, "GPT-4 technical report," CoRR, vol. abs/2303.08774, 2023.
[4] Y. Deng, X. Zhang, W. Zhang, Y. Yuan, S. Ng, and T. Chua, "On the multi-turn instruction following for conversational web agents," Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 (L. Ku, A. Martins, and V. Srikumar, eds.), pp. 9795-8812, Association for Computational Linguistics, 2024.
[5] A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang. "Evaluating very long- term conversational memory of LLM agents," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 (L. Ku, A. Martins, and V. Srikumar, eds.), pp. 13851-13870, Association for Computational Linguistics, 2024.
[6] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber, "Metagpt: Meta programming for multi-agent collaborative framework." " in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024.
[7] "Bing copilot." https://www.bing.comm com/cchtt 2024.
[8] Z. Liu, W. Ping, R. Roy, P. Xu, C. Lee, M. Shoeybi, and B. Catanzaro, "Chatqa: Surpass- ing GPT-4 on conversational QA and RAG," in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024 (A. Globersons, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, eds.), 2024.
[9] X. Jiang, R. Qiu, Y. Xu, W. Zhang, Y. Zhu, R. Zhang, Y. Fang, C. Xu, J. Zhao, and Y. Wang, "Ragraph: A general aeureetal-ugmmeneed graph learning framework," " in Advances Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December sons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, eds.). 2024.
15, 2024 (A. Glober-
11

---

[10] D. Ru, L. Qiu, X. Hu, T. Zhang, P. Shi. S. Chang, C. Jiayang, C. Wang, S. Sun, H. Li, Z. Zhang. B. Wang, J. Jiang, T. He, Z. Wang, Liu, Y. Zhang, and Z. Zhang, "Ragchecker: A fine-grained framework for diagnosing retrieval-augmented generation," in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024 (A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet. J. M. Tomozak, and C. Zhang, eds.), 2024.
[11] C. Lin, Z. Han, C. Zhang, Y. Yang, F. Yang, C. Chen, and L. Qiu, "Parrot: Efficient serving of llm-based applications with semantic variable," in 1 18th USENIX Symposium on Operating Systems Design and Implementation. OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024 (A. Gavrilovska and D. B. Terry, eds.), Pp. 929 945, USENIX Association, 2024.
[12] J. Stojkovic, C. Zhang, f. Goiri, E. Choukse, H. Qiu, R. Fonseca, J. Torrellas, and R. Bian- chini, "TAPAS: thermal- and power-aware scheduling for LLM inference in cloud plat- forms," in Proceedings of the 30th ACM International Conference for Programming Languages and Operating Systems, Volume 2. ASPLOS 2025, Rotterdam, Netherlands, 30 March 2025 April 2025 (L. Eeckhout, G. Smaragdakis, K. Liang, A. Samp- son, M. A. Kim, and C. J. Rossbach, eds.), pp. 1266-1281, ACM, 2025.
Architectural Support
[13] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, "Distserve: Dis- aggregating prefill and decoding for goodput-optimized large language model serving," in 18th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024 (A. Gavrilovska and D. B. Terry, eds.), pp. 193-210, USENIX Association, 2024.
[14] R. Qin, Z. Li, W. He, J. Cui, F. Ren, M. Zhang, Y. Wu, W. Zheng, and X. Xu, "Mooncake: Trading more storage for less computation A kveache-centric architecture for serving LLM chatbot," " in 23rd USENIX Conference on File and Storage Technologies, FAST 2025, Santa Clara, CA. February 25-27, 2025 (H. S. Gunawi and V. Tarasov, eds.), pp. 155-170, USENIX Association, 2025.
[15] A. K. Kamath, R. Prabhu, J. Mohan, S. Peter, R. Ramjee, and A. Panwar, "Pod-attention: Un- locking full prefill-dcoode overlap for faster LLM inference," in Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ASPLOS 2025, Rotterdam, Netherlands, 30 March 2025 3 April 2025 (I.. Eeckhout, G. Smaragdakis, K. Liang, A. Sampson, M. A. Kim, and C. J. Rossbach, eds.), pp. 897-912, ACM, 2025.
[16] H. Dong, T. Johnson, M. Cho, and E. Soroush, "Towards low-bit communication for tensor parallel LLM inference," CoRR, vol. abs/2411.07942 2024.
[17] Q. Li, Zhang, L. Ye, Y. Zhang. W. Wu, Y. Sun, L. Ma, and Y. Xie, "Flash communication: Reducing tensor parallelization bottleneck for fast large language model inference," CoRR. vol. abs/2412.04964, 2024.
[18] T. Guo, X. Zhang, J. Du, Z. Chen, N. Xiao, and Y. Lu, "gllm: Global balanced pipeline parallelism system for distributed Ilm serving with token throttling," 2025.
[19] A. Agrawal, A. Panwar, J. Mohan, N. Kwatra, B. S. Gulavani, and R. Ramjee, "SARATHI: efficient LLM inference by piggybacking decodes with chunked prefills," CoRR, vol. abs/2308. 2abb//308..6669, 2023.
[20] J. Du, H. Zhang. T. Wei, Z. Zheng, K. Wu, Z. Chen, and Y. Lu, "Ecoserve: Enabling cost- effective 1lm serving with proactive intra- and inter-instance orchestration," 2025.
[21] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with pagedattention." in Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023 (J. Flinn, M. I. Seltzer, P. Druschel, A. Kaufmann, and J. Mace, eds.), pp. ACM, 2023.
12

---

[22] L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gon- zalez, C. W. Barrett, and Y. Sheng. "Sglang: Efficient execution of structured language model programs," in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024 (A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, eds.), 2024.
[23] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. December 4-9, 2017, Long Beach, CA, USA (I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, eds.), pp. 5998-6008, 2017.
[24] J. Gu, K. Cho. and V. O. K. Li, "Trainable greedy decoding for neural machine transla- tion," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 1, 2017 (M. Palmer, R. Hwa, and S. Riedel, eds.), pp. 1968-1975
Association for Computational Linguisties, 2017.
[25] S. Lemons, C. L. López. R. C. Holte, and W. Ruml, "Beam search: Faster and monotonic," in Proceedings of the Thirty-Seconi International Conference on Automated Planning and Scheduling, ICAPS 2022, Singapore (virtual), June 13-24, 2022 (A. Kumar, S. Thiébaux, Varakantham, and W. Yeoh, eds.), pp. 222 230, AAAI Press, 2022.
[26] H. Zhang, T. Wei, Z. Zheng, J. Du Z. Chen, and Y. Lu, "Td-pipe: Temporally- disaggregated pipeline parallelism architecture for high-throughput LLM inference," CoRR, vol. abs/2506. 10470, 2025.
[27] ZeroMQ authors, "ZeroMQ: An open-source universal messaging library."
[28] NVIDIA Corporation, "NCCL: Nvidia collective https//ggthub.co////iDDDMMM 2025.
communications
library."
[29] N. Corporation, "Nvidia a100 tensor core gpu architecture." http://reeoorrcsssiiiaaaccaoooma
httplissssppammpeerareettteeeeee white white-paper, paper, 2021.
[30] S. Teams., "Share your wildest chatgpt conversations with one click.," 2023.
[31] P. Patel, E. Choukse, C. Zhang, A. Shah, f. Goiri, S. Maleki, and R. Bianchini, "Splitwise: Effi- cient generative LLM inference using phase splitting," " in 51st ACM/EEE Annual International Symposium Computer Architecture, ISCA 2024, Buenos Aires, Argentina, June 29 July 3. 2024, pp. 118-132, IEEE, 2024.
[32] Z. Lin, H. Xu, G. Chen, X. Zhang, and Y. Lu, "Bullet: Boosting gpu utilization for Ilm serving via dynamic ppatial-temporal orchestration," 2025.
[33] W. Cui, Y. Chen, H. Zhao, Z. Xu, Q. Chen, X. Chen, Y. Zhou, S. Sun, and M. Guo, "Optimizing slo-oriented IIm serving with pd-multiplexing." 2025.
[34] B. Sun, Z. Huang, H. Zhao, Xiao, X. Zhang, Y. Li, and W. Lin, "Llumnix: Dynamic scheduling for large language model serving." in 18th USENIX Symposium Operating Systems Design and Implementation, OSDI 2024, Santa Clara, CA, USA, July 10-12, 2024 (A. Gavrilovska and D. B. Terry, eds.), pp. 173-191, USENIX Association, 2024.
[35] Q Su, W. Zhao, X. Li, M. Andoorveedu, C. Jiang, Z. Zhu, K. Song, C. Giannoula, and G. Pekhimenko, "Seesaw: High-throughput LLM inference via model re-sharding." CoRR, vol. abs/2503.06444 066333, 2025.
[36] Lin, Z. Liu, Y. You, Wang, W. Zhang, and R. Zhao, "Weipipe: Weight pipeline paral- lelism for communictiion-efeetiie long-context large model training." in Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP 2025, Las Vegas, NV. USA. March 1-5, 2025, pp. 225-238, ACM, 2025.
13

---

[37] Z. Sun, H. Cao, Y. Wang, G. Feng, S. Chen, H. Wang, and W. Chen, "Adapipe: Optimizing pipeline parallelism with adaptive recomputation and partitioning." " in Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS 2024, La Jolla, CA, USA, 27 April 2024- I May 2024 (R. Gupta, N. B. Abu-Ghazaleh, M. Musuvathi, and D. Tsafrir, eds.), pp. 86-100, ACM, 2024.
[38] Z. Liu, S. Cheng, H. Zhou, and You, "Hanayo: Harnessing wave-like pipeline parallelism for enhanced large model training efficiency," in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2023, Denver, CO, USA, November 12-17, 2023 (D. Arnod, R. M. Badia, and K. M. Mohror, eds.), pp. 56:1-56:33, ACM, 2023.
[39]
T. Kim, H. Kim. G. Yu, and B. Chun, "Bpipe: (emory-balancee pipeline parallelism for training large language models," in International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and Scarlett, eds.), vol. 202 of Proceedings of Machine Learning Research, pp. 16639-16653, PMLR, 2023.
[40] P. Qi, X. Wan, G. Huang, and M. Lin, "Zero bubble (almost) pipeline parallelism," in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OvenReview.net, 2024.
[41] M. T. Young, P. Qi, M. Lin, and X. Wan, "Balancing pipeline parallelism with vocabulary parallelism," CoRR, vol. abs/2411 1.. .05288,, 2024.
[42] T. Wei, B. Zhu, L. Zhao, C. Cheng, B. Li, W. Lü, P. Cheng, J. Zhang, X. Zhang, L. Zeng, X. Wang, Y. Ma. Hu, S. Yan, H. Fang, and Y. Zhou, "Skywork-moe: deep dive into training techniques for cexture-of-experee language models," CoRR, vol. hass22406.06663 2024.
[43] W. Liu, M. Li, G. Tan, and W. Jia, "Mario: Near zero-cost activation checkpointing in pipeline parallelism," Proceedings of the 30th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, PPoPP 2025, Las Vegas, NV, USA, March 1-5, 2025, pp. 197-211, ACM, 2025.
[44] D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, D. Vainbrand, P. Kashinkunti, J. Bernauer, B. Catanzaro, A. Phanishayee, and M. Zaharia, "Efficient large- scale language model training on GPU clusters using megatron-Im, in International Conference for High Performance Computing. Networking, Storage and Analysis, SC 2021, St. Louis, Missouri, USA, November 14-19, 2021 (B. R. de Supinski, M. W. Hall, and T. Gamblin, eds.), p. 58, ACM, 2021.
14

---

Appendix
A Performance Evaluation on Other Models
gLLM
DynaPipe
10-
100-1
ShanCPt
R2EL (0)
SLO AL. (%))
1.0s
S1.0 AL.
30
59
6.71
12
12
16
30
18
24
180
1,00x
1.0x
1.8x
1.1%
6:14
Azure Corv
12E1-0-
SL. (%6)
SLO (56)
50-
69
10
12
15
Request Rate (raquls)
Request Rate (roquis)
Request Rate (raqul)
Request Rate (roqu))
Qwen3-300-A3B
(b) Mette_llmm.---l--aaarrre
Figure 9: Performance comparison on MoE and Dense LLM models. DynaPipe consistently outperforms the baseline in both average latency and SLO compliance across all workloads.
As shown in Figure 9, DynaPipe consistently outperforms the baseline on both MoE and dense LLM models. The experiments are conducted single node equipped with four NVIDIA A100 40GB GPUs. The results reveal that, under pipeline parallelism, the inter-stage load imbalance caused by sampling overhead is pervasive issue, regardless of the underlying model architecture. By dynamically detecting and adapting to such imbalance, DynaPipe effectively accelerates the inference process across different models. This further demonstrates the strong generality and scalability of the proposed method in a wide range of model scenarios.
B Analysis of System Overhead
Migration Overhead Analysis In our system, dynamic layer redistribution inevitably introduces certain amount of KV cache migration overhead. Since the KV cache is transferred the layer gran- ularity, the communication cost of each migration remains relatively fixed under PCIe interconnect conditions. Meanwhile, the main latency model forward computation strongly correlated with the batch size: when the batch size is large, we overlap KV cache migration with part of the forward computation, effectively hiding most of the communication latency; when the batch size is small, although the overlap is less complete, the additional delay of single layer migration still be kept within 100 ms. Furthermore, when deployed in environments with high-bandwidth interconnects such NVLINK, this overhead can be further reduced significantly.
To mitigate the disturbance caused by frequent migrations, we introduce a hindow-threshee mech- anism into the scheduling policy. The system does not trigger redistribution upon every minor fluctuation in the ratio between sampling latency and single-layer forward computation cost. Instead, the signal is accumulated over sliding window, and redistribution is only executed when the accumu- lated value exceeds a predefined threshold. This design effectively avoids oscillations near decision boundaries. For instance, when the Lampling-to-forwant ratio approaches the boundary between "adjusting 2 layers" and "adjusting layers." the mechanism prevents frequent switching between these two configurations, thereby avoiding redundant migrations and unnecessary costs. Moreover, in this boundary region, the performance difference between the two configurations is inherently minimal. Experimental results show that triggering redistribution only 5-7 times is sufficient to achieve an excellent balance between performance and stability.
Memory Overhead Analysis. To accommodate potential layer redistributions during runtime, we preemptively load the weights of layers that may used in the future, which inevitably leads to certain degree of GPU memory overhead. Taking the Qwen2.5-32B model example: under a static allocation strategy, each GPU typically assigned 16 Transformer layers, whereas after dynamic redistribution, some GPUs may need to load up to 19 layers. For 40GB A100 GPU, this
15

---

corresponds to approximately 7.5% increase in memory usage. Although this adds some memory pressure, the resulting performance improvement significant, reflecting a deliberate trade-off between system performance and memory resource utilization. More importantly, this memory overhead can be further optimized in the future. By introducing an asynchronous weight offloading and loading mechanism, the weights of layers that are not on the current computation path can be temporarily stored in CPU memory. When the scheduler anticipates an upcoming layer redistribution, the system asynchronously preloads the corresponding weights onto the target GPU in advance. This mechanism allows us to maintain high performance while significantly mitigating memory redundancy caused by dynamic layer redistribution.
C Performance Benefit Analysis.
Within the inference pipeline of LLM, the sampling stage typically introduces additional process- ing time after each microbatch's forward computation, resulting in pipeline bubbles. Under high concurrency, these bubbles accumulate continuously and significantly increase the queuing delay of subsequent requests. As illustrated in Fig. bubbles progressively postpones the scheduling of subsequent microbatches into the pipeline. Let the duration of single sampling bubble be t.. If completing the prefill for the first n requests requires microbatches, then the (n + 1) request must wait at least æt, of additional time before it can start execution. Moreover, during decoding, each forward step introduces an extra (PP size 1) sam- pling bubbles, further increasing the overall inference latency. In contrast, with layer redistribution optimization, bubble accumulation can be effectively mitigated, allowing new requests to enter the system more rapidly and reducing e2e latency.
the baseline system, the accumulation of sampling
The performance advantages of DynaPipe are most pronounced in scenarios with small models and high request rates. This phenomenon arises primarily because the per-step forward computation time of small models is short, whereas the sampling latency remains largely constant. Consequently, the relative contribution of sampling to total latency increases, thereby magnifying the effect of pipeline idle time on overall system performance.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction elearly reflect the paper's contributions, outlining the primary research goals and results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The work focuses solely pipeline parallelism, without addressing the combined optimization of tensor, sequence, and expert parallelism, and plans to extend to multi-dimensional parallelism in future research.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
The paper provides complete proofs for both the complexity analysis and the modeling.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include the benchmark settings and hyperparameter configurations neces- sary for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use only publicly available datasets, and the code will be open-soureed after the paper accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
onooooiy))yy))looloooolmmoommlpl for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /ttpppcc/ppulliieee htt//uubicggguue heatiicgguie////eee haaps//CoSSSuumiisce seibbbmmssiooooooccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Our paper includes these details the evaluation section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: While the experimental results demonstrated high stability (with repeated trials of selected experiments confirming consistent outcomes), comprehensive repetition was infeasible due to prohibitively long computation times required for each complete experimental run.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The experimental section comprehensively documents all computational resources used, including hardware specifications and runtime metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurIPS Code of Ethics https: //nerrips/// httppp/licc/thhcssiieeeeeeeeeeee
didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The research fully complies with all provisions of the NeurIPS Code of Ethies in every aspect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This study improves the inference speed of large language models.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
This study utilizes only publicly available, low-risk academic datasets and does not involve any high-risk models or data that could potentially be misused or have dual-use applications.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All third-party resources used in this study have been properly cited in their re- spective sections, with clear attribution of sources, version numbers, and license information (including MIT License and CC-BY 4.0).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
21

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. comdattaeess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This study did not create any new datasets, models, code repositories.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This study utilizes publicly available datasets for analysis and does involve any experiments with human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for rescarch with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This study does not involve human subjects or crowdsourcing experiments, therefore IRB approval was not required.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No LLM tools were used in this study.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: hattps://neurips //nerrrrps.... wwtp//c/////onnnceeccec htt.:/pp..c.co...eeeeeee//////// for what should or should not be described.
23


Reviews:
[{"review_id": "v67skgDXwJ", "reviewer": "Reviewer_icHW", "summary": "This paper identifies the inefficiency caused by costly sampling operation in LLM inference. Specifically, in pipeline parallel inference, the last sampling step takes more time on computation, thus creating idle bubble in the pipeline stages. The authors present empirical evidences to show the non-negligible impact of sampling operation, comparing with the Transformer layer and argued that traditional approach which splits the Transformer layers evenly is efficiently. They thus propose DynaPipe which relies on a latency predictor to adaptively adjust the layer distribution among devices to achieve a better balanced computation. Together with a KV cache migrator, DynaPipe can serve the requests with much lower latency.", "questions": "1. Figure 2(b)'s experiments looks confusing. For 2(b), what is the layer execution time here? I assume for a specific token size, the per-layer execution time should be similar. It also applies to the sampling time. Why the ratio would be different? Is the token size changing in this Figure? And is it a reasonable setting to compare the sampling time with only one layer computation? In the evaluation, Qwen 32 B has 64 layers, and each GPU would hosts 16 Transformer layers in a even setup. I think it is fairer to compare the sampling time with a real per-device execution time.\n2. The siginificance of Figure 2(a) and Figure 2(b) are also unclear. In a computation-bound setup, it is common knowledge that the duration grows linearly with the size of the tokens. I don't see a connection with the paper's idea.\n3. Pipeline parallelism can be used in multi-node setting due to its lighter communication burden compared with Tensor parallelism. Is there any results on a larger-scale setting, like 4 nodes?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The authors identify an interesting problem in LLM serving. Most of the existing work focuses on accelerating the computation in Transformer layer. Sample indeed is a inevitable step in inference. This problem is more critical in generative LLM.\n2. DynaPipe proposes a clear design to efficiently adjust the layer distribution across devices to achieve a more balanced states.\n# Weakness \n1. Some of the experiments results require further clarification. See questions below. \n2. The evaluation could be improved with a larger scale setup.", "was_revised": true, "final_justification": "My concerns are properly addressed. I willl maintain my score.", "rebuttal": "Thank you for your valuable comments. We appreciate your feedback and would like to address your concerns as follows.\n\n**Q1:**\n\nTo begin, we would like to clarify some background details. Our experiments adopt the chunked prefill strategy, which allows prefill and decode requests to be mixed within a single batch and processed jointly during inference. In this setting, the sampling overhead primarily depends on the number of decode requests within the batch, while the forward execution time of each Transformer layer depends on the total number of tokens to be processed (including both prefill and decode tokens).\n\nWe now explain the motivation behind Figure 2, which serves to justify the need for dynamic layer redistribution, as proposed in our system.\n\n**(1) Why does the ratio between sampling time and per-layer forward time vary?**\n\nFigure 2(b) aims to quantify the relative cost of the sampling operation. Specifically, under a realistic inference workload, this figure shows a distribution of the ratio between sampling latency and the forward pass time of a single Transformer layer. From empirical observation, this ratio averages around 3.09, but is not constant—it varies dynamically across different time intervals and workload conditions, ranging approximately from 1 to 5.\n\nThe reasons for this variability include:\n\n- Sampling overhead increases with the number of decode requests, as illustrated in Figure 2(a). This overhead is determined solely by the number of decode tokens $n_d$ in a batch, since only decode requests require sampling operations.\n- The per-layer forward time grows with the total number of tokens, including both prefill and decode tokens, as shown in Figure 2(c). That is, the forward time depends on $n_p + n_d$, where $n_p$ is the number of prefill tokens and $n_d$ is the number of decode tokens.\n- The total token count $n_p + n_d$ per batch is not fixed in real-world serving scenarios. Since requests arrive dynamically, the ratio $n_d / (n_p + n_d)$ fluctuates over time. For example, at the beginning, when there are mostly prefill tokens and few decode tokens, sampling overhead is minimal. However, in later stages—if no new prefill requests arrive and the batch consists primarily of decode tokens—sampling overhead becomes dominant.\n\nGiven these factors, the ratio between sampling time and per-layer forward time varies dynamically during inference. This non-stationarity in workload characteristics renders static layer allocation suboptimal for maintaining balanced pipeline stage runtimes, and therefore motivates DynaPipe’s dynamic layer redistribution mechanism.\n\n**(2) Definition of “layer execution time” in Figure 2(b), and rationale for using a per-layer unit instead of per-device execution time:**\n\nIn Figure 2(b), the term “layer execution time” refers to the average forward time of a single Transformer layer under current batch conditions.\n\nSince all layers share the same architecture and process the same batch of tokens, the forward time per layer remains uniform under consistent conditions. Thus, we choose the per-layer execution time as a standardized unit for modeling and reasoning about the relative cost of sampling.\n\nMoreover, the per-layer unit facilitates scheduling decisions in our system: specifically, DynaPipe's bubble-aware scheduler estimates how many layers must be removed from the final pipeline stage to offset the sampling latency. For example, if the sampling delay is equivalent to four layers of forward computation, we can offload three layers from the final stage and redistribute them upstream, achieving better load balance.\n\nThis modeling approach is not only intuitive and lightweight, but also simplifies the logic for runtime adaptation. We emphasize that quantifying one layer versus multiple layers is fundamentally equivalent, with the primary difference being the granularity of expression. Therefore, the design choice in Figure 2(b) is meant to support scheduling formulation, and does not constitute an unfair or misleading comparison.\n\n**Q2:The siginificance of Figure 2(a) and Figure 2(b) are also unclear. In a computation-bound setup, it is common knowledge that the duration grows linearly with the size of the tokens. I don't see a connection with the paper's idea.**\n\nThe purpose of Figure 2(a) is to show how the sampling overhead varies with the number of sampled tokens, while Figure 2(c) illustrates the relationship between the single-layer forward computation time and the number of tokens to be processed.  These figures reflect the concrete trends in the overheads of both computation stages under different workload conditions, providing a data-driven foundation for building accurate performance prediction models.\n\nSince the overheads of both stages vary with workload, Figure 2(b) further presents the distribution density of their time ratio, showing that the relative cost between sampling and single-layer forward computation is not constant, but fluctuates within a dynamic range.  This variability directly motivates our design of a dynamic layer redistribution strategy.\n\n**Q3:Pipeline parallelism can be used in multi-node setting due to its lighter communication burden compared with Tensor parallelism. Is there any results on a larger-scale setting, like 4 nodes?**\n\nPipeline parallelism generally offers better scalability in multi-node environments due to its lower communication overhead compared to tensor parallelism. In our experiments, we conducted evaluations on a single-node PCIe setup, where the communication cost of tensor parallelism is relatively low. Nevertheless, even under such conditions that favor tensor parallelism, our pipeline-based approach still demonstrates superior performance.\n\nDue to hardware constraints, we were unable to conduct experiments on real multi-node clusters (e.g., 4 nodes). However, we performed a simulated multi-node experiment on a single machine with four GPUs by disabling NCCL's P2P, shared memory, and IB communication to emulate the communication overhead of cross-node setups. Under these simulated conditions, our approach still achieved promising results. For instance, under simulated multi-node settings on the Qwen-32B model and ShareGPT dataset, our DynaPipe system consistently reduced end-to-end latency by 10%–20% compared to gLLM across request rates of 12, 18, and 24, despite an overall increase in latency due to the emulated communication overhead.\n\nIt is worth noting that the original gLLM paper has already demonstrated its superiority over tensor parallelism in real multi-node environments. Given that our DynaPipe outperforms gLLM in simulated multi-node settings, we have strong reason to believe that it would deliver even better performance than tensor parallelism in real-world multi-node deployments."}, {"review_id": "QBEgGO0IgB", "reviewer": "Reviewer_Zxmv", "summary": "This paper proposes DynaPipe, a system that optimizes pipeline parallel inference by dynamically redistributing layers between stages. The authors observe the sampling (occurring at the last stage of the pipeline) causes stage imbalance. The authors propose to reduce the number of layers in the last stage (and redistribute them to previous stages) and improve pipeline balance. The optimal redistribution is determined by a performance model, while asynchronous pre-transferring of KV cache is employed to reduce redistribution overheads.", "questions": "— In Fig.2, it seems both sampling and layer computation time grows near linearly with the number of tokens. What is the source of the variance of sample / layer time (Fig. 2b)? Is it because sampling and layer time grow at a different rate?\n\n— What is the model used in Fig.2? Is the sampling time mainly related to vocabulary size (and the number of tokens to be decoded), rather than model size? If so, will the sampling time become negligible if the model is large enough?\n\n— How does the sample-layer time ratio change during real serving scenarios? The claim that dynamic adjustments are needed can be better illustrated if authors can show the variance of sample-layer time ratio over a time interval, under realistic settings.\n\n—The paper assumes chunked prefill is used. However, production inference services often adopt disaggregated serving (e.g., [3]), where prefill and decoding happen on different GPUs. Does the analysis and the approach proposed in this paper still apply?\n\n—In Sec.3.3, it is stated that \"the system pre-allocates memory and loads the weights for additional layers that may be reassigned\". How many layers are loaded in advance? Will this cause extra memory consumption?\n\n—In Sec.4.3, around line 281-282, why output length is related to sampling overhead? Is it because it takes more time to sample each token, or it's because there's more decoding tokens (rather than prefill)?\n\n—What is the layer partitioning strategy used in the gLLM baseline? While static partitioning is used as baselines in Fig.5, it would be nice if static partitioning (with uneven layer distribution) is compared to in end-to-end evaluation on real datasets, to better justify the need for dynamic adjustments.\n\n[3] https://arxiv.org/pdf/2401.09670", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n 1. The paper makes an interesting observation that the sampling overhead can cause pipeline imbalance during inference.\n 2. The methods proposed in the paper are simple and easy to implement.\n \nWeakness:\n\n— The problem of pipeline imbalance when layers are uniformly partitioned is well known in training, often mitigated by partitioning the layers unevenly (e.g., sec. B.2 in Skywork-MoE[1], configuration option in Megatron-LM [2]). Other optimizations like asynchronous transfer of KV Cache are also standard techniques. Therefore, it would be better if the paper focuses more on justifying the need for dynamic justification of layers (which differentiates the paper from prior works).\n\n—It would be better to quantify the overhead of dynamic layer & KV cache redistribution in the evaluation section.\n \n[1] https://arxiv.org/pdf/2406.06563\n\n[2] https://github.com/NVIDIA/Megatron-LM/blob/45400df7da7fa23e3aff86804e5ac254d9a8d3c0/megatron/training/arguments.py#L2258", "was_revised": true, "final_justification": "The authors have addressed my major concerns satisfactorily.", "rebuttal": "Thank you for your valuable comments. We appreciate your feedback and would like to address your concerns as follows.\n\n**Q1:In Fig.2, it seems both sampling and layer computation time grows near linearly with the number of tokens. What is the source of the variance of sample / layer time (Fig. 2b)? Is it because sampling and layer time grow at a different rate?**\n\nFigure 2(a) illustrates that the sampling overhead grows linearly with the number of sampled tokens. The number of sampled tokens is determined by the number of decode requests in a batch, since only decode requests require sampling operations. Let the sampling overhead be denoted as $t_s$, and the number of decode tokens as $d_n$, then: $t_s = a \\cdot d_n$.\n\nFigure 2(c) shows that the forward execution time of a single Transformer layer increases with the total number of tokens to be processed, including both prefill and decode tokens. Theoretically, due to the quadratic complexity of the attention mechanism, the forward time should grow quadratically with respect to the token count. However, in practice, the attention component contributes only a small portion of the total execution time, with the majority dominated by MLP and other linear computations. As a result, the forward time empirically appears to scale linearly.\n\nThus, we simplify the forward cost per layer as a linear function. Let the forward time per layer be $f_t$, and the number of prefill tokens be $p_n$, then:$t_f = b \\cdot (p_n + d_n)$.\n\nConsequently, the ratio between sampling overhead and forward computation time becomes:$\\frac{t_s}{t_f} = \\frac{a}{b} \\cdot \\left( \\frac{p_n}{d_n} + 1 \\right)^{-1}$\n\nIn the chunked prefill setting, prefill and decode requests are mixed and processed jointly within a single batch. Since the number of decode and prefill tokens varies dynamically across batches due to changing workloads, the ratio $p_n / d_n$ also fluctuates over time. This leads to a dynamic change in the ratio between sampling and forward computation costs.\n\nTo address this challenge and adapt to the evolving workload, we introduce our dynamic layer redistribution design, which enables more efficient pipeline parallelism under mixed prefill/decode batch processing.\n\n**Q2:What is the model used in Fig.2? Is the sampling time mainly related to vocabulary size (and the number of tokens to be decoded), rather than model size? If so, will the sampling time become negligible if the model is large enough?**\n\nYour observation is absolutely correct: the sampling overhead is primarily determined by the vocabulary size and the number of tokens to be decoded, and is largely independent of the model's parameter scale. In very large models such as DeepSeek 671B, the forward computation dominates the total runtime, making the sampling cost relatively negligible.\n\nHowever, as shown in Figure 2, our experiments are conducted on the Qwen2.5-32B model, where the sampling overhead is already significant. More importantly, in many real-world deployment scenarios, particularly in on-device applications, extremely large models are not commonly used. Models such as 32B and 72B are widely adopted in production because they offer a favorable trade-off between performance and efficiency.\n\nAt these model scales, sampling can become a performance bottleneck. Therefore, addressing the sampling overhead remains practically important.\n\n**Q3：How does the sample-layer time ratio change during real serving scenarios? The claim that dynamic adjustments are needed can be better illustrated if authors can show the variance of sample-layer time ratio over a time interval, under realistic settings.**\n\nAs we discussed in our response to Q1, the ratio between sampling time and per-layer computation time dynamically changes according to the proportion of prefill and decode tokens within each batch.\n\nWe will include a visualization of this ratio’s temporal variation in the final version of the paper.    Here, we provide a textual description of its general trend in real-world serving scenarios:\n\nIn the early stage, batches are dominated by prefill requests, with relatively few decode tokens.    As a result, the sampling overhead is small compared to the forward computation time.\n\nAs decoding progresses, the number of decode requests increases, causing the sampling overhead to constitute an increasingly larger proportion relative to the forward computation.\n\nIn the middle stage, due to the dynamic arrival of new requests in the serving environment, the number of prefill tokens fluctuates, causing the ratio to change dynamically.\n\nIn the final stage, as the batch consists solely of decode requests, the ratio reaches its maximum.\n\n**Q4：The paper assumes chunked prefill is used. However, production inference services often adopt disaggregated serving (e.g., [3]), where prefill and decoding happen on different GPUs. Does the analysis and the approach proposed in this paper still apply?**\n\nWe believe that the proposed method remains applicable — and even more critical — in serving architectures where prefill and decode are decoupled. This is because sampling overhead only occurs during the decode stage, allowing for targeted optimization of the decode cluster. Since the decode cluster no longer performs forward computation for prefill tokens, the relative cost of sampling becomes even more prominent, potentially turning into a significant performance bottleneck.\n\nMoreover, in real-world scenarios, especially when serving long text inputs, the computational cost of attention increases substantially. In such cases, the forward computation time exhibits a nonlinear, approximately quadratic relationship with the number of tokens. A simplified model would express the forward time as:$t_f = b \\cdot d_n^2$, while the sampling time remains:$t_s = a \\cdot d_n$, resulting in a ratio of:$\\frac{t_s}{t_f} = \\frac{a}{b} \\cdot \\frac{1}{d_n}$.\n\nThis indicates that the ratio of sampling to forward computation cost dynamically varies with the number of decode tokens. Therefore, even under disaggregated serving architectures, our proposed dynamic layer redistribution strategy remains essential for adapting to workload variations and optimizing overall inference performance.\n\n**Q5:In Sec.3.3, it is stated that \"the system pre-allocates memory and loads the weights for additional layers that may be reassigned\". How many layers are loaded in advance? Will this cause extra memory consumption?**\n\nOur current strategy preloads on each GPU the model weights that may be needed in the future. For example, in a 4-GPU setting, up to three layers may be dynamically reassigned, so each GPU loads an additional three layers of weights in advance. This strategy does introduce some extra GPU memory overhead, but it represents a trade-off between performance and resource consumption. Based on our experimental results, this approach significantly reduces end-to-end latency for individual requests.\n\nA similar strategy is also adopted in the deployment of DeepSeek, where redundant expert models are loaded on each GPU to improve overall system efficiency.\n\nMoreover, we are actively developing an optimization to reduce this additional GPU memory usage. Specifically, the redundant weights are preloaded into CPU memory. During runtime, a predictor anticipates potential layer reassignments and asynchronously transfers the required weights to the GPU while offloading the weights of unused layers. This mechanism retains the flexibility of dynamic scheduling while effectively lowering GPU memory consumption.\n\n**Q6:In Sec.4.3, around line 281-282, why output length is related to sampling overhead? Is it because it takes more time to sample each token, or it's because there's more decoding tokens (rather than prefill)?**\n\nThis is because, in a dataset with a longer average output length, each request undergoes more decoding steps. Under the chunked prefill mechanism, this leads to a higher number of decode requests accumulated within each batch. As the number of decode requests increases, the overall sampling overhead correspondingly grows.\n\n**Q7:What is the layer partitioning strategy used in the gLLM baseline? While static partitioning is used as baselines in Fig.5, it would be nice if static partitioning (with uneven layer distribution) is compared to in end-to-end evaluation on real datasets, to better justify the need for dynamic adjustments.**\n\nThe gLLM baseline adopts an even layer partitioning strategy.  We have also evaluated other static uneven partitioning strategies under real workloads, and Dynapipe consistently outperforms them.  This is because static strategies cannot adapt to the optimal layer distribution during runtime.\n\nConsidering the wide variety of static uneven partitioning schemes, as well as our comparisons against other systems such as vLLM and SGLang, including all baselines in a single figure would lead to visual clutter and hinder readability.  Therefore, we designed an ablation study in Figure 5, where we vary the input/output length ratio to simulate different prefill/decode workload distributions.  This setup indirectly reflects how different static partitioning strategies perform under various load conditions.  The results further demonstrate that Dynapipe consistently outperforms these methods, highlighting the necessity and advantage of dynamic layer adjustment.\n\nThis is particularly important in real-world serving environments, where the engine typically handles mixed workloads rather than a single dataset.  Requests from different applications arrive at different times with diverse input/output length patterns.  In such dynamic scenarios, Dynapipe can more effectively adapt the system to the current request distribution, delivering better and more stable performance."}, {"review_id": "fOMcRdqwzu", "reviewer": "Reviewer_oYmA", "summary": "This paper focuses on a significant but overlooked bottleneck in PP caused by sampling operations during LLM inference. It proposes a dynamic layer redistribution approach and an asynchronous KV cache migration technique to address this bottleneck. Experiments show that the proposed method can reduce the average latency and improve SLO attainment rate compared to existing systems.", "questions": "* The same problem has been well studied for LLM training in [1], where they propose to partition the sampling operations across PP devices and adapt PP scheduling. Compared to trivial layer redistribution, this method provides a unified solution under all configurations, with quite small overhead and balanced memory. Can similar ideas be applied in LLM inference as a better solution (to resolve the limitations mentioned in Weaknesses)? BTW, I would suggest to cite papers on this topic for LLM training  ([1] and potentially more) in Related Work. \n* From my understanding, the batch in an inference service is dynamic, especially for online inference where new requests can come at any time. How to guarantee the optimal solution under such dynamic settings?\n\n[1] Yeung, M. T., Qi, P., Lin, M., and Wan, X. Balancing pipeline parallelism with vocabulary parallelism. arXiv preprint arXiv:2411.05288, 2024.", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n* The target problem is valid and underexplored.\n* The paper is well written.\n* The evaluation is extensive and detailed.\n\n## Weaknesses\n* The proposed method introduces migration overhead (both parameters and KV cache), and this overhead can be huge in some configurations (totally different layers for some stages).\n* The paper lacks memory analyses. For example, layer redistribution will make KV cache imbalanced and may introduce memory overhead. \n* For many cofigurations, there still exists computational imbalance becuase layer redistribution cannot provide perfect solutions for all use cases.", "was_revised": true, "final_justification": "The topic is reasonable and the engineering effort is commendable. I am open to raising my score from 3 to 4, but only after the following points are addressed:\n* The authors should clearly state the limitations; a dedicated limitations section is encouraged.\n* There should be a thorough discussion of existing solutions for both training and inference to help readers better understand the broader context and alternative approaches.", "rebuttal": "Thank you for your valuable comments. We appreciate your feedback and would like to address your concerns as follows.\n\n**Q1: The same problem has been well studied for LLM training in [1], where they propose to partition the sampling operations across PP devices and adapt PP scheduling. Compared to trivial layer redistribution, this method provides a unified solution under all configurations, with quite small overhead and balanced memory. Can similar ideas be applied in LLM inference as a better solution (to resolve the limitations mentioned in Weaknesses)? BTW, I would suggest to cite papers on this topic for LLM training ([1] and potentially more) in Related Work.**\n\nWe will cite the paper *Balancing Pipeline Parallelism with Vocabulary Parallelism* in the related work section. However, we believe the technique proposed in that paper is primarily designed for LLM training scenarios and has certain limitations when applied to inference workloads.\n\nThe paper proposes a vocabulary parallelism approach that distributes the sampling operation, which is originally concentrated in the final pipeline stage, across all GPUs. This effectively amortizes the sampling overhead over all pipeline stages. While this method is effective during training, it largely benefits from the presence of backward propagation dependencies. These dependencies prevent intermediate pipeline stages from continuously executing forward passes, requiring them to wait for backward computations on certain microbatches to complete. As a result, idle periods (\"bubbles\") naturally arise in the pipeline, which can be utilized to perform sampling operations and improve resource efficiency.\n\nHowever, this assumption no longer holds in inference scenarios. First, since there is no backward propagation, each pipeline stage can continuously process forward passes of its assigned microbatches. Second, pipeline parallelism allows at most `pp_size` microbatches to be processed concurrently in the system. When the microbatch at the final stage has not completed its sampling operation, a new microbatch cannot be injected into the pipeline, causing the first stage to be blocked and generating front-end bubbles. Therefore, in inference, sampling operations must be completed as early as possible to ensure continuous and efficient pipeline execution.\n\nIntroducing vocabulary-parallel sampling during inference also poses challenges. Coordinating the sampling operation across all pipeline stages requires careful synchronization. In practice, due to the variability in forward pass durations across pipeline stages, such synchronization may actually delay the sampling completion. For example, suppose GPU3 finishes the forward pass of microbatch 0 and broadcasts a signal to all GPUs to perform the sampling operation. At that moment, GPU2 may have just started executing microbatch 3. It must finish the current forward pass before executing the sampling, resulting in a sampling delay of at least one microbatch's execution time. This delay would hinder the injection of new microbatches into the pipeline and degrade overall throughput.\n\nFurthermore, the vocabulary-parallel mechanism divides the output layer computation into a local computation phase (S) and a global normalization phase (T), where S must occur after the final Transformer layer, and T must wait for all S phases to complete. This introduces additional global communication and synchronization, which is particularly undesirable in inference workloads. It increases scheduling complexity, creates new performance bottlenecks, and introduces pipeline bubbles.\n\nIn contrast, our proposed Dynamic Layer Redistribution technique achieves load balancing without incurring the cost of global synchronization and communication, enabling more efficient pipeline-parallel execution for inference scenarios.\n\n**Q2: From my understanding, the batch in an inference service is dynamic, especially for online inference where new requests can come at any time. How to guarantee the optimal solution under such dynamic settings?**\n\nIndeed, in online inference services, the batch of incoming requests is inherently dynamic. In our experiments, we simulate this scenario by controlling the request arrival rate to mimic an online serving environment. Because requests arrive at different time points, the ratio of prefill to decode requests within each microbatch varies over time.\n\nWe adopt a chunked prefill mechanism, which allows prefill and decode requests to be mixed and executed together within a single microbatch. Under this mechanism, the execution time of each Transformer layer depends on the total number of tokens (prefill + decode), while the sampling overhead only depends on the number of decode requests. As a result, fluctuations in the prefill/decode ratio, driven by the dynamic arrival of new requests, lead to corresponding changes in the relative cost between sampling and single-layer forward computation.\n\nIn such a dynamic setting, static layer partitioning becomes suboptimal, as it cannot adapt to the shifting workload composition, often leading to pipeline bubbles and resource imbalance. To address this, we propose a dynamic layer redistribution mechanism that adjusts to real-time workload variations, effectively minimizing pipeline stalls and improving overall system throughput.\n\n**Response to Weaknesses:**\n\n（1）**Regarding the migration overhead**, the number of KV Cache layers that need to be transferred varies across different pipeline stages. As a result, in certain cases, the last GPU may incur significantly more migration traffic than the first GPU, introducing additional cost.\n\nTo mitigate this overhead, we adopt two key optimization strategies:\n\n1. Asynchronous communication-computation overlap: By overlapping KV cache migration with partial forward computation, we are able to hide part of the communication latency.\n2. Window threshold mechanism (see Section 4.4): Instead of triggering a layer redistribution every time the ratio between sampling cost and per-layer forward cost changes, we accumulate this signal over a sliding window. Only when it reaches a predefined threshold do we trigger a reallocation. This approach prevents frequent layer adjustments caused by transient or local fluctuations, thereby reducing unnecessary migration overhead.\n\nAs shown in Figure 6, when the window threshold is set to 25, the system achieves a good trade-off between performance gain and overhead, effectively suppressing the cost introduced by frequent layer reassignments.\n\nIn fact, the primary motivation for introducing the window threshold is to avoid frequent oscillation between boundary configurations. For example, when the sampling-to-forward ratio hovers near the decision boundary between adjusting 2 layers and adjusting 3 layers, the system may repeatedly switch back and forth—adjusting to 3 layers in one round and reverting to 2 layers in the next. Such frequent changes bring little performance benefit but lead to excessive KV cache migrations, increasing overhead instead.\n\nMore importantly, in this boundary range, the performance difference between adjusting 2 layers and adjusting 3 layers is negligible. Therefore, the window mechanism helps suppress unnecessary jitter, ensuring stable performance improvements while significantly reducing the cost associated with frequent KV cache movements.\n\n（2）**Regarding memory overhead**, dynamic layer redistribution does introduce additional GPU memory usage. However, from a system-wide perspective, this represents a trade-off between performance and memory consumption, as redistributing layers helps eliminate pipeline bubbles and improves overall throughput.\n\nTake Qwen-32B as an example: originally, each GPU holds around 15 transformer layers. After redistribution, this increases to 18 layers, resulting in approximately 16% more GPU memory usage. However, this overhead can be further optimized and mitigated.\n\nWe are currently implementing an enhancement that caches the surplus weights in CPU memory. When the scheduler detects the need for a layer redistribution, it preloads the required weights asynchronously from CPU to GPU, just before they are needed. This allows us to keep the number of resident weights on the GPU low without sacrificing performance, thus significantly reducing redundant memory usage.\n\n\n（3）**Generality**: While there are still other types of computational load imbalance in the system, this paper focuses specifically on the performance bottleneck caused by sampling overhead. We argue that this issue has a significant impact during the inference stage and thus deserves focused attention and targeted solutions.\n\nIt is also worth noting that our proposed approach is orthogonal to other load-balancing techniques, meaning it can be integrated with existing solutions to jointly address broader load imbalance challenges and further improve overall inference efficiency."}, {"review_id": "dKHc3NfO0y", "reviewer": "Reviewer_oJc6", "summary": "DynaPipe is an LLM serving framework that dynamically balances the workload across pipeline stages, by migrating KV cache at background. The evaluation observes significant advantages in performance and SLO guarantee.", "questions": "1. Can the authors quantify the impacts of post-processing on serving performance? or explain what are other factors lead to such huge (up to 49%) performance benefits? It seems that getting such benefits by just addressing the pipeline bubble caused by post-processing is too good to be true.\n\n2. Is the work compatible with novel pipeline execution patterns (e.g., DualPipe)?", "limitations": "The evaluation dataset is a bit limited (only on 2 smaller variants of Qwen). Maybe the authors can discuss more on what LLMs (e.g., small/large ones, MoE/dense ones, ...) are benefited more by DynaPipe", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\nPipeline bubbles caused by post-processing latency are a novelly identified issue.\n\nThe results (up to 49% performance improvement) seem too impressive.\n\nWeakness\n\nWhile bubbles caused by post-processing are a new issue, it is hard to see why addressing this small overhead can lead to such huge performance benefits.", "was_revised": true, "final_justification": "All my key concerns are solved in the rebuttal, but the evaluation benchmarks can still be improved (see limitation). The paper is solid, but addressing pipeline bubbles is not an entirely novel idea. Still, the good parts outweigh the parts that need improvement, so I am still positive about this paper.", "rebuttal": "Thank you for your valuable comments. We appreciate your feedback and would like to address your concerns as follows.\n\n**Q1：Can the authors quantify the impacts of post-processing on serving performance? or explain what are other factors lead to such huge (up to 49%) performance benefits? It seems that getting such benefits by just addressing the pipeline bubble caused by post-processing is too good to be true.**\n\nFirst, we have included complete source code in the submitted package, and all experimental results are fully reproducible.\n\nRegarding the observed performance improvements, our analysis is as follows:\n\nFirst, the pipeline bubbles caused by sampling overhead tend to accumulate over time and significantly impact the waiting time of subsequent requests. As shown in the upper part of Figure 1, in the baseline system, these bubbles gradually accumulate, causing a noticeable delay in when a given microbatch enters the system compared to the optimized pipeline adopted by DynaPipe. This delay depends on the extent of bubble accumulation—the longer the accumulation time, the greater the delay.\n\nFor example, suppose that at time $t_1$, the $(n+1)$-th request arrives. In DynaPipe, since the prefill phase for the previous $n$ requests has just completed, the new request can be seamlessly inserted into the system with virtually zero queuing delay. However, in the baseline system, if the prefill for the previous $n$ requests has not yet finished, and no computation resources are available, the new request must wait until the corresponding microbatch slot becomes available. Thus, the queuing time for this request equals the cumulative delay caused by all preceding sampling bubbles.\n\nAssume that it takes $n/2$ microbatches to complete the prefill for the previous $n$ requests, and that each microbatch execution introduces one sampling bubble with duration $t_s$. Then, the $(n+1)$-th request must wait at least $n/2 \\times t_s$ before it can begin execution. Furthermore, once the request starts decoding, each forward computation step still suffers from an additional $(pp\\_{size} - 1)$ sampling bubbles compared to DynaPipe, as illustrated in Figure 1. Therefore, both the queuing latency and the effective throughput are negatively impacted by the inter-stage idle time introduced by sampling.\n\nAs for why we observe performance gains as high as 49%, we explain this as follows:\n\nThe best-case improvement occurs when using the Qwen2.5-14B model at a request rate of 36. In this scenario, the model is relatively small, resulting in lower per-layer forward computation cost, whereas the sampling overhead remains the same. As a result, the ratio of sampling overhead to forward computation cost increases, making the impact of sampling more pronounced.\n\nMoreover, under high request rates (e.g., 36), the incoming requests quickly accumulate in the queue. In such cases, the queuing delay of at least $n/2 \\times t_s$ can become substantial. For requests with short output lengths, the queuing time may even exceed the actual inference time, drastically reducing system efficiency.\n\nTherefore, DynaPipe's optimization significantly mitigates the idle time caused by sampling bubbles and improves resource utilization during forward computation. This leads to up to 49% end-to-end performance improvement under specific but practically relevant settings.\n\n**Q2: Is the work compatible with novel pipeline execution patterns (e.g., DualPipe)?**\n\nDualPipe is an optimization method designed for large language model training, while our proposed DynaPipe focuses on performance optimization during the inference stage. This distinction arises because training involves not only forward computation but also backpropagation and parameter updates. If dynamic layer redistribution is applied during training, it may disrupt the alignment between the backward pass and the corresponding parameter layers, leading to incorrect or failed parameter updates. Therefore, DynaPipe's dynamic redistribution mechanism is not suitable for training scenarios. Its strengths lie in the inference stage, where it can dynamically adapt to runtime workload fluctuations and improve overall execution efficiency.\n\n**Response to Limitation:**\n\nWe acknowledge that the current evaluation is limited to two relatively small variants of the Qwen model, and the dataset scope is somewhat narrow. However, from a mechanism perspective, DynaPipe is generally applicable to LLMs of varying sizes and architectures.\n\nSpecifically, the overhead of the sample phase mainly depends on the vocabulary size and is largely independent of the model parameter scale. In contrast, the forward computation cost of each layer is closely tied to the model’s weight parameters. As a result, DynaPipe tends to yield more significant benefits for smaller models, while its effectiveness may decrease as model size grows. That said, in practical deployments, especially on edge devices, many use mid-sized models such as 14B or 32B, and our experiments demonstrate that DynaPipe achieves substantial performance gains on these models.\n\nMoreover, DynaPipe is effective for both dense and sparse (MoE) models. Our current experiments are based on dense models. In the case of MoE models, although the expert selection mechanism may lead to load imbalance, pipeline parallelism typically assigns all experts within a layer to the same GPU, thereby avoiding inter-expert imbalance. However, the sample operation is still executed on the final GPU stage, which introduces inter-stage pipeline bubbles. In such scenarios, DynaPipe can still be leveraged to mitigate the performance degradation through dynamic layer redistribution."}]

Author rebuttal:

Meta-review: Summary: The paper identifies a practically important bottleneck in pipeline-parallel LLM serving and presents a simple, implementable mechanism (dynamic layer redistribution with async KV migration and bubble-aware scheduling) that delivers consistent latency reductions, with reasonable overheads and clear engineering.

Strengths: 1. Sampling-time bubbles at the tail stage are real and under-addressed in serving stacks. Reviewers agree the observation is interesting/important.
2. Strong speedups (8–49%) with explanations for when/why gains peak; code provided and experiments reproducible

Weakness: 1. Novelty is incremental relative to training literature on uneven partitions and communication overlap; the contribution is primarily in porting/adapting these ideas to inference with runtime dynamics. 
2. Only two Qwen variants; reviewers requested more sizes and possibly MoE/dense diversity (authors agree to expand)

Decision: Works with common serving patterns (chunked prefill) and shows benefits even in simulated multi-node settings over a strong pipeline baseline (gLLM).  The paper surfaces a real bottleneck and shows a clean, deployable fix with consistent gains across rates/lengths.

Discussion Summary: Authors clarified per-layer forward time as a normalized unit for modeling/scheduling; the scheduler decides how many layers to shift based on this unit. They argued training not same as inference (backward phase, synchronization) and emphasized runtime dynamics; they promised more overhead breakdowns. Main concerns addressed. Not all points raised by oYma addressed on Dynapipe vs Vocabulary parallelism, but can be addressed through future works. Cross review concern on overheads: Authors quantified memory overhead (~10–15%; ~16% in Qwen-32B example) and explained how async overlap + window-threshold=25 restrain migration costs.
Decision: Accept (poster)