PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference
Authors: Jiarui Fang
Jinzhe Pan
Aoyu Li
Xibo Sun
WANG Jiannan
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Model
GPU
Sequence Parallel
Pipeline Parallel
Primary area: infrastructure

Abstract: This paper presents PipeFusion, an innovative parallel methodology to tackle the high latency issues associated with generating high-resolution images using diffusion transformers (DiTs) models.  PipeFusion partitions images into patches and the model layers across multiple GPUs. It employs a patch-level pipeline parallel strategy to orchestrate communication and computation efficiently.  By capitalizing on the high similarity between inputs from successive diffusion steps, PipeFusion reuses one-step stale feature maps to provide context for the current pipeline step. This approach notably reduces communication costs compared to existing DiTs inference parallelism, including tensor parallel, sequence parallel and DistriFusion. PipeFusion enhances memory efficiency through parameter distribution across devices, ideal for large DiTs like Flux.1. Experimental results demonstrate that PipeFusion achieves state-of-the-art performance on 8$\times$L40 PCIe GPUs for Pixart, Stable-Diffusion 3, and Flux.1 models. Our Source code is available at \url{https://github.com/xdit-project/xDiT}.

PipeFusion: Patch-level Pipeline Parallelism for
Diffusion Transformers Inference
Jiarui Fang *
Jinzhe Pan HUST
Aoyu Lit ByteDance
ByteDance [angjaarui1230@manr com
eigensystem@hust.o edu. en
aoyul12000@outloo..ccmm
Xibo Sun
Jiannan Wang
Tencent
The University of Hong Kong
xibosun@tencent.comm com
straunk@connect. hku
Abstract
This paper presents PipeFusion, an innovative parallel methodology to tackle the high latency issues associated with generating high-resolution images using diffusion transformers (DiTs) models. PipeFusion partitions images into patches and the model layers across multiple GPUs. It employs a patch-level pipeline parallel strategy orchestrate communication and computation efficiently. By capitalizing on the high similarity between inputs from successive diffusion steps, PipeFusion reuses one-step stale feature maps provide context for the current pipeline step. This approach notably reduces communication costs compared to existing DiT's inference parallelism, including tensor parallel, sequence parallel and DistriFusion. PipeFusion enhances memory efficiency through parameter distribution across devices, ideal for large DiTs like Flux.1. Experimental results demonstrate that PipeFusion achieves state-of-the-art performance on L40 PCIe GPUs for Pixart, Stable- Sibbl--Diffuuion 3, and Flux. 1 models. Our Source code is available at htttt:: //gi::///itte thub. com/ xdit ceatt-projectxtil
1 Introduction
In recent years, the capabilities of Al-generated content (AIGC) have rapidly advanced, with diffusion models emerging as a leading generative technique in image 1| and video synthesis (566) The model architecture of diffusion models is undergoing significant transformation. Traditionally dominated by U-Net 7 architectures, these models are now transitioning to Diffusion Transformers (DiTs) 4. The context length of diffusion models is also increasing, which is crucial for handling long videos and high-resolution images.
DiTs suffer from quadratic latency growth in long-sequence generation due to attention computation. Given that a single GPU cannot satisfy the latency requirements for practical applications, becomes necessary to parallelize the DiTs inference for single image across multiple computational devices. However, tensor parallelism (TP) X commonly used in serving Large Language Models (L.LMs), inefficient for DiT's due to their large activation volumes. In such cases, the communication costs often outweigh the benefits parallel computation. To address the challenges posed by long sequences, sequence parallelism (SP) [9] [0 is applied for the parallel inference of DiTs, which partitions the input image into patches and needs data exchange across different devices in the attention operations.
Equua contribution.
1 This work was completed when Jiarui Fang, Jinzhe Pan, Aoyu Li, and Jiannan Wang were Tencent.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent studies have highlighted the presence of input temporal redundancy in Diffusion Transformers (DiTs) during inference TT: 6]. This phenomenon manifests as significant similarity in both inputs and activations across consecutive diffusion time steps, presenting opportunities for optimizing parallel computation approaches. DistriFusion [T6] represents a sequence parallelism method that exploits this characteristic in U-Net-based diffusion models. The approach combines fresh local activations with stale activations from preceding steps during attention and convolution operations, effectively overlapping communication costs within individual diffusion timestep computations. However, applying DistriFusion to DīTs reveals substantial memory inefficiencies, as the method requires maintaining full spatial dimensions of attention keys (K) and values (V) across all layers, resulting in memory costs that scale linearly with sequence length.
To more effectively utilize input temporal redundancy, we propose PipeFusion, a novel patch-level pipeline parallelism. PipeFusion partitions both the DiT's layers and input image patches across GPUs, enabling pipelined parallel processing by leveraging temporal redundancy to reuse stale activations from previous timesteps. Firstly, PipeFusion substantially reduces inter-device communication costs compared to existing DiTs' parallel methods. It only transfers the input of the initial layer and the output of the final layer on each device, whereas other parallel approaches, i.e. sequence and tensor parallel, require transmitting activations for every DiT layer. Secondly, PipeFusion offers lower memory consumption by distributing model parameters across multiple devices. This makes it particularly suitable for DiT's with large model parameters, such the 12B Flux.1 model [ In contrast to DistriFusion, PipeFusion reduces the memory buffers required for attention K, to 1/N (N is the parallel degree). Finally, PipeFusion, by utilizing fresh K and V values achieves higher image generation accuracy compared to DistriFusion. Experimental results show that PipeFusion achieves the lowest DiTs inference latency on 8 XLL40 PCle-connected GPUs compared to sequence parallelism, DistriFusion, and tensor parallelism.
longer period,
Background & Related Works
Diffusion Models and Diffusion Transformers (DiTs): Diffusion models generate high-quality images using a noise-prediee crise-prediction deep neural network (DNN) denoted by cg. The process begins with pure Gaussian noise IT A(,,, I) and iteratively denoises it through multiple steps to produce the final image Zo. with T being the total number of diffusion time steps. As shown in Figure I at each step given the noisy image rı. the model co takes Fı. and additional condition (e.g., text, image) as inputs to predict the noise e, within IG. The previous image _t-1 is obtained using:
-t- = Update(#z, (t), Ce = co(re, c)..
(1)
The function Update is specific to the sampler (e.g.. DDIM I77. DPM (8) and involves element- wise operations. Finally, 2o is decoded from the Latent Space to the Pixel Space using Variational Autoencoder (VAE) [9]. The main contributor to inference latency is the forward propagation through
<g. The architecture of eg is transitioning from U-Net 77 to Diffusion Transformers (DiTs) [20- 22], driven by the scaling law that shows improved performance with increased model parameters and training data. Unlike U-Nets that use convolutional layers to capture spatial hierarchies, DiTs segment the input into latent patches and use the transformer's self-attention mechanism (3] to model relationships within and across these patches. The input noisy latent representation is decomposed into patches, embedded into tokens, and fed into a series of DiT blocks, which generally include Multi-Head Self-Attention, Layer Norm, and Feedforward Networks.
condition
Text Encoder
Pixel Space
Updntc(est.))
Latent Space
XT
X
X
VAE
DiTs Backbone
Decoder
Figure 1: Workflow of DiTs inference.
Input Temporal Redundancy: Diffusion Model entails the iterative prediction of noise from the input image video. Recent research has highlighted the concept of input temporal redundancy.

---

which refers to the similarity observed in both the inputs and activations across successive diffusion timesteps [T] [2 A recent study [3] further investigates the distribution of this similarity across different layers and timesteps. Based on the redundancy, branch of research caches the activation values and reuses them in the subsequent diffusion timesteps to prune computation. For example, in the context of the U-Net architecture, DeepCache updates the low-level features while reusing the high-level ones from cache [12]. Additionally, TGATE caches the cross- tross-attention output once it converges along the diffusion process [12]. By contrast, in the domain of DiT models, A-DiT proposes to cache the rear DiT blocks in the early sampling stages and the front DiT blocks in the later stages [24]. PAB [1] exploits the U-shaped attention pattern to mitigate temporal redundancy through a pyramid-styl
broadcasting approach. Finally, DiTFastAttn [14] identifies three types of redundancies, such spatial redundancy, temporal redundancy, and conditional redundancy, and presents an attention compression method to speed up generation.
DiT Parallel Methods
In this section, we delve into the application of tensor parallelism (TP), sequence parallelism (SP), and an asynchronous SP leveraging input temporal redundancy, namely DistriFusion, for parallel inference in DiT's Following this, we introduce novel patch-level pipeline parallel approach named PipeFusion that markedly improves communication efficiency compared to existing techniques.
In the remainder of this article, the following notations are used: p denotes the sequence length, which corresponds to the number of pixels in the latent space. hs represents the hidden size of the model. stands for the number of network layers. N is the number of computing devices.
3.1 Tensor Parallelism Sequence Parallelism
Tensor parallelism (TP) partitions the linear layers of DiTs column-wise and row-wise. This approach requires two all-reduce operations per layer, incurring a communication overhead of 40(p x hs)
per layer (Table [ The communication cost scales linearly with the sequence length, leading to poor scalability, especially for large models such as the 0.6B-parameter PixArt model (Figure 8). Additionally, TP poses implementation challenges for non-standard architectures like MM-DiT Flux/SD3.
Sequence parallelism (SP) partitions images into patches, with each device processing patch as input. Since attention mechanisms require global interactions, devices must communicate partial query (Q), key (K), and value (V) between each other. As shown in Figure 2 two notable SP implementations are DeepSpeed-Ulye Ulysses [9] (SP-Ulysse) (SP-Ulysses) and Ring-Atten Aite-Attention [10] (SP-Ring). SP-Ulysses uses All2All communication to switch between sequence and hidden-size partitioning during parallel attention, while Si-Ring implements distributed Flash Attention via point-to-poin K/V subblock transfers. It is worth noting that SP-Ulysses and SP-Ring be combined to form USP (Unified Sequence Parallel) 25). which is often more efficient than using either method alone.
Deviceo
Impue
Devize1
Inpot
Inguee
SeepSpeed-Ulyssee
Ring-Attention
DistriFusion
Figure 2: Comparison of DistriFusion with Sequence Parallelism methods (DeepSpeed- Ulysedu and Ring-Attention) for attention layers.

---

3.2 DistriFusion: Asynchronous Sequence Parallelism
DistriFusion an asynchronous sequence parallelism method that leverages temporal redundancy in the input data. It uses slightly outdated (stale) activations from the previous timestep, rather than requiring fresh activations at each step. Despite this, DistriFusion maintains image generation accuracy without any perceptible loss in quality.
While DistriFusion was initially applied to U-Net-based SDXL models [66]. it can be extended to DiT models as well. As illustrated in Figure [ DistriFusion also operates by partitioning along the sequence dimension. The key distinction, however, lies in its use of asynchronous all-gather operations to collect K and V activations from remote devices. Due to the asynchronous nature of this communication, the activations for the current diffusion step are not immediately available; instead, they become accessible during the next diffusion step.
DistriFusion exploits this by utilizing a fraction N-1 of the K and V activations from timestep T 1, combined with fraction & of the local and V from diffusion timestep T' This combination is used to compute the attention operation with the local queries at diffusion timestep I'. The communication of K and V for timestep 7' is intentionally overlapped with the network's 's forward computation at timestep T, thus hiding communication overhead during computation.
However, DistriFusion achieves this cramuuunccaiioncompttttttt overlap the cost of increased memory usage. Fach computational device is required to maintain communication buffers that store the complete spatial shape of the K and V activations, which amounts to AL in total. Consequently, the memory cost of DistriFusion does not scale down with the addition of computational devices.
3.3 PipeFusion: Patch-level Pipeline Parallel
PipeFusion simultaneously partitions the input sequence and the layers of the DiTs backbone as shown at the top of F PipeFusion partitions the DiTs model along the direction data flow. Therefore, each partition contains set of consccutive layers and is deployed also partitions the imput image into M non-overlapping patches so that each GPU processes the computation for one patch with its assigned layers in parallel, and the diffusion routine runs efficiently in a pipelined manner. Pipelining requires synchronizations between devices, which would be efficient if the DiT workload is evenly distributed across each GPU. The even partition is easy to achieve DiT contains number of identical transformer blocks.
GPU. PipeFusion
PO
P1
DIT Layaro
DIT Layar1
DIT Layar2
DIT Layar3
DIT Layars
P2
Layard
Layars
Leyar?
P3
Input
Device
Device
Device
Device
Sequence Partition
Backbone Partition
Diffusion timestep
micro-atep--
micro-stap-1
micro-atap=2
micro-atsp-3
micro-atiped
micro-atap-5
micro-step-é
micro-atep-?
0
P2
P3
P1
P3
device
PO
device
P3
device 3
Figure 3: Above: partitioning strategy for Input and DiTs backbone network. Below: Workflow of the PipeFusion as patch-level pipelined parallelism.
Suppose are currently at diffusion timestep T' the previous timestep is T process goes along the reversed order of timesteps. The example in Figure 3 demonstrates the pipeline workflow with 4 and M 4, where the activation values of patches at timestep T are highlighted. Due to the input temporal redundancy, a device does not need to wait for the receiving of full spatial shape activations for the current timestep T to start the computation of its own stage. Instead, it employs the stale activations from the previous timestep to provide context for the current computation. For instance, micro-step 5, the activations for Patchs PO and are from timestep T. while others are from timestep T + 1. In this way, each device maintains the full activations and
the diffusion

---

there is no waiting time after the pipeline is initialized. Bubbles only exist at the beginning of the pipeline. Suppose the total number of diffusion timesteps S, then the effective computation ratio of the pipeline is M As S is commonly chosen to be a large number for high-quality image generation, the effective computation ratio is high. For example, when M = .V = 4 and S = 50, the ratio is equal to 98.5 4.... As long as the patch number M is greater than the parallel degree N, there will be no bubble generation except for the initial startup overhead. We recommend setting M = N. We have analyzed the impact of M on the final performance in Appendix Sec. B
M-S
Moreover, in PipeFusion, a device sends micro-step patch activations to the subsequent device via asynchronous P2P, enabling the overlap between communication and computation. For example, micro-step 4, device () receives P1's activation from diffusion timestep 7 while concurrently comput- ing the activation values for PO timestep T', Similarly, at the next micro-step, the transmission of PO to device I can be hidden by the computation of PI on the device.
Similar to DistriFusion, before executing the pipeline, usually conduct several diffusion iterations synchronously, called warmup steps. During the warmup phase, patches are processed sequentially, resulting in low efficiency. Though the warmup steps cannot be executed in pipelined manner, the workflow is relatively small compared to the entire diffusion process, and thus, the impact on performance is negligible. We have analyzed the impact of warmup and propose the corresponding solutions in Appendix Sec.
PipeFusion theoretically outperforms DistriFusion in terms of generation quality, considering the area of fresh activation. As shown in Figure within a single diffusion timestep, PipeFusion continuously increases the area of fresh activation the pipeline micro-steps progress from diffusion timestep 4 to In contrast, throughout the entire diffusion process, DistriFusion constantly maintains one patch of fresh out of the total M patches. Therefore, the results of PipeFusion closer to the original diffusion method compared to DistriFusion.
Device 0 Activations at Diffusion timestep T (PipeFusion)
Device Activations at Diffusion timestep T (DistriFuuion)
micro-step-4
micro-step-5
micro stopm6
micro-step-7
micro-step-4
miero-stap-5
micro stor--
mioro-step-?
PO
PO
PO
P0
PO
PO
PO
PD
P1
P1
Figure The fresh part of activations during diffusion timestep T of Figure represents fresh data and the light gray represents stable data.
3
The dark gray
The exploitation of input temporal redundancy requires a short warmup phase, since the behavior of diffusion synthesis changes qualitatively over the course of denoising. Accordingly, DistriFusion performs several warmup steps with standard synchronous patch parallelism before switching to its asynchronous mode. For PipeFusion, warmup steps cannot be pipelined and therefore introduce bubbles. However, because the number of warmup steps is typically very small compared to the total diffusion iterations, their overall impact on efficiency limited. We analyze this effect in Sec. B
PipeFusion represents a fundamental departure from prior pipeline parallelization methods, such as GPipe (27 and TeraPipe 28). Unlike GPipe, which splits along the batch dimension and depends on high concurrency across multiple requests, PipeFusion partitions along the sequence dimension to deliver efficient pipelined execution for single image. Unlike TeraPipe, which is designed for causal attention in language models, PipeFusion is tailored to the full-attention mechanism of DiTs and exploits temporal redundancy simultaneously improve efficiency and preserve generation quality.
3.4 Comparison Between Different Parallelisms
Compared with existing parallel solutions, PipeFusion demonstrates superior efficiency in commu- nication and memory usage. Firstly, PipeFusion transmits between computational devices only the activations that serve as inputs and outputs for series of consecutive transformer layers belonging to stage, significantly reducing the communication bandwidth requirement comparison to other methods that transmit the KV activations for all the I. layers. In particular, the total communication cost for PipeFusion 20(p x hs), not associated with the number of layers L.. Secondly, each device in PipeFusion stores only & of the parameters belonging its specific layers. Since the usage of
5

---

Table Comparison between different DiT parallel methods on single diffusion timestep. Overlap
denotes the overlap between communication and computation.
Communication
Memory Cost
Method
Cost
Overlap
Model
KV Activations
Tensor Parallel
40(p hs)
x
P
N
KV
DistriFusion
20(p x h
4
P
LKV)L
SP-Ring
20(p x hs)
P
KV
Y
SP-Ulysses
(p(p h
P
KV
PipeFusion
20 x hs)
N
(KV)L
stale KV for attention computation requires each device to maintain the full spatial KV for the L layers the device, the overhead is significantly smaller than that of DistriFusion and diminishes as the number of devices increases.
N
In the following, summarize the theoretical cost of existing methods in Table I The commu- nication cost is calculated by the product the number of elements to transfer with an algorithm bandwidth (algobw) factor related to communication type For collective algorithms of AllReduce, AllGather and AllToAll, the corresponding algobw factors are 2 n-1 and 1. In the table, we approximate the term O (~=+) to O(1) for simplicity.
Among all methods, PipeFusion has the lowest communication cost, long N < 2L, which is easy to satisfy as the number of network layers L. is typically quite large, e.g., I = 38 Stable- Diffusion-3. In addition, PipeFusion overlaps communication and computation. Ulysses Seq Parallel exhibits a decreasing communication cost with increasing N, outperforming the remaining three methods. However, its communication cannot be hidden by computation. Ring Seq Parallel and DistriFusion have similar communication costs and overlapping behaviors. The distinction lies in the scope of overlapping: computation in Ring Seq Parallel overlaps within the attention module, whereas that in DistriFusion overlaps throughout the entire forward pass.
To analyze the memory costs, denote P as the total number of model parameters. In PipeFusion and tensor parallelism, the memory cost decreases as more GPUs are utilized, which is nice property with the rapid growth of the DiT model size. Both PipeFusion and DistriFusion maintain a KV buffer for each transformer layer leading to significant activation memory overhead, especially for long sequences. The KV buffer in PipeFusion decreases the number of devices N increases, whereas DistriFusion does not exhibit such a reduction.
Experiments
In our experimental setup, we deployed our trials on an Lx (PCle Gen4x16) cluster to eval- uate three prominent DiT models: Pixart 3]: Its backbone is transform model that encompasses 0.6B (billion) parameters. Pixart draws its architectural foundation from the original DiTs framework. with pivotal enhancement: the integration of cross-attention modules designed to incorporate text conditions into the model's processing. Stable-Diffusion-3 [2 (SD3-medium): which leverages a Multimodal (MM)-DIT and its backbone transformer model contains 2B parameters. As the largest open-source variant available, SD3-medium employs 20-step
MaraatchhuurDDiseetteeeeeeeee 1-ux, [1] Its backbone transformer model contains 12B parameters and features an enhanced MM-DiT design. Flux. -edev also relies on a 28-step FlowWWachEEnleeDisrreeechhhhuu
4.1 Performance Results
This section compares the performance of different parallel methods on 8 8x GPUs, connected by PCIe Gen4. This section only presents the latency of the DiT except for the latency of the VAE decoding, since VAE module is the same across different parallel methods. Since the Pixart and SD3 can utilize Classifier-Fres Guidance (CFG) 29), we additionally implemented the CFG parallel. For an input prompt, CFG parallel individually computes the forward tasks for both unconditional

---

guidance and text guidance. It leverages inter-image parallel to perform these two tasks, collecting the results after each diffusion step is completed. For PipeFusion, we select the best latency performance by searching the patch number M from 2, 4, 8, 16, 32. All figures show the average latency of 5 runs. We employ a default warmup step of for both DistriFusion and PipeFusion. The software stack utilized includes PyTorch 2.4.1, CUDA Runtime 12.1.105, and diffusers 0.30.3.
4.1.1 Latency Evaluation
Pixart: Figure 5 shows the inference latency of Pixart on image generation tasks with resolution as 1024px (1024x x 1024), 2048px (2048 x 2048) and 4096px (4096 4096) BxL40.
Latency Pleort on Bat 40 for 10etpx image
Jeeenyy of Plzwet 40 2040px image
Latency Pizzrt Bel 4090 image
200
304.80
17.0
200
5331
7177
122
0.55
3.87
.3.0
25.5
Figure 5: Latency on Pixart of various parallel approaches on two image generation tasks with the
20-Step DEMSSoherMuttiiepScccheeer
Latency 503-medium 1DDlgp image
Lolinny 8D0-mediim 8sL.40 2048px
20.-4
12
2.25
1.15
4.11
(tanus
bamiine
apulystere dtgttter
venggia
frani
apsing
Ap degeeeee
Nosiattio
Legit
Figure 6: Latency on -medium of various parallel approaches on two image generation tasks with the 20-Step Flow MalccaaaeeerDiserre Scheduler.
Lateray of Plact dev on Bx (28 (ecas) 1024lx image
Lalengy of Flta 1-dew (58 stepc) on 0xd.40 20r0pu Image
100
80
70-52
Labona
80
7.28
3.53
40
27.43
20
0
bexoline
op-ulyptazs
op-iig
pipatusion
basoline
speting
pipatasion
(1sps))
ringed
(1200)
riggu
Figure 7: Latency on Flux. conx.I-dew 1-dev of various parallel approaches on three image generation tasks with the 28-Step FrowMacchEuleriisceeee Scheduler.
PipeFusion exhibits the lowest latency among parallel methods across all three tasks. PipeFusion is 1.48% x, 1,55x, and 1.16x faster than the second best parallel approaches on the 1024px, 2048px and 4096px tasks.
The highest latency parallelism is always SP- SPyUsysses, primarily due to the bandwidth bottleneck between CPU sockets by conducting A1l2Al on 8 GPUs via PCle. SP-Ring exhibits significantly lower latency compared to SP-Ulysses. An efficiency method USP 25 can further improve SP performance by hybridizing SP- SP-Ulysses and SP-R SP-Ring. It views the process group a 2D mesh where the columns are SP-Ring groups and the rows are SP-Ulysss groups USP parallelizes the attention head dimension in SP-Ulysses groups and parallelizes the sequence dimension in SP-Ring groups
7

---

USP yields lower latency than SP-Ring, as illustrated by the ulysses=2 and ring-4 columns in the figure.
Additionally, tensor parallel also shows high latency, which aligns with our analysis in section 3.4 DistriFusion exhibits slightly higher lateney compared the USP (ulysses=2 and ring=4), and encounters out-of-memory (OOM) issues on the 4096px task.
Using the CFG parallel, the 8 GPUs are divided into two groups, each performing PipeFusion independently, which can further reduce latency, This approach (cfg=2, pipefusion=4 in the figure) achieves latency reduction of 5.09x, 7.91x, and 8.59x compared to the baseline (1 GPU). PipeFusion achieves perfect scalability in tasks 2048px and 4096px.
Teart
Figure 8: Scalability Analysis of Three DiT' on 8xI40: Leftmost figures depict Pixart performance on 1024px, 2048px, and 4096px images. The second from the right shows SD3 performance on 1024px. The rightmost figure illustrates Flux. 1-dev performance on 1024px. The speedup to GPU baseline is labeled on the points of pipefusion curves.
Stable Diffusion 3: shows the inference latency of SD3-medium image generation tasks with resolution as 1024px, 2048px on 8 XL04 The model is unable to generate 4096px images due to positional encoding limitations. Since MM-DiT applied in SD3 is non-standard transformer layer, arranging column-wise and row-wise weight partitioning poses significant challenges, and we did not implement tensor parallelism. Additionally, due to potential OOM issues with DistriFusion and inferior performance compared with USP, we did not implement it for SD3.
PipeFusion outperforms the best sequence parallelism in these two cases, both of which are also the USP with ulysses=4, ring=2, achieving speedups of 1.57 x and 1.30x. When combined with CFG parallel, the pipefusion=4, cfg=2 scheme on 8 GPUs achieves speedups of 3.11x and 8. 8.16x compared to the baseline (1 GPU).
Figure 7 illlustraees the inference latency of SD3-medium on image generation tasks with resolutions of 1024px and 2048px, respectively, using 8x1.40. Flux.1 employs a similar MM-DiT architecture to SD3 but extends the network depth to 57 layers.
Flux.
In both scenarios, PipeFusion surpasses the best sequence parallelism, which is also the USP with ulysses=4 and ring=2, achieving speedups of 1.23 x and 1.25 x. PipeFusion offers 3.42x and 5.79x speedup compared the baseline (1 GPU) for the two cases.
Scalability: We evaluated the scalability of inference for three models by scaling from to 2, 4, and 8 GPUs. Figure X illustrates the scalability of different parallel approaches. PipeFusion and DistriFusion both apply a single warmup iteration, while for SP we adopt the USP, which searches the optimal combination of Ulysses and Ring degrees. The results show that PipeFusion consistently achieves the best scalability across all five scenarios. Given that PixArt has the smallest model size, tensor parallelization unsurprisingly exhibits the worst scalability.
At 1024px and 2048px resolutions, PipeFusion substantially outperforms both SP and DistriFusion. Even at 4096px, where DistriFusion encounters out-of-memory (OOM) issues and thus provides no valid data, PipeFusion continues to deliver strong performance: on 8 LX0 GPUs, it achieves 32.1s end-to-end latency versus SP's best 37.3s, corresponding to a 1.16x speedup. While this speedup appears smaller than at lower resolutions, PipeFusion's 's communication efficiency remains a clear advantage. Specifically, PipeFusion reduces communication share to only 4.6% of total latency 4096px, compared with 17.9% for SP.
Here, the communication share is defined as
8

---

TEDE, SGPU
Tingie
CommShare
(2)
TR SGPU
which subtracts 1/8 of the single-GPU latency from the observed 8-GPU latency. For example,
32.1 244.80/8 = 4.6% (PipeFusion),
37.3 244.89/8
= 17.9% (SP).
32.1
37.3
In absolute terms, SP's communication cost is 6.69s, whereas PipeFusion lowers it to 1.49s - 78% reduction. This shows that even in the compute-bound regime, PipeFusion robustly suppresses communication overhead.
4.1.2 Memory Efficiency
We collect the memory usage excluding the VAE usage, which involves the computation of the text-encoder and DiTs backbones, as shown inAs shown in Figure The figure includes memory usage for "parameters", which encompasses both the text encoder and transformers, well for "activations", which includes activations and temporary buffers. Note that SP-Ulysses, SP-Ring and USP exhibit similar memory consumption, denoted as SP the figure.
The maximum memory usage of different parallel approaches for Pixart is depicted in Figure Pa The Pixart model consists of 0.6B parameter DiT backbone (2.3 GB on disk) and T5-based text encoders (18GB on disk). So the memory consumption of parameters is dominated by the text encoder. DistriFusion maintains full spatial shape K, V tensors for each layer, resulting in increase in memory consumption as the image resolutions grow. In contrast, PipeFusion only needs to store I/N of the full spatial shape K, V tensors. Consequently, comparable to that of Tensor Parallel and Sequence Parallel. Remarkably, even for tasks with an 8192px resolution, PipeFusion exhibits the lowest memory consumption.
8x GPUs, its memory footprint is
Menary Usage 21
Nerrey Usage 25540 flac
lansor peralial
distitaioo sucuenca pursil
1760
24.5
(a) Max GPU Memory Usage of Various Ap- proaches for Pixart Image Generation Tasks (b) Max GPU Memory Usage GPUs for 3D3-medium and Across Four Resolutions.
1004pe
2040gz
10P4pe
2048pe
Flux. 1024px and 2048px image generation tasks.
Figure 9: Comparison of GPU memory usage across different models and resolutions.
The maximum memory usage of sequence parallel and PipeFusion for SD3-medium and Flux. 1-dev is depicted in Figure 9b. The SD3-medium consists of 2B parameter DiT's backbone (7.8 GB on disk) and text encoders of nearly 19GB on disk. The Flux.1-dev consists of 12B parameter DiTs backbone (23 GB disk) and text encoders of nearly 9.I on disk. The memory cost of PipeFusion is also significantly less than that of sequence parallelism. For the 12B Flux. I-dev PipeFusion significantly reduces the memory footprint of model parameters, However, PipeFusion increases the memory usage of activations due to the consumption of the KV Buffer. Nevertheless, this increase is relatively minor compared to the overall model parameter memory usage. The overall memory usage of PipeFusion is 32% and 36% of SP on 1024px and 2048px cases using Flux.1.
4.2 Quality Results
Table presents the Fréchet Inception Distance (FID) 30 scores for PipeFusion, DistriFusion, and the Original model, evaluated on the Pixart and Flux.1 datasets. lower FID score is indicative of superior performance in terms of image quality. The results clearly demonstrate that PipeFusion consistently outperforms DistriFusion across identical device configurations, achieving significantly lower FID scores. This finding is in direct alignment with the theoretical analysis presented in Sec. 3.3 where PipeFusion is shown leverage fresher activations while effectively reducing the impact of stale ones, thereby enhancing overall performance.

---

Additionally, we provide comprehensive visual comparison in Fig. 10 of Appendix A where image samples generated by PipeFusion, DistriFusion, and the Original model are displayed. The images generated by PipeFusion are nearly indistinguishable from the original images to the human eye.
Model
PixArt-XL--2226-MS
FLUX.I[dev]
Method
Original
DistriFusion
PipeFusion
Original
PipePusion
Device Number
4
8
4
8
4
w/ G.T.
22.78
30.69
41.81
23.60
25.23
28.46
25.03
24.17
25.97
FID
w/ Orig.
8.41
19.81
1.67
3.11
6.09
4.01
5.93
Table 2: FID Scores for Parallel Methods on the Pixart and Flux. 1. w/ G.T. means calculating the FID metrics with the ground-truth images. w/ Orig. means calculating the FID) metrics with the images generated by the 1-Device original implementation.
5 Limitations & Discussion
While this paper has demonstrated PipeFusion's advantages over tensor parallelism and sequence parallelism, we acknowledge that it is not a universal solution. In fact, combining PipeFusion with existing approaches into a hybrid parallelization scheme likely the most practical direction for deployment. Such a design enables efficient scaling of DiT inference across large cross-node configurations: sequence parallelism can be exploited within NVLinn-connected, high-bandwidth settings, while PipeFusion is particularly effective across inter-node, Now-bandwidtt environments.
key limitation of PipcFusion lies in its dependence on multiple diffusion steps. For models with only a handful of steps such as distilled variants like the 4-step Flux. 1 1-schnell [3]] or one-step
temporal redundancy is minimal, making PipeFusion less effective. However, this does not pose major constraint in practice, as of DiT models generally require a substantial number of steps to deliver competitive quality.
diffusion methods 332 33
Another promising extension is the application of PipeFusion to video generation. Architecturally, PipeFusion is already compatible with multimodal DiTs (MM-DiTs) such as Flux.1 and Stable Diffu- sion 3, where cross-modal conditions are injected through unified attention. Recent video models (e.g.. Wan2.1 [34]. Hunyuan Video (35). CogVin Video (36)) adopt similar MM-DiT backbones, suggesting that PipeFusion can be applied with minimal adaptation. From the tensor-shape perspective, both image and video DiTs operate on latent tensors of shape (B, L, with video models flattening spatial and temporal dimensions into unified sequence dimension. Because PipeFusion partitions along the sequence dimension, this structural difference does not affect applicability. From the performance perspective, video models are typically larger and involve longer sequence lengths, which amplifies communication overhead for SP. PipeFusion mitigates this overhead through parameter partitioning and pipelined execution, offering reduced memory footprint and stronger scalability. This suggests that video generation is not only natural but also a highly favorable domain for applying PipeFusion
Conclusion
We introduced PipeFusion, patch-level pipeline parallelism strategy tailored for Diffusion Transform- ers. By reusing temporally redundant activations and overlapping communication with computation, PipeFusion reduces both bandwidth demands and memory footprint while preserving image quality. Experiments on Pixart, Stable Diffusion 3, and Flux. demonstrate consistent latency improvements over strong baselines, achieving up to 1.5x faster inference commodity multi-GPU clusters.
Beyond efficiency gains, PipeFusion highlights broader systems insight: temporal redundancy in diffusion processes can be systematically exploited to design new forms of parallelism. While our study focused on single-node, multi-GPU inference, the approach naturally composes with existing sequence or tensor parallelism, offering path toward scalable deployment on large clusters. We hope this work inspires further exploration of hybrid parallel strategies that close the gap between rapidly growing generative models and the practical requirements of serving them at scale.
10

---

References
[1]
BlackForestl.abs. Announcing black forest labs.
https: https://bllcctforetttaabs ai/ 2024. Accessed: [2024.10].
[2] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[3] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang. Ping Luo, Huchuan Lu, and Zhenguo Li. rixart'/sigmaa Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint MxXiv:2403.04622 2024.
[4] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
[5] OpenAl. Video generation models as world simulators.
https://openal: com/index/ video- wideo-generattnn weneration-modelss-e.e. assdeellsas-worr www-wwordddsmmmlmdeee simulators/ 2024. Accessed: May 2024.
[6] MetaAL. Movie gen: cast of media foundation models. https: // ai. meta. com/
staaticresourcee movie gen- reea-reeeerchhpppp paper 2024. Accessed: 2024-10-14.
[7] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAN 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[8] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:I909.000 08053, 2019.
[9] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309. 14509, 2023.
[10] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:2310.01889, 2023.
[11] Junhyuk So, Jungwon Lee, and Eunhyeok Park. Frdiff: Feature reuse for exquisite zero-shot acceleration of diffusion models. CORR, abs/2312.03517, 2023.
[12] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15762-15772, 2024.
[13] Xinyin Ma, Gongfan Fang, Michael Mi, and Xinchao Wang. Learning-to-cacher Accelerating diffusion transformer via layer caching. arXiv preprint wxXiv:2406.0/733 2024.
[14] Zhihang Yuan, Pu Lu, Hanling Zhang, Xuefei Ning, Linfeng Zhang, Tianchen Zhao, Shengen Yan, Guohao Dai, and Yu Wang. Ditfastattn: Attention compression for diffusion transformer models. arXiv preprint Xrv:::240 085552 2024.
[15] Xuanlei Zhao, Xiaolong Jin, Kai Wang, and Yang You. Real-time video generation with pyramid attention broadcast. arXiv preprint arXiv:2408. /2588, 2024.
[16] Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming- Yu Liu, Kai Li, and Song Han. Distrifusion: Distributed parallel inference for high-resolution diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[17] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2r11v22010.02502, 2020.

---

[18] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprini arXiv:1312.611 14, 2013.
[20] Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng. Xiang Li, Cong Xie, Shibiao Nong, et al. Megascale: Scaling large language model training to more than 10,000 gpus. arXiv preprint arXiv:2402.15622 2024.
[21] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo. Huchuan Lu, et al. Pixart-a: Fast training of diffusion transformer for photorealistic text-to ext-to-imaae synthesis. arXiv preprint arXiv:2310.00426. 2023.
[22] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint AxXiv:241103344: 2024.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[24] Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, and Tao Chen. /delta-dit: A training-fire free acceleration method tailored for diffusion transformers. arXiv preprint arXiv:2406.01122 2024.
[25] Jiarui Fang and Shangchun Zhao. A unified sequence parallelism approach for long context generative ai. arXiv preprint arXiv:2405.07719, 2024.
[26] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint urXiv:230..01952, 2023.
[27] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
[28] Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, and Ion Stoica. Terapipe: Token-level pipeline parallelism for training large-scale language models. In International Conference on Machine Learning, pages 6543-6552 PMLR, 2021.
[29] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12554 12598, 2022.
arXiv preprint
[30] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[31] black-forrst
hugging face.
[32] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6613 6623, 2024.
[33] Zhengyang Geng, Mingyang Deng, Xingjian Bai, Zico Kolter, and Kaiming He. Mean flows for one-step generative modeling. arXiv preprint arXiv:2505. /3447, 2025.
[34] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et Wan: Open and advanced large-scale video generative models. arXiv preprint arXi:2500..23144 2025.
12

---

[35] Weijie Kong, Qi Tian, Zijian Zhang, Rox Min. Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03603 2024.
[36] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 15868, 2022.
[37] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1504.001004 2015.
(38) Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties gan evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11410-11420 2022.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper clearly states its contribution proposing PipeFusion, a novel patch-level pipeline parallelism method for efficient inference of diffusion transformer models. It highlights the key ideas of leveraging input temporal redundancy and reducing communication and memory costs compared existing parallel methods. The claims are supported by the experimental results demonstrating state-of-the-ar performance on multiple models and configurations.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses limitations in Section 5, noting that PipeFusion is not well-suited for models with limited number of diffusion steps, such distilled models or one-step diffusion methods. also mentions that while PipeFusion reduces communication and memory costs, may face challenges in scenarios with extremely long sequences where communication overhead becomes less significant.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
13

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper focuses on proposing and evaluating the PipeFusion method through experimental results, and does not include formal theoretical results, theorems, proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper provides sufficient details reproduce the main experimental results. It describes the experimental setup. including the hardware configuration (8xL40 PCIe GPUs). software stack (Py (PyTo,,h, CUDA Runtime, diffusers), and the models used (Pixart, Diffuu-Diffuiinn 3, Flux. 1). It also explains the methodology for implementing PipeFusion and other parallel approaches, and provides results for various image resolutions and configurations. While the code and data are not explicitly provided, the detailed descriptions allow for replication of the experiments.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
14

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibilityy may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper cites the original sources for the models and datasets used, such as Pixart, Stable-Diffusion 3, and Flux.I. It also mentions the use of publicly available datasets like COCO Captions 2014 for evaluation. While it does not explicitly state the license for each asset, is common practice in the field to assume that publicly available datasets and models are used under their respective standard licenses (e.g.. CC-BY for COCO Captions). The paper does not release any new assets, so there is no need provide licenses for derived assets.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hhapss.c//ubllc/guudde
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
15

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper cites the original sources for the models and datasets used, such as Pixart, Stable-Diffusion 3, and Flux.1. also mentions the use of publicly available datasets like COCO Captions 2014 for evaluation. While it does not explicitly state the license for each asset, is common practice the field to assume that publicly available datasets and models are used under their respective standard licenses (c.g., CC-BY for COCO Captions). The paper does not release any new assets, so there is no need to provide licenses for derived assets.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not report error bars, confidence intervals, or other statistical significance tests for the experiments. It is because the results is stable among tests. presents the average latency and FID scores for multiple runs but does not provide informa- tion on the variability or statistical significance of these results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
16

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper provides detailed information on the computer resources used for the experiments, It specifies the hardware configuration 8xL40 PCIe GPUs with 48GB memory each and mentions the software stack including PyTorch 2.4.1 and CUDA Runtime 12.1.105. Additionally, it reports the average latency for multiple runs, which indicates the time of execution for each experiment.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpgrr//aarrppo.ccchhhba//c/hha
Answer: [Yes]
The research described in the paper does not involve human subjects, sensitive data, any other elements that would require special ethical considerations according the NeurlPS Code of Ethics. The paper focuses on improving the efficiency of diffusion transformer models for image generation, which does not pose risks related to privacy, discrimination, security, or other ethical concerns outlined in the guidelines.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
The paper does not discuss the broader socictal impacts of the work performed. While it focuses improving the efficiency of diffusion transformer models for image generation, it does not address potential positive or negative societal impacts. The research is foundational and does not directly tie specific applications or deployments that could have significant societal consequences.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
The paper does not release any datasets models that pose high risk for misuse. It focuses on the methodology and experimental results of the PipeFusion approach without distributing any models or datasets that could be misused.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The paper cites the original sources for the models and datasets used, such as Pixart, Stblle-Diffusion 3, and Flux. also mentions the use of publicly available datasets like COCO Captions 2014 for evaluation. While it does not explicitly state the license for each asset, is common practice the field to assume that publicly available datasets and models are used under their respective standard licenses (e.g., CC-BY for COCO Captions). The paper does not release any new assets, so there is no need to provide licenses for derived assets.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce or release any new assets such as datasets, models, or code. Therefore, there is no need to document provide documentation for new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsoureing experiments or research with human subjects. It focuses on the technical aspects of parallelizing diffusion transformer models for image generation, which does not require human participation data collection.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any research with human subjects, so IRB approvals equivalent reviews are not applicable.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
The paper does not describe the usage of Large Language Models (LLMs) important, original, non-standard component of the methods. The research focuses on the development and evaluation the PipeFusion method for parallelizing diffusion transformer models, which is technical contribution unrelated to LLMs.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy lattps://neurdl: httpppss.cCCooooo wcc/oonferences/2000 for what should or should not be described.
A Image Quality
In Figure 10 we selected a few sample images generated using prompts from DistriFusion [16] to help readers visually assess generation quality. The images generated by PipeFusion nearly indistinguishable from the original images to the human eye, across various configurations of the number of patches and devices. Note that the FID evaluation results are independent of the specific prompts used for the visual demonstrations.
B Ablation Study
We analyze the impact of the setting of patch numbers M on PipeFusion as shown in Figure [I] According to the analysis presented in Table I M does not affect the communication cost. However, as shown in the figure, the lowest latency is achieved when M is equal to the number of GPUs, N. If M is large, the input size for operators becomes smaller, which can negatively impact the computational efficiency of the operator. In contrast, if M is small, the efficiency of overlapping communication and computation decreases.
Patch Number:
The number of warmup steps can negatively impact the overall performance of PipeFusion. The input temporal redundancy is relatively small during the initial few diffusion timesteps. Therefore, it is usually to employ a few warmup steps to enable synchronous communica- tion without resorting to stale activations, which would introduce waiting time, commonly referred to as bubbles, into the pipeline. As the number of sampling steps increases, the latency will also increase, shown in Table 3 If the proportion of warmup steps is relatively small, action is required. Our experiments involved a relatively small number of sampling steps, specifically 20 and 28. In other related work, it is common to use 50 steps [66] or 100 steps [[2]. which helps to alleviate the overhead of warmup.
Warmup Step:
To mitigate the performance degradation caused by warmup, we can separate the warmup steps from the remaining working steps and allocate different computational resources them. The output feature maps after the warmup steps can then be transmitted from the warmup devices to the
20

---

PipeFusion 2 Devices patch number=2
PipeFusion 4 Devices patch number=4 FID: 25.23
PipeFusion Devices patch number=8
Original Device FID: 22.78
DistriFusion Devices FID: 30.69
DistriFusion 8 Devices
FID: 41.81
FID: 23.60
FID: 28.
Prompt: A double decker bus driving down the street.
Prompt: A brown dog laying on the ground with a metal bowl in frons of him.
MAX
Prompt: multi-colored parrot holding its foot up to beak.
Prompt: A kid wearing headphones and using a laptop.
Prompt: A pair of parking meters reflecting expired times.
Figure 10: Above: Showcases for 1024px Generation Images of PipeFusion and DistriFusion using Pixart. Bottom: FID Scores Evaluated for Pixart and Flux.1. We use 20-step DPM-Solver with the warmup step set to for DistriFusion and PipeFusion. We use the COCO Captions 2014 [37 dataset evaluate the FID scores. During the evaluation, subset comprising 30,000 images is sampled from the validation set and resized to 256px to serve as the reference dataset. Concurrently, each experiment generates 30,000 images of 256px, each paired with a caption derived from the COCO Captions 2014 dataset, as the sample dataset. The quality of images generated by PipeFusion closely resembles that of the original images, regardless of whether 4 or devices are used, and across varying patch numbers. FID above the images is computed against the ground-truth images using Clean-FID 38
21

---

100
Latenay vs Paxch Number BxL40 (Pixart)
Lateecy Patch Number 4aLAD (Posai))
Lutercy Patzzh Number on 2x140 (Poant)
1084px 2048px
1024px
20-0px
Latenay (pen))
4096px
Latonny
1024px 2048px
16
52
Patch Number
Patch
Patch Number
Figure 11: Latency of PipeFusion (without warmup) for various patch numbers M 2x, 4x, and 8xL40 GPUs in image generation tasks at three different resolutions using Pixart.
pixart
sd3-medium
Flux.
Warmup
pp=8
cfg=2, pp=8
pp=8
cfg=2, pp=8
pp=8
0
0.71
0.66
1.05
0.83
5.48
1
0.82
0.69
1.16
0.87
5.53
(+15%)
(+4%)
(+10%)
(+4%)
(+1%)
2
0.91
0.70
1.27
0.92
6.00
(+28%)
(+6%)
(+21% 1))
(+11%)
(49%)
Table 3: Impact the warmup step number to Pixart, sd3-medium and Flux. on 1024px image generation, where pp indicates pipefusion (pp) parallel degree.
working devices. The warmup phase apply sequence parallelism to fully utilize the computational resources.
We found that the setting of warmup steps is highly dependent on the model. By initializing the KV buffer zero and not performing warmup, Pixart can still achieve satisfactory image generation results. Through dynamic detection of the difference between the latent space input and the previous diffusion step, we can automatically set the warmup steps. These optimizations for warm-up will be our future work.
Method
1024px
2048px
4096px
Tensor Parallel
1.22
7.07
36.33
DistriFusion
0.77
2.48
25.24
SP-Ring
1.37
2.46
23.31
SP-Ulysses
2.59
3.54
27.41
PipeFusion
0.66
2.59
22.39
Table 4: PixArt latency (sec) 8XA100 GPUs with NVLink interconnect. PipeFusion consistently delivers the lowest latency.
Evaluation on NVLink connected GPUs:
To evaluate the sensitivity of PipeFusion to hardware interconnect bandwidth, we conducted additional experiments on A101 GPUs connected via NVLink banghhbandidthh in contrast the default 8xL40 PCIe setting. Table reports the latency for PixArt inference across different resolutions. PipeFusion consistently achieves the lowest latency across resolutions and remains robust even under high-bandwidth NVLink settings. While the relative performance gap narrows at higher resolutions due to the increasing dominance of computation cost (i.e., the shift from communication- to computation-bowe is still competitive superior to all baselines. We also emphasize that the FID score is invariant to GPU type or interconnect bandwidth, as it is solely determined by the model architecture and inference algorithm.
regimes), PipeFusion
22


Reviews:
[{"review_id": "IpndZ7MH2k", "reviewer": "Reviewer_2Wj8", "summary": "The paper proposes a parallelization scheme for diffusion model inference and demonstrates favorable performance on a range of models on 8 NVIDIA L40 GPUs. The primary consideration is to minimize latency caused by GPU-to-GPU communication. First, it leverages pipeline parallelism so that the communication is only required for passing the last activations in one device to the other. Second, it performs patch-wise processing and uses the cached activations from the previous denoising time step, leveraging the temporal similarity of activations across neighboring time steps in diffusion model inference. The results demonstrate an efficient inference pipeline that outperforms competing implementations.", "questions": "Q1: Would it be feasible to extend the proposed method to handle DiT-based video model inference?\n\nQ2: Have you considered compressing the activations to further reduce communication cost?\n\nQ3: Is the DiT activation as smooth as the UNet activations across different denoising time steps.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Achieve clear speed gain over competing methods\n- Have a lower memory footprint.\n- I think the proposed method shows a new direction for diffusion model inference optimization. I am unaware of prior works leveraging pipeline parallelism for DiT acceleration.\n\nWeakness\n- A nice integration of known techniques. However, the added value is a bit limited.", "was_revised": true, "final_justification": "After checking the rebuttal and the reviews from the other reviewers, I lean towards keeping my original rating. I feel the merit provided in the paper warrants a publication.", "rebuttal": "Q1: Would it be feasible to extend the proposed method to handle DiT-based video model inference?\n\nThank you for the insightful question. We believe PipeFusion is highly compatible with DiT-based video models and can naturally extend to this domain with minimal adaptation.\n\nFrom a model architecture perspective, PipeFusion already supports Multimodal DiT (MM-DiT) structures, which inject both text and image conditions through a shared unified attention mechanism, enabling joint cross-modal interaction within the same transformer layers. As demonstrated in our experiments with Flux.1 and Stable Diffusion 3, both of which are MM-DiT structures. Notably, the most recently released video generation models—such as Wan2.1, HunyuanVideo, Mochi-1, and CogVideo—adopt similar MM-DiT backbones. PipeFusion can support these models seamlessly.\n\nFrom a tensor shape perspective, both image and video DiTs operate over latent tensors with the shape (batch_size, sequence_length, hidden_size). While image models flatten spatial dimensions (height × width) into sequence tokens, video models flatten temporal and spatial dimensions (frames × height × width) into sequence tokens. Since PipeFusion performs pipeline parallelism along the sequence dimension, this difference does not affect its applicability, making video and image DiTs equally compatible.\n\nMoreover, from a performance standpoint, PipeFusion is particularly well-suited for video models with longer sequence lengths and larger models. Compared to sequence parallelism (SP), PipeFusion incurs significantly lower memory footprint by partitioning model parameters layer-wise across GPUs—an important advantage for video models with more parameters (e.g., StepVideo, ~30B). Additionally, video models typically exhibit longer sequence lengths, amplifying the communication burden for SP. As shown in the table below, our results already demonstrate that for Pixart at 1024px (sequence length 4K), PipeFusion achieves 1.85× communication speedup over SP, and at 4096px (sequence length 65K), the gain increases to 4.49×. We expect even greater benefits in video settings (e.g., Wan2.1 with 1080×720×5s input seqlen is 75,600 tokens).\nPS: Comm. Time is calculated by (8 × L40 latency - 1 × L40 baseline) / 8\n\n|                          | 1024px        | 2048px       | 4096px        |\n|--------------------------|---------------|--------------|---------------|\n| **seqlen**               | 4,096         | 16,384       | 65,536        |\n| **8×L40 PipeFusion Latency (sec)** | 0.82          | 3.27         | 32.1          |\n| **8×L40 SP Latency (sec)**        | 1.22          | 5.07         | 37.3          |\n| **PipeFusion Comm Time (sec)**    | 0.47          | 0.57         | 1.48875       |\n| **SP Comm Time (sec)**            | 0.87          | 2.37         | 6.68875       |\n| **Comm. PipeFusion / SP**         | 185%          | 416%         | 449%          |\n| **1×L40 Baseline (sec)**          | 2.8           | 21.6         | 244.89        |\n\nWe will add a brief discussion on this extension in the paper and plan to include detailed benchmarks on video models in future work.\n\nQ2: Have you considered compressing the activations to further reduce communication cost?\n\nThank you for the insightful suggestion. We agree that compressing activations is a promising direction for further reducing communication overhead in multi-GPU inference.\n\nAs you noted, communication remains a significant component of end-to-end latency in certain scenarios. For instance, in our experiments on Pixart 1024px using 8×L40 GPUs, communication accounts for approximately 47% of the total inference latency (As shown in the above Table). In such cases, activation compression techniques—such as quantization or sparsification—could indeed further reduce bandwidth usage and improve performance. Work, such as SQ-DM, CompactFusion, and TDQ, has demonstrated the potential of such methods.\n\nSQ-DM: Fan Z, Dai S, Venkatesan R, et al. SQ-DM: Accelerating Diffusion Models with Aggressive Quantization and Temporal Sparsity[J]. arXiv preprint arXiv:2501.15448, 2025.\n\nTDQ: CompactFusion: So J, Lee J, Ahn D, et al. Temporal dynamic quantization for diffusion models[J]. Advances in neural information processing systems, 2023, 36: 48686-48698.\n\nCompactFusion: Luo J, Xiao Y, Xu J, et al. Accelerating Parallel Diffusion Model Serving with Residual Compression[J]. arXiv preprint arXiv:2507.17511, 2025.\n\nHowever, we also observed that the benefit of activation compression may diminish for high-resolution inputs. For example, in our 4096px Pixart-8×L40 setup, communication only contributes to 4.6% of the total latency. The program becomes compute-bound, and introducing additional operations (e.g., for compression or decompression) may negate the communication savings and even hurt performance.\n\nWe will add a discussion on activation compression and its trade-offs to the main paper. \n\nQ3: Is the DiT activation as smooth as the UNet activations across different denoising time steps?\n\nThe Δ‑DiT [24] paper highlights that both U-Net and DiT both exhibit high activation similarity between adjacent denoising steps. Empirical evidence (e.g., in DiTFastAttn [14], Learning‑to‑Cach [13]) also suggests that temporal redundancy is significant enough to be exploited for acceleration or compression.\nWe have added a brief discussion of this point in the revised version.\n\nChen P, Shen M, Ye P, et al. *Δ‑DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers*. arXiv preprint arXiv:2406.01125, 2024.\n\nYuan Z, Zhang H, Pu L, et al. *DiTFastAttn: Attention Compression for Diffusion Transformer Models*. *Advances in Neural Information Processing Systems*, 2024, 37: 1196–1219.\n\nMa X, Fang G, Bi Mi M, et al. *Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching*. *Advances in Neural Information Processing Systems*, 2024, 37: 133282–133304."}, {"review_id": "Y00nxAlIDf", "reviewer": "Reviewer_7h7H", "summary": "The paper presents a novel parallelization strategy, dubbed PipeFusion, for efficient inference of Diffusion Transformer. The method partitions both input images and model layers across multiple GPUs, employing patch-level pipeline parallelism to mitigate communication overhead and leverage temporal redundancy. The approach is validated across various state-of-the-art diffusion Transformer models, demonstrating its superior performance in terms of latency, scalability and image quality.", "questions": "Please see the above **weaknesses** section.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The proposed patch-level pipeline parallelism is an innovative and effective approach.\n\n2. The experiment is extensive, covering multiple strong baselines incl. DistriFusion across various Diffusion Transformer architectures incl. Pixart, Stable-Diffusion.\n\n3. Experimental results consistently favor PipeFusion across different image resolutions and model scales.\n\n### Weaknesses\n\n1. The proposed framework is only tested under one system configuration: 8 x L40 GPUs. As the pipeline stages and GPU to GPU communication bandwidth could greatly impact the performance (FID score and throughput) of the proposed method. The authors are suggested to test under different system settings (e.g., communication bandwidth).\n\n2. It would be great to have more studies regarding the performance impact of stale representations, as this is one important reason of why the proposed PipeFusion has lower FID compared to DistriFusion, potentially by testing the FID score under different level of staleness.", "was_revised": true, "final_justification": "My concerns are resolved in the rebuttal.", "rebuttal": "Q1:1. The proposed framework is only tested under one system configuration: 8 x L40 GPUs. As the pipeline stages and GPU to GPU communication bandwidth could greatly impact the performance (FID score and throughput) of the proposed method. The authors are suggested to test under different system settings (e.g., communication bandwidth).\n\nThank you for your valuable feedback. We fully agree that system configuration, especially inter-GPU communication bandwidth, plays a critical role in the performance of parallel inference methods.\n\nTo address this concern, we have conducted additional experiments on an alternative hardware platform: 8×A100 GPUs connected via NVLink, which provides significantly higher communication bandwidth than PCIe. The results are summarized in the table below for the Pixart model:\n\n| Latency (sec)      | 1024px | 2048px | 4096px |\n|----------------|--------|--------|--------|\n| tensor parallel| 1.22   | 7.07   | 36.33  |\n| distrifusion   | 0.77   | 2.476  | 25.24  |\n| sp-ring        | 1.37   | 2.46   | 23.31  |\n| sp-ulysses     | 2.59   | 3.54   | 27.41  |\n| **pipefusion** | **0.66** | **2.59** | **22.39** |\n\nAs shown above, PipeFusion consistently achieves the lowest latency across all resolutions and remains robust even under high-bandwidth NVLink settings. While the relative performance gap narrows at higher resolutions due to the increasing dominance of computation cost (i.e., the shift from communication- to computation-bound regimes), PipeFusion remains competitive or superior across all configurations.\n\nWe would also like to clarify that the FID score is determined solely by the model architecture and inference algorithm, and remains consistent across different hardware settings as long as the number of GPUs and the generated outputs are the same. Therefore, changes in GPU type or interconnect bandwidth do not affect FID evaluation.\n\nQ2: It would be great to have more studies regarding the performance impact of stale representations, as this is one important reason of why the proposed PipeFusion has lower FID compared to DistriFusion, potentially by testing the FID score under different level of staleness.\n\nThank you for highlighting the importance of evaluating the impact of stale representations on diffusion performance. We plan to conduct controlled experiments that vary the level of activation staleness (e.g., by adjusting the freshness ratio or delaying KV usage across more steps), and measure the corresponding FID. This would help isolate the direct influence of stale feature reuse on output quality.\n\nSome works in our reference have begun exploring this space. For instance:\n\n- ∆-DiT [24] studies caching strategies for different DiT layers depending on the sampling stage;\n\n- Learning-to-Cache [13] analyzes layer-wise temporal redundancy and proposes selective reuse policies;\n\n- FrDiff [11] and DeepCache [12] observe that reusing high-level activations introduces minimal degradation while significantly accelerating inference.\n\nWe anticipate combining PipeFusion’s low-communication pipelined inference with more stale steps offers a two‑axis acceleration: reducing both inter-device communication and redundant per-timestep computation, with minimal FID degradation. We will add this point to the Discussion section."}, {"review_id": "omykIoMwte", "reviewer": "Reviewer_9xjo", "summary": "PipeFusion introduces a patch-level pipeline parallelism to accelerate inference for DiTs on multi-GPUs. It divides the input into multiple patches and maps the model layers across GPUs, enabling pipelined parallel processing. Also, PipeFusion leverages the high temporal redundancy between consecutive steps by reusing one-step stale feature maps to reduce the communication cost compared to existing parallelization methods such as tensor parallelism, sequence parallelism, and DistriFusion. Experiments on DiT models, including Pixart, Stable-Diffusion 3, and Flux.1, demonstrate substantial improvements in latency with minimal impact on image generation quality.", "questions": "In your experimental results, you note that for Pixart at 4096 px resolution, the performance gap between PipeFusion and SP narrows, and both methods exhibit similar scalability. While you attribute this to the exceptionally long sequence length making computation the dominant bottleneck rather than communication, the analysis remains relatively high-level.\n\nCould you provide a more detailed breakdown or profiling of the memory cost and computational bottlenecks at this resolution?\nSpecifically:\n1. Can you quantify how the memory footprint (for parameters, activations, and intermediate buffers) scales with resolution and sequence length for PipeFusion and SP?\n2. Is there a point at which the communication-to-computation ratio fundamentally shifts, and can you provide empirical or theoretical evidence for where this occurs?\n3. Have you profiled GPU utilization, memory bandwidth, or kernel efficiency at 4096px to pinpoint what limits further speedup for PipeFusion?\n\n[Significance] A more granular analysis of these aspects would help clarify the practical and theoretical limits of your approach and guide future work.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. It’s a novel approach to combine both patch-level and layer-level partitioning for DiT inference. This is distinct from prior work that primarily focused on sequence or tensor parallelism. \n2. The methods of this paper are easy to follow and adopt. The core concept can be simply handled by specifically writing the schedule in the compiler. The only additional thing to pay attention to is the input temporal redundancy.\n3. Authors provide a thorough comparison with prior parallelization methods and clearly articulate the scenarios where PipeFusion is most beneficial, as well as its limitations.\n\nWeaknesses:\n1. As the discussion of stability section, when computation becomes dominant, the benefit of PipeFusion’s communication reduction is less pronounced, e.g., Pixart 4096px.", "was_revised": true, "final_justification": "The authors have provided a detailed breakdown of the performance, which is very helpful. My main concerns regarding the significance have been addressed.", "rebuttal": "We sincerely thank the reviewer for the insightful and constructive comments.\n\nWe apologize for the misleading wording in the last paragraph of Sec. 4.1.1. Below, we provide clarifications and additional analysis to address your concerns:\n\nFirst, on the statement \"PipeFusion and SP exhibit similar scalability at 4096px\":\n\nWe must point out that the term \"similar\" is imprecise. In fact, PipeFusion still demonstrates clear advantages at 4096px. As shown in Figure 5 (8×L40), PipeFusion achieves an end-to-end (E2E) latency of 32.1s, while SP’s best configuration (Ulysses=2, Ring=4) reaches 37.31s, yielding a 1.16× E2E speedup. Despite entering a compute-bound regime at 4096px, PipeFusion still outperforms SP by 1.16× end-to-end — a strong indication of its robustness even under diminishing communication relevance.\n\nMoreover, analyzing the communication component itself reveals a stark difference: PipeFusion communication share = (32.1 - 244.89/8) / 32.1 = 4.6%. SP communication share = (37.31 - 244.89/8) / 37.31 = 17.9%.This shows that PipeFusion significantly reduces communication overhead even at high resolutions. We will revise the wording and replot the relevant figure to reflect this difference better.\n\nSecond, on the statement \"This is likely due to the exceptionally long sequence, where the communication overhead becomes less significant, thereby diminishing the advantage of PipeFusion’s communication efficiency.\":\n\nThe analysis is vague and overly high-level. We therefore provide a more detailed analysis of how PipeFusion’s speedup varies across different input resolutions—1024px, 2048px, and 4096px.\n\nFact 1: PipeFusion’s speedup for the communication component over SP (measured as SP comm. time / PipeFusion comm. time) becomes increasingly significant as resolution grows. For example, on 8×L40, at 1024px, PipeFusion reduces the latency of communication component to 54%(-46%); at 2048px, it is reduced to 24%(-76%); at 4096px, it is reduced to 22%(-78%). The PipeFusion’s speedup for the communication component becomes more pronounced at higher resolutions. This trend is attributable to longer sequences resulting in an increase in absolute communication time, and PipeFusion only benefits the communication component rather than the computing one.\n\n|                         | 1024px     | 2048px     | 4096px     |\n|-------------------------|------------|------------|------------|\n| **seqLen**              | 4,096      | 16,384     | 65,536     |\n| **8×L40 PipeFusion Latency (sec)** | 0.82       | 3.27       | 32.1       |\n| **8×L40 SP Latency (sec)**        | 1.22       | 5.07       | 37.3       |\n| **PipeFusion Comm. Time (sec)**   | 0.47       | 0.57       | 1.49    |\n| **PipeFusion Comm. Share**        | 57.3%      | 17.4%      | 4.6%       |\n| **SP Comm. Time (sec)**           | 0.87       | 2.37       | 6.69   |\n| **SP Comm. Share**                | 71%        | 47%        | 18%        |\n| **Comm. PipeFusion/SP**           | 54%        | 24%        | 22%        |\n| **Baseline (single GPU, sec)**    | 2.8        | 21.6       | 244.89     |\n\nFact 2: The communication-to-computation ratio decreases as resolution increases. In PixArt, FLOPs are dominated by self-attention (which scales quadratically with sequence length). In contrast, communication cost (based on PipeFusion Table 1) grows only linearly with sequence length. This causes the communication-to-computation ratio to decrease — at 4096px, it is 0.1690 that of 1024px.\nAs a result, although PipeFusion increasingly accelerates the communication component, this component constitutes a shrinking portion of total latency. For example, SP's communication share drops from 71% (1024px) to 47% (2048px) and 18% (4096px).\n\n| Input  | SeqLen  | Compute (GFLOPs) | PipeFusion Comm. (GB) | Comm/Compute Ratio (Normalized) |\n|--------|---------|------------------|------------------------|---------------------|\n| 1024px | 4,096   | 6,507.61         | 0.0175                 | 1.0000              |\n| 2048px | 16,384  | 51,952.85        | 0.0703                 | 0.5032              |\n| 4096px | 65,536  | 623,373.28       | 0.2816                 | 0.1680              |\n\nThese two facts explain the observed behavior: PipeFusion yields stronger absolute gains on the communication part as sequence length increases, but the overall end-to-end speedup becomes less dramatic since communication accounts for a smaller share of total runtime. We will clarify this nuanced trade-off in the revised manuscript.\n\n**For Q2**:  While PipeFusion consistently accelerates the communication component, the relative speedup on end-to-end latency diminishes because computation grows quadratically, whereas communication scales linearly. \n\nHowever, we emphasize that this trade-off is tied to GPU scale. If GPU numbers scale proportionally with compute (e.g., 2 GPUs for 1024px, 8 for 2048px, 32 for 4096px), the communication share remains stable. In such cases, PipeFusion's advantage becomes more pronounced, making it highly effective in large-scale deployments.\n\nThird, on profiling memory, GPU utilization, and kernel efficiency at 4096px. Using DCGM profiling, we summarize our findings in the following table:\n\n| Metric                          | 1024px   | 2048px   | 4096px   |\n|--------------------------------|----------|----------|----------|\n| Param mem. (GB)                | 0.15     | 0.15     | 0.15     |\n| Text encoder mem. (GB)         | 10.75    | 10.75    | 10.75    |\n| Stale KV buffer mem. (GB)      | 0.03075  | 0.123    | 0.492    |\n| Activations mem. (GB)          | 0.91925  | 1.047    | 1.578    |\n| Overall mem. (GB)              | 11.85    | 12.07    | 12.97    |\n| Latency (Sec)                  | 0.82     | 3.27     | 32.1     |\n| Mem Bandwidth Util.            | 5%       | 4%       | 2%       |\n| GPU SM Activity                | 37%      | 65%      | 97%      |\n| PipeFusion Comm / Latency      | 0.57     | 0.17     | 0.046    |\n| SP Comm / Latency              | 0.71     | 0.46     | 0.178    |\n\n**For Q1**: As shown in the above table, PipeFusion’s activation and stale-KV memory footprint remains low and does not pose a scaling bottleneck.\n\n**For Q3**: On 4096px case, the above table shows that GPU SM activity reaches 97%, while memory bandwidth utilization is only 2%. This indicates that the model is compute-bound, not memory-bound, at this resolution. Consequently, further speedup would require optimizing kernel execution, such as through improved CUDA kernels or quantization techniques. These compute-level optimizations are orthogonal to PipeFusion’s communication strategy and can be applied in combination to further enhance performance.\nIn summary, we appreciate your comments, which led us to refine both the empirical explanation and theoretical understanding. We will revise the manuscript accordingly to ensure the significance of PipeFusion is not underestimated."}]

Author rebuttal:

Meta-review: The paper received all positive reviews, leading to a final acceptance recommendation.
Decision: Accept (poster)