Revisiting Parameter Server in LLM Post-Training
Authors: Xinyi Wan
Penghui Qi
Guangxing Huang
Chaoyi Ruan
Min Lin
Jialin Li
Venue: ICLR 2026 Poster (2026)
Keywords: Distributed Training
ZeRO Optimizer
FSDP
Parameter Server
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We transform FSDP (ZeRO3) into a decentralized parameter server with on-demand communications, which relax the synchronization barrier from layer level to minibatch level, and achieves up to 36% acceleration on various LLM post-training settings.

Abstract: Modern data parallel (DP) training favors collective communication over parameter servers (PS) for its simplicity and efficiency under balanced workloads. However, the balanced workload assumption no longer holds in large language model (LLM) post-training due to the large variance in sequence lengths. Under imbalanced workloads, collective communication creates synchronization barriers, leading to under-utilization of devices with smaller workloads. This change in training dynamics calls for a revisit of the PS paradigm for its robustness to such imbalance. We propose **On-Demand Communication (ODC)**, which adapts PS into Fully Sharded Data Parallel (FSDP) by replacing collective all-gather and reduce-scatter with direct point-to-point communication. Compared to FSDP, ODC reduces the synchronization barrier from once per layer to once per minibatch and decouples the workload on each device so that faster workers are not stalled. It also enables simpler and more effective load balancing at the minibatch level. Across diverse LLM post-training tasks, ODC consistently improves device utilization and training throughput, achieving up to a 36\% speedup over standard FSDP. These results demonstrate that ODC is a superior fit for the prevalent imbalanced workloads in LLM post-training. Our implementation of ODC and integration with FSDP is open-sourced at https://github.com/sail-sg/odc.

Published as a conference paper at ICLR 2026
REVISITING PARAMETER SERVER IN LLM POST- TRAINING
Xinyi Wan 122 Penghui Qi Guangxing Huang', Chaoyi Ruan', Min Lin' & Jialin Li
Sea AI Lab National University of Singapore
ABSTRACT
Modern data parallel (DP) training favors collective communication over parame- ter servers (PS) for its simplicity and efficiency under balanced workloads. How- ever, the balanced workload assumption no longer holds in large language model (LLM) post-training due to the high variance in sequence lengths. Under im- balanced workloads, collective communication creates synchronization barriers, leading to under-utilization of devices with smaller workloads. This change in training dynamics calls for revisit of the PS paradigm for its robustness to such imbalance. We propose On-Demand Communication (ODC), which adapts PS into Fully Sharded Data Parallel (FSDP) by replacing collective all-gather and reduce-scatter with direct point-to-point communication. Compared to FSDP, ODC reduces the synchronization barrier from once per layer to once per mini- batch and decouples the workload on each device so that faster workers are not stalled. It also enables simpler and more effective load balancing at the minibatch level. Across diverse LLM post-training tasks, ODC consistently improves device utilization and training throughput, achieving up to 36% speedup over standard FSDP. These results demonstrate that ODC is a superior fit for the prevalent imbal- anced workloads LLM post-training. Our implementation of ODC and integra- tion with FSDP is open-sourced at https:://ittttt com/ sail-sol odc
INTRODUCTION
The development of DP distributed training (Krizhevsky) 2014 Goyal et al. 2017 Li et al. 2020)
has followed two main approaches: the PS architecture and collective communication. Harly large- scale systems such as DistBelief used the PS model to train deep neural networks across hetero- geneous hardware and networks with variable latencies (Dean et al. 2012). In this setup, servers stored the model parameters while workers handled computation, enabling asynchronous or loosely synchronous training that tolerated slower or unreliable machines. Later work expanded on this design by enabling different consistency policies and exploring elastic scalability with continuous fault tolerance (Li al. 2014). With the emergence of dense, homogeneous GPU clusters and high- bandwidth interconnects, collective communication became the mainstream approach for distributed DP. prominent advantage of this paradigm was the opportunity it created for communication- efficient algorithms. Ring-basce based methods, demonstrated in Baidu AllReduce (Research) (017) and Horovod (Sergeev & Del Balso 2018), reduced bandwidth requirements while scaling pre- dictably. This trend was further reinforced by vendor-optimized libraries like NCCL (NVIDIA) b). which made high-performance collectives broadly accessible and easy to integrate into modern training frameworks. fundamentally relies on balanced workloads. This presumption was largely valid for many domi- nant deep learning domains, including vision, speech, and early NLP. As result, the dependency on workload balance was frequently taken for granted or neglected in system design.
important note that the high efficiency of collective communication
Recently, the post-training of LLMs (Ouyang et al. 2022 Guo assumption of balanced workloads that collective communication relies on. Real-world text corpora contain sequences of widely varying lengths (Bai et al. 2024| Yang et al. 2025). As the cost of attention grows quadratically with sequence length (Vaswani et al. (017)) while activation memory grows linearly, this variation leads to persistent computational imbalance across devices. Although
2025) breaks the long-standing
"Equal Contributors

---

Published as a conference paper at ICLR 2026
line of work has focused mitigating this issue with sophisticated packing strategies (Krell al, 2021 Kundu et al. 2024 Yao et al., 2025: Wang et al. 2025). these methods can only reduce the skew, but cannot remove entirely, especially under memory constraints that force minibatches to be split into smaller microhatches (Huang et al. 2019, Qi et al. 2024). This not only narrows the solution space for effective packing, but also increases the number of synchronization points, further amplifying the inefficiency due to imbalanced workloads.
This inefficiency from workload imbalance is particularly severe in contemporary sharded DP, ex- emplified by ZeRO (Rajbhandari et al. 2020) and PyTorch's 's FSDP (Zaao et al. 2023). By sharding parameters, gradients, and optimizer states across devices, FSDP enables memory-efficien scaling trillion-paammeter to trillion parameter models, making it the standard choice for LLM post-training and reinforcement
learning (RI.) pipelines (Hu al. 2024 Sheng et al. 2025 Fu al. 2025 Liu et al. 2024). How- ever, this memory efficiency comes at the cost of increased synchronization (Figure FSDP relies heavily on collective communication: per-layer parameters are reconstructed via all all-gather before the forward pass, and gradients aggregated via reduce-scatter after the backward pass. This fine-grained, layer-level synchronization implicitly assumes balanced workloads, which is precisely the assumption violated in LLM post-training. Our evaluation shows that even with tate-of-the-ar packing strategies, workload imbalance can still result in device idle times of up to 50% during long-sequence supervised fine-tuning (see Table
To bridge the gap between fine-grained synchronization and workload imbalance LLM post- training, revisit the PS idea, and adapt to the modern sharded DP paradigm through On- demand Communication (ODC). We replace the per-layer collectives with point-to-point primi- tives, allowing devices fetch parameters and push gradients independently (Figure 2). This frames FSDP as decentralized PS where server and worker roles are colocated, thus preserving its memory and scaling advantages. While preserving the synchronous optimization semantics, we relax synchronization from the layer level to the minibatch level. This decoupling of device progress significantly mitigates straggler effects and enables a more flexible space for workload balancing.
In summary, this paper presents a novel perspective: compared to collectives, the PS architecture is naturally better suited for LLM post-training due its tolerance for heterogeneous workloads. To retain the key benefits of modern DP schemes, do not build standalone PS. Instead, propose ODC, a communication scheme that brings the workload-tolerane evaluation demonstrates that ODC substantially improves device utilization and end-to-end through- put across diverse LLM post-training tasks, including supervised fine-tuning (SFT) and RL, achiev- ing up 36% speedup conventional FSDP.
of classic PS into FSDP. Our
Device
F
B
F
B
B
B
Device
F
B
B
F
B
B
B
Time
Figure Collective communications introduces per-layer synchronization barriers FSDP.
Device 0
F
B
B
B
B
B
B
Time Saved
Device
F
Time
B
B
B
Forward/backward of consecutive layers
Optimizer Step
Device Idle
Figure 2: On-demand communications relaxes the synchronization barriers to minibatch end.
BACKGROUND
2.1 MINIBATCH, MICROBATCH AND GRADIENT ACCUMULATION
In deep learning, a minibatch refers to the set of training samples processed single optimizer step. However, training LLMs often exceeds the memory capacity required process the desired
2

---

Published as a conference paper at ICLR 2026
minibatch in wackwrrdddb pass. A common remedy is to divide the minibatch into M mi- crobatches and accumulate gradients before performing the optimizer update. For each microbatch m ...... M compute the forward and backward passes to obtain per-parameter gradients g(m)) and then accumulate
M 1m= Wm g!m),, where Wm encodes the aggregation policy (e.g.,
Wm 1 for summation, or proportional weighting when averaging by tokens or samples).
2.2
SYNCHRONIZATION BARRIERS FSDP
All-gather
Reduce-scatter
Before
After
Before
After
Device 0
Paramo
Paramo
Param1
Grado
Grad1
0+
Device 1
Param1
Paramo
Param1
Grad2
Grad3
1+3
Figure 3: all- -ather and reduce-scatter
In FSDP, both parameters and gradients partitioned across devices. FSDP primarily uses all- gather to materialize parameters and reduce-scatter to aggregate gradients. The mechanics of reduce-scatter and all- all-gather are illustrated in Figure
The communication pattern unfolds as follows. During the forward pass, before computation on a specific layer begins, its full parameters reconstructed each device via all-gather operation. These reconstructed parameters are then discarded immediately after use to save memory. A similar all-gather process occurs during the backward pass. Additionally, after gradients computed for layer, they are aggregated and distributed using a reduce-scatte operation, leaving each device with only its corresponding shard of the total gradient. The overall communication flow is shown in Figure 4 In practice, modern implementations overlap these communications with computation (e.g., pre-fetching parameters for the next layer during the current layer's 's execution) hide the latency, but this overlap does not remove the underlying synchronization points.
Layer forward
Idle
Layer backward
Idle
Device
AG
AG
RS
Device
Layer forward
Layer backward
layers
layers
Figure 4: Communication pattern of FSDP within microbatch. The left panel shows forward communication (all-gather parameters), and the right shows backward communication (all- (all-gather parameters & reduce-scatter gradients). AG = all- all-gather; RS reduce-scatter
These per-layer collectives create fundamental synchronization barriers that are the root cause of inefficiency under imbalanced workloads. All devices must complete the all-gather before layer's forward computation can begin, and they must all complete the reduce-scatter before gradient accu- mulation can proceed. This tight coupling forces devices advance at the same pace, meaning faster devices must idle and wait for the slowest one before moving to the next layer.
More formally, let batching solution PM specify the assignment of training samples to M mi- crobatches each device. Denote by Tm.a:((MM) the time to execute layer I of microbatch on device under Par- For model with L layers, the minibatch runtime bounded by the slowest device at each per-layer step:
M
T(PM) Σ Σ max Tm.a, (Par).
significant body of research has focused on finding an optimal batching solution, P*, that mini- mizes T(PM). However, detail in Section these approaches face fundamental limitations.

---

Published as a conference paper at ICLR 2026
3 ON-DEMAND COMMUNICATIONS
To address the inefficiency of FSDP caused by imbalanced workload, we step back from the pre- vailing focus on complex batching strategies and re-examine first principle of data parallelism: per-device computations are independent. Standard FSDP violates the spirit of this independence by using collective communication, which imposes fine-grained synchronization barriers. These barriers, which force devices to wait for the slowest one, are the direct cause of idle time. They are an artifact of the communication model, not requirement of the training algorithm itself, and are therefore fundamentally avoidable.
To address this root cause, we propose ODC, new communication scheme that relaxes synchro- nization to much coarser granularity without altering the training semantics (Figure D. ODC preserves FSDP's memory layout and computational graph but replaces its synchronous collectives with point-to-point operations. Specifically, we decompose the collective calls. An all-gather is replaced by a series of targeted gather requests, where a device fetches only the specific parameter shards it needs from its peers. Similarly, reduce-scatter is broken down into a series of scatter- accumulate operations, where device pushes its computed gradients directly to the devices that own the corresponding gradient shards. This process is illustrated in Figures With ODC, each de- vice operates independently, fetching parameters or pushing gradients as soon as it is ready, thereby eliminating the synchronization-innueed stalls.
critical feature of ODC is that these point-to-poin data transfers are non-intrusive. When one device initiates gather or scaater--ccumullte request to another, it does not interrupt the ongoing
computation on the target device. We show how this is enabled in Section 3.2
Before
After
Before
Acc
Accumulate to
Paramo
Param1
Grad1
GradO
Device
Param1
Param1
Grad1
Accumulate to
Gather (to device 0)
Scatter-accumular (from device 0)
Figure 5: gather and scatter-accumulate.
3.1 ODC AS A DECENTRALIZED PARAMETER SERVER
The classic PS architecture Dean al. 2012 Li et al. 2014) separates model state from model computation, where a set of server nodes is responsible for storing the model's parameters and optimizer states. Meanwhile, set of worker nodes pulls parameters from the servers, performs the forward and backward computations on its local data, and then pushes the resulting gradients back to the servers. The servers then aggregate these gradients and apply the updates. This design decouples the progress of individual workers and provides natural tolerance for stragglers, which a key advantage for the imbalanced workloads common in LLM post-training.
As shown in Figure 6 ODC paradigm reframes FSDP as a modern, decentralized PS. Instead of using dedicated server nodes, we colocate the server and worker roles by evenly partitioning param- eters, gradients, and optimizer states across all devices. Each device acts as a server by owning and managing a shard of the model's parameters and optimizer state. Simultaneously. acts as a worker by executing the forward and backward passes its assigned data. This decentralized, co-located design mirrors the memory layout of FSDP and avoids the network bottlenecks of a centralized PS. While colocated roles has precedent in some PS systems (Tiagg al. 2020), approach is novel in its direct integration with FSDP's sharding mechanism.
Ultimately, by replacing FSDP's pay-layer collectives with on-demand point-to-poin communica- tion, our method gains the imbalance tolerance of a PS while retaining the core benefits of FSDP: memory efficiency, decentralization, scalability, and simplicity.
4

---

Published as a conference paper at ICLR 2026
Data
Gather
2. Calculation
Workers
3. Scatter-accumllate
Gradients
Parameters
Parameter Servers
Optimize) Step
4. Optimizer Step
Optimizer States
Device 0
Device N 1
Figure 6: The architecture of FSDP with ODC, in which FSDP be decentralized parameter server, with server part and worker part highlighted in this figure.
3.2 IMPLEMENTATION
ODC workers often push pull data servers while colocated workers concurrently perform com- putations, making it essential to minimize server interference. Communication primitives must also support ODC's on-demand nature, where workers control the flow and servers cannot anticipate requests. Existing message-based libraries like MPI (Gabriel et al., 2004) and NCCL/NVIDIA b) require explicit, ordered participation from both sender and receiver, making them neither transpar- ent nor on on-demand, and prone to deadlocks if not carefully scheduled.
ODC instead leverages native RDMA-based interfaces: CUDA IPC (NVIDIA) a) for intra-node and NVSHMEM (NVIDIA c) for inter-node communication. RDMA enables transparent data trans- fers without active server involvement, except for gradient accumulation, which is handled by lightweight daemon. The communication kemnel is built on Triton-Distributed (Zheng et al.) 2025), a Triton (Tillet et al. 2019) wrapper that exposes RDMA functionalities directly in Python Triton kernels, eliminating the need for low-level CUDA C code. We put more implementation details at Appendix B and will open-source our implementation community usage.
Integrating ODC into FSDP is straightforward: it only requires replacing collective communication calls with ODC primitives and retrieving accumulated gradients the minibatch end.
SIMPLIFIED LOAD BALANCING WITH ODC
Due the variation in sequence lengths, naive padding strategy significantly suffers from compu- lation waste. To mitigate this, Krell et al. (2021) introduced the strategy of sequence packing, which concatenates multiple samples into single sequence with appropriate attention masks, improving utilization and balancing workload across microbatches. This approach has been broadly adopted and extended by subsequent work (Bai et al. 2024 Kundu et al., 2024 Yao al. 2025 Wang al.,
2025). with efficient support in modern libraries like FlashAttention (Dao et al. 2022 Dao 2023).
However, existing sequence packing methods operate the microbatch level, which faces several fundamental limitations under FSDP. First, the size of a microbatch is bounded by device memory, limiting the number of samples per microbatch and leaving substantial variance in workload across devices. This effect is amplified in long-sequence training regimes, such LongAlign (Bai et al.) 2024) and RL for LLM reasoning (Guo et al. 2025), where extended contexts further constrain der-ievice capacity. Second, for sample of sequence length activation memory typically scales as (ss) while runtime scales as O(s²) (e.g., due to attention), creating a fundamental mismatch between memory and compute. Consequently, compute alignment can be infeasible under memory
5

---

Published as a conference paper at ICLR 2026
constraints. For instance, if microbatch contains single sample at the maximum sequence length, no feasible packing of shorter samples can match its runtime.
By replacing collective operations with ODC, approach decouples the execution of microbatches across devices. This eliminates synchronization barriers inherent in FSDP and removes the implicit requirement for uniform number of microbatches per device. This insight allows for significant simplification workload balancing strategy. Specifically, our strategy shifts the balancing objec- tive from the fine-grained microbatch level to the coarser minibatch level. We first partition the global set of training samples across devices with the sole goal of balancing the total computational load. Subsequently, each device independently packs its local subset of samples into microbatches, governed only by its local memory constraints. This shift in granularity not only simplifies the pack- ing algorithm, but also achieves superior load balancing by operating on a larger, less constrained set of samples. We leave the detailed packing algorithms in Appendix C
5 EVALUATIONS
5.1 SETUP
We evaluate ODC on two major LLM post-training tasks: SFT and RL. For SFT, use a) Lon- gAlign (Bai et al. 2024). dataset for extending LLM context windows, and b) open-source trajec- tories from SWE-Smith YYang et al. 2025). an agent model for software engineering tasks released by the SWE-Bench team (Jimenez et al. 2023). For RL, we run GRPO (Guo et al. 2025 Liu et al.) 2025) implemented in verl (Sheng et al. 2025) on AIME prompts (Li et al. 2024), ) which includes
problems from Olympiad-levell math contest. Notably, we only record the model training time in RL,
ignoring forward-on -nly parts like actor rollout. The sequence length distributions of these datasets shown in Figure
LongAlign
SWE-Smith
AIME (Prompt Response)
Peeguoncy
Mean 16540
Mcan 34724
FRo cny
Mcan 9066
10000 2000 30000 40000 50000
60000
10000
20000
30000 40000
50000 60000
5000
10000
15000
20000
25000
Sequenee Length
Sequence Length
Sequencce length
Figure 7: Sequence length distributions of evaluation datasets.
We evaluate ODC on the DeepSeek- R1 --is--Dtll--UUS -wenn family of models (Team 2024 Guo et al. 2025), with varying size from 1.5B to 32B. The models are trained on up to 32 NVIDIA A100 80G GPUs, with NVSwitch for intra-node communication and RoCE RDMA (800 Ghps per node) for inter-node communication. Notably, for RL experiment we run only up to 14B model using 16 GPUs, the inference time would be too long for 32B model. Additionally, we validate the correctness of ODC by verifying the training convergency in Appendix
Each method in our evaluation is a combination of communication scheme and load balancing algo- rithms. For communication scheme, we have a) Collective baseline using collective all- all-gather and reduce-scatter; b) ODC our approach introduced in Section 3, For load balance algorithms, we include a) LocalSort adapted from Bai et al. (202)); within each device's minibatch, sequences are sorted by length but not packed. b) L.-MMicr- a heuristic- bbased packing baseline designed mini- mize workload imbalance across devices within the same microbatch. In RIL experiments, we show that it substantially faster than the native implementation verl (Sheng et al. (2025), underscoring its effectiveness as strong baseline. c) LB-Mini our algorithm introduced in Section which bal- ances workload at the minibatch level. As LB-Mini can produce different number of microbatches for different devices, it applies only to ODC. Detailed implementations
be found in Appendix C Unless otherwise specified, the maximum number of tokens in a microbatch is constrained by the maximum sequence length of a single sample in the dataset.

---

Published as a conference paper at ICLR 2026
Model, Devices
Madel, Devices SFT-LongAlign
32 Devices SFT-LongAlign
Samples Secord
SFT-LongAlign
Samples per Socordd
Samples par Secord
SFT-LangAlign
DDC LEatil
DECL
22-
Caliniiee L8
Calecd Li
300
Sulbo lee
102-
Dubisive
Edlind Lo
13
200
50
20
Number Samples mimbatch
Number Samples Minibatch
Number Samples Miribatch
Number samples Minbatcch
1.58 Model, 8 Devices
75 Model. 8 Devices SFT-SWE-Smith
148 Model. 16 Devices SFT-SWE-Smith
328 Model, 32 Devices SFT-SWE-Smith
3FT-5WE-Smith
Summp per Secordd
200
Samo Nes par Socira
coc quat on
socia stuo
DECLa Navo
Doc Mere
Ube Lendion
dic Locatin
Samp DAT
Ooe Locqler
100
40-
85-
Number Samples Minibatch
Number of Samples Miribatch
Number of Samples Miribatch
Number Samples Minbatch
Figure 8: Samples per second on SFT datasets (LongAlign and SWE-Smith) across different model scales and minibatch sizes. ODC consistently improves throughput over Collectives in both un- packed (LocalSort) and packed (LB-Micro, LB-Mini) scenarios.
1.5B Model, 8 Devices RL-AIME
7B Model, 8 Devices RL-AIME
14B Model, 16 Devices RL-AIME
Samples per Second
Samples per Second
Sampless Second
200
1000
-9
GDC Min
DDC Mi
400
GBC 1p-Wer
GBC LB Min
GBC LD-Mtre
GBC LB-18N
800
Colleciie Notive
Collcctve L3-Mero
300
Collccire harhee
Colective Li-tern
150
Collectiva hative
Collectoe Li-Heen
100
200
16
16
2
8
16
Number of Samples in Minibatch
Number Samples Minibatch
Number of Samples Minibatch
Figure 9: Samples per second on RL with AIME prompts. In addition the methods in Section 5.1 we also evaluate the default load balancing algorithm in verl, denoted as Native, LB-Micro is sub- stantially faster than Native, underscoring effectiveness as strong baseline.
5.2
MAIN RESULTS
Figure presents the evaluation results on SFT tasks. ODC consistently improves throughput over the collective baseline both unpacked (LocalSort) and packed (LB-Micro, LB-Mini) settings, with the most pronounced gains observed under packing, reaching up 36% speedup. All methods perform similarly when the minibatch size one, since in this case ODC synchronizes after every sample, just like collective.
Figure shows in RL tasks ODC achieves up to 10% speedup over collective baseline, although the gains are less pronounced than in SFT. This is primarily due to: a) implementation constraints in verl, which require identical numbers of samples per device and thus limit the effectiveness of LB- Mini. While relaxing this constraint is feasible, we did not do so, the current solution is easier to integrate; and b) a less long-tailed sequence length distribution compared to SFT datasets (Figure 7
At small minibatch sizes, LB-Mini often outperforms LB-Micro. This reflects the benefits of its minibatch-level balancing, which permits devices to process different numbers of microbatches. As the minibatch size increases, however, LB-Micro has more flexibility balance workloads effec- tively, which narrows the performance gap between the two methods. The detailed timing data as well as bubble rate reported in Appendix G
5.3 PARAMETRIC STUDY
The effectiveness of ODC compared to collectives depends on several factors: a) Minibatch size: the number of samples per minibatch per device; b) Max length: the maximum sequence length in the dataset; to control this factor while maintaining the overall distribution, we adjust each sample by uniformly truncating repeating tokens fixed ratio; e) Packing ratio: the maximum number
7

---

Published as a conference paper at ICLR 2026
of tokens allowed in microbatch divided by the max sequence length (e.g., with max sequence length of 16K and packing ratio of 2, a microbatch may contain up to 32K tokens); d) Devices: the total number of devices.
To isolate the impact of each factor, we adopt a controlled methodology: starting from a fixed golden setting (Table l]. vary one factor time while holding others constant. As shown in Figure To the acceleration ratio peaks at moderate minibatch sizes before declining as larger batches give the baseline more flexibility; increases with sequence length, since longer sequences amplify the quadratic compute cost and exacerbate imbalance; it decreases with packing ratio, which improves the baseline's 's packing efficiency; and grows with the number devices, as more devices introduce greater heterogeneity.
Model
Dataset
minibatch Size
Devices
Packing Ratio
1,5B
LongAlign (Max 64K)
4
Table 1: Golden setting for the parametric study. Each experiment varies at most one factor.
Acceleration Ratio on Minibatch Size
Accetoration (30
Acceleration Ratio on Max Length
ODC + LB Mini
20
GDC LB Nin
20-
ODC L.BMicro
GDC MR-Mirr
20
Longth
Accel eration Ratio Packing Ratio
Acceleration Ratio on Devices
Acceleration
20
DDC Micro
20-
30-
16
32
Packing Ratto
Devites
Figure 10: Acceleration ratio of ODC compared to collective with LB-Micro in parametric study,
5.4
BENCHMARK ON COMMUNICATION PRIMITIVES
Payload: 1 Layer of 1.5B Model
Payload: 1 Layer of 7B Model
Bandwicth (GBrs)
Bandwidth (GB/S)
Scottor 10DCI
Scottor Accumulsto (CDC) Heduce-Scallie (Callecive) Gather (0DC)
150
Seduce-Scailer (Collcciv))
100
Alt-Gateer (Colecct
Gather (SDC)
100
All-Gather (Co lecivel
16
32
16
32
Payload: Layer of 14B Model
Payload: Layer of 32B Model
Bandwiath (Gais)
Bandwiath (GBI5)
Seat-ee-rrummdiee (9OC)
200
Sentieederrmmalet (002)
(calecive)
Scotter(Callcchee
(D05)
100
100
16
16
Number of Devices
Number of Devices
Figure 11: Benchmarking communication primitives against collectives. Within a node, ODC has comparable performance with collective. But significantly slower than collective cross node.
We compare the bandwidth of ODC primitives (gather and scatter-accumulate) against collectives (all-gather and reduce-scatter) in NCCL. For fairness, ODC primitives are launched synchronously: each device issues operations in the same order, with barriers inserted before and after each primitive. Results are shown in Figure II Within a single node (up to 8 devices), ODC achieves bandwidth comparable to collective. However, once communication spans multiple nodes, ODC lags signifi- cantly behind collective. We leave more discussion and how to mitigate this inter-node inefficiency in Section
8

---

Published as a conference paper at ICLR 2026
6 DISCUSSION
6.1 CHALLENGES ON INTER-NODE COMMUNICATION EFFICIENCY
Collective primitives often highly optimized by exploiting hierarchical interconnects in multi- node settings. For example, an all-gather operation might first perform an inter-node broadcast followed by an intra-node broadcast to minimize costly inter-node traffic. ODC does not increase communication volume, but changes the topology: it uses point-to-poing RDMA and thus forgoes these hierarchical optimizations (see Appendix DD. However, we argue that larger DP scale typically amplifies straggler effects under imbalance, increasing the benefit of ODC's decoupled progress (see Figure 10P. Furthermore, several ways can effectively mitigate this communicate overhead.
Overlapping Communication with Computation. ODC retains the standard FSDP optimization of overlapping communication with computation. This is particularly effective because commu- nication volume per microbatch is constant with sequence length (s), whereas computation scales as O(s²). For long sequences, the large computational cost effectively hides the communication latency. Consequently, despite using non-hierarchical communication pattern, ODC shows no significant slowdown in our long-context evaluations (see Section 5.2)
Hybrid Sharding. When the tokens per microbatch is too small hide communication costs, hybrid sharding provides an effective solution. Similar to ZeRO++ (Wang et al., [024), parameters and gradients are sharded only within a node, while optimizer states remain sharded across nodes, This design eliminates cross-node parameter gather and gradient scatter-ac ccatter-accumulatee at the cost of higher per-node memory usage, which is manageable trade-off given that activation memory requirements are lower. As shown in Appendix this strategy effectively mitigates ODC's additional overhead.
6.2 FUTURE WORK
ODC is an initial effort toward adapting PS to modern sharded DP. We believe this is a foundational step that opens several promising directions for future research.
ODC-specific Optimizations While our current ODC implementation uses direct point-to-pon tont-t communication. its communication graph be further optimized. For instance, a device could fetch parameter shard from a peer on the same node that has already cached it, effectively creating hierarchical communication path similar to topology-aware collectives.
Relaxing Synchronization Guarantees Our current design intentionally preserves a synchronous update the minibatch boundary to maintain identical training semantics. However, this barrier could be relaxed. Extending ODC to support classic asynchronous SGD schemes (Recht er al.) 2011] such as bounded-stalenes, updates (Chen et al. 2016 Ho et al. 2013), could further reduce idle time and improve hardware utilization, particularly in highly heterogeneous environments. This would, however, require a careful analysis of the convergence implications for LLM training.
Elasticity and Fault Tolerance A significant advantage of PS-style architectures is their natural support for elasticity and fault tolerance (Dean al., 2012, Li et al. 2014). Collective-based sys- tems, in contrast, are notoriously brittle and difficult to resize (Jiang et al. 2020 Narayanan et al.) 2021 Duan et al. (024).. Integrating these capabilities into ODC would improve the resilience and flexibility of large-scale, long-running LLM training jobs.
7 CONCLUSION
This paper revisits PS and adapts its principles to solve a critical bottleneck in modern sharded DP training for LLM post-training. We identified that the per-layer all- all-gather and reduce-scatter collectives in FSDP create fine-grained synchronization barriers, which amplify the straggler effeets caused by workload imbalance.
We proposed ODC to replace these collectives with point-to-point operations, effectively relaxing synchronization from the layer level to the minibatch level. This approach, which reframes FSDP decentralized PS, decouples device execution and enables more effective load balancing. Em pirically, ODC delivers consistent throughput and utilization gains across a range of Long-sequence SFT and RI. tasks.

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our experiments, we open-source our implementation, including: a) the core communication library of ODC, and b) the code patch that integrates ODC into FSDP at https://github. com/ sail hail-sq/ooe -sg/odc.
REFERENCES
Yushi Bai. Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. Longalign: A recipe for long context alignment of large language models. arXiv preprint arXiv:2401.1805 2024.
Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting dis- tributed synchronous sgd. arXiv preprint arXiv: (ete::66400081,,, 2016.
Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 16344 2022.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Mare' aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012.
Jiangfei Duan, Shuo Zhang, Zerui Wang, Lijuan Jiang, Wenwen Qu, Qinghao Hu, Guoteng Wang, Qizhen Weng, Hang Yan, Xingcheng Zhang, et al. Efficient training of large language models on distributed infrastructures: survey, arXiv preprint arXiv:2407.20011 2024.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, et al. Areal: A large-scale asynchronous reinforcement learning system for language reasoning. arXiv preprint arXiv:2505.244990 2025.
Edgar Gabriel, Graham E Fagg, George Bosilca, Thara Angskun, Jack J Dongarra, Jeffrey M Squyres, Vishal Sahay, Prabhanjan Kambadur, Brian Barrett, Andrew Lumsdaine, et al. Open mpi: Goals, concept, and design of next generation mpi implementation. In European Parallel Virtual Machine/Message Passing Interface Users' Group Meeting, pp. 97-104. Springer, 2004.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint (tti:::77060066771 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint (50.::250/.19444 2025.
Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B Gibbons, Garth A Gibson, Greg Ganger, and Eric P Xing. More effective distributed mi via stale synchronous parallel parameter server. Advances in neural information processing systems, 26, 2013.
Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, et al. Openrlhf: An easy-to-use, scalable and high-perffrmance rihf framework. arXiv preprint arXiv:2405. 11143, 2024.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee. Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems. 32, 2019.
Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, and Chuanxiong Guo. unified architec- ture for accelerating distributed (DNN) training in heterogeneous (GPU/CPU) clusters. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 463-479, 2020.
10

---

Published as a conference paper at ICLR 2026
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023.
Narendra Karmarkar and Richard M Karp. The differencing method of set partitioning. Computer Science Division (EECS), University of California Berkeley, 1982.
Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination:: Accelerating large language models without impacting per- formance. arXiv preprint arXiv:2107.02027 2021.
Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint ArX1v:144..5997, 2014.
Achintya Kundu, Rhui Dih Lee, Laura Wynter, Raghu Kiran Ganti, and Mayank Mishra. Enhancing training efficiency using packing with flash attention. arXiv preprint arXiv:2407.09100 2024.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9, 2024.
Mu Li, David G Andersen, Alexander Smola, and Kai Yu. Communication efficient distributed machine learning with the parameter server. Advances in neural information processing systems, 27, 2014.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006. / 15704, 2020.
Zichen Liu, Changyu Chen, Xinyi Wan, Chao Du, Wee Sun Lee, and Min Lin. Oat: A research- friendly framework for Ilm online alignment. https: /ttgtttttttttt////// com/ saillalllsse mall-sg/oatt 2024.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20 207883 2025.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Effi cient large-scale language model training on gpu clusters using megatron-lm. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. 1-15, 2021.
NVIDIA. Cuda c++ programming guide.
http:://oos..niiiia.oomm cuda/
a. n.d. NVIDIA. Nvidia collective communications library (neel).
cuda C- www.raaming-guuuuu......
ht:ps::/ddeveooper..wvv
com/ncel b. n.d.
NVIDIA. Nvidia openshmem library (nvshmem) documentation. https://docs.ni
com/ homm//vshhmee//aii/ineee. htmml n.d.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744 2022.
Penghui Qi, Xinyi Wan, Guangxing Huang, and Min Lin. Zero bubble (almost) pipeline parallelism. In The Twelfth International Conference on Learning Representations, 2024.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEFE, 2020.
11

---

Published as a conference paper at ICLR 2026
Benjamin Recht, Christopher Re, Stephen Wright, and Feng Niu. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. Advances in neural information processing systems, 24, 2011.
Baidu Research.
Baidu allreduce.
http:s//ggthii...mmmmiiidrrrrrrr
baicu-a a Treduce 2017.
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in tensor- 2018.
flow. arXiv preprint /
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.
2024.
Philippe Tillet, Hsiang-Tssin Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10-19, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Guanhua Wang, Heyang Qin, Sam Ade Jacobs, Xiaoxia Wu, Connor Holmes, Zhewei Yao, Samyam Rajbhandari, Olatunji Ruwase, Feng Yan, Lei Yang, et al. Zero++: Extremely efficient collective communication for large model training. In The Twelfth International Conference on Learning Representationss 2024.
Zheng Wang, Anna Cai, Xinfeng Xie, Zaifeng Pan, Yue Guan, Weiwei Chu, Jie Wang, Shikai Li, Jianyu Huang, Chris Cai, et al. Wib-llm: forkload-balanced 4d parallelism for large language model training. arXiv preprint 17X1v:2503.17924, 2025.
John Yang. Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint 2rXiv:2504.21796 2025.
Yongqiang Yao, Jingru Tan, Kaihuan Liang, Feizhao Zhang, Yazhe Niu, Jiahao Hu, Ruihao Gong, Dahua Lin, and Ningyi Xu. Hierarchical balance packing: Towards efficient supervised fine- tuning for long-context 1lm. arXiv preprint arXiv:2503.0768) 2025.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences scaling fully sharded data parallel. arXiv preprint arXiv:2304. /277, 2023.
Size Zheng, Wenlei Bao, Qi Hou, Xuegui Zheng, Jin Fang, Chenhui Huang, Tianqi Li, Haojic Duanmu, Renze Chen, Ruifan Xu, et al. Triton-distriuted:: Programming overlapping kernels on distributed ai systems with the triton compiler. arXiv preprint arXiv:2504. /9442, 2025.
12

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
LLMs were used to polish the writing of this paper and to assist in generating code for producing graphs used to present the evaluation results.
B IMPLEMENTATION DETAILS OF ODC
For intra-node communication, use CUDA-IPC (VVIDIA! a), which supports native read/write operations remote GPU tensors as if they were local. As result, no custom GPU kernels are required for intra-node communication. For inter-node communication, we implement a custom kernel using the put_mem and get_ment mem primitives provided by Triton-Distriuted (Zheng et al., 2025).
The implementation of gather is straightforward: each rank pulls data from all other ranks using get_mem. Empirically, we find that limiting the communication payload per transfer helps stabilize RDMA traffic, server may receive RDMA read requests from multiple clients simultaneously.
The implementation of scatter-accumulate slightly more involved. After a worker pushes data to server using put .mmm it notifies the server the same RDMA channel. The server runs a lightweight daemon process that polls for notifications and performs gradient accumulation upon receipt. As the polling does not occupy GPU SMs, worker (compute) process. Because server can receive concurrent pushes from multiple clients, we allocate dedicated buffer for each client to enable parallel data transfers and maximize throughput. Since requests from any single client are serialized, only one buffer per client is required. This design bounds the buffer memory each server M/ / per client, resulting total of M/N x N M per server, where N is the number of GPUs and M is the number of elements in a transformer layer.
observable slowdown of the colocated
C SEQUENCE PACKING STRATEGIES USED IN EXPERIMENT
We use the Karmarkar- Karp algorithm (Krrmarkr & Karp 1982 balance computational work- loads by solving the number partitioning problem! Our approach builds on the implementation in the Verl framework (Sheng et al. 2025) but adds crucial modification to prevent out-of-memory (OOM) errors. We modify the microbatch, partition partition is memory-feasioe before accepting memory-constrained settings. The key implementation details are provided in Listing [
setrrbbachppartttionn function to iteratively validate that any proposed solution. This ensures robust execution even in
C. LB-MICRO AND LB-MINI
As detailed in Section 5.1 we compare two primary load-balancing strategies: LB-Micro and LB- Mini. LB-Micro performs workload balancing at the microbatch level, adhering to the conventional constraint that all devices must process an identical number microbatches. In contrast, LB-Mini, enabled by ODC framework, balances the workload at the coarser minibatch level. The funda- mental advantage of LB-Mini is that it removes the rigid constraint on the number of microbatches per device, allowing for more flexible and effective load distribution. We highlight the implementa- tion differences between these two approaches Listing
C.2 VERL NATIVE TWO-LEVEI. PARTITIONING STRATEGY
The packing method in the Verl framework is subject to two main constraints. First, assumes that the number of training samples assigned to each device is the same. Second, because of layer- level synchronization, all devices must process an equal number microbatches. For these reasons, Verl uses two-level hierarchical heuristic approach, the implementation of which can be found in
Listing 2
13

---

Published as a conference paper at ICLR 2026
C.3 OPTIMIZED TWO-LEVEL PARTITIONING STRATEGY
The native partitioning strategy in Verl is suboptimal because it balances workloads at the global batch level, prior to splitting the data into minibatches. This approach fails ensure balance within each individual minibatch. To correct this, we optimize the implementation by first partitioning the data into minibatches and then performing load balancing across devices for each minibatch. This reversal yields substantial throughput improvements, shown in Figure Our optimized implementation detailed in Listing
Listing 1: Helper functions.
def. karmarkar_karp
compute__ose costs: List [int],
Input list of computational costs The target number of partitions If true, enforce equal size
k partitions: int,
equal_ size: bool
Returns the partitions indexes """Split input into partitions to balanco workload"""
[Lstt [int]]:
def get_compute_cone aeellen_sst: List[int)] -> List [int]: ""*Get the compute costs given the sequence lengths, "n"
det (omccooosseeeen seqeeeeeeeeeeeeeeeeeeee :st: List Listlint]) -> int:
"""Check it the micronatch will COM; returns it OCK, else 0."m
def ninibatch_partttionn
globbllseqlen__e List[int),, werld_size: int -> List [List compute_costss cos__compue__costt seegle____sees__tt____
equal_ size True
squal_size False set False for SFT with ODC+LB Mini
partiticn_lst Karmarkar_karg
compute_costs, k_partitions=sooldddiee oqual_size=oqua__ee
return partition_lst
def microbatch_partittion
minibatch_seqqeen_ [ist[int]
-> |ist||isti
vet_cempute_com costs (minibatch_ seqlen 1st)
k. partiiiooss
while True:
microbatchh_arritionn...s
karmarkar_karpt
Kinibatch_compute__ee
k_partitions, equal_size=Palse)
is_com check_oom (minibatch__eello__1ll
same_micra rmme_micro_in_ee True
hame_micro_iee = False # set False for ODC+LB_Mii
if same_mit cime_micro_in_de
Ensure all ranks have equal number of microbatches.
(is_oom)
if is_oom == 0:
break # Found valid packing conf configuration. k_partitions +-
return microbatch__aartitioo....
Listing 2: Pseudocode for workload balancing and packing algorithms.
def veri_native_ppo_sttee
@ebbal_seelen__stt. List [int], world size: int, minibatch_sizer int
"""PPO step with two-level partitioning. *** global_seqlen_n arrarray (glcbal_seel (iebbal_segle__stt)
step 1: Balance global batch across ranks. = The fel following block runs in parallel on each device.
sized_size)
for rank, ppobatch in enumerate (rabchto_ppooaacch)
14

---

Published as a conference paper at ICLR 2026
in parallel
For simplicity, we assume PPO epoch is 1. ppobatch = shuffle (ppobatch)
step 2: Split local batch into minibatches.
minibatches = [ppobatch + minibatch_size]
for i range (0, len (epobbatc,)) minibatch_size)]
for minibatch minibatches:
= global_seglen maalennnmmmiiiaacch .tolisst
.oolist ()
# Step 3: Partition minibatch into microbatches.
plcrobatchhporttiion
minibatch_ minibatchheselln__1stt for microbatch in microbatch_partitinn.. I Perform PРO update on the microbatch.
Listing 3: Pseudocode for workload balancing and packing algorithms.
def veri_ prrl_optimizen ppo s
global_ seall_seeeen__eee 1st: List [int], world size: int, minibatch size: int
):
"**рPо step with optimized twc-level partitioning. """
For simplicity, global_seqlen__1s s (gooblaggggaaleeee__eee
assume PРО epoch is 1.
minibatch_size == world_size
Step 1: Split global data inte minibatches
global_minibatches
[i:i minibatch_size)
for i in range len [egggldbal__eeglen_lll sizibatch_size)
for global_minibate in glcbal minimatchss:: glbbalsseleenng
Step 2: Halance minibatch across ranks. rank_to_minibateh minibatch_partitioon miniilacinniccchh werld_sizel
for rank, minibatch in enumerate (rank minibatch):
# run parallel
minibatch_seglen_lss = perta__seeglon_nmmnneeeeee ttliiit tolisst ()
i Step 3: Fartition minibatch into microbatches.
milrrbbttch_parritionnaa microbatchhppartit.nn
Sunnbattch_seglen_Jae
for microbatch in microlat.c__aattttio..
# do ppo update
D COMMUNICATION VOLUME COMPARISON
Method
Intra-node
Inter-node
Total
Collective all-gather (Ring)
G-1 (D 1) K
+ 1) K
(D 1) K
ODC gather
GG 1) K
(D-G)*
- (D-1) 1) +
Collective reduce-scatter (Ring)
G-1
(D 1) K
(D-
100-)) 1)*
(D - 10-1)=
G
ODC ccatter-accumllate
(G-1)
(D-G)+
(D (D-1)*K
Table 2: Comparison of per-client communication volume for collectives and ODC. Both meth- ods send the same total volume ((D - 1) K), but ODC increases inter-node traffic since clients gather/scattrr-accummili
independently.
Let D denote the total number devices, the number of devices per node, and K the size of the per-device local parameters gradients. Under these assumptions, Table Table[]summarrizs summarizes the per- client communication volume for collectives versus ODC. We can observe that ODC increases the cross-node communication volume, which might in turn slows down end-to-end communication.
15

---

Published as a conference paper at ICLR 2026
E ZERO++ STYLE HYBRID SHARDING
which is particularly effective for shorter sequence lengths. To simulate this setting, we truncate each sequence in LongAlign to 1/8 of its original length, resulting in dataset with maximum length of 8k and average length of 2k. As shown Figure 12] hybrid sharding achieves acceleration comparable to full sharding-upp 28% 38% when comparing ODC against collectives.
We evaluate the hybrid sharding strategy introduced in Section 6.1
14B Model, 16 Devices Sim 8K
32B Model, Devices Sim 8K
Samples per Second
Samples per Second
Li-iini
LB-Hit
600
Collective LB-Wil
Li-Nlers
300
Ce
(B-Wiro
Colleciiee LocatSort
Localbort
400
200
200
100
2
Number Samples Minibatch
Number Samples Minibatch
Figure 12:
Comparing ODC with Collectives using hybrid sharding.
It is worth noting that hybrid sharding incurs higher memory usage compared to fully sharded train-
ing; a detailed memory usage comparison is provided Figure T3
Memory Consumption by Sharding Strategy
Memory Consumption (GB)
40
Others
Optimizer
Gradient
30
20
10
GPUs, Fulty Shard
148 Model on 16 GPUs, Hybrid Shard
32B Model GPUs, Fully Shard
328 Model on 32 GPUS, Hy brid Shard
148 Model
Figure 13: Memory consumption ODC hybrid and full sharding.
Devices: Collectives wihhout packing
Devices: Collectives with packing
12
ODC with LB-Kin
CDC with LS-Miri
100
200
250
Step
Step
Devices: 16; Minibatch Size: 1
Devices: 16; Minibatch Size: 4
12
Collecivees packing with LB-Min
12
Collectives packing
Loss
GDC LS-Mirl
100
200
300
400
500
20
40
50
80
100
120
Figure 14: Training loss curves on 8k samples from LongAlign with a 1.5B model. ODC and Collective produce almost identical loss curves.
16

---

Published as a conference paper at ICLR 2026
F CONVERGENCY VERIFICATION
To validate the correctness of our implementation, we compare the loss curves when training a 1.5B model from scratch (to produce clearer loss-descent trajectory) on 8k samples from LongAlign. Results are shown in Figure 14
G DETAILED EXPERIMENT DATA
We show the detailed timing data for SFT and RL in Table As shown in Tables 5 and we also report the bubble rate, defined the ratio of device idle time- time-caused by workload imbalance to the total run time, as estimated by the packing algorithm (Tables and
We observe that the acceleration achieved by ODC closely correlates with the bubble rate predicted by the packing algorithms, indicating that the performance gains primarily stem from reducing idle time due workload imbalance.
Samples Per Second Per Device
Model Dataset
Method
Minibs=2
8
16
1.5B
AIME
Collective Native
496.1
549.9
614.1
658.3
1.5B
AIME
Collective LB-Micro
636.6
716.4
739.7
755.2
1.5B
AIME
ODC LB-Micro
698.0
(+10%) 786.0
(+10%)
804.8
(+9%)
809.9
(+7%)
1,5
AIME
ODC LB-Mini
700.0 (+10%) 784.6
(+10%) 805. 1 (+9%) 811.8 (+7%)
7B
AIME
Collective Native
175.1
199.6
230.5
259.3
7B
AIME
Collective LB-Micro
235.9
273.0
284.4
290.8
7B
AIME
ODC LB-Micro
248.1
(+5%)
302.6
(+11% 11)
309.0 (+9%)
312.4 (+7%)
7B
AIME
ODC LB-Mini
248.5 (+5%)
302.5
(+11%) 309.8
(+9%) 312.6 (+7%)
14B
AIME
Collective Native
74.4
94.5
107.2
130.9
14B
AIME
Collective LB-Micro
106.4
129.6
137.6
140.7
14B
AIME
ODC LB-Micro
101.0 (-5%)
137.2 (+6%)
142.9
(+4%)
145.1 (+3%)
14B
AIME
ODC LB-Mini
101.9 (4%)
143.1 (+10%) 144.4 (+5%) 145.6 (+3%)
Table 3: Timing Data for RI. For the percentage, ODC LB-Micro and ODC LB-Mini is comparing against Collective LB-Micro, ODC LocalSort is comparing against Collective LocalSort.
Bubble Rate (%)
Model
Devices
Dataset
Method
Minibs=2
4
8
16
1.5B
AIME
Collective LB-Micro
15.73
6.56
3.47
1.65
1.5B
AIME
Collective Native
33.83
27.61
20.81
13.13
1.5B
8
AIME
ODC LB-Micro
10.26
0.51
0.05
0.01
1.5B
AIME
ODC LB-Mini
10.26
0.51
0.05
0.01
7B
AIME
Collective LB-Micro
20.79
7.48
3.93
1.90
7B
AIME
Collective Native
40.15
32.41
23.40
13.45
7B
AIME
ODC LB-Micro
16.85
0.53
0.06
0.01
7B
AIME
ODC LB-Mini
16.85
0.53
0.06
0.01
14B
16
AIME
Collective LB-Micro
28.35
10.77
5.91
2.48
14B
16
AIME
Collective Native
46.68
36.36
26.63
14.83
14B
16
AIME
ODC LB-Micro
22.89
0.61
0.04
0.01
14B
16
AIME
ODC LB-Mini 22.89
0.61
0.04
0.01
Table 4: Bubble Rate Data for RL
17

---

Published as a conference paper at ICLR 2026
Samples Per Second Per Device
Model
Dataset
Method
Minibs=1
2
4
8
1,5B
LongAlign
Collective LocalSort
150.7
173.8
218.7
253.4
1.5B
LongAlign
ODC LocalSort
150.9 (+0%)
186.9
(+8%)
239.5 (+10%)
269.5 (+6%)
1,5B
LongAlign
Collective LB-Micro
150.7
212.8
299.4
352.9
1.5B
LongAlign
ODC M.-MMicro
150.9 (+0%)
214.4 (+1%)
348.5
(+16%)
434.6 (+23%)
1.5B
LongAlign
LB-Minn
150.9 (+0%)
232.9 (+9%)
401.3
(+34%)
432.0
(+22%)
1.5B
SWE-Smith
Collective LocalSort
86.3
93.1
111.0
117.9
1.5B
SWE-Smith
ODC LocalSort
87.1 (+1%)
97.0 (+44%)
119.4 (+8%)
125.1 (+6%)
1.5B
SWE-Smith
Collective LB-Micro
86.3
112.0
140.9
152.9
1.5B
SWE-Smith
ODC LB-Micro
87.1 (+1%)
132.2 (+18%)
171.5
(+22%)
171.7 (+12%)
1.5B
SWE-Smith
ODC LB-Mini
87.1 (+1%)
142.2 (+27%)
172.0
(+22%)
172.0 (+12%)
7B
LongAlign
Collective LocalSort
52.7
60.4
75.5
86.4
7B
LongAlign
ODC LocalSort
5-.6
64.9 (+8%)
82.4 (+9%)
92.3 92.3 (+7%)
7B
LongAlign
Collective LB-Micro
52.7
74.3
104.2
122.0
7B
LongAlign
ODC LB-Micro
74.8 (+1%)
119.8 (+15%)
145.7 (+19%)
7B
LongAlign
ODC LB-Mini
82.1 (+11%)
139.6 (+34%)
145.7 (+19%)
7B
SWE-Smith
Collective LocalSort
31.5
33.7
39.7
42.1
7B
SWE-Smith
ODC LocalSort
31.6 (+0%)
35.1 (+4%)
42.9 (+8%)
44.7 (+6%)
7B
SWE-Smith
Collective LB-Micro
31.5
40.5
50.9
54.5
7B
SWE-Smith
ODC LB-Micro
31.6 (+0%)
47.5 (+17%)
60.4 (+19%)
60.3 (+11%)
7B
SWE-Smith
ODC LB-Mini
(+0%)
51.2 (+26%)
60.9 (+20%)
60.2 (+10%)
14B
LongAlign
Collective LocalSort
20.0
25.0
29.5
35.5
14B
LongAlign
ODC LotalSort
19.6 (-2%%
25.8 (+3%)
32.9 (+12%)
38.5 (+9%)
14B
LongAlign
Collective LB-Micro
20.0
31.0
45.1
53.9
14B
LongAlign
LB
19.6
(-0%)
49.9 (+11%)
68.9 (+28%)
14B
LongAlign
ODC LB-Mini
(-2%%)
33.4 (+8%)
61.4 (+36%)
69.0 (+28%)
14B
SWE-Smith
Collective LocalSort
12.3
14.9
16.5
18.1
14B
SWE-Smith
QDC
11.9 (-3%)
15.0
16.7 (+2%)
18.2 (+1%)
14B
SWE-Smith
Collective LB-Micro
12.3
18.3
22.9
25.8
14B
SWE-Smith
ODC I.-Miicro
20.3 (+11%)
27.3 (+19%)
27.5 (+7%)
SWE-Smith
ODC LB-Minn
11.9 (-3%)
23.0 (+26%)
27.4 (+19%)
27.6 (+7%)
32B
LongAlign
Collective LocalSort
10.3
12.9
17.3
20.7
32B
LongAlign
ODC LocalSort
10.3 (+1%)
13.6 (+5%)
17.7 (+2%)
21.3 (+3%)
32B
LongAlign
Collective LB-Micro
10.3
17.3
25.6
32.5
32B
LongAlign
ODC LB-Micro
10.3 (+1%)
17.0 (-2%)
28.1 (+10%)
39.8 (+23%)
32B
LongAlign
ODC LB-Mini
10.3 (+1%)
18.4 (+7%)
33.6 (+31%)
39.4 (+21%)
32B
SWE-Smith
Collective LocalSort
7.6
8.2
9.0
10.5
32B
SWE-Smith
ODC LocalSort
7-2%)
8.3 (+1%)
9.4 (+4%)
11.0 (+5%)
32B
SWE-Smith Collective LB-Micro
7.6
11.1
13.9
15.3
32B
SWE-Smith ODC LB-Micro
7-4%))
12.6 (+13%)
16.1 (+16%)
16.5 (+8%)
32B
SWE-Smith
ODC LB-Mini
(-2%%
14.3 (+29%)
16.1 (+16%)
16.5 (+8%)
Table 5: Timing Data for SFT. For the percentage, ODC LB-Micro and ODC LB-Mini is comparing against Collective LB-Micro ODC LocalSort is comparing against Collective LocalSort.
18

---

Published as a conference paper at ICLR 2026
Bubble Rate (%)
Model
Devices
Dataset
Method
Minibs=
2
4
8
L.5B
LoggAlign
Collective LB-Micro
66.81
52.63
35.28
22.08
1.5B
LongAlign
Collective LocalSort
66.81
61.26
52.22
42.82
1.5B
LongAlign
ODC MB-Micro
66.81
52.48
26.31
1.73
1.5B
LongAlign
ODC LB-Mini
66.81
48.58
14.81
0.02
1,5B
LongAlign
ODC LocalSort
66.81
58.43
48.29
39.70
1.5B
SWE-Smith
Collective LB-Micro
52.36
36.28
20.00
10.74
1,5B
SWE-Smith
Collective LocalSort
52.36
48.22
37.74
33.28
1,5B
SWE-Smith
ODC LB-Micro
52.36
25.96
1.99
0.06
1.5
SWE-Smith
ODC LB-Mini
52.36
20.66
0.71
0.05
.5B
SWE-Smith
ODC LocalSort
52.36
46.27
32.40
29.18
7B
LongAlign
Collective LB-Micro
63.85
48.31
30.31
17.77
7B
LongAlign
Collective LocalSort
63.85
58.03
48.95
39.65
7B
LongAlign
ODC LB-Micro
63.85
48.14
21.23
0.39
7B
LongAlign
ODC LB-Mini
63.85
42.71
7.19
0.03
7B
LongAlign
ODC LocalSort
63.85
54.99
45.11
36.71
7B
SWE-Smith
Collective LB-Micro
49.70
33.87
17.41
9.77
7B
SWE-Smith
Collective LocalSort
49.70
45.69
35.53
31.18
7B
SWE-Smith
ODC LB LB-Micro
49.70
22.86
1.29
0.08
7B
SWE-Smith
ODC LB-Mini
49.70
16.81
0.59
0.05
7B
SWE-Smith
ODC LocalSort
49.70
43.75
30.21
27.08
14B
16
LongAlign
Collective L.-Miccro
72.28
57.52
37.75
24.41
14B
16
LongAlign
Collective LocalSort
72.28
66.02
59.19
48.97
14B
16
LongAlign
ODC LB-Micro
72.28
56.78
29.52
0.02
14B
16
LongAlign
ODC LB-Mini
72.28
53.48
14.43
0.02
14B
16
LongAlign
ODC LocalSort
72.28
64.20
52.93
42.46
14B
SWE-Smith
Collective L.B-Micro
56.53
35.00
18.45
9.30
14B
16
SWE-Smith
Collective LocalSort
56.53
47.47
42.04
35.51
14B
SWE-Smith
ODC LB-Micro
56.53
24.87
0.47
0.05
14B
16
SWE-Smith
ODC LB-Mini
56.53
16.13
0.42
0.04
14B
SWE-Smith
ODC LocalSort
56.53
45.88
39.27
32.04
32B
32
LongAlign
Collective LB-Micro
73,01
55.19
35.59
20.43
32B
32
LongAlign
Collective LocalSort
73.01
67.49
57.00
50.50
32B
32
LongAlign
ODC LB-Micro
73,01
54.64
26.06
0.01
32B
32
LongAlign
ODC LB-Mini
73.01
50.91
10.67
0.01
32B
32
LongAlign
ODC LocalSort
73.01
64.16
49.55
37.76
32B
32
SWE-Smith
Collective LB-Micro
54.03
32.54
16.69
8.95
32B
32
SWE-Smith
Collective LocalSort
54.03
50.42
44.74
36.49
32B
32
SWE-Smith
ODC LB-Micro
54.03
20.66
0.29
0.03
32B
SWE-Smith
ODC LB-Mini
54.03
10.11
0.30
0.03
32B
32
SWE-Smith
ODC LocalSort
54.03
49.25
40.55
32.16
Table 6: Bubble Rate Data for SFT
19


Reviews:
[{"review_id": "vqi35qfnR7", "reviewer": "Reviewer_xMDa", "summary": "This paper proposes On-Demand Communication (ODC), which removes per-layer synchronization barrier from training LLM in FSDP. Because of the imbalanced workload in post-training, with a simplified load balancing strategy, ODC helps speedup over FSDP by replacing collective calls with P2P communication.", "questions": "Can ODC extend to support MoE models in FSDP? Though not efficient as expert parallel method, it might be better than the standard FSDP?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-motivated and straightforward to implement.\n- An excellent complement to FSDP that only requires changing communication operators—theoretically and experimentally equivalent (minor precision differences from batch size variations have minimal impact).", "weaknesses": "The practical applications of ODC may be quite limited, requiring specific scenarios with load imbalance. In SFT, the paper only tested LongAlign and SWE-Smith; in RL, updating the actor is not the main bottleneck, and currently, partial-rollout or fully asynchronous training are more commonly used to improve overall system throughput.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NnCSydLbrX", "reviewer": "Reviewer_ycWJ", "summary": "This paper addresses a critical limitation of modern data-parallel training (e.g., FSDP) in LLM post-training, where variable sequence lengths cause severe workload imbalance and render collective communication inefficient due to strict synchronization barriers. The authors propose ODC, a novel adaptation of the PS paradigm within FSDP that replaces collective all-gather/reduce-scatter with point-to-point communication. By decoupling device synchronization to the minibatch level and enabling asynchronous gradient exchange, ODC eliminates workload-induced stalls and facilitates fine-grained load balancing. Experiments across diverse LLM post-training tasks show consistent gains in device utilization and throughput, demonstrating that PS-style communication is not obsolete, but better suited to real-world LLM training dynamics. The approach is practical and novel.", "questions": "- Under the setting of activation recomputation, is ODC still applicable?\n- What are the potential limitations of ODC compared to FSDP?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Novelty: The proposed method adapts the parameter server concept into the FSDP framework by replacing collective all-gather/reduce-scatter operations with point-to-point communication, thereby reducing synchronization overhead.\n- Practicality: The authors implemented ODC and provided extensive experimental results demonstrating its reliability.\n- Readability: The paper is well-structured and easy to follow.", "weaknesses": "No obvious drawbacks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WvQYmKLnvy", "reviewer": "Reviewer_7zfy", "summary": "The paper proposed on-demand-communication, which separate worker and parameter sever as old-fashion PS.", "questions": "Try to profile some real world workload, and real-world system setting (e.g., how to do data loading, data preprocessing (padding, batching), distributed FSDP+prefetch, LoRA, etc), for post training is a good way to find where is the real bottleneck.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. good idea to call back to parameter server. Personally I like this call-back.\n2. clear presentation and experimental results", "weaknesses": "1. With PS, one major issue is we have inconsistency of parameters (i.e. parameter delay). For example, two GPUs , GPU0 pull the weights before some xyz updates, then GPU1 pull the same weights with xyz updates, then they are training on different weights but same iteration number, This parameter inconsistency will hurt model convergences. There is no discussion on model convergence in the whole paper.\n\n2. Indeed, FSDP/ZeRO they already incorporated async parameter pre-fetch, hiding weights updates behind OS (optimizer states) updates etc. This communication hiding mechanism works pretty well and I don't see the real motivation for deleting all the global collective into pair-wise p2p communication. The statement of this paper in L134 \"but this overlap does not remove the underlying synchronization points.\" is wrong, because, every GPU can pre-fetch a number of layers weights all together at once, they don't need to wait for each other at every layer's fwd or bwd\n\n3. In real world post-training, we do either batching or padding to make every GPU has same sequence length of input, thus the straggler problem this paper try to solve does not exists in real-world application scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KHT2nj6a4s", "reviewer": "Reviewer_sQnr", "summary": "The paper analyzes synchronization inefficiencies in Fully Sharded Data Parallel (FSDP) training when applied to LLM post-training workloads characterized by heterogeneous sequence lengths and constrained microbatch sizes. The core claim is that per-layer collective communication (all-gather in forward/backward and reduce-scatter in backward) introduces fine-grained synchronization barriers, causing straggler-induced idle bubbles. This assumption of balanced workloads holds in vision and speech models but fails in long-context supervised fine-tuning and RL workloads, where compute scales quadratically with sequence length (attention) while memory scales linearly.\nThe proposed method decomposes collectives into point-to-point RDMA gather and scatter-accumulate primitives, enabling asynchronous parameter fetch and gradient push at the granularity of microbatch availability. The authors demonstrate that this transformation turns FSDP into a decentralized parameter-server architecture that retains ZeRO-style memory partitioning while removing per-layer synchronization points. Synchronization is preserved only once per minibatch to maintain identical global optimizer semantics.\nEvaluations on SFT (LongAlign and SWE-Smith datasets) and RL (AIME prompts with GRPO) across model scales 1.5B–32B show up to 36% throughput improvement and strong correlation with predicted imbalance (bubble rate).", "questions": "- What is the asymptotic slowdown of ODC communication when inter-node degree grows? Can you provide a model predicting crossover vs collectives?\n- Does the minibatch-level synchronization impose idle bubbles in the best-case balanced regime? If so, is dynamic minibatch staggering feasible?\n- In RL, how would delayed optimizer updates or local advantage normalization interact with async shard availability?\n- What is the concurrency behavior of the gradient accumulation daemon at high client cardinality? Any queue saturation or tail latency concerns?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Precise identification of bottleneck in sharded data parallelism. The per-layer max-runtime aggregation inherently propagates worst-case slowdowns, validated with measurements showing up to 50% idle bubbles.\n- The introduced method preserves DP correctness constraints: a) parameter, gradient, optimizer sharding identical to FSDP. b) deterministic synchronous update at minibatch boundaries, and c) No model semantic change.\n- Lightweight integration: <100 LOC modification using CUDA-IPC + NVSHMEM RDMA via Triton-Distributed.", "weaknesses": "-  Benchmarking shows significant bandwidth degradation vs NCCL rings when D > G.\nInter-node performance constraints remain unresolved in this work.\n- The degree of compute–communication overlap is asserted but not rigorously quantified (e.g., timeline traces, GPU utilization per stage). This limited control-flow characterization does not give much confidence to the reader on the effectiveness in overlapping.\n- Comparison set omits strong baselines. There is no experimentation vs ZeRO++ hierarchical collectives, scheduling-based overlap strategies, or pipeline-staleness variants.\n- RL system constraints to some extend weaken conclusions. To be specific, forcing equal samples per device, reduces the effectiveness of proposed minibatch-level packing (LB-Mini).\n- Claims of benefits growing with scale lack large-cluster support are not supported by the results since the scaling experiments are not extended to >32 GPUs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)