Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling
Authors: Hongtao Xu
Wenting Shen
Yuanxin Wei
Ang Wang
Guo Runfan
Tianxing Wang
Yong Li
Mingzhen Li
Weile Jia
Venue: NeurIPS 2025 poster ()
Keywords: long context fine-tuning
context parallelism
supervised fine-tuning
LLM
Large language model
Primary area: infrastructure
TL;DR: Efficient Long Context Fine-tuning through Dynamic Data Scheduling

Abstract: Long-context supervised fine-tuning (Long-SFT) plays a vital role in enhancing the performance of large language models (LLMs) on long-context tasks. To smoothly adapt LLMs to long-context scenarios, this process typically entails training on mixed datasets containing both long and short sequences. However, this heterogeneous sequence length distribution poses significant challenges for existing training systems, as they fail to simultaneously achieve high training efficiency for both long and short sequences, resulting in sub-optimal end-to-end system performance in Long-SFT.
In this paper, we present a novel perspective on data scheduling to address the challenges posed by the heterogeneous data distributions in Long-SFT. We propose Skrull, a dynamic data scheduler specifically designed for efficient long-SFT. Through dynamic data scheduling, Skrull balances the computation requirements of long and short sequences, improving overall training efficiency. Furthermore, we formulate the scheduling process as a joint optimization problem and thoroughly analyze the trade-offs involved. Based on those analysis, Skrull employs a lightweight scheduling algorithm to achieve near-zero cost online scheduling in Long-SFT. Finally, we implement Skrull upon DeepSpeed, a state-of-the-art distributed training system for LLMs. Experimental results demonstrate that Skrull outperforms DeepSpeed by 3.76x on average (up to 7.54x) in real-world long-SFT scenarios.

Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling
Hongtao Xu
Wenting Shen
Yuanxin Wei Ang Wang Guo Runfan Weile Jia2t
Tianxing Wang Yong Li
Mingzhen Lift
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences State Key Lab of Processors, Institute of Computing Technology, CAS
Alibaba Group
SSun Yat-sen University
Corresponding authors
Abstract
Long-contex, supervised fine-tuning (Long-SFF) plays vital role in enhancing the performance of large language models (LLMs) on long-context tasks. To smoothly adapt LLMs long-context scenarios, this process typically entails training on mixed datasets containing both long and short sequences. However, this heterogeneous sequence length distribution poses significant challenges for existing training systems, as they fail to simultaneously achieve high training efficiency for both long and short sequences, resulting in sub-optimal end-to-end system performance in Long-SFT. In this paper, we present a novel perspective on data scheduling address the challenges posed by the heterogeneous data distributions in Long-SFT, We propose Skrull, dynamic data scheduler specifically designed for efficient long- SFT. Through dynamic data scheduling, Skrull balances the computation requirements of long and short sequences, improving overall training efficiency. Furthermore, we formulate the scheduling process as a joint optimization problem and thoroughly analyze the trade-offs involved. Based on those analysis, Skrull employs lightweight scheduling algorithm to achieve near-zero cost online scheduling in Long-SFT. Finally, we implement Skrull upon DeepSpeed, state- of-the-art distributed training system for LLMs. Experimental results demonstrate that Skrull outperforms DeepSpeed by 3.76x average (up to 7.54x) in real-world long-SFT scenarios.
Introduction
Long-context capabilities important for large language models (LLMs) to handle various tasks such as long document summarization, question answering. multi-turn dialogue and code generation. Mainstream LLMs such as Llama [1 10), Qwen 19 and GPT-4 [88] support the context
window of up to 128K tokens. Google's 's Gemini 9 can even achieve up to 1M tokens per context window. Typically, additional training phases like long-context supervised fine-tuning (Long-SFT) as well as long-context continue pre-training (Long-CPT)) are employed to extend the context length. For example, Llama3 [10] is fine-tuned with 99, 99.8 short sequence (averaging under IK tokens) and 0.1 11% long sequence (averaging around 37K tokens). Qwen2.5-Turbe [19 gradually extends context length by training on 40% long sequences and 60% short sequences. Training on those meticulous gathered datasets enables smoothly adaptation of LLMs to longer context while still maintaining the performance on short context tasks.
However, this heterogeneous data distribution in Long-SEr SFT poses significant challenges for existing distributed LLM training frameworks [14 20 [5), exhibiting sub-optimal efficiency. For instance, the heterogeneous data distribution poses a dilemma for parallelism and memory-reduction strategies. Specifically, long sequences necessitate context parallelism and other memory redurtireduction approaches
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

due to their tremendous memory requirements. However, those approaches compromise the training efficiency for short ones due to the overheads like unnecessary communication and GPU under- utilization. Moreover, the wide sequence length distribution in long-SFT worsen the mismatch of computation characteristies Attention module, which exhibit quadratic computational complexity leading to another dilemma for load balance problem.
and linear memory consumption 7 6
To tackle the above challenges, we propose Skrull, dynamic data scheduler dedicated for Long-SFT scenarios. Skrull efficiently handle the unique data distributions in Long-SFT scenario through two main components: Distributed-Awaz
Context Parallelism (DACP) and Global Data Scheduling (GDS). DACP selectively shards sequences and schedules them across different workers to minimize the performance degradation while maintains the ability of handling long sequence. GDS enlarge the scope of scheduling and improve the GPU utilization during training. The two components collaborate with each other different scheduling granularities. Furthermore, to achieve the optimal performance, we formulate the scheduling process as joint optimization problem and design a lightweight heuristic algorithm to solve it runtime. Experimental results demonstrate that Skrull improves the end-to-end training performance by 3.76x average (up to 7.54x) compared to DeepSpeed, sate-of-the-art distributed LLM training framework.
Our key contributions are summarized as follows:
We provide a new perspective of data scheduling to address the heterogeneous sequence length distribution.
We propose new context parallelism called DACP based on fine-grained data scheduling, which maintaining both the processing capabilities for long sequences and efficiency for short sequences, enabling efficient training on heterogeneous data distribution in long-SFT scenario.
We implement coarse-grained global data scheduling (GDS) and further formulate GDS and DACP as a joint optimization problem through performance modeling.
We design a lightweight heuristic algorithm and achieve performance gains by 3.76x on average (with a peak improvement of 7.54x) in real-world datasets.
2
Preliminaries
Data Parallelism (DP). Data parallelism
15
24 20
partitions the training samples to multiple
workers and each worker maintains a complete model weight replica. each iteration, all workers process subset of global batch independently and then synchronize the gradients across all DP ranks. However, due to the inherent synchronization semantic in DP, the load balance becomes a noticeable problem, especially in long context scenarios.
Context Parallelism (CP). Context parallelism partitions the input tensor along the sequence length dimension and distributes it to multiple workers 16 [0]. CP is emerging as inevitable parallel strategy when handling long context. In the Transformer architecture, the primary challenge of CP stems from the parallelization of Attention module because each tokens needs to attend to other tokens in the sequence. Consequently, the communication in CP is inevitable. Notably, DACP, proposed in this paper, leverages data scheduling to minimize the overheads caused by CP and is orthogonal to specific CP implementations.
Observation
3.1
Heterogeneous Sequence Length Distribution
As shown in Figure la we observe pronounced variance in the sequence length distribution across real-world Long-SF datasets, including Wikipedia 22, LMsysChatIM (5) and ChatQA2-Long SFT exhibits bimodal pattern,
[ Among them, the sequence length distribution of ChaiQAA2-long-SE where the proportions of long and short sequences are nearly equal. Specifically, approximately 40% of sequences are shorter than 8K tokens, while the remaining 60% exceed this threshold. As comparison, long-tail distributions represent another typical pattern in Long-SFT datasets. In Llama3's 's internally collected SFT datasets [10], we find that 99.89% of sequences are under IK

---

tokens on average, while the remaining 0.1 0.11% % are upproximately 37K tokens, showcasing extremely skewed long-tail distribution. Due data accessibility constraints, we plot the sequence length distribution of Wikipedia and I.Msyshhatl M in Figure Ta which have the identical feature with Llama3's 8 Long-SFT dataset. Table lists the portions under different lengths thresholds for these three datasets, highlighting the heterogeneous sequence length distribution in Long-SFT.
Density
Cha10A2 Long-SFT SFT
0K
5K
15K
20K
24K
29K
34K
39K
TFLOPe
200
150
Donoty
LMeysChat1M
OK
10K
20K
29K
39K
18K
59K
68K
78K
50
Denaity
2
Wikipadia
Sequence Length
128K
GP-2
OK
2K
4K
8K
10K
OP+1
CP-4
CP-18
Sequence Length
(a) Sequence length distribution.
(b) Performance (FLOPS) with different CP workers.
Figure 1: Sequence length distribution different datasets, and corresponding performance impact.
Table 1: Percentage of sequence length in real-world datasets.
Dataset
<IK
<4K
<8k
<32K
<128K
Longest
Wikipedia
87.8 87.88%
99.34%
99.92%
99.99%
100.0%
78K
LMsysChatIM
87. 12%
99.35%
99.87%
99.98%
99.99%
1643K
ChatQA2-Long-STT
21.92%
31.48%
40.43%
99.86%
100.0%
99K
3.2 Performance Degradations for Short Sequences
In this section, we discuss observation on the performance degradations and GPU under-utilization for short sequences in Long-SFT training. During the training process, context parallelism degree and other memory reduction strategies such as gradient accumulation set to accommodate the longest sequence in datasets to avoid out-of-memory errors (OOMs). However, these training settings degrade their performance for the shorter sequences, which make up the majority in Long-SFr SFT datasets. As shown in Figure Tb we test the performance of Attention module (6] under different CP degrees. Results demonstrate, especially for the short sequences, higher CP degree exacerbates kernel execution efficiency. Additionally, context parallelism also brings unnecessary communication overhead to short sequences. Also, the memory reduction strategies tailored to long sequence lead to low GPU memory utilization for the most time.
Skrull
We introduce design of Skrull and the efficient implementation for online Long-SFT training in this section. Figure Saare2eill illustrrae the workflow of Skrull. From the perspective of data scheduling, Skrull consists of two parts: (i) Global data scheduling (GDS): For every iteration, Skrull takes the global batch input and employs coarse- -rarined scheduling to generate the optimal micro-batcher for each ranks. (ii) Distributed-awarc Context Parallelism (DACP): Taking the micro-batch produced in GDS, Skrull further employs finer-grained scheduling to selectively distribute the sequences and assign them to different CP workers. For the convenience formulation, we sequentially introduce DACP in Section
4.1
GDS in Section 4.2 and the efficient implementations in Section

---

Global Data Scheduling (GD5)
Distrbbted Award Context Parallolis (DNCP)
Depres
Buration
Stuctare
Co
Roll-Back
Model Contie
Schoualin Algoriten Chosfication
Schetuling Algorther
Camm Comp Overlapping
Iteration Start
Scheduling g Algorithi
Duration
Perf
Memory
FLOPS
MI Officee Profiing
Buckediorr
Communication
Sepunner Botchig
COS Banding
CPres
Compatation
(a)
(b)
(c)
Figure 2: Workflow of Skrull. (a) Offline profiling: Given model and training settings, provides performance estimation for data scheduling. (b) GDS: produce optimal batching strategies for DACP. (c) DACP: dynamically scheduling data to specific hardware with balanced workload and minimum overheads. (d) Performance gains of DACP: it shows how the reduced communication volumn and overlapping improve the performance.
4.1 Distributed-aware Context Parallelism
To simultaneously achieve high efficiency for all the sequences, we propose distributed aware context parallelism (DACP). As shown in Figure[]c), DACP dynamically determines whether to distribute sequences to avoid unnecessary overheads or not. On the one hand, DACP preserves the original context parallel settings to maintain the ability of handling long sequences. On the other hand, DACP selectively schedules short sequences entirely within single device to minimize the degradation. Therefore, based on distinct computational characteristics, DACP classifies sequences into two categories: (i) distributed sequences requiring context parallelism, and (ii) local sequences needing efficient processing and intended to reside entirely within a single device. Notably, these sequences are still processed within a shared CP group without increasing the number of GPUs used for training. Furthermore, as illustrated in Figure d), DACP brings additional opportunity to overlap the communication of distributed sequences and the computation of local sequences in Attention module due to the inherent independence between distributed and local sequences.
However, the scheduling process presents significant challenges. First, inappropriate sequence classification may lead to out-of-memory errors (OOMs). Second, the local sequences are varying length and pose load imbalance issue across CP ranks. To fully explore the relationship between scheduling plans and performance gains, we first analyze the computation and memory features in Appendix
C Through offline profiling, we model the computation (see in Appendix C.2 by FLOPs function and latency estimation function Teomp. Additionally, we map the sequence length to the memory consumption and derive the BucketSize C' which indicates the capacity of total sequence length per ranks. The BucketSize C' plays vital role in measuring the memory constrain during Skrull's 's scheduling. More details are listed in Appendix [.I]] Similarly, we model communication volume function Volume and latency function Toomm detailed in Appendix CC33 Finally, formulate the scheduling process as an optimization problem as follows. The frequently used notions listed in Table 2
DACP Formulation. We first define the sequence classification array E {o, 1) K (0 for local sequence and for distributed sequence) and local sequence assignment matrix P {0, K)Kx (1 for assignment and 0 for not). For example, D3 indicates that the k: th sequence with length of Sy is scheduled to be computed in distributed manner. Similarly, Pxj = indicates that the th sequence is assigned to device j, implying Dj = O. Given a micro-batch comprising K sequences with lengths Sy (k (0, (O....,K' 1)), SucketSize C' and CP degree N, the scheduling process of

---

Table 2: Symbols used in this paper.
Symbol
Description
Symbol
Description
Sk
Length of the k-th sequence in batch.
BucketSize per rank.
DR
Distribute k-th sequence or not.
Pkj FLOPs
Assign k-th sequence to CP rank j
N
CP degrees.
FLOPs estimation function.
Teomp
Computation cost estimation.
Tcomm
Communication cost estimation.
Bgij
Assign k-th sequence to i th DP rank and j - th micro-batch.
ws
DP degree
Volume
Communication volume count function.
DACP can be formulated as follows:
Objective
min TDACP = min max (Time,)
(1)
org
Subject to
Time,
max (I (Teom (V) , Teump (Local,)) + Tammp (Dist), Vj
(2)
Local
Σ
FI.OPs (Pej Sk); Vj
Dist N
Σ
FLOPs (Dxx Sx)
(4)
V Volume DR SE)
(5)
PNs Pkj Pkj+ + + Dx=1, Vk
(6)
Σ S. Pl.-Pag k
D& Sk SC. C, Vj
(7)
Here, optimization goal is find the optimal D and to minimize TDACP, which represents the total duration in one micro-batch. As shown in Equation [ TDACP is determined by the maximum execution time Time, across all CP ranks j. Specifically, of two components: (1) the overlapping term, defined as the maximum of the communication time T Toomm (V) and the computation time (Local,) for local sequences, and (2) the computation time Tommp (Dist) for distributed sequences. Here, Teomm depends on the communication volume V. as modeled in Equation Similarly, Teomp utilizes the results from Equations and 4 which compute the FLOPs for local sequences on CP rank j and distributed sequences, respectively. Finally, Equation 6 ensures the completeness of data scheduling, while Equation enforces the memory constraint.
described in Equation 2 Time, consists
4.2 Global Data Scheduling
Section 4.1 discusses the data scheduling in the scope of micro-batch. However, only relying scheduling in DACP is insufficient. The reasons are as follows.
First, the heterogeneous sequence length distribution also leads the to severe load imbalance across different micro-batches, resulting in the sub-optimal training efficiency Long-SFT scenarios. Second, to achieve maximum performance gains in DACP, meticulous miiro-batching strategy is essential. For example, pairing long and short sequences with appropriate memory pressure can expand the valid scheduling space for DACP. Specifically, micro-batches with large total sequence lengths increase the risk of OOMs and limit the optimizations in DACP, such as selective sharding. In contrast, hicro-batches with small total sequence lengths introduce GPU under-utilization, degrading the end-to-end performance. Therefore, as shown Figure (bb, Skrull employs Global Data Scheduling (GDS), which derives the optimal micro-batching strategy from the global batch. We limit the scheduling scope to the global batch because it represents the maximum scope that maintains mathematical equivalence for mainstream optimizers such Adam [3] and AdamW [17].
We re-formulate the scheduling process as a joint optimization problem that integrates both DACP and GDS. We first define the batching matrix Buiz {0, 10,1]Kx4 which
Joint Formulation
5

---

indicates whether the k-th sequence is scheduled into the j-th ticro-batch of DP rank i. Given global batch B consisting of K sequences with lengths Sk. re-formulate the scheduling process as follows:
Objective
min max arg
Times;
(8)
Subject to
BRy 1, 4k
(9)
E Buy Sk CC*N Vj
(10)
Times- = TDACP ( [Bay Sk), 4k:
(11)
Here, TDACP represents the function in Equation Equation ensures all sequences are assigned exactly once. The memory constraint in Equation 10 prevents the OOMs while the Time,y shown in
Equation [ provides cost estimations for each micro-batch using DACP formulation in Equation|] [ As shown in Equation X the total execution duration per iteration is determined by the DP rank with the longest cumulative execution time across its micro-batches.
Overall, the optimization target is to the minimize the total execution time per iteration by deducing the optimal scheduling plan, which represented by a combination of Buaj, Da and Pag.
4.3 Efficient Online Scheduling
Although some solvers like 4 can derive the optimal scheduling plan, its long solving time makes impractical for scheduling during runtime. To achieve online scheduling during Long-SFT, we resort to design lightweight heuristic scheduling algorithm. Notably, our scheduling algorithm is integrated into the DatalLadee and introduces near-zero overhead to the training process.
4.3.1 Memory vs. Computation: Trade-off Analysis
Memory and Computation are the key factors related the performance, shown in the formula- tions of scheduling. We should achieve optimal performance while not violating memory constraint, presenting trade-off. Therefore, we first analyze the trade-off between Computation and Mem- ory when deducing the scheduling strategies, highlighting the considerations when designing the scheduling algorithms.
Sequence classification: deduce the array D. We analyze the sequence classification (array D in Section 4.1). From the perspective of Computation, D impacts the communication volume and computation of sharded sequences (Equation 5 4). More sharded sequences will incur more performance degradation, which comes from both communication overhead and kernel execution (refer to Section 3.2). However, from the perspective of Memory, more distributed sequences will bring more balanced memory consumption (Equation 7.. which can lower the risk of OOMs, as the remaining local sequences with varying lengths are hard to be assigned evenly. Besides, although the overlapping in DACP can alleviate the performance degradation problem to some extent (Equation 2 it is still non-trivial to decide the optimal classification array D.
Local sequence assignment: deduce the matrix P. Then, we analyze local sequence assignment, which is represented by P. From the perspective of Computation, P impacts the Equation 3 which implies the computation workload in each CP ranks, thus affects the load balance. The ideal situation is to balance the local sequences for computation balance among CP Howevvr, from the perspective of Memory, the scheduling which balances the computation leads to the unbalance of memory consumption, which increases the risk of OOMs.
Unfortunately, we cannot balance the computation and memory at the same time. The reason is that, after applying FlashAttention 7 K) the correlation between computation complexity and and sequence length (n) is O(n²), however, the correlation between memory is O(n). Moreover, with the sequence length increasing, the portion of Attention module gradually dominates the computation load, making it more difficult to balance the computation and memory. Worse still, the model

---

configuration (e.g., KV heads, hidden size) also impacts. Due to the limited page, we list the details in Appendix
Therefore, we need to carefully deal with the memory footprint balance and the computation com- plexity balance and we design the following heuristics.
4.3.2 Heuristics
Scheduling Algorithm of DACP. We first summarize three principles of algorithm design in DACP. (i) Avoid sharding: We strive to avoid sequence sharding and assume that all sequences will be handled locally first. (ii) Prioritize computation: We prioritize balancing computation over memory to achieve better performance. (iii) Roll-back mechanism: We continuously monitor the estimated memory consumption and revert decisions when necessary. The roll-back mechanism guarantees the memory constrains outlined in Equation 7 and Equation TO, while enabling more aggressive scheduling attempts based on (i) and (ii). Our heuristic for DACP listed in Algorithm [ Given a micro- batch containing K sequences with lengths S[K] and a predefined BucketSize C, the algorithm outputs the sequence classification and assignment results in the form of an array ret. In this array, value of 1 at the i-th position indicates that the i-th sequence is to be sharded, while a value v (O..... ws 1) indicates that the i-th sequence assigned to CP rank v entirely. To better balance computation while ensuring memory constrains, we maintain two arrays during DACP scheduling: RemainBucket RB and Loads L, which represent the current memory budget and computation load, respectively. We first sort the sequences in ascending order. For the each sequence, we sequentially assign to the bucket (as well as CP rank) with minimum T, to avoid sharding and prioritize balancing computation (line 6-8). If the bucket cannot accommodate the sequence, we attempt to assign it to the bucket with the maximum RB to avoid sharding (line 10-12). If both attempts fail, we classify the sequence as distributed sequence and attempt shard it (line 14-16). However, if the bucket with minimum of RB cannot handle the sub-sequence after sharding, this indicates that the earlier process incorrectly classified inappropriate sequences local sequences within this bucket. To address this, we employ a roll-back mechanism (line 18 and Appendix [B.I). This mechanism identifies a local sequence in the bucket, shards it to reduce memory pressure, and resumes the assignment process. If the roll-back fails due to the absence of local sequences in the bucket, we return DACP scheduling error. In such cases, GDS will also revert the batching plan (see the Section 4.3.2 Notably, every assignment updates RB and I. through the predefined functions Updatel.ocal and UpdateAll. The details of these functions including RollBack are further elaborated in Appendix B.I
Scheduling Algorithm of GDS. Algorithm Serrithmmeeemnn demonstrates the heuristic scheduling algorithm of GDS. Given a global batch containing K sequences with lengths [[ DP world size ws and DP rank dp_rank, the algorithm returns the scheduling result mós, which consists of multiple
as inputs for Algorithm [ We summarize three principles in our algorithm design. (i) Prioritize computation: We prioritize balancing computation across DP workers. To achieve this, we estimate the FLOPs (Appendix C.2) and employ a bin-packing algorithm to balance computational workloads at a coarse granularity (line 1). (ii) Pair long and short sequences: We sort the sequences within each DP rank and batch them in interleaved manner (line 7). This approach ensures that long sequences are assigned more evenly across micro-batches. Additionally, each micro-batch contains several short sequences, enhancing both task overlapping and load balancing. (iii) Improve memory utilization: We estimate the total memory requirements and try to improve the concurrency with less number of micro-batches. Thanks to the roll-back mechanism (line 8), this method maximizes memory utilization while not increase the risk of OOMs. As shown in line 5, we gradually increase the number of micro-batches if the scheduling fails and requires rolk-back.
Evaluation
Experimental Setup. We conduct experiments using testbed consisting of 4 nodes interconnected via a high-ppeffrmance InfiniBand network, with each node equipped with 8 Nvidia H100 GPUs connected via 900GB/s NVLink. Then, We implement Skrull on top of DeepSpeed, a state-of-the-art distributed L.LM training system and enable Zero-2 optimization as our baseline. Additionally, we implement sorted batching method in Align [ for more comparison, which sort the dataset by sequence length and select random consecutive groups for each batch to improve the Iong-SFT
7

---

Algorithm Heuristic scheduling algorithm of DACP
Require: SeqNum K, SeqLens S[K], BucketSize C, CP degree N
Ensure: Scheduling Result ret [K]
I: Sort(SegLens, ascending=True)
i= N do
3:
C, -
b Initialization
4: end for
5: for 0 K do
6:
f 1--argmi([
Find rank with minimum workload
RB [ 2
then ret +1 UpdateLocal
8:
9:
else
10:
1 < arggax((BB) RB)
if RB S then ret t UpdateLocaalf, t)
12:
13:
else
14:
<- argmin (RB)
15:
[B]]2S then ret 1 UpdateAll(1)
16:
Distribute the sequence
17:
else
18:
Assert RollBack(t, RB, L)
19:
iti-1
Roll-back avoid OOMs
20:
continue
21:
end if
22:
end
23:
end if
24:
end for
25:
return rel
Algorithm 2 Heuristic Scheduling Algorithm of GDS
Require: SeqNum K, SeqLens S[K]. BucketSize C, CP degree N. DP WorldSize ws, DP Rank dp_ rank
Ensure: Micro-batches mbs
1: Bin ws] + (innacc((ss.FFFLP((((111))
Coarse-fine fined balance
2: Subset
Bin Bin (d__rank],, ánit [Sum (amm@sssst)/CC x N 1
Sort(Subsed ascending=True)
4: while imat < K + do
5: init +- imst + 1, mbs enil do mube aphend/ Subsett :: inill)
6: for
Pair long and short sequences
8: if Sum(mns -1] 2 C' x N or not scheduling in DACP(mbs[ then
9:
Continue
b Rollback if overload or DACP sheduling fails
10: end if
end for end while
13:
return mbss
training efficiency. We evaluation our optimizations on Qwen2.5-0.5B and Qwen2.5-7B using the three real-world datasets described in Section 3. Although Wikipedia and LMsysChat M not specifically gathered for Long-SFT, we still choose them as our evaluation datasets due to their
long-tail distribution, which is exactly identical Meta's in-house Long-SFF dataset [10). In contrast,
ChatQA2-long-SPP dataset [1] is specifically gathered for Long-SFI and exhibits bimodal distribution of data length, which is also similar to the dataset mentioned in [9 Through offline profiling,
configure the BucketSize to 26K and 13K for Qwen2.5-0.55 and Qwen2.5-7B. respectively. Further details regarding BucketSize configuration can be found in Appendix C.T All the experiments share the same training settings with <DP=4, CP=8, BatchSize=642, zero-2 enabled and selective recomputation strategy except for training Qwen-2.5-7B with CharQA2-long-SFT dataset. Due the increased memory requirements, we adjust its parallel settings with <DP-2, CP=16, BatchSize=40> The global batch size is equal to size multiplied by BatchSize. Due to the limited page, we list precision validation in Appendix A

---

baseline
baseline- DACP
Daselin+GDS+DACC
sorted batching
7.54x
2.30x
1.24
2.85x
1,00
1.00d .0y
2,7x
1.00x
10x
1.00x
00tx
Wilpedia (4,8,64)
LMsyeChatt M (4,8,84) Owen-O.S SB
CharQA2 SFT (4,8,64)
Wikpedia (4.8 (4.8.64)
LMsysChat11M (4,8,84) (b) Qwen-78
ChatQA2-Lon
(2.18.40)
Figure 3: Overall performance and step-by-step evaluation. The settings represent the DP degree, degree and batch size, respectively.
2.00
3.0
BatchSiiz aize BurkettSine
148
1.75
328
2.5
2.3%%
1.50
Speedup
1.35x
1.30x
1.37%
Speedup
2.0
1.25
1.784
00M
1.5
1.00
1.18x 1.24x
0.75
0.80
1.0
1,00x
1.00x
0.50
0.5
(8K)
12 (18K)
20 (28K)
54 (32K)
Baseline
DACP
DACP + GDS
Performance impact BatchSize and BucketSize.
Fine-luning with lora.
Figure 4: Left (a) shows the performance impact under different BatchSize and BucketSize. Right (b) shows Skrull's effectiveness and compatibility with Lora larger models.
Overall Performance. Figure illlstrrts the speedup achieved by Skrull, with the performance measured in terms of average iteration time. Skrull significantly outperforms the sorted batching strategy Experimental results demonstrate that Skrull outperforms DeepSpeed and sorted batching method by an average of 3.76x and achieve peak improvement of 7.54x. The average speedups for Qwen-0.5B and Qwen-7B are 5.50x and 2.03x, respectively. We attribute this difference to the variation in BucketSize, which directly influences the valid data scheduling space. Also, Skrull outperforms sorted batching method by an averge of 3.45x with the peak improvement of 6.85x. Additionally, from the perspective of datasets, the performance on Wikipedia and LMsysChat I M are similar due to the similar data distribution, which both exhibit long-tail feature. this distribution, the short sequences dominate the datasets thus showcasing more optimization potential. In contrast, long sequences also account for the majority bimodal distribution, leading relatively small optimization space. Specifically, when training Qwen-7B with this datasets, the major sequence length exceeds the BucketSize thus leading to limited speedup. We can further extend the BucketSize by combining more optimization techniques like parameter-efficient fine-tuning (PEFT)
dataset, which exhibits
Step-by-step Evaluation. Additionally, conduct step-by-step evaluation with the training settings mentioned above. As shown in Figure 3 we successively enable DACP and GDS to test the effectiveness of each part in Skrull. Experimental results show that both components are effective and can cooperate well to further improve the end-to-end system performance in Long-SFT.
Performance Impact of BatchSize and BucketSize. To investigate the performance impact of BatchSize and BucketSize, we conduct experiments ChatQA2-long-SSF using Qwwe2.55-0550 with the default setting of <DP=1, CP=8, BatchSize=64, Bucketšize=26K>. As shown in Figure fa), adjust the BatchSize from to 54 and the end-to-end speedup also improves. We attribute this

---

performance gain to the expanded scheduling scope afforded by larger batch sizes. However, as the BatchSize increases further, the sampled batches gradually converge to the sequence length distribution of the dataset, causing the performance gains stabilized within reasonable range. Additionally, we also evaluate the effect of BucketSize. Figure 4 shows that increasing the BucketSize from 8K to 32K (values in parentheses) improves the speedup until an out-of- (OOM) error occurs. This indicates that while a larger BucketSize enhances performance, it also raises the risk of OOM errors. Therefore, it is important set appropriate BucketSize, highlighting the importance of performance modeling module Skrull.
Table 3: Scheduling Strategies Comparison
Table 4: Latency Comparison per Iteration
Method
Speedup
Iteration
Baseline
RR
Skrull
RR w/ roll-back
1.17x
1
(36, 36)
(10, 49)
(21,
RR w/o roll-back
OOM
2
(40, 40)
(29, 47)
(37,50)
Skrull w/ roll-back
1.40x
3 (35, 35)
(14, 45)
(32, 49)
Skrull w/o roll-back
OOM
4
(46, 46) (46, 47)
(45, 49)
Case Study In this section, present quantitative analysis of the training process of Skrull. We conduct experiments using the Qwen2.5-0.55 model with the ChatQA2-Lonn--SH dataset under the configuration <DP=1, CP=8, BatchSize=64>. To evaluate the effectiveness of our heuristics implementation, we compare it against a round-robin (RR) scheduling strategy, which assigns sequences in a simple round-robin manner (details in Appendix B.2). Additionally, we test both scheduling algorithms with and without roll-back mechanism to further assess the trade-off design in Skrull. As shown in Table 3 Skrull significantly outperforms the RR scheduling. We analyze this result by examining quantitative data in the first four iterations. Table 4 (presented in tuple format) reports the minimum and maximum peak memory usage (in gigabytes) across all GPUs during each iteration. Compared to the baseline, both Skrull and RR scheduling allocate more sequences locally at the cost of increased memory imbalance. While such imbalance is acceptable as long as it dose not exceed memory capacity, it raises the risk of OOMs. Therefore, as shown in Table 3 without the roll-back mechanism, both scheduling strategies result in OOMs, underscoring the importance of this safeguard. In contrast to RR scheduling, Skrull achieves a better computational balance (indicated by the speedup) while adhering to memory constraints, demonstrating the effectiveness of its trade-off design in Skrull.
6 Related Works
From the perspective of data engineering. those works 23 training datasets for long context fine-tuning, From the perspective of training system, Long 3 adopts a sorted batching strategy to optimize system efficiency in long context fine-tuning phase. Chunkflow 2 organize the training data into fixed size chunks, enabling controllable peak memory consumption and reduced pipeline bubbles. Additionally, some works employ dynamic parallelism settings X to handle varying length sequences, which is similar to long-SFT. In contrast, Skrull adopts fixed parallelism settings and is orthogonal to those methods. Another type of works are parameter efficient finetuning (PEFT) I1 5 and Skrull is also effective for this methods.
19 10 involve meticulously gathering
7 Conclusion
In this paper, we provide a new perspective of data scheduling to enhance the training efficiency in Long-SFT scenarios. The heterogencous data distribution in Long-SFT poses dilemmas for existing training systems on configuring parallelism strategies and ensuring the load balance. To tackle those challenges, we propose Skrull, a dynamic data scheduler dedicated for Long-SFT. Through dynamic data scheduling, Skrull achieves efficient training on both long sequences and short sequences. Additionally. we formulate the scheduling process as joint optimization and adopt a lightwight scheduling algorithm. Experimental results demonstrate that Skrull outperforms DeepSpeed by 3.76x on average (up to 7.54x) in real- long-SFT. Furthermore, we believe that Skruil can serve as an effective solution in other scenarios especially when dealing with mixture of long and short training data, such reinforcement learning from human feedback (RLHF).
10

---

Acknowledgments
This work is supported by the following funding: Beijing Natural Science Foundation (4254087), National Science Foundation of China (62502501, 92270206, 62377233), China National Postdoctoral Program for Innovative Talents (BX20240383), Strategic Priority Research Program Chinese Academy of Sciences (XDB0500102), and Alibaba Research Intern Program. The model training were performed on the robotic Al-Scientist platform of Chinese Academy of Science and Alibaba Cloud Platform for AI (PAI).
References
[1] nvddiuChaaQQA2-lonnSSST datasets hugging face, URL htttts:/huggiiggfaa
oo/Aattascts//vvadda///ttttt---- SFT-data.
[2]
pesstoowilppdaaccco22220001--0-0 datasets at hugging face,
URL https://
cn- 20230720-
[3] Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li. LongAlign: A recipe for long context alignment of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1376-1395, Miami, Florida, USA. November 2024. Association for Computational Linguisties. doi: 10.18653/v 1186553v1/20244 1/2024. finiinin www.inng--ennll.44444 emnlp.74. URL https://aclanthhollee org/ foraaniiinnngns emnlp. 74/
[4] Suresh Bolusani, Mathieu Besançon, Ksenia Bestuzheva, Antonia Chmiela, João Dionísio, Tim Donkiewicz, Jasper van Doommaaen, Leon Eifler, Mohammed Ghannam, Ambros Gleixner, Christoph Graczyk, Katrin Halbig, Ivo Hedtke, Alexander Hoen, Christopher Hojny, Rolf der Hulst, Dominik Kamp, Thorsten Koch, Kevin Kofler, Jurgen Lentz, Julian Manns, Gioni Mexi, Erik Mühmer, Marc E. Pfetsch, Franziska Schlösser, Felipe Serrano, Yuji Shinano, Mark Turner, Stefan Vigerske, Dieter Weninger, and Liding Xu. The scip optimization suite 9.0, 2024. URL
27772
[5] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. LongLoRA: Efficient fine-tuning of long-context large language models. URL https://
arxiv Frxiv.org/oo org.oog///aaoo. httggg/bs//309/.. 12307 _eprin:: 2309.12307.
[6] Tri Dao. FlashAttention--:: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
[7] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[8] Hao Ge, Fangeheng Fu, Haoyang Li, Xuanyu Wang, Sheng Lin, Yujie Wang, Xiaonan Nic, Hailin Zhang, Xupeng Miao, and Bin Cui. Enabling parallelism hot switching for efficient training of large language models. In Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, pages 178-194, ACM. ISBN 979-8-4007-1251-7, doi:
10.1 10.114//694475536655555
URL httph:ttt///at acm. org htgg/doi/10..o /115/1194441536668898 3695969
[9] Google Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL http::tttts:xxiiee http:/awwww.....aaaaaaaaaaa wwtp://///aaaaaaaaaaaaa wwwp///3....0555
[10] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Farcia-Olano, Diego Perino, Dicuwke Hupkes, Egor Lakomkin, Ehab

---

AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang. Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria T'impouuelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy. Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajiwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro. Roshan Sumbaly, Ross Taylor. Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shao- liang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao. Ujiwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero. Vladan Petrovic. Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet. Xiaodong Wang. Xiaofang Wang. Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mcjia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern. Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph. Helen Suk, Henry Aspegren, Hunter Goldman. Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, -apt-see Gaya, Jeff Mareus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings. Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt,
12

---

Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar. Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li. Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinotr, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho. Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li. Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla. Vlad Ioneseu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang. Ying Zhang. Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models. URL ptatp://arxizn www.aawwww.waaaaaaaaaaaaaa
org/arg.///...o. abs///b///// 2407 21783
[11] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022. URL http://ablp. umi-trier de/ bt/db/contie http//oo///cc///122/000 html# HuSWALMWC22
[12] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminadabi, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. System optimizations for enabling training of extreme long sequence transformer models. In Proceedings of the 43rd ACM Symposium on Principles of Distributed Computing, pages 121-130. ACM. ISBN 979-8- 4007-0668-4. doi: 10.1 10.1145/3662111 258.666682188366886666
URL htt.h::///// //d1 acm. org//oii/10 1145/
3662158.3662805
[13] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxiv. org/ abs//412, 6980
[14] Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large trans- former models. ML.Sss URL http:///pooceeings.llll...eeee www.:diinnn.mm................ee httpa///00000..10cceceeeeeeeeeee
files/ fatpee//palepp/e/////aapppp html
[15] Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li. Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: experiences on accelerating data parallel training. Proc. VLDB Endow,, 13(12):305533013 August 2020. ISSN 2150-8097. doi: 10.14778/3415478.4455555 URL https:////d org/ 10.14778/3415477 3415530
[16] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ringattention with blockwise transformers for near-infinite context. In The Twelfth International Conference Learning Representations, 2024. URR..https:///// htttpss:/opperreiieeeeeee wtwpiiiwwwweeefttrne net:/foruumIt--Wallllllllaa
[17] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL. htt.:/tttp::/oo/mtie wtw.pperrreiiee.eee/tuual net/forum? id id-Bkg6RiCqy7
13

---

[18] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shappiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang. Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier. Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dumning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider. Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk. David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak. Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pan- tuliano. Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard. Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Bric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such. Natalie Summers. Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang. Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: http::ttt:///rtttt. //axxvv wwepxxvv.org///// arg//bs//an arg/abs/2303. 2203. 2303.08774
[19]
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. URL httttttp///xii>> org/abg............ 25112
[20] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimiza- tions toward training trillion parameter models. In Proceedings of the International Conference
14

---

for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press, 2020.
ISBN 9781728199986.
[21] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyann Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnie, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. URL https: //arxiv org/ abs/ 2307. 09288 _eprint: 2307.09288.
[22] Xiulong Yuan, Hongtao Xu, Wenting Shen, Ang Wang, Xiafei Qiu, Jie Zhang, Yuqiong Liu, Bowen Yu, Junyang Lin, Mingzhen Li, Weile Jia, Yong Li, and Wei Lin. Efficient long context fine-tuning with chunk flow. In Forty-second International Conference Machine Leurning, 2025. URL https: /ttp////ppe/eveeeeeeeeeeee net forum?id=rz220gf
[23] Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou. LongSkywork: A training recipe for efficiently extending context length in large language models. URL httpt:tttttttiiti .rg/ abs/2406. 00605
_eprin:: 2406.0060S
[24] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023. URL httph:: /tttps://arxivv org/ abs/
2304. 11277
[25] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. LMSYS-chat-Ime A large-scale real-world LLM conversation dataset. In The Twelfth International Conference on Learning Representutions, 2024. URL https: pestps::/openreee //penreview net/forum?ic http:forrumiddBB0Dxxxxrrrt BOFDKXEWtO
15

---

A Precision Validation
To evaluate the loss equivalence, compare the loss carve between Skrull and standard training method when training Qwen2.5-0.5B on LMsysChat IM. The data scheduling in Skrull alter the accumulation order and we can observe slightly numerical differences due the non-associativity of floating-pott operations. However, Skrull do not alter any contents and orders in each global batch, the optimization trajectory remains equivalent. Therefore, shown in Figure Skrull does not influence the convergence.
Tharing
Figure 5: Loss comparison between Skrull and standard training method.
B Heuristic Scheduling Algorithm
B.1 Function Definition of Heuristic Algorithm
Algorithm 3 Function Definations in scheduling algorithm for DACP
Require: SeqNum K, SeqLens S[K). Buckets C, CP degree N, Loads L[N], RemainBucket RB[N], DACP scheduling result rel.
1: function UPDATEL wAA.TELLLCAAL.. AMPDATELLCCAA(idda rank)
2: RB[rank] + RB frank] S(idz]
> Update remaining bucket capacity Update current load
3: L[rank] + L [rank] FLO Ps (S(iddz)
4: end function
5: function UPDATEALL(dda
6: for = 0 to N do
7:
R. R. Slide] /N
Distribute across all buckets D Update all loads
8: L[i + L FLOPs (idx)) N)
9: end for
10: end function
11: function RoIL.BACK(rankRRBB
12:
for i = 0 to K - 1 do
if ret rank then
14:
ret --1
Distribute the sequence
RB [rak]] < RB [rank]n]] S|i + S/1/NN
16:
if L is not None then
L[rank] L[rank] end if
18:
19:
return True
Success Roll-back
20:
end if
end for
22:
return False
Roll-back Failed
23:
end function
16

---

Algorithm 4 Round-Robin Scheduling Algorithm
1: Input: SeqNum N, SeqLens S, Buckets C, WorldSize ws
2: Output: PartitionIds P
3: for = to m do
4: t FindMMaxxuccketsddsS
5:
if' C 2 STi] then
6:
P
fit the max bucket
7:
else
8:
j FindMinBucketsldss)
9:
if C > S wss then
10:
P
partition the sequence
11:
else
Assert RollBack(j, C)
4 with roll-back
13:
iti-1
14:
continue
15:
end if
16: end
17:
end for
B.2 Round-robin Scheduling Algorithm
C Performance Modeling
C.1 Memory Estimation
Due to limited pages, we discuss the memory estimation methodology of Skrull in this section. The key point of this section is the determination of BucketSize C, which maps memory capacity to sequence token length.
We first analyze the memory consumption during LLMs training. The memory consumption can be roughly categorized into components: the static memory and the dynamic memory. The static memory, which typically includes model parameters and optimizer states, remains roughly constant throughout the training process given specific model configurations and parallelism strategies. In contrast, the dynamic memory or activation memory, varies with the input workload. In transformer architectures, activation memory is proportional to the sequence length. For instance, the Linear module, LayerNorm and Attention on module (using FltehAttention 77 6)) exhibit linear relationship with sequence length. Therefore, we can estimate activation memory for given sequence length using the following equation:
Memory(S) = aS + 3
(12)
Here, the coefficient a and constant 3 is determined at offline profiling. Notably, some memory reduction strategies, such as gradient checkpoints, only affect the and B. We can still apply offline profiling method to estimate activation memory. In our implementation, we found that 3 is usually negligible. Additionally, we employ sequence packing to eliminate padding and enhance performance, allowing us to directly use the total sequence length for memory estimation. Consequently, through offline profiling, we can deduce the BucketSize C' under various settings.
C.2 Computation Estimation
In this section, we describe the methodology used to estimate the computational cost Tromp-
Accurately modeling the computational cost as function of sequence length S is non-trivial. Simply assuming linear or quadratic relationship with sequence length is insufficient because the computational FLOPs of TransformerLayer are dominated by the Linear and Attention in modules, exhibiting a hybrid of linear and quadratic dependencies on S. The relative contributions these components vary depending the specific model configuration. Therefore, we formulate function of FLOPPs to provide roughly computational cost estimation given specific model configuration and sequence length S.
17

---

Given the model configuration of hidden dimension h, key/value hidden dimension h&ve and training batchsize (usually be when employ sequence packing), the FLOPs is estimated as the Equation
FL.OPs(S)) 20 b h22 Sk + hkv Sk +4* Sh
(13)
For each sequence, the Teomp be estimated
Teomp - aFLOPs+
(14)
where all the a and / is determined when offline profiling.
Furthermore, as shown in Figure 6 we plot the relationship between FLOOPs and sequence length for Qwen-2.5-0.5B and Qwen-2.5-7B. The results highlight the distinct characteristies of long and short sequences. For short sequences, both computational workload and activation memory consumption scale roughly linearly with sequence length. However, for the long sequences, the computational workload grows rapidly due to the dominance of the quadratic term, while memory consumption still remains linear, leading to the problem of trade-off between balancing computation and memory, which is discussed in detail in Section 4.3 where we present insights into our heuristic algorithm design.
Additionally, the transition point at which the quadratic term dominates varies depending on the model configuration. demonstrated F Qwen-2.5-7B, which has larger hidden dimension h, exhibits a more rapid increase in FLOPs compared to Qwen-22.-0.555 Although Qwen-2.5-0.55 has slower FLOPs increase, we take example further discuss the distinct characteristics between long and short sequences. In Qwen-2.5-0.5BB the quadratic term begins to dominate only when the sequence length exceeds approximately 4K, exhibits roughly linear relationship in short sequences. However, when S 32K, the total computational workload is 30 times greater than when S 4K, while the memory consumption increases only 4-fold. These estimations further elucidate the distinct characteristics of long and short sequences.
Jecpaeicca Leogen
Figure 6: FLOPs VS Sequence Length Qwen-2.5 0.5B and 7B
C.3 Communication Estimation
For the T Toomm, we can simply profile offline ways. Concretely, when the communication volume is smaller then threshold, the fixed overhead of communication dominates the latency. However, with size increased, the fixed overhead become negligible and the latency is approximately proportional to communication volumes. We can deduce the thresholds, fixed overhead and the estimation function through a simple profiling. As shown in Table we plot the communication performance profiling results. Therefore. we can fit the Equation 16 according to communication volume in different hardware environments. Then, we can derive the communication volume according to sequence length S under different model configurations as shown in Equation 15 where hiddengy and b means hidden dimension of Key/Valur and batch size.
Volume = b * Sh hiddengg
(15)
T Toomm (a.V + Fisedd
(16)
18

---

Table 5: Collective Communication Latency Profiling.
Size Latentttcyy(us)
All gather
All to All
Reduce scatter
All reduce
2
53.29
80.62
59.48
84.65
4
72.52
78.63
79.26
113.3
8
97.86
110.9
104.7
168.4
16
199.3
163.2
177.4
312.2
32
286.2
277.5
269.5
479.2
64
488.6
502.4
458.8
859.7
128
910.6
939.2
864.3
1642.9
256
1758.4
1803.9
1663.9
3197.9
512
3416.4
3411.2
3239.5
6181.2
1024
6467.9
6629.6
6294.3
12126
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We list our contributions in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the BatchSize will limit the optimization in our evaluation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

We list the details about how we analysis the memory and computation in Appendix in this pdf.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide pseudocode and evaluation settings in paper
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [NA]
Justification: We use public data in our experiments. Due to some approval process, we will make our code public as soon as possible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips //nips.cc/ppl wccppublicgguuuiicceeeeeee.oo.pe
http://ooooSuuissssooeee Stt.SSbbbiiiiooooiiiii.oo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We list in evaluation sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
Our experimental results are averaged by multiple runnings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list in evaluation testbed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaaraaa..cpppaaaa//aaaaaa
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is about system optimization.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is about system optimization.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, cite the related paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
25

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The paper do not involve usage of LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
26


Reviews:
[{"review_id": "9BodxGrXkr", "reviewer": "Reviewer_eLuU", "summary": "This paper tackles the efficiency problem in long-context supervised fine-tuning (Long-SFT) for LLMs, where mixing long and short sequences causes system inefficiencies. The authors introduce Skrull, a dynamic data scheduler combining Distributed-Aware Context Parallelism (DACP) and Global Data Scheduling (GDS). Their approach balances computation and memory on the fly, achieving up to 7.5× speedup over DeepSpeed in realistic benchmarks.", "questions": "How robust is Skrull if the input sequence distribution changes over time — does profiling need to be repeated?", "limitations": "Skrull depends on accurate profiling; if cost models are wrong, it may underperform or run out of memory. It currently works with DeepSpeed — porting it to other training frameworks may require extra engineering effort.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\nThe paper addresses a highly practical and timely challenge for modern LLM training. Its system-level method is orthogonal to model or optimizer changes, making it easy to integrate with other techniques. The design is well explained, with clear trade-off modeling and solid experimental validation on real datasets.\n\nWeaknesses\nThe experiments focus on models up to 7B parameters; it’s unclear how well Skrull scales to even larger models. There is no analysis of Skrull’s impact on training convergence or final task accuracy. Also, it lacks direct quantitative comparison with other recent long-context scheduling methods.", "was_revised": true, "final_justification": "This is a good paper with solid results. The authors further provide detailed rebuttals that addressed my concerns. Thus, I will keep my rating.", "rebuttal": "We sincerely appreciate Reviewer **eLuU** for recognizing the value of our work, noting that it **\"addresses a highly practical and timely challenge for modern LLM training\".** In accordance with Reviewer eLuU's suggestions, we have conducted additional experiments to further demonstrate Skrull's effectiveness, precision, and provide a more quantitative comparison with other approaches such as LongAlign [1]. We sincerely thank Reviewer eLuU for the valuable feedback. All additional results will be included in the final version. Below, we provide detailed responses to Reviewer **eLuU's** concerns and questions.\n\n## Responses to Weaknesses\n### ***Concerns 1: How Skrull scales to even larger models***\n\nFirstly, we would like to clarify that Skrull addresses system inefficiencies caused by the dataset characteristics in long-context SFT. Theoretically, such inefficiencies (e.g. load imbalance, redundant communication, kernel degradation) persist regardless of model size. Also, we believe that as model size increases, the kernel degradation problem caused by finer sequence splitting is alleviated due to higher computational intensity.\n\nTo empirically validate this, we tested the Qwen2.5-14B and 32B models on the LMsysChat dataset under the setting <DP=4, CP=8, BatchSize=64>. We adopted LoRA [2] to rapidly evaluate Skrull's effectiveness with larger models and its compatibility with other efficient fine-tuning techniques. The results show that Skrull achieves a 2.37×, 2.54x end-to-end speedup compared to the standard method, respectively. The detailed breakdown is as follows:\n\n\n\n|          | 14B | 32B |\n|:--------:|:--------:|:---------|\n| Baseline     |  1.00×       | 1.00x |\n| DACP       | 1.18×       | 1.24x |\n| DACP + GDS      | 2.37×  | 2.54x |\n\n***Table 1**. Speedup with larger models.*\n\n\n\n**Our experiments demonstrate that Skrull remains effective when scaling to larger models.** Furthermore, our additional experiments show **Skrull's compatibility with other long-context fine-tuning techniques**. We believe Skrull, a precision-lossless system optimization, can provide the long-context community with a more efficient and easily integratable training solution.\n\n\n### ***Concern 2: Analysis of Skrull’s impact on training convergence***\n\nWe thank for Reviewer **eLuU**'s meticulous reviews. As mentioned with other reviewers, we add the loss carve of both Skrull and standard training methods for comparison. Due to the NIPS rebuttal rules, we cannot include loss carves figure directly or external links in the OpenReview to demonstrate Skrull's precision and convergence. Instead, we present some loss value at different training iterations. After 1000 iterations, the loss of both methods converge to about ~0.24. The slight numerical differences are due to hardware-level fluctuations, which do not affect convergence. We will include complete loss carves figure in our final version.\n\n\n| Iteration | Standard | Skrull |\n|:-----------:|:------------:|:------------:|\n| 0         | 7.45541573 | 7.45541573 |\n| 10        | 6.27256107 | 6.27239227 |\n| 20        | 4.24731827 | 4.24617065 |\n| 30        | 2.67359886 | 2.67313461 |\n| 40        | 2.24764290 | 2.24854412 |\n| 50        | 1.88435841 | 1.87571697 |\n| 60        | 1.63205013 | 1.62057266 |\n| 70        | 1.49548502 | 1.48315649 |\n| 80        | 1.38979225 | 1.39394617 |\n| 90        | 1.62660141 | 1.60026455 |\n| 100       | 1.17248878 | 1.17774509 |\n\n***Table 2**. Loss carves comparison during training Qwen2.5-0.5B on LMsysChat1M.*\n\nWe highly acknowledge the importance of optimization precisions and we conduct the module tests in our early proof of concept validation, including the attention module and the training impact of scheduling data within global batch. However, Skrull functions as a system-level optimization, it only alters the assignment of training data to specific GPUs and does not involve any algorithm-level modifications. This helps to explain why we did not initially include this section in our original submission.\n \n\n### ***Concern 3: more direct quantitative comparison with other methods.***\n\nWe sincerely appreciate this suggestion and we have conducted additional experiments for a more comprehensively comparison. As mentioned with other reviewers, we compare with LongAlign [1], which involves sorted batching method to sort the dataset by sequence length and select random consecutive groups for each batch to improve the long-SFT training efficiency. The results are listed in the following Table 3.\n\n\n| Model  | Wikipedia         | LMsysChat1M        | ChatQA2-Long-SFT     |\n|:----------:|:-----------------:|:------------------:|:--------------------:|\n| Qwen2.5-0.5B       | 1.10× *(7.54×)*   | 1.13× *(7.51×)*    | 1.03× *(1.46×)*      |\n| Qwen2.5-7B         | 1.06× *(2.30×)*   | 1.07× *(2.73×)*    | 0.99× *(1.08×)*      |\n\n***Table 3**. Speedup comparison between Sorted Batching (main value) and Skrull (in parentheses)*\n\n\nExperimental results demonstrate that **Skrull significantly outperforms the sorted batching strategy**. Although sorted batching achieve more balance batching by grouping the training sequence by length, it is still a \"static\" batching method. In contrast, Skrull can achieve dynamic data scheduling **guided by performance modeling**. Additionally, Skrull implements DACP to reduce the communication overheads during long-context training. The results will be reflected in our final version.\n\n-----\n\n## Responses to Questions\n### ***Questions 1: How robust is Skrull if the input sequence distribution changes over time — does profiling need to be repeated?***\n\nWe sincerely thank for Reviewer **eLuU** for pointing out this key detail of our implementation. Our profiling is aimed to predict the performance characteristics according to data sequence length. The relationship between sequence length and performance is affected by the model, hardware configuration and training system settings (parallel settings, gradient checkpointing, etc.). If all the settings are same, we do not need to repeat offline profiling even if the sequence distribution changes. However, if the scope of sequence length beyond the profiling scope (eg. maximum sequence length extended), we suggest to repeat for more accurate prediction.\n\n-----\n\n## Comments on Limitations\n\n### ***Skrull relies on accurate profiling***\n\nWe thanks for this insightful comments and we are highly agreed with Reviewer **eLuU**. Skrull depends on accurate profiling because we want to maximize the performance gains. Especially for memory, inaccurate modeling incurs the risk of OOM. But in practice, the profiling results exhibit stable and accurate. We will put some predicted and realistic data in our appendix to illustrate it. \n\n### ***Skrull's portability and engineering efforts***\n\nSkrull is orthogonal to training frameworks. However, it does require some engineering effort to implement, mainly focus on the logic of Dataloader and some custom modules.\n\n-----\n\n## Summary\n\nWe thank Reviewer **eLuU** again for meticulous reviews of our paper, which help us improve both the clarity and strength of our manuscript. We hope our responses have fully addressed your concerns.\n\n[1] Yushi Bai et al., *LongAlign: A Recipe for Long Context Alignment*.  \n\n[2] Edward J. Hu et al., *LoRA: Low-Rank Adaptation of Large Language Models*."}, {"review_id": "BbkCHoyy2k", "reviewer": "Reviewer_N1BH", "summary": "This paper addresses training efficiency losses incurred by training on heterogeneous mixtures of short- and long-context data, as is common in SFT stage of LLM training. The authors propose a novel dynamic data scheduler designed to dynamically schedule long-context SFT for efficiency improvements. The authors show on common datasets that sequence length distributions are highly variable both inter and intra dataset. They propose DACP and GDS, which together compose Skrull. They present analyses of the marginal gains of adding DACP and GDS to the DeepSpeed framework, reporting strong results across three datasets.", "questions": "Some of the writing is garbled and ungrammatical. This does not terribly distract from the content, but is distracting and not publication-ready. Please fix. lines 302-303 for example. \n\nSee weaknesses.\n\nNit:\nFig 2 is quite difficult to read when printed, and in PDF format needs to be heavily zoomed in. Please fix the tiny bold/italicized words.", "limitations": "none", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper structure is easy to follow.\nThe formulation of the problem is lucid, the approach is clearly explained, the results are strong.\nThe efficiency gains of the proposed method seem significant, and likely to be useful to others.\n\nWeaknesses:\nThe writing is occasionally unclear, ungrammatical, or both. This can be easily improved to make the paper much easier to read.\n\nThe section 4.3 discusses the tradeoffs between memory and computation that are central to evaluating Skrull, but at no point are the actual memory costs associated with the experimental results actually presented. Isn't this important to evaluating the technique?", "was_revised": true, "final_justification": "Solid paper with thorough experiments and good results. Will be of interest to others and likely to be replicated.", "rebuttal": "We sincerely appreciate Reviewer **N1BH** for recognizing the value of our work, noting that our work is *\"easy to follow, lucid in problem formulation, clearly explained, with significant results and likely to be useful to others.\"* We are grateful for the constructive feedback, especially regarding writing clarity, figure readability, and the need for further analysis of trade-offs in Skrull. We address each of these points below and will incorporate all necessary revisions in our final version.\n\n### 1. Writing Clarity.\n\nWe thank the reviewer for pointing out that parts of the writing are unclear or ungrammatical (e.g., lines 302–303). We fully agree that writing quality is essential for clarity and readability. Following this suggestion, we have carefully revised the entire paper to improve the writing, correct grammatical issues, and ensure all sentences are concise and well-structured. The final version will be thoroughly proofreading and ready for publication.\n\n### 2. Readability of Figure 2.\n\nWe appreciate the reviewer’s suggestion regarding the legibility of **Figure 2**, especially when printed. We have re-designed this figure to use larger fonts and avoid excessive stylization (e.g., tiny bold/italicized text). We will ensure that all figures in the final version are visually accessible and clear in both digital and print formats.\n\n### 3. Evaluation and Analysis of Trade-offs in Skrull.\n\nWe appreciate the reviewer’s insightful comment regarding the lack of actual memory usage data to evaluate the trade-offs between computation and memory in Skrull.\n\nWe provide additional quantitative data based on training with ChatQA2-Long-SFT under the configuration `<DP=1, CP=8, BatchSize=64>` using Qwen2.5-0.5B. Additionally, we include a naive scheduling algorithm for comparison, which dispatches sequences in a simple round-robin manner. To further evaluate the effectiveness of memory trade-offs in Skrull, we test the round-robin scheduling algorithm both with and without a roll-back mechanism.\n\nWe measure the balance of computation using wall-clock time, where lower time indicates more balanced computation under the same workload. Similarly, we evaluate memory balance by tracking the minimum and maximum peak memory usage across all GPUs.\n\n\n| Method                    | Speedup |\n|:---------------------------:|:---------:|\n| Baseline                  | 1.00×   | \n| Round-robin w/ roll-back | 1.17×   | \n| Round-robin w/o roll-back| OOM     |\n| Skrull                    | 1.40×   | \n\n\n***Table 1**. Comparison between different scheduling strategies.*\n\n\n> *Note 1: Higher speedup (i.e., lower wall-clock time) indicates better computational balance.*\n\n> *Note 2: The roll-back mechanism is a component of Skrull used to avoid out-of-memory.*\n \n\n\n| Iteration | Baseline (Min, Max) | Round-robin w/ roll-back (Min, Max) | Skrull (Min, Max) |\n|:-----------:|:--------------------:|:-------------------------:|:--------------------:|\n| 1         | (36, 36)             | (10, 49)                             | (21, 51)           |\n| 2         | (40, 40)             | (29, 47)                             | (37, 50)           |\n| 3         | (35, 35)             | (14, 45)                             | (32, 49)           |\n| 4         | (46, 46)             | (46, 47)                             | (45, 49)           |\n| 5         | (49, 49)             | (49, 49)                             | (49, 49)           |\n| 6         | (35, 35)             | (16, 46)                             | (29, 50)           |\n| 7         | (33, 33)             | (13, 50)                             | (20, 48)           |\n| 8         | (47, 47)             | (37, 52)                             | (47, 47)           |\n\n***Table 2**. GPU memory usage per iteration (in GB).*\n\n\nThe results in Tables 1 and 2 demonstrate that **Skrull achieves optimal performance while respecting memory constraints**. Compared to the baseline, both Skrull and round-robin scheduling assign more sequences locally, reducing communication and computation degradation at the cost of increased memory imbalance. While such imbalance is acceptable as long as memory capacity is not exceeded, it does increase the risk of out-of-memory (OOM) errors — as observed in the round-robin scheduling without roll-back.\n\nThese results highlight the importance of the **roll-back mechanism** and **cost modeling** in Skrull. Compared to round-robin with roll-back, Skrull provides better computational balance and lower memory imbalance, demonstrating the effectiveness of its scheduling algorithm.\n\nWe will include a detailed version of this analysis, along with additional quantitative data, in the final version.\n\n---\n\n### Summary\n\nWe thank Reviewer **N1BH** again for the valuable feedback. We believe that the additional analyses and the improved presentation will further strengthen the clarity, completeness, and readability of our paper. All suggested corrections will be reflected in the final version."}, {"review_id": "yXqeIGhAD7", "reviewer": "Reviewer_9acd", "summary": "This paper investigates how to conduct efficient training in the presence of both long and short data sequences. It proposes a method named Skrull, which includes Distributed-Aware Context Parallelism (DACP) and Global Data Scheduling (GDS). Experimental results demonstrate that this method achieves a significant efficiency improvement (3.76x) compared to DeepSpeed in real-world long-SFT scenarios.", "questions": "See weakness", "limitations": "None", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### **Strengths:**\n\n1. The proposed method is novel and its solution is well-supported by theory, making the approach appear solid.\n    \n2. The experimental results are impressive. The average speedup of 3.76x, with a peak of 7.54x, strongly validates the effectiveness of the method. This is a great contribution to the long-context community.\n\n### **Weaknesses:**\n\n1. The organization of the paper seems imbalanced. The experiments section is too brief (only one page), which slightly weakens the evidence supporting the method's effectiveness. The analysis is also insufficient. For instance: 1) There is a lack of discussion regarding more hyperparameters. 2) The paper claims the \"Scheduling Algorithm of GDS\" has almost no overhead, but this still should be presented with quantitative data.\n    \n2. This method alters the construction of batches. Although the authors state on page 5, lines 164-165, that \"We limit the scheduling scope to the global batch because it represents the maximum scope that maintains mathematical equivalence for mainstream optimizers such as Adam [13] and AdamW [17],\" this assertion alone is relatively weak. The authors should provide a comparison of model performance (e.g., loss curves or downstream benchmark results) between a model trained with Skrull and one trained with a standard method.\n    \n3. The number of baselines is limited. The only comparison made is against DeepSpeed. The authors could consider comparing their method against other strategies like sorted batching, such as LongAlign, as mentioned in the related work section on line 305.\n\n### **Note:**\n\nI highly recognize the contribution of this paper in terms of efficiency, and I believe it is an important contribution. However, as an academic paper, its current presentation has several issues, as detailed in the weaknesses above. If these issues are addressed, I would be willing to raise my score.", "was_revised": true, "final_justification": "The authors addressed most of my concerns in their rebuttal. My concerns are majorly regarding the comprehensiveness of the experiments. Through their response, the authors gives very detailed results. Thus, I raised my rating from borderline reject to borderline accept.", "rebuttal": "We sincerely thank Reviewer **9acd** for taking the time to review our work and for highly recognizing the novelty and effectiveness of our approach, especially noting that it is **\"a great contribution to the long-context community.\"** We appreciate Reviewer **9acd's** constructive feedback on the current presentation, which we will carefully address in the final version. Below, we summarize the reviewer’s concerns and describe how we plan to resolve them.\n\n## Revision Summary\n\nWe will add several subsections to enhance the Evaluation and Experimental Analysis sections. To allocate space, we will move part of Skrull's implementation details to the appendix and extend the experimental section by approximately one page.\n\n**First, we will add three new subsections to Section 5 (Evaluation):**\n\n1. **Precision Validation** (after \"Overall Performance\"): We will present loss curves comparing Skrull and standard training to demonstrate their training equivalence.\n2. **Impact of BucketSize** (after \"Performance Impact of BatchSize\"): We will analyze how this hyper-parameter affects performance and highlight the importance of accurate cost modeling.\n3. **Case Study** (at the end of Evaluation): We will analyze a realistic training case with memory footprints and wall-clock time to illustrate the trade-off between memory and computation, as discussed in Section 4. (Due to limited characters, see the quantitative data and analysis in our response to Reviewer N1BH (Point 3) ).\n\n**Additionally, we will include a paragraph quantifying the scheduling overhead**, demonstrating that Skrull enables online scheduling with negligible cost.\n\n**Second, we will expand the baseline comparisons:**\n\n1. We include **sorted batching in LongAlign** [1] under our experimental settings and integrate the results into Figure 3. Our method significantly outperforms the sorted batching.\n2. We evaluate Skrull on larger models and test compatibility with algorithm-level optimizations such as LoRA [2], further highlighting Skrull’s versatility and ease of integration.\n\n\n------\n\n## Responses to Concerns\n\n### ***Concern 1: The experiments section is too brief. (1.1) Lacking of discussion regarding more hyper-parameters. (1.2) Lacking of quantitative data to illustrate the overhead of Skrull.***\n\nWe thank Reviewer **9acd** for this useful comments. We have included new experiments and discussions.\n\n●**1.1 Experiments on more hyper-parameters:** Besides the \"BatchSize\", we add a subsection to **discuss the performance impact of \"BucketSize\"**.\n\nWe test the end-to-end speedups with multiple BucketSize settings using Qwen2.5-0.5B in ChatQA2-Long-SFT dataset with the setting <DP=1,CP=8,BatchSize=64>. The table below demonstrate that different \"BucketSize\" setting significantly influence the performance. \n\n| BucketSize | Time (s) | Speedup |\n|:-------:|:---------:|:-------:|\n| baseline | 155 | 1.00x |\n| 6K      | 236       | 0.66×   |\n| 8K      | 193       | 0.80×   |\n| 10K     | 177       | 0.88×   |\n| 12K     | 170       | 0.91×   |\n| 14K     | 165       | 0.94×   |\n| 16K     | 162       | 0.96×   |\n| 18K     | 153       | 1.01×   |\n| 20K     | 143       | 1.08×   |\n| 22K     | 132       | 1.17×   |\n| 24K     | 126       | 1.23×   |\n| **26K** | 119      | 1.30×   |\n| 27K     | 116       | 1.34×   |\n| 28K     | OOM    | -       |\n| 30K     | OOM    | -       |\n| 32K     | OOM    | -       |\n\n***Table 1**. Speedup with different BucketSize.*\n\n**Although larger BucketSize brings more performance gains, it also increases the risk of out-of-memory (OOM) errors.** Therefore, it is important to set appropriate \"BucketSize\". In Skrull, BucketSize is determinted by the memory modeling results in offline profiling.  Under this training setting and model size, the predicted ideal BucketSize is range from 26K to 27K, thus we choose 26K as default BucketSize in our evalution section. **The experimental results further demonstrate the effectiveness of performance modeling in Skrull.**\n\nAdditionally, We also evaluate Skrull's performance in larger model and compatibility with other SFT optimizations such as Lora[2]. We test Qwen2.5-14B and 32B model in the manner of Lora using LMsysChat1M. The table below demonstrate Skrull can work seamlessly with Lora, showcasing the versatility of Skrull. We believe Skrull, a system-level long-SFT optimization, can work seamlessly with more algorithm perspective optimization and provide community a effective and easy to use training solution.\n\n|          | 14B | 32B |\n|:--------:|:--------:|:---------|\n| Baseline     |  1.00×       | 1.00x |\n| DACP       | 1.18×       | 1.24x |\n| DACP + GDS      | 2.37×  | 2.54x |\n\n***Table 2**. Speedup with larger models.*\n\n\n●**1.2 Scheduling overhead anaylsis and quantitative data in Skrull.**\n\nFirstly, the scheduling process is implemented in the DataLoader. Skrull only request the sequence length information in DataSampler and generate the scheduling plans to DataLoader. From the feature of DataLoader, Skrull's scheduling is doing on CPUs and is fully asynchronous with the model training on GPUs. Therefore, **the overheads can be overlapped during the training.**\n\nSecondly, we also record the quantitative costing during realistic training with Qwen2.5-0.5B in ChatQA2-Long-SFT. The scheduling cost (including both GDS and DACP) is ranged from 10ms to 50ms. In contrast, training costs always elapsed 1 second or more per iteration, confirming **Skrull’s overhead is negligible and fully amortized**. \n\n\n### ***Concern 2: Provide a comparison of model performance between Skrull and standard training.***\n\nWe sincerely thank the Reviewer **9acd** for this important point. Although Skrull reorganizes sequences within the global batch for system efficiency, it does **not** alter the global batch content or its optimizer interaction. Since gradients are accumulated and averaged within each global batch before the optimizer step, the optimization trajectory remains equivalent, except for negligible numerical differences.\n\nNevertheless, to empirically validate this, we compare the training loss between Skrull and standard methods. After 1000 iterations, both methods converge to a loss of ~0.24, as shown in the partial results below:\n\n\nWe sincerely thank the reviewer for this important feedback. We would like to further clarify that Skrull do **not** alter the global batch content and the global batch orders from the perspective of model optimization. Since gradients are accumulated and averaged within each global batch before the optimizer step, the optimization trajectory remains equivalent. Skrull reorganizes the calculation orders (gradients accumulation orders) within global batch and the way they are executed in GPU (splited or not, in GPU 0 or GPU 1) for system efficiency. **However, we are highly agreed with reviewers that the precision validation is crucial to optimization.** Therefore, to empirically validate this, we compare the loss curves between Skrull and standard methods. Due to the rules of rebuttal, we can only show some points in the following table.  After 1000 iterations, the loss of both methods converge to a loss of ~0.24. The slightly numerical differences are caused by hardware-level numerical fluctuation, which do not influence the convergence. We will plot the figure of whole loss curves comparison in our final version.\n\n\n| Iteration | Standard | Skrull |\n|:-----------:|:------------:|:------------:|\n| 0         | 7.45541573 | 7.45541573 |\n| 10        | 6.27256107 | 6.27239227 |\n| 20        | 4.24731827 | 4.24617065 |\n| 30        | 2.67359886 | 2.67313461 |\n| 40        | 2.24764290 | 2.24854412 |\n| 50        | 1.88435841 | 1.87571697 |\n| 60        | 1.63205013 | 1.62057266 |\n| 70        | 1.49548502 | 1.48315649 |\n| 80        | 1.38979225 | 1.39394617 |\n| 90        | 1.62660141 | 1.60026455 |\n| 100       | 1.17248878 | 1.17774509 |\n\n***Table 3**. Loss carves comparison during training Qwen2.5-0.5B on LMsysChat1M.*\n\n### ***Concern 3: The authors could consider comparing their method against other strategies like sorted batching.***\n\nWe thank for this helpful comments. To further evaluate our optimization, we add sorted batching strategy in LongAlign[1] as our baseline. As shown in the following table, Skrull outperform sorted batching significantly. Although sorted batching achieve more balance batching by grouping the training sequence by length, it is still a \"static\" ahead-of-time batching method. In contrast, Skrull can achieve dynamic just-in-time data scheduling guided by performance modeling. Additionally, Skrull also implement DACP to reduce the communication overheads during long context training. The results will be reflected in our final version.\n\n| Model Size | Wikipedia         | LMsysChat1M        | ChatQA2-Long-SFT     |\n|:----------:|:-----------------:|:------------------:|:--------------------:|\n| 0.5B       | 1.10× *(7.54×)*   | 1.13× *(7.51×)*    | 1.03× *(1.46×)*      |\n| 7B         | 1.06× *(2.30×)*   | 1.07× *(2.73×)*    | 0.99× *(1.08×)*      |\n\n***Table 4**. Speedup comparison between Sorted Batching (main value) and Skrull (in parentheses)*\n\n----\n## Summary\nWe thank Reviewer **9acd** again for their thoughtful comments and suggestions. We believe that the revisions will significantly improve the clarity of our paper, and we look forward to presenting the enhanced version.\n\n[1] Yushi Bai et al., *LongAlign: A Recipe for Long Context Alignment*.  \n[2] Edward J. Hu et al., *LoRA: Low-Rank Adaptation of Large Language Models*."}]

Author rebuttal:

Meta-review: The paper introduces Skrull, a novel dynamic data scheduler designed to accelerate the supervised fine-tuning of large language models on datasets with mixed long and short sequences. By intelligently balancing computational loads and memory usage across GPUs, Skrull addresses system inefficiencies inherent in long-context training. As highlighted by reviewers, paper's core strengths are novelty, practicality and very promising speedup results (up to 7.54x).

Initially, all three reviewers shared concerns about the insufficient experimental results, lack of clarity and polish. Reviewer eLuU questioned Skrull's scaling to larger models and its impact on convergence, while also wanting comparisons to other schedulers. Similarly, Reviewer 9acd found the experiments section too brief, asking for more baselines like sorted batching, analysis of the scheduling overhead, and proof (e.g., loss curves) that the method didn't alter model performance. Reviewer N1BH focused on the lack of quantitative data to validate the paper's claimed memory-computation trade-offs and pointed out minor presentation issues. The authors effectively addressed every concern in their rebuttal by providing new experiments on larger 14B and 32B models, a direct speedup comparison against the "sorted batching" method, detailed loss curve data confirming mathematical equivalence, and tables showing the negligible scheduling overhead and memory usage, which ultimately satisfied the reviewers and led to increased scores. Meta reviewer also notes that authors do not provide any test loss/accuracy numbers. Authors are highly encouraged to revise their manuscript to address above concerns.
Decision: Accept (poster)