Title: Hierachical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM
Venue: NeurIPS 2025
Accepted: yes
Keywords: Deep Learning

LLM

SFT

LongContext
Abstract: Training Long-Context Large Language Models (LLMs) is challenging, as hybrid training with long-context and short-context data often leads to workload imbalances. Existing works mainly use data packing to alleviate this issue, but fail to consider imbalanced attention computation and wasted communication overhead. This paper proposes Hierarchical Balance Packing (HBP), which designs a novel batch-construction method and training recipe to address those inefficiencies. In particular, the HBP constructs multi-level data packing groups, each optimized with a distinct packing length. It assigns training samples to their optimal groups and configures each group with the most effective settings, including sequential parallelism degree and gradient checkpointing configuration. To effectively utilize multi-level groups of data, we design a dynamic training pipeline specifically tailored to HBP, including curriculum learning, adaptive sequential parallelism, and stable loss. Our extensive experiments demonstrate that our method significantly reduces training time over multiple datasets and open-source models while maintaining strong performance. For the largest DeepSeek-V2 (236B) MoE model, our method speeds up the training by 2.4$\times$ with competitive performance. Codes will be released at https://github.com/ModelTC/HBP.

Hierarchical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM

Abstract

Training Long-Context Large Language Models (LLMs) is challenging, as hybrid training with long-context and short-context data often leads to workload imbalances. Existing works mainly use data packing to alleviate this issue, but fail to consider imbalanced attention computation and wasted communication overhead. This paper proposes Hierarchical Balance Packing (HBP), which designs a novel batch-construction method and training recipe to address those inefficiencies. In particular, the HBP constructs multi-level data packing groups, each optimized with a distinct packing length. It assigns training samples to their optimal groups and configures each group with the most effective settings, including sequential parallelism degree and gradient checkpointing configuration. To effectively utilize multi-level groups of data, we design a dynamic training pipeline specifically tailored to HBP, including curriculum learning, adaptive sequential parallelism, and stable loss. Our extensive experiments demonstrate that our method significantly reduces training time over multiple datasets and open-source models while maintaining strong performance. For the largest DeepSeek-V2 (236B) MoE model, our method speeds up the training by 2.4 × with competitive performance. Codes will be released at https://github.com/ModelTC/HBP.

1 Introduction

Large Language Models (LLMs) [1, 2, 3] have achieved state-of-the-art performance in tasks like machine translation, summarization, and code generation. However, many applications demand to process and understand long-context information [4, 5, 6], such as summarizing books, analyzing legal documents, or retaining context in multi-turn conversations. This underscores the necessity for long-context LLMs that can efficiently process long input sequences.

As mentioned in [1], incorporating long context during the Supervised Fine-Tuning (SFT) [7] stage is highly necessary. On the other hand, general short-context data is also crucial to maintain the model's general capabilities. However, this hybrid dataset composition introduces significant challenges, primarily in terms of speed and accuracy. For speed, long-context data intensifies training inefficiencies due to imbalanced workloads; For accuracy, long-context data can degrade performance on short-context tasks, affecting the model's general capabilities. These challenges hinder the efficiency and effectiveness of SFT for long-context LLMs.

The workload imbalance caused by the hybrid of long and short data arises from two main aspects: (1) within mini-batch imbalance [8], which is caused by excessive padding from randomized mini-batch

High

Comm

a) Naive Packing

Short

Medium

Short

Short

Long

Short

Short

Short

Short

Medium

Medium

Medium

Medium

Long

Long

Long

b) Hierarchical Balance Packing

Low

Comm

High

Comm

128K Packing Group

128K Packing Group

16K Packing Group

32K Packing Group

Long

Medium

Medium

High

CR

Low

CR

High Attention Complexity Variance       High ABR

Long

Short

Long

Short

Short

Short

Low Attention Complexity Variance       Low ABR

construction, and (2) across mini-batch imbalance [8], which is caused by uneven computation distribution over data parallel replicas. Existing approaches mainly address this issue using data packing [6, 9], which combines variable-length data into fixed-length mini-batches. While data packing helps mitigate workload imbalances, it can also introduce new challenges. First, data packing alters the data distribution, which might affect the models' performance. Second, the complexity of attention computation for short- and long-context data differs significantly. The attention complexity of the packed samples is the sum of the attention complexities of all samples within each packing group. As illustrated in Figure 1, naive packing leads to high variance in attention complexity by directly mixing long and short samples, which causes imbalanced attention computation and workload imbalance. Third, handling long-context data requires sequential parallelism (SP) and collective communication for attention computation [10, 11], which short data do not need. When mixed, short-context data also requires sequence parallel communication, leading to wasted communication time. Larger models, such as DeepSeek-V2 (236B) [12] MoE (Mixture of Experts) models, introduce higher communication overhead due to the increased number of parameters.

To overcome the limitations of data packing, we propose Hierarchical Balance Packing (HBP), an innovative method that proposes multi-level data packing instead of conventional single-level data packing. HBP consists of three key components: (1) What are the optimal packing groups? (2) How to assign training samples to their optimal group? (3) How can a long-context LLM be trained with that data? Firstly, we propose hierarchical group auto-selection to determine the optimal packing-length group set and corresponding configurations, including the packing length, Gradient Checkpointing configuration [13], and the SP degree (how many partitions the data is divided into). Secondly, we propose balanced packing to allocate each sample to the optimal group, aiming to minimize imbalanced attention computation and communication overhead. Thirdly, we adopt alternative training between different packing groups, along with curriculum learning and a stable loss normalizer to stabilize the training process.

We validate the effectiveness of our method through extensive experiments in multiple settings. For example, on datasets Tulu3 [14] (32K) + Longcite [15] (128K), our approach speeds up 2.4 × (57.1 to 23.8 GPU Days) on DeepSeek-V2 (236B) [12]; On datasets OpenHerme [16] (4K) + Longcite (128K), our approach reduces training time from 2.95 to 2.04 GPU Days about 1.45 × speeds up on LLama3.1-8B [1]. More importantly, our method preserves performance on both short- and long-context datasets while achieving significant efficiency gains. Experiments on various models at different scales like LLama3.1-8B [1], Qwen2.5-32B [2], Qwen2.5-72B [2], and DeepSeek-V2 (236B) demonstrate consistent improvements, showing the effectiveness and generalizability of HBP.

2 Related Works

2.1 Long-Context LLM

Long-Context Extension . Long-Context Extensions aim to enhance LLMs' capabilities in handling long contexts. Current research can be broadly categorized into approaches that require fine-tuning

and those that operate in a zero-shot manner. Zero-shot approaches often leverage techniques such as prompt compression [17] or specially designed attention mechanisms [18, 19]. On the other hand, fine-tuning methods primarily focus on extending position encoding, such as RoPE-based approaches [5], or utilizing memory-augmented architectures [20].

Long-Context Supervised Fine-Tuning . For Long-Context SFT, research mainly concentrates on generating long-context datasets [21] and establishing corresponding benchmarks [22, 15, 23]. LongAlign [6] also focuses on workload balance and accuracy degradation issues. LongAlign proposed using packing and loss-reweighting to mitigate these issues. However, they failed to recognize the imbalanced attention computation and wasted communication overhead due to the packing of short- and long-context data. FlexSP [24] dynamically organizes training samples into different data groups and uses flexible sequence parallelism to enable hybrid training. Its online data organization introduces significant overhead (5-15 seconds) each iteration and fails to handle attention computation complexity across data-parallel (DP) groups. In contrast, our method introduces only negligible overhead and achieves a better computation balance of attention.

2.2 Data Packing

Data packing [9] is a more practical approach compared to randomly organizing data batches in LLM training. It reduces padding within batches and minimizes idle time across different data-parallel groups. Common packing methods include Random Packing [25], Sorted Batching [6], First Fit Shuffle (FFS), First Fit Decrease (FFD), Best Fit Shuffle (BFS), Shortest-Pack-First HistogramPacking (SPFHP) [26], Iterative sampling and filtering (ISF) [8]. However, those packing methods operate on a fixed length. HBP operates data globally by introducing multiple packing groups of varying lengths, enabling more flexible and efficient handling of hybrid training with both short- and long-context data.

3 Problem Analysis

In this section, we first define the notations in Table 1 and introduce performance metrics in Section 3.1. We then conduct a preliminary analysis of the commonly used packing methods in Section 3.2 and training strategies in Section 3.3.

3.1 Measuring Metrics

Dist Balance Ratio (DBR) [8] quantifies the computational balance inter-devices based on length. A lower DBR 1 indicates more balanced workload distribution across different devices.

Padding Ratio (PR) [8] measures the proportion of wasted computations resulting from intra-device padding based on input length. A lower PR 1 indicates fewer padding tokens.

Attention Balance Ratio (ABR) is proposed to quantify the imbalance in attention computation for different data inter-devices. Previous metrics estimate the computational cost of attention based solely on the length of the input with full attention. However, using packing algorithms, attention computation becomes a significant factor in the overall cost. Consider the packing of a 4 K sequence as an example, both { 1 K, 1 K, 1 K, 1 K } and { 2 K, 2 K } have the same total length. However, their

actual attention computation differs significantly, with complexities of 4 K 2 and 8 K 2 , respectively. The Attention Balance Ratio (ABR) and example are given by 2. A lower ABR indicates a more balanced attention computation between different DP groups, resulting in less idle time.

Communication Ratio (CR) is proposed to measure the overhead of SP communication. T comm i in CR 3 represents the total number of tokens that require SP communication in the current iteration, while T i denotes the total number of tokens in the current iteration. A lower CR 3 indicates that fewer tokens require SP communication, resulting in reduced communication overhead.

Average Tokens (Ave-T) is defined as the average number of tokens processed per iteration, serving as a measure of the model's workload. A larger Ave-T 3 indicates higher training efficiency.

3.2 Packing Analysis

Importance of Packing . In Table 3, we conduct three batching alternatives, random batching, ISF packing batching [8] (comparison between different packing methods is shown in Appendix A), sorted batching [6] (ensures that the sequences within each batch have similar lengths) at three sequence length 4K, 32K, 128K using Tulu3 [14] dataset. Both sorting and packing reduce DBR and PR, speeding up training. On longer sequences like 128K, packing achieves higher Ave-T than sorting, improving GPU utilization and benefiting mixed-length datasets.

Limitation of Packing . Although packing can partially address the efficiency issues associated with hybrid training, several limitations remain. Imbalance Problem of Attention Complexity: Packing samples with varying sequence lengths leads to differing attention computation complexities. Directly mixing these samples can cause imbalanced workloads and inefficient resource utilization. As illustrated in Table 2, the ABR increases significantly with the sequence length growth, indicating a rise in device idle time. More details are shown in Appendix D. SP Communication Overhead: long sequences require communication for attention computation, whereas short sequences do not. Directly mixing them can lead to extra communication overhead. As shown in Table 2, the CR reaches 1 when the sequence length increases, indicating that all short-context data are involved in unnecessary communication.

3.3 Training Strategy Analysis

Given the GPU resources and the data to be trained, we can select from various training strategies, provided that the VRAM requirements are satisfied. The main factors to consider are the degree of SP and the configuration of Gradient Checkpointing (GC), which is the number of layers where GC is enabled. If the SP degree is small, the VRAM demand is high, which forces an increase in

Long

Short

Short

Short

Short

Medium

Medium

Medium

Medium

Long

Long

128K Packing Group

16K Packing Group

32K Packing Group

Long

Long

Long

Long

Data Pool

Short

Short

Short

Short

Medium

Medium

Medium

with SP = 1

Iteration-1

with SP = 2

Iteration-300

with SP = 8

Iteration-500

No

Comm

High

Comm

Low

Comm

Total Training

Iteration 2610

Total Training

Iteration 190

Total Training

Iteration 300

Medium

Medium

Short

Short

Adaptive SP

Curriculum Learning

Short

Hierarchical

Groups Auto-

n

Profiling

16K

32K

Groups

128K

Data Prepare Pipeline: Hierarchical Packing Groups

Balance

Packing

Packing

the number of GC layers and leads to excessive additional computation. On the other hand, if the SP degree is large, although the VRAM demand is reduced, it introduces additional communication overhead. Therefore, there is a trade-off between the SP degree and GC configuration. Moreover, the optimal strategy varies for different sequence lengths. Table 4 shows that the optimal strategies for 32K, 64K, and 128K are different.

4 Hierarchical Balance Packing

In this section, we first describe how to determine optimal packing length groups (Section 4.1), then explain how each sample is assigned to a group (Section 4.2). Finally, we introduce a dynamic training pipeline for HBP (Section 4.3). The overall framework is shown in Figure 2.

4.1 Hierarchical Groups Auto-Selection

To determine the optimal packing length groups, we design a profile-based auto-selection algorithm as described in Algorithm 1. It operates in two stages: (1) finding the best training strategy for predefining a possible sequence length set (e.g., 8K, 16K, 32K, 64K, 128K) based on naive packing. (2) Deriving the final packing groups by optimizing communication overhead.

Algorithm 1 Hierarchical Groups Auto- Selection

1: Inputs: Lengths L , Profile Time P , Strategy S

2: Stage-1: Find the best training strategy

3: Initialize P ← [] , S ← []

4: for each l ∈ L do

5: s = ( sp, ckpt ) ← FindBestSpCkpt ( l )

6: P. add ( ProfileTime ( s )) , S. add ( s )

7: end for

8: j ← argmin ( P )

9: s best ← S [ j ] , l best ← L [ j ]

10: s max ← S [ - 1] , l max ← L [ - 1]

11: Stage-2: Optimize packing groups for communication

12: l 1 ←⌊ l best /l best.sp ⌋ , l 2 ←⌊ l max /l max.sp ⌋

13: if l 2 > l best then

14: L p ← [ l 1 , l best , l 2 , l max ]

15: else

16: L p ← [ l 1 , l best , l max ]

17: end if

18: return L p

Algorithm 2 Balance Packing

1: Inputs: Dataset D = { x 1 , x 2 , . . . , x n } , hier- archical packing groups L p = { l 1 , l 2 , . . . , l n }

2: G = { G 1 , G 2 , . . . , G n } : packed data group

3: B = { B 1 , B 2 , . . . , B n } : final batched data

4: GroupData ( D,L p ): splits D subsets [ D 1 , D 2 , . . . , D n ] by packing groups L p .

5: Packing G i = ( D i , l i ): packing D i by l i

6: GreedyFill G i = ( G i , l i , [ D i - 1 , . . . , D 1 ] ): fill data from smaller groups to reduce PR.

7: Balance Batching( G i ) : construct balance batches by sorting packed data according to attention complexity A in Section 3.1.

8:

9: Initialize B ← [ ]

10: [ D 1 , . . . , D n ] ← GroupData( D , L p )

11: for i = n to 1 do

12: G i ← Packing( D i , l i )

13: G i ← GreedyFill( G i , l i , [ D i - 1 , . . . , D 1 ] )

14: B i ← Balance Batching( G i ), B .add( B i )

15: end for

16: return Shuffle( B )

Medium

Medium

Naive 32K Packing Group

Short

Short

Short

Short

Medium

Short

Short

Short

Short

32K Packing Group

Medium

Medium

Medium

SP = 1

SP = 2

No

Comm

L <= 16K

L > 16 K

SP = 2

SP

Comm

16K Packing Group

SP

Comm

Naive Packing Groups [32K]

Optimized Packing Groups [16K, 32K]

16K Packing Group

input length l , we compute the optimal degree of SP and the configuration of the GC s = ( sp, ckpt ) with the FindBestSpCkpt function, which achieves the optimal trade-off between the degree of SP and the configuration of the GC. More details of FindBestSpCkpt are shown in Appendix I.1.

Specifically, given a packing length l , we iterate all possible sp degrees and ckpt GC configurations and profile their iteration time. The best combination ( sp, ckpt ) is then found by a greedy method. Once all input lengths are processed, the algorithm selects the best strategy by identifying the index j that minimizes the profiling times in P . The optimal strategy and input length are:

Additionally, the maximum input length l max and its corresponding strategy s max are also retained.

Stage 2: Optimize packing groups for communication .

After calculating the optimal l best and the corresponding s best , we can optimize it further to reduce communication overhead and obtain the final packing group lengths. Taking 32K packing groups in Figure 3 with SP=2 as an example, each SP process handles a split of 16K tokens. For samples longer than 16K, additional communication is required anyway. For samples shorter than 16K, we can group them into 16K packing groups, which is equivalent to training with 16K packing groups using SP=1, thereby reducing communication.

For l best, the smallest packing group l 1 is derived based on its optimal degree of SP s best.sp, ensuring no communication overhead during sequential parallelism. Similarly, the smallest packing group l 2 for l max is also considered. If l 2 is smaller than l best, it will be merged into the l best range. Finally, the hierarchical packing groups L p = { l 1 , l best , l 2 , l max } is obtained. More implementation details are shown in Appendix I.

4.2 Balance Packing

After obtaining the optimal hierarchical pack groups, we distribute the dataset samples into different groups while ensuring that the metrics outlined in Section 3.1 (DBR, PR, ABR, and CR) are well-optimized, which conventional packing struggles with. We first divide the entire dataset into sub-datasets [ D 1 , D 2 , . . . , D n ] based on L p . For each D i , the following steps are executed:

Packing : We pack the data in D i to length l i . This ensures low PR and DBR. Note that arbitrary packing methods are feasible.

GreedyFill : We use the remaining unpacked data for a greedy fill, as larger groups struggle to fill the packed data using only their samples. A simple example illustration is shown in Appendix E.

Balance Batching : Constructing balance batches B i based on attention complexity using heuristic solution sorting support maintaining balanced attention computation across different packing groups. Heuristic solutions are an efficient, approximate solution for large-scale datasets. More evidences are shown in Appendix D.

Loss

1.4

1.2

1.0

0.8

0.6

0

20

40

60

80

w/o CL

with CL

100

Step

0

200

400

600

800

1000

Step

0.5e+5

1.0e+5

1.5e+5

2.0e+5

128k

32k

4k

0

200

400

600

800

1000

Step

1

2

3

4

5

Average Token Loss

128k

32k

4k

The procedure of balanced packing is illustrated in Algorithm 2. Since our approach inherently involves multiple levels, i.e., hierarchical packing groups, it automatically separates short- and longcontext data, avoiding wasted communication overhead and imbalanced attention computation and reducing CR and ABR significantly. We also achieve low PR and DBR at multiple levels, thanks to GreedyFill. More details about the implementation are shown in Appendix J.

4.3 Dynamic Training Pipeline

Since HBP involves multi-level inputs, it is essential to design a dynamic training pipeline, enabling hot switching of different packing groups. Adaptive SP (pre-initialized multi-SP achieving zero overhead) is proposed to ensure efficient training with multi-level packing. Curriculum Learning and a Stable Loss Normalizer are proposed to improve the performance in hybrid training.

Adaptive Sequential Parallel : Each packing group is assigned an optimal training configuration s = ( sp , ckpt ) , and we adopt an alternating scheme that selects the best SP degree and GC setting per packing group, as illustrated in Figure 2.

Curriculum Learning Strategy : Training on long-context tasks presents challenges because initiating training without instructional capabilities can result in significant fluctuations in the training loss, as illustrated in Figure 4. Thanks to our inherent hierarchical structure, it is straightforward to adopt a curriculum learning strategy that begins with general short-context data in the early stages of training. As training progresses, we transition to a hybrid approach that alternates training both shortand long-context data, as illustrated in Figure 2.

Stable Loss Normalizer : The training stability introduced by data packing is an important problem, as it impacts the data distribution. Previous work [6] has analyzed loss calculation, identifying two primary loss normalizers L token (Token-Mean) and L sample (Sample-Mean):

where B l represents the local batch size within the DP group, and T i denotes the number of loss tokens in the batch i . They argued that instability lies in the inconsistent loss of normalization. [27] also proposes sum loss without normalization to mitigate the effect of norm discrepancies. However, the sum loss introduces a trade-off: as the sequence length increases, the gradient values escalate disproportionately ( 1e+5 ), as shown in the left part of Figure 5. To address the above issues, we empirically observe T ave (Average Token) of the global batch size B g across all DP groups can serve as a stable loss normalizer . The stable loss normalizer is derived from theoretical considerations, with the objective of guaranteeing that each token contributes equally to the aggregate loss. This formulation mitigates biases introduced by heterogeneous sequence lengths, varying data-parallel group sizes, and gradient accumulation strategies. We illustrate the effect of different normalization strategies under a simple setting: a data-parallel (DP) group of size 2 , where each rank processes a local batch of 2 samples.

Token-Mean : In the token-mean approach, the loss for each DP rank is normalized by its local token:

The global loss is then averaged across ranks:

This formulation normalizes losses rank-wise without accounting for global token distribution, potentially introducing bias when sequence lengths vary across ranks.

Sample-Mean : In the sample-mean method, losses are first normalized per sample and then averaged:

The global loss becomes:

This strategy balances individual samples but fails to reflect the total token count, giving undue weight to shorter sequences.

Stable Loss Normalizer : To eliminate such bias, we normalize by the global average token length:

Each DP's loss is then normalized by T ave, And the final loss is:

This normalization ensures that the contribution of each token is equal, regardless of which sample or rank it belongs to, thereby producing an unbiased global loss.

5 Experiments

5.1 Experimental Setup

Implementation Details . We use large-scale datasets: Tulu3 (32K)[14] for general tasks and LongCite (128K)[15] for long-context tasks. Both have shown strong performance across various benchmarks. They also enhance the model's ability to handle input lengths from 0.1K to 128K tokens. We conduct experiments with the following models: LLaMA 3.1 [1], Qwen-2.5 [2], and DeepSeek-V2 (236B). Most models are trained on 32x H100 80GB GPUs using the DeepSpeed [28], while DeepSeek-V2 (236B) is trained with the Megatron-LM [29] with 256x H100 80G GPUs. In our experiments, we use DeepSpeed-Ulysses's [10] sequence parallelism approach, and the ring-attention [11] method is also applicable. We conducted ablation experiments using the LLaMA3.1-8B model. For the Longsite dataset, approximately 2k samples are uniformly sampled. The loss normalizer for baselines without special instructions is Token-Mean, while HBP uses Ave-Token. GPU days are the evaluation metric to estimate the total training time. We use a learning rate of 1e-5, weight decay of 0.01, and adopt AdamW as our optimizer.

Evaluation. We comprehensively evaluated the LLM's performance using OpenCompass [30]. For general tasks, several benchmark datasets were assessed, including MMLU [31], MMLU PRO [32], CMMLU[33], BBH [34], Math [35], GPQA Diamond [36], GSM8K [37], HellaSwag [38], MathBench [39], HumanEval [40], MBPP [41], IFEval [42], and Drop [43]. For long-context tasks, the evaluation including Ruler [44], NeedleBench [45], LongBench [46], and Longcite.

5.2 Main Results

In Table 5, we compare our method HBP with LongAlign [6] and packing method ISF [8]. We also notice that LongAlign improves the average general tasks to some extent; it sacrifices performance in long tasks e.g. , Ruler-128K. In contrast, our method maintains strong performance both in general short tasks and long tasks. The improvements are consistent across a wide range of model sizes, from 8B to 236B parameters. Notably, for the largest MoE model, DeepSeek-V2 (236B), our method achieves an impressive 2.4 × training speed-up, reducing training time from 57.1 to 23.8 GPU days. Full results are shown in Appendix C.

5.3 Ablation Results

Importance of Hybrid Training . Table 9 shows that both short- and long-context data are essential for maintaining general and long-text capabilities. "Longcite (8K)" includes only sequences up to 8K. All settings use naive packing. Removing long-context data (row 1) harms long-text performance, while removing short-context data (row 2) weakens the general ability. Row 3, with partial short data, confirms these effects.

Components of HBP . In Table 6, we show that the Attention Balance Ratio (ABR) and Communication Ratio (CR) can be reduced significantly with hierarchical packing. In particular, ABR drops from 0.506 to 0.288, and the CR drops from 1.0 to 0.173. By balancing the batching data with a similar complexity of attention computation, we have much more balanced mini-batches with low ABR, from 0.288 to 0.002. Overall, we achieve a 1.4 × speedup. Similar results can be observed in the larger LLaMA-3.1-70B model.

Curriculum Learning . Table 7 presents the impact of curriculum learning on HBP. Starting with short tasks and gradually mixing in long-context tasks benefits training and convergence. We applied a similar curriculum strategy to the naive ISF baseline using advanced sampling, which also showed improvements. This confirms the general effectiveness of curriculum learning for long-context SFT. Notably, HBP naturally separates short and long contexts, making curriculum learning easier to apply.

Iterations for Curriculum Learning training . Table 8 presents detailed ablation studies across different models. As shown, even a simple curriculum learning setup, adding 100 steps of early-stage

training with short samples, already yields noticeable improvements. To ensure training stability, we ultimately adopt a setting with 500 steps of short-sample-only training in our final experiments.

Stable Loss Normalizer . We compared several loss normalization methods by training models using different normalizers while keeping all other training configurations consistent. As shown in Table 10, the Ave-Token loss normalizer achieved the highest performance in both general tasks, Average (AVE) 57.6, and long context tasks LongBench 43.1, Ruler-128K 70.8, and LongSite 71.2.

Importance of Hierarchical Groups Auto-Selection . Table 11 compares manual (row 2) and automatic (row 3) packing groups. Automatic selection of groups (based on Appendix F) achieves greater speedup, reducing training time from 4.25 to 3.73 GPU days.

Compared with FlexSP . As shown in Table 12, HBP achieves lower ABR, DBR, and PR than FlexSP, while maintaining a similar CR, contributing to faster training (4.35 to 3.73 GPU days), even considering only the model computation time. Furthermore, FlexSP introduces a per-batch grouping overhead (5-15 seconds), which further slows down training (see Appendix B). HBP performs global data-level operations in advance with negligible data overhead.

Negligible Overhead . The overhead of HBP is negligible: Auto-Selection takes ∼ 3 minutes and Packing 5 seconds, together contributing less than 2% of training time (see Appendix G).

Dataset Generalization . To verify the generalization of our method, we conducted experiments on different datasets (OpenHermes[16], LongWriter[23]). Our method also achieves effective results on these datasets. Results are shown in the Appendix H.

6 Conclusion and Limitations

In this paper, we proposed Hierarchical Balance Packing (HBP), a novel strategy to address workload imbalances in long-context LLM training through multi-level data packing and a dynamic training pipeline. Due to limitations in computational resources, we have not conducted experiments on longer contexts, such as 256K or 512K. Additionally, we have not validated HBP on other post-training tasks, such as RLHF or DPO. We leave these explorations to future work.

References

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , 'The llama 3 herd of models,' arXiv preprint arXiv:2407.21783 , 2024.

A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al. , 'Qwen2. 5 technical report,' arXiv preprint arXiv:2412.15115 , 2024.

Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, X. Dong, H. Duan, Q. Fan, Z. Fei, Y. Gao, J. Ge, C. Gu, Y. Gu, T. Gui, A. Guo, Q. Guo, C. He, Y. Hu, T. Huang, T. Jiang, P. Jiao, Z. Jin, Z. Lei, J. Li, J. Li, L. Li, S. Li, W. Li, Y . Li, H. Liu, J. Liu, J. Hong, K. Liu, K. Liu, X. Liu, C. Lv, H. Lv, K. Lv, L. Ma, R. Ma, Z. Ma, W. Ning, L. Ouyang, J. Qiu, Y. Qu, F. Shang, Y. Shao, D. Song, Z. Song, Z. Sui, P. Sun, Y. Sun, H. Tang, B. Wang, G. Wang, J. Wang, J. Wang, R. Wang, Y. Wang, Z. Wang, X. Wei, Q. Weng, F. Wu, Y. Xiong, C. Xu, R. Xu, H. Yan, Y. Yan, X. Yang, H. Ye, H. Ying, J. Yu, J. Yu, Y . Zang, C. Zhang, L. Zhang, P. Zhang, P. Zhang, R. Zhang, S. Zhang, S. Zhang, W. Zhang, W. Zhang, X. Zhang, X. Zhang, H. Zhao, Q. Zhao, X. Zhao, F. Zhou, Z. Zhou, J. Zhuo, Y. Zou, X. Qiu, Y. Qiao, and D. Lin, 'Internlm2 technical report,' 2024.

S. Chen, S. Wong, L. Chen, and Y. Tian, 'Extending context window of large language models via positional interpolation,' arXiv preprint arXiv:2306.15595 , 2023.

B. Peng, J. Quesnelle, H. Fan, and E. Shippole, 'Yarn: Efficient context window extension of large language models,' arXiv preprint arXiv:2309.00071 , 2023.

Y. Bai, X. Lv, J. Zhang, Y. He, J. Qi, L. Hou, J. Tang, Y. Dong, and J. Li, 'LongAlign: A recipe for long context alignment of large language models,' in Findings of the Association for Computational Linguistics: EMNLP 2024 . Miami, Florida, USA: Association for Computational Linguistics, Nov. 2024, pp. 1376-1395. [Online]. Available: https://aclanthology.org/2024.findings-emnlp.74

L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , 'Training language models to follow instructions with human feedback,' Advances in neural information processing systems , vol. 35, pp. 27 730-27 744, 2022.

Y. Yao, J. Tan, J. Hu, F. Zhang, X. Jin, B. Li, R. Gong, and P. Liu, 'Omnibal: Towards fast instruct-tuning for vision-language models via omniverse computation balance,' arXiv preprint arXiv:2407.20761 , 2024.

S. Wang, G. Wang, Y. Wang, J. Li, E. Hovy, and C. Guo, 'Packing analysis: Packing is more appropriate for large models or datasets in supervised fine-tuning,' arXiv preprint arXiv:2410.08081 , 2024.

S. A. Jacobs, M. Tanaka, C. Zhang, M. Zhang, S. L. Song, S. Rajbhandari, and Y. He, 'Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models,' arXiv preprint arXiv:2309.14509 , 2023.

H. Liu and P. Abbeel, 'Blockwise parallel transformer for large context models,' Advances in neural information processing systems , 2023.

DeepSeek-AI, 'Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model,' 2024.

M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su, 'Scaling distributed machine learning with the parameter server,' in 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14) . Broomfield, CO: USENIX Association, Oct. 2014, pp. 583-598. [Online]. Available: https://www.usenix.org/conference/osdi14/technical-sessions/presentation/li_mu

N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, Y. Gu, S. Malik, V. Graf, J. D. Hwang, J. Yang, R. L. Bras, O. Tafjord, C. Wilhelm, L. Soldaini, N. A. Smith, Y. Wang, P. Dasigi, and H. Hajishirzi, 'Tülu 3: Pushing frontiers in open language model post-training,' 2024. [Online]. Available: https://arxiv.org/abs/2411.15124

J. Zhang, Y. Bai, X. Lv, W. Gu, D. Liu, M. Zou, S. Cao, L. Hou, Y. Dong, L. Feng, and J. Li, 'Longcite: Enabling llms to generate fine-grained citations in long-context qa,' arXiv preprint arXiv:2409.02897 , 2024.

Teknium, 'Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants,' 2023. [Online]. Available: https://huggingface.co/datasets/teknium/OpenHermes-2.5

H. Jiang, Q. Wu, X. Luo, D. Li, C.-Y. Lin, Y. Yang, and L. Qiu, 'Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression,' arXiv preprint arXiv:2310.06839 , 2023.

Y. Sun, L. Dong, B. Patra, S. Ma, S. Huang, A. Benhaim, V. Chaudhary, X. Song, and F. Wei, 'A length-extrapolatable transformer,' in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 14 590-14 604. [Online]. Available: https://aclanthology.org/2023.acl-long.816/

C. Han, Q. Wang, H. Peng, W. Xiong, Y. Chen, H. Ji, and S. Wang, 'LM-infinite: Zero-shot extreme length generalization for large language models,' in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , K. Duh, H. Gomez, and S. Bethard, Eds. Mexico City, Mexico: Association for Computational Linguistics, Jun. 2024, pp. 3991-4008. [Online]. Available: https://aclanthology.org/2024.naacl-long.222/

C. Packer, S. Wooders, K. Lin, V. Fang, S. G. Patil, I. Stoica, and J. E. Gonzalez, 'MemGPT: Towards llms as operating systems,' arXiv preprint arXiv:2310.08560 , 2023.

W. Xiong, J. Liu, I. Molybog, H. Zhang, P. Bhargava, R. Hou, L. Martin, R. Rungta, K. A. Sankararaman, B. Oguz et al. , 'Effective long-context scaling of foundation models,' arXiv preprint arXiv:2309.16039 , 2023.

Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia, 'Longlora: Efficient fine-tuning of long-context large language models,' arXiv preprint arXiv:2309.12307 , 2023.

Y. Bai, J. Zhang, X. Lv, L. Zheng, S. Zhu, L. Hou, Y. Dong, J. Tang, and J. Li, 'Longwriter: Unleashing 10,000+ word generation from long context llms,' arXiv preprint arXiv:2408.07055 , 2024.

Y. Wang, S. Wang, S. Zhu, F. Fu, X. Liu, X. Xiao, H. Li, J. Li, F. Wu, and B. Cui, 'Data-centric and heterogeneity-adaptive sequence parallelism for efficient llm training,' arXiv preprint arXiv:2412.01523 , 2024.

X. Contributors, 'Xtuner: A toolkit for efficiently fine-tuning llm,' 2023.

M. M. Krell, M. Kosec, S. P. Perez, and A. Fitzgibbon, 'Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance,' arXiv preprint arXiv:2107.02027 , 2021.

N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu et al. , 'T \ " ulu 3: Pushing frontiers in open language model post-training,' arXiv preprint arXiv:2411.15124 , 2024.

S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He, 'Zero: Memory optimizations toward training trillion parameter models,' in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis . IEEE, 2020, pp. 1-16.

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro, 'Megatron-lm: Training multi-billion parameter language models using model parallelism,' arXiv preprint arXiv:1909.08053 , 2019.

O. Contributors, 'Opencompass: A universal evaluation platform for foundation models,' 2023.

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, 'Measuring massive multitask language understanding,' Proceedings of the International Conference on Learning Representations (ICLR) , 2021.

Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang et al. , 'Mmlu-pro: A more robust and challenging multi-task language understanding benchmark,' arXiv preprint arXiv:2406.01574 , 2024.

H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin, 'Cmmlu: Measuring massive multitask language understanding in chinese,' 2023.

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, , and J. Wei, 'Challenging big-bench tasks and whether chain-of-thought can solve them,' arXiv preprint arXiv:2210.09261 , 2022.

H. Saxton, Grefenstette and Kohli, 'Analysing mathematical reasoning abilities of neural models,' arXiv:1904.01557 , 2019.

D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, 'GPQA: A graduate-level google-proof q&a benchmark,' in First Conference on Language Modeling , 2024. [Online]. Available: https://openreview.net/forum?id=Ti67584b98

K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, 'Training verifiers to solve math word problems,' arXiv preprint arXiv:2110.14168 , 2021.

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, 'Hellaswag: Can a machine really finish your sentence?' in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2019.

H. Liu, Z. Zheng, Y. Qiao, H. Duan, Z. Fei, F. Zhou, W. Zhang, S. Zhang, D. Lin, and K. Chen, 'Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark,' 2024.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, 'Evaluating large language models trained on code,' 2021.

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , 'Program synthesis with large language models,' arXiv preprint arXiv:2108.07732 , 2021.

J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou, 'Instruction-following evaluation for large language models,' 2023. [Online]. Available: https://arxiv.org/abs/2311.07911

D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, 'DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs,' in Proc. of NAACL , 2019.

C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y. Zhang, and B. Ginsburg, 'Ruler: What's the real context size of your long-context language models?' arXiv preprint arXiv:2404.06654 , 2024.

M. Li, S. Zhang, Y. Liu, and K. Chen, 'Needlebench: Can llms do retrieval and reasoning in 1 million context window?' 2024. [Online]. Available: https://arxiv.org/abs/2407.11963

Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, Y. Dong, J. Tang, and J. Li, 'Longbench: A bilingual, multitask benchmark for long context understanding,' 2023.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the contributions and scope of the paper?

Answer: [Yes]

Justification: The main claims in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper provides a brief discussion of the limitations in the Conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper is primarily based on empirical observations and proposes heuristic approaches, which are not fully supported by formal mathematical proofs or a complete set of theoretical assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental results can be fully reproduced based on the information provided in the paper, and the authors have indicated that the code will be released in the future.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The code and data are not yet publicly available, and although the authors plan to release them later, they are currently not provided with sufficient instructions in the supplemental material to ensure faithful reproduction of the results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full details are provided in the main text and supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information about the computational resources used, including details in both the main text and the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research presented in the paper fully conforms to the NeurIPS Code of Ethics in all respects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All resources used in the paper comply with their respective licenses and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Packing Strategy Results

Table 13 illustrates the complexity of different packing strategies and their corresponding final training times. It also shows that the training indicators Data Balance Ratio (DBR), Padding Ratio (PR), and Attention Balance Ratio (ABR) are strongly correlated with the training time, emphasizing the effectiveness of these indicators. Based on the computational complexity of packing strategies and their training time, and accuracy performance, we ultimately selected ISF as our naive packing baseline.

B Results of FlexSP data grouping

The experiments were conducted with a sequence length of 128K using 32 GPUs. The results for FlexSP were obtained by directly testing with the officially released code. The FlexSP data group method incurs a non-negligible overhead of 5-15 seconds per batch, which cannot always be hidden by the training time shown in Table 14 Rows 3 and 4.

C Full Results

Tables 15 and 16 present our complete results for general tasks and long-context tasks, respectively. These results collectively validate the effectiveness of our HBP method. The other dense models all use the GQA architecture, while Deepseek-V2 adopts the MLA structure, which involves more communication due to a larger number of K and V heads, similar to MHA. HBP effectively reduces the communication overhead, making the speedup more significant.

D More Details of Attention Imbalance Problem

Imbalance of attention complexity: As deduced from Megatron-LM paper, gives a packed sequence containing n sub-sequences with total lengths, each transformer layer computes cost can be approximated as: 24 Bsh 2 +4 B ∑ s 2 i , where s i is the length of the i-th sub-sequence, B is the batch size, and h is the hidden size. Therefore, variance in attention complexity ∑ s 2 i directly impacts compute cost.

Effectiveness of Balance Batching:

Sorting is indeed a heuristic solution, and it cannot be strictly proven mathematically to guarantee perfect balance. However, based on the results across different datasets, it achieves near-perfect balance (low ABR approaching 0).

Heuristic solutions are a common and efficient approximate solution for solving problems on largescale datasets. Our method ensures that iterations maintain attention to balance across different datasets over 99.6% of training shown in Table 17, demonstrating the strong generalization capability of our method.

Balance Evidence: The left side in Figure 6 shows baseline packing with high attention cost variance across DP groups, while the right side shows our sorting-based method, which significantly reduces this variance and improves efficiency.

0

20

40

60

80

100

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Time (s)

Forward Time (w/o sort)

dp1

dp2

dp3

dp4

0

20

40

60

80

100

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

Forward Time (with sort)

dp1

dp2

dp3

dp4

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Seq Length Square Sum

1e10

Attention Complexity (w/o sort)

dp1

dp2

dp3

dp4

0

20

40

60

80

100

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1e10

Attention Complexity (with sort)

dp1

dp2

dp3

dp4

Step

Step

Long

Long

Long

Long

Long

Long

Long

Long

Medium

Short

Short

Long

Long

Long

Long

Long

Long

Long

Long

Pad

Pad

Pad

128K Packing Group

With Padding

128K Packing Group

With GreedyFill

Low

AVE-T

High

AVE-T

E More Details of GreedyFill.

GreedyFill aims to reduce intra-pack padding tokens and improve Ave-T. It operates within local packing groups by selecting a small number of sequences from other shorter packing groups to efficiently fill residual space, as shown in Figure 7.

F More Details of Hierarchical Groups Auto-Selection.

The necessity of auto-selection: Table 18 presents an example of 32K token-length training, showcasing various SP degrees and GC configurations. The second row highlights the minimal configuration s = (2 , 28) that satisfies memory constraints. While this configuration adheres to

memory requirements, it fails to achieve optimal performance. In contrast, the fourth row illustrates a more balanced and effective configuration with s = (8 , 8) , achieving the fastest speed. These experiments demonstrate that, given a specific packing length, there are significant performance differences among various SP degrees and GC configurations.

The final selection groups result: Table 19 presents the training speed of the optimal SP degrees and GC configurations for various sequence lengths L . The second row of the table shows that our optimal configuration is groups = 16K, with the corresponding training strategy being s = (1 , 28) . This indicates that the optimal training strategy is distinct for different packing groups. The evidence above emphasizes the importance of searching for the best groups and their corresponding training strategies, i.e., Auto-Group Selection.

G Overhead Analysis

Balance Packing and Auto-Selection overhead relative to training time. Both steps together remain consistently negligible: Auto-Selection requires at most 15 minutes and Packing only 5 seconds, while training spans from 168 to 1400 minutes. Overall, the combined overhead is below 2% of training time, confirming that preprocessing costs are insignificant compared to model training.

G.1 Overhead of Profiling

Memory Profiling. The memory profiling cost depends on the sequence length set ( Seq ) processed by the device:

Time Profiling. The total number of search strategies ( S ) is determined by the packing length set ( L ) and the Sequence Parallel ( SP ) settings. profile_iter can be adjusted based on actual iteration times, with typical values ranging from 3 to 10 :

Total Cost (Example). For instance, with profile_iter = 5 , L = { 16 K , 32 K , 128 K } and SP = { 2 , 4 , 8 } , Seq = { 4 K , 8 K , 16 K } , for LLaMA3.1-8B:

Cost t = 3 × 3 × 5 = 45 × iteration_time ,

Costtotal = Cost t + Cost m = 60 × iteration_time .

Considering the 3000+ training iterations, totaling 60 iterations (2%) is negligible compared to the overall training time.

G.2 Overhead of Balance Packing

For our training dataset of 1 million samples, the overhead introduced by Balance Packing is consistently within 5 seconds.

Complexity. The computational complexity of Balance Packing can be expressed as:

where N is the number of samples, M is the number of packing groups, C is the number of ISF iterations, and L denotes the selected packing set.

Since C ≪ N , M ≪ N , and L ≪ N , the overall complexity reduces to a sublinear class relative to N . Hence, scaling to substantially larger datasets incurs only minimal additional overhead.

G.3 Scalability of the Proposed Method

Large-scale dataset. Our profiling is performed for only a small number of iterations ( < 100 ). As the dataset size increases, the overall overhead ratio ( < 1% ) becomes even lower. Meanwhile, the complexity of Balance Packing is O ( N ) , so the additional cost introduced is negligible as the dataset scales.

H Dataset Generalization

H.1 OpenHermes

We also provide some of our experimental results on OpenHermes. For example, Table 21 shows that the results are consistent with Tulu3 under different packing strategies. The label 22 shows that our HBP is also effective in a 128K training setting.

H.2 LongWriter

Figure 8 and Table 23 present the results of our HBP hybrid training on Tulu3 and LongWriter. The x-axis represents the user instruction required length, and the y-axis represents the model output length based on the Long-Writer paper. A stronger linear correlation indicates better instructionfollowing capability of the model. The left side of the Figure shows the original baseline results without using the Long-Writer dataset. The middle shows the results of naive packing, and the right shows the results of HBP. The experiment demonstrates that our method is equally effective, achieving consistent acceleration across both models.

104

104

104

Output Length

Output Length

Output Length

103

103

103

102

102

102

102

103

104

102

103

104

102

103

104

RequiredLength

RequiredLength

RequiredLength

I Implementation Details of Hierarchical Groups Auto-Selection

I.1 FindBestSpCkpt

Algorithm 3 FindBestSpCkpt Function

1: Initialize P ← [] , O ← [] 2: for each sp ∈ SP do 3: ckpt ← GreedyProfileCkpt ( l ) 4: P. add ( ProfileTime ( l, sp, ckpt )) , O. add ( sp, ckpt ) 5: end for 6: j ← argmin ( P ) 7: return O [ j ]

Algorithm 4 GreedyProfileCkpt

1: Inputs: l, sp, c min , c

max 2: s 1 ← ( sp, l, c min ) 3: s 2 ← ( sp, l, c max ) 4: m r 1 ← ProfileMemory ( s 1 ) 5: m r 2 ← ProfileMemory ( s 2 ) 6: m ave ← ( m r 2 -m r 1 ) / ( c max -c min ) 7: c o ← c max -m r 2 /m ave

Algorithm 3 determines the optimal gradient checkpointing strategy by evaluating all possible sp strategies.

Initialization : Start with empty lists P and O for profiling times and configurations, respectively.

Iterate Over Strategies : For each strategy sp ∈ SP :

-Compute the best gradient checkpointing configuration ( ckpt ) using the GreedyProfileCkpt function.

-Use ProfileTime to profile the execution time for the model with the given configuration and append it to P .

Find Optimal Strategy : Identify the index j of the minimum profiling time in P using argmin ( P ) .

Return Best Configuration : Return the SP degree and corresponding gradient checkpointing configuration O [ j ] .

I.2 GreedyProfileCkpt

The algorithm 4 estimates the number of gradient checkpoint layers required for a given strategy l and sp . Here, we provide a more detailed explanation. We use the "pynvml" library to monitor GPU memory usage for memory analysis.

Memory Profiling:

Step 1: Given an input length L, enable Gradient Checkpointing for all layers (32 for 7B), and record the remaining GPU memory as m1.

Step 2: Given the same input length L, enable Gradient Checkpointing for only a subset of layers (e.g., 20 layers), and record the remaining GPU memory as m2.

Step 3: From the above, we can estimate the average memory saved per layer with Gradient Checkpointing: ave_m = (m1 - m2) / (32 - 20)

By running only 10 iterations, we can obtain a reliable estimate of the memory saved per layer when using Gradient Checkpointing.

Memory And Time Consumption Report:

For a given input length L, we set different sequence parallelism (SP) configurations. Based on the remaining GPU memory and the previously estimated ave_m, we determine the number of layers to apply Gradient Checkpointing. Then, we run 10 iterations to record the corresponding memory usage and iteration time.

Full workflow:

Initialization : Obtain the configurations using the minimum and maximum number of gradient checkpointing layers (based on empirical observations):

Memory Profiling : Profile the remaining memory for s 1 ( m r 1 ) and s 2 ( m r 2 ).

Memory Slope Calculation : Compute the average memory slope (ave m ) as:

Checkpointing Layer Estimation : Estimate the required number of checkpoints ( c o ) as:

J Implementation Details of Balance Packing

J.1 GroupData

Given a data set D and predefined hierarchical lengths L p , we evaluate the length of each data set x to determine the interval ( l i -1 , l i ) to which it belongs, assigning it to the corresponding group D i . The detailed procedure is outlined in Algorithm 5.

J.2 GreedyFill

For a given packing group G i with the corresponding length l i , we iterate through the smaller dataset partitions ( D i -1 , D i -2 , . . . , D 1 ) and greedily fill the group g within the current G i . The detailed procedure is illustrated in Algorithm 6.

J.3 Attention Balance Sort Function

First, we calculate the attention complexity for all data within the given packing group G . Then, we sort the elements in G i based on their attention complexity and construct mini-batches according to the global token number requirements as shown in Algorithm 7.

Algorithm 5 GroupData Function

Algorithm 7 Attention Balance Sort Function

2

Algorithm 6 GreedyFill Function

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of workload imbalance when fine-tuning large language models on mixed short- and long-context data. The authors propose Hierarchical Balance Packing (HBP), which creates multiple packing-length groups, assigns each example to its optimal configuration group via \"balanced packing\", and employs a dynamic training pipeline (adaptive sequence-parallelism, curriculum learning, and a stable loss normalizer). Experiments on models from 8B to 236 B parameters (e.g., LLaMA3.1-8B, Qwen2.5-32B, DeepSeek-V2) and datasets (Tulu3, LongCite, OpenHermes, etc.) show that HBP significantly improves attention balance ratio and communication ratio, yielding up to 2.4$\\times$ speedup with negligible or positive impact on downstream task performance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. **Well motivated and Important Problem**: Data packing plays an important role in improving both the efficiency and effectiveness of LLM training. This paper presents a practical method to tackle that challenge.\n\n2. **Novelty in Multi-Level Packing** Unlike previous single-length packing, HBP's two-stage group auto-selection (Alg. 1) and balanced packing (Alg. 2) jointly optimize packing lengths, sequence-parallel degree, and gradient-checkpoint configuration. This approach effectively takes into account Attention Balance Ratio and Communication Ratio, significantly improving both of them.\n\n3. **Comprehensive Empirical Validation.** Experiments span small to very large models and standard general-task (MMLU, BBH, HumanEval, etc.) and long-context benchmarks (Ruler, LongBench, LongCite). HBP consistently yields 1.3–2.4× speedups while maintaining or improving average task accuracy (Table 5, 6).\n\n### Weakness\n1. **Ablation Study**: There is an inherent trade-off among Padding Ratio, Attention Balance Ratio, Communication Ratio, and Average Tokens. I strongly encourage a systematic, pairwise analysis of these metrics—ideally visualized as a Pareto front—to clarify how different packing configurations balance efficiency and performance across these objectives.\n\n2. **Clarity**: Figures 2 and 3 would benefit from more descriptive captions that walk the reader through each step of the main algorithm. Additionally, the manuscript uses numerous abbreviations (e.g., ABR, CR, DBR, PR, SP for sequence parallelism, GC for gradient checkpointing), which should be better illustratd in the context, especially for captions of figures and tables.\n\nQUESTIONS:\n* The term \"hierarchical balancing\" is somewhat misleading. While the algorithm does employ multiple packing groups, it doesn’t follow a clear hierarchical (tree-like) structure. Typically, \"hierarchical\" implies successive merging or nesting—e.g., combining shorter-length groups into longer ones. It would help if the authors clarified why they chose this terminology and described any implicit hierarchy in their grouping process.\n\n* Introducing curriculum learning and adaptive sequence parallelism departs from the usual random-sampling assumptions in stochastic optimization, which could, in extreme cases, impair convergence. Are there practical downsides—such as degraded performance on certain tasks, increased risk of loss spikes, or sensitivity to hyperparameters—that practitioners should be aware of?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Hierarchical Balance Packing (HBP), a method to enhance the efficiency of supervised fine-tuning for long-context large language models. HBP mitigates workload imbalances inherent in hybrid long/short-context data training through a novel approach that combines multi-level packing groups, strategic sample allocation, and a dynamic training pipeline. Experiments show that HBP significantly accelerates training speed across various models and datasets without compromising performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. **The paper is well motivated**. The paper clearly analyzes the limitations caused by naive packing strategies in hybrid long/short context training, and proposed Hierarchical Balance Packing as an effective solution.\n2. **The experiments are sufficient**. The paper validates HBP's effectiveness across multiple models of varying scales and several datasets. The ablation studies clearly demonstrate the contribution of each component of HBP to performance and efficiency, which supports the paper's core claims. \n3. The paper is well-written and easy to follow.\n\n**Weakness**\n1. **The hierarchical groups auto-selection is inefficient**. It requires an additional profiling stage to determine the optimal configuration, yet the manuscript lacks a thorough discussion of the associated computational overhead (e.g., time cost) and the robustness of this process across different hardware environments. \n2. The empirical findings would be strengthened by more rigorous theoretical analysis. Specifically, while the Stable Loss Normalizer proposed in Section 4.3 shows promising results, the paper does not provide a clear theoretical justification for why the Average Token strategy effectively overcomes the limitations of existing normalization methods.\n\nQUESTIONS:\n1. In Section 4.1, Algorithm 1 leverages a profile-based approach for hierarchical group selection. However, the paper does not clarify how the profiling was conducted, such as the number of iterations or evaluation settings. Could the authors provide details on the profiling procedure, and quantify the computational overhead of this process?\n2. The GreedyFill step uses shorter samples to fill the packs of longer samples. Does it reintroduce the communication overhead and attention imbalance problems that HBP aims to solve? \n3. Table 9 shows the importance of both short- and long-context data. Does the mixture ratio of short- and long-context data in the training set affect the performance of HBP?\n\nLIMITATIONS:\nPlease see the comments in Weakness and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn summary, the authors propose an efficient hierarchical packing algorithm for long-context SFT for LLM. The traditional packing algorithm only has one packing group and faces two problems: 1) imbalanced attention complexity, 2) Inefficient Sequence-Parallelism (SP) communications. To tackle this challenge, the proposed method divides the training dataset into multiple packing groups with different sequence lengths to save the SP communications. Within each group, Balanced Packing is performed to maintain balanced attention computation among different batches. To support this multi-length input packing groups, a curriculum learning strategy is adopted. The proposed training strategy is tested on multiple models on an H100 cluster and is shown to improve the training efficiency with improved model performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This paper studies an important problem in large-scale training and is well-motivated\n2. The proposed solution achieves faster training speed on multiple models with improved model performance\n\nWeakness:\n1. The proposed method is purely heuristic and lacks in-depth analysis \n2. Some descriptions in the paper are not clear and confusing (Please refer to Questions)\n3. The proposed packing method seems to be much more complicated than the traditional packing, but such overhead is not measured in the paper\n4. The proposed training framework has to come with a curriculum learning strategy due to its multi-level nature. It is unclear whether this learning strategy could lead to worse accuracy for certain models/datasets\n5. Seems the proposed method is only validated on an H100 cluster. However, to show the generalizability of the proposed method, it would be better to test it on multiple cluster setups where the hardware resource is different (e.g., A100 clusters, different CPUs)\n6. The proposed method is limited only to SFT and does not apply to other tasks such as PPO/GRPO.\n\nQUESTIONS:\n1. Table 2 seems important to motivate the proposed method but it lacks details. How are the numbers calculated/measured? Is it a general phenomenon? \n\n2. The Curriculum Learning Strategy seems important for the proposed training strategy but it is not introduced in detail. For example, the proposed method is applied in two different clusters, which results in a different number of packing groups with different sequence lengths. How would the curriculum learning strategy vary for these two cases? \n\n3. It would be great to know whether the proposed method is compatible with various parallelism strategies such as DP/PP/TP/FSDP.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new data packaging method with automatic training configuration optimization (wrt gradient checkpointing and sequence parallelism). \nThe author designed several useful metrics to effectively measure the training efficiency in terms of waste of padding, waste of communication, balance of attention FLPOs and communications, etc.. These metrics could serve as helpful optimization objectives for future works as well.\nThe author designed automatic optimization method to decide the optimal packing length, assign data to these groups, and design the optimal training configurations for these groups. \nExperiments demonstrate that the proposed method could speed up training time to 1.33-2.4x across different model sizes, while maintaining high performance on both short- and long-context tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The set of metrics and optimization objectives is general and useful for future works to design better algorithms.\n2. The proposed method improves the training efficiency without harming the performance.\n3. The author identified an important but have been overlooked issue with existing data packaging approaches, and proposed a method that not only keeps the natural distribution of varied-length data but also minimizes the waste of compute and load imbalance in current training strategies.\n4. Extensive experiments show that the proposed method is effective and performant.\n\nWeaknesses:\n1. The writing could be further improved--the paper used a large space to repeatedly highlight the key contributions and the limitation of the current packaging methods. Limited space was left for explaining the method and showing the results. In particular, a brief hint on what the notations stand for would be helpful than just assuming everyone to repeatedly look back at the notation table. In addition, some details might be missing, e.g., there is no explanation on what is FindBestSpCkpt function in 4.1 while both pseudocode and the main text uses it. \n2. The proposed packaging method focuses only on the length distribution. However, it ignores the content distribution shift after clustering the data purely based on lengths. For example, if one domain only has short sentences, it will only appear at the early training stage, leading to catastrophic forgetting for this domain.\n3. The paper uses profile-based automatic optimization, however, it did not discuss the overhead brought by this optimization process. In addition, it applies the method to Tulu3, which is a mid-scale post-training dataset. However, sequence packaging is often used in pretraining stage, where the paper lacks discussion on the scalability of the proposed method to large-scale pretraining datasets.\n4. The search space only contains sequence parallelism and gradient checkpointing, while more dimensions, such as tensor parallelism, fsdp, etc., are not considered. In addition, sequence parallelism also has different categories, such as DeepSpeed version (using all-to-all communication and split over attention head) and DistFlashAttention version (using p2p communication and splitting over sequence length), while the method's optimization considers a very limited optimization space.\n\nQUESTIONS:\n1. Could the author provide more details about the overhead brought by the optimization process and discuss how scalable it is if someone considers applying it to pretraining-scale datasets?\n2. Do you think clustering the data by lengths would impact the training data distribution, thus leading to very different bahaviors when different domains have very different length biases? How do you mitigate this impact?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on training Long-Context Large Language Models (LLMs), with particular emphasis on data packing when mixing long-context and short-context data. The paper points out that existing data packing methods suffer from inefficiencies and imbalances. To address these issues, the paper introduces Hierarchical Balance Packing (HBP), which tackles the challenges of optimal packing group selection, data assignment, and training with packed data through three components: Hierarchical Groups Auto-Selection, Balance Packing, and Dynamic Training Pipeline. Experimental results demonstrate that HBP effectively improves training efficiency while maintaining strong performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces a variety of measuring metrics for data packing , providing valuable insights for future analyses of data packing strategies.\n2. The paper constructs a comprehensive pipeline to address different challenges in data packing. Furthermore, the automatic packing capability of HBP reduces the difficulty of parameter tuning.\n3. HBP significantly improves the training efficiency of Long-Context LLMs while maintaining strong performance across different benchmark datasets, thoroughly validating its effectiveness.\nWeaknesses:\n1. Section 3.1 introduces multiple measuring metrics and explains their relevance to LLM’s training, but the paper lacks experiments demonstrating how variations in individual metrics concretely affect LLM’s training performance and efficiency.\n2. The paper claims that the HBP algorithm introduces only negligible overhead, but lacks detailed quantitative data showing how much time is cost by the Hierarchical Groups Auto-Selection and Balance Packing components, respectively.\n\nQUESTIONS:\nIn Section 4.3, the paper proposes the Stable Loss Normalizer and empirically selects the Average Token method, but lacks an in-depth analysis of the underlying reasons. Furthermore, it remains unclear whether the stability advantages of the Average Token normalizer apply exclusively to HBP, or if they also generalize to other methods. Could the authors clarify this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}