SkyLadder: Better and Faster Pretraining via Context Window Scheduling
Authors: Tongyao Zhu
Qian Liu
Haonan Wang
Shiqi Chen
Xiangming Gu
Tianyu Pang
Min-Yen Kan
Venue: NeurIPS 2025 poster ()
Keywords: pretraining
language models
context window
Primary area: deep_learning
TL;DR: We investigate an efficient and effective context-window scheduling method for language model pretraining.

Abstract: Recent advancements in LLM pretraining have featured ever-expanding context windows to process longer sequences. However, our controlled study reveals that models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget. This finding motivates us to explore an optimal context window scheduling strategy to better balance long-context capability with pretraining efficiency. To this end, we propose SkyLadder, a simple yet effective approach that implements a short-to-long context window transition. SkyLadder preserves strong standard benchmark performance, while matching or exceeding baseline results on long-context tasks. Through extensive experiments, we pretrain 1B-parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating that SkyLadder yields consistent gains of up to 3.7% on common benchmarks, while achieving up to 22% faster training speeds compared to baselines.

SkyLadder: Better and Faster Pretraining via Context Window Scheduling
Tongyao Zhu
Qian Liu Haonan Wang's Shiqi Chen
Xiangming Gu² Tianyu Pang" Min-Yen Kan
National University of Singapore "Sea AI Lab *City University of Hong Kong
tongyao wwugyya..uuuu..o. zhutu. nus. liuqian sea@gmail........ Liugian.seadgmail com
Abstract
Recent advancements in LLM pretraining have featured ever-expanding context windows to process longer sequences. However, our controlled study reveals that models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget. This finding motivates to explore an optimal context window scheduling strategy to better balance long-context capability with pretraining efficiency. To this end, we propose Sky- Ladder, simple yet effective approach that implements short-to-long context window transition. Skyl SkyLadder preserves strong standard benchmark performance, while matching or exceeding baseline results on long-context tasks. Through ex- tensive experiments, we pretrain 1 B--parameter models (up to 32K context) and 3B-parameter models (8K context) on 100B tokens, demonstrating that SkyLadder yields consistent gains of to 3.7% on common benchmarks, while achieving up to 22% faster training speeds compared to baselines".
Introduction
The evolution of language models has been marked by a consistent expansion in context window sizes (Figure left). While early models like GPT [39] and BERT [8] were limited context windows of 512 tokens, subsequent models have pushed significantly beyond these bounds. GPT-2 [40] doubled this capacity to 1024 tokens, and with Large Language Models (LLMs) exceeding 1B parameters. this trend has continued: Llama [51] has a 2048-token window, followed by Llama-2 [52] (4096 tokens), and Llama-3 [13] (8192 tokens). The need for models to handle longer sequences during inference has fueled the rush to expand the context window. As models pretrained with longer context windows reduce document truncation and preserve coherence [9], there a widespread belief that such models should perform comparably to, even surpass, their shorter-context counterparts.
We question the common belief that larger context windows do actually improve performance. Close inspection of previous work reveals that there has yet to be a fair experimental setup for comparing models across different context windows while adhering controlled experiments, we test how changing only the context window size during pretraining impacts their performance. As shown in Figure (right), our results indicate that models pretrained using shorter contexts always outperform long-context models, when assessed by their average performance across popular benchmarks. In addition, we verify that the performance gap is not eliminated by using advanced document packing strategies [13, 9, 44].
fixed token budget. Using tightly
To ensure the model can ultimately process long sequences, the model still needs be exposed long sequences. However, given the finding that shorter context windows enhance performance on
Corresponding author.
Projecct code is https: http//ggttttuttteeeeeeeeeeeeeeet wtt.://..commaaaaegge. SkyyLadder
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Gommaz Llansd
Context Window Size
8192
Gerrma
Average Accuracy
OLMo
LinmaZ
45.0
Queen2
2048
GPT
BLOOM
QUEen
42.5
Pythia
Liamal
1k
40.0
2k
GP12
1024
37.5
8k
512
2020
2022
2024
20
40
60
100
Year of Release
Training Tokens (B)
Figure 1: Left: Pretraining context window of LLMs grows over the recent years. Right: Average performance (in %) across nine downstream tasks for IB- 1B-parameter models with different pretrained context window sizes (color-coded). Increasing the context window degrades the overall performance.
downstream tasks, we face a trade-off between long-context capability and pretraining effectiveness. We propose SkyLadder, a simple yet effective context window scheduling strategy designed to balance both objectives. SkyLadder does this by progressively expanding the size of the context window during pretraining, beginning pretraining with minimal short context window (e.g.. tokens) and progressively expanding it to the long target context window (e.g.. 32,768 tokens).
Empirical results IB-parameter models (up to 32K context window) and 3B- parameter models (up to 8K context window) on 100B tokens demonstrate that SkyLadder outperforms naive long- context pretraining baselines, in both short- and long-context evaluation tasks. For example, models trained with SkyLadder demonstrate significantly higher accuracy on standard benchmarks (e.g.. HellaSwag), and reading comprehension tasks (e.g.. HotpotQA), while still maintaining competitive performance on long-context evaluations like RULER. We further investigate the mechanisms behind the superior performance by observing the training dynamics, and discover that SkyLadder exhibits more concentrated and effective attention patterns.
Overall, we suggest that the length of the context window is an important dimension in pretraining and should be scheduled over the course of training. We recommend a progressive approach that begins with small context of 8 tokens and gradually increases according to linear function of training stops. Given a target context window (c.g., 32K), we suggest that allocating approximately 60% of the total training tokens this expansion phase leads to stronger downstream performance compared to baselines. This scheduling strategy optimally enhances both training efficiency and model capability, offering practical recipe for improving pretraining in language models.
2 Related Work
contar: w Windol
Context Window
Context Whooew
Context Window Scheduling. Early work ex- plored gradually increasing the context win- dow in smaller models like BERT and GPT-
32K
32K
32K
improve training stability and efficiency [35, 28, 21]. Notably, Li et al. [28] proposed length warmup for more stable training but did not show clear performance gains, while Jin et al. [21] focused on training acceleration in 400M models. We extend these findings by demon- strating, for the first time, that context window scheduling significantly boosts both efficiency and performance at much larger scales (up to 3B parameters). A parallel approach from Pouransari et al. (38) segments training documents by length, but et al. [10] caution that such segmentation can introduce domain biases, longer texts often cluster in specific domains such books. Recent devel- opments in continual pretraining with long context windows [37, 55, 12], can also viewed through the lens of context window scheduling with different strategies (illustrated in Figure 2). Our work represents the first demonstration of both effectiveness and efficiency of context window scheduling, providing empirical evidence of its benefits in both standard and long-context benchmarks.
Training Tokers
Training Tokens
Treining Takens
(4) context window scheduling (ours)
(b) constant long context
(c) continual pretraining long context
Figure 2: Schematic comparison of training-time context window scheduling.
Long-Context I Language Models. Long- language models have received a lot of attention due to their ability to capture extended dependencies across large textual windows. Most existing

---

Pretraining Corpus
Packed Sequences
Packing
(eoo) Taken
Thiinin
Trucction & Concatmmation
Applying Altention Mask
Doc
Poc
Diee
Mideel with Contest Window
Due
MauulMutt hara Doc Head
Documentt varying longth
kergtt
Figure 3: An illustration of the workflow for pretraining data preparation highlights several critical decisions. Key considerations include the method of data packing, the type of attention mask to employ (causal or intra-doc mask), and determining the appropriate context window length L.
approaches follow a continual pretraining paradigm [10, 57], which extends pretrained backbone model longer contexts through specialized fine-tuning or additional training. Several works propose to intervene in the positional embeddings to accommodate longer sequences [1, 31, 1,, 37, 4, 22]. while others perform extended pretraining on longer-sequence corpora [12, 55, 32, 63]. Our approach differs from previous methods as we train native long-context models from scratch, rather than modifying a pretrained model post-training. Compared with naive long-context pretraining baseline with constant schedule, tasks, underscoring the benefits of training from scratch. These findings show that our method can be promising direction for future research building language models with longer context windows.
approach delivers substantial gains on multiple long-context
3 How Context Window Affects Pretraining
To investigate this in a fair and comparable manner, we pretrain language models from scratch with context windows ranging from 512 16,384 tokens under a fixed total number of tokens, evaluating via perplexity and downstream task benchmarks. We examine how the context window size impacts model performance, analyzing how data packing and masking strategies interact with window size.
How does context window affect pretraining? ?
3.1 Packing, Masking and Context Window
Most modern LLMs based on decoder-only transformer architecture [54] with fixed context window size denoted by L. In contrast, the pretraining corpus, = {dz, d2 d3...., dn consists of documents with varying lengths different from L. Therefore, key step before pretraining is to pack the documents into sequences of length L.. Formally, a packed sequence C₁ is constructed d 1 Trunc(d,,n) where represents concatenation, and Trunc denotes truncation documents to ensure len (C,) = L. Following previous works [44, 64), document boundaries within C; are explicitly marked using -nd-of-sequence ([EOS]) tokens.
C Trune (d4,) di,2
After the sequences are packed, the inputs are passed into transformer layers for next-token prediction training. A crucial component of these layers is the attention mechanism, which can be formulated as A., = q, ky. and then (ttn(X) = Softmax (A M). In decoder-only models, a mask M is applied to introduce constraints. A common approach is each position can only attend to previous tokens by masking out (setting to oo) attention scores corresponding future positions: M,j masking scheme, known as intra-doc mask [64, 13). imposes constraint that only allows tokens to attend each other if they belong to the same document. Let each document have start index sa and end index ea, the masking can be denoted as M when 3 d such that sæ S i,j < ed and otherwise. The model is trained with the standard cross-entropy loss on the packed sequences of length L. The workflow for pretraining data processing is illustrated Figure 3.
causal mask, which ensures that
for j and Maj otherwise. A recently proposed
j < i, and Mini
3.2 Preliminary Study on Context Window Size
As per Section 1, we initiate our study by investigating the impact of context window size on model performance through controlled experiment. Specifically, we pretrain language models with varying context window sizes, while preserving all other experimental settings. This enables pure analysis of the context window's 's influence on model performance. Through this analysis, we aim to understand whether longer context windows inherently lead to better or worse model performance.
3

---

Randam
InnuDoc
Villddea P7L
10.2
IntraiDrc:
300M
10
54
512
14
196
312
16
25
63.
56
156
7.5
10.0
12.5
15.0
26
56
Connect Windew (L)
Contaxt Window (L)
Training Tokent (B)
Coutext Window
(a) Validation perplesity
Validation perplesity differennt packing and masking strategtes across varying contect windowe
(c) Validation perplexitty of modelss without postional encoding across
(4)
Distribttio context windowe sizes for IntraDoc and Random methods
models with varping sizes across different contect window
content windows sizes
Figure 4: Ablation studies of different factors on different context window sizes. Note that the validation PPL is obtained on the validation documents with sliding window size of 512 tokens. The packing strategy in (a) is Random, and the model sizes in (b) and (c) are 1B and 120M, respectively. Note that the context window in (d) means the number of available preceding tokens when making next-token prediction (calculation details in Section A.6)
Key Variables. The context window size determines the number of tokens included in the context for each packed sequence. However, discussed earlier, several additional factors influence the content within the context window: (1) Packing methods determine which documents constitute the context window, and different packing strategies can significantly alter the composition of token sequences; (2) Masking methods decide whether cross-document attention is enabled within the same context window. The choice of masking affects how the information from different documents interacts during training.
Packing and Masking. To study the impact of packing, we employ two strategies: random packing and semantic packing. For random packing, documents are randomly concatenated without a specific ordering. For semantic packing, inspired by Shi et al. [44], we retrieve and concatenate semantically relevant documents from the corpus, aiming to keep them within the same context window. After experimenting with both dense retriever [20] and lexical retriever BM25, we found that BM25 gives stronger performance and chose it as our focus. For masking, the baseline approach is causal masking, where each token attend to all preceding tokens within the same context window, regardless of document boundaries. Conversely, recent studies [64, 9] show that disabling cross- document attention, thereby enabling intra-document attention, improves performance. For clarity in subsequent discussions, we denote random packing with causal masking as Random, BM25 packing with causal masking as BM25, and random packing with intra-document masking as IntraDoc.
Training. We pretrain models from scratch using the TinyLlama codebase [61], and study models with 120M, 360M and IB parameters. Given the substantial computational cost associated with retrieval in semantic packing, we randomly select around 30B tokens from the CommonCrawl (CC) subset of the SlimPajama dataset [46] as the pretraining corpus. All models undergo training for up to 100B tokens (~3.33 epochs). To ensure consistency across experiments, other settings, retaining the same batch size and learning rate schedule for all context windows. All models also incorporate Rotary Positional Encoding (RoPE) [47] to encode positional information. Appendix A.3 and A.4 give further model architecture details and training settings.
strictly control all
Evaluation. For all model sizes, we use perplexity (PPL) on validation documents from the original dataset as key metric, in line with established practices [10, 24, 17J. Note that when comparing models across different context windows (e.g., a 2K-context model and an 8K-context model), we must ensure the evaluation sequence fits within the shorter model's 's context window maintain fair comparison. We also evaluate 1B models on downstream standard benchmarks: HellaSwag [60], ARC-Easy and ARC-Challena
[6]. Winogrande [42], CommonsenseQA [48], OpenBookQA [34], PIQA [2], Social-QA [43], and MMLU [16]. We employ the OLMES suite [15] for the evaluation, it has been shown to provide reliable and stable results with curated 5- demonstrations [12].
3.3 Experimental Results
Figure I presents the main experimental result, obtained using the Random setting with 1 (B-parameter models. The results indicate that context window size significantly influences the performance of LLMs, with shorter contexts generally leading to better performance. To further investigate the

---

factors contributing to the observation, we perform comprehensive analysis to examine potential variables that may affect the conclusion. Figure shows our results, and we derive four key findings:
Findings: (1) The advantage of training on shorter contexts consistent across model sizes; (2) This advantage is independent of the packing and masking methods employed: (3) It is also unrelated to the use of positional encoding; (4) The best packing and masking strategy is IntraDoc, which outperforms others probably because it introduces a larger number of short contexts during pretraining.
Findings (1) and (2). As shown in Figure 4, regardless of the model size in (a) or the packing and masking methods in (b), a shorter context window for pretraining generally results higher average performance on benchmarks. The finding on benchmarks is consistent with the trend of validation PPL, where shorter context windows always yield lower PPL.
Finding (3). When using shorter context windows, one might hypothesize that the model learns positional encoding patterns for nearer positions more frequently, leading to better performance on standard benchmarks. To test the hypothesis, we systematically ablate RoPE by completely excluding it during pretraining, following prior work [25]. Figure 4(c), models trained with short-context windows still outperform their long-context counterparts, even in the absence of positional encoding, This suggests that the advantages of shorter contexts are independent of positional encoding.
Finding (4), From Figure 4(b), we observe that IntraDoc achieves the best validation PPL. across all context window sizes compared to Random and BM25, alongside consistently higher performance on standard benchmarks (c.f. Appendix A.7.1). This raises the question: why does IntraDoc excel? We attribute the advantage to the context window size distribution of IntraDoc, which implicitly increases the prevalence of shorter contexts. As illustrated in Figure 4(d), despite the sequence length of 8K, fewer than I 1% of context windows actually reach this limit. While prior work links the success of IntraDoc to reduced contextual noise [64], we identify complementary factor - reduced average context window size
key factor in its strong performance. That is, we hypothesize that the effectiveness of IntraDoc may also be closely tied to short context windows.
4 SkyLadder: Context Window Scheduling
We now present SkyLadder for progressively expanding the context window during pretraining.
4.1
Method
Saquence Langlt
Sequence Lengthe L
Inspired by learning rate scheduling, we ex- plore whether dynamically scheduling the con- text window from short to long during pretrain- ing could lead to performance improvements. This method can be implemented by applying multiple local "mini" causal masks to long, packed sequence. We illustrate this masking strategy Figure 5.
Random
IntroDoc
conting Toree
SkyLLdder
IntraDoc Skyl
Figure 5: An illustration of SkyLadder with Ran- dom and IntraDoc. The example shows packed sequence (length L) consisting of two documents. For SkyLadder, the context window w starts from a small value and dynamically adjusts during train- ing, eventually converging to the masking patterns of Random or IntraDoc.
Formally, we define a local window length w. The associated mask Ma is defined as fol- lows: M1 = 0 when & w < S i, and My = oo otherwise, where [ calculates the largest multiple of w that is less than equal to i, effectively defining a block wise attention mask for the query token position i. We linearly adjust the window size upwards by a constant factor per training step t: w(t) min min(we, w, + [at]), where and represent the ending and starting context window sizes, respectively. Here, a denotes the rate of expansion, and corresponds to the training step. As the training progresses, when the dynamic context window size w(l) even- tually reaches the desired (long) context window size = was it remains fixed at that value. At this point, the attention mask is equivalent to full causal mask. Notably, this method modifies
5

---

Table 1: Performance (aceuracy in %) of different 1B models pretrained 100B CC tokens standard benchmarks. denotes statistical improvements over the baseline (described $A.7.3).
Method
Avg.
ARC-E
ARC-C
CSQA
HS
OBQA
PIQA
SIQA
WG
MMLU
Random 46,3
58.0
32.7
49.6
43.0
40.2
64.8
46.4
51.9
29.9
+ SkyLadder
50.0 (+3.7)
65.4"
35.6"
56.8'
47.0"
42.8
64.8
48.9"
56.0"
32.4'
IntraDoc
47.4
61.8
33,4
52.7
45.6
38.0
64.3
45,7
54.8
30,5
49.3 (+1.9)
64.8"
33.8
55.4*
47.9"
39.4
66.1*
48.0"
56.4
31.8*
Table 2: Performance (accuracy in %) of IB models pretrained on 100B CC tokens with different meth-
ods on reading comprehension and long-context benchmarks, Detailed setup is in Appendix A.7.3.
Reading Comprehension Benchmarks
Long Benchmarks
Method
Avg.
HotpotQA
SQuAD
NQ
TriviaQA
RACE-h
Avg.
MDQA
RULER
Random
25.5
6.5
37.0
15.8
37.7
30.7
15.3
17.7
12.8
+ SkyLadder
30.2 (+4.7)
12.4
40.2
20.4
43.0
35.0
14.3
18.3
10.3
IntraDoe
28.7
11.4
39.0
18.2
42.3
32.3
13.0
15.3
10.6
+ -SkyL.adder
29.1 (+0.4)
11.0
38.5
20.4
41.5
34.3
13.2
15.6
10.7
the effective context window through masking, independent of how the sequences are packed. As such, this mask M., can be integrated with M which maintains the attention boundaries between documents; it can seamlessly combined with most packing and masking strategies.
4.2 Experimental Setup
We follow the same setup in Section 3.2 to pretrain language models with 8K context 100B tokens. We set W. = 32 and a = 1/8 by default, which means that a model roughly needs 64K steps (around 64B tokens) to reach the final desired context window of L 8192. All baseline and Skyl Ladde models are implemented with Flash Attention 2 [7] (pseudocode in A.5). We fix all other hyperparameters, such as the learning rate schedule, batch size, etc., for fair comparison. Due to resource constraints, we do not perform extensive hyperparameter search to obtain the best combinations for w(t), a, and w,, our ablation study, we show that these hyperparameters have negligible impact on performance, as long as they are within a reasonable range.
For evaluation, we the same suite mentioned in Section 3.2 with standard benchmarks. To evaluate the performance of long-context question answering within an 8K length, we utilize the 30-document setting from the Multi-Documedt QA (MDQA) benchmark [30]. This is widely-adopted benchmark that is shown to be reliable for models of IB scale [38, 64], with an average length of approximately 6K tokens. We also select synthetic tasks within RULER [18], defined by Yen al. [59]. We choose the setup of the task that fills up the model's target context window I.
4.3 Experimental Results
Tables 1 and present the main results, highlighting significant improvements achieved by SkyLadder across standard benchmarks, reading comprehension tasks and long-contex, benchmarks. For instance, compared to the Random baseline, integrating SkyLadder yields notable performance gains on standard tasks such as MMLU (+2.5%), ARC-E (+7.4%) and HellaSwag (+4%). This suggests that models with SkyLadder excel at learning common knowledge during pretraining. Additionally, our method further improves the performance of the strong baseline IntraDoc across many benchmarks. Meanwhile, for realistic long-contess benchmarks like MDQA, SkyLadder matches or exceeds baseline performance. For RULER, the performance difference is likely because of fluctuation caused by its synthetic nature and small size [55]. More long-context evaluation can be found in Section A.7.3, confirming that SkyLadder is comparable with or better than baselines long- context evaluation. In addition to Random and IntraDoc, we also verify that LkyLadder improves the performance of the BM25 model on both short and long tasks (Section A.7.4).
To address potential concerns that benefits observed in short contexts may stem from the high level of noise in CC, we conduct additional experiments using the Fine Pro dataset [65], carefully curated high-quality dataset containing 100B tokens. As shown in Table 4, improved data quality indeed leads to substantial performance gains. However, our key findings remain consistent: IntraDoc

---

Table 3: Performance (in %6 of 1B models pretrained on 100B Python code data. We follow the protocol of Huang et al. [19] to evaluate on HumanEval [3] and BigCodeBench [66]. t is the sampling temperature. SkyLadder shows consistent improvement especially for 32K-contex models.
HvmanEval
BigCodeBench
Greedy
Sampling (t = 0.8)
Greedy
Sampling (t = 0.8)
L
Method
Pass @
Pass@ 10
Pass@ 100
Pass@1
Pass @ 10
Pass @
32K Random
17.7
32.4
51.8
9.0
16.1
19.7
+ SkyLLadder
21.3
37.7
59.8
9.4
20.6
24.3
8K
Random
22.0
37.2
61.0
9.9
19.3
23366
SkyLLadder
23.2
38.2
63.4
11.3
20.0
24.1
Table 4: Performance (average accuracy over tasks, in %) of IB models pretrained Web-Po with an 8K context window.
Table 5: Performance (average accuracy in %) for models of different sizes.
Size
Method
Standard
Long
Method Standard
Long
Random
40.1
5,8
120M
Random
52.5
11.1
SkyLadder
41.2 (+1.1)
5.1 (.0.7)
SkyLadder
55.2 (+2.7)
12.3 (+12)
Random
47.2
8.9
360M
IntraDoc
54.3
12.7
SkyL.adder
49.6 (+2.4)
8.9
Skyl adder
54.8 (+0.5)
13.9 (+1.2)
Random
57.0
15.8
3B
SkyLadder
60.5 (+3.5)
19.3 (+3.5)
continues to outperform Random, and SkyLadder consistently delivers significant improvements over both baselines. This demonstrates that our method generalizes corpora of varying quality.
We further examine whether SkyLadder generalizable beyond natural language tasks. Follow- ing Ding et al. [9], pretrain 1B code models on 100B Python code with the Starcoder tokenizer [29]. We observe a lower training loss (~~ 0.9) for code pretraining compared to natural language (~~ 2.1), suggesting that the structure in code makes the training easier. However, shown in Table 3, there is still significant improvement when applying SkyLadder under both greedy decoding and sampling setups, especially when the target context length is 32K. This demonstrates the potential of SkyLaader to coding and possibly other reasoning tasks beyond natural language modelling.
4.4 Scalability Experiments
We examine whether SkyLadder's improvements persist as we scale up the model parameters and extend the context window size. We use the largest model and context size that our compute permits.
Model Size. We conduct experiments across three model sizes: 120M, 360M, and 3B parameters on the Fineweb-Pro dataset. Table demonstrates that models utilizing SkyLadder consistently achieve better standard benchmark performance on all model sizes. For long context tasks, our method does not benefit 120M models, possibly due to their limited capacity processing long sequences. However, the performance gain on 3B models is prominent. We observe a positive scaling trend: as the model size grows, the performance improvement also increases, indicating the potential of applying method to even larger models beyond our current scale. We leave it as future work to explore larger models as requires significantly more compute.
Context Window Size. To examine whether SkyLadder can effectively scale to longer context windows, we train IB models with a 32K context window on 100B FineWeb- Pro tokens. We adjust a to 1/2 to ensure that the final context window expands to 32K before the end of pretrain- ing. As shown in Table 6, our model demonstrates strong performance on both standard and long benchmarks. In addition, the performance difference of SkyL (0.9%) between the 8K and 32K models is largely reduced com- pared with the baseline approach (1.8%), which alleviates
Table 6: Performance (%) of 1B models trained on 100B FineWee-Pro tokens with 32K context window.
Method
Standard
Long
Random
50.7
9.7
SkyLadder
54.3 (+3.6)
13.5 (+3.8)
IntraDoc
54.0
13.0
SkyLadder
54.9 (+0.9)
14.4 (+1.4)
7

---

15.8
Pf 512
PPL A
Valibaiion PM MM
12.8
15.8
12.6
13.4
12.4
124
152
12.2
152
1222
15.0
12.0
15.0
12.0
12
52
2.56
Expansion Rate (1/e)
Initial Window Length (Wa)
Figure 6: Validation PPL 512 and 8K contexts of models with different expansion rate (left) and initial window length w, (right).
Table 7: Comparison of 1B models trained with 32K context window with different scheduling methods. Numbers are average accuracy (%)
Table 8: Comparison of relative training time and compute efficiency for 1B Models with different context window sizes L. FLOPs calculation fol- lows Zhang al. [61]. larger context window leads to more efficiency gains.
Method
Long
Standard
Constant Long (32K)
9.7
50.7
Method
Time (%)
FLOPs (1020)
Linear (32-+32K, default)
13.5
54.3
Stepwise Linear (32-+32K)
13.3
55.3
Random (8K)
100.0%
11.6
Sinusoidal (32-+32K)
14.2
54.2
SkyLadder
86.9% (-13.198
9.9 (-14.%%
Exponential (32-+32K3
11.5
54.7
Random (32K)
100.0%
25.5
Cont. Pretrain (4K->32)
10.0
52.9
-SkyLadder
77.8% (-22.2%)
18.8 (-26.3%)
the performance degradation described in earlier study. Notably, compared to the baseline Random approach, SkyLadder trains the model on progressively shorter contexts during earlier stages. This reveals counterintuitive insight: naively training model with long context window is not always optimal, even if the model is evaluated on long contexts. In contrast, strategic scheduling of the context window during pretraining can yield better results.
4.5
Ablation Study
We now examine the impact of hyperparameters in SkyLadder scheduling. To manage computational costs, adopt default setup of pretraining 120M models with 8K context on 100B CC tokens.
Expansion Rate. We investigate the impact of the expansion rate a in Figure 6 (left). We choose different a ranging from slowest (1/ (1/12) to fastest (1). Our findings reveal that, for short contexts, performance generally improves as the expansion rate slows down. However, selecting an excessively slow rate (e.g., 1/12) can negatively affect long-context performance due to insufficient training on longer contexts. Therefore, we recommend setting to 1/8 for good balance.
Initial Context Window. As the final context window length we is fixed to L, the sole remaining hyperparameter is w.. Intuitively, setting w, to excessively large value (e.g. close to L) leaves little room for scheduling, resulting in sub-optimal performance. In Figure 6 (right), we demonstrate that when w, is set to a relatively small value (e.g., 8), great performance short and long contexts. This suggests that there is still potential for further improvement in our default setup. Therefore, recommend starting with small context window, such as 8 tokens.
be achieved for both
Scheduling Type. The default scheduling method in SkyLadder is linear scheduling. We evaluate different context window scheduling types (more details Table 20 and Figure 12 in Appendix A.7.4):
(1) Stepwise Linear rounds window size w(t) to multiples of IK, resulting in step function; (2) Sinusoidal increases quickly at the early stage then slows down; (3) Exponential starts slow but accelerates sharply; (4) Continual pretraining setup trains with 4K context windows for ~97B tokens, then switches 32K context for the final 3B tokens. Table shows that linear and sinusoidal schedules outperform the exponential variant on long tasks, likely because the exponential schedule, with extended short-context pretraining at the beginning, fails to adequately train on long contexts. Last, the most commonly used continual pretraining setup performs poorly overall, suggesting abrupt context changes harm both short and long performance. These findings suggest that context window scheduling is superior to both constant long-context pretraining and continual pretraining.

---

Overall, we conclude that the schedule should start from a small wy and the expansion should be gradual. We leave it future work to study more advanced schedules and discover optimal configurations. For instance, it is possible that the schedule needs to be adjusted for various model sizes. More ablations for combination with BM25, hybrid attention, cyclie schedules and scheduling under compute budget can be found in Appendix A.7.4.
4.6 Analysis and Discussion
Training Efficiency. We observe a significant boost in training efficiency when employing Sky- Ladder in Table 8. On 8K models, Skyl.ader accelerates training time by 13% due to the reduced context window caleulating attention. With a 32K context window, the efficiency gain becomes more pronounced: our method saves 22% of training time while achieving better performance. The FLOPs saving is larger than the actual time because of reduced attention calculation.
Attention Pattern. We next investigate why SkyLadder, despite being trained on short contexts overall, consis- tently outperforms the bascline. As language models rely on attention mechanisms to encode context information, we study how attention patterns change. Specifically, dur-
30
3.5
Atten tion Emiropy
Atteationn Sink
ing pretraining, we monitor the dynamics of (i) attention entropy (solid lines in Figure 7), where lower entropy is associated with better downstream performance [62]; (ii) attention sink [56], where the initial token in the context receives disproportionately high attention. We utilize the metric in Gu et al. [14] to quantitatively measure the ampli- tude of attention sink. As shown in Figure 7 (dashed lines), compared with the baseline Random, SkyLadder demon- strates reduced attention entropy, suggesting more con- centrated attention pattern. However, slower emergence and lower amplitude of attention sink are simultaneously observed. This suggests that SkyLadder's attention is con- centrated on the key information in the context instead of the initial token, which accounts for the performance gain.
-33
3.2
Training Tokens (B)
Baseline
Attention Sink
SkyLadler
Attention Entropy
Figure 7: Dynamics of attention sink and entropy during pretraining 1B mod- els (8K context). SkyLadder delays the emergence of attention sink while lower- ing the overall entropy, indicating a more effective attention pattemn.
Max Attention Logit
Training Stability. To further understand the reasons behind SkyLadder's better performance, we analyze the impact of pretraining context length training dynamics, We pretrain 120M-pann -arameetee models with different context lengths. We first monitor the maximum attention logits (Smax = maxij 4 kj for all i. throughout pretraining, following the methodology of K2 [50]. A large attention logit indicates that an attention head is malfunctioning and may cause numerical instability. In Figure 8, we observe that pretraining with a long context of 16K tokens leads exploding max attention logits, while a shorter window leads to lower attention logits.
100
Max Atlention Logit
80
20-
0-
Training Tokens (B)
2K
8K
Figure 8: Max attention logits during training of models of different context lengths (in different colors).
Next, we study the loss and gradient behavior by comput- ing four stability metrics over the first N 30K steps of pretraining, where L, denotes the training loss and G2 is the gradient norm before clipping:
Loss Volatility: measures local fluctuations of loss over a sliding window (w = 10), computed as 3 ΣN Std(44-wtt.... L.). Lower values indicate more stable training.
Loss Smoothness: the average loss change between consecutive steps, N-I ΣM Lr Li-il Smaller values mean smoother convergence.
Mean Loss Ratio [28]: measures temporary increases in loss relative to the best loss so far,
N-T Enz
min tin where smaller values indicate fewer loss spikes.
L.

---

Table 9: Training stability metrics during pretraining of 120M models with different context lengths. All metrics are averaged over the first 30 billion tokens. + indicates that smaller values are better.
Context
Volatility + (w=10)
Smoothness
Mean Loss Ratio +
Avg Grad Norm +
IK
0,023
0.019
1.014
0.335
2K
0.026
0,023
1.017
0,338
4K
0,030
0,029
1.020
0.340
8K
0.036
0.036
1.025
0.347
16K
0.041
0.042
1.036
0.416
Average Gradient Norm: NV Et= min(Gr, 1), where larger values indicate more aggressive
gradient updates.
In Table 9, longer-context models show higher volatility, less smooth loss curves, more frequent upward spikes, and larger gradient norms, all indicating less stable optimization. In contrast, short- context models converge more smoothly with smaller fluctuations and more controlled gradient updates. Together, these results reveal that short-context pretraining is inherently more stable, both in attention behavior and optimization dynamics. The reduced numerical instability and smoother convergence likely enable more consistent gradient signals and better overall convergence, explaining their superior downstream performance.
Comparison with Related Work. We compare method with another approach for improving pretraining in Table 10. As discussed in Section 2, Pouransari al. [38] proposed Dataset Decomposi- tion (DD) by segmenting a document into sequences of varying lengths and using a curriculum during pretraining. However, this approach inevitably in- troduces domain bias, as the document lengths in different domains are different [10]. This explains why DD with only one short-to-lone cycle fails to outperform the IntraDoc baseline. To mitigate this, the authors suggested iterating through multiple cy-
Table 10: Comparison between SkyLadder and Dataset Decomposition (DD) on 1B mod- els trained with 100B Fine Prro tokens. Numbers are in average performance in %.
Model
Standard
Long
IntraDoc
54.3
12.7
+ SkyLadder
54.8 (+0.5)
13,9 (+1.2)
+ DD cycle)
53.9 (-0.4)
12.3 (-0.4)
+ DD (8 cycles)
54.5 (+0.2)
13.5 (+0.8)
cles of long and short data, which does improve performance substantially. In contrast, our method achieves better performance by avoiding such biases by not altering the data order based on length. In Appendix A.7.4, we experimented with various cyclic schedules but did not observe any im- provements. In fact, we noticed loss spikes between cycles (Figure 14), indicating potential issues with domain shifts. This further supports that our method is safer since it does not disrupt the natural ordering and distribution of the data. More discussion with other related works [28, 21] is in Section A.8, where demonstrate that our work provides novel insights that scheduling the context window over the entire training time improves both efficiency and performance.
Conclusion
We conduct a comprehensive controlled study of the impact of context window on pretraining, revealing that a shorter context window is more beneficial to the model's performance on standard benchmarks. This debunks the trend of pretraining with longer context windows. We therefore propose SkyLadder to schedule the context window from short to long during pretraining, which gives substantial improvement in downstream performance and computational efficiency. We conclude that context window scheduling is an important dimension for pretraining, and deserves more consideration. In the future, we plan to explore more dynamic and performant scheduling strategies that adapt according to model size or pretraining data distribution.
10

---

References
[1] Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, and Lingpeng Kong. Why does the effective context length of ilms fall short?, 2024. URL
https:// /ttpa::///xvvvvv/aa org/
httpa//.rrg/bb//11...... ab2/2410....... 18745.
[2] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
[3] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https: //arxiv. ht///bb/222 /2007 03374.
[4] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL https://arxiv org/abs/2300 /206. 15595.
[5] Yu-An Chung, Hung- Yi Lee, and James Glass. Supervised and unsupervised transfer learning for question answering, 2018. URL httpt://arxiiv http:///aaaaiv.oaaaa//////////// arggabs//711 .05345.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: /awxxaaxvvvo//// orgoorggg/ooo aaaaaa//100 .04457.
[7] Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
(8] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 Minneapolis, Minnesota, June 2019, Association
(Long and Short Papers), pages for Computational Linguistics. doi: 10. 11.11865/11/N11-11 142191133/ URL https: //c/aacaanntoooyyyyoo org/N19- 1423/.
[9] Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. Fewer truncations improve language modeling. In Proceedings of the s1 International Conference on Machine Learning, pages 11030-11048, 2024.
[10] Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Proceedings of the 41st International Conference Machine Learning, pages 14125-14134, 2024.
[11] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for shot language model evaluation, 07 2024. URL attps://zenee org wrgrrecords// 12608602.
[12] Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectivelyy), 2025. URL https: : /atxxivivvv//////////a/////aaal/ org/ htgg/bbs/4411 0261/
11

---

[13] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models, 2024. URL https: /ttp////xvvvv..///// www.iii.rgg/...eee..oo.ee /2007 21783.
[14] Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang. Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view, 2024, URL https:/ /trps: org/ abs/////..... 10781.
[15] Yuling Gu, Oyvind Tafjord, Bailey Kuchl, Dany Haddad, Jesse Dodge, and Hannaneh Ha- jishirzi. OLMES: A standard for language model evaluations. In Luis Chiruzzo, Alan Ritter, and Lu Wang. editors, Findings of the Association for Compulational Linguistics: NAACL 2025. pages tional Linguistics. ISBN 979-8-89176 195-77 URL https: hatpe////aaaaahlggaeeeee 002/ findings- naainngs-nacc..882 282/.
Albuquerque, New Mexico, April 2025. Association for Computa-
[16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[17] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optiman large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 97817 9781718871088
[18] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's 's the real context size of your long-context language models? In First Conference on Language Modeling, 2024.
[19] Siming Huang, Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang. Yuan Qi. Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models, 2024. URL https: /ttp:v//////vvvva waw.///11.1 04905.
[20] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2021. URL https: / aaxiivi org/ brrg/bbs/21) abs//a12..100 09118.
[21] Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Chia- Yuan Chang, and Xia Hu. Growlength: Accelerating LLMs pretraining by progressively growing training length, 2023.
URL https: org/ abs/ /210.. 2310.00576. 00576.
[22] Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. LLM maybe longlm: Selfextend L.M context window without tuning. In Forty-first International Conference on Machine Learning, 2024.
[23] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers). pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1/17711111177 URL https: http:////aaaaaatooaaa/ao org/P17-1147/
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: http://aaxvvv...//ee
08361.
[25] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023, Curran Associates Inc.
12

---

[26] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-We Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 11452-466 2019. doi: 10.1 1 a a_00276. URL
www.iloggyrrgg/ss/ otg//19-1022000 1026//
[27] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1/655//11D17--0822 URL https: httpa///aaaaaathhlooeeoc org/D17-1082/.
[28] Conglong Li, Minjia Zhang, and Yuxiong He. The stability-efficiency dilemma: investigating sequence length warmup for training gpt models. In Proceedings of the 36th International Conference Neural Information Processing Systems, pages 26736 26750,, 2022.
[29]
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poiries João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt Dan- ish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!. 2023. URL https: //ale://///xxii /arxiv. org/ /////b////// 2305 061611.
[30] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 10.1 a 00638. URL httpps:/accaatthlllee
[31] LocaILLaMA. NTK-aware scaled rope allows Ilama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degration, 2023. URL https://www http:///www.rrditt.mm htteiiittcom///eee/eeeeeeeeeeeet com/:/LocalLLaM.
wttpraaaaaaaaaa//aeooo//aaaao//e (ttpl///////tawwreesseee vope_allows_o sawlllosslllaaas lwt.aalaaeaaaaaaaaaaaaeaaaaealaa have/.
[32] Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush. A controlled study on long context extension and generalization in lims, 2024. URL https: wrgg ab////bb///..... 12189.111111.
[33] Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and Weipeng Chen. Base of RoPE bounds context length, 2024. URL http::///axxvvvorrepee abs/ 14591.
[34] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391 Brussels, Belgium, October- November 2018. Association for Computational Linguisties. doi: 10.18653/v1/D18-122 1260. URL https://aclettt http///ccaaathhoogyyyy Drg/D18-1260/.
[35] Koichi Nagatsuka, Clifford Mooni-Bediako, and Masayasu Atsumi. Pre-training a BERT with curriculum learning by increasing block-size of input text. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 202 pages 989-
996, 2021.
13

---

[36] Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5336-5358, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1// m2a.0222maclmmminnn9.. URL https://
aclanthology wtw.//22......oooo wwgg/2222.aaclmmaanrnre naacl- main.. m91m
[37] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttps://openreeeee
net/ foott/foummmdda WHBf:idd--d-duuu
[38] Hadi Pouransari, Chun-Liang Li, Jen-Hao Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, and Oncel Tuzel. Dataset decomposition: Faster Ilm training with variable sequence length curriculum. Advances in Neural Information Processing Systems, 37:36121- 36147. 2024.
[39] Alec Radford. Improving language understanding by generative pre-training. OpenAl blog, 2018.
[40] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[41] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pages 2383 2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 18.186/3/v 1/D16- 1264. URL https: / httpa/aaaaaanoooo...ooooooooolla wrt. 1264/.
[42] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 2021.
[43] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNIP--JCNI.P), pages 4463-4473, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10. 10.166///1//11914555 1454. URL bhttps//acclntheen
atg/Di9--a 1454/.
[44] Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Victoria Lin, Noah Smith, Luke Zettlemoyer, Wen-tau Yih, and Mike Lewis. In-context pretraining: Language modeling beyond document boundarics. The Twelfth International Conference on Learning Representations, 2024.
[45] Lestie Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pages 464-472. IEEF, 2017.
[46] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. sl slimpajama-
https: http://wwwceeebbreeeeeeee seerebras..et//log..... net/blog/ jma, co/ htoo/daaaees////eeoooooo wwweeeebbrr/Siiimnee /ot.a//aaaaeeeeeeeerrr//aaaaaaa/ www.:iimaaaaaaa21a1a11 627B.
www.ttookenneeeeeeeeee cleaned and wtw.:deupppiaatedeeeeeiiiiiooooe 6 2023. URL tttts://huggiing
[47] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 127:13 2024.
[48] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense@A:
A question answering challenge targeting commonsense knowledge. In Proceedings of the
14

---

2019 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long and Short Papers), pages 4149- 4158, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653//1/119-144 URL. https:::::: httpss///aaaaaatooo.oooooo
1421.
[49] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report, 2025. URL paattps://arxiie .wwpsiiiorrr..oooo arg/abs/// /203. 19786.
[50] Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui. Hao Ding. Mengnan Dong, Angang Du, Chenzhuang Du, Dikang Du, Yulun Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong. Yangyang Hu, Zhenxing Hu, Weixiao Huang. Zhiqi Huang. Zihao Huang. Tao Jiang, Zhejun Jiang, Xinyi Jin, Yongsheng Kang, Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Li, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma. Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi. Feifan Song. Jianlin Su. Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng. Chensi Wang, Dinglu Wang, Feng Wang, Haiming Wang, Jianzhou Wang. Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shuyi Wang. Yao Wang. Yejie Wang. Yiqin Wang, Yuxin Wang, Yuzhi Wang, Zhaoji Wang. Zhengtao Wang. Zhexu Wang, Chu Wei, Qiangian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xie, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang, Ying Yang. Zhen Yang. Zhilin Yang. Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye. Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang, Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang, Yizhi Zhang, Yongting Zhang. Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng. Shaojie Zheng, Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi k2: Open agentic intelligence, 2025. URL https: // //trx//axvvv///////////e.//a.av. org/ hts///b//250000 20534.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https: :/apxii:///xxiv..// org/ abs/ /lls/2250 2202 13971.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jían Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CORR, https///30.0928... 2023. doi: 10.48550/ARXI1.2307.0000 URL httpt://doo..coooo org/ www.o/////...///////////////..// ariv 2307 092888
[53] Bo-Hsiang Tseng, Sheng-Syun Shen, Hung-Yi Yi Lee, and Lin- Lin-Shan Lee. Towards machine comprehension of spoken content: Initial toefl listening comprchension test by machine, 2016. URL https://arxiv org/ aa//abb//1100 06378.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon,
15

---

U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. htt.tttss: neerrdd ccc wwtpcppr.pppppp///et fiwes/ /tlee//paper/211///lla 4faee245477ee.fa00ccccctataaaaaa Paper. pdf.
[55] Haonan Wang, Qian Liu, Chao Du, Tongyao Zhu, Cunxiao Du, Kenji Kawaguchi, and Tianyu Pang. When precision meets position: B 6 breaks down rope in long- training, 2024.
URL. http///axvvvvrroee .rgg ab///41s....... 2211 13476.
[56] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp////ppeeeeeeeee//e/ee net/ forum? 7 NG7
[57] Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oğuz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-contens scaling of foundation models. In North American Chapter of the Association for Computational Linguistics, 2023.
[58] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 236992380 Brussels. Belgium, Detober-November 2018. Association for Computational Linguistics. doi: 10. 18653v 10665//1/DD11.. 1259. URL https: // aac anthologhygy...... . org
org/D18- /D18-1181111 1259/.
[59] Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-conten language models effectively and thoroughly. In International Conference on Learning Representations (ICLR), 2025.
[60] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[61] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024. URL httpp:://aaaie org/abs////////a/.oo/ooooeee abs/2401. 02385.
[62] Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, and Dong Yu. Attention entropy is a key factor: An analysis of parallel context encoding with iil-attentionbbasee pre-trained language models, 2024. URL https: //arxiv. org/ abs/2412 16545.
[63] Liang Zhao, Tianwen Wei, Liang Zeng, Cheng Cheng, Liu Yang, Peng Cheng, Lijie Wang, Chenxia Li, Xuejie Wu, Bo Zhu, Yimeng Gan, Rui Hu, Shuicheng Yan, Han Fang, and Yahui Zhou. Longskywork: A training recipe for efficiently extending context length in large language models, 2024. URL ht
2406. 00605.
[64] Yu Zhao. Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Miloš, Yuxiang Wu, and Pasquale Minervini. Analysing the impact of sequence composition on language model pre-training. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, edi- tors, Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 7897-7912, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. https://acll tholoooyyyyyyyooo org/ 2024. acl- 1ong..2g20 427/
URL
[65] Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu. Programming every example: Lifting pre-training data quality like experts at scale, 2024. URL hnttps://arxt org/ abs/ 2409. 17115.
[66] Terry Yue Zhuo Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong. Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateck Yaday, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian
16

---

Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. URL https:/ pttctps://areee htt////xxvvrrrr/ee org/ ab/////s..2/...///// 15877.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract summarizes the findings of our preliminary study and the essence of the SkyLadder method: we systematically study the impact of context length on pretrain- ing and empirically verify the effectiveness of our approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of our study in Section A.1.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer:
[NA]
Justification:
This paper is not theory paper and we do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed hyperparameter setting (Section 3.2, A.4), implementation pseudocode (A.5) and model configuration (A.3). We implement most of the experiments in TinyLlama, popular public project for pretraining, which makes it highly reproducible.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our pretraining code is based on TinyLlama. an open-source pretraining framework. The data based on open-source pretraining corpora like SlimPajama and Fineweb-Pro. We include the code in the supplementary materials, and will open source the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tttps://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
ctt.ps.ccppublcc//// weeiii/gguieeee guwdee/ide//// CCoeeSumeeeeaeeeeeeee woabbbmssssoon ssisnnollll. ssionnyccyy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the training and testing setup in detail in Section 3.2 for our preliminary study, and in Section A.7.3 for SkyLadder experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct McNemar test for our main results in Table The details for testing are in A.7.3. Due to the excessive cost of pretraining, we do not perform repeated runs of the same setup. However, our claims are supported by multiple variations of the pretraining and comprchensive ablation studies in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We share the compute information in Section A.9.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ttttps://neet httpp/s//////./././/eee http:///weurpppcccppb.eeeae cc/public/Etteeeeeeeeeeeeeeeeeee hetllii//ttistttteee csQuidelinnen ines?
Answer: [Yes]
Justification: We fully anonymize the paper and the code. All experiments fully conform with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impact of our work in Section A.2.
21

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work is not meant release a pretrained language model or publish dataset.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We explicitly mention and cite the source of the datasets (SlimPajama, FineWeb- Pro) and the implementation codebase in paper (TinyLlama). License in- formation is described in Section A. All of them are open-souree projects available for public use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
22

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide instructions to reproduce our experiments with the code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This project does not involve human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This project does involve human subjects crowdsourcing.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We did use LLMs for developing our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ cattps://neuree wwtccc////coocccccccccceeoo /00e:///// for what should or should not be described.
24

---

Table 11: Model configurations for pretrained language models.
Model
Tinyllama 1B
Tinyllama 120M
Tinyllama 360M
Llama3. 2 3B
Vocab Size
32000
32000
32000
32000
Layers
22
12
18
28
Heads
32
12
16
24
Embedding Dim
2048
768
1024
3072
Intermediate Size
56332
2048
4096
8192
Normalization
RMSNorm
RMSNorm
RMSNorm
RMSNorm
Normalization
10-
1 10 1 No
1 10
10
Query Groups
4
16 No
8
Bias
No
No
10000 if L 8K
RoPE
10000
10000
100000
1000000 if L 32K
A
Appendix
A.1
Limitations
While we perform extensive experiments study the impact of context window on pretraining and demonstrate the effectiveness of SkyLadder, we acknowledge that there are still limitations to be addressed. First, we conduct experiments up 3B-model scale and 32K context length, while the latest large language models are typically much larger and capable of processing longer contexts. However, pretraining large model with long context window requires prohibitive computational resources beyond our budget. Within our computational capabilities, we have tried to demonstrate the generalizability of SkyLadder across corpora, context window size. model size, and downstream tasks. Thus, we leave it as future work to apply SkyLadder to larger models. Second, we do not include a theoretical analysis explain the effectiveness of Skyl Lkyle as we mainly focus on empirical insights. We suggest that future work may investigate the relationship between the context window and the training compute to obtain the optimal context window schedule.
A.2 Broader Impacts
The work aims to investigate the impact choices on context windows in language model pretraining and proposes a way to speed up pretraining by scheduling context windows. On the positive side, this improves the efficiency of language model pretraining, making more accessible and reducing the carbon footprint. Moreover, it enhances the performance of pretrained language models, which may result in better downstream performance applications. There might be potential misuse of pretrained language models, which is beyond the scope of this work.
A.3 Model Architecture
In Table I1, we list the architecture choices of the models trained, including the 120M, 360M, and IB models based on the TinyLlama architecture [61]. The 3B model is based on Llama3.2 architecture [13].
A.4 Training Configurations
We include details of the training configurations in Table 12. All models, irrespective of size or context window length, are trained this same set of hyperparameters. For most of the hyperparameter values, we follow the TinyL [61] project, therefore, our results are highly reproducible.
A.5 Implementation
We provide the pseudocode for implementing SkyLadder with Flash Attention 2 [7]. The only change is to apply local causal masking with size w, and combine them with the original document boundaries under the IntraDoe scenario. can easily be integrated into any model before calculating attention. The rest of the training pipeline remains unchanged.
25

---

Table 12: Hyperparameters setup for pretraining the language models. All pretrained models follow the same structure.
Parameter
Value
Optimizer
Adam W
AdamW-St W
0.9
Adam
0.95
Learning Rate Schedule
Cosine
Peak Learning Rate
4e-4
Minimum Learning Rate
4e-5
Warmup Steps
2000
Gradient Norm Clipping
1
Total Steps
100,000
Global Batch Size
1,048,576 (229) tokens
Weight Decay
0.1
SkyLadder with Flash Attention
# q, k, v: RoPE-encoded query, key, value tensors
# _oc__oundaries: EOS token positions per document
# is is_intradoc: intra-document attention flag
training.step: current global step
# L: maximum context window length
# get current window size min(L, get_current_ saatt__aaa_______ lenggglttttttrrnnnnn
# breakpoints every w tokens (and at document boundaries if using IntraDoc masking)
mask_ booudorouddriesss arange L, w)
if is is_indrdo::: mask_boundaries np. ap.unionnd (mask (mask_ boundaries, foc_boundaries)
compute max segment length cumulative lengths for flash attention max_seqlen get_max_seqles cu_seqlens net_cu_seq]eee maask_ounddries, L)
L)
attn - flash_ attn_ func
9. v, cu_seqlens, max max_seqlen, causal-True
)
A.6 Definition of Per-token Context Window
In Figure 4(d), show the context window distribution difference between IntraDoc and Random. To clarify, the context window size refers to the number of preceding tokens available in the context window when making the next token prediction. This is different from (a) and (b), where the context length is the model's pretrained context window.
Formally, consider token at index i and attention mask matrix M, where entry Mt, 0 indicates that token can attend to token j. and -oo otherwise. The context window size C, for the i-th token defined as C. Σ 1 [M,, 0}. where 1 {-} is the indicator function that returns when M., and otherwise. essence, C; the number of tokens available as context for the i-th token, and the distribution of C, over all pretraining tokens is in Figure 4(d).
26

---

For Random, the causal mask is triangular: the i-th token has a context window size equal to (i.e., C C2 2, etc.). Thus, the distribution C; is uniform. contrast, IntraDoc effectively shortens the context length by limiting the cross-document attention.
A.7
Additional Results
A.7.1
Context Window Study
Randoca
BM25
IntraDoc
10-
36k
8k
Valiain
Validatoon PTC.
Valiatiin
1k
14-
512
73
100
103
91
80
Trininn Tokers (B)
Taising Tokers (B)
Training Tokers (B)
Figure 9: Validation perplexity (evaluated sliding window of 512) models with different context lengths.
165
Bk, 9- 10*
- Bk, 9-
100
162
Validarion PPP
BM25
Aveerage Accuncy
= 9=
9.5
15.5
2k, 1-
Valitastoon PML
15.2
16k,
Randem BM25
14.5
14.3
%
Evalintion Length
Training Tokerss (B)
Figure 10: Left: Evaluation perplexity of models with different packing or masking strategies. Right: Downstream performance over tasks of different models.
1008
3200
6032
Evaluation Length
Figure 11: Validation perplexity vs training tokens with different context windows and base of ROPE, 0. Evaluation done on a sliding window of varying length (x-axis) on the validation documents.
In Figure We plot the validation perplexity of models with different context windows under the Random, IntraDoc, and BM25 settings. We observe consistent trend that shorter-context model has lower evaluation perplexity on a shorter sequence under all settings.
In Figure 10, we plot the evaluation perplexity and downstream performance of models with different packing or masking strategies. We conclude that overall, IntraDoc achieves the best performance, with consistently lower PPL and a higher downstream accuracy. We think that this is partially due the shorter context window that the IntraDoc model is trained on.
A.7.2 Ablations for Context Window Study
Base of RoPE. It has been shown that the value of RoPE may have a significant impact on the model's long context performance, and a longer context requires larger base [33]. Therefore, we increase the RoPE base to 100,000, which is sufficiently large according to Men et al. [33]. In Figure 11, we observe an improvement for long-context models on long-context evaluation. However, the large gap between shorter and longer model still remains, therefore rejecting the hypothesis that the RoPE base the key contributing factor the superior performance of short-context models.
27

---

Table 13: Performance of 3B models long tasks of atrreval-augmeeneed generation (evaluated by exact-match scores) and reading comprehension benchmarks (accuracy in %).
Retrieval Augmented Generation
Reading Comprehension
Model
Avg.
NQ
TriviaQA
HotpotQA
PopQA
Avg.
TOEFL
QuALITY
Random
30.3
24.3
45.2
29.3
22.5
37.1
43.5
30.6
SkyLadder
35.5
27.8
52.7
32.3
29.3
39.4
48.0
30.9
Table 14: Many-shot ICL performance (accuracy) on text classification benchmarks. Numbers in parentheses denote the number of labels for each task.
Model
Avg.
DBpedia (14)
AGNews (4)
Amazon (2)
Yelp (2)
SST2 (2)
Random
73.9
17.4
68.6
94.3
94.7
94.5
SkyLadder
76.5
25.5
75.8
94.1
95.0
92.2
A.7.3 SkyLadder Evaluation
Statistical Test We test the statistical significance of the performance difference between our models and baselines in Table 1. We use McNemar test as the two models are evaluated on the same set of questions. The original OLMES suite samples 1000 examples from each benchmark's 's full evaluation suite. contrast, when conducting the McNemar test, we evaluate models on the full set to obtain more statistically meaningful results. We note that OpenBookQA only has 500 questions, making it harder to obtain statistical significance.
Reading Comprehension For reading comprehension, we evaluate the following benchmarks: Hotpot QA (2-shot) [58], SQuAD (4-shot) [41], NaturalQuestions (NQ) (2-shot) [26], TriviaQA (2-shot) [23], and RACE-high (0-shot) [27]. We follow the setup by Zhao et al. [64], where NQ and TriviaQA use retrieved documents as contexts. For RACE, we use lm- Im-evaluation. haallaatii---hanesss [11] to compare the PPL. between options.
Long-context Evaluation We provide additional long-contess evaluation with an 8K context. This is to mitigate the performance instability of using synthetic benchmarks small models. We first follow [38] to evaluate model accuracy reading comprehension benchmarks TOFEL [5, 53] and QuALITY [36]. Next, we evaluate the model's performance on Retrieval Augmented Generation (RAG), where the model is provided with many relevant but potentially noisy contexts and needs to locate the correct information. As shown in Table 13, SkyLadder consistently performs better than the baseline across all evaluated RAG and reading comprehension datasets, highlighting its ability to locate correct answers within a lengthy con- text. addition, we test the in-context learning ability of the models on text classification bench- marks [64, 44J. Results in Table 14 suggest that SkyLadder shows significant gain for tasks with many labels, such as DBpedia, while achieving comparable high performance on binary tasks.
largest 3B model
Closed-book QA We additionally evaluate the closed-book QA performance of our models without access to any document. We use the evaluation protocol Zhao et [64] to measure the exact match. In Table 15, we notice a signif- icant improvement our methods compared to the baselines for answering closed-book ques- tions. This is consistent with the results that our models show improvements standard bench- marks that contain commonsense knowledge.
Table 15: 1B model (trained CC) performance (exact match %) on closed-book QA tasks.
Closed-book QA
Model
NQ
TriviaQA
Average
Random
6.1
11.9
9.0
+ SkyLadder
17.5
13.2
IntraDoc
7.8
14.7
11.3
+ SkyLadder
8.2
17.4
12.8
A.7.4 SkyLadder Ablations
Combination with BM25 Packing
Skyleadder only changes the context length via masking without altering the underlying data, it is orthogonal to any advanced data packing method such as Shi
28

---

Table 16: Performance (%) of 1B models with different schedule types. All models are trained on the same 100B CommonCrawl tokens with final context length of 8K. BM25 packing, when combined with SkyLadder, significantly boosts per- formance on long tasks.
Table 17: Performance (%) of 1B models with different schedule types. All models are trained on the same 100B FineWeb-Pro tokens with final context length of 8K. Short-to-long schedul- ing is consistently better than long-to-short scheduling.
Standard Avg.
Long Avg.
Standard Avg.
Long Avg.
Random
46.3
15.3
No Scheduling
52.5
11.1
BM25
47.5 (+1.2)
(+1.)
Short-to-Lond
55.2 (+2.7)
12.3 (+1.2)
SkyLadder
49.8 (+3.5)
17.0 (+1.7)
Long-to-Shooe
52.6 (+0.1)
10.7 (-0.4)
Table 18: Evaluation perplexity for Gemma3-like models with different evaluation context lengths Le. All models are trained with 100B tokens on CommonCrawl.
Model
L
512
L,
4K
L.
8K
Random 120M
15.9
13.4
13.0
+ SkyLadder
15.5
12.9
12.4
Random - 360M
12.1
10.2
9.8
SkyLadder
11.6
9.7
9,4
al. [44], Ding et [9]. In Table 16, we combine the SkyLadder with the BM25 packing method. We show that the model achieves even better performance on both short and long context evaluation than BM25 without scheduling, which also better than the Random baseline. This reveals that our method can be combined with more advanced packing techniques to further boost performance.
Combination with Hybrid Attention We note that a recent interesting trend in pretraining models with long context is to use hybrid attention structure. For instance, Gemma3 [49] uses mixture of global and local attention layers to balance efficiency and performance the long-conters model. We curious about the generalizability of SkyLadder to such architecture, and follow Gemma3's strategy with a global-to-local ratio of 6:1. The results are presented in Table 18. We observe that SkyLadder consistently outperforms the baseline across all evaluation lengths, verifying its applicability. Importantly, SkyLadder works along the time dimension and is combinable with different attention variants, as long there is context window to be scheduled. We also verified the effectiveness of SkyLadder on alternative model structures. In Table 19, we pretrain models following the Qwen2.5-0.5B structure, and obtain consistent gains as well.
Long-to-Short Schedule A possibility that SkyLadder works better than baseline on standard benchmarks, which are typically short, might be that the training data mix has more short-context data after applying the mask. To study the effect of pure data distribution. we conduct an ablation of reversing the original short-to-long schedule and name it as the long-to-shon schedule. This schedule spends the same number of tokens (64B) in the changing phase, before the constant training phase in T= 8K for another 36B tokens. In Table 17, we show that the long-to- schedule not helpful to the model's performance in both short and long evaluation tasks. This highlights that the context window needs to be scheduled, rather than simply having a data mixture long and short contexts.
We explore various types of short-to-long scheduling following different functions as mentioned in Section 4.5. Table 20 shows the details of the schedule as a function of f, and Figure 12 shows an illustration of the different schedule types. In Table show
Alternative Schedule Types
Table 19: Evaluation perplexity for Qwen2.5-0.5B models under different evaluation context lengths Le. All models are trained with 100B tokens on CommonCrawl.
Model
L. 1K
L.
4K
L.
8K
Random
14.8
13.1
12.5
SkyLadder
14.3
12.7
12.1
29

---

Table 20: Functions for different context window schedule types. We set w,, 32 and We 32768 in experiments. The for rounding is set to 1024.
32K
Context Window
Constant
Linear
20K
Stepwise
Sinusolial
Schedule
Function
10K
Exponential
Constant Linear
We W. (we
Cont. Pretrain
Wis)
an us Lizz
0
Stepwise
max(wa, (W,T
Training Tokens (B)
Sinusoidal
W/ (we wis) sin
2( wa)
Figure 12: Illustration plot of various scheduling types.
Exponential
Wa
Mi D0
that smoother increase following the sinusoidal schedule works the best for long-context evaluation, while also achieving strong performance on standard benchmarks.
2003
8000
3.0
7000
4.5
7000
4.5
6000
Cunoo. Rind
4.0
5000
4000
4000
3.5
3.5
5000
3000
2000
2000
3.-
1000
1000
2.5
Training Tokens (B)
Training Tokens (B)
Grddell 144 155
Buacire
== 1/2
== 1/10
Lows
Pamp 1/4,
1/4
1/12
Contest Window
Figure 14: An illustration of the cyclic schedules with gradual increases or jumps. Dashed lines represent the context length for each step, and solid lines are the loss evaluated at an 8K length. e represents the number of cycles.
Figure 13: An illustration of the effect of differ- ent a. Dashed lines represent the current context window w for each step, and solid lines are the loss evaluated 8K length.
Expansion Rate We illustrate the effect of the rate of expansion a in Figure 13. As the evaluation is done on 8K contexts, models with a lower rate (and shorter context window) will have higher models' loss converges to
loss the evaluation length is nut-of-distribution. However, eventually, low level after the schedule reaches 8K. The detailed numbers of validation loss after pretraining can be found in Table 21. Following previous work [10, 17, 24], we consider a loss difference larger than 0.01 as significant. We conclude that setting reasonable rate of 1/8 balances both short and long-context loss, which is the default setup for our main experiments.
Cyclic Schedule Inspired by the cyclic schedule learning rate [45], we also wonder if cycles are helpful in the schedule. In Figure 14, we show two cyclic schodules. In the "Jump" schedule, w(t) will decrease to w, immediately after reaching L. On the other hand, the "Gradual" schedule means an "M" shape alternating between We and w. Notably, in the discontinuous Jump schedule, we notice a significant increase in long-context perplexity when train only short contexts for extended period. However, as long as w increases back to L, the performance will return.
In Table 22, we show that these schedules have no major impact on the final performance. This highlights that the method does not introduce additional bias data selection: different from existing methods such as Pouransari et al. [38] that proposes to train short data first, followed by long data, we not assume such curriculum on data. We argue that the context window size should be independent of the data lengths to avoid bias in training only certain domains of data.
30

---

Table 21: Validation loss with different expansion rates. A box is colored red if it is significantly worse (difference > 0.01) than the best the column. Le is the evaluation context length. All models of size 120M and trained on 100B tokens.
Rate (1/a)
Tokens to Reach 8K (B)
Lg = 512
Le 4K
Lg = 8K
2.751
2.563
2.522
2.741
2.551
2.514
32
2.740
2.551
2.515
8
64
2.732
2.553
2.519
9
72
2.731
2.553
2.519
80
2.732
2,555
2.522
2.730
2.554
2.521
12
96
2.729
2.557
2.526
Baseline (Constant)
2.780
2.590
2.349
Table 22: Validation loss with cyclic schedules. Le represents the evaluation context length. All
models of size 120M and trained on 100B tokens.
Type
Number of Cycles
Tokens per Cycle (B)
Le
512
Le 8K
Random
2.780
2.549
SkyLadder
2.732
2.519
Gradual
4.5
16
2.743
2.530
Jump
2.744
2,532
Gradual
2.5
32
2.732
2.521
Jump
5
16
2.733
2.521
Gradual
1.5
64
2.728
2.524
Jump
3
32
2.727
2.522
Initial Window Length We show the effect of having different wg, the initial window length when the training starts. In Table 23, show that the optimal starting length is 8 tokens. The trend is the same across both a = 1/4 and a = 1/8. This suggests that the starting length should be sufficiently small, irrespective of the expansion rate. It also reveals that prior studies, such as Jin et al. [21] and Pouransari et al. [38] that start with an initial length of 256 could be suboptimal.
Compute Budget We show that when the total number of tokens is limited, our method can still improve language model performance. In Table 24, we choose 12.5B, 25B, and 50B total tokens as the computing budget, and vary the expansion rate so that reaches L at the same point during training. We observe that under different token budgets, the performance trend is the best: gradually expanding the context window gives better performance than rapid increase.
Sliding Window Expansion A possible alternative to SkyLadder (using local causal masks by default) is to use sliding window attention with a window size of w(() that changes with the training time. Formally, the mask becomes:
wi-wsy ifi-wjjs
Mr, = -00 otherwise.
so that each token in the context has fixed preceding context of size w. When w(t) reaches L, the mask becomes equivalent causal mask. We compare the performance of the two in Table 25 and observe that the sliding window approach shows slightly better performance in long tasks and worse performance standard benchmarks. This likely because overall there are more tokens with longer preceding contexts for the sliding window approach. In both cases, SkyLadder outperforms the Random baseline. We think that future work could further investigate the differences between SkyL Ladder implementations with causal and sliding window attention, such as the formation of attention sink [14]. There could also be possible combinations of the two: for instance, using local mask first to disable distraction, and enabling sliding windows as the training progresses.
31

---

Table 23: Final validation loss after training 120M models 100B tokens with different w, when a = 1/4 and a = 1/8. Le represents the context length evaluation. cell is colored red if its loss has difference larger than 0.01 from the column's 's best. W, 8192 equals scheduling.
Le 512
Le
1K
L. 8K
1/4
4
2.731
2.546
2.510
8
2.730
2.545
2.508
16
2.733
2.551
2.513
32
2.740
2.551
2.515
64
2.742
2.557
2.520
128
2.748
2.564
2.528
256
2.750
2.566
2.527
1/8
4
2.727
2.549
2.515
8
2.725
2.545
2.510
16
2.729
2.550
2.516
32
2.732
2.553
2.519
2.735
2.553
2.519
128
2.743
2.564
2,530
256
2.748
2.567
2.531
8192
2.780
2.590
2.549
Table 24: Final validation loss under different training token budgets and expansion
with 120M
models. Le represents the context length used for evaluation. "% of Token Budget" means how many tokens spent in the expansion phase with w(() increasing. Under all token budgets, observe consistent improvement when we spend around 64% in expansion, and 36% the stable phase.
α Tokens to L (B)
% of Token Budget
Le
512
L
4096
Le
8192
Token Budget
12,58
1
8
64%
2.912
2.732
2.698
2
4
32%
2.933
2.746
2.709
4
2
16%
2.958
2.767
2.729
8
8%
2.976
2.782
2.743
Baseline
3.008
2.823
2.790
Token Budget
25B
1/2
16
64%
2.829
2.650
2.617
1
8
32%
2.841
2.656
2.619
2
4
16%
2.851
2,665
2.626
4
2.873
2,683
2.645
Baseline
2.918
2.734
2.700
Token Budget
508
1/4
32
64%
2.771
2.590
2.556
1/2
16
32%
2.781
2.596
2.560
1
8
16%
2.789
2,603
2.564
2
4
8%
2.795
2.607
2.567
Baseline
2.839
2.652
2.616
A.8 Additional Comparison with Related Work
We acknowledge that there are several prior works discovering similar pattern of short-to-long pretraining. For instance, Li al. [28] discover that using sequence- warmup for the initial steps in pretraining improves model stability. However, they mostly focus on stability in training loss and not show clear performance gain across multiple evaluations and larger scales. Moreover, we demonstrate that the benefits of scheduling model's context window go beyond only the warmup stage. Table 21 first row, simply warming up the model with 8B tokens results suboptimal performance compared to slower expansion rate. This validates that the context window should be
32

---

Table 25: Performance (%) of 1B models with different masking schemes. All models trained on the same 100B Fine tokens with a final context length of 8K. Both implementations of SkyLadder outperform the baseline, and the sliding window approach excels at long tasks with slight performance drop on standard benchmarks.
Model
Standard Avg.
Long Avg.
Random
52.5
11.1
+ SkyLadder w/ local causal
55,2 (+2.7)
12.3 (+12)
SkyLadder w/ sliding window
54.4 (+1.9)
12.8 (+1.7)
considered as a factor to schedule over the entire training course, which also differentiates us from Li et al. [28] that only consider the warmup stage.
Another related work is Jin et al. [21] where the authors use progressive sequence lengths to accelerate training. However, their method leads to worse performance under the same token budget, while our SkyLadder shows both time saving and performance improvement with the same number of tokens. We suspect that this might be because of the suboptimal schedule they used. Moreover, their study limited to observing the training loss of small models (up to 410M parameters), while we comprehensively show performance gain across multiple corpora, model sizes, context sizes, and wide variety of tasks. Overall, we systematically conduct controlled experiments on the impact of context window scheduling in pretraining, providing insights to explain these previous studies.
A.9 Compute Information
We conducted all of our experiments for models with IB size internal cluster of NVIDIA A A100 nodes with 40G memory. Experiments with 3B models were conducted on H100 nodes. There are additional preliminary experiments that we did not include in the paper, which account for fraction of the total compute. The detailed computation for each experiment is as follows: For the preliminary study on context window, pretraining IB model with 100B tokens (with 8K context) takes around 200 hours on node of 8 A A100s. Models of different sizes scale accordingly. For instance, plotting Figure 4(a) and (b) requires total of 159 days of pretraining on single node. For SkyLadder experiments, the baseline pretraining using various corpora takes the same time, and SkyLadder speeds up the training by 13% to 22% depending on the context length.
A.10 Dataset Statistics
In this section, we provide detailed statistics of the datasets used in our study. These include the document length distributions of the pretraining corpora, the characteristics of the evaluation datasets, and the input length statistics of standard reasoning benchmarks.
Table 26 reports the document length statistics for the two pretraining corpora, CommonCraw/ and Fine WemeWerrrro Both distributions are strongly right-skewed, indicating that long documents are rare. Compared to Fin-Wor- CommonCrawi generally contains longer documents, while Fine Web-Prorr has been more carefully cleaned and filtered.
Table 26:
Document length statistics of the pretraining corpora, measured in tokens per document. Mean, median, and standard deviation describe the central tendency and variation. P25 and P75 indicate the 25th and 75th percentiles, while skewness and kurtosis capture distribution asymmetry and tail heaviness.
Dataset
Mean
Median
StdDev
Min
Max
P25
P75
Skewness
Kurtosis
CommonCrawl
1973
1067
4567
594,272
1867
21
820
FineWcb-Pro 1364
849
2295
230,949
507
1481
15
533
Table 27 shows the input length characteristics of common reasoning and knowledge benchmarks, including ARC, CSQA, HellaSwag, OBQA, PIQA, SocialIQA, Winogrande, and MMLU. While these benchmarks consist of relatively short contexts, they remain standard for assessing model's factual consistency and reasoning ability. Importantly, a long-context model should maintain stable behavior even when the user provides short query.
33

---

Table 27: Input length characteristies of common reasoning and knowledge benchmarks. Although relatively short, these tasks are crucial for measuring knowledge and reasoning consistency.
Metric
ARC-C
ARC-E
CSQA
HellaSwag
OBQA
PIQA
SIQA
WinoG.
MMLU
Mean
222
216
146
508
129
224
226
149
540
Std
20
16
7
32
29
6
4
512
Min
191
194
134
435
116
191
210
140
155
Max
401
336
203
576
192
440
262
170
3144
Finally, Table 28 summarizes the characteristics of the evaluation datasets used in the reading comprehension and long-context evaluation. These include QA benchmarks such MDQA, RULER, SQuAD, HotpotQA, NQ. TriviaQA, and RACE. The datasets differ substantially in input length, reflecting the diversity of reasoning depth and context complexity.
Table 28: Length statistics of reading comprehension and QA evaluation datasets. These benchmarks capture varying levels of input complexity, from short factual QA to multi-hop reasoning tasks.
Metric
MDQA
RULER
SQuAD
HotpoiQA
NQ
TriviaQA
RACE
Mcan
5150
7259
1048
5010
583
566
492
287
745
81
993
21
28
121
Min
4172
6209
923
3587
536
529
122
Max
6755
8061
1174
7842
633
643
1323
Overall, the datasets used in this work span a wide range of input lengths and domains, from large- scale pretraining corpora to short and long-context evaluation benchmarks, ensuring that our analysis both comprehensive and representative.
A.11 Licenses of Assets
We mainly use the following public datasets or codebases in this paper: SlimPajama [46] following the CommonCrawl Foundation Terms Use", FibeWW [65] with ODC-By 1.0 license, and TinyLlama [6]] with an Apache 2.0 License.
34


Reviews:
[{"review_id": "AoFcNAtCow", "reviewer": "Reviewer_Tqtf", "summary": "This paper proposes SkyLadder, a simple yet effective context window scheduling strategy for pretraining LLM. Instead of using a fixed long context window throughout training, SkyLadder gradually increases the window size from small to large, improving both model performance and training efficiency. Through extensive experiments on models up to 3B parameters and context lengths up to 32K, the authors show that SkyLadder consistently outperforms fixed-window baselines across standard benchmarks, long-context tasks, and multiple masking strategies, while also reducing computational cost.", "questions": "1. What training scale was used in the \"Continual Pretraining\" experiment (Tab. 7)? Was an evenly spaced expansion (e.g., 4K → 8K → 16K → 32K) used for context window sizes?\n2. Does SkyLadder affect the model's ability to generalize to unseen sequence lengths? Is there a risk that the model’s generalization ability to longer sequences may be compromised by the gradual expansion of the context window during training?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper conducts extensive and sufficient experiments to support its conclusions.\n2. The proposed context window scheduling strategy offers valuable insights for training long-context LLM\n3. The proposed SkyLadder method improves both model performance and training efficiency.\n\nMajor Weaknesses\n1. The paper provides solid experimental evidence for the effectiveness of context window scheduling, but the overall innovation is limited.\n2. Although the authors note that experiments were limited to models below 3 billion parameters due to computational constraints, the absence of validation on larger-scale models (particularly 7B-level LLM), undermines the persuasiveness of the method's scalability and practical applicability.\n3. Although the paper conducts a series of experiments based on the TinyLlama baseline, the lack of comparative analysis with other advanced models, such as LLaMA 3.2 3B, Mistral 3B, and Qwen 2.5, makes it difficult to demonstrate the superiority of the proposed method.", "was_revised": true, "final_justification": "The response addresses W1 to some extent; however, I am not fully satisfied, as attention masking appears to be a general design choice. W2 and W3 remain unaddressed, and the authors should demonstrate the applicability of the method. The other questions have been addressed appropriately. After considering the other reviews, I will not oppose accepting this paper; however, I will not advocate for its acceptance either.", "rebuttal": "Thanks for your valuable feedback. We appreciate that you find our study insightful. Here are our responses to your concerns in the Weaknesses section:\n\n***The paper provides solid experimental evidence for the effectiveness of context window scheduling, but the overall innovation is limited.***\n\nThanks for acknowledging the rigor of our study. We want to clarify that the main contribution of our paper is to provide a new insight that context window scheduling can boost the downstream task performance, which is \"novel\" compared to previous works. As discussed in the Related Work section, previous works that explore related ideas are mostly designed to accelerate training, without demonstrating consistent performance gains. Our work is the first to conduct a large-scale, controlled study (up to 3B models, 32K context) showing that context window scheduling not only improves training efficiency but also significantly boosts downstream task performance.\n\nFurthermore, unlike approaches that select documents of certain lengths (which risk introducing domain biases), SkyLadder dynamically adjusts the effective context window through attention masking, mitigating the distribution shift issues in long-context training. This combination of scale, performance improvements, and bias mitigation constitutes an important contribution beyond prior works.\n\n\n---\n\n***Scaling to larger models.***\n\nWe are also curious about further scaling and acknowledge this as a limitation. We are unable to finish the pretraining of 7B models during the rebuttal period, as training a larger model naturally requires more tokens beyond our compute capability. However, we believe our findings have provided strong evidence of scalability:\n\n- **Positive scaling trend**: In Table 5, As shown in Table 5, the performance gain from SkyLadder grows larger as model size increases (from 120M to 3B parameters), suggesting its potential in larger models. \n- **Consistent gain across different settings**: We demonstrate consistent improvements across different corpora, masking strategies, evaluation tasks, and model architectures. This shows that SkyLadder is not specific to a single configuration and should generalize to larger models\n\nThese results support the scalability of our method to larger models, which we leave as future work. \n\n---\n\n***The lack of comparative analysis with other advanced models, such as LLaMA 3.2 3B, Mistral 3B, and Qwen 2.5.***\n\nWe appreciate the reviewer's suggestion to include comparisons with advanced models. However, a direct comparison is not meaningful due to fundamental differences in training setups:\n- First, our paper looks at pretraining a model *from scratch*. These public LLMs are trained on orders of magnitude more data (>10T tokens), and their superior performance is largely attributed to the large data scale. \n- The pretraining data for these models is not publicly available, making it impossible to control for differences in data quality or preprocessing. \n- Our goal is not to release a new state-of-the-art model, but to isolate and study the effect of context scheduling under a fixed token budget, where we can rigorously control for all other variables.\n  \nOur 3B model exactly follows the Llama3.2 3B architecture (Table 10), and we train two models on exactly the same tokens to verify the effectiveness. This is a strictly controlled setup that rules out confounding factors such as improved data cleaning. \n\nFor other architectures, we note that Mistral has no 3B model officially. Therefore, we additionally pretrain Qwen 0.5B models from scratch on 100B tokens to show the generalizability of our approach. The table below shows the PPL on the held-out validation set. \n\n| Model      | PPL (1K) | PPL (4K) | PPL (8K) |\n|------------|----------|----------|----------|\n| Random   | 14.8     | 13.1     | 12.5     |\n| + SkyLadder  | **14.3**     | **12.7**     | **12.1**     |\n\nSkyLadder improves the validation PPL across all evaluation lengths. \n\n| Model      | Average | ARC-E | ARC-C | HellaSwag | MMLU  | CSQA  | OBQA  | SocialIQA | PIQA  | WinoGrande |\n|------------|---------|-------|--------|-----------|-------|--------|--------|-------------|--------|--------------|\n| Random   | 42.6    | 51.9  | 28.5   | 39.3      | 28.6  | 43.7   | 33.2   | **43.9**   | 61.2  | **53.4**     |\n| + SkyLadder  | **44.3**| **55.6**| **31.0** | **40.3**  | **29.7**| **48.2**| **35.2**| 43.7       | **62.4**| 52.2         |\n\nOn common benchmarks, SkyLadder also performs better overall. These results confirm that our method is applicable to Qwen models and generalizes across architectures. We have also verified its effectiveness on hybrid-attention models such as Gemma 3 (Table 17; Appendix Line 742).\n\n---\nHere are our responses to the specific questions in the “Questions” section.\n\n***What training scale was used in the \"Continual Pretraining\" experiment (Tab. 7)? Was an evenly spaced expansion (e.g., 4K → 8K → 16K → 32K) used for context window sizes?***\n\nTable 7 shows results for 1B models pretrained on 100B high-quality Fineweb-Pro tokens, with a final target length of 32K. We did not employ gradual expansion for the Cont. Pretrain setup in Table 7. However, we agree that it is interesting to examine the evenly spaced expansion as suggested, and we conducted additional experiments. The results are shown below:\n\n| Model                                 | Standard Avg. | Long Avg. | MDQA | RULER |\n|--------------------------------------|----------------|-----------|------|-------|\n| Constant                             | 50.7           | 9.7       | 12.8 | 6.6   |\n| SkyLadder                            | **54.3**       | **13.5**  | **18.4** | **8.5**   |\n| Cont Pretrain (4K -> 8K -> 16K -> 32K) | 52.9           | 10.4      | 15.3 | 5.4   |\n\n\nAlthough this gradual continual pretraining setup outperforms the baseline, SkyLadder still achieves the best performance across all tasks. This result indicates that the schedule should be applied from the beginning of pretraining, rather than only during the continual pretraining stage.\n\n---\n***Does SkyLadder affect the model's ability to generalize to unseen sequence lengths?***\n\nThanks for this interesting question. Our answer is that it **improves the generalization ability to unseen lengths**, as shown by the following observations.  \n\nFirst, we directly evaluate pretrained checkpoints trained with an 8K window on sequences longer than 8K, following the protocol in [1]. The table below shows the perplexity (PPL) on the GovReport dataset for different evaluation lengths (each column) up to 32K:\n\n| Model      | 8K   | 12K   | 16K   | 24K   | 32K   |\n|------------|------|-------|-------|-------|-------|\n| Random     | 6.9  | 24.4  | 99.4  | 364.1 | 579.8 |\n| + SkyLadder  | **6.7**  | **9.6**   | **22.0**  | **62.6**  | **99.5**  |\n\nInterestingly, the SkyLadder model, with gradual expansion, achieves lower perplexity beyond the training length. In contrast, the PPL of the Random baseline quickly explodes. \n\nSecond, we also implemented long-context continual pretraining, where both models are extended from 8K to 32K using 2B additional tokens from [2]. We evaluate the models on the RULER benchmark. \n\n| Model      | 32K  | 16K  | 8K   | 4K   |\n|------------|------|------|------|------|\n| Random     | 27.2 | 38.6 | 41.8 | 51.1 |\n| + SkyLadder  | **29.8** | **40.3** | **47.9** | **55.7** |\n\nIn this case, the SkyLadder model outperforms the baseline as well. This confirms that gradual context expansion leads to a stronger model with better generalization to unseen lengths.\n\nWe hope our responses have addressed your concerns and clarified the contributions of our work. We would greatly appreciate it if you could consider raising your score. \n\n---\n***References***\n\n[1] Peng et al. YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024. \n\n[2] Fu et al. Data Engineering for Scaling Language Models to 128K Context. ICML 2024."}, {"review_id": "aNtByoUcfc", "reviewer": "Reviewer_FJPN", "summary": "This paper reveals that models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget. Motivated by this observation, the paper aims to better balance long-context capability with pretraining efficiency. To this end, it proposes SkyLadder, a method that progressively expands the context window during pretraining. SkyLadder can be integrated with strategies such as Random, BM25, and IntraDoc. Experimental results demonstrate that SkyLadder effectively achieves a trade-off between efficiency and long-context performance.", "questions": "See weaknesses", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Interesting finding: models pretrained with shorter context windows consistently outperform their long-context counterparts under a fixed token budget.\n2. The method is simple, effective, and efficient.\n3. Experimental results and analysis are good.\n\n\n Weaknesses:\n1. Is the 100B token budget used in Figure 1 sufficient? The observed results may be due to an insufficient number of tokens. I suggest conducting additional experiments with a larger token budget to validate the findings. In addition, is this due to the length characteristics of these nine datasets?\n\n2. The experiments in this paper are conducted on models with up to 3B parameters. It remains unclear whether the conclusions can generalize to larger-scale language models.\n\n3. It would be better to extend Tables 1 and 2 to include more pre-training corpora, rather than just the 100B CC.\n\n4. It would be beneficial to evaluate the proposed method on a broader range of long-text benchmarks to provide a more comprehensive assessment.\n\n5. Why does Random+SkyLadder outperform IntraDoc+SkyLadder in both Table 1 and Table 2? Is this due to the small context window?\n\n6. Since the evaluation in this paper is highly related to input length, I suggest including information about the dataset lengths to provide a clearer presentation.", "was_revised": true, "final_justification": "Most of my questions have been answered, but the only remaining question is whether this article can be generalised to larger LLMs. Since I am not an expert in this field, I cannot be sure that my assessment is entirely accurate.", "rebuttal": "Thanks for your valuable feedback. We are delighted that you find our work interesting. Here are our responses to each question:\n\n***1. More tokens for Figure 1.*** \n\nWe conducted additional experiments by training 1B models from scratch on 200B Fineweb-Pro tokens, with a context length of 1K, 4K, and 16K. Results on downstream tasks are shown below: \n\n| Training Length | Avg   | ARC-E  | ARC-C  | CSQA  | HellaSwag | OBQA   | PIQA   | SocialIQA | Winogrande | MMLU   |\n|-----------------|-------|--------|--------|-------|-----------|--------|--------|-----------|------------|--------|\n| 1K              | **56.6** | **74.1** | **43.5** | **61.8** | 61.0     | **49.4** | **73.6** | **51.2** | **59.8**      | **34.6** |\n| 4K              | 54.5   | 71.7   | 40.6   | 58.1   | **61.1** | 45.2   | 71.3   | 49.7      | 58.1       | **34.6** |\n| 16K             | 53.7   | 69.1   | 40.5   | 56.8   | 60.9      | 43.2   | 72.6   | 48.6      | 57.9       | 33.9   |\n\nThe trend remains consistent: shorter context windows lead to better overall performance under a fixed token budget. We plan to extend training to even larger budgets (e.g., 500B tokens) in future work.\n\nThe following table represents the length characteristics of these datasets:\n| Metric | ARC-C  | ARC-E  | CSQA  | HellaSwag | OBQA  | PIQA  | SocialIQA | Winogrande | MMLU  |\n|--------|--------|--------|-------|-----------|-------|-------|-----------|------------|-------|\n| Mean   | 222    | 216    | 146   | 508       | 129   | 224   | 226       | 149        | 540   |\n| Std    | 20     | 16     | 7     | 32        | 9     | 29    | 6         | 4          | 512   |\n| Min    | 191    | 194    | 134   | 435       | 116   | 191   | 210       | 140        | 155   |\n| Max    | 401    | 336    | 203   | 576       | 192   | 440   | 262       | 170        | 3144  |\n\nThese benchmarks are commonly reported when a base model is released. While these benchmarks are relatively short, they are standard for assessing a base model’s knowledge and reasoning abilities: we do not want a long-context model to hallucinate simply because the user provides a short query. \n\n---\n\n***2. Generalization to larger models.*** \n\nWe also hope to validate SkyLadder on larger models like 7B. However, such experiments require more tokens and compute, which is infeasible during the rebuttal period. Our current results already demonstrate generalizability across multiple model sizes, context lengths, datasets, and tasks. As shown in Table 5, the relative gains of SkyLadder increase with model size (from +1.1% for 120M to +3.5% for 3B), indicating that the benefits are likely to be even larger as model size grows.\n\n---\n\n\n***3. Extend Table 1 and 2 to more corpora.*** \n\nWe reported the performance of pretraining on Fineweb-Pro in Table 4. We report the performance of these models on standard benchmarks below:\n\n\n| Model      | Average | ARC-E | ARC-C | CSQA | HS   | OBQA | PIQA | SIQA | WG   | MMLU |\n|------------|---------|-------|-------|------|------|------|------|------|------|------|\n| Random     | 52.5    | 69.0  | 39.2  | 56.8 | 55.7 | 43.0 | 70.2 | 48.5 | 56.0 | 33.7 |\n| + SkyLadder | **55.2** | 72.6  | 40.3  | **60.7** | **60.7** | **48.4** | 71.8 | **49.1** | **58.1** | **35.3** |\n| IntraDoc   | 54.3    | 72.2  | 41.0  | 58.3 | 59.8 | 45.4 | **72.8** | 48.7 | 55.8 | 34.7 |\n| + SkyLadder  | 54.8    | **72.8** | **42.9** | 58.2 | 59.6 | 46.2 | 72.4 | 48.6 | 57.6 | **34.8** |\n\nModels with SkyLadder added show steady performance gains on these commonly used benchmarks, even when trained on a different corpus. \n\nFor benchmarks listed in Table 2, the scores are as follows:\n\n| Model      | RC Avg. | HotpotQA | SQuAD | NQ   | TriviaQA | RACE-h | Long Avg. | MDQA | RULER |\n|------------|---------|----------|-------|------|-----------|--------|-----------|------|--------|\n| Random     | 20.4    | 1.9      | **25.4**  | 15.0 | 25.4      | **34.0**   | 11.1      | 15.5 | 6.7    |\n| + SkyLadder  | **21.2**    | **7.5**      | 24.3  | 13.9 | **26.5**      | 33.8   | 12.2      | 15.9 | 8.4    |\n| IntraDoc   | 20.2    | 5.9      | 24.1  | 11.8 | 25.3      | 33.7   | 12.1      | 14.8 | 9.5    |\n| + SkyLadder  | 20.7    | 6.6      | 23.3  | **14.0** | 25.8      | 33.9   | **13.9**      | **17.3** | **10.4**   |\n\nSkyLadder achieves better or comparable performance on these benchmarks as well, demonstrating its generalizability.\n\n---\n\n\n\n***4. More long-context benchmarks.***\n\nWe evaluate our largest 3B model on additional long-context tasks (Tables 12 and 13 in Appendix A.7.3). Key results are summarized below:\n\nFirst, we follow a related work [1] to evaluate models on TOEFL and QuALITY. SkyLadder achieves better performance on these benchmarks. \n\n| Model      | TOEFL | QuALITY |\n|------------|-------|---------|\n| Baseline   | 43.5  | 30.6    |\n| + SkyLadder  | **48.0**  | **30.9**    |\n\n\n**Many-shot ICL**: We follow [2] and [3] to test on text classification. Numbers in brackets are the number of labels for each task. \n| Model      | DBpedia (14) | AGNews (4) | Amazon (2) | Yelp (2) | SST2 (2) | Average |\n|------------|--------------|------------|------------|----------|----------|---------|\n| Baseline   | 17.4         | 68.6       | **94.3**   | 94.7     | **94.5** | 73.9    |\n| + SkyLadder  | **25.5**     | **75.8**   | 94.1       | **95.0** | 92.2     | **76.5**|\n\n\nSkyLadder shows significant gain for tasks with many labels. Both models achieve comparable performance on binary tasks, possibly because the performance is almost saturated above 90%. \n\n**Retrieval-Augmented Generation**: We follow [4] for evaluating long-context RAG performance on multiple QA datasets. We fill the entire 8K context for this setup. \n\n| Model      | NQ   | TriviaQA | HotpotQA | PopQA | Average |\n|------------|------|----------|----------|-------|---------|\n| Baseline   | 24.3 | 45.2     | 29.3     | 22.5  | 30.3    |\n| + SkyLadder  | **27.8** | **52.7**    | **32.3**    | **29.3**  | **35.5**    |\n\nSkyLadder outperforms the baseline by a large margin across all QA datasets in the RAG task, which is an important application scenario for long context models. \n\nOverall, as shown by these additional results, SkyLadder can be better than or comparable with the baseline on multiple long context evaluations while achieving significant performance gains on standard benchmarks.\n\n---\n\n\n***5. Random + SkyLadder outperforms IntraDoc + SkyLadder***\n\nWe thank the reviewer for spotting this intriguing phenomenon. First, we note that both SkyLadder and IntraDoc create shorter contexts; thus, the gain of applying SkyLadder to IntraDoc is smaller compared to Random. Towards the end of training, Random+SkyLadder (denoted as RS) is trained on overall longer contexts compared to IntraDoc+SkyLadder (IS): the RS model is able to see contexts beyond the current document. Thus, RS is exposed to more diverse and noisy multi-document contexts. Such exposure likely teaches the model to filter out irrelevant content and focus on key information. Thus, it shows better performance on the benchmarks, especially the longer ones with noisy contexts.  \n\n---\n\n\n***6. Information about dataset lengths***\n\nAs we are unable to attach figures in the rebuttal, we have to present the statistics as a table. We first show the distribution of the pretraining corpora we used: \n\n| Dataset     | Mean | Median | StdDev | Min | Max   | P25  | P75  | P90  | P95  | P99   | Skewness | Kurtosis |\n|-------------|------|--------|--------|-----|--------|------|------|------|------|--------|------|------|\n| CommonCrawl | 1973 | 1067   | 4567   | 45  | 594272 | 651  | 1867 | 3601 | 6005 | 16261 | 21   | 820  |\n| Fineweb Pro | 1364 | 849    | 2295   | 1   | 230949 | 507  | 1481 | 2557 | 3810 | 9733  | 15   | 533  |\n\nBoth distributions are right-skewed, which means long documents are rare. Overall, the CC corpus has longer documents, as Fineweb-Pro has been carefully cleaned and processed. \n\nAs for the evaluation datasets, besides the ones we reported earlier in \\#1 in this response, the statistics for other benchmarks are as follows:\n \n| Metric | MDQA | RULER | SQuAD | HotpotQA | NQ  | TriviaQA | RACE |\n|--------|------|-------|-------|----------|-----|----------|------|\n| Mean   | 5150 | 7259  | 1048  | 5010     | 583 | 566      | 492  |\n| Std    | 287  | 745   | 81    | 993      | 21  | 28       | 121  |\n| Min    | 4172 | 6209  | 923   | 3587     | 536 | 529      | 122  |\n| Max    | 6755 | 8061  | 1174  | 7842     | 633 | 643      | 1323 |\n\nWe will put more dataset information in the revised version of the paper.\n\nWe hope the above response addresses your concerns, and we would appreciate it if you could consider raising your score. \n\n---\n\n***References***\n\n[1] Pouransari et al. Faster LLM Training with Variable Sequence Length Curriculum. NeurIPS 2024. \n\n[2] Zhao et al. Analysing the Impact of Sequence Composition on Language Model Pre-training. ACL 2024. \n\n[3] Shi et al. In-context Pretraining: Language Modeling Beyond Document Boundaries. ICLR 2023. \n\n[4] Yen et al. HELMET: How to Evaluate Long-context Language Models Effectively and Thoroughly. ICLR 2025."}, {"review_id": "dcrDIgEMoa", "reviewer": "Reviewer_78Fj", "summary": "In this paper, the authors conduct a comprehensive controlled study on the impact of context window size during pretraining, revealing that shorter context windows lead to better performance on standard benchmarks. This finding challenges the prevailing trend of using longer context windows for pretraining. Based on this insight, the authors propose SkyLadder, a scheduling strategy that gradually increases the context window from short to long during pretraining. SkyLadder yields substantial improvements in both downstream performance and computational efficiency.", "questions": "As mentioned in the weaknesses, I would like to ask:\n\n1. Could the authors provide a more in-depth analysis of why short-context pretraining leads to better performance?\n\n2. Why does the proposed SkyLadder method fail to improve performance on certain reading comprehension tasks, as shown in Table 2?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. Overall, I appreciate the writing quality of this paper. The plots are clear and well-organized, and the authors provide a solid background on long-context pretraining. All implementation details, ablation studies, and analyses are clearly presented and easy to follow.\n\n2. The authors conduct extensive experiments across different datasets, model sizes, hyperparameters, and settings to support their motivation. The proposed method is simple yet effective.\n\n3. The thorough experiments and analyses contribute valuable insights that could benefit future research on language model training.\n\n\n**Weaknesses:**\n\n1. While the paper is well-written and clear, the core idea is not particularly surprising. Moreover, the motivation behind the proposed method appears to be primarily result-driven, with limited theoretical or in-depth analysis. For example, it would be helpful if the authors could analyze *why* shorter contexts work better—possibly from the perspective of learning dynamics or optimization behavior.\n\n2. In Table 2, the gains from applying SkyLadder are not entirely consistent—e.g., 3 out of 5 tasks show performance drops. This weakens the claim that SkyLadder improves reading comprehension. It would be helpful if the authors could elaborate more on these results and explain the underlying factors.", "was_revised": true, "final_justification": "Thanks to the authors for the thorough analysis, which solves my concerns on (1) the novelty of idea and (2) why the proposed method fails to improve performance on certain reading comprehension tasks. Therefore, I remain positive about this paper.", "rebuttal": "Thanks for your helpful feedback. We appreciate that you found our work clear and valuable. Here are our responses to your questions: \n\n***The core idea is not particularly surprising.***\n\nOur contribution is the first large-scale, controlled study across multiple model sizes, datasets, and masking strategies, demonstrating that shorter windows consistently outperform long ones under a fixed token budget. These findings challenge the current trend of pretraining with longer context windows (Figure 1).\n\nPrevious works may apply short-to-long pretraining for efficiency reasons without showing a clearer performance gain, as discussed in Section 2. We show that proper context scheduling can improve performance on both short and long tasks, in addition to training speedup. We verified this effect across model sizes and pretraining corpora. This reveals that the schedule context length is a critical yet underexplored dimension of pretraining design. \n\n---\n\n***More analysis on why short context pretraining performs better.***\n\nShort-context pretraining provides more stable learning dynamics overall. We identify three key observations below:\n\nFirst, as increasing the context window mostly affects the attention layer, we choose it as the focus of our study. In Section 4.6 Figure 7, we plot the dynamics of attention sink and entropy during the pretraining: we observe that **SkyLadder delays the emergence of attention sink**. This is consistent with the findings of concurrent work [1] that short-context pretraining results in a lower attention sink. Overall, short-context models have more active attention heads that do not sink to the first token, leading to more capacity in processing the context.\n\nSecond, we monitor the max attention logits $S_{\\text{max}} = \\max_{i,j} q_i \\cdot k_j$ across the training steps, following a recent study of K2 [2]. A large attention logit suggests that a head is malfunctional and leads to numerical instability. We pretrained 120M models with different context lengths (each row), and get $S_{\\text{max}}$ as training progresses (each column represents a checkpoint pretrained with X tokens). \n\n| Training Context | 2B   | 5B   | 10B  | 15B  | 20B  | 30B  | 40B  |\n|-------|------|------|------|------|------|------|------|\n| 1K    | 19.2 | 28.2 | 32.3 | 33.0 | 33.9 | 34.9 | 35.3 |\n| 2K    | 23.7 | 34.8 | 37.3 | 37.7 | 37.2 | 36.5 | 36.0 |\n| 8K    | 16.3 | 36.1 | 46.8 | 49.5 | 51.2 | 52.8 | 54.0 |\n| 16K   | 21.3 | 49.8 | 56.7 | 63.5 | 69.3 | 77.7 | 84.4 |\n\nWe observe that the 16K model shows growing $S_{\\text{max}}$, and still increases steadily at the end of training. In contrast, the 1K model exhibits smaller $S_{\\text{max}}$ overall, and the value quickly saturates after 5B tokens. Thus, **short-context models give smaller max attention logits**. \n\nFinally, we directly monitor the training dynamics by investigating the training loss and gradient behavior. We compute four stability metrics for the first $N = 30K$ steps of pretraining, where $L_t$ is the training loss and $G_t$ is the gradient norm before clipping:\n\n- **Loss Volatility:** measures local fluctuations of loss over a sliding window ($w=10$), computed as $\\frac{1}{N} \\sum_{t=1}^{N} \\text{Std}(L_{t-w+1}, \\dots, L_t)$. Lower values indicate more stable training.  \n- **Loss Smoothness:** the average loss change between consecutive steps, $\\frac{1}{N-1} \\sum_{t=2}^{N} |L_t - L_{t-1}|$. Smaller values mean smoother convergence.  \n- **Mean Loss Ratio:** measures temporary increases in loss relative to the best loss so far, $\\frac{1}{N-1} \\sum_{t=2}^{N} \\frac{L_t}{\\min(L_1, \\dots, L_{t-1})}$, where smaller values indicate fewer loss spikes.  \n- **Average Gradient Norm:** $\\frac{1}{N} \\sum_{t=1}^{N} \\min(G_t, 1)$, where larger values indicate more aggressive gradient updates.\n\n| Context | Volatility ↓ (w=10) | Smoothness ↓ | Mean Loss Ratio ↓ | Avg Grad Norm ↓ |\n|---------|--------------------|--------------|-------------------|-----------------|\n| 1K      | **0.023**          | **0.019**    | **1.014**         | **0.335**       |\n| 2K      | 0.026              | 0.023        | 1.017             | 0.338           |\n| 4K      | 0.030              | 0.029        | 1.020             | 0.340           |\n| 8K      | 0.036              | 0.036        | 1.025             | 0.347           |\n| 16K     | 0.041              | 0.042        | 1.036             | 0.416           |\n\nWe find that **longer-context models exhibit higher loss volatility, less smooth loss curves, more frequent upward loss spikes, and larger gradient norms**, indicating less stable optimization dynamics during training. This suggests that **short-context pretraining is inherently more stable**, potentially providing more reliable gradient updates and facilitating better convergence.\n\nDue to the rebuttal policy, we are unable to present figures to visualize the dynamics. We will plot these interesting results in the updated version of our paper. \n\n---\n\n***Why does the proposed SkyLadder method fail to improve performance on certain reading comprehension tasks, as shown in Table 2?***\n\nWe note that in the reading comprehension setup, Random+SkyLadder performs the best across all tasks. In contrast, for the IntraDoc setup, adding SkyLadder is overall comparable to the baseline. For tasks improved by SkyLadder, the difference is large. For tasks that with a performance drop, the decrease is very small (<0.8%). Our evaluation setup mainly follows [3], where a performance fluctuation is also observed at around 1-2%.\n\nSeveral factors may explain this fluctuation:\n\n- **Few-shot prompting noise**: We prepend 2-4 demonstrations with passages and questions to each test sample. The context thus includes extra noisy information that the model needs to handle, causing performance variations.\n- **Open-ended generation**: These QA tasks require the model to output free-form text, which naturally gives more variations than multiple-choice questions. \n- **Strict exact-match metric**: The generated text is evaluated by an exact match to the answer. Small differences in phrasing or format may be marked incorrect.\n\nAll these factors may contribute to performance fluctuation. Even though  IntraDoc+SkyLadder does not seem to improve reading comprehension significantly, it still improves training efficiency and standard benchmark scores. \n\nOverall, we hope that we have addressed your concerns by carrying out the above analysis. We plan to include these results in the revised version of the paper, and would appreciate it if you could consider updating your score. \n\n----\n\n***References***\n\n[1] Barbero et al. Why do LLMs attend to the first token?. Arxiv 2504.02732\n\n[2] Kimi Team. Kimi K2: Open Agentic Intelligence. Technical Report 2025.\n\n[3] Zhao et al. Analysing The Impact of Sequence Composition on Language Model Pre-Training. ACL 2024"}, {"review_id": "poUGgxFRx4", "reviewer": "Reviewer_hz6k", "summary": "This paper uses tightly controlled experiments to demonstrate that the context length during large language model (LLM) pretraining can greatly affect the performance of the model on shorter-context tasks. More fine-grained experiments show that the document packing and masking strategies can also affect the performance. Based on these findings, the authors propose SkyLadder, a context length scheduler that dynamically adjusts the context length of the sequence during LLM pretraining by introducing \"mini\" causal masking that splits documents into several smaller segments. This scheduler improves the model performance while reducing the overall training time.", "questions": "1. The paper mentioned in Section 2 that for fair comparison, the context is limited to 512 tokens, which is the shortest context window of the tested candidate models. This seems to give an advantage to short-context windows as they were naturally trained with similar lengths. Given that RoPE, and especially NoPE can actually be extended to unseen lengths, have you ever tested the performance of these models on long-context tasks?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The controlled experiments and the analysis in Section 3 are very useful and provide insights on the design of the pretraining process. Some conclusions, such as the effect of packing/masking strategies and the experiments without positional encoding, can shed light on future practices.\n2. All the experiments, including those in Sections 3 and 4, are controlled with detailed explanations on reproduction. The conclusions are convincing.\n3. This paper provides comprehensive analysis, including the effect of context length scheduling on attention patterns, scheduling type, and training efficiency.\n4. The experiments are costly with multiple runs of the training processes. This systematic analysis done in this paper, as well as practices such as the expansion rate selection, can greatly save the efforts of future researchers.\n\n## Weaknesses\n\n1. Changing the context length is not a new idea. In past work, as early as Longformer, or many open LLMs with technical reports like DeepSeek v3, all mentioned that they would first pretrain the model on short contexts, and then gradually expand the context length till the maximum length. Though many of them did not systematically detail how the scheduling is done and the packing/masking solutions, they should not be too much different from SkyLadder.\n2. Continuing from the above point, using a baseline that is done without scheduling can be misleading, as it is not the common practice of actual LLM pretraining. In Section 4.6 the authors discussed DD, while this is not the only method. It might be better to use a more appropriate method such as 2-stage training, i.e., short-context pretraining + long-context adaptation.", "was_revised": true, "final_justification": "I will maintain my rating.\n\nThe long-context training scheduling isn't a new idea and has been explored by various research papers and technical reports. The findings mentioned in the rebuttal are not new. Here I will mention one more paper:\n\nhttps://ai.meta.com/research/publications/effective-long-context-scaling-of-foundation-models/ (published in NAACL 2024)\n\nSome of the ideas discussed by SkyLadder has already been discussed in this paper, e.g. longer isn't better, and context schedule is important. They have similar conclusions though in different settings.\n\nThe added experiments answered my question; thanks.", "rebuttal": "Thanks for your valuable feedback and for recognizing the value of our controlled experiments. Here are our responses to your questions:\n\n***Changing the context length is not a new idea.***\n\nWe fully acknowledge prior works that employ the idea of growing the context length. However, as discussed in Section 2, these approaches were introduced primarily for efficiency reasons and lacked rigorous and public evaluation of their impact on final model performance. To our knowledge, no prior work has conducted a controlled, large-scale study across multiple model sizes, datasets, and masking strategies to quantify:\n1. How pretraining context length affects downstream task performance under a fixed token budget.\n2. How different scheduling strategies compare in terms of performance.\n\nFor (1), our controlled experiments challenge the common assumption that longer context windows are better if compute permits. We show that naïve long-context pretraining does not consistently improve performance. This overturns the default “longer is better” expectation and highlights context length as a critical but underexplored dimension in pretraining design. \n\nFor (2), while existing models often extend context in multiple stages, there has been no public and systematic analysis of how the schedule pace affects outcomes. In Table 7 (illustrated in Figure 11), we show that different schedules, despite reaching the same final 32K context, can lead to significantly different performance. This shows that a systematic investigation of the schedule is necessary and deserves more open and public research. \n\n---\n\n***It might be better to use a more appropriate method such as 2-stage training, i.e., short-context pretraining + long-context adaptation***\n\nWe agree that current models often adopt a two-stage procedure: an extended period of training at a fixed mid-range length (e.g. 4K or 8K), followed by long-context continual pretraining on a much smaller token budget with gradual expansion. We therefore include a Cont. Pretrain baseline following this setup and compare it with SkyLadder.\n\nWe experimented with pretraining 1B models with 32K context. The SkyLadder schedule starts from 32 at the beginning and gradually expands towards 32K. In contrast, the Cont. Pretrain setup spends 95% of training tokens at 4K length, before expansion to 32K with the rest 5% tokens. The performances are as follows:\n\n| Model                                 | Standard Avg. | Long Avg. | MDQA | RULER |\n|--------------------------------------|----------------|-----------|------|-------|\n| Constant 32K       | 50.7           | 9.7       | 12.8 | 6.6   |\n| Cont Pretrain (4K  $\\rightarrow$ 8K  $\\rightarrow$ 16K  $\\rightarrow$ 32K) | 52.9           | 10.4      | 15.3 | 5.4   |\n| SkyLadder (32 $\\rightarrow$ 32K)       | **54.3**       | **13.5**  | **18.4** | **8.5**   |\n\nWhile two-stage training improves over the constant baseline, SkyLadder achieves the highest performance on all tasks. This demonstrates that scheduling should be planned over the entire pretraining process, not only during the adaptation stage. This controlled comparison shows that SkyLadder provides genuine gains over the commonly used heuristic approaches.\n\n---\nHere are our responses to the “Questions” Section: \n\n***Given that RoPE, and especially NoPE can actually be extended to unseen lengths, have you ever tested the performance of these models on long-context tasks?***\n\nWe evaluated whether short-context models can match long-context models when extended to 8K using RoPE adjustments or NoPE. \n\nFirst, we take RoPE-based models pretrained with 512–4K contexts and apply three common extension methods (Linear, Dynamic NTK, and YaRN) to scale the context window to 8K. The table below reports evaluation perplexity (PPL) at 8K. For reference, a model pretrained directly at 8K achieves a PPL of 8.25. \n| Pretraining Context Length | Linear   | DynamicNTK | YaRN   |\n|-----------------|----------|---------|--------|\n| 512             | 176.20   | 72.47   | 11.79  |\n| 1k              | 152.18   | 27.91   | 9.21   |\n| 2k              | 71.29    | 12.22   | 8.41   |\n| 4k              | 17.18    | 8.47    | 8.23   |\n\n**Observation**: YaRN provides the best results, but only the 4K→8K model approaches the 8K-pretrained baseline. Models trained on very short contexts (512 or 1K) perform poorly when extrapolated, showing that post-hoc scaling cannot fully substitute long-context pretraining.\n\nNext, we evaluate these extended models on the MDQA-30 document task. For reference, a model pretrained directly on 8K achieves a score of 17.7.\n\n| Pretraining Context Length | Linear | DynamicNTK | YaRN |\n|----------------------------|--------|------------|------|\n| 512                        | 0.4    | 0.8        | 5.9  |\n| 1k                         | 0.5    | 1.3        | 9.5  |\n| 2k                         | 1.0    | 4.5        | 15.4 |\n| 4k                         | 6.2    | 12.3       | 14.0 |\n\n**Observation:** Even with the strongest extrapolation techniques, short-context models underperform the 8K-pretrained model on MDQA-30. The performance gap widens as the pretraining context length decreases.\n\nFor NoPE models, we encountered training stability issues and were able to do small-scale experiments (120M parameters, with 15B tokens). Due to the limited model size, we only evaluated perplexity (PPL) on held-out validation text, with $L_e$ indicating the evaluation context window.\n\n| Pretraining Context | $L_e=512$ | $L_e=1K$ | $L_e=2K$ | $L_e=4K$  | $L_e=8K$ |\n|-------------------------|---------|--------|--------|--------|--------|\n| 512                     | 18.9    | 37.4   | 213.9  | 767.9  | 1454.1 |\n| 1K                      | 19.3    | 17.9   | 61.0   | 447.0  | 2107.3 |\n| 2K                      | 19.8    | 18.3   | 17.2   | 61.6   | 1308.5 |\n| 4K                      | 20.6    | 19.0   | 17.8   | 16.9   | 37.6   |\n| 8K                      | 21.6    | 19.8   | 18.5   | 17.5   | 16.5   |\n\n**Observation**: PPL increases sharply whenever the evaluation length exceeds the pretraining length. While NoPE models show some extrapolation ability, they still fail to match the performance of models trained natively with longer contexts.\n\nOverall, both RoPE-based extensions and NoPE provide only limited long-context capability. **Short-context models cannot fully substitute dedicated long-context pretraining**, especially on complex long-document tasks. This highlights the necessity of long-context training. Our paper thus investigates the challenges in long-context pretraining and proposes SkyLadder to improve both long and short performance. \n\nWe hope that our response has addressed your concerns, and we would greatly appreciate it if you could consider increasing the score."}]

Author rebuttal:

Meta-review: The paper analyzes the effect of context window length on LLM pretraining and, based on these observations, introduces SkyLadder, a short-to-long context window curriculum. The method improves accuracy under a fixed token budget on standard tasks, while matching or exceeding performance on long-context tasks, with similar training speed improvements as well.

While some observations (e.g., the effect of context length during training) are already known in the community, and the method itself is not highly novel (e.g., related to long-context fine-tuning), the reviewers agreed that the context length analysis is well-designed and the experiments are thorough, spanning datasets, model sizes, and hyperparameters. The motivation is clear, and the proposed method is simple and effective. A remaining concern is that experiments extend only up to 3B parameters; the rebuttal offered some arguments, based on scaling trends, that the method should generalize to larger models.

Overall, the AC agrees that the paper makes a solid empirical contribution, with carefully designed analysis and a simple method that could be of significant practical benefit to the community. The paper is therefore recommended for acceptance.
Decision: Accept (poster)