Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
Authors: Divyat Mahajan
Sachin Goyal
Badr Youbi Idrissi
Mohammad Pezeshki
Ioannis Mitliagkas
David Lopez-Paz
Kartik Ahuja
Venue: ICLR 2026 Poster (2026)
Keywords: Teacher Forcing
Multi-Token Prediction
Pretraining
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose future summary pretraining to better capture long range dependencies and enhance reasoning capabilities of LLMs

Abstract: Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitations largely attributed to teacher-forced training. Multi-token prediction (MTP) partially mitigates these issues by predicting several future tokens at once, but it mostly captures short-range dependencies and offers limited improvement. We propose future summary prediction (FSP), which trains an auxiliary head to predict a compact representation of the long-term future, preserving information relevant for long-form generations. We explore two variants of FSP: handcrafted summaries, for example, a bag of words summary of the future of the sequence, and learned summaries, which use embeddings produced by a reverse language model trained from right to left. Large-scale pretraining experiments (3B and 8B-parameter models) demonstrate that FSP provides improvements over both NTP and MTP across math, reasoning, and coding benchmarks.

Published as a conference paper at ICLR 2026
BEYOND MULTI-TOKEN PREDICTION: PRETRAINING LLMs WITH FUTURE SUMMARIES
Divyat Mahajan :. Sachin Goyal
Badr Youbi Idrissi? Mohammad Pezeshki
Toannis Mitliagkas'
David Lopez-Paz'
Kartik Ahuja
Mila, Université de Montréal
Carneiie Mellon University
*FAIR Meta
ABSTRACT
Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writ- ing. with these limitations largely attributed to teacher-forced training. Multi- token prediction (MTP) partially mitigates these issues by predicting several fu- ture tokens at once, but mostly captures short-range dependencies and offers limited improvement. We propose future summary prediction (FSP), which trains an auxiliary head to predict a compact representation of the long-term future, pre- serving information relevant for long-form generations. We explore two variants of FSP: handcrafted summariex, for example, bag of words summary of the fu- ture of the sequence, and learned summaries, which use embeddings produced by a reverse language model trained from right to left. Large-scale pretraining exper- iments (3B and 8B-parameter models) demonstrate that FSP provides improve- ments over both NTP and MTP across math, reasoning, and coding benchmarks.
INTRODUCTION
Early progress in large language models was primarily driven by massive scaling of both data and compute (Brown al. 2020: Kaplan et al. 2020). However, the returns from this vanilla scaling approach are beginning to diminish as we encounter the "data wall" (Sutskever (2024). This has renewed efforts toward algorithmic advances, including new architectures and pretraining objectives, that can extract more predictive signal from a fixed amount of training data.
Next-token prediction (NTP) with teacher forcing- training models by conditioning on ground-truth history when predicting the next token-is foundational to current pretraining methods. However, this approach introduces train-inference mismatch known exposure bias: during inference, the model must rely on its own outputs rather than the ground truth, leading to compounding errors and degraded long-range generation quality (Bengio et al., 2015). Moreover, teacher forcing can induce vaining-timm shortcut learning well, where the model exploits local cues from the ground-truth prefix instead of capturing true long-range dependencies (Bachmann & Nagarajan 2024]. These issues manifest most clearly in tasks demanding extended reasoning, narrative coherence, and open- ended creativity (Papalampidi et al. 2022 Nagarajan et al. 2025).
An appealing alternative to teacher forcing is teacherless training, where the model learns from its own generations rather than relying on ground-truth histories. However, this approach is compu- tationally intensive and challenging to parallelize. As a practical compromise, recent research has explored multi-token prediction (MTP) methods (Gloeckle et al.] (2024). which train auxiliary heads to predict several future tokens simultaneously. MTP has demonstrated success in large-scale sys- tems such as DeepSeek- V3 (Liu et al. 2024) and Qwen-3 (Yang et al. 2025). In MTP, each time step augments the standard next-token prediction (x++1) with auxiliary heads that predict additional fu- ture tokens (e.g., It+2 and beyond). However, these methods typically assume independence among the predicted tokens given the prefix, resulting in poor approximations of the true joint distribution over long future spans.
"Correspondence divyatmahajjan@maail. com. Work done during DM and SG's internship at Meta.

---

Published as a conference paper at ICLR 2026
NTP:
MTP:
St
[++
FT
Figt
F1+1
21+2
2r-
FSP-BoW:
FSP-RevLM:
Beg-of-wordss (multi-hot vector)
Hidden Representation Fleverse LM
Z-gt
Ugt
2+++1
T+++2
TT-
Figure 1: A comparison of future-aware pretraining objectives. All methods take prefix Zge as input. NTP: Only predicts the immediate next token. MTP: Uses multiple auxiliary heads, each predicting specific future token. FSP-BoW: Our proposed hand-crafted summary method that predicts a multi-hot bhag-of-tokens" summary of a long future window using single auxiliary head. ESP-RevLM: Our proposed learned summary method predicts compact hidden representation of the future, which is generated by reverse language model (RevLM).
In this work, we propose a different approach: rather than predicting multiple future tokens individ- ually, train single auxiliary head to predict summary representation of the future sequence. aims to push as much information as possible about the future into a single target vector, while filtering out information that is inherently unpredictable. At each time step f, given the future kens (Tz+2, Te+3,... xt+-r). ), we construct summary vector a(t, 7) to supervise the auxiliary head. We explore two complementary approaches for constructing future summaries: simple token-level aggregation method, and our main contribution that learns representation of the future sequence.
Hand-crafted summary. We train the auxiliary head to predict all the future tokens that will occur in future window, without requiring to know their exact positions. Concretely, at each step t. we define a multi-hot vector a(t, T) over the vocabulary, akin to bag-of-words, where a(t, r) I(i (II+2) ............. (etr }), and train the model with a binary cross-entropy objective.
Learned summary. Handcrafted summaries such as the one proposed above can be noisy as not all future tokens equally relevant. To address this, we propose learn compact summary of the future. We do this by training reverse language model (g,) on reversed sequences, so that its hidden representation att, 7) = gr (x22222) serves as rich embedding of the future context. The auxiliary head of the forward model is then trained to match this representation with an &g loss.
We use the lens of synthetic tasks, path-star graph and sibling discovery (Bachmann & Nagarajan 2024 Nagarajan et al.) 2025) to clarify the conceptual difference between MTP and our approach. On path-star graph, hand-crafted summaries deliver strong gains over MTP, which only predicts immediate future tokens, demonstrating the benefit of long-horizon supervision. However, in sibling discovery. where the future includes content unrelated to the current local sequence, handcrafted summaries struggle, they treat all future tokens equally. In contrast, our learned summary vectors focus on the informative parts of the future and achieve consistent gains.
We then scale to real-world pretraining the 8B parameter level, conducting a systematic evalua- tion across six methods, including DeepSeek-MTP and multiple handcrafted summary variants- a breadth of comparison that is rare due to its high computational cost. The proposed future summary prediction with learned summaries yields substantial improvements over NTP and MTP baselines, with gains of up to 5% on math and coding benchmarks that demand long-horizon reasoning and planning. These results demonstrate that future summary prediction is not only effective in con- trolled synthetic settings but also translates into meaningful gains for large-scale L.M training.
FUTURE-AWARE PRETRAINING
2.1 BACKGROUND ON NEXT-TOKEN AND MULTI-TOKEN PREDICTION
Let define a sequence of tokens as X X2.... Xx) sampled from a distribution Px, where each token is discrete random variable supported on {1, (1,2....,V)
2

---

Published as a conference paper at ICLR 2026
Next-token Prediction (NTP). The standard next-token prediction objective is:
LNTP (r, -ExmPx
log Po (xs))
(1)
where Pg is the predictor, optimized via
8* arg min LNTP (X, PX)
typical parameterization is:
Po(#+41 (gg)) softmax Ju In Is (a(()))
(3)
where fa is the transformer backbone, fn. is processing head, and fu is the unembedding layer (Figure 2
Multi-token Prediction (MTP). MTP aims to jointly predict multiple future tokens, as follows:
LoTP-Joime (x,,iiiiiiXXX Po) -Ex-Px
log Po Zetr rree)
(4)
Since modeling the exact joint distribution is intractable, a common simplification is to model marginal distribution of future tokens given the prefix (Gloeckle et al. 2024 Liu et al. 2024):
(X, Po) = -E Fx
ΣΣ 1[ log Pa(xe+a (gg))
(5)
Following Gloeckle et al. (2024), the predictor uses separate auxiliary heads for each k:
Po xgg))= = softmax fu fn (
(6)
Po(x=- ((g)) = softmax fu She o fa(Fst)). 4k > 1,
where fh. are auxiliary transformer blocks specialized for predicting future tokens Ti+k-
This design reduces teacher forcing by predicting Ti++ from F<t only, rather than conditioning the full prefix J5++R-1- This is the key principle behind multi-token prediction, i.c., reduced teacher forcing by requiring the model to predict block of future tokens each step.
We discuss other variants of MTP such as DeepSeek-MTP and random future token MTP (Thankara) et al. 2025) in Appendix B
2.2 FUTURE SUMMARY PREDICTION (FSP)
In MTP, we use a set of auxiliary heads to predict a block of immediate future tokens. A key limitation of this approach is that does not know exactly where the informative signal in the future sequence lies. Informative signals in future could occur far away in the sequence and be well beyond the number of future tokens (k) that MTP predicts. A trivial approach overcome this could be to predict all the future tokens. However, having one auxiliary head per future token is not scalable, and limits the amount of future tokens we can utilize during training.
Towards this, we propose Future Summary Prediction (FSP), that predicts a compact summary of the (Iong) future sequence rather than each token individually.
Let alt, T) represent summary of future tokens ... The learning objective is:
LEsP (X X, Pa) = (X, + Ex-Px La (As((zz)) alt, att, (().
where Py is the next-token predictor, As the summary predictor, and la is the loss between pre- dicted and ground-truth summaries. The architecture is:
Po(x++1 ( = softmax fu fn fo fF(s))
(8)
Ap(Igt) Sh o fa(
3

---

Published as a conference paper at ICLR 2026
Unlike MTP, FSP requires only single auxiliary head fh. making it more scalable.
Future summary att, 7)
2++1
The key question, then, is how to construct an effective future summary. In this work, we investigate two approaches:
MainBlock fu(-)
AuxBlock (.)
(1) Hand-erafted future summaries. We define binary vector a(t, T) E RV indicating whether token appears in the future window: a(4, r); = Ii E
Transformer backbone f.(-)
ft+r)).
Given logits from (z(z)) we minimize reweighted binary cross- entropy loss:
la (As((s)) = = Σ
w(i) (a, (og (000g(110))) (1-4(z))
Figure 2: An abstraction of the architecture that subsumes NTP, MTP,
1=1
(10)
where a; = T(i E (I++2..... r+++]), 2; is the i-th logit of As(age)
and FSP.
a is the sigmoid, and w(i) reflects the importance of token (e.g. term frequency-inven inverre document frequency, tf idf
(2) Learned future summaries. Predicting every future token via hand-crafted summaries as discussed above could be noisy. For exam- ple, not all tokens in the future are informative and thus hand-crafted summaries that account for all of them be wasteful. Instead of predicting all the future tokens, we therefore propose predict a learned representation of the future tokens relevant to predicting the current token 214 We learn
this representation via reverse language model Qu (RevLM), trained "right-to-left" sequences:
LRevLM (X,,LM(XCCC Qu) = -E Σ log Qu (2(42)
(11)
RevLM shares the same architecture, and we take its hidden state the summary vector:
a(t, T -) = 9h o go (120-2).
(12)
We then train A& (xgg)) to match this representation via the Eg loss:
la (Ao( a(t, - 0)) | 9h g (x20422))
(13)
Why Future Summary Prediction? The key advantage of future summary prediction is its abil- ity to reduce dependence on teacher forcing when modeling long future sequences. To intuitively measure teacher forcing, consider for each ground-truth token exposed to the model, how much in- formation is the model required to predict about unseen tokens? the model predicts more such information, then we have reduced teacher forcing. Next-token prediction (NTP) uses the highest degree of teacher forcing, since the model always conditions on ground-truth histories to predict just the token. Multi-token prediction (MTP) partially relaxes this by asking the model to pre- dict short blocks of future tokens, thereby reducing teacher forcing locally. However, MTP remains constrained by the short horizon of its predictions. In contrast, our proposed approach predicts sum- maries of long future sequences, substantially reducing teacher forcing by requiring the model to reason about rich, global properties of the target trajectory.
ANALYZING FUTURE SUMMARIES
Future summary prediction provides unified view of a broad family of pretraining objectives, cov- cring MTP and its variants that sample random future tokens (Thankaraj et al. 2025 Gerontopoulos al. 2025), and proposed bag-of-words, (FSP-BoW) and RMverseLM (ESP-RevLMM) summary prediction objectives.
To better understand how the FSP framework clarifies the tradeoffs of each objective long- planning. we consider the graph modeling benchmarks introduced in prior works. Our analysis yields following insights:
4

---

Published as a conference paper at ICLR 2026
Long future summaries matter. On the canonical path-star task (Bachmann & Nagarajan 2024). we find that MTP with short range future prediction fails to generalize, highlighting the need for auxiliary objectives that incorporate Long-range future information.
Adaptive future summaries matter. On modified sibling discovery task (Nagarajan
et al. 2025), we show that incorporating every future token with hand-crafted summaries is suboptimal, highlighting the need for learned summaries that retain key information.
3.1
LONG FUTURE SUMMARY IS IMPORTANT
G(2,6)
100%
100%
Example Graph:
100
G(2,8)
80
Accuracy (%)
65.7%
Example Sequence:
start/goal
path
60
x <graph>>
5
2
4
44.8%
47.9%
NTP:
<graph> 5
2
Difficult prediction lask
40
<graph> 5.
1
2
4
<graph>
5
<graph>>
5
2
4
Too easy due to shortcut
20
<graph>
5.
1
1
5
FSP:
NTP
MTP
FSP-BoW
<graph>
1 2
- Mix of 1- to 4-hop reasoning
Figure 3: Analysis of FSP-BoW on the path-star task, which tests long-horizon planning. Left: Accuracy (mean over 5 random seeds) of different pretraining objectives on degree 2 graphs with path lengths 6 and Standard NTP generalizes poorly, and MTP's accuracy degrades as the path length increases, while FSP-BoW achieves perfect accuracy. Right: An illustration of why NTP fails while FSP-BoW succeeds, where the input context is shown in grey and the target is in beige.
We consider the path-star graph, a directed acyclic graph (DAG) G(d, I) composed of d paths, each of length 1, originating from a central start node Ustan- The model given the adjacency list of the graph in the prefix, and the task is to generate the path from Usart to designated end node Vend-
Let the target path be (Nstur; 11; 12; ... Vend) and the input prefix be p = (Adj(G), Uaan; Mend). Then NTP with teacher forcing predicts an intermediate node in the path as Po (p. Ugs). As shown by |Bachmaan & Nagarajan (2024) NTP often learns shortcut solutions: the model can recover 4/41 directly from U; by scanning the adjacency list in p, without learning the underlying long-range plan (Figure 3) This leads to gradient starvation (Pezeshki et al. 2021). where the supervision signal for the actual planning task is lost. Once the shorteut is learned, meaningful gradient information remains only for predicting the first step vn, as it is the only difficult token.
A natural remedy is to reduce teacher foreing via future prediction approaches, which require the model to predict tokens further ahead. This makes shortcuts less effective, as multiple lookups in the adjacency list would be needed to predict future tokens. Hence, we consider the MTP approach, which predicts the immediate future token, and the handcrafted summary method FSP-BoW, which compresses the information about all the future tokens from the path. By summarizing the entire future trajectory, PSP-BoWW substantially reduces teacher forcing, encouraging the model to plan the full path instead of exploiting shortcuts.
We conduct experiments for two graphs G(2.6) and G(2, 8), with all the approaches pretrained from scratch using the GPT-Mini architecture (details in Appendix C.I). At inference, we discard the auxiliary head used in future prediction, and the task is to generate the complete path given the prefix p. The evaluation metric checks whether the generated path exactly matches the true path. Results in Figure 3 (left) show that both NTP and MTP fail to generalize (both obtained perfect training accuracy). and the accuracy of MTP degrades further for the scenario with longer path G(2, 8). Hence, predicting just the immediate future token is not enough, and FSP-BoW tackles this by efficiently compressing all the future tokens from the path, enabling it to achieve perfect accuracy in both cases. Appendix C.1 (Table 4) shows that increasing the number of auxiliary heads in MTP
5

---

Published as a conference paper at ICLR 2026
provide some improvement, but practical limits exist: even with four additional future heads MTP cannot solve the longer path graph G(2, 8).
3.2
ADAPTIVE FUTURE SUMMARY IMPORTANT
Convergence Speed Relative to NTP
Graph:
1.4
ISP Bow
12-
PSP-RoVIM
P
Pm
1.0
0.8
S
S3
SP
S
S
0.8
Example Sequence:
0.4
0.2
x
S
S
3
P-
5
S3
p
5
S
Sg
p
0.0
4
8
10
this is the prefix
Relevant future
Not predictable from the prefix
Number of Components
Figure 4: Analysis on the modified sibling discovery task which requires adaptive future summaries. Left: Convergence speed (mean (s.c.) over 3 random seeds) relative to NTP, where lower values imply faster convergence. FSP-RevL.M converges faster than NTP while FSP-BoW improves only in the cases with few components. Right: Task setup illustration- given the prefix, only future to- kens in the highlighted component are informative. FSP-BoW, which summarizes all future tokens, suffers from irrelevant information, whereas FSP-RevLM summarizes only the informative aspects.
Long handcrafted summaries often incorporate all future tokens, even though only a subset may provide meaningful supervision signals, while the rest can introduce noise. This motivates the need for an adaptive future summary. We illustrate this with modification of the sibling discovery task. Figure g (right) depicts the setup, the model must generate sequences made of concatenated, inde- pendent components, where each component lists its children nodes first, followed by their parent (e.g., nodes St. S3. S3 followed by parent Pt).
Intuitively, the causal factorization implied by the DAG (parent followed by children) implies goal-conditioned approach: by conditioning on the parent, the model can easily capture the sibling dependencies. Under NTP, the model estimates Pa (S without the parent, making sibling rela- tionships harder to learn and requiring more samples. Future prediction can help, when the model incorporate the parent information. This
predicts the parent from St. then the representation enables goal-conditioned planning, the model can predict S conditioned on both S and the parent, allowing sample efficient learning of sibling relationships (see) Nagarajan al, (2025) for details).
However, not all future tokens are equally informative. As Figure 4 illuttaatess, future tokens from a different component do not provide relevant signal for predicting SJ. Handerafted summaries that include all future tokens may therefore be affected by the irrelevant information, whereas learned summaries remain robust, as the reverse language model learns representation that emphasize only the predictive signals needed to infer the next token.
We verify this empirically by comparing FSP-BoW and FSP-RevLMM in experiments with varying number of total components. All models are pretrained from scratch with the GPT-Mini architecture (details in Appendix C.2 and the inference task requires generating a coherent sequence (siblings followed by parent for each component). At convergence, all models produce coherent sequences; to quantify the learning speedup over NTP, we report the ratio of steps to convergence relative to NTP, with lower values indicating faster learning. Since all methods eventually reach perfect consistency at convergence, time to convergence is a reliable metric for comparison. Results figure A (left) shows that FSP-BoW improves over NTP only when the number of components is small, with gains disappearing for more than six components. In contrast, FSP-RevLM consistently achieves faster convergence across all component sizes, confirming the benefit of adaptive future summaries.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 SETUP
We pretrain 3B- and 8B-parameter models on corpora of 250B and IT tokens, respectively, covering diverse domains. The majority of the data comes from DCLM-like sources and GitHub repositories, supplemented by specialized material in mathematics, programming, and related areas. Models are evaluated across a diverse set of benchmarks: ARC-Easye AaCC-ayy/Challenge (Clark et al., (2018) for general reasoning, MBPP (Austin et al. 2021) and HvmanEval+ (Liu et al. 2023) for code generation, and GSM8K (Cobbe et al. 2021) and Math-500 (Hendrycks et al.] (021) for mathematical reasoning. Details of the pretraining corpus and hyperparameters are provided in Appendix[D.I]
We first benchmark our primary method, FSP-RevLM, against the baselines: next-token prediction (NTP), standard multi-token prediction (MTP) (Gloeckle et al.) 2024), and DeepSeek-MTP (DS- MTP) (Liu et al.] (024). Further, natural way to improve MTP for longer horizons is to add multiple auxiliary heads, each predicting a token farther into the future, but this approach quickly becomes impractical. We therefore constrain both MTP and DS DS-MTP to single auxiliary head predicting the immediate future token. This design choice keeps the comparison consistent and aligned with the proposed FSP, since it uses single auxiliary head.
Building on this unified single auxiliary head framework, we conduct an analysis of how MTP can be enhanced by predicting richer future targets instead of the immediate future token. In addition to predicting the learned future summary (FSP-RevLM), we compare handcrafted future summaries over short- and long-rane long-range windows. This includes the proposed multi-hot future summary (FSP- B)W) as a contributed baseline, and a random-token summary baseline (radomly sampling token from future), in line with prior works (Thankaraj et al. 2025] Gerontopoulos et al. 2025).
Note regarding experiment design. All experiments are conducted under iso-data conditions, meaning that all the methods are trained on identical datasets. For the proposed FSP-RevLM, this implies that both the forward and reverse models are trained on the same data. In line with standard practice distillation, we do not perform iso-compute comparisons that include the teacher model's (ReverseLM) cost in the reported compute budget. In practical scenarios, the computational cost of training the teacher model is typically amortized, hence it can be treated as a one- nimetime overhead that is excluded from comparisons of student models (Gemma et al. 2024| 2025).
Note regarding FSP-RevLM implementation. In our experiments with FSP-RevLM, the reverse model is the same size as the forward model (and other baselines), and it is trained for the same number of steps. As a result, FSP-RevLMM roughly doubles the total compute cost compared to standard NTP training. While FSP-RevLM increases training FLOPs, we believe this tradeoff is reasonable in today's compute-rich, data-limited scaling regime. The field has effectively hit the data wall, whereas available compute continues to grow. Progress increasingly depends on using this growing compute-per por-token budget to extract more value from fixed datasets. In this context, methods that deliver measurable gains without requiring additional data, even higher compute cost, are valuable.
4.2 RESULTS
At the 8B scale (Table []. future-summary supervision via the ReverseLM (FSP-RevLM) consis- tently improves the performance across different evaluation tasks. On ARC-Easy, FSP-RevLM (76.6%) provides significant improvement over NTP (71.8%) and MTP (73.6%), and it also leads on ARC-Challenge and MATH. For code generation, it achieves the highest score on MBPP and ties with MTP on HumanEval+, showing that the benefits of predicting future summaries generalize across both reasoning and program synthesis tasks. GSM8K is the one task where NTP (71.6%) holds a lead, though FSP-RevLM (70.5%) still narrows the gap relative to MTP (67.8%).
At the 3B scale (Table 2). MTPPSeek-MTP is a strong baseline and obtains better performance than RevLM on most tasks, except math reasoning. More importantly, DSP-RevLMM exhibits larger relative improvements scale increases from 3B to 8B parameters, and becomes more favourable than DS-MTP. Further, note that even at the 3B scale, FSP-RevLM still beats MTP on ARC and math reasoning tasks, and performs comparably on MBPP, suggesting that even at smaller scales,
7

---

Published as a conference paper at ICLR 2026
Task
NTP
MTP
DS-MTP
FSP-RevLM
ARC-Basy
0.718 (0.000)
0.736 (0.000)
0.617 (0.003)
0.766 (0.000)
ARC-Challenge
0.531 (0.000)
0.552 (0.000)
0.426 (0.002)
0.559 (0.000)
GSM8K
0.716 (0.003)
0.678 (0.007)
0.704 (0.003)
0.705 (0.004)
MATH
0.342 (0.008)
0.309 (0.006)
0.335 (0.014)
0.351 (0.017)
MBPP
0.657 (0.004)
0.672 (0.008)
0.678 (0.006)
0.683 (0.006)
HumanEval+
0.478 (0.019)
0.541 (0.011)
0.526 (0.013)
0.541 (0.009)
Table I: Pretraining at 8B scale. We benchmark the proposed FSP-RevLM approach against NTP,
MTP, and DS-MTP. Results (mean + s.e. over 3 seeds) report pass 16 for code/math tasks and accuracy for ARC. ESP-RevLM achieves the strongest overall performance. with large gains over the baselines ARC tasks and MATH, and competitive results with (DS) MTP on code benchmarks.
Task
NTP
MTP
DS-MTP
FSP-RevLM
ARC-Easy
0.263 (0.002)
0.272 (0.005)
0.293 (0.000)
0.277 (0.000)
ARC-Challenge
0.263 (0.001)
0.245 (0.002)
0.274 (0.008)
0.255 (0.000)
GSM8K
0.410 (0.003)
0.411 (0.001)
0.417 (0.003)
0.436 (0.003)
MATH
0.213 (0.004)
0.196 (0.009)
0.201 (0.004)
0.212 (0.002)
MBPP
0.521 (0.007)
0.526 (0.004)
0.537 (0.007)
0.524 (0.001)
HumanEval+
0.301 (0.009)
0.321 (0.015)
0.348 (0.022)
0.305 (0.006)
Table 2: Pretraining at 3B scale. We benchmark the proposed FSP-RevLM approach against NTP, MTP, and DS-MTP. Results (mean + s.c. over 3 seeds) report pass @ 16 for code/math tasks and accuracy for ARC. At this smaller scale, DS-MTP is a strong overall baseline, but FSP-RevLM outperforms it math reasoning tasks, and also provides substantial gains over MTP ARC and math reasoning tasks. Further, as we scale the approaches to 8B parameters, FSP-RevLM scales more favorably than DS-MTP, overtaking on most tasks.
learning to predict future summaries may provide more effective auxiliary signal than immediate future token prediction with MTP.
4.3 ANALYZING MTP WITH DIFFERENT FUTURE SUMMARIES
Table 3
presents our analysis of different suture-summary strategies as auxiliary head targets at 8B scale. We focus on the standard MTP architecture, without comparing to DS-MTP as it modifies the input to auxiliary head, to isolate the effect of different future targets on the auxiliary head.
Our results show that random-token handcrafted future summaries perform MTP with immediate future token prediction, and performance further degrades as the future win- dow (r) increases. In contrast, the proposed multi-hot or bag-of-word. handcrafted future sum- maries yield meaningful improvements over MTP, especially on math reasoning tasks, with both shorter (r 12) and longer (r 100) future windows. For example, FSP-BoW with 7 12 achieves 33.1% on MATH (+2.2 points) and 69.9% on GSM8K (+2.1 points), while even longer windows (r = 100) further amplifies the performance on GSMSK (71.4%, +3.6 points).
than standard
Finally, our learned future summaries (FSP-RevI.M) outperform MTP across all evaluation tasks, with especially pronounced gains on math reasoning: 35.1% on MATH (+4.2 points) and 70.5% on GSM8K (+3.5 points). Further analysis (Figure(3) for these math reasoning tasks shows that learned summaries promote greater output diversity across different pass@k k settings, compared to vanilla MTP with immediate future token prediction.
In Appendix D.2 (Table 6)
we replicate these findings at 3B scale, where both handcrafted and learned future summaries improve tasks. Additional ablations explore the effects of omitting reweighting in FSP-BoW and predicting learned summaries from deeper layers of the M (ev.-.
vanilla MTP, again most prominently math reasoning
8

---

Published as a conference paper at ICLR 2026
Method
MBPP
GSM8K
MATH
HumanEval+
ARC-Challenge
ARC-Easy
MTP
0.672 (0.008)
0.678 (0.0077
0.309 (0.006)
0.541 (0.011)
0.552 (0.000)
0.736 (0.00I0)
MTP-SKip v:4
0.658 (0.006)
0.639 (0.004)
0.277 (0.020)
0.508 (0.000)
0.494 (0.003)
0.722 (0.000)
MTP-Skip 2:12
0.623 (0.002)
0.621 (0.006)
0,287 (0.018)
0.486 (0.010)
0.512 (0.000)
0.710 (0.003)
MTP-Skip 7:32
0.611 (0.008)
0.598 (0.007)
0.271 (0.005)
0.459 (0.007)
0.379 (0.000)
0.564 (0.0000
FSP-BoW 7:12
0.609 (0.005)
0.699 (0.006)
0.331 (0.016)
0.508 (0.005)
0.562 (0.000)
0.737 (0.000)
FSP-BoW v:100
0.671 (0.002)
0.714 (0.009)
0.331 (0.007)
0.500 (0.019)
0.459 (0.000)
0.662 (0.000)
FSP-RevLM
0.683 (0.006)
0.705 (0.004)
0.351 (0.017) 0.541 (0.009)
0.559 (0.000)
0.766 (0.000)
Table 3: Analysis of future-summary strategies at 8B scale. We evaluate the effect of different future-summary prediction approaches against vanilla MTP. Results (mean + s.e. over seeds) report pass @ 16 for code/math tasks and accuracy for ARC. Handcrafted multi-hot summaries (FSP- BoW) improve over standard MTP, especially on math reasoning (e.g., GSM8K and MATH), while learned summaries (ISP-RevLM) provide the largest gains across math reasoning and ARC tasks.
GSMBK
MATH
0.7
MTP
MTP
FSP-RevLM
FSP-Rev
0.3
Pass@k
Pass@K
0.5
0.2
0.3
0.1
8
16
16
K (# Attempts)
K (# Attempts)
Figure 5: Enhanced diversity through learned future summaries. We compare standard MTP, which predicts the immediate future token, with FSPM which enriches the auxiliary head tar- get using learned future summaries. FSP-RevLM substantially increases output diversity compared MTP on GSM8K and MATH benchmarks.
RELATED WORK
Beyond Multi-token Prediction. Multi-token Prediction (MTP) extends Next-token Prediction (NTP) by introducing auxiliary heads to predict future tokens (Gloeckle et al. 2024), thereby pro- viding richer learning signal. Variants such as DeepSeek-MTI (Liu et al. 2024h, joint-token pre- diction (Ahn et al. 2025), and next-latent prediction (Teoh et al. 2025) differ in how they structure these auxiliary predictions, but they retain common limitation: scaling auxiliary heads to capture long-range dependencies becomes impractical. More recent approaches attempt to improve effi- ciency by predicting randomly selected or non-sequential future tokens (Gerontopoulos al. 20225] Thankaraj et al., 20255 Liu et al. (2025). However, heuristically sampling future tokens risks missing the most informative long-range signals. Our approach (FSP-ReyLM) addresses this by predicting a learned summary of the future, such that it can extract meaningful long-range information.
Realted the proposed hand-crafted summary approach (FSP-BoW), (Yin et al., 2024) and Fry- denlund (2025) incorporate objectives that predict an unordered set-level summary of future tokens, conceptually similar to our FSP-BoW and its binary cross-entropy formulation. However, our for- mulation is more general as we allow re-weighting schemes such as tf-idf scores within the BCE objective and is implemented via auxiliary heads to be consistent with the multi-token prediction architecture (Gloeckle et al.) 2024). More importantly, our contribution is not limited to a specific BoW variant: we introduce a unified framework for future summary prediction that clarifies the trade-offs between different forms of future supervision. For instance, BoW summary directly resolves the path-star star graph failure by aggregating the long-range information, yet it remains sub- optimal when only part of the future is relevant, observed in our sibling discovery experiments.

---

Published as a conference paper at ICLR 2026
These insights motivate moving beyond hand-crafted summaries toward learned future summary via reverse LM, which adaptively captures the most informative aspects of the future.
Finally, regarding SemFormer (Yin et al. 2024), beyond its BoW-style supervision, its central idea is to introduce special planning tokens that are trained to predict a latent embedding of the future sequence, encouraging planning. While conceptually related, it applies future supervision only designated planning tokens and relies on auto-encoding objective to learn future embeddings. In contrast, our approach requires no special planning tokens, enforces future-summary alignment at every position in the sequence through auxiliary heads, and, learns future embeddings via the Re- verseLM. yielding a fundamentally different supervision signal.
Leveraging the right-to-left signal. Leveraging reverse, or "right-to-left", order during training has shown empirical benefits across multiple learning paradigms. The Belief State Transformer (BST) (Hu et al. 2024) employs dual forward and backward encoders to predict both the next token after a prefix and the previous token before a suffix. This bidirectional training encourages the model to form compact belief state, though it does not explicitly address teacher forcing. In contrast, our FSP-RevI.M incorporates a reverse model with a different objective: mitigating dependence on teacher forcing. While trained standard left-to-right sequences, FSP-RevI.M aligns the forward model's embeddings with those of a reverse model, effectively distilling the reverse order into the forward language model.
The most closely related work aimed at reducing teacher forcing is Twin Networks (Serdyuk et al,, 2017), which trains reverse RNN and matches the forward hidden states those of the reverse model to encourage long-range future dependence. While similar at a high level, our contributions go beyond the specific ESP-RevLM mechanism: we present broader perspective in which future summary prediction serves framework for understanding and designing pretraining objectives, together with evidence showing when simpler approaches fail and why learned summary coupled with a reverse LM becomes necessary. Moreover, scaling this idea to Transformers is non-trivial. Just as TwinNet anticipated aspects of our reverse component, earlier work also explored multiple future tokens parallel token blocks prediction from given prefix (Tschannen et al. 2023 Monea et al. 2023) well before Gloeckle et al. (2024), though without establishing MTP as a broadly effective objective for large-scale' Transformers, Glocckle et al. (2024) deserve credit for identifying a formulation that works in modern LLM training, and our results extend this line by showing that both MTP and future-summary prediction can be cleanly integrated into Transformer pretraining and scaled to yield robust gains.
Another related approach, Meet-in-the-Middle (MiM) (Nguyen al. 2023). jointly trains forward and backward models with shared parameters and employs an agreement regularizer to align their output distributions. Our method differs in two key aspects: (1) we perform distillation with the reverse model rather than parameter sharing, and (2) we avoid the somewhat impractical assumption required by MiM that the forward and reverse output distributions must match exactly.
CONCLUSION
In this work, we highlighted a key limitation of existing multi-token prediction methods: the diffi- culty of scaling auxiliary heads for long-horizon future prediction. Towards this, we proposed Future Summary Prediction (FSP), a novel pretraining framework that shifts the auxiliary objective from predicting specific future tokens to predicting a (learned) summary of the future. Experiments on 8B models, using both hand-crafted and learned summaries, demonstrate that FSP delivers a stronger, more robust training signal, improving over NTP and MTP challenging reasoning and coding tasks. Our findings indicate that focusing on abstract, predictable aspects of the future is a promis- ing strategy for designing more efficient and effective pretraining objectives for next-generation large language models. Future work can focus on designing computationally efficient approaches for learning future summaries (embeddings), potentially without training a reverse model.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors thank Sachin Mehta for suggestions on the real-world pretraining experimental setup, and thank Vaishnavh Nagarajan and Nanda H Krishna for their detailed feedback on the draft. The authors also gratefully acknowledge helpful discussions with Aniket Didolkar, Andrei Nicolicioiu, Mathurin Videau, Sarthak Mittal, Sharut Gupta, Vineet Jain, and Moksh Jain. This research was fully funded by Meta through Meta's 's AI Mentorship (AIM) program with Mila. loannis Mitliagkas in his role as Divyat Mahajan's academic advisor, acknowledges support by NSERC Discovery grant 20011N-221--065111 and a Canada CIFAR AI chair. Divyat Mahajan acknowledges support via FRONT doctoral scholarship (https://doi... www.ooo...gg//oooooooooo org/ 10. 69777/354785)) for his graduate studies.
REFERENCES
Kwangjun Ahn, Alex Lamb, and John Langford. Efficient joint prediction of multiple future tokens. arXiv preprint arXiv:2503.21801 2025.
Zeyuan Alu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint Stt::2309.133166 2023.
Kushal Arora, Layla EI Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of eπror accumulation in language generation. In Findings of the Association for Computational Linguistics: ACI. 2022, pp. 700-710, 2022.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint (tt::2108.07332, 2021.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In International Conference on Machine Learning, pp. 2296-2318 PMLR, 2024.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Lukas Berglund, Meg Tong. Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Kor- bak, and Owain Evans. The reversal curse: Lims trained on" is b" fail learn" is a". arXiv preprint 2rXiv:2309.12200 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. httLhttps://rroceeett www.diinnnnnnnnnnnnneeeeeeee www.nneurii.... cc/ wwwppppfileeeee/ppppppppppaeeeep /il.le//ppperr/20///lleeee pdf
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 0544500..55.,... 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.11668 2021.
11

---

Published as a conference paper at ICLR 2026
Arvid Frydenlund. Language models, graph searching, and supervision adulteration: When more supervision less and how to make more more. In Proceedings of the 63nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. pp. 29011-29059, 2025.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima. Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2/01.0000 101.00027, 2020.
Team Gemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint 2rXiv:2408.001 2024.
Team Gemma, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint (.XXi::2533.197866 2025.
Anastasios Gerontopoulos, Spyros Gidaris, and Nikos Komodakis. Multi-token prediction needs registers, 2025. URL
10518
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Proceedingy of the 41 International Conference on Machine Learning, pp. 15706-15734. 2024.
Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403. 13799, 2024.
Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. 7-forcing: Training stochastic recurrent networks. Advances in neural information processing systems, 30, 2017.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Edward Hu, Kwangjun Ahn, Qinghua Liu, Haoran Xu, Manan Tomar, Ada Langford, Dinesh Jayaraman, Alex Lamb, and John Langford. The belief state transformer. arXiv preprint arXiv:2410.23506 2024.
Jared Kaplan, Sam McCandlish,
Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language mod- els. ArXiv, abs/2001.055 08361,, 2020. URL https://api. http::::/pp..sematttessteee semanticscholar www.ccoooaa.r..ooorrroueee org/CorpssII:
210861095
Jeffrey Li, Alex Fang, et al. Datacomp-Im: search of the next generation of training sets for language models, 2025.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation Learning to solve and explain algebraic word problems, 2017. URL https:
//arxiv. wttrxiv.org/bbs////. 04146
Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558-21572 2023.
Xiaohao Liu, Xiaobo Xia, Weixiang Zhao, Manyi Zhang, Xianzhi Yu, Xiu Su. Shuo Yang, See- Kiong Ng. and Tat-Seng Chua. L-mtp: Leap multi-token prediction beyond adjacent context for large language models. arXiv preprint ar0iv:2505. 77505, 2025.
12

---

Published as a conference paper at ICLR 2026
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the
finest collection of educational content, 2024. URL
Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13588 2023.
Vaishnavh Nagarajan, Chen Henry Wu, Charles Ding, and Aditi Raghunathan. Roll the dice & look before you leap: Going beyond the creative limits next-token prediction. Forty-second International Conference Machine Learning. PMLR, 2025.
neogithub. Github code dataset, 2022.
gode
Anh Nguyen, Nikos Karampatziakis, and Weizhu Chen. Meet in the middle: A new pre-training paradigm. Advances in Neural Information Processing Systems, 36:5079-5091, 2023.
Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poz- nanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lam- bert, Pete Walsh, Pradeep Dasigi. Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Victoria Graf, Akari Asai, Akshita Bhagia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasi- ades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Jiacheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerquin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjonsberg. Shannon Ze- jiang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Merrill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer. Pang Wei Koh. Ali Farhadi. Noah A. Smith, and Hannaneh Hajishirzi. Olmo 3, 2025. URL htt.:tppp/arrrii.oo//aa//2222222 13961
Pinelopi Papalampidi, Kris Cao, and Tomas Kocisky. Towards coherent and consistent use of entities in narrative generation. In International Conference Machine Learning, pp. 17278 17294. PMLR, 2022.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems, 34:1255-1222 1272, 2021.
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou, Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. arXiv preprint arXiv:2001 1000:2000.000633, 2020.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities of neural models, 2019, URL httpss//arxxi..rr///////90... .1557
Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. Twin networks: Matching the future for sequence generation. arXiv preprint arXiv: 2017.
Ilya Sutskever. Sequence to sequence Icarning with neural networks: What decade. NeurlPS 2024 Test-of-Time Talk (YouTube video), 2024. URL httpps:/www.yutteee cot.bbtcchcc////tc//c/c TyBBqasHLZs Presented at NeurlPS 2024, Vancouver, Canada.
Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward $ Hu, Pratyusha Sharma, Riashat Islam, Alex Lamb, and John Langford. Next-latent prediction transformers learn compact world models. arXiv preprint arXiv:2511.05%6 1059663, 2025.
Abitha Thankaraj, Yiding Jiang. J. Zico Kolter, and Yonatan Bisk. Looking beyond the next token, 2025. Utt.https://arxiv httgg//bb//2..... 11336
Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances Neural Information Processing Systems, 36:46830-46855 2023.
13

---

Published as a conference paper at ICLR 2026
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint WrXX::25055009388 2025.
Yongjing Yin, Junran Ding, Kai Song, and Yue Zhang. Semformer: Transformer language models with semantic planning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18669-18680, 2024.
Weizhe Yuan, Jane Yu, Song Jiang, Karthik Padthe, Yang Li, Dong Wang, Ilia Kulikov, Kyunghyun Cho, Yuandong Tian, Jason E Weston, and Xian Li. Naturalreasoning Reasoning in the wild with 2.8m challenging questions, 2025. URL httpt://///xxi wwwprrwww...rgaaaeeeeeeeeeeee 2tt.b//2250211131 13124
14

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORKS
Pitfalls of Next-token Prediction. While next-token prediction (NTP) is the standard loss for modern LLMs, its shortcomings are increasingly evident. A core issue is the train- test mismatch: training conditions on ground-truth tokens (teacher forcing), while inference conditions the model's own generations (autoregression), leading to exposure bias and compounding errors over long horizons (Bengio et al., 2015 Arora al. 2022). Beyond the auto-regresive error accumu- lation, teacher forcing may encourage learning spurious correlations or "shortcuts" rather than the relevant learning signal. [achhmann & Nagarajan (2024) describe this as the "Clever Hans cheat," where the model exploits easy local cues in the prefix that correlate with the target, thereby absorb- ing gradient signal that would otherwise teach genuinely lookahead -eeendet structure in tasks such as their path-star graph problem. Nagarajan et al. (2025) further argue that NTP is myopic and data-ineffie cuta-inefficient on tasks requiring an implicit long-range "leap of thought" (egg. Sibling Discovery) such as their sibling díscovery task. Our approach tackles these failures by providing a more robust, long-range training signal that discourages such shortcuts.
Beyond Multi-token Prediction. Multi-token Prediction (MTP) addresses limitations of Next Token Prediction (NTP) by using auxiliary heads predict future tokens (Gloeckle et al. 2024). Within this MTP family, DeepSeek MTP (Liu et al.] (024) uses recursive auxiliary design: when predicting a future token, each auxiliary head conditions on the backhone state and the intermediate representations produced by earlier auxiliary heads. Closely related, Joint Token Prediction (Ahn) jet al. 2025) keeps a similar slight-teacher-foreie
flavor but predicts all future tokens in paral- lel, while Next Latent Prediction (Teoh et al. 2025) replaces discrete future-token targets with a representation-spet states. But the key issue with all these approaches is that scaling the auxiliary heads for long- raage dependencies is impractical. Recent work aims to improve efficiency and capture longer depen- dencies by radomly predicting information from the future sequence. (Geonnoooouoss al. (2025) introduce register tokens, predicting tokens k steps ahead without architectural changes. Thhankaraj al. (2025) insert lookahead tokens containing future subsequences, while Liu et al.) (2025) use leap-based strategy to predict non-sequential future tokens. However, heuristically sampling ran- dom future tokens still poses the risk of missing informative long-range signals. Our PSP-RevLM addresses this by predicting a learned summary of the future, such that it can extract meaningful long-range information.
alignment objective that matches the current hidden state to future hidden
A relevant predecessor from the RNN literature that in principle avoids scaling auxiliary heads is ProphetNet (Qi et al., 2020), which uses shared self-attention mechanism together with relative positional encodings to predict future n-grams without introducing separate heads. While this may seem related to our bag-of-words summary (FSP-BoW), ProphetNet still supervises each future token individually and preserves positional order. In contrast, FSP-BoW operates on summary vector that discards positional information, and applies a binary cross-entropy objective over an unordered future-token summary, yielding fundamentally different training signal.
Further, (Yin et al., 2024) and Frydenlund (2025) incorporate objectives that predict an unordered set-level summary of future tokens, conceptually similar our FSP-BoW and its binary cross- entropy formulation. However, our formulation is more general as we allow re-weighting schemes such tf-idf scores within the BCE objective and is implemented via auxiliary heads be consis- tent with the multi-token prediction architecture (Gloeckle et al. 2024), rather than as a standalone modification. More importantly, our contribution is not limited specific BoW variant: we in- troduce a unified framework for future summary prediction that clarifies the trade-offs between different forms of future supervision. For instance, a BoW summary directly resolves the path- star graph failure by aggregating the long-range information, yet it remains suboptimal when only part of the future is relevant, as observed in our Sibling Discovery experiments. These insights motivate moving beyond hand-crafted summaries toward learned future summary via reverse LM, which adaptively captures the most informative aspects of the future.
Finally, regarding SemFormer (Yin et al. 2024), beyond its BoW-style supervision, its central idea is to introduce special planning tokens that are trained to predict a latent embedding of the future sequence, encouraging planning. While this shares the high-level goal of learning future rep- resentation as in our approach FevM there are several differences. SemFormer applies the embedding-matching loss only at designated planning tokens (typically once per sequence). limiting
15

---

Published as a conference paper at ICLR 2026
where future information is explicitly enforced. In contrast, approach introduces planning tokens and enforces the future embedding matching loss at every position in the sequence via the auxiliary prediction head, which enables us to employ an architecture that is fully aligned with the multi-token prediction approach (Gloeckle et al. 2024). Moreover, SemFormer learns future emheddings via the auto-encoding objective (left-to-right order of future sequence), whereas our FSP-RevLM computes the summary using reverse LM (right-to-left order of future sequence), resulting in fundamentally different supervision signal.
Leveraging the right-to-left signal. Leveraging reverse, or "right-to-left", order during training has shown empirical benefits across multiple learning paradigms. Reverse Training (Golovneva al. 2024) augments the dataset with reversed sequences to teach bidirectional dependencies and mitigate the Reversal Curse (Berglund et al.) 2023). The Belief State Transformer (BST) (Hu et al, 2024) employs dual forward and backward encoders predict both the next token after a prefix and the previous token before a suffix. This bidirectional training encourages the model to form a com- pact belief state, though it does not explicitly address teacher forcing. In contrast, our FSP-RevLM incorporates a reverse model with a different objective: mitigating dependence on teacher forcing. While trained standard left-to-right sequences, FSP-RevL.M aligns the forward model's embed- dings with those of reverse model. effectively distilling the reverse order into the forward language model. Another related approach, teee-in-the-Middll (MiM) (Nguyen al. 2023). jointly trains forward and backward models with shared parameters and employs an agreement regularizer to align their output distributions. Our method differs in two key aspects: (1) we perform distillation with the reverse model rather than parameter sharing, and (2) we avoid the somewhat impractical assumption required by MiM that the forward and reverse output distributions must match exactly.
The most closely related work aimed at reducing teacher forcing is Twin Networks (Serdyuk et al.) 2017). which trains a reverse RNN and matches the forward hidden states to those of the reverse model to encourage long-range future dependence. While similar at high level, contributions go beyond the specific ReP-ReLLM mechanism: we present a broader perspective in which future summary prediction serves framework for understanding and designing pretraining objectives, together with evidence showing when simpler approaches fail and why learned summary coupled with a reverse LM becomes necessary. Moreover, scaling this idea Transformers is non-trivial. Just as TwinNet anticipated aspects of our reverse component, earlier work also explored multiple future tokens or parallel token blocks prediction from given prefix (Tschannen et al., 2023 Monea et al. 2023) well before Gloeckle et al. (2024), though without establishing MTP as broadly effective objective for large-scale Transformers. Glosekle et al. (2024) deserve credit for identifying formulation that works in modern LLM training, and our results extend this line by showing that both MTP and future-sumin summrry prediction can be cleanly integrated into Transformer pretraining and scaled to yield robust gains.
Another somewhat related line is z-forcing (Goyal et al. 2017), which uses a reverse RNN to infer latent variables that are then injected into the forward RNN to compute its hidden states. In contrast, our reverse LM is not performing latent-variable inference; it is used solely to generate targets for an auxiliary prediction head. Furthermore, unlike Z- Z-forcing, model at inference the reverse LM is purely a training-time component, whereas Z-forcing depends the reverse RNN during inference well.
approach does not require the reverse
16

---

Published as a conference paper at ICLR 2026
B DETAILS ON MULTI-TOKEN PREDICTION VARIANTS
In Section 2 we discussed MTP and the proposed future summary prediction to address a major limitation in MTP. We summarize other variants proposed in the literature and their limitations here.
Incorporating slight teacher forcing in MTP. Note that in the standard MTP we did not allow for any teacher forcing from the "auxiliary" tokens, i.e, the auxiliary head predicted future tokens Iz+k given only the prefix zgg.- This was modified by popular variant proposed by DeepSeek (Liu et al,) 2024p. where condition the auxiliary head Sh, with full prefix -- predict the future kens k, but with reduced teacher forcing from the "auxiliary" tokens (1++1, (5(+4-1) as well.
r
LDS-MTP (X) 1:000-1Eppp Ex
ΣΣ
1t+ k s T] log Po (
(14)
The parameterization is:
Poleet1 (xg)) = softmax fu fa o o (2
(15)
Po(xet (ggeh-1)) = softmax fu She (fo (tt. (.tt,.,,.....))))
4k:
Here, the auxiliary (future) tokens (Ie+ bypassing the backbone, which reduces teacher forcing compared to standard NTP.
are injected directly into the auxiliary heads,
A notable architectural feature of DS-MTP is its recursive nature: the auxiliary head fh takes as input not only the backbone representation (*(g)) but also the representations produced by the previous auxiliary heads (fr,.... S when predicting the future token ø4. creating chain of dependent predictions across depths.
Joint-token prediction (Ahn et al., 2025) shares a similar flavor of slight teacher forcing but com- putes all future token predictions in parallel rather than recursively (same parametrization as 15), which the authors argue encourages the model to maintain a belief state over future tokens jointly. Another related variant, Next Latent Prediction (Teoh et al. 2025), also incorporates slight teacher forcing in a similar spirit, but differs in its objective: rather than predicting discrete future tokens, supervises the model's current hidden state match the hidden states the model would produce at future positions, using self-supervised alignment loss operating purely in representation space.
However, all these approach suffers from the short-horizon limitation as MTP: predicting multiple future steps would require adding separate auxiliary heads for each token, which quickly becomes impractical and limits scalability long future sequences.
Randomly sampled future information. In this work, emphasized that standard MTP and DS- MTP often fail to capture the most informative future tokens, as both primarily focus on predicting the immediate next tokens via the auxiliary head. The TRELAWNEY approach of Thankaraj et al. (2025) adopts a complementary strategy: rather than altering the model architecture, it augments the training data by inserting window of future tokens, bounded by special markers. This encourages the model to predict a randomly sampled block of future tokens, thereby reducing reliance on teacher forcing. Moreover, TRELAWNEY allows random subsequences to be sampled from the future, with the goal of exposing the model to a richer and more informative signal than what is available in the immediate next few steps. In similar spirit, MuToR (Gerontopoulos et al. 2025) interleaves learnable register tokens into the input sequence, each tasked with predicting future token at randomly sampled offset, and L-MTP (Liu al.) 2025) introdcues a leap-based mechanism that skips over adjacent tokens and instead predicts non-sequential future tokens in single forward pass, aiming to capture longer-range dependencies while also enabling inference acceleration.
However, all these approaches based on heuristic random sampling can still fail to capture critical information from tokens far in the future, limiting its robustness for long-horizon planning tasks.
17

---

Published as a conference paper at ICLR 2026
C SYNTHETIC EXPERIMENTS
C.I PATH-STAR GRAPH
Experiment Setup. We adopt the dataset generation procedure from the official code repository! of Bachmann & Nagarajan (2024). Specifically, we construct set of 50 distinct nodes and generate
multiple instances of path-star graph sequences by randomly sampling nodes from this set. The training set consists of Tlrain 200,000 sequences, and evaluation is performed on Rest = 20,000 sequences.
For modeling, we employ the GPT-Mini architecture with the following configuration:
Total Layers: 12
Embedding Dimension: 384
Total Attention Heads: 6
MLP expansion factor:
We summarize the other relevant hyperparameters below.
Learning Rate: 3e 4
Batch Size: 256
Weight Decay: le - 2
Total Epochs: 500
Gradient Norm Clipping: 1.0
Experiment with multiple auxiliary heads in MTP. A naive approach to incorporate longer- term future dependencies in MTP is by increasing the number auxiliary heads. Table[4reportt the results: adding more auxiliary heads improves performance on G(2, 6). due reduced teacher forc- ing. However, simply scaling the number of auxiliary heads is not a practical solution for modeling long-range dependencies, shown by the poor performance on the longer path graph G(2, 8).
Method
G(2,6)
G(2,8)
NTP
0.45 (0.01)
0.45 (0.03)
MTР
0.66 (0.17)
0.48 (0.03)
MTP (aux heads: 2)
0.89 (0.09)
0.46 (0.01)
MTP (aux heads: 3)
0.98 (0.01)
0.47 (0.06)
MTP (aux heads: 4)
0.97 (0.01)
0.48 (0.12)
FSP-BoW
1.00 (0.00)
1.00 (0.00)
Table 4: Analyzing the performance of MTP path-star graphs with an increasing number of aux- iliary heads. Augmenting MTP with additional auxiliary heads improves performance on G(2, 6), but practical limits exist: even with 4 aux heads, MTP fails to solve the longer path graph G(2, 8).
C.2 SIBLING DISCOVERY
Experiment Setup. We build on the dataset generation procedure from the official code reposi- tory of Nagarajan al. 22025). While the original sibling discovery task presents sequences from single component, we modify the setup by concatenating sequences from multiple components. Each component defined by unique parent pi and three child nodes S S St. The support of each conditional distribution P(S* P*) is disjoint across children -.ere each child S can take N possible values, and these sets of values are non-overlapping across children. As a result, with K components, the total number of distinct child value combinations is 23K In our experiments, vary the number components E 2, 4, 6, 8, 10 while fixing = 100. For training, we randomly
18

---

Published as a conference paper at ICLR 2026
Attention Weights Layer 11
00.00
R2 m0
03.30
181220
(8,350
220
0.2
0.3
Alleniion Widgle
Figure 6: Each subplot corresponds to different attention head. Red squares highlight queries belonging to a particular component, showing that they attend predominantly to keys within the same component, forming a subgroup- wise pattern. While the average across heads shows some cross-component diffusion, several heads exhibit clean intra-componet attention, indicating that the reverse model preserves the structural organization of the future input rather than mixing information across unrelated parts of the sequence.
sample child values for each component to construct Thrain 20,000 sequences, and evaluate on Thest = 3,000 sequences.
We use the GPT-Mini architecture with the following specifications.
Total Layers: 12
Embedding Dimension: 384
Total Attention Heads: 6
MLP expansion factor:
We summarize the other relevant hyperparameters below.
Learning Rate: 3e 4
Batch Size: 256
Weight Decay: le 2
Total Epochs: 150
Additional Experiment: Interpretability of attention weights from RevLM. Interpreting the learned future summaries is crucial for understanding how RevLM captures information from future context. For the sibling-discovery experiment (Section 3.2), conducted preliminary analysis using attention weights from the final layer, which directly contribute to the future-summary repre- sentation. We observe that queries associated with particular component the sequence attend predominantly to keys from the same component, indicating limited cross-component interaction. While the average across heads shows some diffusion of attention beyond component boundaries. three individual heads exhibit clear subgroup-wy that the learned future summaries preserve the informative structure of the future input by prioritiz- ing intra-component tokens. Prior work shows that hidden states of LLMss naturally encode detailed information about surrounding tokens (see Section Physics of L.LMs (Allen-Zhu Lij (023)), supporting the intuition that reverse-direction representations patterns. Extending this analysis with additional interpretability tools, such as linear probing and activation patching, remains promising direction for future work.
structure without such leakage. This suggests
capture meaningful long- rangerange
19

---

Published as a conference paper at ICLR 2026
D REAL WORLD PRETRAINING
D.1 EXPERIMENTS SETUP
Pretraining dataset composition. Our pretraining corpus is constructed cover a wide range of domains, aiming to equip models with both broad knowledge and strong reasoning ability. The bulk of the data drawn from a DCLM-style mixture (Li et al., 2025) and large-scale code sources such as GitHub (neogithub) 2022). To strengthen performance we further supplement with mathematics and scientific data, including the DeepMind Mathemat- ics dataset (Saxton et al. 2019)), Proof Pile 2 (ArXiv, Open WebMath, Algebraic Stack) (Azerbayev) ctt al. 2.23). and Stack Exchange from the Pile (Gao et al. 2020). Additional curated resources in- clude Fine Web-Edu (Lozhkov al., 2024), the Natural Reasoning Dataset (Yuan et al. 2025). and AQuA (Ling et al. 2017). Together, this mixture balances large-scale general text with carefully chosen reasoning-focused datasets.
more specialized skills,
Model Architecture. Both the 3B and 8B models follow the LLaMA 3 architecture. The 8B configuration aligns with the official LLaMA setup, while the 3B model scaled-down variant that preserves architectural ratios.
Parameter
3B Model
8B Model
Hidden dimension (d)
2432
4096
Layers (L)
26
32
Attention heads
19
32
KV heads
(GQA)
FFN expansion
4x
4x
FFN multiplier
1.3
1.3
Sequence length
2048
8192
RoPE 0
10,000
500,000
Table 5: Architectural configurations for the 3B and 8B models.
Grouped Query Attention (8B only). The 8B model uses Grouped Query Attention (32 query heads, 8 KV heads; 4:1 ratio), reducing KV-cache memory by 4x during inference and enabling longer context lengths without proportional memory growth.
Extended RoPE. The 8B model uses rope_ toee_theta50000000 to support 8K context length, following LLaMA 3's long-context scaling strategy. The 3B model uses the standard 0 10.000.
SwiGLU
Networks. Feed-forward blocks use SwiGLU activations with 4x expansion and 1.3 multiplier to preserve effective hidden dimensionality under gating.
RMSNorm (Pre-Norm). We adopt pre-norm RMSNorm (e 10-5 5) for improved stability and computational efficiency compared to LayerNorm.
Depth-Scaled Initialization. Parameters are initialized using depth- scaling propor- tional to 1/V2T., improving stability for deep networks.
Attention Implementation. Attention is implemented using PyTorch Sealed Dot-Product Attention (SDPA), which dispatches to optimized kernels (c.g.. FlashAttention-22) depend- ing on hardware.
Mixed Precision and Parallelism. Training uses BF16 precision for numerical stability and memory efficiency. We employ Fully Sharded Data Parallel (FSDP), which shards parameters, gradients, and optimizer states across GPUs to enable large-scale training.
20

---

Published as a conference paper at ICLR 2026
Hyperparameters. All models were trained on NVIDIA H200 GPUs using fully sharded data parallelism (FSDP) for memory-efficien scaling. We use a cosine learning rate schedule with linear warmup at the start of training, followed by smooth cosine decay to zero. The initial learning rate set to 3 x 10-3 for 3B models and 3 x 10-1 for 8B models, reflecting standard scaling practices for larger model sizes. Unless otherwise specified, models trained with BF16 mixed precision, gradient clipping for stability, and Adam W optimization. Main hyperparameters are listed below.
* 3B Models
Batch Size (per GPU): 16
Total GPUs: 128
Sequence Length: 2048
Total Training Steps: 60k
8B Models
Batch Size (per GPU): 2
Total GPUs: 256
Sequence Length: 8192
Total Training Steps: 240k
Note that these choices lead to total tokens seen during training as 250B tokens for the 3B parameter model, and IT tokens for the 8B parameter model.
Evaluation Metries. We evaluate all tasks using the pass @ metric. For multiple-choice bench- marks such as ARC (Easy and Challenge), we report pass@1 which equivalent standard accu- racy, For mathematical reasoning and code generation benchmarks, we report pass @k with A > 1, measuring whether least of independently sampled generations solves the problem correctly. All results are reported as the mean and standard eтror over 3 independent random seeds.
generate samples across range of decoding temperatures (0, 0.1, 0.2..... 0.9, 1.0) while keeping other decoding hyperparameters fixed. For each method and task, we report the performance corresponding to the temperature that yields the highest pass k. This ensures a fair comparison, as different training objectives may prefer different temperatures.
For @ass@k evaluation,
Importantly, auxiliary prediction heads are used only during training to shape representation learn- ing. At test time, all models are evaluated exclusively using their standard next-token prediction heads, ensuring that improvements stem from improved pretraining dynamics rather than additional inference- pecrencee-imeeeee components.
21

---

Published as a conference paper at ICLR 2026
D.2 ADDITIONAL RESULTS
D.2.1 ANALYSIS OF FUTURE SUMMARIES (3B).
Table O presents our analysis of different future-summary strategies as auxiliary head targets at the 3B seale. Similar to the 8B case, we focus on the standard MTP architecture, without comparing to DS-MTP since it modifies the input to the auxiliary head, allowing us to isolate the effect of different future targets. Our findings are similar: random-skip handcrafted summaries underperform relative words approaches perform well. In our experiments,
to MTP, while the proposed we observed consistent improvement over MTP with the proposed handerafted summaries (FSP- BoW) on both Math and GSM8K tasks, across the 3B and 8B model scales. For the 3B ablation, tested removing tf-idf reweighting as well and found that, in most cases, did not provide benefits, which led us keep the tf-idf reweighting for our 8B experiments. The only exceptions where no tf- idf reweighting performed better were GSM8K at T = 12 and HumanEval+ at T 10. Additionally, experimented with using deeper layers at the 3B scale and found that the last layer performed better on ARC Challenge and ARC Easy, though it was slightly worse on MBPP and Math. Based on these findings, we chose to use the last layer for our 8B- experiments.
Method
MBPP
GSMBK
MATH
HumanBval+
ARC-Challenge
ARC-Easy
MTP
0.526 (0.004)
0.111 (0.001)
0.196 (0.009)
0.321 (0.015)
0,245 (0.002)
0.272 (0.005)
MTP-Skip m:
0.519 (0.004)
0.368 (0.007)
0.181 (0.005)
0.309 (0.004)
0.241 (0.900)
0.282 (0.000)
MTP-Skip 7:12
0.485 (0.011)
0.354 (0.006)
0.177 (0.006)
0.287 (0.009)
0.264 (0.008)
0.262 (0.000)
MTP-Skip 7:
0.467 (0.007)
0.354 (0.008)
0.189 (0.004)
0.278 (0.017)
0.243 (0.000)
0.240 (0.003)
JSP-BoW-an #:12, no if-idf
0.518 (0.006)
0.431 (0.004)
0.201 (0.009)
0.301 (0.009)
0.250 (0.000)
0.251 (0.000)
FSP-BoW 7:12
0.521 (0.005)
0.419 (0.008)
0.204 (0.002)
0.305 (0.012)
0,251 (0.000)
0.262 (0.004)
FSP-BoW r:100, no uf-idf
0.512 (0.007)
0:116 (0.005)
0.203 (0.008)
0.309 (0.004)
0,228 (0.003)
0.238 (0.000)
FSP-BoW 7:100
0.524 (0.001)
0.417 (0.004)
0.209 (0.003)
0.293 (0.012)
0.251 (0.006)
0. 0.262 (0.000)
FSSP-RevI.. depth:
0.528 (0.004)
0.428 (0.003)
0.217 (0.006)
0.305 (0.013)
0.243 (0.007)
0.265 (0.000)
PSP-RevI.M
0.524 (0.001)
0.436 (0.003)
0.212 (0.0022
0.305 (0.000)
0.255 (0.000)
0.277 (0.000)
Table 6: Analysis of future summaries at 3B scale. We evaluate the effect of different future- summary prediction approaches against vanilla MTP. Results (mean + s.e. over seeds) report pass@ 16 for code/math tasks and accuracy for ARC. Handcrafted multi-hot summaries (FSP-BoW) improve over standard MTP, especially on math reasoning (c.g., GSM8K and MATH), while learned summaries (FSP-RevLMy) provide the largest gains across math reasoning and ARC tasks.
D.2.2 EVALUATING PERFORMANCE ON NEXT-TOKEN PREDICTION
Our main comparison evaluates pretrained base models on downstream reasoning benchmarks, since performance on these tasks is a strong indicator for the capabilities we ultimately care about after post-training. For completeness, we also report standard language-modeling negative log-likelihood (NLL) and next-token accuracy (NTA) on two held-out validation sets, Dolei-Think and Dolmino (Olmo et al. 2025), which are excluded from the pretraining corpus. We compute NLL/NTA over approximate ly 4M tokens.
metrics, next-token
The results are shown in Table 7 and Table X for the 3B and 8B parameter model respectively. downstream reasoning benchmarks are substan- tially larger and more consistent than the differences observed NLL/NTA, which are often small. Thus auxiliary objectives influence the learned internal representations in way that improves gener- alization on downstream tasks, even when the standard next-token prediction metrics remain similar.
Across methods, we observe that improvements
22

---

Published as a conference paper at ICLR 2026
Metric
Dataset
NTP
MTP
DS-MTP
FSP-RevL.M
NLL
Dolci-Think
3.1813 (0.0000)
2.9746 (0.0000)
3.2272 (0.0000)
2.8012 (0.0000)
(lower is better)
Dolmino
1.0266 (0.0000)
1.0401 (0.0000)
1.0239 (0.0000)
1.0264 (0.0000)
Dolci-Think
0.4057 (0.0000)
0.4078 (0. (0.0000)
0.4046 (0.0000)
0.4214 (0.0000)
NTA
Dolmino
0.7450 (0.0000)
0.7457 (0.0000)
0.7491 (0.0000)
0.7513 (0.0000)
Table 7: Performance on next-token prediction at 3B scale. We report negative log-likelihood
(NLL) and next-token accuracy (NTA) Dolci-Think and Dolmino. Results are averaged over random seeds (mean + SE). Lower NLL corresponds to better predictive calibration.
Metric
Dataset
NTP
MTP
DS-MTP
FSP-ReyLM
Dolci-Think
1.3535 (0.0000)
1.3604 (0.0000)
1.3571 (0.0000)
1.3634 (0.0000)
NLL
Dolmino
0.9330 (0.0000)
0.9386 (0.0000)
0.9217 (0.0000)
0.9373 (0.0000)
(lower is better)
Dolci-Think
0.6118 (0.0000)
0.6123 (0.0000)
0.6128 (0.0000)
0.6124 (0.0000)
NTA
Dolmino
0.7652 (0.0000)
0.7679 (0.0000)
0.7707 (0.0000)
0. 0.7696 (0.0000)
Table 8: Performance on next-token prediction at 8B scale. We report negative log- ligelikeldhood (NLL) and next-token accuracy (NTA) Dolci-Think and Dolmino. Results are averaged over random seeds (mean +: SE). Lower NLL corresponds to better predictive calibration.
23


Reviews:
[{"review_id": "bayEfX4K35", "reviewer": "Reviewer_r6NQ", "summary": "The authors propose a novel pretraining objective called Future Summary Prediction (FSP). Instead of predicting specific future tokens, FSP trains an auxiliary head on the LLM to predict a single, compact summary of the long-term future of a sequence. This forces the model to develop a more global understanding and plan ahead, significantly reducing its reliance on myopic, shortcut-based learning.\n\nFSP augments the standard NTP loss with an auxiliary loss that predicts a summary of a long future window. The paper explores two ways to create this future summary:\nHand-crafted Summary (FSP-BCE): This is a simple but effective approach where the summary is a \"bag-of-words\" (a multi-hot vector) of all unique tokens that will appear in a long future window. The model is trained to predict the presence of these future tokens, without needing to know their exact order or position. This forces the model to look far ahead, making it much harder to rely on simple shortcuts.\nLearned Summary (FSP-RevLM): This is the paper's main and most powerful contribution. It addresses a weakness in handcrafted summaries, which can be \"noisy\" by including irrelevant future information. A separate reverse language model (RevLM) is trained to read sequences from right to left.The hidden state of this RevLM, after it has processed the future part of a sequence, serves as a rich, compact, and adaptive summary of that future. The main (forward) model is then trained to predict this learned summary vector. The RevLM naturally learns to emphasize what is important and predictable about the future, filtering out noise.", "questions": "1. Does this technique only boost model accuracy or do you see any boost in inference speed per token by generating multiple tokens at a time?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Novelty: This paper proposes a new dimension towards thinking about multi-token prediction in LLMs\nSuperior Performance on Benchmarks: At the 8B scale, FSP-RevLM consistently outperforms NTP, MTP, and DeepSeek-MTP across a range of benchmarks. It shows significant improvements on:\nReasoning: Up to a 5% absolute gain on ARC (AI2 Reasoning Challenge).\nMath: A 4.2% gain on MATH and 3.5% on GSM8K compared to standard MTP.\nCoding: Achieves the highest score on MBPP and is competitive on HumanEval+.", "weaknesses": "The speedup due to this change is not mentioned in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FbHbEQuVGD", "reviewer": "Reviewer_QvQ8", "summary": "This paper presents two methods for improving transformer training.  One predicts the future bag-of-words with an additional head (FSP-BCE) and another predicts the hidden representation of a backwards running transformer (FSP-RevLM).  There are some results on path-star, sibling discovery, and 8b scale pre-training.  I think this is a nice topic, but I see a few issues with the paper.  First, the presentation of two very different methods gives the papers a somewhat disjointed feel.  The advantages and disadvantages of each method feel like they are not discussed enough.  For example, on some datasets, FSP-BCE will converge to predicting a marginal distribution over all the tokens in the dataset, if the sequences are sufficiently long and diverse.  In this case, it seems like it would not be very useful.  It also seems like this the effectiveness of this method will be sensitive to how tokenization is done.  For example, if we used a character-level model, it seems like it would be almost completely useless.  I don't think that's a decisive disadvantage by itself, but a paper focusing on this method exclusively could explore these tradeoffs.  \n\nAnother serious issue is that I believe that the RevLM method has been proposed before in the literature, (Serdyuk et. al 2017), and while the method here has some differences, it seems like the paper should be written so that the novelties in the proposed method can be properly highlighted.", "questions": "The paper has experiments on the \"path star\" task, but why didn't you also run on the more general (and much harder) star graph task?  \n\nWhat's the added computational cost of training the RevLM model on the right-to-left sequences?  Presumably it's twice the number of total parameters used during training and a little over twice the cost to train?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper deals with an important topic, which is how to move beyond the next-token prediction objective in a practical way.  I believe that the experiments are also correctly implemented and described in a clear way.  Some of the empirical improvements, especially with FSP-RevLM are also impressive.", "weaknesses": "In my opinion, the biggest issue with this paper is that I don't believe that the FSP-RevLM method's novelty is correctly explained with respect to the existing literature.  There is a paper called Twin Networks, published in ICML 2017 (Serdyuk et. al), which I believe is essentially the same approach, of predicting the hidden state of a backward-trained RNN.  Now, there may be a difference here, in that this paper uses transformers instead of RNNs.  However, those differences should be discussed and this paper should be cited.  I found another paper, not cited in this work, called \"ProphetNet\" from (Qi et. al) in 2020 which predicts future n-grams, which also seems quite closely related to the bag-of-words prediction in this paper.    The z-forcing paper (Goyal et. al 2017) also predicts the hidden states of a backwards running RNN.  See section 3.3 in the arxiv paper.  The fact that this very relevant and fairly old work is not discussed, gives me concerns.  \n\nIt's less important, but there is a recent paper \"Joint Token Prediction\" (Ahn et. al 2025) which deals with learning belief states using a multi-token prediction method.  However, I am willing to accept this not being cited due to the work being quite recent.  \n\nAnother issue I have is that FSP-BCE seems like it could be sensitive to the horizon that you use, depending on how the dataset is constructed.  Perhaps empirically it's not an issue, but it seems like you can run into trouble with method if the sequences become long enough, and eventually the distribution over future tokens become similar (for example if our dataset is drawn from one very long sequence, then this would be an issue).  However, if the dataset splits all the sequences into different documents, perhaps this won't be an issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1N2QGX6sHQ", "reviewer": "Reviewer_YWeo", "summary": "This paper proposes Future Summary Prediction (FSP) to address the limitations of next-token prediction (NTP) and multi-token prediction (MTP). The authors introduce two variants of FSP and conduct experiments on 3B and 8B models, demonstrating consistent improvements over NTP and MTP across mathematical reasoning, general reasoning, and coding benchmarks.\n\nKey Reasons:\n1. The paper presents a well-motivated and principled pretraining objective that addresses genuine limitations of existing approaches.\n2. The interpretability of the learned representations and the computational cost of the approach are insufficiently analyzed.\n\nSupporting Arguments\n\nFSP-RevLM achieves consistent gains across multiple benchmarks, validating the empirical effectiveness of the method. However, the underlying mechanism—why future summary prediction improves model performance—remains underexplored.", "questions": "1. What is the computational overhead of training the reverse language model? Can it be jointly optimized with the main model?\n2. Have you considered applying FSP to other popular open-source LLM architectures?\n3. Why does FSP-RevLM slightly underperform NTP on GSM8K? Have you analyzed task-specific failure cases?\n4. Do you plan to evaluate FSP on larger models (e.g., 70B) or alternative architectures (e.g., MoE, RWKV)?\n5. Is there a way to interpret or visualize the learned future summaries from RevLM to better understand the captured information?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed FSP shifts the modeling focus from predicting individual tokens to capturing the global structure of future content—a fresh and principled direction for language modeling.\nExperiments at the 3B and 8B scales are comprehensive, with detailed comparisons against NTP, MTP, and DeepSeek-MTP across multiple benchmarks.", "weaknesses": "Although FSP-RevLM achieves strong empirical results, the learned future representations lack interpretability, making it unclear what specific future information is being encoded.\nThe computational cost of training the reverse language model is not discussed in detail. The paper does not include comparisons of training time or resource consumption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y7DA6nPFyg", "reviewer": "Reviewer_WozN", "summary": "The work proposes an improved multi-token prediction objective for LLMs. Instead of just predicting the next token, or next few tokens, the method proposes to predict some summary of the future tokens. The summary can either be a simple bag of words set, or a learned embedding from another transformer that encodes the future. They show at the 3B and 8B scale, that future summary prediction outperforms conventional multi-token approaches in a variety of reasoning tasks (ARC, GSM8K, etc.) as well as on some pedagogical tasks.", "questions": "Aside from my concerns above,\n\nIs there a connection between this objective and JEPA / SSL approaches, where we are trying to enforce similarity between a view of the context, and a view of the future?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting idea of predicting a learned summary of the future, instead of raw tokens, to make representations capture more of the future.\n- Good evaluations of their multi-token prediction objective against other multi-token prediction baselines, at somewhat large scales (3B-8B) and over serious reasoning benchmarks.\n- Paper is generally well written.", "weaknesses": "**This method has increased param / training budget, while baselines do not**\n\nThis method requires the training of an additional encoder, RevLM, and the experiments do not mention the additional size of this encoder or the computational cost incurred from using it. \n\n There should be some baseline that is fair in the number of training flops / parameters, since your method gets access to an additional encoder. For example, you should train the MTP baseline with the same number of parameters / flops as your method.\n\n\n**Learned Future Summary is still susceptible to failure modes** \n\nIt's also not clear to me how the learned future summary is \"adaptive\", in the sense that the future summary embedding is being trained on a \"fixed\" objective of previous token prediction, given the future. I suspect it's possible to come up with some adversarial sequence for the RevLM to destroy its representation. \n\nHere's one speculative idea. Imagine we also have a stargraph where the paths lead to other stargraphs.  The forward LM starts in the origin of the 1st stargraph, and the RevLM starts at the origin of another stargraph at the end of the path of the first stargraph. Then the RevLM will learn a shortcut solution for predicting the previous token, and making the learned future summary useless. \n\n### Minor\nSibling discovery task: the evaluation metric is vague, where the method's convergence speed is shown to be faster than NTP. But what is the definition of a converged method, is there a raw metric like accuracy score? The relative convergence speed to NTP convergence is not useful if NTP is bad, for example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)