Title: On the Bias of Next-Token Predictors Toward Systematically Inefficient Reasoning: A Shortest-Path Case Study
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reasoning

Chain-of-thought

Next-token prediction

Transformers

Algorithm

Shortest-Path
Abstract: Recent advances in natural language processing highlight two key factors for improving reasoning in large language models (LLMs): (i) allocating more test-time compute tends to help on harder problems but often introduces redundancy in the reasoning trace, and (ii) compute is most effective when reasoning is systematic and incremental, forming structured chains of thought (CoTs) akin to human problem-solving. To study these factors in isolation, we introduce a controlled setting based on shortest-path tasks in layered graphs. We train decoder-only transformers on question–trace–answer triples using a custom tokenizer, comparing models trained on optimal bottom-up dynamic programming traces with those trained on longer, valid traces involving backtracking. Surprisingly, under the same training-token budget, the latter models generalize better to unseen graphs. This benefit is not due to length alone—injecting arbitrary redundancy into reasoning traces fails to help and can even hurt performance. Instead, we find that generalization correlates with the model's confidence in next-token prediction, suggesting that long, coherent, and locally incremental traces make the training signal easier to optimize.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to understand how chain-of-thought reasoning traces and test-time compute aid reasoning performance. More specifically, they aim to understand the impact of the structure, efficiency, and length of reasoning traces on performance of a model trained from scratch on a shortest-path graph task. The authors make several observations: reasoning traces enable models to do this task at all for deeper graphs (directly outputting the shortest path works well only for short graphs). Then, inefficient depth-first search style reasoning traces work better than those reflecting an optimal dynamic programming solution. The authors attribute this to the fact that the reasoning steps follow each other more directly there, with less specific computation that has to be done for each step (each step easier to predict). They further study the impact of test-time compute through deterministic redundancy (if the model can predict where the redundancy occurs, it can use that to do additional computations). This does not help, meaning the actual steps you output matter. Finally, non-systematic structure in the reasoning traces works worst, again attributed to fact that the model cannot predict the next token accurately here (similarly for randomized redundancy). This indicates structure over content is more important in the reasoning trace.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Interesting and novel finding that inefficient reasoning works better than optimal reasoning traces, convincingly tied back to the fact that the intermediate steps in the inefficient traces follow more naturally from each other (each token easier to predict)\n- The paper is very clearly written, with good figures\n- Sound experimental setup,\n\nWeaknesses:\n- The results and findings would be stronger and tie more to real-world LLMs if experiments with pretrained models were done. For example through finetuning on the graph task with different style reasoning traces (essentially the same thing as done here but with pretrained models).\n- The findings would also be strengthened when an additional task would've been investigated. Right now it's unclear whether the findings would generalise beyond graph-based tasks where things like depth-first search are less obviously applicable (e.g. countdown).\n\nQUESTIONS:\n- The list of types of reasoning traces is a bit hard to follow, after continuing reading the paper I understood but perhaps you could already annotate which of each is the DP and DFS one (l133-l142)\n-  Line 210-211; to say that more test-time compute doesn't help, you would have to compare on graphs of the same length, do you do that? I don't see how you can get this conclusion from the $\\eta=0$ curves in fig 4.\n\nLIMITATIONS:\nMostly properly addressed, but could discuss how training from scratch might give different results than test-time compute on pretrained models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

comments: SUMMARY:
The authors build a sandbox in which a small decoder-only transformer (Phi-3) must solve shortest-path problems on layered graphs. The transformer is then trained on `question -> trace -> answer` triples and compare the effect of **efficient traces** (in which the traces are optimal bottom-up DP) vs. **inefficient traces** (in which the traces involve backtracking). Finding that the **inefficient traces** lead to better generalization, the authors argue that step-by-step backtracking better matches the inductive bias of autoregressive LMs.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The authors set up a clean, controlled experimental sandbox to study changes in trace structure and length.
- By keeping the total token budget fixed, the authors are able to cleanly focus on changes in trace structure.
- The paper is clear, well written, and original.

**Weaknesses**
- The biggest weakness of this paper is that the experiments are only done on one experimental setting (shortest-path in layered graphs), for one model (Phi3) that is **extremely** small. It is not clear if the inductive bias narrative transfers to models with B's of params and more emergent planning abilities. It is also not clear if the lesson about inefficient traces being more helpful would translate to other tasks.

QUESTIONS:
1. Have the authors tried the same experiment on another algorithmic benchmark, or on natural-language datasets (e.g., math)? Do the same trends hold?
2. Have the authors tried the same experiment on varying model sizes? Is there a point at which inefficient traces stop being optimal?
3. The eureka jump is intriguing. Have the authors inspected attention patterns / neuron activations before and after the jump for more insight?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
In a carefully controlled setting, the authors investigate the reasoning behavior of modern LLMs. By systematically altering the reasoning trajectories in training samples, they observe the models’ learning dynamics. They arrive at a surprising conclusion: LLMs are better at following structured, step-by-step reasoning, rather than finding creative shortcuts to solve problems.

STRENGTHS AND WEAKNESSES:
Strengths:

1.The custom tokenizer designed for the shortest-path problem, along with the experimental setup, is very novel.

2.The study offers valuable insights into how Chain-of-Thought training data can be constructed in specialized scenarios.

Weaknesses:

1.The conclusions are based on a single synthetic dataset, which may be too narrow. The highly structured nature of the input makes the task closer to sequence prediction than genuine reasoning.

2.The explanation for why structured but inefficient reasoning outperforms optimal strategies is not entirely convincing. The weaker performance of dynamic programming (DP) traces may result from limited coverage in training, as DP trajectories are less repetitive and harder to learn than DFS ones. The better results with DFS may simply reflect that such traces are more learnable for LLMs.

QUESTIONS:
1.How was the graph data used in training and testing constructed? In particular, were there any rules governing how nodes were connected? I’m concerned the model may have learned node-connectivity patterns rather than actual reasoning rules.

2.The study focuses solely on synthetic data. Do the authors expect their findings to hold on publicly available datasets? Have they tested whether the same bias appears there?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a synthetic testbed to test how well transformers can learn to find the shortest path in a directed acyclic graph with different search strategies. They experiment with a less efficient depth-first-search-like strategy and a more efficient dynamic programming strategy. They find through extensive experiments involving training Phi3 models from scratch that transformers are able to better generalize to unseen graphs when they are trained on the less efficient search strategy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - The testbed reveals interesting behavior where transformers learn to find the shortest task more generalizably using a less efficient search technique.\n - The paper represents an interesting exploration in fundamental capabilities of transformers when trained with idealized data (therefore providing insight into the upper limits of transformer capabilities on this task).\n\nWeaknesses:\n - The exploration is rather limited in terms of graph depth and it is unclear if the findings generalize to larger graphs/CoTs. It would be very interesting to explore the scaling trend of the inefficient vs efficient searching approaches on larger graphs.\n - Some claims are not supported by the experimental results. (see below)\n\nQUESTIONS:\nFrom Figure 2, it seems that the redundancy setting is not exactly comparable to the \\eta = -5 setting: The mean CoT length in the redundancy setting is 12% larger than that of \\eta = -5, and the variance is 25% smaller.\n\nLine 188: \"...good generalization on the larger graph instances can only be achieved when the model is allowed to produce a reasoning trace.\"\n  Since L = 7 (line 183), the model should have seen graphs with a maximum depth of 7, right? If so, then depth-7 graphs would not be \"larger graph instances.\" Or by \"larger graph instances\" did you mean the graphs with depths within the training distribution that are larger than other graphs from the training distribution? What is the distribution of graph depth of the training examples?\n\nThe analysis on the model's confidence of the next step prediction is unclear: For \\eta = 0 and \\eta = +5, isn't low confidence expected? If there are multiple valid next steps at a given point in the exploration, then the model should assign uniform probability to each of the valid next steps. In this sense, next token confidence would only be a good proxy for learnability _if the task is not too ambiguous._ How often do the exploration traces encounter points where there are more than one valid next steps (for \\eta = -5, 0, and +5)?\n\nLine 221: Why would it necessarily be the case that \"incremental steps [immediately building] upon previous steps\" would result in higher next-token confidence?\n\n\nThe paper does not make many grammatical errors, but the writing can be much improved. Some of sentences are difficult to read and can be made much clearer. I list (non-exhaustively) some of the issues below and I recommend the authors carefully re-read the paper and correct all such errors. The below list also contains some questions about the content:\n\nFigure 1: The edge costs are drawn as nodes which is misleading. I strongly suggest the authors draw the edge costs as labels rather than nodes.\n\nLine 81: Spurious hyphen?\n\nLines 87-89: (minor) These lines seem to be spaced much farther apart than other lines.\n\nFigure 2: I believe \"Effect of...on...\" rather than \"Dependency on...of...\" is closer to the intended meaning.\n\nLine 115: So does a larger or smaller value of \\eta produce more \"structurally efficient\" reasoning traces? It is not easy to understand the meaning \"structural efficiency\" if no precise definition is provided aside from the algorithm in Figure 2.\n\nLine 127: How are reasoning steps \"disregarded\"? Are they randomly removed from the reaoning trace? If so, they would have no effect on the subsequent generation of the transformer, won't they? For systematic redundancy: Is it only possible to increase the length of the CoT by an integer >= 2 (since *each* reasoning step is repeated)?\n\nLine 146: Is this the percent increase or decrease?\n\nLine 171: Duplicate \"iv)\".\n\nFigure 3a: This plot needs a legend. The colors are also somewhat random. I recommend using a gradient of three colors to better depict the sequence of 3, 5, and 7-depth graphs.\n\nLine 204: Does \"dashed\" and \"full\" refer to the plot in Figure 4a? The figure hasn't even been referenced yet, so you should avoid referring the line style in the plot here.\n\nLine 208: \"less\" -> \"fewer\"\n\nFigure 4: I strongly suggest that legends be added to the plots in this figure.\n\nLine 210: (minor) Use \"---\" or \\textemdash for em dash, and remove spaces around each dash.\n\nLines 215-218: This sentence is unclear and difficult to understand. What does \"understand\" mean here, in the context of \"harder to understand and mimic for the trained model\"? What does it mean for the model to understand an exploration order?\n\nLine 229: \"the model transitions from repeating the same reasoning steps from 20% to 3% of the times\" This is ambiguous. Does this mean that, prior to this transition point, the model is repeating steps in 20% of the validation examples? Training examples? Or are 20% of the reasoning steps repeated across all reasoning steps in all validation examples? (or training examples?)\n\nLIMITATIONS:\nSee Weaknesses above. The authors acknowledge that their testbed is rather artificial and their findings may not necessarily generalize to real-world settings. However, I do believe it represents an interesting exploration in fundamental capabilities of transformers when trained with idealized data (therefore providing insight into the upper limits of transformer capabilities on this task).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}