Title: Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, SLM, latent reasoning

Abstract: Enhancing the reasoning capabilities of Large Language Models (LLMs), especially under parameter constraints, is crucial for versatile applications. However, with limited computation per decoding step, LLMs struggle to predict some hard tokens, such as logical connectives that guide subsequent reasoning directions. Prior work addresses this with uniform latent thinking. After the first, standard forward pass, instead of verbalization, last-layer hidden states are fed back as inputs for extra iterations to refine predictions. Yet we identify a latent overthinking problem: easy token predictions that are already correct after the first pass are sometimes revised into errors when uniformly pushed to second iterations.
To address this, we propose Think-at-Hard (TaH), a dynamic latent thinking method that iterates deeper only at hard tokens. TaH employs a lightweight neural decider to identify which tokens are likely wrong after the standard forward pass. Selected positions enter latent iterations, where the model shifts from general next-token prediction to focused hard-token refinement. TaH applies Low-Rank Adaptation (LoRA) modules and residual connections to the LLM backbone to better facilitate such objective shift. We also introduce a duo-causal attention mechanism, extending standard causal attention with an additional depth dimension for cross-iteration information flow.
Experiments show that TaH consistently boosts LLM reasoning performance. Compared with strong single-iteration Qwen3 models finetuned with the same data, it delivers a 4.0–5.0% average accuracy gain across five challenging benchmarks using the same number of parameters and only 6% additional iterations. With less than 3% additional parameters, the gains further increase to 5.3–5.4%. Relative to AlwaysThink which performs two iterations at every token, the gains are 8.1–11.3% and 8.5–12.6%, respectively.

TL;DR: We use a neural decider to dynamically initiate latent iterations only where needed