Training Large Language Models to Reason in a Continuous Latent Space
Authors: Shibo Hao
Sainbayar Sukhbaatar
DiJia Su
Xian Li
Zhiting Hu
Jason E Weston
Yuandong Tian
Venue: COLM 2025 (2025)
Keywords: chain of thought
reasoning
TL;DR: To explore the potential of LLM reasoning in an unrestricted latent space instead of using natural language, we introduce a new paradigm Coconut (Chain of Continuous Thought)

Abstract: Large language models (LLMs) are restricted to reason in the "language space", where they typically express the reasoning process with a chain-of-thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning. For example, most word tokens are primarily for textual coherence and not essential for reasoning, while some critical tokens require complex planning and pose huge challenges to LLMs. To explore the potential of LLM reasoning in an unrestricted latent space instead of using natural language, we introduce a new paradigm Coconut (Chain of Continuous Thought). We utilize the last hidden state of the LLM as a representation of the reasoning state (termed "continuous thought"). Rather than decoding this into a word token, we feed it back to the LLM as the subsequent input embedding directly in the continuous space. Experiments show that Coconut can effectively augment the LLM on several reasoning tasks. This novel latent reasoning paradigm leads to emergent advanced reasoning patterns: the continuous thought can encode multiple alternative next reasoning steps, allowing the model to perform a breadth-first search (BFS) to solve the problem, rather than prematurely committing to a single deterministic path like CoT. Coconut outperforms CoT in certain logical reasoning tasks that require substantial backtracking during planning, with fewer thinking tokens during inference. These findings demonstrate the promise of latent reasoning and offer valuable insights for future research.

Published as a conference paper at COLM 2025
Training Large Language Models to Reason in a Continuous Latent Space
Shibo Hao Sainbayar Sukhbaatar', DiJia Su', Xian Li', Zhiting Hu?, Jason Weston Yuandong Tian'
FAIR at Meta, UCC San Diego
https::/ggttubb.cmmm http://hhhh..cooooooooootrrrrros
Abstract
Large language models (LLMs) are restricted to reason in the "language space", where they typically express the reasoning process with a chain-of- thought (CoT) to solve a complex reasoning problem. However, we argue that language space may not always be optimal for reasoning. For example, some critical tokens require complex planning, making them difficult to compute in single forward pass, while many other tokens contribute little to the actual reasoning process. To explore the potential of LLM reasoning in unrestricted latent space instead of using natural language, we introduce a new paradigm COCONUT (Chain of Continuous Thought). We utilize the last hidden state of the LLM as a representation of the reasoning state (termed "continuous thought"), Rather than decoding this into a word token, we feed it back the LLM as the subsequent input embedding directly in the continuous space. This latent reasoning paradigm leads to the emergence of an advanced reasoning pattern: the continuous thought can encode multiple alternative next reasoning steps, allowing the model perform breadth-first search (BFS) to solve the problem, rather than prematurely committing to single deterministic path like COT. COCONUT outperforms CoT on certain logical reasoning tasks that require substantial search during planning, and shows better trade-off between accuracy and efficiency. We hope these findings demonstrate the promise of latent reasoning and offer insights for future research.
1 Introduction
Large language models (LLMs) have demonstrated remarkable reasoning abilities, emerging from extensive pretraining on human languages (Dubey et al., 2024; Achiam et al., 2023). While next token prediction is an effective training objective, it imposes a fundamental constraint on the LLM as a reasoning machine: the explicit reasoning process of LLMs must be generated in word tokens. For example, a prevalent approach, known as chain-of-thoughe (CoT) reasoning (Wei et al., 2022), involves prompting or training L.LMs to generate solutions step-by-step using natural language. However, this stands in stark contrast to findings from studies on human cognition. Neuroimaging studies have consistently shown that the language network production - remains largely inactive during various reasoning tasks (Amalric & Dehaene, 2019; Monti et al., 2012; 2007; 2009; Fedorenko et al., 2011). Further evidence indicates that human language is optimized for communication rather than reasoning (Fedorenko et al., 2024).
set of brain regions responsible for language comprehension and
significant issue arises when LLMs use language for reasoning: the amount of reasoning required for each particular language token varies greatly, yet current LLM architectures allocate nearly the same computing budget for predicting every token. Some critical tokens require complex planning and pose huge challenges to LLMs, while most tokens in
Work done at Meta.

---

Published as a conference paper at COLM 2025
Chain-of-Thought (CoT)
Chain of Continuous Thought (CoCONUT)
Last hidden states are used as input embeddings
output token
x
Ai4j
[Answer]
[Answer]
(sampinn)
last hidden state
Large Language Model
Large Language Model
Input embedding
input taken
[Question]
Ag
Ai+1
A++2
[Question]
<bot>
<eot>
Figure 1: A comparison of Chain of Continuous Thought (CoCONUT) with Chain-of- Thought (CoT). In CoT, the model generates the reasoning process as a word token sequence (e-g,, [xi, xi+ in the figure). COCONUT regards the last hidden state representa- tion of the reasoning state (termed "continuous thought"), and directly uses it the next input embedding. This allows the LLM to reason in an unrestricted latent space instead of language space.
reasoning chain are generated solely for fluency, offering minimal contribution to the reasoning process and consequently reducing overall efficiency. While previous work has attempted to fix these problems by performing additional reasoning before generating some critical tokens (Zelikman et al., 2024) or encouraging LLMs to generate succinct reasoning chains (Madaan & Yazdanbakhsh, 2022; Nayab et al., 2024; Han et al., 2024), these solutions remain constrained within the language space and do not solve the fundamental problems. On the contrary, it would be ideal for LLMs to have the freedom to reason without any language constraints, and then translate their findings into language only when necessary.
In this work we instead explore LLM reasoning in a latent space by introducing novel paradigm, COCONUT (Chain of Continuous Thought). It involves a simple modification to the traditional CoT process: instead of mapping between hidden states and language tokens using the language model head and embedding layer, COCONUT directly feeds the last hidden state (a continuous thought) the subsequent input embedding for the next token (Figure 1). This modification frees the reasoning from being within the language space, and the system can be optimized end-to-end by gradient descent. To enhance the training of latent reasoning, we employ a multi-stage training strategy inspired by Deng et al. (2024), which effectively utilizes language reasoning chains to guide the training process.
This proposed method leads to an efficient reasoning pattern: Unlike language-based reasoning, continuous thoughts in COCONUT can encode multiple potential next steps simultaneously, allowing for a reasoning process akin to breadth-first search (BFS). While the model may not initially make the correct decision, it can maintain many possible options within the continuous thoughts and progressively eliminate incorrect paths through reasoning, guided by some implicit value functions. This advanced reasoning mechanism surpasses traditional CoT, even though the model is not explicitly trained or instructed to operate in this manner, seen in previous works (Yao et al., 2023; Hao et al., 2023).
search
We further validate the feasibility of latent reasoning through additional analyses on three datasets. On other planning-inten -intensivee tasks, such ProntoQA (Saparov He, 2022) and our newly introduced ProsQA, COCONUT consistently achieves higher accuracy than CoT while generating fewer tokens. For math reasoning tasks on GSM8k (Cobbe et al., 2021), augmenting the LLM with six continuous thoughts doubles its performance. Additionally, COCONUT surpasses the strong baseline iCoT (Deng et al., 2024) and offers a superior trade- off between accuracy and efficiency compared to COT. We believe these results underscore the significant potential of latent reasoning and offer valuable insights to guide future research.
2 Related Work
thain-of-thought (CoT) reasoning. We use the term chain-of-th thotught broadly to refer to methods that generate intermediate reasoning process in language before outputting the
2

---

Published as a conference paper at COLM 2025
final answer. This includes prompting LLMs (Wei et al., 2022; Khot et al., 2022; Zhou et al., 2022), or training LLMs to generate reasoning chains, either with supervisse finetuning (Yue et al., 2023; Yu et al., 2023) reinforcement learning (Wang et al., 2024; Havrilla et al., 2024; Shao et al., 2024; Yu et al., 2024a; Guo et al., 2025). Madaan & Yazdanbakhsh (2022); Nayab et al. (2024); Han et al. (2024) designed methods to make LLMs generate shorter reasoning chains. Recent theoretical analyses have demonstrated the usefulness of CoTT from the perspective of model expressivity (l'ng et al., 2023; Merrill & Sabharwal, 2023; Li et al., 2024). By employing CoT, the effective depth of the transformer increases because the generated outputs are looped back to the input (Feng et al., 2023). These analyses, combined with the established effectiveness of CoT, motivated our design that feeds the continuous thoughts back to the LLM as the next input embedding. While CoT has proven effective for certain tasks, its autoregressive generation nature makes it challenging to mimic human reasoning on more complex problems (LeCun, 2022; Hao et al., 2023), which require planning and search. There are works that equip LLMs with explicit tree search algorithms (Xie et al., 2023; Yao et al., 2023; Hao et al., 2024), or train the LLM on search dynamics and trajectories (Lehnert et al., 2024; Gandhi et al., 2024; Su et al., 2024). In our analysis, we find that after removing the constraint of a language space, new reasoning pattern similar to BFS emerges, though the model is not explicitly trained this way.
Latent reasoning in LLMs. Previous works mostly define latent reasoning in LLMs as the hidden computation in transformers (Yang al., 2024; Biran et al., 2024). Yang et al. (2024) constructed a dataset of two-hop reasoning problems and discovered that it is possible to recover the intermediate variable from the hidden representations. Biran al. (2024) further proposed to intervene the latent reasoning by "back-patching" the hidden representation. Shalev et al. (2024) discovered parallel latent reasoning paths in LLMs. Another line of work has discovered that, even if the model generates CoT to reason, the model may actually utilize a different latent reasoning process. This phenomenon is known as the unfaithfulness of CoT reasoning (Wang et al., 2022; Turpin et al., 2024). To enhance the latent reasoning of LLMs, previous research proposed to augment it with additional tokens. Goyal et al. (2023) pretrained the model by randomly inserting a learnable <pause> tokens to the training corpus. This improves LLM's performance on a variety of tasks, especially when followed by supervised finetuning with <pause> tokens. On the other hand, Pfau et al. (2024) further explored the usage of filler tokens, e.g. "g.""..." parallelizable problems. However, Pfau et al. (2024) mentioned these methods do not extend the expressivity of the LLM like CoT; hence, they may not scale to more general and complex reasoning problems. Wang et al. (2023) proposed to predict a planning token as discrete latent variable before generating the next reasoning step. Recently, it has also been found that one can "internalize" the CoT reasoning into latent reasoning in the transformer with knowledge distillation (Deng et al., 2023) or special training curriculum which gradually shortens Co'T (Deng et al., 2024). Yu et al. (2024b) also proposed to distill a model that can reason latently from data generated with complex reasoning algorithms. These training methods can be combined to our framework, and specifically, we find that breaking down the learning of continuous thoughts into multiple stages, inspired by iCoT (Deng et al., 2024), is very beneficial for the training. Other work explores alternative architectures for latent reasoning, including looped transformers (Giannou al., 2023; Fan et al., 2024), diffusion models in sentence embedding space (Barrault et al., 2024), depth-recurrent models (Geiping et al., 2025) or energy-based models (Gladstone et al., 2025). Different from these works, we focus on general multi-step reasoning tasks and aim at investigating the unique properties of latent reasoning in comparison to language space. Zhu et al. (2025) developed a theoretical construction of 2-layer transformer with COCONUT, leveraging the idea that continuous thoughts serve as superpositional representations of multiple reasoning paths. It proves to solve the directed graph reachability problem more efficiently than the best-known theoretical results based on discrete COT.
" and concluded that they work well for highly
3 COCONUT: Chain of Continuous Thought
In this section, we introduce our new paradigm COCONUT (Chain of Continuous Thought) for reasoning in unconstrained latent space. We begin by introducing the background and
3

---

Published as a conference paper at COLM 2025
CoT (Supervised line-tuning)
[Question]
[Step
[Step 2]
[Step
[Answer]
CoCONUT
[Question]
<bot><<eot>
[Step
[Step 2]
[Step N]
[Answer]
(Stage
00
[Question]
<bot>
<eot>
[Step 2]
[Step
[Answer]
(Stage
[Question]
<bots
<eot>
[Answer]
discrete token
(Stage
calculating loss
continucus thoughts
continuous thought
Figure Training procedure of Chain of Continuous Thought (CoCONUT). Given training data with language reasoning steps, at each training stage we integrate additional con- tinuous thoughts (c = this example), and remove one language reasoning step. The cross-entropy loss is then used on the remaining tokens after continuous thoughts.
notation use for language models. For input sequence (x1, XT), the standard large language model M can be described as:
H; Transformer (E+)
M (x,-1 x<() softmax(Whi)
where E, =
is the sequence of token embeddings up to position t; H, E Rfxd is the matrix of the last hidden states for all tokens up to position I; h is the last hidden state of position ie., h4 = H, el()ee((() is the token embedding function; is the parameter of the language model head.
Method Overview. the proposed COCONUT method, the LLM switches between the "language mode" and "latent mode" (Figure 1). In language mode, it operates standard language model, autoregressively generating the next token. latent mode, it directly uses the last hidden state the next input embedding. This last hidden state represents the current reasoning state, termed as a "continuous thought". Special tokens <bot> and <eot> employed to mark the beginning and end of the latent thought mode.
Training Procedure. In this work, we consider a problem-solving scenario where the model receives a question input and is expected to generate answer through reasoning process. We leverage language CoT data to supervise continuous thought by implementing multi-stage training curriculum inspired by Deng al. (2024). As shown in Figure 2, in the initial stage, the model is trained on regular CoT instances. Assume the total number of training stages (excluding the initial stage) set to N (N in this example). In the subsequent stages, at the k-th stage, the first k reasoning steps in the CoT are replaced with k x continuous thoughts where is hyperparameter controlling the number of latent thoughts replacing a single language reasoning step. Following Deng et al. (2024), we also reset the optimizer state when training stages switch.
During the training process, we optimize the negative log-likelidood loss as usual, but mask the loss questions and latent thoughts. It is important to note that the objective does not encourage the continuous thought to compress the removed language thought, but rather to facilitate the prediction of future reasoning. Therefore, it's possible to learn more effective representations of reasoning steps compared to human language.
IIf language reasoning chain is shorter than & steps, then all the language thoughts will be removed.

---

Published as a conference paper at COLM 2025
Training Details. Our proposed continuous thoughts are fully differentiable and allow for back-propagation. We perform л + forward passes when n latent thoughts are scheduled in the current training stage, computing new latent thought with each pass and finally conducting an additional forward pass to obtain a loss on the remaining text sequence. While we have avoided redundant computations by reusing the KV cache across each forward pass, the sequential computation of COCONUT poses challenges on parallelism for the existing training infrastructure. Further optimizing the training efficiency of COCONUT remains an important direction for future research (Pöppel et al., 2025).
Inference Process. The inference process for COCONUT is analogous to standard language model decoding, except that in latent mode, we directly feed the last hidden state as the next input embedding. One practical challenge is determining when the model should switch between latent and language modes. As we focus on the problem-solving setting, we insert a <bot> token immediately following the question tokens. For <eot>, we consider two potential strategies: a) train a binary classifier on latent thoughts to enable the model to autonomously decide when to terminate the latent reasoning, or b) always pad the latent thoughts to constant length. We found that both approaches work comparably well. Therefore, we use the second option in our experiment for simplicity, unless specified otherwise.
4 Continuous Space Enables Latent Tree Search
In this section, we provide a proof of concept on the advantage of continuous latent space reasoning. On ProsQA, a new dataset that requires extensive planning ability, COCONUT outperforms language space CoT reasoning. Interestingly, our analysis indicates that the continuous representation of reasoning can encode multiple alternative next reasoning steps. This allows the model to perform a breadth-first search (BFS) to solve the problem, instead of prematurely committing to a single deterministic path like language COT.
We start by introducing the experimental setup (Section 4.1). By leveraging COCONUT's ability to switch between language and latent space reasoning, we are able to control the model to interpolate between fully latent reasoning and fully language reasoning and test their performance (Section 4.2). This also enables us to interpret the latent reasoning process as tree search (Section 4.3). Based on this perspective, we explain why latent reasoning can make the decision easier for LLMs (Section 4.4).
4.1 Experimental Setup
Dataset. We introduce ProsQA (Proof with Search Cuestion-Ansseriing). a new logical reasoning dataset. A visualized example is shown in Figure Each instance in ProsQA con- sists of directed acyclic graph (DAG) of logical relationships between concepts, presented as natural language statements. The task requires models to determine logical relationships by finding valid paths through this graph, demanding sophisticated planning and search strategies. Unlike previous logical reasoning datasets like ProntoQA (Saparov & He, 2022), ProsQA's DAG structure introduces complex exploration paths, making it particularly challenging for models to identify the correct reasoning chain. More comprehensive details about the dataset construction and characteristics can be found in Appendix A.
Experimental Setup. We use a pre-trained GPT-2 model as the base model for all experi- ments. The learning rate is set to 1 x 10-4 while the effective batch size is 128. We train COCONUT model following the training procedure in Section 3. Since the maximum reasoning steps in ProsQA is 6, we set the number of training stages to N = 6 in the training procedure. In each stage, we train the model for 5 epochs, and stay in the last stage until the 50 epochs. The checkpoint with the best accuracy in the last stage is used for evaluation. As reference, we report the performance of (1) CoT: the model is trained with CoT data, and during inference, the model will generate complete reasoning chain to solve the problem. (2) no-CoT: the model is trained with only the question and answer pairs, without any reasoning steps. During inference, the model will output the final answer directly.
5

---

Published as a conference paper at COLM 2025
Final
Reasoning Process
(km6)
Category
Ours (k=5)
Ours (k=5)
Correct Label
Ours (k=4)
Ours (k=4)
Incorrect Label
Method
Ours (k=3)
Method
Ours (k=3)
Wrong Target
Longer path
(k-2)
Ours (k=2)
Hallucination
Ours (k-1)
Ours (K-1)
Ours (k=0)
Ours (k=0)
CoT
CoT
80
100
100
200
300
400
500
Accuracy (%)
Count
The accuracy of final answer (left) and reasoning process (right) of multiple variants of COCONUT and baselines on ProsQA
Figure 3:
To understand the properties of latent and language reasoning space, we manipulate the model to switch between fully latent reasoning and fully language reasoning, by manually setting the position of the <eot> token during inference. When we enforce COCONUT to use k continuous thoughts, the model expected to output the remaining reasoning chain in language, starting from the k step. In our experiments, we test variants of COCONUT 2,,,,,,,,6, 4,3, Note that all these variants only differ in inference time while sharing the same model weights.
ProsQA with E {0, 2,
Metrics. We apply two sets of evaluation metrics. One of them is based on the correctness of the final answer, regardless of the reasoning process. It is also the main metric used in the later sections (Section 5.3). To enable fine-grained analysis on ProsQA, we define another metric the reasoning process. We classify reasoning chain into (1) Correct Path: The output is one of the shortest paths to the correct answer. (2) Longer Path: A valid path that correctly answers the question but is longer than the shortest path. (3) Hallucination:
(4) Wrong Target: A valid path in the graph, but the destination node is not the one being asked. These four categories naturally apply to the output from COCONUT (k = 0) and CoT, which generate the full path. For COCONUT with > 0 that outputs only partial paths in language (with the initial steps in continuous reasoning), we classify the reasoning as a Correct Path if a valid explanation can complete it. Also, we define Longer Path and Wrong Target for partial paths similarly. If no valid explanation completes the path, it's classified as Hallucination. In no-CoT and COCONUT with larger k, the model may only output the final answer without any partial path, and it falls into (5) Correct Label or (6) Incorrect Label. These six categories cover all cases without overlap.
The path includes nonexistent edges or is disconnected.
4.2 Overall Results
Figure 3 presents comparative analysis of various reasoning methods evaluated on ProsQA. The model trained using CoT' frequently hallucinates non-existent edges or outputs paths leading to incorrect targets, resulting in lower answer accuracy. In contrast, COCONUT, which leverages continuous space reasoning, demonstrates improved accuracy as utilizes an increasing number of continuous thoughts. Additionally, the rate of correct reasoning processes (indicated by "Correct Label" and "Correct Path") significantly increases. At the same time, there is notable reduction in instances of "Hallucination" and "Wrong Target," issues that typically emerge when the model makes mistakes early in the reasoning process.
This improvement can be attributed to the inherent advantage of latent reasoning: it miti- gates premature commitment to definitive choices by allowing the model to progressively refine its decisions through subsequent steps. Consequently, incorrect options systemati- cally eliminated over time, culminating in higher final accuracy. An intuitive demonstration of this benefit is provided by the case study depicted in Figure

---

Published as a conference paper at COLM 2025
Question:
Rost
Ta gegt node
Every grimpus yimpus, Every worpus a jolpus, Every zhorpus a sterpus. Alex is grimpus on Every lumps is yumpus. Question: Alex gorpus bompus?
SGF
Distrcttive noce
Sam
Child the root Grandchild
CoT
node
Ground Truth Solution
Alex is a lempus. Every
Alex a grimpus.
botty
Every ghipus rorpus, Every rorpusi bompus, #Ra Alex (s bompus
Every scrompus is yumpus, Every yumpus rempus, Every rempus gorpus. #** Alex is gorpus X
REERAL
PGS
(Hallucination)
Win
CoCoNUT (k=1)
[Thought] <eot>
CocoNUT (k=22
Every lempus a scrompus. Every scrompus brimous, X
<bot> [Thought] [Thought] <eot>
Every torpus bompus,
Aee
#*# Alex a brimpus
#B0 Alex is bompus v/
(Wronn Target)
(Correct Fath)
Figure 4: A case study of ProsQA. The model trained with CoT hallucinates edge (Every yumpus is rempus) after getting stuck in a dead end. COCONUT (k=1) outputs a path that ends with an irrelevant node. COCONUT (k=2) solves the problem correctly.
0.01 (h=0)
2e-3 (hel)
DUS
0.33 (h=2)
CoCONUT (k-1)
76-4
(1~01
3e-3 (h=D)
CacoNuT (c=2)
<bot> [Touught] <eot>
<bot> [Thought] (Thought] <eot> Every corpus
PCA
PC Tempus")
(h-O)
PC rorpus
(hml)
p( ((1eepppmm"" (mm")((pss)))))))
pCro """ra (p("""
0.32
(l-2)
5e-5
(h=1)
0.12
(h-1)
0.87
Figure 5: An illustration of the latent search trees. The example is the same test case as in Figure 4. We show the probability of the first concept predicted by the model following latent thoughts (e.g., "lempus" in the left figure). It is calculated as the multiplication of the probability of all tokens within the concept. The height of a node (denoted as h in the figure) is defined as the shortest distance to any leaf node in the graph
4.3 Interpreting the Latent Reasoning as Tree Search
To better understand COCONUT, we probe the latent reasoning process by forcing the model to explicitly generate language reasoning steps following intermediate continuous thoughts (Figure 5). Using the example presented in Figure 4, at the initial reasoning step, the model must select which immediate child node of Alex' to consider next, specifically from the set lempus, sterpus, zhorpus, grimpus. The distribution over these candidate next steps is visualized in Figure 5, left. In the subsequent reasoning step, these nodes expand further into an extended set of potential paths, including all grandchildren of AAlex' (Figure 5, right).
We define the predicted probability of a concept following continuous thoughts as value function (Figure 5), estimating each node's potential for reaching the correct target. Interest- ingly, the reasoning strategy employed by COCONUT is not greedy: while "lempus" initially has the highest value (0.33) at the first reasoning step (Figure 5, left), the model subsequently assigns the highest value (0.87) to "rorpus," a child of "grrimpus"" rather than following "lempus" (Figure 5, right). This characteristic resembles a breadth-first search (BFS) ap- proach, contrasting sharply with the greedy decoding typical of traditional CoT methods. The inherent capability of continuous representations to encode multiple candidate paths enables the model to avoid making immediate deterministic decisions. Importantly, this tree search pattern is not limited to the illustrated example, but constitutes fundamental mechanism underlying the consistent improvement observed with larger values of in CoCONUT.
7

---

Published as a conference paper at COLM 2025
4.4 Why is Latent Space Better for Planning?
Building upon the tree search perspective, we further examine why latent reasoning benefits planning fasks-specificallly why maintaining multiple candidate paths and postponing deterministic decisions enhances reasoning performance. Our hypothesis is that nodes explored in the early reasoning stages are inherently more challenging evaluate accurately because they are farther from the final target nodes. In contrast, nodes positioned closer to potential targets, having fewer subsequent exploration possibilities, can be assessed accurately with higher confidence.
To systematically test this, we define the height of node as its shortest distance to any leaf node and analyze the relationship between node height and the model's estimated value. Ideally, correct node-one that can lead to the target node-should receive high estimated value, whereas an in- correct node-oneee one that cannot lead to the target node-should receive a low value.
First thoughts
0.5
Incorrect
Value
0.4
0.2
0.0
Empirical results across the test set (Figure 6) sup- port our hypothesis: nodes with lower heights con- sistently receive more accurate and definitive prob- ability evaluations. Conversely, nodes with greater heights exhibit more ambiguous evaluations, reflect- ing increased uncertainty.
1
3
5
Height
Second thoughts
0.3
These findings underscore the advantage of latent space reasoning. By delaying deterministic decisions and allowing exploration to proceed toward termi- nal states, latent reasoning significantly enhances the model's ability to differentiate correct paths from incorrect ones, thereby improving performance on complex, -intennn-ittensiee tasks compared to tradi- tional greedy methods. Motivated by these empirical observations, Zhu al. (2025) presented a theoretical construction of a 2-layer transformer with COCONUT, demonstrating that it can solve the directed graph reachability problem using O(n) steps, where n is the number of vertices. This efficiency arises from the use of superposition to represent multiple explored vertices in parallel. On the contrary, the best known theoretical result for constant-depth transformers with discrete CoT requires O(z²) steps (Merrill & Sabharwal, 2023). Experiments also confirm that the construction aligns well with the empirical solution obtained via training dynamics.
0.2
0.1
0.0
2
3
Height
Figure 6: The correlation between the predicted value of correct/incon incor-- rect nodes and their heights.
Empirical Results with COCONUT
After showing the promising tree search pattern of COCONUT, we validate the feasibility of LLM reasoning in continuous latent space through more comprehensive experiments, highlighting its better reasoning efficiency over language space, as well as its potential to enhance the model's expressivity.
5.1 Experimental Setup
Math Reasoning. We use GSM8k (Cobbe et al., 2021) the dataset for math reasoning, It consists of grade school-level math problems. To train the model, we use a synthetic dataset generated by Deng et al. (2023). We use two continuous thoughts for each reasoning step (i.e., c = 2), and set the number of training stages to 3.
Logical Reasoning. Logical reasoning involves the proper application of known conditions to prove or disprove a conclusion using logical rules. We use the ProntoQA (Saparov & He, 2022) dataset, and our newly proposed ProsQA dataset, which is more challenging due to
8

---

Published as a conference paper at COLM 2025
GSM8K
ProntoQA
ProsQA
Method
Acc. (%)
# Tokens
Acc. (%)
Tokens
Acc. (%)
# Tokens
CoT
42.9
25.0
98.8 =0.8
92.5
77.5 +1.9
49.4
No-CoT
10.5
2.2
93.8 =0.7
3.0
76.7 11.0
8.2
iCoT
30.0*
2.2
10.3
98.2 10.3
8.2
Pause Token
16.4 111
2.2
77.7 121.0
3.0
75.9 10.7
8.2
COCONUT (Ours)
34.1 +155
8.2
99.8 =0.2
9.0
97.0 =0,3
14.2
w/o curriculum
14.4 +0.8
8.2
=0.4
9.0
76.1 =02
14.2
w/o thought
21.6 +0.5
2.3
99.9
=0.1
3.0
95,5
8.2
pause thought
24.1 4.0.7
2.2
100.0
40.1
3.0
96.6 +0.88
8.2
Table 1: Results on three datasets: GSM8k, ProntoQA and ProsQA. Higher accuracy indicates stronger reasoning ability, while generating fewer tokens indicates better efficiency. "The result is from Deng et al. (2024).
more distracting branches. We use one continuous thought for each reasoning step (i.e.,
c = 1), and set the number of training stages to 6.
More details of datasets and training settings are described in Appendix A and Ap- pendix B.3.
5.2 Baselines and Variants of COCONUT
We consider the following baselines: (1) CoT, and (2) No-CoT, which were introduced in Section (3) iCoT (Deng et al., 2024): The model is trained with language reasoning chains and follows carefully designed schedule that "internalizes" CoT. As the training goes on, tokens at the beginning the reasoning chain are gradually removed until only the answer remains. During inference, the model directly predicts the answer. (4) Pause token (Goyal et al., 2023): The model is trained using only the question and answer, without a reasoning chain. However, different from No-CoT, special <pause> tokens are inserted between the question and answer, which provides the model with additional computational capacity to derive the answer. The number of <pause> tokens is set the same continuous thoughts in COCONUT.
We also evaluate some variants of COCONUT: (1) w/o curriculum, which directly trains the model in the last stage. The model uses continuous thoughts to solve the whole problem. (2) w/o thought: We keep the multi-stage training, but don't add any continuous latent thoughts. While this is similar to iCoT in the high-level idea, the exact training schedule is set to be consistent with COCONUT, instead of iCoT, for strict comparison.
(3) Pause as thought: We use special <pause> tokens to replace the continuous thoughts, and apply the same multi-stage training curriculum as COCONUT.
5.3 Results and Discussion
We show the overall results on all datasets in Table 1. Using continuous thoughts effectively enhances LLM reasoning over the No-CoT baseline. For example, by using continuous thoughts, COCONUT achieves 34.1% accuracy on GSM8k, which significantly outperforms No-CoT (16.5%). We list several key conclusions from the experiments as follows. More discussions are in Appendix B.7.
"Chaining" continuous thoughts enhances reasoning. Language CoT proves to increase the effective depth of LLMs and enhance their expressiveness (Feng et al., 2023). Thus, generating more tokens serves as a way to inference-time scaling for reasoning (Guo et al., 2025; Snell et al., 2024). This desirable property holds naturally for COCONUT too. On GSMSk, COCONUT outperformed other architectures trained with similar strategies, includ- ing COCONUT (pause thought) and COCONUT (w/o thought). Particularly, it surpasses the

---

Published as a conference paper at COLM 2025
latest baseline iCoT (Deng et al., 2024), which requires more carefully designed training schedule.
Additionally, experimented with adjusting the hyperparameter which controls the number of latent thoughts corresponding to one language reasoning step (Figure 7, II). As we increased from 0 to to 2, the model's performance steadily improved. This further validates the potential of continuous thoughts to scale up to harder problem. In two other synthetic tasks, we found that the variants of COCONUT (w/o thoughts pause as thought), and the iCoT baseline also achieves impressive accuracy. This indicates that the model's computational capacity may not be the bottleneck in these tasks. In contrast, GSM8K involves more complex contextual understanding and modeling, placing higher demands on computational capability.
Continuous thoughts are efficient representations of reasoning. Compared to traditional CoT, Co- CONUT generates fewer tokens while achieving higher accuracy on ProntoQA and ProsQA (Ta-
(I) Reasoning Efficiency on GSM8k
Accuracy (%)
40
Language
Continuous
ble 1). Although COCONUT does not surpass CoT on GSM8k, it offers superior trade-off between reason- ing efficiency and accuracy (Figure 7, 1). To illustrate this, we train series of CoT models that progres- sively skip (or "internalize" (Deng et al., 2024)) the initial {0, 1,,,,,,,,ALJ reasoning steps, and plot their accuracy versus the number of generated to- kens (labeled as "language" in the figure). These CoT models quickly lose accuracy as the token gener- ation budget decreases. In contrast, by applying Co- CONUT training strategy-replacing each language reasoning step with two continuous thoughts- the accuracy drop is substantially mitigated, maintain- ing higher performance even when fewer tokens are generated. Another interesting observation is that, when we decode the first continuous thought, it often corresponds to possible intermediate variables in the calculation (Figure 9). This also suggests that the con- tinuous thoughts are more efficient representations of reasoning.
30
20
10
Generated Tokens
(II) Coconut GSM8k
Accuracy (%)
35
30
0
2
# Thoughts per step (c)
Figure 7: Efficiency comparison of reasoning space and COCONUT with difference c.
6 Conclusion
In this paper, we introduce COCONUT, a new paradigm for reasoning in continuous latent space. Experiments demonstrate that COCONUT effectively enhances LLM performance across variety of reasoning tasks. Reasoning in latent space gives rise to advanced emergent behaviors, where continuous thoughts can represent multiple alternative next steps. This enables the model to perform BFS over possible reasoning paths, rather than prematurely committing to a single deterministic trajectory as in Chain-of-Thought (CoT) reasoning. Further research is needed refine and scale latent reasoning to pretraining, which could improve generalization across a broader range of reasoning challenges. We hope our findings will spark continued exploration into latent reasoning, ultimately advancing the development of more capable machine reasoning systems.
Acknowledgements
The authors express their sincere gratitude to Jihoon Tack for his valuable discussions throughout the course of this work.
We discuss the case of larger in Appendix B.8.
10

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar Xi1::33033008774 2023.
Marie Amalric and Stanislas Dehaene. A distinct cortical network for mathematical knowl- edge in the human brain. Neurolmage, 189:19-31, 2019.
Loic Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta Costa-jussà, David Dale, et al. Large concept models: Language modeling in sentence representation space. Xrviv preprint arXi::2412.0822 2024.
Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. Xiv preprint ar Xiv::2406. 12775, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. Xrv preprint arXit:2110.1416. 2021.
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint ar Xire:2311.01460, 2023.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arviv preprint or Xrxo:24405.1483 14838, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar xxX::2407.21783 2024.
Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. Looped transformers for length generalization. Xiv preprint ar arXiv:2409.15647, 2024.
Evelina Fedorenko, Michael K Behr, and Nancy Kanwisher. Functional specificity for high- level linguistic processing in the human brain. Proceedings of the National Academy of Sciences, 106339111422811111166 16433, 2011.
Evelina Fedorenko, Steven T Piantadosi, and Edward AF Gibson. Language primarily tool for communication rather than thought. Nature, 630(8017):555-586, 2024.
Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36, 2023.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint ar Xivv:2404.03688 2024.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint drXin=2502.05171, 2025.
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dim- itris Papailiopoulos. Looped transformers as programmable computers. In International Conference on Machine Learning, pp. 11398-11442. PMLR, 2023,
Alexi Gladstone, Ganesh Nanduru, Md Mofijul Islam, Peixuan Han, Hyeonjeong Ha, Aman Chadha, Yilun Du, Heng Ji, Jundong Li, and Tariq Iqbal. Energy-based transformers are scalable learners and thinkers. ar Xiv preprint trXiv:2507.0200 2025.
11

---

Published as a conference paper at COLM 2025
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Syn- naeve. Better & faster large language models via multi-token prediction. arXiv preprint #rXiv:2404.19737, 2024.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arxiv preprint arXiv:2310.02226, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint XrX::22501.12948, 2025.
Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget-aware Ilm reasoning. arXiv preprint ar XrXi:2241..18577 2024.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXio:2305.14992, 2023.
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Lim reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. arXio preprint ar arXiv:2404.05221, 2024.
Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. ar Xiv preprint ar Xio:24033004422 2024.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
Yann LeCun. path towards autonomous machine intelligence version 2022--0027- Open Review, 62(1):1-62, 2022.
Lucas Lehnert, Sainbayar Sukhbaatar, Paul Mevay, Michael Rabbat, and Yuandong Tian. Beyond a:: Better planning with transformers via search dynamics bootstrapping. Xiv preprint ar Xiv:2402.14083. 2024.
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers trans- formers to solve inherently serial problems. ar Xiv preprint ar ArX:::2402. 12875, 2024.
Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango. arXiv preprint ar axXiv:2209.07686, 2022.
William Merrill and Ashish Sabharwal. The expresssive power of transformers with chain of thought. arviv preprint ar XrXiv:2310007923 2023.
Martin M Monti, Daniel N Osherson, Michael J Martinez, and Lawrence M Parsons. Func- tional neuroanatomy of deductive inference: language-independent distributed network. Neuroimage, 3703):0055-1066 2007.
Martin M Monti, Lawrence M Parsons, and Daniel N Osherson. The boundaries of language and thought in deductive inference. Proceedings of the National Academy of Sciences, 106(30): 12554-12559, 2009.
Martin M Monti, Lawrence M Parsons, and Daniel N Osherson. Thought beyond language: neural dissociation of algebra and natural language. Psychological science, 23(8):914-922, 2012.
Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicola- maria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on llm reasoning and cost. Xiv preprint arXio:2407.1988 19825, 2024.
12

---

Published as a conference paper at COLM 2025
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXio:2404.15558, 2024.
Korbinian Pöppel, Maximilian Beck, and Sepp Hochreiter. Flashrnn: 1/o-aware optimization of traditional mnss on modern hardware. In The Thirteenth International Conference on Learning Representations, 2025.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thought. arXiv preprint Xiio:2210.01240, 2022.
Yuval Shalev, Amir Feder, and Ariel Goldstein. Distributional reasoning in Ilms: Parallel reasoning processes in multi-hop reasoning. arviv preprint arXir:2406.13858 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ar preprint ar drXio:2402.03300, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arviv preprint arXiv:2408..03: 2024.
DiJia Su, Sainbayar Sukhbaatar, Michael Rabbat, Yuandong Tian, and Qinqing Zheng, Dualformer: Controllable fast and slow thinking by learning with randomized reasoning traces. Xiv preprint ar arXto:2410.09918 2024.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't always say what they think: unfaithful explanations in chain-of-thougg prompting. Advances in Neural Information Processing Systems, 36, 2024.
Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. Xiv preprint Xrx::2212.1000 2022.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426-9439, 2024.
Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessandro Sordoni. Guiding language model reasoning with planning tokens. Xiv preprint Xxx::2310.05707, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2455 2022.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min-Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36, 2023.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint ar arXiv.2402.16837, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Adoances in Neural Information Processing Systems, 36, 2023.
Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Efficient training of Ilm policy with divergent thinking. Xiv preprint ar 2rXiv:2406.0567) 2024a.
13

---

Published as a conference paper at COLM 2025
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint urXio:2399.12284 2023.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arxiv preprint ar Xix:22407 0.6002., 2024b.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arviv preprint ar Xxtx:2309.05655 2023.
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Good- man. Quiet-star: Language models can teach themselves to think before speaking, arv preprint ar drXiv:2403.09629, 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint ar arXiv:2205.100 10625,, 2022.
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by superposition: A theoretical perspective on chain of continuous thought. Xiv preprint arXiv=2505.12514, 2025.
A Datasets
A.1
Examples
We provide some examples of the questions and CoT solutions for the datasets used in our experiments.
GSM8k
Question "John cuts his grass to inches. grows .5 inches per month. When it gets to inches he cuts it back down to 2 inches. It cost $100 to get his grass cut. How much does he pay per year?" Steps = ["<4-2=2*", *<2.55=4", "<12/4=3x", "<100+3=300>"]
Answer "300"
ProntoQA
Question = "Brimpuses are not luminous. Shumpuses are amenable. Each yumpus is lorpus. Gorpuses are shumpuses. Each zumpus is a grimpus. Gorpuses are rompuses. Dumpuses are not floral. Lempuses are cold. Brimpuses are impuses. Every lorpus is floral. Every rompus is transparent. Grimpuses are muffled. Rompuses fast. Wumpuses are bitter. Every sterpus is orange. Each lorpus is a vumpus. Yumpuses are feisty. Each yumpus is lempus. Gorpuses are gorpuses. Every lorpus is a sterpus. Stella is a brimpus. Stella is zumpus. True or false: Stella is not floral."
yumpuses. Rompuses are wumpuses. Zumpuses are
snowy. Zumpuses
Steps = ["Stella is a zumpus. Zumpuses are gorpuses." " "Stella is a gorpus. Gorpuses are rompuses.". " "Stella "Stella is a yumpus. Each yumpus is lorpus.", "Stella is lorpus. Every lorpus floral.". "Stella is floral.""
a rompus. Rompuses are yumpuses."
Answer "False"
14

---

Published as a conference paper at COLM 2025
# Nodes
Edges
Len. of Shortest Path
# Shortest Paths
23.0
36.0
3.8
1.6
Table Statistics of the graph structure in ProsQA
Dataset
Training
Validation
Test
GSM8k 385,620
500
1319
ProntoQA ProsQA
9,000
200
800
17,886
300
500
Table 3: Statistics of the datasets.
ProsQA
Question = "Every shumpus is a rempus. Every shumpus is a yimpus. Every terpus is fompus. Every terpus is a gerpus. Every gerpus is a brimpus. Alex is rempus. Every rorpus is scrompus. Every rorpus is yimpus. Every terpus is a brimpus. Every brimpus Every shumpus is a timpus. Every yimpus is boompus. Davis is a shumpus. Every gerpus a Iorpus. Davis is a fompus. Every shumpus is a boompus. Every shumpus is rorpus. Every terpus is lorpus. Every boompus is timpus. Every fompus is a yerpus. Tom is dumpus. Every rempus is a rorpus. Is Tom lempus scrompus?"
a lempus. Tom is terpus.
Steps = ["Tom is a terpus.", "Every terpus is a brimpus.", "Every brimpus is lempus." "]
Answer "Tom is lempus."
A.2 Construction of ProsQA
To construct the dataset, we first compile a set of typical entity names, such as "Alex" and "Jack," along with fictional concept names like "lorpus" and "rorpus," following the setting of ProntoQA (Saparov & He, 2022). Each problem is structured as a binary question: "Is [Entity] |Concept A] or [Concept BI?" Assuming [Concept A] is the correct answer, build a directed acyclic graph (DAG) where each node represents an entity or a concept. The graph is constructed such that path exists from [Entity] to [Concept A] but not to [Concept BJ.
Algorithm 1 describes the graph construction process. The DAG is incrementally built by adding nodes and randomly connecting them with edges. To preserve the validity of the binary choice, with some probability, we enforce that the new node cannot simultaneously serve as descendant to both node 0 and 1. This separation maintains distinct families of nodes and balances their sizes prevent model shortcuts.
After the graph is constructed, nodes without parents are assigned entity names, while other nodes receive concept names. To formulate question of the form "Is [Entity] [Concept A] or [Concept BJ?"", we designate node 0 in the graph as |EEtity], as |Concept AJ, and a leaf node labeled 2 as [Concept BJ. This setup ensures a path from [Entity] to [Concept A] without any connection to [Concept B], introducing moderately complex reasoning path. Finally, to avoid positional biases, [Concept A] and [Concept B] randomly permuted in each question.
leaf node labeled
A.3 Statistics
We show the size of all datasets in Table 3.
15

---

Published as a conference paper at COLM 2025
Algorithm 1 Graph Construction for ProsQA
edges
nodes - [0,1}
labels {0: 1,]
Labels: 1 (descendant of node 0), 2 (descendant of node 1), 3 (both), 0 (neither).
groups + {0: 1 10),2: (1).3: (}
idx t- 2
while idx N do
For each new node, randomly add edges from existing nodes n in nodes t- poisson(1.5)
rand random()
if rand < 0.35 then
candidates groups(0) U groups[1]
Cannot be descendant of node 1.
else if rand <. 0.7 then
candidates groups[0] U groups[2]
Cannot be descendant of node 0.
else
candidates nodes
end if
n n_in m_in_nodes nodes <- min(len(candidaales) n m_i_inodi_iii__es
weights +- (depth_to_root(( 1.5 + Vc E candidafes]
> Define sampling weights to prioritize deeper nodes. This way, the solution reasoning chain is expected to be longer. in_nodes + random_choicel achhiceccandidates, n_ n_in_nodes, prob = weights sum(weights))
cur label +- 0
for in_idx in_nodes do
cur_label cur label labels(in idx]
Update label using bitwise OR.
edges.append (in_idxx idx))
end for
groups(cur _abell append labels[idx) <- cur lahel nodes &- nodes LJ {idx}
idx t- idx +1
end while
16

---

Published as a conference paper at COLM 2025
First thoughts
Second thoughts
0.6
volue
0.4
0.4
0.2
0.2
Top
Top
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Percentlee
Percenalle
Figure 8: Analysis of parallelism in latent tree search. The left plot depicts the cumulative value of the top-1, top-2, and top-3 candidate nodes for the first thoughts, calculated across test cases and ranked by percentile. The significant gaps between the lines reflect the model's ability to explore alternative latent thoughts in parallel. The right plot shows the corresponding analysis for the second thoughts, where the gaps between lines are narrower, indicating reduced parallelism and increased certainty in reasoning as the search tree develops. This shift highlights the model's transition toward more focused exploration in later stages.
B
More Discussion on Empirical Results
B.1 COCONUT Generates Better Reasoning Chains
As shown in Figure 3, even when COCONUT is forced to generate a complete reasoning chain, the accuracy of the answers still higher than CoT. The generated reasoning paths are also more accurate with less hallucination. From this, we can infer that the training method of mixing different stages improves the model's s ability to plan ahead. The training objective of CoT always concentrates on the generation of the immediate next step, making the model "shortsighted". In later stages of COCONUT training, the first few steps are hidden, allowing the model to focus more future steps. This is related to the findings of Gloeckle et al. (2024), where they propose multi-token prediction as a new pretraining objective to improve the LLM's ability to plan ahead.
B.2 Analysis Parallelism in Latent Tree Search
We present an analysis of the model's ability to explore alternative latent thoughts in parallel. As shown in Figure 8, the model makes use of the latent space to explore multiple paths in parallel, and this ability is more pronounced in the early stages of the search.
B.3 Training Details
Math Reasoning. By default, we use latent thoughts (i.e., 2) for each reasoning step. We analyze the correlation between performance and in Section 5.3. The model goes through stages besides the initial stage. Then, we have an additional stage, where we still use 3 x continuous thoughts as in the penultimate stage, but remove all the remaining language reasoning chain. This handles the long-tail distribution of reasoning chains longer than 3 steps. We train the model for 6 epochs in the initial stage, and 3 epochs in each remaining stage.
Logical Reasoning. We use continuous thought for every reasoning step (i.e., c 1). The model goes through 6 training stages in addition to the initial stage, because the maximum number of reasoning steps is 6 in these two datasets. The model then fully reasons with continuous thoughts to solve the problems in the last stage. We train the model for 5 epochs per stage.
For all datasets, after the standard schedule, the model stays in the final training stage, until the 50th epoch. We select the checkpoint based the accuracy on the validation set. For
17

---

Published as a conference paper at COLM 2025
180
180"
0.20
LM head
0.13
Large Language Model
<bot>
sprints
runs 60 meters each sprint. How many total meters does he run week?
Figure 9: A case study where we decode the continuous thought into language tokens.
inference, we manually set the number of continuous thoughts to be consistent with their final training stage. We use greedy decoding for all experiments.
B.4 Hyperparameter Searching
We perform hyperparameter search over learning rates in the set 1 x 10 10-1, x 10-5 and batch sizes in 32, 64, 128, 256. The maximum num- ber of training epochs set to 50, as we observe convergence across all runs within this limit. Additionally, we tune the number of training epochs per stage for COCONUT over the set 3,5,5 individually for each task.
B.5 Clock-Time Reasoning Efficiency Metric
We present a clock-time comparison evaluate reasoning efficiency. The reported values represent the average inference time per test case (in seconds), with batch size of 1, measured on an Nvidia A100 GPU. For the no-CoT and CoT baselines, we employ the standard generate method from the transformers" library. Our results show that clock time is generally proportional to the number of newly generated tokens, detailed in Figure 7.
Method
GSM8k
ProntoQA
ProsQA
No-CoT
0.03
0.03
0.08
CoT 0.26
0.85
0.47
COCONUT
0.09
0.11
0.15
Table 4: Inference time (in seconds) comparison across tasks and methods.
B.6 Interpretation of Continuous Thoughts
In Figure 9, we show a case study where we decode the continuous thought into language tokens. The first continuous thought can be decoded into tokens like "180", 180" (with a space), and "9" Note that, the reasoning trace for this problem should be 3 x x 60 9 x 60 540, x 3 60 3 180 540. The interpretations of the first thought happen to be the first intermediate variables in the calculation. Moreover, it encodes a distribution of different traces into the continuous thoughts. This is consistent the analysis in Section 4.3.
B.7 More Discussions on Empirical Results
Performance Differences among Different Datasets. We discuss the performance dif- ferences among different datasets, to understand which tasks benefit more from latent reasoning.
18

---

Published as a conference paper at COLM 2025
Real-World vs. Synthetic Domains: GSM8k represents real-world, open-domain question-answering task. Unlike the synthetic datasets used in our study, it de- mands more complex contextual understanding and modeling, which can place greater demands on computational capabilities. This hypothesis is supported by the observation that CoCONUT outperforms all other latent reasoning methods, and its accuracy steadily improves as the number of thoughts per step (c) increases from 0 to 2. Additionally, GSM8k requires diverse commonsense and world knowledge. This may give CoT an advantage, as it aligns closely with the pretraining objectives of the underlying language model, enabling it to better leverage its knowledge compared to COCONUT.
Complex reasoning tasks often require the model to "look ahead" to determine whether particular step is optimal (also known as planning). Among the datasets in our experiments, GSM8k involves grade-school-leve math word problems that allow for intuitive judgment of the next reasoning step. Simi- larly, ProntoQA includes distracting branches of limited size, making it relatively straightforward to identify the correct next step. In contrast, ProsQA, based on randomly generated Directed Acyelic Graph (DAG) structure, presents a signifi- cant challenge to the model's planning abilities. Our experimental results suggest that tasks requiring extensive planning benefit more from latent space reasoning (including COCONUT, some of its variants, and iCoT) than from reasoning using language tokens (CoT).
Planning Requirements:
The LLM still needs guidance to learn continuous thoughts. In the ideal case, the model should learn the most effective continuous thoughts automatically through gradient descent on questions and answers (i.e., COCONUT w/o curriculum). However, from the experimental results, we found the models trained this way do not perform any better than no-CoT. With the multi-stage curriculum which decomposes the training into easier objectives, COCONUT is able to achieve top performance across various tasks. The multi-stage training also integrates well with pause tokens (COCONUT- pause thought). Despite using the same architecture and similar multistage training objectives, we observed small gap between the performance of iCoT and COCONUT (w/o thoughts). The finer-grained removal schedule (token by token) and few other tricks in iCoT may ease the training process. We leave combining iCoT and COCONUT as a future work. While the multi-stage training used for COCONUT has proven effective further research is definitely needed to develop better and more general strategies for learning reasoning in latent space, especially without the supervision from language reasoning chains
B.8 Using More Continuous Thoughts
In Figure 7 (II), we present the performance of COCONUT on GSM8k using E 20, When experimenting with c 3, we observe a slight performance drop accompanied by increased variance. Analysis of the training logs indicates that adding three continuous thoughts at once particularly during the final stage transition - leads to a sharp spike in training loss, causing instability. Future work will explore finer-grained schedules, such as incrementally adding continuous thoughts one at a time while removing fewer language tokens, as in iCoT (Deng et al., 2024). Additionally, combining language and latent reasoning- e-g. generating the reasoning skeleton in language and completing the reasoning process in latent space-cou could provide promising direction for improving performance and stability.
B.9 COCONUT with Larger Language Models
We experimented with COCONUT on GSMSk using Llama 3.2-3B and Llama 3-8B (Dubey et al., 2024) with 1. We train them for 3 epochs in Stage followed by 1 epoch per subsequent stage. The results are shown in Table 5.
We observe consistent performance gains across both Llama 3.2-3B and Llama 3-8B models compared to the no-CoT baseline, though these improvements are not as pronounced as those previously demonstrated with GPT-2. One possible reason is that larger models have
19

---

Published as a conference paper at COLM 2025
Model
no-CoT
CoCoNUT (Ours)
Llama 3.2-3B
26.0
31.7
Llama 3-8B
42.2
43.6
Table 5: Experimental results of applying COCONUT to larger Llama models. We report per- formance comparisons between models without CoT reasoning (no-CoT) and our proposed COCONUT method.
already undergone extensive language-focused pre-training, making the transition to latent reasoning more challenging.
We emphasize that the primary goal of this paper is to highlight the promising attributes of latent-space reasoning and initiate exploration in this new direction. Universally surpassing language-based CoT likely requires significant research efforts dedicated to latent space pre-training. We are encouraged by recent progress in this area (Geiping et al., 2025; Barrault et al., 2024). While these recent models provide scalable methods for latent representation learning, their latent spaces have not yet been explicitly optimized for reasoning. Integrating these recent advancements with COCONUT presents exciting and promising avenue for future research.
20


Reviews:
[{"review_id": "EIWVmqnq5H", "reviewer": "Reviewer_BZuh", "summary": "This paper introduces a novel reasoning paradigm for LLMs called Coconut (Chain of Continuous Thought), which departs from conventional language-based reasoning (e.g., CoT) by operating directly in the latent embedding space of the LLM rather than the language space. Instead of decoding a hidden state into a token, Coconut feeds the hidden state back into the model as the next input, enabling latent-space reasoning steps that are unconstrained by language syntax or semantics. The authors demonstrate that this mechanism supports breadth-first search-style planning, encodes multiple reasoning paths simultaneously, and achieves superior results to CoT on logical and planning-intensive tasks such as ProntoQA and GSM8K. Coconut is shown to improve both accuracy and efficiency, with fewer reasoning steps and better final outcomes.\n\nPros:\nThis paper draws inspiration from human brains. It extends the LLMs to have the capability to think as it desires. The basic idea seems to be very impactful and transformative.\n\nCons:\nThe current results are quite premature. The GSM results are much worse at this point.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper draws inspiration from human brains. It extends the LLMs to have the capability to think as it desires. The basic idea seems to be very impactful and transformative.", "reasons_to_reject": "I think the current results have not fully unleashed its potential. However, I don't think it's a deal breaker. I think this paper provides very good insights.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MvuZiywBRH", "reviewer": "Reviewer_eCiG", "summary": "The paper proposes a novel methodology to enable LLMs to reason/think in latent space instead of language tokens as is commonly done by CoT. \nThey purport the benefits of thinking in a latent space are that a model need not commit to a specific path ahead of time and can explore tree-like search over many paths at once. Secondly, it can also be more efficient to think in the latent space where efficiency is measured in terms of the number of forward passes of the Transformer required to achieve a certain quality.\n\nThe proposed method, COCONUT (Chain Of CONtinUous Thought), has two thinking modes - language space and latent space. In the latent space, instead of decoding the final embedding into an explicit token, we pass it as is to the next forward pass of the transformer. \nTo train COCONUT, the authors implement a multi-stage curriculum as follows. We start with a model trained on regular CoT data. We assume the CoT data splits into a number of reasoning steps. In each stage we replace the first k reasoning steps, with kc continuous thoughts (where c is a hyperparameter controlling the number of latent thoughts per reasoning step). Then the loss is optmized while masking the question and the continuous thoughts.\nSpecial beginning and end marker tokens are used to demarcate between continuous thought and language thought modes.\nThe authors then proceed to show the efficacy of COCONUT first on a synthetic dataset ProsQA and then on a more general LLM reasoning task GSM8k.\n\n**ProsQA analysis**:\nThe dataset requires the model to perform a graph search on a DAG to figure out the path to the answer. The authors take a pre-trained GPT-2 model and apply COCONUT fine-tuning on top of it specific to the ProsQA dataset. Compared to CoT where the model is trained to commit to a path right from the beginning, COCONUT allows the model to explore many different paths in a BFS-like manner and achieve a higher final accuracy of reaching the goal.\nThe authors also perform an interesting interpretability analysis to support that COCONUT's continuous thoughts are performing a tree-like search.\n\n**General Reasoning Tasks**:\nThe authors focus on GSM8k and PRontoQA as the reasoning datasets for this part of the analysis. They compare with a CoT, No-CoT and iCot baselines. In addition, they ablate performing COCONUT without the stage-wise curriculum, and with a fixed pause token as the thought as well.\nOverall, they find that on GSM8k, COCONUT significantly improves over No-CoT (but still underperforms CoT). However at a fixed token budget it is able to beat CoT since CoT requires a lot of intermediate tokens for its superior accuracy to show up.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea is novel and the thoughtful and detailed experiments with the synthetic dataset show that the approach indeed has the ability to teach the model to explore multiple solution paths in parallel which is main drawback of CoT reasoning.\n\n1. The paper studies an important and relevant problem. It attempts to address a focal drawback of CoT reasoning which can have a significant impact on the space of LLMs. Moreover, the paper is well-written and easy to read.", "reasons_to_reject": "Overall, it feels like although the paper introduces a very nice technique, the experiments showing its efficacy are limited in scope.\n\n1. The insightful analysis seems limited to only 2 datasets GSM8k and ProsQA out of which one is purely synthetic. ProntoQA seems very saturated at their model scale as even No-CoT gets over 90% accuracy. \n\n1. Apriori it is unclear how the method can be applied to settings where CoT does not neatly decompose into reasoning steps.\n\n1. It is also not clear whether the method can be applied in a general manner for all reasoning tasks or it needs dataset specific training for each dataset. Evidence for the possibility of the former could be built by evaluating on a wider suite of simple reasoning tasks.", "questions_to_authors": "1. Could you elaborate more on the KV cache reuse you mention in line 153?\n\n1. Could you also elaborate a bit on the different techniques you tried for changing optimizer state across stages?\n\n1. Can this method work in the absence of CoT data?\n\n1. Does this need to be done for each dataset separately? A more general recipe?", "extra_scores": {}}, {"review_id": "AwGjAbS9MX", "reviewer": "Reviewer_NeU4", "summary": "The paper proposes a new reasoning paradigm for language models, called COCONUT (Chain of Continuous Thought), which allows reasoning in a continuous latent space rather than traditional language-based chains of thought (CoT). By using the last hidden state as an input embedding for subsequent reasoning steps, COCONUT avoids language constraints, enhancing reasoning efficiency. The authors use a multi-stage curriculum training scheme that gradually shifts from language-based reasoning to continuous thoughts, helping models learn effective latent reasoning representations. Experiment results with pre-trained GPT2 on synthetical datasets show that COCONUT is most effective in planning intensive tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. One of the early works in latent CoT reasoning. Many follow-ups after this.\n\n2. Sufficient analysis on accuracy, efficiency, and possible tree-search related explanation.\n\n3. Show better performance on GSM8K than previous latent CoT methods.", "reasons_to_reject": "1. Limited effectiveness in terms of performance on real-world reasoning task, GSM8K. \n\n2. Only GPT2 model is used in the experiments.\n\n3. The proposed method directly feeds the last layer output representation as the input embeddings, which makes the training procedure non-parallelizable sequence-wise, as in normal language models. Might affect scalability.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "W1ShHOsuA7", "reviewer": "Reviewer_Hbwo", "summary": "The paper proposes a new approach for Continuous CoT, feeding the output token before the unembedding layer to the first layer at the next timestep, thereby creating a differentiable pathway across timesteps. The method leverages an iterative training procedure, where standard CoT steps are progressively replaced from the left with continuous ones. The authors analyze the behavior of the learned system, and draw connections with search-based procedures, showing the model can encode a distribution over next steps, rather than committing to a single one. This demonstration is aided by a reasoning dataset proposed by the authors, ProsQA. \n\nThe authors show that latent space reasoning is able to make a better use of the compute allocation across tokens, showing that for a fixed token budget, it outperforms standard CoT.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors explore the relatively new field of continuous reasoning, and show promising results. \n2. The paper is clearly presented, and overall easy to follow. The authors do a good job at investigating the underlying mechanisms of Coconut, designing targeted experiments and ablations\n3. The authors propose a new synthetic dataset, ProsQA, to aid understsanding of reasoning in search-based settings", "reasons_to_reject": "1. The proposed training procedure is convoluted, requiring multiple training phases\n2.  The connection to BFS is somewhat flaky; the authors argue that Coconut \"allows the model to perform a breadth-first search (BFS) to solve the problem, instead of prematurely committing to a single deterministic path like language CoT.\" To this end, the authors show one example of a BFS-like behavior with the model. The authors allude to observing this beyond this single example. Is there a way to quantify this, or to show this repeating beyond a single example ? The current paper lacks strong evidence to support the BFS claim in its current form.", "questions_to_authors": "1. Is the observed BFS-like behavior a property of Coconut on the given ProsQA dataset, or do you believe this behavior to emerge on other reasoning datasets not explicitly built around search ? \n2. For figure 3, was greedy decoding to generate the potential valid generation ?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Coconut, a novel and impactful approach for LLMs to perform reasoning in the continuous latent space, rather than solely through language tokens. The paper has received unanimously positive reviews.

The method enables more efficient and parallel exploration of reasoning paths, akin to breadth-first search. The detailed analysis, particularly with the synthetic ProsQA dataset, provides compelling evidence for Coconut's unique capabilities in managing multiple reasoning trajectories.

While reviewers initially raised valid concerns regarding current empirical performance on real-world tasks like GSM8k (though showing significant improvement over No-CoT), the reliance on dataset-specific training, and the use of GPT-2 only, the authors provided thorough and convincing rebuttals in the discussion period. They effectively framed these limitations as inherent to a pioneering work in an emergent field, outlining clear paths for future research (e.g., large-scale latent space pre-training, applicability to larger models like Llama 3 with new results, and system-level optimizations for scalability). The authors' additional experiments and clarifications further strengthened the paper, addressing most open questions. The reviewers recommended to study continuous thoughts further to fully realize the potential of this method. Additionally, it is recommended to study more domains beyond math and logical reasoning.

As such, I recommend acceptance.
Decision: Accept