Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
Authors: Eric Zelikman
Georges Raif Harik
Yijia Shao
Varuna Jayasiri
Nick Haber
Noah Goodman
Venue: COLM (2024)
Keywords: reasoning, internal monologue
TL;DR: Language models can teach themselves to reason using internal monologue.

Abstract: When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%→10.9%) and CommonsenseQA (36.3%→47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.

Published as a conference paper at COLM 2024
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
Eric Zelikman Stanford University ezelikman@cs. stanford. edu
Georges Harik Notbad AI Inc
Yijia Shao
Stanford University
georgesègmail.ed
shaoyj@cs. s stanf
Varuna Jayasiri Notbad Al Inc
Nick Haber Stanford University
Noah D. Goodman
Stanford University
ppjayasiri@gmail.c com
nhaber@sttafford.. edu
ngoodman@stant tanadmaan@ttaffrdd.eeem edu
Abstract
When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to correct answer. This is a highly constrained setting ideally, language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended forcing technique. Encouragingly, generated rationales disproportionatel] help model fifficullt-t-prediie tokens and improve the LM's s ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%~100.9%% 16.3%%+47..%% and observe perplexity improve-
the theory of mind underlying
and CommonsenseQA ment of difficult tokens natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks step towards LMs that can learn to reason in more general and scalable way.
"Life can only be understood backwards; but it must be lived forwards."
-Soenn Kierkegaard
1 Introduction
Much of the meaning of text is hidden between the lines: without understanding why statements appear in a document, a reader has only a shallow understanding, Moreover, this has been repeatedly shown to be true for LMs as well, in the contexts of tasks ranging from commonsense reasoning
theorem proving to programming (Wei al., 2022b; Nye et al., 2021; Zelikman et al., 2022; 2023a; Kojima et al., 2022). Reasoning about implications of text to predict later text has consistently been shown to improve LM performance on variety of tasks, but methods for allowing LMs to learn from their reasoning (e.g, Zelikman et al. 2022) have focused on solving individual tasks or predefined sets of tasks (e.g., Wei et al. 2021b). These works rely carefully curated datasets to provide either specific reasoning tasks or in some cases, the reasoning itself. We instead ask, if reasoning is implicit in all text, why shouldn't we leverage the task of language modeling to teach reasoning?
1

---

Published as a conference paper at COLM 2024
Original Text
Foouuhh
Though,
Troghh
Troug
moogg,
Teughs
Thoghh
Toought
!) Think
2) Talk
Helps Prediction
3) Learn
M Te
Sampled Thought
LM
2
Reward
Harms
Prediction
Sampled
Thought
LXG
Discard
Figure 1: Quiet-STaR. We visualize the algorithm as applied during training to a single thought. We generate thoughts, in parallel, following all tokens in the text (think). The model produces a mixture of its next-token predictions with and without a thought (talk). We apply REINFORCE, as in STaR, increase the likelihood of thoughts that help the model predict future text while discarding thoughts that make the future text less likely (learn). represents one that
Note the second sampled thought
useful in predicting "4".
In particular, the Self-Taught Reasoner (STaR, Zelikman et al. 2022) showed that LMs can bootstrap their reasoning ability on peestion-answeriige (QA) datasets by sampling rationales to attempt to answer questions, training rationales if they led to correct final answer, and then repeating this to iteratively solve more difficult problems. Yet, training from curated QA datasets limits the scale and generalizability of the rationales. QA datasets, especially high-quality ones, require thoughtful curation and will inherently only ever cover subset of reasoning tasks. Thus, we extend STaR instead of the LM learning to reason on particular tasks like mathematical QA, we train an LM to generate reasoning that helps infer future text from large internet text corpus. As result, we allow the LM to learn from the diverse tasks present in language (Weber et al., 2021). This builds on an intuition essential to the current language modeling paradigm, namely, that "language models are unsupervised multitask learners" (Radford et al., 2019). Thus, as in STaR, we leverage the LM's pre-existing reasoning ability to generate rationales and train the LM on them with REINFORCE-bas reward (Williams, 1992). We refer to this technique as Quiet-STall as it can be understood as applying STaR "quietly", training the model to think before it speaks.
Broadly, Quiet-STaR proceeds by generating rationales after every token to explain future text (think), mixing the future-text predictions with and without rationales (lalk), and then learning to generate better rationales using REINFORCE (learn). We apply Quiet-STaR to Mistral 7B (Jiang et al., 2023) using the web text datasets OpenWebMath (Paster et al., 2023) and Colossal Clean Crawled Corpus (C4, Raffel et al. 2020). We find that, even without dataset-specific fine-tuning, Quiet-STaR results in improvements to zero-shot direct- reasoning abilities on CommonsenseQA (4.33%+44722% and GSM8K
and that these improvements consistently increase with the number of tokens used in the LM's internal thoughts. Lastly, we qualitatively investigate patterns in the generated rationales.
In solving this task, we make the following contributions:
1. We generalize STaR to learn reasoning from diverse unstructured text data. To our knowledge, this is the first work explicitly training LMs to reason generally from text, rather than on curated reasoning tasks or collections of reasoning tasks.
2. We propose and implement a parallel sampling algorithm that makes our training procedure scalable, generating rationales from all token positions in given string.
3. We introduce custom meta-tokens at the start and end of each thought to allow the LM to learn that it should be generating a rationale and when it should make prediction based on that rationale.
4. We apply a mixing head retrospectively determine how much to incorporate the next-token prediction from a given thought into the current next-token prediction.
5. We show that non-myopic loss, including multiple tokens ahead for language modeling, improves the effect of thinking.
6. On multiple tasks, we demonstrate that thinking allows the LM to predict difficult to- kens better than one trained the same web text, improving with longer thoughts.
2

---

Published as a conference paper at COLM 2024
# #Toought Abead Tokens
CommonsenseQAA Accuracy (%)
Thought, Ahead Tokens
24, 12
46%
24, 12
GSM8K Accuracy (%)
16,8
16,8
12,
44%
12,4
10,
10.4
8,
42%
K,4
Boseline
Basching
40%
38%
36%
34%
10
30
50
70
80
90
100
10
20
40
60
80
90
100
Training Step
Training Step
(a) GSM8K
(b) CommonsenseQA
Figure 2: Generalization Results. We evaluate the extent to which the model trained with Quiet-STaR generalizes to directly answering problems that require reasoning. The left plot (a) shows the zero-shot accuracy GSM8K, while the right plot (b) shows the zero-shot accuracy on CommonsenseQA, without any fine-tuning. In both plots, the x-axis represents training steps, and each line corresponds a different number of thinking tokens used during Quiet-STaR training. The y-axis measures the zero-shot direct accuracy on the respective datasets. We also include an inference normalized version of this plot in Figure Note the baseline corresponds to naive continued pretraining.
2 Related Work
2.1 Reasoning in Language Models
There have been many works training and exploiting language models to solve difficult tasks by first training them to reason through them. For example, Rajani et al. (2019) fine- tuned language model human-written reasoning traces placed before multiple-choice answers to commonsense reasoning questions, finding it outperformed language model trained to directly predict the answers. Shwartz al. (2020) demonstrated that language models, when provided with some scaffolding, can generate these helpful chain-of-thoure solutions without additional supervision. Later, Nye al. (2021) demonstrated that "scratch- pads" required less scaffolding when the language models were more capable, result later reinforced by Wei et al. (2022b), emphasizing informal tasks, and further strengthened by Kojima et al. (2022), demonstrating this behavior could be accomplished zero-shot. Most recently, Wang Zhou (2024) showed further that for commonsense-qqueston answering, one could force language model to leverage chain-of-thouse from emitting any valid answer tokens unless was confident. However, once again, these approaches only work for question-answer dataset, and Wang Zhou (2024) relies on heuristics to identify when the model has output answer tokens. Somewhat like TRICE (Phan et al., 2023), we the relative improvements in the log-likelihood of the target text across rationales as an estimate of quality, but we simply subtract the mean reward and do not incorporate more complex control variates.
reasoning by preventing
2.2 Training Language Models Reason
One direction that researchers have used to train language models to reason or improve their reasoning training the language model on mined reasoning traces or reasoning-like data (Rajani et al., 2019; Wei et al., 2021a; Lewkowycz et al., 2022; Chung et al., 2022; Gunasekar et al., 2023). Although this approach has been demonstrated to be effective, it comes with drawbacks. It requires either manual annotation, which is sensitive to the capability of the annotators and off-policy for the language model (i.e., the distribution of reasoning is not text that the language model would otherwise likely have generated). This approach is also expensive, difficult to scale, and provides no clear path to solving problems harder than those that the annotators capable of solving.
3

---

Published as a conference paper at COLM 2024
Algorithm 1: Quiet Self-Taught Reasoner (Quiet-STaR)
Input: Language model l,, training steps num steps, sequence length I, thought length learning rate a, batch size b, number of thoughts Rthoughts: number of ground truth tokens used for supervising each thought Ntrue
Output: Language model 0 that generates rationales to predict future text
for i 0 to num_steps do
Sample batch of sequences X of length
h f- hidden_statesp, (X)
for j = 1 to in parallel using attention mask do
log pinit
+ Im_heade (hini
// Predict next tokens
fij+Mlrue
1:
T₁- - generate tokens, ( (X <start thought>|, Htthughts) // Generate thought
Ti [T; <ej=<tetthhouhtt
I1 thought
hidden ([xy; Tji Xirtttrne--
log pijtTrue + 1m_dead,, (h thought
thought
1:j4-Ruue thought
1/ Predict next tokens w/ thought
f mixing heady, (h Jiit
F. Mirue
talk log jijtiiur +
--jj:1 Mirue log pinit
Wjj Myrree
log P jij+
thought
// Mix logi ts
ENLL + log ylal
Pij++Miue (X (X)+1s++mmo+1)))
"j = log p talk
iif+Miue (X &iit
log P
plalk
(X (Xm+lej+
CREINFORCE t- - -1; > 0] V log Pe, (T)l <start
V - VelNuL NLLL + V CEINNOOR
0/+1 + 0 - a و Vel;
// Update model parameters
return 0 @num_steps
Another direction for teaching reasoning relies on a language model's own generated reasoning, which can be seen as building on large body of literature on self-play (Silver et ata,,,2177 Anthony et al., 2017; Polu & Sutskever, 2020). These include methods such as the Self-Taught Reasoner (Zelikman et al., 2022), which demonstrated that language model iteratively trained on its reasoning that led to correct answers could solve increasingly difficult problems, Later work aimed to leverage additional information or assumptions such Huang et al. (2022) which demonstrated that the algorithm proposed in STaR could still work if one assumed that the majority-vote answer was correct (although this has lower ultimate performance). Further work has generalized the results of Zelikman et al. (2022), such as Uesato et al. (2022) which demonstrated additional usefulness to "process-based" supervision where incorrect reasoning traces were filtered, recently V- STaR (Hosseini et al., 2024) that demonstrates that training a verifier to guide generation also improves performance, as well as TRICE (Hoffman et al., 2024) which maximizes the marginal likelihood of the correct answer given several reasoning traces per problem. Finally, related work has also explored learning intermediate reasoning in the constrained setting of making mathematical statements, where statements in the model's intermediate reasoning could be constrained to only be valid mathematical statements (Poesia et al., 2023). We include further discussion of related reasoning works in Appendix F.
2.3 Meta-tokens
Recently, growing body of work has demonstrated the usefulness of custom tokens optimized to perform specific functions in the context of a neural network for this reason, they have also been referred to as "function vectors." (Todd et al., 2023). One of the original instantiations of this was prompt-tuning (Lester et al., 2021) (and relatedly prefix-tuning (Li & Liang, 2021)), where the embeddings corresponding to the tokens of prompt could be optimized to better accomplish a task. Others have applied meta-tokens to compress long prompts (Li et al., 2023; Jung & Kim, 2023) for efficiency. Most relevant to this work, Mu et al.
4

---

Published as a conference paper at COLM 2024
(2024) optimized a token such that, when the tokens after it could not attend to the tokens before it (i.e., a context compression token), it would provide sufficient information to future tokens. Although we do not focus on compression, token that a affects attention and controls complex downstream behavior. In one related work, Goyal et al. (2023) show that learning single "pause" token (essentially representing each token as two tokens) improves LM performance. However, unlike the thought tokens in our work, this pause token does not initialize thought instead, can be seen acting as the entirety of the thought. We find that reasoning in language is significantly more helpful.
share the problem of learning
3 Problem Statement
In this work, we introduce an auxiliary 'rationale' variable between each pair of observed tokens of the sequence. We then aim to optimize language model with parameters 0 with the capacity to generate intermediate thoughts (or rationales) such that
e* arg max [logPe (Xim (ooi, rationaley (xoz))]
Note that, in principle, this provides no advantage over an optimal language model that already correctly models the language's distribution over strings. Yet, in practice, extensive prior work has shown that language models benefit from intermediate rationales on reason- ing tasks (Nye et al., 2021; Zelikman et al., 2022; Wei et al., 2022b). Some work has aimed to explain the effects of thain-of-though reasoning, namely attributing it to "locality of experience" (Prystawski et al., 2024). More broadly, reasoning allows model to decompose a challenging computation into smaller steps. In effect, we train the model to learn which decomposition and planning steps are effective in predicting future text. Also note that we formulate the objective as accurately predicting the remaining sequence, rather than only the next token. Once again, for an optimal LM these would be equivalent. However, we find that the non-myopic formulation leads to more effective loss for learning rationales.
4 Quiet-STaR
4.1 Overview
Quiet-STaR operates with three main steps (Figure 1):
1. Parallel rationale generation (think, Subsection 4.2): In parallel across π tokens x; in an input sequence X0:,, we generate y rationales of length f: c; = (ca..... Cit) resulting in x rationale candidates. We insert learned <|startofthoughh)>>>>> and < </endofthought/> tokens to mark each rationale's start and end.
2. Mixing post-rationale and base predictions (talk, Subsection 4.3): From the hidden state output after each rationale, we train "mixing head" - a shallow MLP produc- ing a weight determining how much the post-rationale next-token predicted logits should be incorporated compared to the base language model predicted logits. This approach eases distribution shift early in finetuning, due to introducing rationales.
3. Optimizing rationale generation (learn, Subsection 4.4): We optimize the rationale generation parameters (start/en) end tokens and LM weights) to increase the likelihood of rationales that make future text more probable. We use REINFORCE to provide a learning signal to rationales based on their impact on future-token prediction. To reduce variance, we apply a teacher-forcing trick to include in the loss the likelihood of predicting not only the token after the thought but also later tokens.
4.2 Parallel Generation
A key challenge in Quiet-STaR is efficiently generating rationales at each token position in the input sequence. Naively, this would require a separate forward pass for each token, which becomes computationally intractable for long sequences.
We allow for highly parallel generation by first observing that an inference pass of a language model produces probability distribution over the next tokens for all input tokens. Naturally,
5

---

Published as a conference paper at COLM 2024
Base Text
Thought Token
600
000
000
d
b"
c'
d'
b
Thought
a
b
d
Token
d
Thought Token 1
b
C
d'
b
Base
b
d'
Text
Generation Paths
Parallel Inference Mask
By constructing an attention mask that allows all thought tokens to pay attention to themselves, all preceding thought tokens within the same thought, and the preceding text, we can generate continuations of all of the thoughts in parallel. Each inference call is used to generate one additional thought token for all text tokens.
Figure 3: Parallel Generation.
this allows us sample one next token from each token in the input. If one has generated successor from each token, it is not possible to simply continue with the original sequence. For example, imagine predicting the next token after each token of "< bos > the cat sat" one might generate "yes orange saw down" each successor by itself is reasonable next token to a prefix of the sequence, but the list of tokens is a set "counterfactual" continuations of these prefixes. We can, however, leverage these continuations to generate hidden thoughts for each observed token.
To do this efficiently, cache each forward pass and concatenate diagonal attention mask to the previous attention mask: each generated token now attends to all of the tokens that used to generate Moreover, this parallelized next-sampling token procedure can be repeated arbitrarily many times (or at least, until one runs out of memory). We visualize this procedure in Figure and highlight additional ways to make this algorithm faster in Appendix B. In particular, in Figure: (left) each vertical bottom-to-top iteration corresponds to one horizontal (left-to- right) iteration of Figure 3 (right).
well to itself (but not to token on other "counterfactual" paths).
4.3 "Mixing" (Residual) Heads
When starting with pre-trained model, thoughts will initially be out of distribution, and hence harm language modeling performance. To smooth the transition to thinking, we introduce learned interpolation between the LM predictions with and without thoughts. Given the end-of-thought token's hidden state and the hidden state of the original text token, the mixing head outputs a weight that determines the extent which the post- thought prediction logits will be used. We use a shallow multi-layer perceptron for this head, outputting scalar for each token. We include implementation details Appendix A.
4.4 Optimizing Rationale Generation
4.4.1 Optimizing Start-of-Thought and End-of-Thought Tokens
The <|startofthought)> and </endofthought)> tokens serve learned meta-tokens that control the model's rationale generation. Optimizing the representation of these tokens, especially the < <staaooffhouuhht token, is crucial but challenging due to the discrete nature of the rationale tokens. We initialize the start and end token embeddings to the embedding corresponding to the dash, "---" which often appears in text data to denote pause or thought. This leverages the language model's preexisting knowledge. In addition, to allow these embeddings to be optimized more quickly, apply (hyperparameter) weight to

---

Published as a conference paper at COLM 2024
000
d
e
9
h
_LM
h PLM
i LLM
Thought
</t>
g
h
Figure 4: Forward Pass and Teacher Forcing. We visualize single forward pass of our algorithm. Solid lines denote language model computation, while dashed lines indicate tokens inserted via teacher forcing, and the mixer represents the mixing head. In particular, we visualize predicting three tokens ahead. Thought generation is shown in more detail in Figure 1 and Figure 3.
the gradients of these embeddings during the update step. Intuitively, the start thought tokens can be understood as putting the model into "thinking mode" and the end thought token can be understood as telling the model when it's done thinking.
4.4.2 Non-myopic Scoring and Teacher-forcing
Because we do not expect thoughts to be useful in predicting every token, we would prefer the model's reward to depend less on the exact next word in the text following the thought and more on the following semantic content. There are two primary challenges here. First, unlike in typical language modeling with transformers, only the thoughts corresponding to a given next-token prediction receive a gradient from that prediction consequence our parallel sampling strategy. We could address this by adding loss terms for future tokens by sampling the tokens before. However this would result in much higher entropy for language modeling in general and lower-quality generated text, because it would train the LM to par- tially disregard its preceding tokens. Instead, we use the parallel attention mask to compute the log probabilities of the true next tokens, applying teacher forcing by assuming the model selected the correct next ground-truth token (as implicit in normal language modeling with transformers). Note that the loss for each future token also depends on mixing weight computed from the end thought token and the previous observed token. The number of future tokens included in the loss hyper-parameter. We apply the same teacher-forcing technique to insert the start and end tokens. We visualize this procedure in Figure 4.
4.4.3 Objective
We use REINFORCE to optimize the likelihoods of the rationales based on their usefulness: the logeliielihood of the Ntre true next tokens Xm+ +1 under the language model given previous observed tokens and particular rationale (plalk Mirue shorthand for the mixed
prediction probabilities after thinking, see Algorithm 1). To reduce variance, we generate multiple rationale continuations for each token in the input sequence (loosely inspired by TRICE, Phan et al. (2023)). We thus define the reward for each rationale Tj as the difference palk P and the average across rationales for that token (plalk (p)sttmme))
between
= log Pij1 Merue (Xj+1i++tmm +1) - -log ttalk Mirue
talk
(Xj+1:f+Wrautt)
We then use this reward REINFORCE loss term to update the language model parame- ters 0 to increase the likelihood of rationales that perform better than the average:
Vol VoL RRINFORCE -rj- lolog po (
7

---

Published as a conference paper at COLM 2024
We found it useful to exclude the negative reward from the REINFORCE loss term, as it led to more stable training, though may introduce some bias.
This loss term encourages the model to generate rationales that improve its predictions of future tokens compared to the average prediction across all generated rationales for that token. The gradients from this loss are used to update both the LM parameters and the start-of-thought and end-of-thought token embeddings, with a (hyperparameter) weight applied to the gradients of the start-of-hhought and end-of-thought token embeddings to accelerate their optimization. By iteratively optimizing these parameters, Quiet-STaR trains the model to generate more useful rationales throughout training. Lastly, we also include log-likelihood loss, CNLL, ensure that the LM learns to optimize the talking heads and also receives a next-token prediction signal for the base LM head".
Experiments and Results
Intuitively, not all tokens require equal amounts of thought. For example, consider the sen- tence "the person is run-": although there is inevitably some probability of the token being something other than "ing" (For example, in this very text, the token following "run" is "-"), as a standalone sentence without context, additional thinking is unlikely to improve well- trained model's prediction. Indeed, we conjecture that for most chunks of most online text, additional thought has little to no impact. Indeed, early in our exploration, we observed that Quiet-STaR does not benefit all tokens equally. Thus, we design our experiments to inves- tigate whether our approach useful in predicting tokens that do require thought. We eval- uate 1) whether Quiet-STaR improves language model's 's ability to directly predict answers in datasets that require reasoning; and, 2) the distribution of impacts resulting from thinking tokens. We conduct all of our experiments starting with the base version of Mistral 7B (Jiang et al., 2023). We also include internal reasoning examples in Appendix J, for reference.
We perform most of our experiments by training on WpenWebMath (OWM) (Paster et al., 2023), a crawl that emphasizes more technical webpages. We selected OWM because we anticipated it would have higher density of tokens benefiting from reasoning, which our ex- periments support. We also evaluate Quiet-STaR on C4 (Raffel et al., 2020), a widely used LM pretraining corpus with more diverse text, and again show significant albeit smaller benefits.
5.1 Downstream Performance
In this subsection, we evaluate the extent to which Quiet-STaR improves the zero-shot rea- soning capabilities of the LM on CommonsenseeA (Talmor et al., 2018) and GSMSK (Cobbe et al,, On CommonsenseQA, we find that Quiet-STaR improves performance by 10.9% compared to the base LMM As shown in Figure 2, this improvement consistently increases with the number of tokens used in the LM's rationales, indicating that more thorough reasoning through the thought tokens is translating to better direct question-answering performance. Similarly, on GSMSK, Quiet-STaR results in a 5.0% boost over the base LM,, and once again, performance scales with the length of the rationales generated during Quiet-STaR training. For reference, in Figure 2, we include a baseline corresponding to train- ing the same LM on the same dataset without thought tokens. We observe that in multiple curves, performance appears to eventually deteriorate we anticipate this because we are not training these downstream tasks, the roles of the thought tokens may change over time. We also find a benefit of our non-myopic objective, which we discuss in Appendix D.
We find that training with Quiet-STaR on C4 (Raffel et al., 2020) also improves performance on GSM8K (5.9% - 8.1%) and CommonsenseeA (36.3% + 42.6%) but by a smaller margin. Specifically, for our C4 evaluation, we train Mistral 7B with 16 thought tokens and 4 true tokens ahead and otherwise the same setup.
We can compare these improvements to those offered by pause tokens (Goyal et al., 2023), which can be seen as a constrained version of Quiet-STaR where each token is represented by
1 Due to our linear mixing, equivalent shifting the mixing weight toward the base prediction.
8

---

Published as a conference paper at COLM 2024
Accuracy on GSM8K vs Number of Majority Vote Samples
50-
GSM8K Accuracy (%
Quict-STaR + CoT
Baseline (CoT)
40
30
2
3
6
8
Samples for Majority Vote
Figure Zero-shot performance on Quiet-STaR applied to thain-of-thought on GSMSK. We visualize how using Quiet-STaR trained Mistral model can improve chain-of-thought performance. We use an 8-thought-token-rrane -ooiient model and use its internal thoughts to improve the tokens in zero-shot chain-of-thought (Kojima et al,,, 2022)
two tokens and the second "pause" token acts the entirety of the thought. In particular, our setup is most comparable to their pause token fine-tuning, as we also finetune a pretrained model. Their results indicate that pause token fine-tuning provides minor gains over the base model on CommonsenseeAA observing an improvement from 26.9% to 28.8%; on GSM8K, Goyal et al. (2023) found that pause token fine-tuning harms performance. Moreover, on both tasks (and most of their evaluated tasks), they observed that additional thought tokens harmed performance. Moreover, they discuss the "lukewarm effect of pause-finetume standard-pretee pretrained model" (Goyal et al., 2023). This suggests that allowing the model to generate rationales leads to more effective reasoning compared to using fixed "pause" tokens. Note that unlike Goyal et al. (2023), we do not fine-tune on the downstream tasks.
Overall, these downstream results validate that training a language model to predict the subtext between the lines of general text data substantially improve its reasoning capabilities, even on datasets it was not explicitly trained on. The fact that longer rationales consistently lead to better outcomes, and that Quiet-STaR outperforms the constrained pause token approach, supports the notion that Quiet-STaR is successfully teaching the model to leverage its own generated thoughts to reason more thoroughly about the input.
5.2 Improvement Distribution
As visualized in Appendix Figure 7, find that average there is little improvement in the LM's ability to predict arbitrary tokens. But, when we visualize the distribution of relative improvements, there is disproportionate improvement on more difficult (i.e., high loss) tokens. This reflects that some text tokens are substantially harder and benefit more from careful thought.
In Appendix Figure 8, we aim to provide some insight into the kinds of tokens where the improvements occur. Namely, while thinking appears help for many tokens in the exam- ple, inspection suggests it disproportionately helps predict tokens where recalling relevant information is useful, such as the name of an applicable theorem or the start of a proof's next step. Notably, this would align well with the framing proposed by Prystawski et al. (2024).
5.3 Quiet-STaR and Shain-offTThoghh
While there are natural parallels between chain-of-thought prompting and our approach, they are orthogonal and complementary. In zero-shot thain-of-thought. a user actively prompts the model to think 'out loud', otherwise using its ordinary production distribution (Kojima et al., 2022); Quiet-STaR instead allows a model to think quietly at every token, with distribution trained to be useful. We investigate using silent, Quiet-STaR, rationales while generating explicit CoT reasoning, Because our goal is generalist reasoning that requires no task-specific input at all, we used zero-shot prompt ("Let's think step by step.") without

---

Published as a conference paper at COLM 2024
in-context examples. Our experiments indicate that internal rationales allow the model to generate more structured and coherent chains of thought, shown in Appendix E and visualized in Figure 5. The majority vote accuracy over 8 samples (cot-maj@8) increases from 40.6% to 47.7% with Quiet-STaR, as evaluated on a sample of 128 GSM8K test items. Note that each -hahuht solution is sampled with temperature 0.7.
In principle, one could imagine that the improvement comes from the computation asso- ciated with the additional tokens that the model is generating. However, seeing this is the setup used by Goyal et al. (2023), which finds no improvements in à stronger "pause- finetuning" setup (after finetuning directly on the downstream task, while we do not), this explanation is less likely though, of course, it is an important one to consider. Notably, this concern a key challenge for reasoning-relatee faithfulness, i.e. how do we know that the internal processing meaningfully corresponds to the generated thought tokens?
work, and is related to the problem of
Conclusion
Quiet-STaR represents a step towards language models that can learn to reason in general and scalable way. By training the rich spectrum of reasoning tasks implicit in diverse web text, rather than narrowly specializing for particular datasets, Quiet-STaR points the way to more robust and adaptable LMs. Our results demonstrate the promise of this approach, with Quiet-STaR improving downstream reasoning performance while generating qualitatively meaningful rationales. We also discuss the limitations of the current results in Appendix K, such as our inability to guarantee generalization to larger scales. We believe this also opens many potential future directions- - for example, one may aim to ensemble thoughts to further improve the predictions for future tokens. Moreover, if the LM can predict when thought will be useful, for example by putting the mixing head before the prediction, then the predicted mixing weight could be used to dynamically allocate compute during generation. Lastly, this work is a step towards allowing inference computation to be spent directly for additional performance. Future work can build on these insights to further close the gap between language model and human-like reasoning capabilities.
Ethics and Reproducibility Statement
This work raises some important ethical questions, many of which also apply to STaR. For example, it is impossible to know that the reasoning expressed by the model in language accurately represents the internal processing of the model (i.e., faithfulness). In addition, regardless of faithfulness, there are no safeguards against harmful or biased reasoning patterns if the model finds them useful. Relatedly, we note that CommonsenseQA is known to have many biased questions and low-quality answers (Geva et al., 2019), but we use it in line with prior work (Zelikman et al., 2022; Goyal et al., 2023). Thus, aside from improving language modeling, it is unclear in what capacity the rationales themselves should be used. Lastly, for reproducibility, we open-source our code at thtb com/ noem ikman/ www.a//aaaattta/.aaiaieeeaaeaaee hutp///ueitttrrreeeeeeee
Acknowledgements
We particularly thank Xindi Wu, Michael Li, and Qian Huang for their helpful and detailed comments, as well as Xuechen Li, Jan-Philipp Fränken, Yuhuai Wu, Gabriel Poesia, Winnie Xu, Omar Shaikh, Fan-Yun Sun, Joy He- Ale-Yuyaa, Omar Khattab, and William Yin for useful discussions. In addition, we would like to acknowledge that this work was supported by NSF Grant #2302701.
References
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at COLM 2024
Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. Fireact: Toward language agent fine-tuning. arXiv preprint arXio:2310.05915, 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. arXiv preprint Xie:2210.114166 2022.
"Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv, 2021.
eprint: 2110.14168
Kanishk Gandhi, Dorsa Sadigh, and Noah D Goodman. Strategic reasoning with language models. Xiv preprint #xXi:23055191655 2023.
Mor Geva, Yoav Goldberg, and Jonathan Berant. Are we modeling the task or the annotator? investigation of annotator bias in natural language understanding datasets. arXiv preprint ar 19v: 07888 2019.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arxiv preprint ar arXiv:2310.02226 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint ar arXiw:2308.08995 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arv preprint XxXi:2306.11644 2023.
Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. Language models can teach themselves to program better. In The Eleventh International Conference on Learning Representations, 2023. URL httphtttopeerrewee
John Hewitt, John Thickstun, Christopher D Manning, and Percy Liang. Backpack language models. Xiv preprint arXiv:2305.16765 2023.
Namgyu Ho, Laura Schmid, and Se-Young Yun. Large language models are reasoning teachers. arv preprint ar arXiv:2212.10077 2022.
Matthew Douglas Hoffman, Du Phan, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A Saurous. Training taain-of-thought via latent-variable inference. Advances in Neural Information Processing Systems, 36, 2024.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXio preprint ar arXit:2402.0645)) 2024.
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexan- der Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. Xrv preprint ar arXim:2305.02300 0.200, 2023.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. ar ar preprint ar frii:22210.16600, 2022.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with sombel-softmax. Xiv preprint 1611.0111411 2016.
11

---

Published as a conference paper at COLM 2024
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arx preprint or arXi::2310.06825 2023.
Hoyoun Jung and Koung-Joong Kim. Discrete prompt compression with reinforcement learning. Xiv preprint Xix::308.008555 2023.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. demonssrate--sarch-peeee Composing retrieval and language models for knowledge-intensive nlp. arviv preprint 1rXiv:2212.14024, 2022.
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into self-improving pipelines. arviv preprint ar Xrii:2310.03714 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners, 2022. URL https://arxivv org/ org/abs/ 2205. 11916.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClellan Jane X Wang, and Felix Hill. Can language models learn from explanations in context? arXiv preprint XrXi:22004002329, 2022.
Jack Lanchantin, Shubham Toshniwal, Jason Weston, Sainbayar Sukhbaatar, et al. Learning to reason and memorize with self-notes. Advances in Neural Information Processing Systems, 36, 2024.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint aro::22004.,6611, 2021.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Michael Y Li, Emily B Fox, and Noah D Goodman. Automated statistical model discovery with language models. ar Xie preprint ar Xix::2402.17879, 2024.
Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, et al. Explanations from large language models make small reasoners better. arXiv preprint ar arXio:2210.06726, 2022.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. ar preprint arXiv:2101.00199 2021.
Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerin. Compressing context to enhance
inference efficiency of large language models. arXio preprint arXiv:2310.06201, 2023.
Jiacheng Liu, Ramakanth Pasunuru, Hannaneh Hajishirzi, Yejin Choi, and Asli Celiky- ilmaz. Crystal: Introspective reasoners reinforced with self-feedback. arXiv preprint arXiv:2310.049211 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arviv preprint ar Xiv:2308.099 09583, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self. Feedback, 2023.
12

---

Published as a conference paper at COLM 2024
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint ar arxi:11312.660 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937.
PMLR, 2016.
Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2024.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arxiv preprint ar Xrix:2112.00111, 2021.
Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback loops with language models drive in-context reward hacking. arXiv preprint ar arXfo:2402.00627,,00 2024.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. arXiv preprint arXie:2310006766, 2023.
Du Phan, Matthew Douglas Hoffman, Sholto Douglas, Tuan Anh Le, Aaron T Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, Rif A Saurous, et al. Training chain-of-thoughh via latent-variable inference. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah D Goodman. Certified reasoning with language models. Xiv preprint ar Xiv:2306.04031, 2023.
Stanislas Polu and Ilya Sutskever. Generative Language Modeling for Automated Theorem 239900333. .eprint:
Proving. CoRR, abs/ 20090009333, 2020. URL 2009,03393.
Ben Prystawski, Michael Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. Advances in Neural Information Processing Systems, 36, 2024.
Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. Autoact: Automatic agent learning from scratch via self-planning. arXiv preprint arXiv.2401.05268, 2024.
Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al. Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. arXiv preprint ar arXii:2310008555 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, (1(14)):1667,022 2020.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. Explain yourself! leveraging language models for commonsense reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4932-4942, 2019.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36,
2024.
13

---

Published as a conference paper at COLM 2024
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arxio preprint ar XxX:::17770634// 2017.
Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming
Puzzles. In Thirty-fifth Conference on Neural Information Processing Systems, 2021. URL
https://openreeviw..t hwtp:/wwweeeerrrreoeeee net/forumiid=fe hCc4RBrg.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprint ar XrXi::3303.1133 2023.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unsuper- vised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Pp. 4615-4629, 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. aix preprint ar Xix:v:11122 17t.01172.00815, 2017.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa:
A question answering challenge targeting commonsense knowledge. arXiv preprint ar 1xXi:1811.00993 2018.
Eric Todd, Millicent L Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. ar Xiv preprint ar arXiv:2310.1 152113, 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey lrving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. Neural Information Processing Systems (NeurIPS 2022) Workshop on MATH-AI, 2022.
Ruocheng Wang, Eric Zelikman, Gabriel Poesia, Yewen Pu, Nick Haber, and Noah D Goodman. Hypothesis search: Inductive reasoning with language models. arXiv preprint arXiv:2309.05660, 2023.
Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. aiv preprint ar XxX:::2402.10200, 2024.
Lucas Weber, Jaap Jumelet, Elia Bruni, and Dieuwke Hupkes. Language modelling as multi-task problem. arXiv preprint ar x1X1v:2101.11287, 2021.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models zero-shot learners. In International Conference on Learning Representations, 2021a.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. ar preprint arXiv:2109.01652 2021b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Lan- guage Models, October 2022a. URL /ttp:///axiv..... org/ abs/ 07682. 27X6v:220..07682 [cs].
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models, 2022b. URI.nttss///rxxd.orr///////
11903.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8:229-256, 1992.
14

---

Published as a conference paper at COLM 2024
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR 2023), 2022.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D. Goodman, and Nick Haber. Parsel: Algorithmic reasoning with language models by composing decompositions, 2023a.
Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXio:2310.023000 2023b.
Hugh Zhang and David C Parkes. Chain-of-thought reasoning is a policy improvement operator. arXiv preprint ar arXio:2309.08589 2023.
Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, and Uri Alon. In-context principle learning from mistakes. arv preprint arXit:2402.05403, 2024.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXit:2210.03493, 2022.
Wenting Zhao, Justin T Chiu, Claire Cardie, and Alexander M Rush. Hop, union, gen- erate: Explainable multi-hop reasoning without rationale supervision. arXiv preprint arXiv:2305.12237 2023.
Hattie Zhou, Azade Nova, Hugo Larochelle, Aaron Courville, Behnam Neyshabur, and Hanie Sedghi. Teaching algorithmic reasoning via in-context learning. arv preprint axXiv:2211.09066 2022.
Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large language models can learn rules. ar preprint arXit:2310.07064, 2023.
15

---

Published as a conference paper at COLM 2024
Appendix
A Hyperparameter Choices
Optimization and Evaluation For optimization, we use the AdamW optimizer with warmup of 20 steps, learning rate of le 6, weight decay of 0.001, and batch size of (along with any necessary gradient accumulation to keep this fixed across runs). Moreover, our <|startofthought// and </endofthought embedding gradient weight is 1e2 and our policy weight is 1e6. We sample with temperature T = 1 during training and use greedy decoding for the thoughts during evaluation. We treat our samples as importance samples by computing the REINFORCE loss at temperature T = 3. Because we do not prompt the model with any examples, we directly compute the probability of the correct answer, conditioned on generating an answer for example, for multiple choice questions between .... E, we compute the accuracy over the logits for tokens corresponding to ... E. Lastly, for our training. select random span of 256 tokens from each sample (or pad if there are fewer than 256 tokens).
Mixing Head For our mixing head, we use three-layer MLP with ReLU activation, taking in vector of two times the size of the hidden state of the language model (as we concatenate the two predictions to determine their weights), and outputting a scalar. This scalar is then used to weigh the logits from the LM head with and without thinking to make prediction from a given token.
Computation
We train all of our models on a single node of eight 80GB H100s.
B Faster Parallel Sampling
In this section, we highlight some simple ways to further accelerate the parallel generation algorithm. For example, note that one can reduce the attention's memory cost by computing the diagonal attention simply as elementwise (rather than pairwise) dot-products. That Sar.aarass
is, given two input embedding sequences of shapes (b, t, and (b, 1, 1,),d) where is the number of timesteps ahead, b is batch size, is sequence length, and d is embedding dimension, we do not need to compute their pairwise attention of shape I,,,,, we only need compute the attention for the paired elements along the diagonal of shape f,). Additionally, to avoid generating continuations for all of the tokens (for example, if one wanted to apply value function to determine where thoughts would be most useful), one can index into this generated attention mask. Notably, however, this also requires manipulation of the other inputs during the forward pass such as the positional embeddings.
C Compute-Adjusted Plots
We also visualize Figure where we normalize by the number of thought and talk tokens used for training.
D Measuring the Impact of Multiple Thoughts Per Sequence and Multiple Tokens Ahead
We perform simple ablation on our 12-hought-token--ahhdd baseline, namely asking whether sampling multiple thoughts per sequence is necessary. We find that although simply computing the reward as the difference between the losses with and without thought proves to be a strong baseline, using multiple thoughts consistently outperformed it (by roughly 0.5% on GSM8K generalization and 3% CommonsenseQA generalization). However, the exact number of thoughts had little impact: varying between 2, 3, and thoughts per sequence appeared to result in consistent improvement with additional
16

---

Published as a conference paper at COLM 2024
Commonsenseod Accuracy (%)
46%
GSMBK Accurucy (%)
44%
9%
42%-
Thought, Tokens
Thought, Tokens
40%
8%
38%
12,
7%
26%
10,
K,4
4
Baseline
Baseline
500
1000
1500
2000
2500
3000
3500
500
1000
1500
2000
2500
3000
3500
Training Step
Training Step
(a) GSM8K
(b) CommonsenseQA
Figure 6: Compute-Normalized Generalization Results. We visualize the performance curves normalized by the number of inference calls used.
thoughts, but additional thoughts per sequence beyond two improved performance by only 0.1-0.3% on both GSM8K and CommonsenseeA. Similarly, we found that predicting more than one token ahead helped by 0.3% on GSM8K and 3.1% on CommonsenseQAi but with 12-hhought-tokens, we did not find that additional tokens ahead, beyond two, improved performance. However, qualitatively, we found that the rationales appeared more coherent with additional tokens-ahead supervision. This suggests a need for more specific evaluations of generation quality in future work.
E Rationale-Augmented Rationales
Because the language model can insert thoughts anywhere, we perform preliminary study where we investigate the reasoning generated by the language model when it has access to internal rationales while it generates external rationales (i.e. thains-of-thoughe to solve problem. Because these chains thought are themselves generated by chains of thought, one might refer to this model to a fixed random question in GSMSK, looking at outputs the base Mistral 7B, the outputs of Mistral fine-tuned for the same number of steps Open MetMahhh (Paster et al., 2023), and the rationales of our best Quiet-STaR model. We sample from all of them directly, at temperature t 1.
net-of-thought (or NoT).. We compare five solutions from the
E.1 Base Mistral 7B
First Response:
A: Let's think step by step. How many fresh eggs remains after the three her eat for breakfast and four goes into the muffin mix for her friends. We calculate it 16-3-4. The amount equals fresh eggs. So, the dollars it makes daily equals what multiplied by $2. 12 2. The answer is $24.
Second Response:
Q: Janet's ducks lay 16 eggs day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg, How much dollars does she make every day at the farmers' market?
A: Let's think step by step. She eats 3 daily for breakfast. She bakes 4 daily into muffins for her friends. That leaves 9. She sells 9 for \(21), \(9\times 2=\$18 \). So Janet makes $18 selling duck eggs each day.
17

---

Published as a conference paper at COLM 2024
66. Janet sells average of fresh duck eggs daily the farmers' market. If she sells them for $2 per egg how much does she make per week, assuming she sells the farmers' market most every day?A: If Janet sells on average 12 fresh duck eggs daily on the farmers' market. This is equal to \(12\times 71), so 12 \(Vtimes\) (841). The total amount she would then make would be \(8\tiimes 21) dollars, 168 per week. So Janet makes $168 selling duck eggs per week.
Third Response:
Janet's s ducks lay eggs per day. She eats three for breakfast every morning and
bakes muffins for her friends every day with four. She sells remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
A: Let's think step by step.
How many eggs does she have left every day? This is important because it will help us find how many eggs she sells each day. What can we do to answer the question?
If we write the relationship between how many duck eggs Janet has left, how many she eats, and how many she bakes for the muffins, then we can see how many eggs she has left to sell the farmers' market.
Let's s write an equation to solve our dilemma. Subtract equation for eggs eaten and equation for eggs baked to find the eggs remaining. Let's see what we can find.
We can see that Janet has 6 eggs leftover each day. $6\times\$2 = \$12$ per day. Therefore Janet receives additional $12 day for her eggs at the farmer's market.
Remember, this just for one day, so now she makes per day in additional income, because she sells her eggs at the farmer's market each day.
#### Answer
$12
Fourth Response:
Q: Janet's 's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
Let's think step by step. First, you want to find out the number eggs total that Janet collects each day. Take this many dollars three and subtract from the money she gets per egg, which is $2. The answer to that question is $5.
Fifth Response
Q: Janet's 's ducks lay 16 eggs day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How every day at the farmers' market?
dollars does she make
A: Let's think step by step.
For the first step, let's s think of the day whole, using some variables to keep $n$ the number of eggs that Janet's ducks lay
track of the numbers of eggs. per day. Then the number eggs that she eats per day will be $(3)$ Next, she makes $mS muffins. For each muffin, she uses $4$ eggs, so we can take from the eggs Janet's 's ducks lay to represent the number she uses: $m) Now, to get her back
starting number of eggs, need add these eggs back on, giving us eggs Janet sells next step.
day brings her $2$$(n)$ 5. This brings to our
Now we will break down all the eggs from our starting number, Sn$ =, into the pieces that set up previously. For the beginning number, we have: This brings back starting number, and can simplify this expression.
18

---

Published as a conference paper at COLM 2024
Moving from the top 1ine the second line, multiply the numbers wi within each parenthetical group, Moving from the second line third line, we add the products together to find the total. Finally, we can distribute and determine the final product: $= 360.
So the total number of eggs that Janet's ducks lay day is $360. Since Janet sells each egg she has for $2, her total earnings the farmers' market is: $\boxed(\S 7
E.2 Open Wrath- Mistral 7B
First Response:
Janet's s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. sells the remainder the farmers' market daily for $2 per fresh duck egg. How much
in dollars does she make every day at the farmers' market?
A: Let's think step by step. How much does she make every day? Well per the question, that is $2 per duck egg- Now, how many duck eggs per day does she sell? Well, that's minus the eggs she eats minus the eggs she uses to ma ke muffins.
She eats eggs every morning, $16-3 135.
She makes muffins, which uses eggs. $16-3-4= Therefore, the number of duck eggs she is selling is $5$.
Clearly, duck egg costs $2. Therefore, the total amount she makes selling the eggs is $10s $25. Now, this be written as fraction as, "ten two-dollars, $10 2 bucks.
Converting to fraction, or "a tenth of ten," so $10 bucks$ = $1$ $10 bucks$
[reveal-answer ( Solution[/reveal-nnse [hidden-answer a="fs-id1 155--116721111
Second Response:
Janett s ducks lay 16 eggs per day. eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
A: Let's think step by step. On what basis
the question affer considering the question? What do we know given and how can we use what we' given to arrive at an answer. 3 is given, is given, 16 is given, does anyone have any thoughts?
A: Well 16 minus 3 minus 4 equals 9, and then I subtract from 9 and that is 4 extra so five more eggs.
And we know that times Ten
much?
M: Yes.
A: If she eats 3 duck eggs for breakfast, in the morning equals \(3 \times 2 61). If
she eat 4 duck eggs baking muffin for her friends equals \(4 \times 2 = 8\),
total of money and that she got is $6 plus which equals 14.
weight gain is because it is is that what you put up there so 5 times 2 ducks and subtract that from that from 16 and you got no what did we write down? Are you able to figure out now what the question is, and what is what the question really is. You got 14 the weight loss you, you subtracted that from 16 which equals 2, $14 minus 2, remainder is what?
A: 12.
M: The question is what is Janet ducks lay 16 eggs, she eats 12 for sale. You subtract plus minus so when you eat the muffin equals 5, times two, the 12, minus the money so times 2, $-12$.
\times 9 181) minus $10, so more eggs... Still times $2, total comes to to $20.
A: understand there is the ones day one 3 plus 4 eats who makes and so and so on.
M: But maybe in order to answer it correctly you have to look little close. Typically tell you how here go.
in the procedures you have that they give you first,
19

---

Published as a conference paper at COLM 2024
M: Now if you increase the 16, you have that you would need determine minus Then subtract and but you M: Which is $10, and then 14, which is is the remainder
minus and that adds up the remainder, which have 5. The next answer then, which you do is now 5 times 2, which is 10.
M: Is $6 and $8 and that is total of $14, and now that you subtract 2, and that means 11.
that you now have minus 5, which
The next thing is to the yield for the second situation you have got 16 minus plus 4 and that comes to 9. This is very easy to do instantaneously even the first part,
16, 3, 4 and 9, and you happen to know what lives, and you just count and you have five and five then go ahead. 5 times comes to 10, $10 then actually you have just added 4.5,
your total is $12, so how much you sell from 16 and then minus 12 comes to 4.
16 plus $2. But
to this is $10 plus $2
M: So what is the remainder of 16 minus 3 plus 4 minus 5? Matters you see three is missing. Four you have. Five eggs. So 5 times 2, so $10 remains at $10, so reduce, what happened question 16 minuses 3 plus 4, and presumably you've done this before, if not then you add up 16 minus 3 plus 4 gives us 9 and what is the remainder after 57 So what is 5 times 2? So 10. 2, 3, plus 2 = and so your answer so far is 16 minus plus 4, and give the basket minus 2 remains and now you have to determine now what is the total for the second part? You have
what happens
added with subtracted the next the
Third Response:

 Janete t ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
Let's think step by step. The "how much in dollars" part means that one egg worth one dollar. So, twelve ducks lay 16 eggs, which means that Janet nets $20 from selling them. $16 muffins will give us 4 eggs, which nets us $8 So, we paid $108 for our ingredients to make 16 muffins. Hence, we lost 96 dollars. Hello, heroine, have to do better than that! Look into averages. Let's say that pay 5 dollars one egg. Now, $12 our product and $5 our investment. Our net wealth will be $12-5+12 24 dollars. Is that correct?
Fourth Response:

 Q: Janets ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
A: Let's think step by step. First, Jane's ducks lay 16 eggs perday. So there are 16 eggs Monday, 16 Tuesday, 16 Wednesday, 16 Thursday, 16 Friday, and so on. Second, Janet eats of the 16 eggs that the ducklay day for breakfast. She only eats the duck eggs, not her baked goods (muffins) other food. Therefore, she eats Monday, Tuesday, 3 Wednesday, 3 Thursday, and so on. Third, she only bakes the first 4 non-eaten eggs into muffins. Therefores she sells the remaining 12 non- eaten eggs Monday, 12 Tuesday, 12 Wednesday, 12 Thursday, and so on. Fourth, Janet sells duck eggs Monday, Tuesday, Wednesday, Thursday, Friday the 2 each. Third, she eats Monday, Tuesday, 3 Wednesday, 3 Thursday, Friday at the farmers'
market. Lastly, she makes 12 Monday, 12 Tuesday, 12 Wednesday, 12 Thursday, and so on in dollars every day at the farmers' market. The worksheet below summarizes and column
these steps. The number formed by the intersection of correspanding terms will be the correct answer. The product of the (3, J) term and the (J, 8) term 15150. So, Janet makes $1, 150 per day!
(Note: Step and Step are repeating, but the (3,7) term in Step 8 is not the same as the (3,7) term in Step 3.
Fifth Response

 Q:
Janets ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at
20

---

Published as a conference paper at COLM 2024
the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
A: Let's think step by step. What do we know?
Janet's 's ducks lay 16 eggs per day. eats three for breakfast every morning and bakes muffins for her friends every day with four.
conclude at least that Janet needs least seven eggs for And what do we need to find exactly?
From these facts, consumption per day.
We need to find how much in dollars Janet makes every day at the farmers' market.
Let's s write an equation solve it find the answer. know:
-(116---=911)
So daily profit is times $2, (00timmes\S2=\$133
Hence her profits at market on a given day are \(118)) Answer: \($181).
E.3 Quiet-STaR-Trained Mistral 7B
First Response:

 Q: Janet's s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day the farmers' market?
A: Let's think step by step.
The ducks lay 16 eggs day.
This is the total number of eggs.
3 of the eggs are eaten for breakfast.
This is the number eggs are eaten breakfast.
4 of the eggs are used to make muffins.
This is the number of eggs that are used make muffins.
16 3 4 =
9 is the number of eggs that sold at the farmers' market.
of the eggs are sold at farmers' market.
$2
price each egg
$2 \times 9 18$
18 is the total price of the eggs.
The total price of the eggs is $18.
Second Response:
21

---

4. Published as a conference paper at COLM 2024
Let's think step by step.
1. The ducks lay eggs per day.
2. She eats for breakfast every morning.
She bakes muffins for her friends every day with 4.
She sells the remainder at the farmers' market daily for $2 per fresh duck egg-
The number of eggs she sells at the farmers'
market 16
6. The number of eggs she sells at the farmers'
market is 16
4.
7. The number of eggs she sells at the farmers'
market is 16
4.
8. The number of eggs she sells at the farmers'
market is 16
The number of eggs she sells the farmers' market 16 3
4. 10. The number eggs she sells the farmers' market is
11. The number of
Third Response:

 Q: Janet's s ducks lay 16 eggs day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg, How much dollars does she make every day at the farmers' market?
A: Let's think step by step.
The ducks lay 16 eggs per day.
This is the total number of eggs.
3 is the number of eggs that Janet eats for breakfast every morning.
is the number of eggs that she uses in the muffins that she bakes for her friends every day.
16
9 is the number of eggs that she sells at the farmers' market daily for $2 per fresh duck egg.
$2 $2
$2
9 $2
18
So, 18 is the amount in dollars that she makes every day at the farmers' market.
The amount in dollars that she makes every day at the farmers' market 18.
A: 18

Fourth Response:
A: Let's think step by step.
The number of eggs that Janet's ducks lay per day is 16.
of these, she eats 3 for breakfast every morning.
This is
of these, she bakes muffins for her friends with 4.
This is 4.
22

---

Published as a conference paper at COLM 2024
The number of eggs that left over is 4.
This is 16 7.
This is 9.
The number of eggs that she sells at the farmers' market daily for $2 per egg is 9.
The total in dollars that she makes every day the farmers' market is $2.
This is 2.
This is 18.
total dollars that she makes every day the farmers' market is 18.
A: 18
Fifth Response
A: Let's think step by step.
ducks lay 16 eggs per day.
This is the total number of eggs.
3 eggs are eaten for breakfast every morning.
eggs used to make muffins for her friends every day.
16 eggs are sold at the farmers' market for $2 each every day.
The 16 eggs that are sold the farmers' market the remaining eggs.
16
9 eggs are sold at the farmers' market for $2 each every day.
(2)
18 is the total in dollars that she makes every day at the farmers' market.
18 is the answer.
A: 18
F Additional Works on Learning to Reason
For completeness, we highlight that many other works have explored teaching language models to reason (often with algorithms similar to or inspired by STaR), either from their own rationales, from interaction with an environment, or from hand-constructed dataset. For example, works explore this in the context of multihop question answering (Zhao et al., 2023), math (Luo et al., 2023; Uesato et al., 2022), machine translation (Gulcehre et al., 2023). Several works investigate teaching language model agents to reason in planning (Chen et al., 2023; Gandhi et al., 2023; Qiao et al., 2024), or to use specific tools or memory (Yao et al., 2022; Lanchantin et al., 2024; Schick et al., 2024), while others investigate how one may distill the reasoning from large language model into smaller language model Ho et al. (2022); Li et al. (2022); Hsieh et al. (2023). Notably however, Pan et al. (2024) demonstrates
23

---

Published as a conference paper at COLM 2024
that these feedback loops may result in reward hacking. Zelikman et al. (2023b) shows how a bootstrapping loop can be implemented where model repeatedly improves a code- improver using the same code-improver and Haluptzok et al. (2023) shows how language models can bootstrap their programming ability with programming puzzles Schuster et al. (2021). Other works have employed similar strategy for using language models to solve inductive reasoning tasks or to model real-world systems (Wang et al., 2023; Qiu et al., 2023; Zhu al., 2023; Li et al., 2024).
Some works have investigated how models can learn from their reasoning mistakes in- context (Shinn al., 2023; Madaan al., 2023; Zhang al., 2024; Liu et al., 2023). Many studies have also focused on the ability of LMs to learn from in-context reasoning examples (Lampinen et al., 2022; Zhou et al., 2022) correspondingly, Khattab al. (2022) and Khattab et al. (2023) show how the sets of examples used to prompt model to reason can be optimized in the context demonstrated that one can improve zero-shot question-answering in language models by using a variety of zero-shot prompts for reasoning.
multi-step reasoning pipeline. Furthermore, Zhang et al. (2022)
G Distribution of Improvements
We also visualize the distribution of improvements across tokens in the training set.
Original Token Prediction Loss
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Improvement in Loss Given Thought
Figure 7: Distribution of changes in log probability. We visualize the distribution of changes in log probability resulting from the generated thoughts across the evaluation dataset. We visualize the density of changes in log probability relative to the LM without thoughts, colored based on log density. The distribution is skewed, with most tokens unaf- fected, while small fraction of hard tokens see substantial improvements from the thoughts. This matches our intuition that most tokens in general web text do not require significant reasoning to predict, but thoughts are disproportionately beneficial for challenging tokens.
24

---

Published as a conference paper at COLM 2024
H
Contribution Visualization
Then used the Moivre's theorem and I got: $(cos5@+1sin5es
compared the imaginary parts and got somethang like: 4sin5e-5ccos144iinn- 55n
is very close $(16cos
but not sane.
where do make te mistake?
Thanks for any help! ;)
Have you tried use Akoos"Ahtheraasiiir2theett =15 Fan
May 15 17 at 19:13
You didn't err. You just didn't finish. May 15 '17 at 19:35
hint
In your Last Line, factor by $ysin [
replace
Aisin"((thetaai by s1-Ncos"2 (Ntheta)s
Figure Contribution Visualization. We provide an example text where we visualize the degree to which introducing thoughts helped at tokens throughout text. Green indicates that the thought before that token made that token easier to predict, while yellow indicates that it made it harder to predict. More impactful thoughts have higher opacity.
I Handling Instability
Several aspects of this task have the potential to introduce instability. First, and perhaps most importantly, the utility of generated thought (or thought token) is function of the mapping from the thought to its contribution to language prediction; however, the mapping from the thoughts to this contribution is learned based on the thoughts themselves. This means that, even if one were to generate thought that allowed the perfect prediction of the next token, the loss could receive no signal from if the mixing head's weight on that generation was 0. One solution we explored was to use the Gumbel-Softmax trick with suraight-througee estimator Jang et al. (2016), but with many consecutive softmax operations observed vanishing gradients. This introduces a fundamental challenge in reinforcement learning. Approaches like DQN (Mnih et al., 2013), PPO (Schulman et al., 2017), and A3C (Mnih al., 2016) often resolve these trade- offs by learning a state value function, which estimates the expected future reward from each state. However, the reward functions associated with this environment are unstable (as noted earlier, due to the also-changing mixing heads) - consequently, our preliminary explorations with these techniques was not promising. While are far from the first to note that optimizing rationales is seafforeeeen--eearnin Parkes, 2023; Phan al., 2023), the need for the rationale to avoid harming the base model performance introduces additional complexity. Essentially, the more complex the mapping from LM output to the next token prediction, the more instability we observed. On the other hand, when trained without any interpolation, i.e. ablating the mixing head and only using the language model prediction after thoughts, the model quickly learned to simply ignore the thoughts (and saw no generalization to any downstream tasks).
trade-off,
task (Zelikman et al., 2022; Zhang
We explored the use of separate heads for thinking and talking (here, we use talking to refer to directly outputting hidden state or logits, rather than mixing weight). In particular, we explored both linear layers from the hidden states and MLPs, initialized to contribute residually to the base language model outputs, in order to generate thoughts and next-token
25

---

Published as a conference paper at COLM 2024
predictions similar to what the language model would have otherwise generated. However, instability prevented learning.
we observed that, in all instances, the previously-mentin Consequently, we aimed to remove minimize all components that could transform the language model's outputs, both with and without its rationales. We also note that our choice to language model to output weight combining multiple states (as done by our mixing head) is essentially an attention mechanism allowing the model to attend to its thinking. This has similarity to the approach taken in Backpack language models (Hewitt et al., 2023), which also learn to predict weights to apply to summed input embeddings to model future text, rather than allowing the language model output arbitrary embeddings. Despite this constraint, Backpack language models appear to have comparable performance to traditional language models (Hewitt et al., 2023).
J Internal Reasoning Examples
While there is no explicit regularization in Quiet-STaR for thoughts to be human- interpretable, they are generated from the same transformer trained to model language, hence likely to be at least partially understandable. We discuss why this design choice benefits the training stability in Appendix For reference, we include examples of thoughts generated that were helpful to the model in predicting future tokens in OeenWeb.aath First, in one case, recalling that one should start with magnesium to produce magnesium nitride allows it to better predict that the first step of the procedure involves heating magnesium.

>>>>>>>>>>>>>>>>>>>> Magnesium reacts with nitrogen to form magnesium nitride. The chemical formula for this reaction is Mg+N_2 --N_2-> MgN_2. What is the product, or what are the products, of this reaction?\n'nJ 12, 2016\n\nThe formula for magnesium nitride is {g}_{3} (N)_(2}S. \n\n#### Explanation: \n\nAs do many active metals, magnesium nitride can st 1 --, so the equation of the reaction that forms magnesium nitride is\n\nsMg N_2 A\oo <lendofthought formed by heating the metal (fier'
In some cases, the most useful thoughts appear to be near-continuations that correspond more closely to the target text, e.g.
An integer $n$ is odd $n 2k+1$ for some integer $k$. An\nTo prove that SA BS, must show that SA Visubseteq B$ and $B Vsubseteq A$. The first of these tends too| thougthouttt|> trickiest for students
in some sense to be the more difficult<]
Lastly, we include an example from answering CommonsenseQA. Notably, this thought occurs while reading the question and hence was not used predict the final answer.
<
 Q: Talking to the same person about the same thing over and over again is < Ssttarto endof thought something someone can what?'
one-to-one correlationnn\n(b) a one-to<]
K Limitations
This work proposes new framework for learning to reason, and in doing explores solutions to variety of meta-learning challenges. However, to solve these challenges, certain simplifications were necessary. For example, it would be valuable to understand whether these techniques work when a model is trained from scratch. We have also only applied Quiet-STaR to 7 billion parameter model, albeit a powerful one. The same techniques applied to a better model would likely yield disproportionately better results, as has often been observed for gains from reasoning (Wei et al., 2022a).
Quiet-STaR results in a substantial overhead, generating many tokens before generating every additional token. (See Appendix C for compute-adjusted performance results.) However, this can also be seen as an advantage: typically, a language model can generate the next token based the current context, and while there techniques to improve
26

---

Published as a conference paper at COLM 2024
sampling quality, there is general way to leverage additional compute to enhance next-token prediction. In the current implementation, we do not support dynamically predicting when to generate, end, rationale. However, this would be natural extension. For instance, if the mixing head was prediction from the base language model, before any thought, rather than after the thought, one could apply a threshold to prevent generating thoughts that would not be incorporated. We expect that this is a more difficult task, as predicting the usefulness of thought simpler when one has already generated the thought. In other words, the task of predicting whether thought will be useful is easier when one has access the thought and not just the context used to generate it.
27


Reviews:
[{"review_id": "m6BhhjIYID", "reviewer": "Reviewer_G2Zv", "summary": "The paper proposed a novel method (Quiet-STaR) that generates rationales at each token to explain the future text, improving their predictions. Quiet-STaR addresses – how to teach LM to produce and use internal thoughts and predict beyond just the next word. They propose a parallel sampling algorithm that operates token by token to make their training procedure scalable. They introduced custom meta-tokens similar to previous works to teach LM when to generate a rationale. Interestingly, they added a residual module that allows their model to determine how much thought needs to be put into the next token prediction. The paper shows that Quiet-STaR outperforms the zero-shot performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your excellent questions!\n\nFirst, to the question about baselines, we need to emphasize that the key goal of this paper is to show that language models can improve their general inter-token reasoning ability. We attempted to analyze / control for this in the experiments visualized in Figure 2. In general, this method is complementary to CoT, since the model can generate an inner thought before each token of a CoT (as shown in Figure 5 and Appendix E). We will update our paper to note that we chose a base language model because it allows us to study this problem directly, without introducing side-effects from (typically unreleased) RLHF and chat-style fine-tuning strategies used to fine-tune these models.\n\nAs for why the start and end thought initializations were selected, it was indeed somewhat arbitrary. We wanted a single token to indicate the start and end of an intermediate thought, similar to what might be used in an actual text, ideally without imposing too many additional biases. We’ll add a note about this. \n\n> One simple explanation could be that the model now has more tokens to generate\n\nSeeing as this is the setup of the pause token paper (Goyal et al 2023), which finds no improvements in a stronger “pause-finetuning” setup (they finetune directly on the downstream task, while we don’t), this explanation is less likely -- though, of course, it is an important one to consider. Notably, this concern is a key challenge for reasoning-related work, and is related to the problem of faithfulness, i.e. how do we know that the internal processing meaningfully corresponds to the generated thought tokens? We'd be happy to elaborate on this more in our limitations section in a revised final version.\n\n> How did the authors determine which tokens are difficult?\n\nIn the context of discussing Figure 7, “difficult” was used to refer to high-prediction-loss tokens, but this could be more explicit. We’ll add “(high loss)” to make this clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written, but some details are missing. The motivation behind each model component was well explained.  \n\n- The idea of teaching a model to generate a meta-token and then act in a certain manner (in their case, meta-token is “thought” and then generates thoughts) is interesting and novel. It is a step towards making these large models more self-aware and controllable.", "reasons_to_reject": "The paper lacks a stronger baseline. Vanilla Mistral-7B is generally not great at reasoning with CoT. An instruction-tuned model might be a better baseline. \nThe paper lacks an ablation study and quantitative analysis. These would substantially improve the understanding of how each component contributes to the model's performance.", "questions_to_authors": "Questions: 1. How did the authors come up with a token “ - - - ” as start and end token embeddings? Did the authors try any other tokens?\n2. It is unclear how thought generation improved the model's performance. One simple explanation could be that the model now has more tokens to generate, which gives it more time to calculate the final answer. It might not be that the model learned to reason better. \n3. The improvement distribution section is a bit confusing. How did the authors determine which tokens are difficult?", "extra_scores": {}}, {"review_id": "rxGeOQlFd2", "reviewer": "Reviewer_fwta", "summary": "This paper introduces Quiet-STaR, an extension of the previous framework STaR, which enhances Language Models (LMs) by enabling them to generate rationales at each token to explain future text, thus improving prediction accuracy. Quiet-STaR leverages the LM's inherent reasoning abilities to generate rationales and employs a REINFORCE-based reward mechanism to train the LM, resulting in improved rationale generation. Notably, Quiet-STaR demonstrates promising potential in significantly enhancing the zero-shot performance of Mistral 7B on two reasoning tasks, CommonsenseQA and GSM8K.\n\nDespite the novel approach and promising results, this paper could benefit from improved writing, clarity, and more comprehensive experiments with diverse models and tasks and competitive baseline methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the great questions and many encouraging words! \n\nFirst, to the question about baselines, specifically Self-improvement (Huang et al., 2022) and V-STaR (Hosseini et al., 2024): while these are both excellent papers, they aim to solve a very different problem. Fundamentally, these techniques aim to train a language model to perform well on a given question-answer dataset, specifically by optimizing on questions from that dataset. Instead, our goal is more general: can a language model learn to generate rationales that help predict arbitrary future text?\n\nBoth are excellent and inspiring papers. Notably, Huang et al 2022 differentiates itself from STaR primarily by using majority vote instead of ground truth answers, but does not claim to outperform STaR given ground truth – in Quiet-STaR, the continuation of the text is our ground truth. As the length of predicted future text increases, the chance of multiple predictions matching also decreases, so using majority vote becomes impractical. V-STaR is also excellent (note, preprinted the month before the COLM deadline) but for us the future text is the ground truth so the verifier would just be the LM.\n\n> The idea of generating a rationale for each token may not be reasonable, as the rationale for a single token may not be contextually meaningful… Instead of generating a rationale for each single token, how about sampling thought based on a chunk of tokens?\n\nAbsolutely, using only a single token ahead would result in a much noisier reward signal – this is why we use the multi-token ahead teacher-forcing trick (see Section 4.4.2).\n> Figures 1, 3, and 4 are not well explained… For example, it is hard to understand the parallel generation process based on Figure 3 and the sketchy description in Section 4.2.\n\nThank you for this point! We would be happy to add additional explanations for these figures using the additional page for the final version. Do you have a particular change in mind for Figure 3? We tried to directly visualize the attentions of the thought tokens in both parts of the figure.\n> In Figure 1, the second sampled thought seems like Thought_5. Why does it appear as Thought_6?\n\nThe second sampled thought shows an example of a thought that would not help with predicting “4”, so the model should not generate thoughts like that.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The idea of Quiet-STaR, which enables LM to learn to reason from diverse text, is novel and interesting. \n+ This paper introduces several innovative designs, including a parallel sampling algorithm for efficient generation, customized meta-tokens for marking each rationale’s start and end, and a mixing head for controlling the next-token prediction from a given thought. These contributions have the potential to benefit future research in this area.\n+ The experimental results are promising, with Quiet-STaR demonstrating notable performance improvements of 10.9% on CommonsenseQA and 5.0% on GSM8K.", "reasons_to_reject": "-\tThe idea of generating a rationale for each token may not be reasonable, as the rationale for a single token may not be contextually meaningful.\n-\tFigures 1, 3, and 4 are not well explained in the context, causing difficulty in understanding the key components of Quiet-STaR. For example, it is hard to understand the parallel generation process based on Figure 3 and the sketchy description in Section 4.2. It would be nice to have a comprehensive framework of Quiet-STaR and a detailed explanation for each component.\n-\tExperiments are relatively weak. Only one base model (Mistral 7B) was evaluated on two reasoning tasks. Only one baseline method (Chain-of-Thought) was compared. It would be better to test with other models (e.g., Llama 2 or 3) in a diverse task setting and compare with more competitive baseline methods such as Self-improvement (Huang et al., 2022) and V-STaR (Hosseini et al., 2024).", "questions_to_authors": "-\tInstead of generating a rationale for each single token, how about sampling thought based on a chunk of tokens? Would this sampled thought be more meaningful?\n-\tIn Figure 1, the second sampled thought seems like Thought_5. Why does it appear as Thought_6?", "extra_scores": {}}, {"review_id": "6PNCZfnARH", "reviewer": "Reviewer_vjbp", "summary": "The paper introduces Quiet-STaR, a novel framework that extends the capabilities of Self-Taught Reasoner (STaR) to enable language models (LMs) to infer unstated rationales in arbitrary text. It tackles challenges such as computational costs, lack of initial knowledge on generating internal thoughts, and the need to predict beyond individual tokens. By employing tokenwise parallel sampling, learnable tokens indicating the start and end of thoughts, and extended teacher-forcing techniques, Quiet-STaR facilitates the generation of rationales. Notably, it significantly improves the LM's ability to predict difficult tokens and answer challenging questions directly. Through continued pretraining on internet text corpora, Quiet-STaR achieves substantial zero-shot improvements on tasks like GSM8K and CommonsenseQA, while enhancing the LM's performance in understanding natural text. Overall, this line of research holds significant promise and is likely to draw increasing interest and participation.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging, thorough, and positive feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper pioneers the exploration of training language models (LMs) to reason comprehensively from text, marking a significant advancement in natural language understanding. Through this pioneering approach, the study not only broadens the scope of LM capabilities but also showcases substantial enhancements in performance across various downstream tasks. This pioneering aspect highlights the paper's contribution to pushing the boundaries of LM-based reasoning.\n* The authors propose several innovative techniques, including a parallel sampling algorithm, meta-tokens, and a mixing head, to address challenges encountered during training. These techniques represent cutting-edge solutions that enhance the scalability, efficiency, and effectiveness of the training process.", "reasons_to_reject": "N/A", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "48UqOTK4RJ", "reviewer": "Reviewer_PS48", "summary": "This paper proposes Quiet-STaR, which trains the model to generate (pseudo-)internal thoughts for predicting every token on the general language modeling task.\nDespite the absence of finetuning on downstream tasks/datasets, the model trained by Quiet-StaR achieves significantly better performance on two reasoning datasets (GSM8K and CommonsenseQA) than the baseline model does.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the supportive and thought-provoking comments and questions!\n\n> As Quiet-STaR incurs significant extra computation overhead, would it be possible to achieve similar performance by spending the extra computation on scaling up the model size & training data (or even just use a larger model during the inference, noting that the extra overhead during inference is also significant)?\n\nAt least in the current framing, Quiet-STaR is presented largely as a continual training / post-training technique, so most likely the best comparison is compute-equivalent training without new parameters. The most expensive runs in our work have compute comparable to training on tens of millions of tokens, roughly a millionth of the pretraining compute (supposing Mistral 7b was trained on trillions of tokens). In contrast, continued training on OWM / C4 results in little improvement (or in some of our baselines, actually results in deteriorations – we’ll also update the figure to make it clearer that our baseline corresponds to naive continual pretraining). \n\nAs for inference cost, the question is definitely trickier. LM scaling laws suggest that a sufficiently large model trained on sufficiently much data should likely perform as well as Quiet-STaR, even on difficult tokens. However, it’s not clear how we might estimate the potential size of such a model or its training corpus. Perhaps the simplest answer is that, given a (sufficiently large) pretrained model, Quiet-STaR allows us to trade inference compute for additional performance in a general way (with other recent works like Rapaso 2024 also being exciting steps in this direction). Indeed, we hope this lays the groundwork for more adaptive computation strategies.\n\n> I doubt the argument that \"We expect that this is a more difficult task, as predicting the usefulness of a thought is simpler when one has already generated the thought,\" which was used to justify Quiet-STaR generating many tokens before generating every additional token.\n\nWe absolutely agree! We mention this in the conclusion and limitations but dynamically predicting the amount of thinking tokens to generate is an essential future direction. This line is mostly saying that predicting the utility of a generated thought is easier when one has the generated thought (indeed, this must be true if there is variance in the utility of thoughts and a thought provides some information about its usefulness). We’ll clarify this in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ I personally believe the general topic of “generating internal thought” is worth researching and this work is a great effort under this topic. Compared to previous works, the proposed method itself is technically good.\n    + Quiet-STaR doesn’t rely on randomness (e.g., Goyal et al. (2023) insert pseudo tokens at random locations).\n    + It is also used in an unsupervised fashion (e.g., unlike Deng et al. (2023), Quiet-STar doesn’t need any external supervision of the internal thoughts or even its locations) so it is quite scalable.\n+ Quiet-STaR is very effective given the significant performance improvement on downstream reasoning tasks despite the model not training on the downstream tasks.\n+ It is great to show the generated internal thoughts may contain meaningful reasoning. This indicates that Quiet-STaR does some effective and could-be-interpretable learning. I also would be happy to see more research on that in the future, and it may contribute to some interpretability research.\n+ The analysis of improvement distribution is insightful. It is great to empirically show that some tokens need more computation than others do, a hypothesis that is mentioned by many works on “adaptive computation” (I personally still love to see some strong empirical evidence tho it is quite intuitively reasonable).\n\n### Reference\nYuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. *arXiv preprint arXiv:2311.01460, 2023.*\n\nSachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. *arXiv preprint arXiv:2310.02226, 2023.*", "reasons_to_reject": "+ As Quiet-STaR incurs significant extra computation overhead, would it be possible to achieve similar performance by spending the extra computation on scaling up the model size & training data (or even just use a larger model during the inference, noting that the extra overhead during inference is also significant)? It is okay even if the answer is YES as I think this work’s value is more about performance improvement, but now the paper lacks such an analysis.\n+ [**Not a major concern and could be regarded as a discussion on future work**] I doubt the argument that `We expect that this is a more difficult task, as predicting the usefulness of a thought is simpler when one has already generated the thought`, which was used to justify Quiet-STaR generating many tokens before generating **every** additional token.\n    + Some other works on “adaptive computation (i.e., the model spending different computations on different tokens)” succeed in doing this kind of computation prediction ahead of the real computation. For example, Raposo et al. (2024) predicted whether a token needs to pass a layer or not ahead (tho this work doesn't use the technique of pseudo-tokens/internal thought, it is still about computation allocation so I mention it here).\n    + Also, thinking about humans' thinking process, we are able to recognize some tokens (e.g., the result of an addition problem) are more difficult than others (e.g., I am sing-*ing*).\n+ [**Not a major concern**] I am curious about how the internal thoughts (for difficult tokens) are changed during the training process. Intuitively, the generated thought may make little sense at the beginning, and finally, it can contain some meaningful reasoning as the paper has already shown. This analysis of training dynamics could provide insights into how the REINFORCE loss term works.\n\n### Reference\nDavid Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. *arXiv preprint arXiv:2404.02258, 2024.*", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method called Quiet-STaR, that trains the LM to automatically produce internal thoughts that benefit the next token predictions on any arbitrary text. The paper is well-written, and the improvements over two tasks (GSM8K and CommonsenseQA) are significant.

Pros:
- All reviewers agree this work is novel and is an important step towards the general direction of LMs "generating internal thought".
- The method introduces innovative designs such as parallel sampling, mixing heads, and non-myopic losses.

Cons:
- Some reviewers mention the experiments are relatively weak as only one base model (Mistral-7B pre-trained) and one baseline method (CoT) was used.

Overall I think this is a good paper and presents an important step towards making LMs think more in a scalable way, thus I'm voting accept.
Decision: Accept