Thinking LLMs: General Instruction Following with Thought Generation
Authors: Tianhao Wu
Janice Lan
Weizhe Yuan
Jiantao Jiao
Jason E Weston
Sainbayar Sukhbaatar
Venue: ICML 2025 poster (2025)
Keywords: thought generation
CoT
preference learning
RL
Primary area: deep_learning->large_language_models
TL;DR: We use RL to improve the thought process.

Abstract: LLMs are typically trained to answer user questions or follow instructions similarly to how human experts respond. However, in the standard alignment framework they lack the basic ability of explicit thinking before answering. Thinking is important for complex questions that require reasoning and planning -- but can be applied to *any* task. We propose a training method for equipping existing LLMs with such thinking abilities for general instruction following without use of additional human data. We achieve this by an iterative search and optimization procedure that explores the space of possible thought generations, allowing the model to learn how to think without direct supervision. For each instruction, the thought candidates are scored using a judge model to evaluate their responses only, and then optimized via preference optimization. We show that this procedure leads to superior performance on AlpacaEval and Arena-Hard, and  shows gains from thinking on  non-reasoning categories such as marketing, health and general knowledge, in addition to more traditional reasoning & problem-solving tasks.

Thinking LLMs:
General Instruction Following with Thought Generation
Tianhao Wu Janice Lan' Weizhe Yuan 13
Jiantao Jiao? 2 Jason Weston Sainbayar Sukhbaatar'
Abstract
time and think before answering complex questions.
LLMs are typically trained answer user ques- tions follow instructions similarly to how hu- man experts respond. However, in the standard alignment framework they lack the basic ability of explicit thinking before answering. Thinking is important for complex questions that require reasoning and planning - but can be applied to any task. We propose training method for equip- ping existing LLMs with such thinking abilities for general instruction following without use of additional human data. We achieve this by an iterative search and optimization procedure that explores the space of possible thought genera- tions, allowing the model to learn how to think without direct supervision. For each instruction, the thought candidates are scored using judge model to evaluate their responses only, and then optimized via preference optimization. We show that this procedure leads to superior performance on AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning categories such as marketing, health and general knowledge, in addition to more traditional reasoning & problem- solving tasks.
One approach is to generate thoughts as text, which takes advantage of the natural language capabilities of LLMs. LLMs are pre-trained on text containing human-written thoughts, which are hence encoded into the model. Chain-of- Thought (CoT) (Wei et al., 2022) is a widely used prompting technique that elicits such behavior by asking the model to write down its reasoning steps. However, the usage of CoTT has been mostly limited to math and reasoning tasks. Meta- analysis by Sprague et al. (2024) found CoT methods to be unhelpful on tasks that do not involve math and logic.
In this paper, we focus on general instruction following instead of focusing on math or logic tasks. We argue that "thinking" should have broad utility. For example, in a creative writing task, internal thoughts can be used to plan overall structure and characters. In other tasks, internal thoughts can be used for understanding the user instruction better. Of course, it is likely that less thinking is required for simpler tasks, and more thinking for more complex ones. In general, we hypothesize that such Thinking L.MSs will have an advantage on all sufficiently complex tasks, The emergence of models like OpenAl-O1 (OpenAI) and DeepSeek-R R1 (Guo et al., 2025) also support our motivation.
However, it is challenging to train model to think due to the lack of supervised training data. Although pre-training data does contain valuable information, coverage can be limited in certain domains as internal thoughts are often omitted in human writing. Existing post-training datasets typically consist of human responses, or preferences over responses, with no information thought processes. The same is true for existing reward models. Combined with the difficulty and cost considerations of collecting human thought data, these factors impose a barrier in training Thinking LLMs.
1. Introduction
Large Language Models (LLMs) are based on the Trans- former architecture (Vaswani et al., 2017) that predicts the next token at each step. Each token takes the same amount of compute, so when LLMs are prompted with a user instruc- tion, they have a fixed compute budget generate the first response token regardless of the instruction's is complexity. One way to increase the compute budget for harder instruc- tions is to allow LLMs to think internally before outputting an response. This is similar to humans who will take more
The goal of this paper is to investigate the possibility of con- verting existing LLMs into Thinking LLMs that work across wide variety of tasks, without any additional data. To this end, we introduce Thought Preference Optimization (TPO) that further trains an instruction seaeunttion-uuned LLM to make it capa- ble of having internal thoughts. Our method is simple and reuses many parts of existing training pipelines. The LLM is first instructed to produce an output sequence that can be divided into thought and response parts. The thought part is considered internal, and not part of the response shown
'Meta FAIR University of California, Berkeley Tianhao
"New York University. Correspondence Wu cthw@bereelcy.ddu., Sainbayar Sukhbaatar <sain- bar@ mor@meta.com>..
Proceedings of the 42st International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Thinking LLMs: General Instruction Following with Thought Generation
Prepend thought prompt
Sample muliiple autputs
User Instruction
Evaluste only responses
Preference pairs
Think before responding
thought
Chosen output
Rejected output
LLM
Judge
Instruction
response
DPO training
Figure/ 1. Thought Preference Optimization: We start by prompting the LLM to generate thoughts before its response. After sampling different outputs, feed the response parts the judge model which determines the best and worst ones. Then the corresponding full outputs as chosen and rejected pairs for DPO optimization. We perform multiple iterations of this training.
to the user. We optimize this thought and response output through iterative Reinforcement Learning from AI Feedback (RLAIF) training. We rely on a standard judge model that is trained to evaluate responses only, and implicitly judge the quality of the thoughts via the induced responses. This has the advantage of not requiring human curated thoughts or special judge model capable of evaluating thoughts. Through preference optimization the thoughts are then op- timized to improve the resulting responses. This contrasts with recent works such as Snell et al. (2024); Kumar et al. (2024b) which use additional supervision signals to guide models in self-refinement or coff-correttin. Instead of di- rectly guiding the internal thought process, we allow the model independently learn think.
Generie Thought Prompt
Respond the following user query in a comprehensive and detailed way. You can write down your thought process before responding. Write your thoughts after "Here is my thought process:" and write your response after "Here is my response:".
User query: (user instruction]
Specific Thought Prompt
Respond the following user query in a comprehensive and detailed way. But first write down your internal thoughts. This must include your draft response and its evaluation. After this, write your final response after "<R>"
User query: (user instruction
Given the multitude of evidence showing the effectiveness of CoT on logic-based tasks like math or coding, we focus our experiments on general instruction following instead. We train on diverse user instructions and evaluate our mod- els on AlpacaEval and Arena-Hard, benchmarks that test general instruction following. We obtain strong win rate of 52.5% and 37.3% respectively on them, outperforming the direct LLM counterpart without explicit thinking. We also conduct more fine-grained evaluations to determine which types of instructions benefit from thought. Surprisingly, we observe that thinking not only benefits topics like reasoning and problem solving, but also leads to better performance on categories that are not typically considered in the rea- soning domain, such as general knowledge, marketing, and health. This opens up a new opportunity to develop Think- ing LLMs aimed at general instruction following rather than specializing in more narrow technical fields.
Thought Prompting. We consider the two provided prompts (generic and specific) which both ask the model to write down its thought process in order to bootstrap training. The specific prompt asks for specific thought format: writing a draft and evaluating it. Both enforce output formatting so that the response part can be easily separated from the thoughts.
Figure 2.
instruction, we prompt the model to generate its thought process followed by the response. Sampling multiple such outputs, then preference optimization to improve the quality of thoughts (and paired responses) based solely on the quality of the responses.
2.1. Generating Thoughts from Thinking LLMSs
Ideally, thought generation should be simple and compati- ble with existing LLM infrastructures. Hence, we keep the model architecture the same, as an autoregressive Trans- former, although our method is potentially compatible with any model that outputs a sequence of tokens. At inference time, the core process is that the output consists of two parts: thought part followed by response part, both of which are in natural language. After generation, instead of directly sending that entire token sequence to the user, we prepro- cess it by splitting it into the two parts, and only sending the response part.
2. Thought Preference Optimization
We now describe our Thought Preference Optimization (TPO) method for teaching LLMs to think before respond- ing, depicted in Figure 1. We start with a typical instruction-tumn LLM that outputs response directly after the user instruction. We assume that there is no provided labeled thought data that we can finetune on, which makes training much more challenging. Instead, as a starting point to bootstrap our training process, for a given training user
At the beginning of our training, we achieve this by prompt-
2

---

Thinking LLMs: General Instruction Following with Thought Generation
ing the model write its thought process. We consider two possible thought prompis, shown in Figure 2. In or- der to separate thought from answer, we need the model to follow strict format. The thought prompts contain fixed keywords the model should use so that we can use simple string matching to locate where the response part begins.
use of thought generation. We employ the Reinforcement Learning from AI Feedback (RLAIF) paradigm (Bai et al., 2022; Zhu et al., 2024) where we generate from the model and rank its responses using reward model that acts as a judge. In particular, we use iterative Direct Preference Optimization (DPO) (Rafailov et al., 2024; Xu et al., 2023) for its simplicity and efficacy.
Thought Prompt Types Although the training process will change and optimize the type of thoughts, the initial thoughts are still important as they act as a starting point. The first thought prompt given in Figure 2 (top) is more generic and leaves up the model what the thoughts will contain. We also experiment with a more specific thought prompt, given in Figure 2 (bottom), that specifies that the thought should contain a draft response and its evaluation. Such specific prompts give us more control over the content of the thoughts, but also requires expert knowledge about what type of thoughts are helpful in LLMs.
Unlike conventional RLAIF, we will not feed the whole model output to the judge. Instead, the judge can only see the response part of the outputs, so the thought part cannot influence its judgement. We chose this approach for several reasons. First, there is a lack of a judge model that is capable of evaluating internal thoughts. Building such a judge is inherently challenging because it is hard to collect human thoughts. In any case, even if such data was collected, it is not clear if human- human-written -witten thoughts will be equally useful for LLMs.. Secondly, the ultimate goal is to provide better responses to the user. Thus, it might be better to optimize the final objective instead of relying on an auxiliary objective that might not align well.
Making Thoughts Internal As we mentioned, the thought part will be hidden from the end user, and only the response part will be provided them. This differenti- ates our outputs from CoT prompting, where the reasoning steps typically become part of the overall response, some- times without there being a clear distinction. While the latter might be useful in certain cases like solving math problems, in general the user expects to receive a response without excessive intermediate reasoning steps. Hiding the thought part allows it to take many forms that are usually not inter- esting to the user: making mistakes, drafting responses and evaluating them, trying to understand the question better, etc. Of course, we can also give an option to reveal the thought part to the user for the purpose of interpretability and for analysing the underlying thought process behind the response.
Our training starts with a seed model MMg that is instruction- tuned directly respond to the user instruction. We also need dataset of user instructions {æ,} to begin training the model. At each training iteration f, we feed instructions to the current model MM along with our thought prompt p as described in Section 2.1:
Me(p x.;) + (2/ J.
Here "+" means the prompts are concatenated as input con- text to the LLM. For each input, we sample A: < K outputs, each containing thought of and response sy parts.
Building Preference Pairs After extracting the response parts 3¹. we feed them to the judge model J for scoring. For pointwise judge models that take a single response and output scalar score, the process is simple:
In theory, the thoughts can take any form that is comprised of generated tokens, and do not even have to be in natu- ral language. Their primary goal is to allow the model to perform extra computation to improve the quality of the response (Pfau et al., 2024). However, thoughts in natural language have several benefits such as taking advantage of human-uwiiiie LLM pre-training data, and allowing hu- mans inspect and interpret the behaviour of the model. Hence, we use this setting because current LLMs can gener- ate thoughts well in natural language.
JZ - s R.
We also consider judge models that take pair of responses and output the winner. In this case we apply the judge model to all possible pairs (u." y/¹ } from the set of responses. This includes swapping positions in order to reduce the position- bias of the judge. Once we have all pairwise winners, we convert those to individual pointwise scores si using ELO scoring performed in Wu et al. (2024). See Appendix B for more details.
2.2. Optimizing Thoughts via Preference Optimization
Next, we select the highest and lowest scoring responses as "chosen" and "rejected"' samples to construct a preference pair. Note that the preference pairs contain both thought and response parts.
While our initial thought prompting generates thoughts via the instruction tuned model, they are not optimized to be actually useful in making the response better. We find they typically underperform thoughtless direct responses, which instruction -uunodd----dddddddooe LLMs have been heavily optimized for. Therefore, we need to train our model so makes better
Pair = {p + I; + 2. + + P p 24 - 2/ + v where c = a 58 r = argmin s/
3

---

Thinking LLMs: General Instruction Following with Thought Generation
Table Bennchmark win rates (%) for AlpacaEval (length-controlldd (LC)) and Arena-Hard. We compare our method Thought Preference Optimization (TPO) to the direct response baseline. 1lama-3-8B- ----ss-uttt and Llama-3-88 3-8-nnttee using Thought Prompting, The latter, which does not perform well, is used as initialization for the first iteration of TPO training. TPO optimizes thought generation during iterative training, which then outperforms the baselines. We also include several well-known LLMs as reference which are typically larger than our TPO model.
Method
AlpacaEval (LC)
Arena-Hard
L-ama-3-8B-bnstin -aassettcrtt-
Llama-3-8B 3---3--n--nstctc
24.9
20.6
Alama-3-88-Instruue Thought prompt
17.3
14.1
Direct response baseline
48.4
33.0
TPO
52,5
37.3
Larger models
GPT-4 (06/13)
30.2
37.9
Alama-3-70b-instruu
34.4
46.6
Mistral Large (24/02)
32.7
37.7
Qwen2 72B Instruct
38.1
36.1
Using this process, the model can learn which thought led to better response.
two responses, outputs its preference in natural language after generating COT. ArmoRM is 8B reward model that directly outputs scalar score to a single response.
Iterative Training Once we have built preference pairs, use them with the DPO loss to train the current model Mr. This gives us new model MM+1 that will be used for the next training iteration. Note that we do not use data derived from previous iterations for training the current iteration, under the assumption that they are lower quality. In addition to DPO, we also experiment with the IRPO loss (Pang et al., 2024) that combines DPO with the NLL loss.
For initial experiments, we use the synthetic instructions from Yuan et al. (2024b) for training. These instructions are generated from Llama-2-70B-ON ing consisting of random samples from the Open Assis- tant dataset (Köpf al., 2024). For later experiments, we switched to UltraFeedback (Cui et al., 2023), which contains actual human instructions. Each training iteration uses 5000 instructions that were not part of the previous iterations.
using 8-shot prompt-
Length-Control It known that some judge models tend to favor longer responses (Dubois et al., 2024; Yuan al., 2024a). This length-bias causes the response length to grow with each training iteration, resulting in an overly verbose model. To mitigate this, we implement a length-control (LC) mechanism. Let us define a normalization function N (1 = (1t - mean: (1t)) / std& (45). We recompute the scores by penalizing longer responses
We generate K = responses per prompt using tempera- ture 0.8 and top-p of 0.95. We train for 10 epochs in each iteration and select the best checkpoint using a validation set of 1500 prompts randomly sampled from UltraFeed-- back. We perform up to 4 iterations. We usually set the length-contrr parameter p E [0, 0.5], with O equivalent to no length-control. Unless otherwise specified, we use the specific thought prompt trained using the ArmoRM judge on UltraFeedback instructions as the default setup.
- N(s PN.)
As a baseline, we train the same seed model that outputs responses directly without any thinking (note, this can still perform CoT as part of the response due to its initial instruction training). We train this baseline in the exactly same way, using the same judge, data and loss. This allows us to directly measure the effect of the thoughts on response quality.
The hyper-parameter controls the strength of the length- control mechanism. Note we normalize both the score and the length to align them into similar scale.
3. Experiments
3.1. Setup
For evaluation, we use two public benchmarks that test gen- eral instruction following capability: AlpacaEval 2 (Dubois et al., 2024) and Arena-Hard (Li et al., 2024). Both bench- marks perform auto-evaluation using gpt-4-1 106-prev as a judge. In addition, we also create our own benchmark to perform fine- ane-grained analysis.
We use Maama-3-8B-Instruus (Dubey et al., 2024) as a seed model in our training. As a judge model, we consider two choices of model: Self- Taught Evaluator (STE) (Wang et al., 2024b) and ArmoRM (Wang et al., 2024a). STE is a LLM- as-a-Judge model based on Llama-3-708B - Given
4

---

Thinking LLMs: General Instruction Following with Thought Generation
50
Win rate (%)
Win rate (%)
30
40
30
20
Direct baseline
Direct bascline
TPO
TPO
0
3
Traiiing Iteration
TTrining Iteration
Figure 3. Comparing Thought Preference Optimization to the direct baseline in each iterations on AlpacaEval (left) and Arena-Hard (right).
3.2. vlpacaEval Results
our model the best model on the leaderboard with a such small size². It performs similarly to much larger models like GPT-4 (06/13) Mistral Large (24/02). Detailed re- sults comparing different experimental setups are shown in Appendix Table 6.
The highest win rate our model TPO achieves is 52.5%, which is +4.1% % better than the direct baseline, as shown in Table 1. It is also a +27 increase over the seed model and puts our method in 3rd position on the leaderboard just after GPT-4 Omni and GPT-4 Turbo. This is an impressive result given the small size (8B) of our model.
3.4. Fine-grained Evaluation
While the above benchmarks evaluate overall performance, they lack granularity to inform which types of instructions benefit from thinking. To obtain a more fine-grained eval- uation, we build our own evaluation using UltraFeedback. We take instructions not used in training, and assign them individually one of 20 categories until each category has 200 samples.
In Figure 3, we plot the win rate for different iterations of training. We can see that before training (iteration 0) the direct baseline performs much better. This is expected as the seed model is instruction tuned to directly output response. Simply prompting the model to write down its thought processes actually hurts performance. This agrees with the findings by Sprague et al. (2024) who showed CoT prompting only helps math and logic related tasks.
To measure the performance on this dataset, we compare responses generated from our TPO model against responses from the direct baseline model. We use the evaluation prompt from Arena-Hard and GPT4 (gpt-4- as a judge. The ordering of responses randomized reduce position- bias. Figure 4 shows the win rates of TPO all 20 cate- gories. Surprisingly, we observe that non-reasoning cate- gories obtain large gains through thinking. This includes language and translation, marketing and health. We also see improvement in reasoning categories like research and analysis, math and calculations. See Appendix A for more details and additional results.
However, after several iterations of training TPO catches up with the baseline, even outperforming it in iteration 3 and 4. This is a promising indication that the model is adapting to think in a way that uses those thoughts to improve its responses. A more detailed breakdown of results is given in Appendix Table 5. The best performing setup uses Ultra- Feedback instructions for training with an ArmoRM judge for both TPO and direct models. For TPO, both generic and specific thought prompts produced similar outcomes, but the latter performs slightly better (both Table and Figure 3 use this setup).
3.5. Analysis and Ablations
3.3. Arena-Hard Results
In this section, we present analysis and several ablation results further understand TPO.
Results on the Arena-Hard benchmark are shown in Ta- ble (right) and Figure 3 (right). They follow similar trend to the results from AlpacaEval. Thinking performance is poor with the initial seed model at the start of training, but with more training iterations it matches the direct base- line and starts to outperform TPO reaches win rate of 37.3%, which is +4.3% better than the baseline. This makes
Qualitative analysis of thoughts In Figure 6, present an example of non-reasoning task to illustrate the broader utility of thinking. While writing a poem is not typically
Excluding some closed models like GPT-4 where the size is not disclosed. Compared against Aug. 31st 2024 version. https: g thub. com/1 mm sys/areeee aarna haanaaaare rd auauto
'As of Jan. 30th 2025. a wappcc__evaall
lab. g github.
5

---

Thinking LLMs: General Instruction Following with Thought Generation
56-
Win rate (%)
64
62-
50-
Roweerth and
Analysts
Language and Thhne alln
Marketingg and Sall
Mall and Caleslations
Presooo Develogmen
Willucse
General Knowledge
Bosanting ane Probleer Sekkinng
Educcation and Leaninn
Art and Dosgnn
Thooe Leteure
Progammiing und
Sctence and Tecimolog
Cameera mnnmrre/ioond Joonll Piologee
Data Amdyais and Viualtoiin
Enterta tontaateeezzz... and Hunse
Contea Wiliis
Businso and Fiaasoo
Gaae Devilgmestt
Creative Witring
Figure 4. Fine-Graite Sine-Grained evaluation on unseen instructions from UltraFeedback, broken down by category. We measure the win rate of TPO against the direct baseline as judged by GPT4,
viewed as a reasoning task, it can benefit from better plan- ning and understanding of the instruction. This is also a good example of when makes sense to hide the thought process. Figure 5 shows the thinking of TPO model trained with the specific thought prompt on a factoid ques- tion (which dog breed is smallest). We see that the model thinks first about the question, then evaluates its draft re- sponse. We provide more thought examples in Appendix C.
Table 2. Comparing the generic and specific (Figure 2) thought prompts. We measure the length (in characters) of the thought and final answer the Ultraeedbackk train prompts, as well as AlpacaFval (I.C) win rates before and after one training iteration.
Seed Model / Iter
Type
Thought Len
Response Len
Win Rate
Direct
2878 / 2787
24.9% 37.5%
Generic
799 / 606 /193/ 1613
1717 1638
17.3% / 32.7%
Specific
1300 1431
16.4% / 31.0%
Thought Prompt Types Table 2
compares different thought prompt types in terms of different metrics across iterations. As we have observed previously, the seed model performs poorly when it is asked to generate thoughts com- pared to directly generating a response, which is the result of the model being well optimized to do the latter. However, after one iteration of training, we see the gap is shrink- ing. Between the two thought prompt types we try, there is not much difference in terms of win rate, but the thought and answer lengths vary greatly. Also, the specific thought prompt leads to longer responses and have a higher average ArmoRM score (see Appendix Figure 21).
Length- We find that length (LC) is a must during training, especially with the STE judge. When no LC is performed, the average response length grows rapidly (Wu et al., 2024). For instance, after just 1 itera- tion of training of the direct baseline, the average response length grows around 15%. If we utilize our LC technique when building preference pairs, we can maintain the same length or even decrease the length if necassary. In our ex- periments, we tune the LCC coefficient, p by measuring the average length of chosen and rejected samples, and choose the smallest value such that the former is not longer than the latter. This simple method was very effective, and both TPO and the direct baseline model did not grow their response length much during training, as demonstrated in Table 2.
Thought lengths As shown in Table 2, the specific thought prompt has the longest thought generations because it asks the model to generate a draft response and evaluate it within the thought. Although we did not directly supervise the thought process, we found the model learns to shorten and condense the thought throughout the training. Table 2 shows the length changes after training iteration, but more detailed statistics across training are given in Appendix Ta- ble 4. We find that after 4 iterations of training, the thought part is shrunk by 61% % and 30% for the generic and specific thought prompts respectively. We also measure thought lengths by category (Appendix Figure 14) and find that cate- gories such as research, art and writing have longer thoughts, while conversation, language, math and general knowledge are on the opposite end.
Training Instruction set and Judges We experiment with two training instruction sets: the synthetic instruction set generated by few-shot prompting Llama-2-70B, and UI- traFeedback more curated instruction set that contains human instructions. As shown in Appendix Table 5 and Table 6, we do not observe large difference between these two datasets in terms of final performance on the bench- marks. We also experimented with two different judges for training, ArmoRM and the STE judge. We find that when using either judge TPO outperforms the direct baseline on AlpacaEval as shown in Table 5. However, STE required
6

---

Thinking LLMs: General Instruction Following with Thought Generation
Example of Factoid Question (specific thought prompt)
Example of Non- Non-Reasoning Instruction (generic thought prompt)
Instruction: What breed dog is smallest?
Instruction:
Wrile me poem in the style of neruda
Internal Thoughts
"Okay, let's tackle this query. I'm thinking of popular toy breeds that are widely recognized for their small size. Chihuahua, Pomeranian, and Poodle are some of the breeds that come to mind. But, I need to dig deeper to ensure provide the most accurate answer.
Here is my thought process:
Neruda's style is known for being passionate, sensual, and evocative. need to tap into that emotional depth.
Draft Response:
His poetry often explores themes of love, politics, and social justice. I should choose theme that resonates with Neruda's 's concerns.
The smallest dog breed recognized by kennel clubs and breed associations is the Chihuahua, specifically the Long Coat variety. According to the FCI (I (Fédération Cynologique Internationale), the Chihuahua has an av- erage weight range of 1.5-3 kg (3.3-.66 lbs) and an av- erage height range of 15-23 cm (6-9 inches).
Neruda's writing also infused with sense of mysti- cism and the mysterious. I'll try to incorporate some enigmatic elements.
Here is my response:
Evaluation:
Whispers in the Rainy Night
My draft response looks good, but I'm not entirely satisfied. While Chihuahua indeed contender for the smallest breed, I'm not certain if it's the absolute smallest. I should provide more context and potentially mention other breeds that might be considered smaller. Additionally, I could include some interesting facts or tidbits to make the response more engaging.
In the city's i secret heart, where shadows play Like lovers' fingers tracing the city's gray The rain whispers secrets, nocturnal refrain As I stand here, lost in your eyes' sweet pain (continued ...)
Final Response
Figure 6. Example of TPO responding to non-reasoning instruc- tion to write a poem. This example demonstrates how thinking can be useful in understanding the task and planning accordingly even for creative writing. The thought part green is not part of the response.
<R>
The Smallest Dog Breed:
Meet the Chihuahua
While there might be variations in size within breeds, the Chihuahua is widely recognized as the smallest dog breed by kennel clubs and breed associations (contin- ued...)
adds an NLL loss on the chosen samples, so that their av- erage log probability does not decrease, which can occur with DPO. However, in our experiments and setting, we did not see a noticeable difference in performance. After one training iteration, TPO using IRPO gives 31.6% AlpacaE- val LC win rate, which almost matches the 32.0% of DPO. Here we used the STE judge and the synthetic dataset for training.
Figure 5. Example of TPO answering simple factoid question. This model is trained with the specific thought prompt, so it writes a draft and evaluates the thought part (in green).
more length-control to be applied during training. On both benchmarks, we obtained better results with the ArmoRM judge.
Math Domain To further understand the performance on the math domain where CoT techniques are often ap- plied, we evaluate our model on the GSMSK dataset (Cobbe et al., 2021) that contains grade-school math word problems. Since we have the correct answers in this dataset, we can more accurately measure performance compared to relying on judge. We first test the seed model without a thought prompt, but observe it uses CoT anyway due to its instruct training, obtaining 79.2% accuracy. We thus append "Out- put only the number answer." to the problems so that the model will not perform CoT in its response. It still performs surprisingly well in this answer-only mode, reaching 69.7% accuracy. However, on closer inspection noticed that the seed model still performs CoT--like arithmetic operations like "16 3 4 = 9. 9 * 2 = 18" in its output. We then evaluate direct models trained on the synthetic prompts and the STE judge. After training, this number drops to 51.3%. Performance is even lower for TPO models. This indicates
Parsing errors For the initial seed model we use prompt- ing to maintain the desired output format, where thoughts and answer are separated, see Figure 2. However, we find during training that without additional safeguards, the rate of parse errors keep fluctuating, and is sometimes hard to control. In order teach the model follow the constraint, we add responses with parse errors rejected examples in the preference data creation process. Typically, the ratio of parse errors as rejected is no more than 10% in each iteration, but we observe for the specific prompt template, it is much harder to control. We found that sampling again if a parse error occurred helped alleviate the issue during evaluations.
DPO vs IRPO loss In addition to training with the DPO loss, we also experimented the IRPO loss that has shown promising results in math tasks (Pang et al., 2024). IRPO
7

---

Thinking LLMs: General Instruction Following with Thought Generation
that our experimental setup is not suited for math tasks. In fact, only 2.2% of our training instructions are categorized into the math category (see Appendix Figure 7). Such a drop in math performance is also observed by Meng et al. (2024); Zhu et al. (2024) in a general instruction tuning setup. How- ever, we do see some examples where the TPO model is able to correct its mistake through reflection, while the direct model simply stops at the mistake, as shown in Appendix Figure 20. For more detailed results, see Appendix Table 3.
It is now widely used for math and reasoning tasks, and most current LLMs are finetuned to do CoT by default for those types of tasks (Dubey et al., 2024). Other works like Pfau et al. (2024) show that the model equipped with CoT might be able to perform hidden thinking using filler tokens. However, CoT usage has had more limited use in other types of tasks. Meta-analysis by Sprague et al. (2024) found that CoT techniques have little benefit outside of math and logic related tasks.
Computational Cost The primary computational costs of TPO arise from two main stages: first, the generation of responses, which involves thoughts and responses, and second, the subsequent training process (DPO) that utilizes these generated responses. The length of these responses and their associated 'thoughts' is key factor in this cost. According to Table 2, initially, the total response length with the generic template was 12.57% less than the Direct base- line, while the specific template yielded responses 21.36% longer. After the first iteration, the generic template's 's re- sponses were 19.48% shorter, and the specific template's 's responses were 9.22% longer than the baseline. Thus, while the computational expense relative to the baseline varies depending on the template - some configurations being less desource-intensire and others more so the aggregate impact on computational cost remains relatively minor.
Training to Think: Nye et al. (2021) trained a model to write intermediate calculations into a scratchpad section be- fore writing the final answer, which improved performance in math and coding tasks. Similarly Lehnert et al. (2024) showed that Transformers can solve complex planning tasks if they are trained to write A search traces before outputting the solution. However, these methods rely on supervised training so ground-truth thought data is required. STaR (Zelikman et al., 2022) removes this constraint by generat- ing both thought and answer from a model using few-shot prompting. Then the generations are filtered by the correct- ness of the answer to be used for supervised finetuning. Its generalization Quiet-STaR (Zelikman et al., 2024) aims to insert thought segments into unstructured text. This involves sampling a sequence of thought tokens after every input to- ken, then training using a REINFORCE based loss that optimizes the likelihood of subsequent input tokens. While it showed promising results in multi-choice reasoning and math tasks, the training mechanism is complex and compute heavy. V-STaR (Hosseini et al., 2024) trained a DPO verifier on both correct and incorrect solutions and uses the verifier to select the response in inference time. IRPO (Pang et al., 2024) also trains a variant of DPO on math and reasoning problems to learn CoTs, assuming access to gold labels on the training set. Similarly. Self-Notes (Lanchantin et al., 2023) allows the model to deviate from the input at any time to write its thoughts, but relied on supervised training data in symbolic tasks. None of these methods have been applied to general instruction following using LLMs. Concurrent work DeepSeek-RR R1 (Guo et al., 2025) used RL to finetune the base model with a structural thinking template against a rule based reward and shown impressive results on par with OpenAI-01(OpenAA.
4. Related work
Reasoning in Language vs. Vectors: Thinking in words takes advantage of the natural language understanding capa- bility of LLMs. While thinking in continuous values might provide more bandwidth, the Transformer architecture al- ready can compute continuous vectors as hidden states and feed them to the next layer. However, these hidden vectors do not feed back into the model at the next token, and thus are not accessible to the future lower layers (Fan et al., 2020). Word tokens on the other hand are fed back to the model immediately, i.e. during inference the previous output token is fed as input to predict the next token making it possi- ble to condition all future computations on them (Merrill & Sabharwal, 2024). Another advantage of word tokens is that there exist simple sampling mechanisms which al- low thoughts to take different paths each time (Wang al., 2023), which can be used to improve results e.g. via major- ity vote.
System 2 methods: Many system 2
methods emerged in recent years that add intermediate steps at inference time be- fore producing a final answer. Those steps typically involve prompting the model with a certain goal, such as verifica- tion of the answer (Dhuliawala et al., 2024), rephrasing user questions (Deng et al., 2023), selecting sentences to attend to (Weston & Sukhbaatar, 2023), etc. Briakou et al. (2024) developed a method for translation incorporating intermediate steps of drafting and revising. Our method has a similarity with these methods in the first step because it
Chain-of-Thought (CoT): CoT prompting (Wei et al., 2022) demonstrated that LLMs perform better at reasoning tasks when they are encouraged to write down intermediate reasoning steps. Since the type of thinking in CoT is dictated by the prompt instruction, there are now many different variations of it facilitating different types of reasoning, such as decomposing into smaller problems (Zhou al., 2023).
8

---

Thinking LLMs: General Instruction Following with Thought Generation
uses prompting on the initial seed model, but further opti- mizes the thoughts during training iterations. Concurrent work by Kumar et al. (2024a) trains models to self-correct, while Yu et al. (2024) distill system 2 methods into system with supervised finetuning. Rather than focusing on general thinking these works teach the model specific skills.
All our experiments are based on 8B parameter sized models. However, it is worth investigating the effect of thinking on larger scale models. Given the compute requirements of such experiments, we leave that to future work.
References
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022.
5. Conclusion
We introduced Thinking LLMs, which think in natural lan- guage before writing response for general instruction- following tasks. To train such models, we proposed a new training recipe called Thought Preference Optimization for teaching Thinking LLMs to improve their thoughts. Un- like prior methods (Snell et al., 2024; Kumar et al., 2024b), which directly supervise the thought generation process through techniques like self-correction self-refinement, we instead provide incentives for the model to generate its own thoughts, without explicitly teaching it how to think. In our experiments, we train and evaluate the models in the general instruction following sctup. The results show that, although the early iteration of the Thinking LLM perform poorly compared to the typical direct response model, af- ter multiple iterations of training using TPO, our method outperforms the baseline. Further, fine-grained evaluations reveal that thinking helps in categories that are not usu- ally associated with reasoning or chain-of-thought methods. This is an encouraging result and hopefully leads to wider adoption of Thinking LLMs in non-reasoning domains.
Briakou, E., Luo, J., Cherry, C., and Freitag, M. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. arXiv preprint arXiv:2409.06799 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14 468, 2021.
Cui, G., Yuan, L.., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Uttraeedback: Boosting language models with high-quality feedback, 2023.
Deng, Y., Zhang, W., Chen, Z., and Gu, Q. Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint S.Xi::2311.02205 2023.
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., and Weston, J. Chain-of-verificattio reduces hallucination in large language models. In ACL (Findings), pp. 3563-3578. Association for Computa- tional Linguistics, 2024.
Impact Statement
We experimented with two different thought prompts, and observed some performance differences between them. It is likely that certain thought types are suited for certain tasks, and direct responses would even work better certain sit- uations. Therefore, training on a diverse set of thought prompts and allowing the model to switch between them could potentially lead to further improvements in perfor- mance.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21173 2024.
Dubois, Y., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024.
While we see improvement in overall performance with TPO, evaluation on GSM8K showed degraded math perfor- mance. As we discussed, this is likely due to our setup not being oriented toward such tasks. Incorporating more math instructions during training and having access to a judge capable of evaluating of their answers are likely solutions.
Fan, A., Lavril, T., Grave, E., Joulin, A., and Sukhbaatar, S. Addressing some limitations of transformers with feed- back memory, arXiv preprint arXiv:2002. 0940000...402..... 2020.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma. S., Wang, P.,, Bi, X., et al. Deepseek-rl: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.
In the current version of the method, thought lengths are purely determined by model itself. There is no steerability in terms of changing the number of thought tokens. Adding such functionality could be useful as longer thoughts in- crease computation and corresponding cost per user instruc- tion. We could use techniques like Yuan et al. (2024a) for this purpose.
2025.
Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sor- doni, A., and Agarwal, R. V-STar: Training verifiers for self- reasoners. In First Conference on Language Modeling, 2024.

---

Thinking LLMs: General Instruction Following with Thought Generation
Köpf, A., Kilcher, Y., Rütte, D., Anagnostidis, S., Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stan- ley, O., Nagyfi, R., et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Pfau, J., Merrill, W., and Bowman, S. R. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404. 15758, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Kumar, A., Zhuang, V., Agarwal, R., Su, Y., Co-Reyes, J. D.,
Singh, A., Baumli, K., lqbal, S., Bishop, C., Roelofs, R.,
Zhang. L. M., McKinney, K., Shrivastava, D., Paduraru, C., Tucker, G., Precup. D., Behbahani, F., and Faust, A. Training language models self-correct via reinforce- ment learning. 2024a.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling IIm test- time compute optimally can be more effective than seal- ing model parameters. arXiv preprint Nxiv:2408.033144 2024.
Kumar, A., Zhuang. V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., Iqbal, S., Bishop, C., Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024b.
Sprague, Z., Yin, F., Rodriguez, D., Jiang, D., Wadhwa, M.. Singhal, P., Zhao, X., Ye, X., Mahowald, K., and Durrett, G. To cot or not to cot? thain-of-thoughee helps mainly math and symbolic reasoning. arXiv preprint arXiv:2409. 2tii:2409.11883, 2024.
Lanchantin, J., Toshniwal, S., Weston, J. E., Szlam, A., and Sukhbaatar, S. Learning to reason and memorize with self-notes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit. J., Jones, I.,, Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Prucessing Systems, volume 30, 2017.
Lehnert, L., Sukhbaatar, S., Mevay, P., Rabbat, M., and Tian, Beyond *#: Better planning with transformers via search dynamics bootstrapping. In First Conference on Language Modeling, 2024.
Wang. H., Xiong, W., Xie, T., Zhao, H., and Zhang. T. Inter- pretable preferences via nulti-obbecctiv reward modeling and mixture-of-experte arXiv preprint arXiv:2406. 228425406.12845. 2024a.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to Sigh-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
Wang, T., Kulikov, I., Golovneva, O., Yu, P., Yuan, W., Dwivedi- Yu, J., Pang, R. Y., Fazel-Zarandi, M., We- ston, J., and Li, X. Self-taught evaluators. ArXiv, abs/2408 02646668.0666..... 2024b.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-free reward. arXiv preprint arXiv:2405. 14734, 2024.
Wang. X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Conference on Learning Representations, 2023.
Merrill, W. and Sabharwal, A. The expressive power of transformers with chain of thought. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Nye, M., Andreassen, A., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. Show your work: Scratchpads for intermediate computation with language models. ArXiv, abs//112.00111 14, 2021.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., brian ichter, Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
OpenAl. Introducing openai ol ol-preview, 2024. URL
Weston, J. and Sukhbaatar, S. System 2 attention (is something you might need too). arXiv preprint arXiv:2311. 1.111.1119,,,,,,1111111111999111 2023.
https::/opents com/index/ index/ int introoucing--pen.n ool -olo-oo-pppeeiee preview/
Pang. R. Y.. Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimiza- tion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Wu, T., Yuan, W., Golovneva, O., Xu, J., Tian, Y., Jiao, J., Weston, J., and Sukhbaatar, S. Meta-rewarding language models: Self-improving alignment with llm-as-at a-meta- judge. ArXiv, abs/2407. 19594, 2024.
10

---

Thinking LLMs: General Instruction Following with Thought Generation
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things more cringe than others: Iterative preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682 2023.
Yu, P. Xu, Weston, J., and Kulikov, I. Distilling system into system 1, arXiv preprint arXiv:2407.06022 2024.
Yuan. W., Kulikov, I., Yu, P., Cho, K., Sukhbaatar, S., We- ston, J., and Xu, Following length constraints in- structions. arXiv preprint arXiv:2406.17744 2024a.
Yuan, W. Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. E. Self-rewarding language mod- els. Forty-first International Conference on Machine Learning, 2024b.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STar: Bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, 2022.
Zelikman, E., Harik, G. R., Shao, Y., Jayasiri, V., Haber, N., and Goodman, N. Qui::-STar: Language models can teach themselves to think before speaking. In Pirst Conference on Language Modeling, 2024.
Zhao, W., Ren, X., Hessel, J., Cardie, C., Choi, Y., and Deng, Y. Wildchat: Im chatgpt interaction logs in the wild. International Conference on Learning Representations, 2024.
Zhou, D., Schärli, N., Hou, L.., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. H. Least-to-most prompting enables complex reasoning in large language models. ICLR, 2023.
Zhu, B., Frick, E., Wu, T., Zhu, H., Ganesan, K., Chiang, W.-L.., Zhang, J., and Jiao, J. Starling-7b: Improving help- fulness and harmlessness with rlaif. In First Conference on Language Modeling, 2024.

---

Thinking LLMs: General Instruction Following with Thought Generation
20
Pereentage (%6
15
10
5
Langnngg and Translliio lon
Programming and Codlag
Rescurch and Anolysis
Educacton Learniing
Conteut Writing
Bessoning and Prublem Solving
General
Chextive Writing
Selence and Technology
Busi besss and
Health Wellness
Mash and Calcalations
Converatiion Dhalogue
Madesting and Salns
Development
Dada Analosis and Visulization
Teavel and Art and Design
Entertainmonn and Humor
Persuaal Developnont
Figure 7. Training instruction distribution: we categorize our training instructions into 20 topics. The language and translation category
has the most examples, while the math category only 2.2% of the data.
A. Fine-grained evaluation details
We first generate 20 category names by prompting -lama-3.1-70B-nnstii model to summarize the categories present given many instructions from WildChat (Zhao et al., 2024). We select 200 instructions from UltraFeedback per category for evaluation, excluding those that used in training. Figure 7 shows the ratio of each category in our training instructions from UltraFeedback. This shows that some categories have much fewer samples in the training data, which might negatively affect their performance.
We show the prompts we used: to generate categories in Figure 10, assign categories in Figure 11, and evaluate with the GPT4-1 106 judge Figure 12. The judge can give evaluations of: A>>B, A>B, A=B, B>A, B>>A, which are counted as 0,7,,,,000 0.5, 0.25, 0 respectively. The scores are averaged over instructions determine the win rate in the category. Figure (a) shows the win rate of the generic thought prompt, which is lower average compared to the specific thought prompt (see Figure 4). However, it still outperforms the direct baseline complex categories like math and coding.
To further validate our results, we perform a similar evaluation using alternative judge, STE. The prompt we used with this judge is shown in Figure 13. Here, we test 500 instructions per category and evaluate in both orders. Figure 8 (b) and (c) show the win rate of the specific and generic thought prompts respectively. Again, we see that the specific thought prompt works better. Compared to the GPT4-based evaluations, we see more variation across the categories, perhaps due lack of tie option the judge. We also see that TPO does better when the training also uses the STE judge (Figure (d)).
Overall, we observe that ranking of categories by the win rate varies lot between different judges and thought prompt types, making it challenging to draw conclusion about which category benefits most from thinking. However, several categories consistently improve from TPO such as math. language and translation, and reasoning.
In addition, we also classify AlpacaEval instructions into 20 categories following (Yuan et al., 2024b) and present the default benchmark win rates in Figure We see large improvements some categories like music and art, entertainment and cooking. This is interesting because those categories are not considered as reasoning tasks in the standard literature. However, note that some categories might not have sufficient samples (the total number of samples is only 805) even though we removed two categories that have less than 10 samples.
B. ELO computation
For each prompt x, we have up to K corresponding responses denoted as {v' ,..., ". We then evaluate each pair of different responses (y"", y) using the pairwise judge.
12

---

Thinking LLMs: General Instruction Following with Thought Generation
Table GSM8K Accuracy when the instruction asked output the final numeric answer only vs using the original prompts. Our training setup degrades performance of both the baseline and TPO models.
Model
Data
Training Iter
Original
Answer-Only
Direct baseline
0
79.2%
69.7%
Direct baseline
Synthetic
4
77.6%
51.3%
TPO
0
77.1%
57.2%
TPO (generic)
Synthetic
78.2%
21.4%
TPO (generic)
UltraFeedback
78.5%
10.2%
TPO (specific)
UltraFeedback
70.6%
43.4%
Table Thought length (characters) TPO on UtraFeeddack validation set during training iterations. UltraFeedback using ArmoRM judge.
We tested models trained on
Iteration
Generic thought prompt
Specific thought prompt
0 (seed)
815
2125
605
1560
467
1430
346
1320
314
1494
The result of single battle between judgments (y"", y²) is defined as:
the pairwise-judge prefers m wins
If the pairwise-judge prefers wins If tie error.
We then construct a battle matrix B the combination of the battle results:
B 1 1 1] + 1 11]
The next step is to convert the battle matrix into ELO score. Following the same process in Wu et al. (2024), determine the ELO score s for each response y² by solving the following maximum likelihood estimation problem:
arg max
Σ
Ban log
1 1om
ISm,nK
C. Thought Examples
Here we show several examples of thought and response outputs. Figure 15 a non-reasoning question and TPO still takes advantage of thinking by recalling relevant information. Figure 16 is example where the model solves complex problem by thinking of a draft that outlines the main steps before actually writing the code. In contrast, the direct response baseline makes a mistake, as shown in Figure 17.
We do observe some failure cases of thinking. For example, we observe the model learns to generate multiple round the drafting-evaluation process even if it is not told to do so the prompt. While this can helpful, we find overthinking sometimes causes the model to get "lost in the thought", without giving an final answer else providing wrong answer. An example of this is shown in Figure 18.
The next example shows how the model thinks before training in Figure 19. Here, the thought process is correct and helps to get closer to the answer, and the model only gets an addition wrong at the last step. This shows that thinking can be helpful, but the model is still bad at arithmetic.
13

---

Thinking LLMs: General Instruction Following with Thought Generation
Table 5. Breakdown of AlpacaEval results for different judge models, training instruction sets ("Data") and Thought Prompts comparing our method, TPO, to the direct baseline. We report the standard LC win rate. In each case TPO outperforms the direct baseline, but only after several iterations of training.
Training Setup
Training Iteration
Model
Judge
Data
Thought Prompt
2
4
Direct
STE
Synthetic
24.9
34.8
40.3
43.6
42.6
TPO
STE
Synthetic
Generic
17.3
32.0
40.2
47.6
46.8
Direct
Armo
Synthetic
24.9
35.3
43.2
45.7
48.1
TPO
Armo
Synthetic
Generic
17.3
33.8
44.1
49.8
51.1
Direct
Armo
UlraFeedback
24.9
37.6
41.1
45.2
48.4
TPO
Armo UlraFcedback
Generic Specific
17.3
32.8
43.0
47.0
50.2
TPO
Armo
UlraFeedback
16.4
31.0
44.9
52.5
50.8
Table 6. Breakdown of Arena-Hard results for varying experimental setups. We report the default win
Training Iteration
Model
Judge
Data
Thought Prompt
1
2
3
4
Direct
STE
Synthetic
20.6
29.6
32.5
33.1
33.2
TPO
STE
Synthetic
Generic
14.1
24.0
30.6
27.2
24.3
Direct
Armo
Synthetic
20.6
27.1
31.4
31.1
30.4
TPO
Armo
Synthetic
Generic
14.1
25.7
31.0
32.8
30.1
Direct
Armo
Ulraacedback
20.6
29.9
33.0
31.7
32.2
TPO
Anmo UlraFeedback
Generic Specific
14.1
27.2
28.1
30.8
31.8
TPO
Armo
UlraFeedback
12.4
26.6
33.2
33.7
37.3
14

---

Thinking LLMs: General Instruction Following with Thought Generation
55.0
Win rake (8)
62.5
60.0
47.5
Mott Cakulaabez
Coclig
Traniatioo
Discation sodi
Dusirco aod
Finnne
Bousningg azil Proble--alve
Comvenation (o Dikkguo
Pusonall Devilommmn
Entertai and Humor
Come Develoyenoo
Vinudoottion
Wiiting
Goneral
Thooel aod
Content Winting
Art aud Design
Marloting sod Stuee
Langang
(a) Generic Thought Prompt judged by GPT4
60
Wia rate (%)
50
40
Chnura
Langrag and
Reasoning and
Prodeam Sool Scor
Maah and Calculistoos
Rntestimmon and Honvr
Coneerationn
Education and Leanlngg
Soienoe und Teednology
Reseurch and Analysie
Data Analysle and Vioudlaa
Health and Welltess
anminng Chiling
Truaal sai Laterro
Busisass and Finnce
Art Design
Developmont
Marketing and
Pemoooll Develppmrr
(b) Specific Thought Prompt judged by STE
Win rate (56)
50
45
40
Tescuning and
Edleation aud
Tanming
Interraimeer. Mad Hamoe
Math and Calculankon co00
Doooorch aod Aualysi
Oecaral Kiunohdgge
Thved and Lebaue
Programm and
Codling
Doliga
Dostosss and
Conten Witting
Health and Wallhoe
Auadysi
and Viraiiattioo
Developmenz
Cloative
Marketing and Salee
Cumerratio
Longuage Thanlatioo
(c) Generic Thought Prompt judged by STE
Win rake (96)
60
55
30
Langhage and
Eatertalment and Hamoor
Modh and Calculation
Progr Programminn sennaiiing and
Edunation and Leaniing
Grearal Kanslagg
Gome Development
Finance
Dats Auslyde
Vinoalaaniio
Creative
Rownuch ana Analyt
Persccal Develggmnn
Health and Wellnss
Hlweel and
Science and Techuolog logz
Commraation
Busi
Content
(d) Generic Thought Prompt judged STE when STE is also used during training (iteration 3)
Figure 8. Additional Fine-grained evaluation results. We show the win rate of TPO against the direct baseline trained in the same setup.
The different plots correspond setups differing in evaluation judge model, thought prompt type, and the training judge.
15

---

Thinking LLMs: General Instruction Following with Thought Generation
Direct baseline
TPO
LC Win rate (%)
20
Philoeopky
Sclenoe, Technologg
Bosiness Marlosing
Hintory Social
Interaction
Music Arts
Exerciee, Health
Tosh, Gadgets
DIY. Hobbies
Thoeel
Macellatoos
Literature,
Writing
Cooingg
Algottikama
Logic
Lang sugge
Came
Development
Figure 9, Fine-gras grained evaluation on AlpacaEval instructions. We measure length-controlled win rates within cach category for cach model.
Prompt to come up with categories
Given the following list possible instructions, define a maximum of 20 categories that would cover the types of intructions, for example recipes, reasoning tasks, general knowledge etc. Try to cover as many of the instructions as pos- sible with the maximum 20 categories, while keeping the categories high-level, concise, simple, and easy undersland.
<NEW INSTRUCTION> (query)
<NEW INSTRUCTION> (query)
Figure Prompt generate categories based of instructions.
16

---

Thinking LLMs: General Instruction Following with Thought Generation
Prompt to assign categories
Below is an instruction that I would like you to analyze:
<instruction
{user instruction
</instruction>
Categorize the instruction above into one of the following categories: General Knowledge
Math and Calculations
Programming and Coding
Reasoning and Problem-Solving
Creative Writing
Content Writing
Art and Design
Language and Translation
Research and Analysis
Conversational Dialogue
Data Analysis and Visualization
Business and Finance Education and Learning
Science and Technology
Health and Wellness
Personal Development Entertainment and Humor Travel and Leisure
Markeling and Sales
Game Development Miscellaneous
Be sure provide the exact category name without any additional text.
Figure 11. Prompt to assign one of the 20 categories to user instructions
17

---

Thinking LLMs: General Instruction Following with Thought Generation
Prompt to evaluate responses with GPT4
<system>>
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants the user prompt displayed below. You will be given assistant A's answer and assistant B's answer. Your job to evaluate which assistant's answer is better.
Begin your evaluation by generating your answer to the prompt. You must provide your answers before judging any answers.
When evaluating the assistants' answers, compare both assistants' answers with your answer. You must identify and correct any mistakes or inaccurate information.
Then consider the assistant's answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it more helpful and appropriate ask for clarifications based assumptions. Relevant means all parts of the response closely connect asked. Concise means the response is clear and not verbose excessive.
more information from the user than providing an answer
appropriate to what is being
Then consider the creativity and novelty of the assistant's answers when needed. Finally, identify any missing im- portant information in the assistants' answers that would be beneficial to include when responding to the user prompt.
After providing your explanation, you must output only one of the following choices as your final verdict with label:
1. Assistant significantly better: [A>>B]]
Assistant slightly better: [[A>B]
3. Tie, relatively the same: [[=B]]
4. Assistant B is slightly better: [[>>A]
5. Assistant B is significantly better: [[>>>A]]
Example output: "My final verdict is tie: [A=B]]]
< </system>
< User Prompt|>
{prompt}
<|The Start of Assistant A's Answer| {generation]
< The End of Assistant A's Answer|>
TTe Start of Assistant B's Answer|> <>
Ignnraation)]
< The End of Assistant B's Answer|>
Figure 12, Prompt to evaluate using GPT4 judge in the fine-grained evaluation.
18

---

Thinking LLMs: General Instruction Following with Thought Generation
Prompt to evaluate responses with STE judge
Please act as an impartial judge and evaluate the quality the responses provided by two AI assistants to the user question displayed below, You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as helpfulness, relevance, accuracy, depth, creativity, and level of detail their respouses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do allow the length of the responses tion. Do not favor certain names of the assistants. Be objective possible. After providing your explanation, oulput your final verdict by strictly lollowing this format: "[[]]" if assistant A is better, "[B]]" if assistant B is better.
influence your evalua-
[User Question] [input}
(The Start of Assistant A's Answer] (generation)
(The End of Assistant A's Answer]
[The Start of Assistant B's Answer]
(generation2]
[The End of Assistant Answer]
Figure 13. Prompt to evaluate using STE judge in the fine-grained evaluation.
400
350
Thought Len
300
250
Creative
Whiting
Reweurch and Analysis
anr Saiee
Data Anal Analysii and Visustizstionn
Gome Develogmentt
Content Writing
Personal Development
Programming and Codiig
Science and Technology
Health and Wellness
Thaee and
Edunacion and Learning
Reasoning and Probilen-S Sol Vitee
cimmettt and Humor
Converss tiennl Dialagui
Language and Transiation
Geanral Knoow ledlg
Businoss
Math
Generic Thought Prompt
1800
Thought (chars)
1600
1400
1200
1000
Gane
Developurent
Resoucch and Analyale
Leiaare
Art and Design
Marlorting and Sodtes
Pensonall Devolopment
Creedivve Wasting
Health and Wellnose
Content Weiting
Programming and Cooling
Data Analpale and Vanultattion
Selence and Techuologg
Educcat fon and Leaening
Entertain terreieennn Humorr
Ressoning and Prohlem-Solvi
Convesattoonl Didlogae
Geperal Knowleigg
Language and Tra
Business
Moth
(b) Specific Thought Prompt
Figure 14. Length of thoughts generated by TPO models.
19

---

Thinking LLMs: General Instruction Following with Thought Generation
Example of Non Reasoning Instruction (generic thought prompt)
Instruction: What type of soil is suitable for cactus?
Here is my thought process:
need think about the specific requirements of cacti determine the suitable soil type.
recall that cacti are native to dry and arid regions, so the soil must be adapted to conserve water.
T need consider the pH level of the soil, as well.
Here is my response:
Cacti thrive well-draining soils that mimic their natural desert habitats. suitable soil type for cactus cultivation typically possesses the following characteristics:
High drainage: Cacti detest waterlogged soil, mix with excellent drainage is essential. A combination of perlite, vermiculite, or sand helps prevent rool rot.
Neutral to slightly alkaline pH: Most cacti prefer a slightly acidic to neutral soil pH (6.0-7.5), which similar to their native soils in the desert Southwest United States and Mexico.
3 Low organic matter: Cacti don't require a lot of nutrients, and excessive organic matter can retain too much water. low-to-medium organic matter content (around 5-10%) is suitable.
4 Good seration: A well-balanced mix allows for air pockets and oxygen circulation around the roots, promoting healthy growth.
general-purpose cactus potting mix typically contains blend of:
Coarse grit or perlite (40-60%)
Peat moss coconut husk (10-20%)
Vermiculite (10-20%)
Composted cactus mix or worm casting (5-10%)
Small amounts of dolomitic limestone or crushed eggshells (pH adjustment and calcium supplementation)
You can also create your own cactus soil mix by combining these components in the above proportions. Avoid using regular potling soil, it retain too much water and potentially harm your cactus. By using soil that meets these criteria, you'll be well your way to growing happy and thriving cacti.
Figure Example of TPO answering non-reasoning question. This example demonstrate how the model learns to provide key points in the thought process (in green) and then expands on these points in the actual answer.
20

---

Thinking LLMs: General Instruction Following with Thought Generation
TPO solves a complex task (specific thought prompt)
Instruction:
Write program to compuse the Blo scores chess tournament.
Internal Thonghts
Initial Thoughts: Oh boy, this sounds like a challenging query! hope can break it down into manageable parts. I'll need to understand rating system,
design program to calculate ratings each player tournament. This might involve some structures, loops, and conditional statements. I'll need to consider edge cases, like incomplete matches or uneven player numhers.
Draft Respouse: Here's possible approach:
The program will require the following imputs: A list players with their initial Elo ratings A list of matches with the winner and loser for each round Optional: K-factor (default to 32, a common value) The algorithm
through each match, applying the rating formula update the ratings. For each match:
Calculate the expected score for the winner and loser based on their current ratings.
2 Update the winner's rating using the Elo formula:
new.cating ing rating K + (1 expt [eexpecLed.winnes observed "2 / 4001))
Update the loser's 's rating similarly: new rating rating
exp (eexpected loser observed loser) '2 400))
4 Assign the updated ratings back to the players.
To handle the tournament structure, I'll need the matches. This might involve recursive function or a loop with memoization.
the matches data structure, like graph or matrix, efficiently
Evaluation the Draf Respouse: This respouse provides general outline However, it lacks specific implementation details, such as data structures, and the algorithm's 's flow is not clearly defined. The response could benefit from more concrete examples and more detailed breakdown of the calculation process.
program's 's requirements the rating formula.
Final Response:
Elo Rating Calculator for Chess Tournament
Python program calculates ratings formula with optional K-factor.
tournament given match results and initial ratings. the staudard
nath
BlcRatinngs louttinggsllulateenn (self, k_fastoz=221: mels.k_factor k_factor
def oeleclate_ eeoosesaa ratingl, sasing2):
inroaloulate expected score player expected
recings nath.pow (10, irasing? rating))
expedsed sereoeed_sco
update rating rating, expected score, scoor,, ranopdate player's Bic zating based
zosuignes
ofrgin_of_ of_toooryy result expoctad_scom nengracing sating self.k_facteer mergir.u_.viettrr cutarn ravgrating
def celcclate res rngingteee tself, natches, initial retingsl: mulateulate Rio recings for each player after the tournamene retings initial mastisi__rtings.co ratingg.ccpyy(
for viacer, losez in matcaes:
expected_xinnar
ratings[[oser]]
expecielLosaer 1 expectes_vvanat
razings [vinncr] racings [loaw:] self, uud..epddees_ede ret ings coaec), snpected_oaee, 0)
uttauuddao___attle (xamee)s(ximmee)) expectet 2)
zotarn sasings
def nain(::
EXampiс usager
calculator - SloBstinggsaccuattoo tactoor))))
initial ratinge ('Flayerl' 1200, "Player2': 1906, "Player3"; 1100;
(1ayee2"),, ("P1aver2", "Blayoz3"), ("Playez3', 'Flayer1')] natches = ['''aayerr', tinal_ratiag
www.oliccaccccceccc.ooo pnint (*Final Satings:")
imatches, initioi_ratinga)
for playes, sating final iiiiigg.iieall) itenal): print (c"iplayeri (ratingi")
":
This program defines a BloRat ingCalculator class encapsulate the Elo rating logic. The calculate wwwp:laateexpetee..coo eepeectdd.ecooee method calculates the expected rating based on their result and expected score. The calculate.rating method iterates through the matches, updating the ratings accordingly.
for player based on their rating and opponent's rating. The update. rating method updates player's 's
In the main function, we demonstrate the usage by calculating the Elo ratings for three players after tournament with specified set of matches and initial ratings. Feel free to modify the K-factor, initial ratings, and matches to sult your specific tournament needs.
Figure 16. TPO with the specific thought prompt answering complex reasoning task.
21

---

Thinking LLMs: General Instruction Following with Thought Generation
Direct baseline fail to answer correctly
Instruction: Write a program to compute the Elo scores chess tournament.
Here's a Python program calculate Elo ratings a chess tournament. This script assumes the tournament a round- robin, and each player plays every other player once.
class BlcRating:
(aelf, initiel ralinggg1110)
sels, faccore inilial_retinng
aulf. wuli.iaitial_
"E": indeisi_ Expected gane anisiai_raatnng Rating
celaciate_ oxtpeessepccccceettteeeeee helf, ratingi, rating2):
vangaloulate the expected outcome (ein probebility? player
(1 nath.pow(10, leating? sa.ingl) 400);
def update ratingiself, ratingl, rating?, outconel:
Vanipdate net ing player after the game, www..ccleciitt..o.. btccuittt_xxpecttooottooooo (esting), rating?) it outcome 1: Hin
self. factioa_ccccoos."
ratingl self.sacing (1 4]
alld gatcoma 0: Lees
wel...a.inn............... factorsss"
ratingl
(e
alset Deas
selt. rating factorss (3"] ratingl relt.raciiggtactoree return celt.rating ractors (13' (131)
def cournmmenttttt (solf, deayers_ratinge sutcomes): e*rRun the cournanent and update ratings.
for i
(Len(players for 1 in zangeli poterrrstttinaal.....
wsnpplavera_aatineai
rating), rating? plagers_ratings eutcome outtooee
saling] sel.fuppaae ingaainnnn,,, saling2, outesse)
rating2 selt.opdate ratattttrrtinnrrtinnn... ratingl, ouscome)
Opponent's outcone
players iatings players ratings(]] ratingl, ratingo
piayars_ratings
Exancle
BlcRating
players (1Playerll 'Player2', 'Player3")
initial_ratin [1400, 1300, 1250]
111. 11, 11, 1, D., [0, 0]1 (3), (0),
aatiias_atttr_ourraaaa prins(""inal Ratinçs:")
too.ranttournnnnnt moittaal_rccingaccom outcomas)
playes, sating in sipfplayers, hirrrpaaaaterrtttcaaeeee print (f"(payyez): trating["]
How works:
Initialize EloRating with optional initial rating (default: 1200).
Calculate the expected outcome (win probability) for player using the Elo formula.
3. Update player I's rating based on the outcome: win (+e*()-e)), loss (-K*e), or draw (+K*(
4. Run the tournament by iterating over all player pains, updating their ratings accordingly.
Output the final ratings for each player.
Note: This script assumes a round-robin tournament where each player plays every other player once. For other tournament formats, you'll noed to adjust the pairing logic.
Figure 17. The direct baseline makes mistake in update. update.rating.
22

---

Thinking LLMs: General Instruction Following with Thought Generation
Example of Overthinking (specific thought prompt)
Instruction: Make one line python code get list of primes from 1 200 lambda function and list comprehension
Internal Thoughts
Query Analysis: Okay, so we need generate a list of prime numbers from 200 using Python, and we're supposed to lambda functions and list comprehension. That's an interesting combination! Draft Response: Here's my initial attempt:
primes 1ist (filter (lambda x: all (x&i for in range (2,x)) x)) and x>1, range (1, 201))>
Evaluation: Hmm, this code works, but it's not very efficient. The all () function will short-circuit soon as it finds divisor, which is great for correctness, but not ideal for performance. Also, the lambda function is doing a lot of work. I'm not sure if this is the most Pythonic way to solve this problem.
Final Response:
Here the Python code using lambda function and list comprehension to generate of primes from 200:
primes st (fi ter (1ambda x: all (x&i>l for range (2,x)) x==2, (p for p
range 201) if all(p8i>) for i in range (2,p1))))
Wait, what?! That's a terrible idea!
Let's 's start fresh! Here's much simpler and efficient solution:
primes = P for in range (2, 201) is all (psi>1 for i in range (2, int (p++0.5)+1))]
Still not perfect...
Okay, okay! Let's try again! (continued ...)
Figure 18. Example of failure from our TPO model trained with the specific thought prompt. The model answers correctly in the first draft but incorrectly the second and third time.
23

---

Thinking LLMs: General Instruction Following with Thought Generation
Example of thought process helping but still incorrect (before training)
Instruction: A number is OxA4F1 in hex. What number is this in decimal?
Direct response:
nice hex-to-decimal conversion!
In hexadecimal, A4F1 represents the decimal number:
10 (A) x 16"2 2560 4 (4) 16 64
(1) x =6
Adding these up, we get: 2560 64 2625 So, the decimal equivalent of OxA4F1 2625.
Response with the generic thought prompt:
Here is my thought process:
need to convert a hexadecimal number to a decimal number. know that each hexadecimal digit can represent bits (or half a byte), and each byte can represent a decimal value from 0 to 255. also know that can convert each bexadecimal digit to its corresponding decimal value using the standard mapping: A=10, B=11, C=12, .. F=15.
To convert the hexadecimal number OxAAF1, I will start by converting each digit separately:
A is 10 in decimal
is 4 in decinal
* FF is 15 in decimal
* #1 is 1 in decimal
So, the decimal equivalent of hexadecinaal number OxA4F1 is:
10 * 16'3 + 4 16"2
15 x 16 + 16"0
10 4096
* 256
15
16
=40960 1024 + 240
=42125
Here is my response:
The decimal equivalent of the hexadecimal number 0xA4F1 is 42125.
Figure 19, Both responses generated using the seed model Llama-3-8B --s--ustt before training, with and without the generic thought prompt. We see that even though the answer wrong (correct answer is 42225), the thinking helps to get it closer.
24

---

Thinking LLMs: General Instruction Following with Thought Generation
Example of thought process helping with self-correction in math tasks
Instruction: Tara bought 8 packs of 5 canvas bags for 84 each. She painted them and sold them at craft fair for 88 each. How much profit did she carn on her bags?
Direct response:
packs of bags each means 8 x 40 bags.
40 bags cost x = 32 dollars.
Selling them for dollars each means 40 x 320 dollars.
The profit is 320 32 288. The answer is 288.
Response with the specific thought prompt:
Here is my thought process:
* **Internal thoughts:**
**Draft Response:**
Okay, let's break this down step by step. Tara bought 8 packs of canvas bags, each with 5 bags. So, she purchased total of 8 x 5 = 40 bags. Each pack cost $4, so her initial investment was 8 $4 $32. Then, she sold each bag for 88. To find the total amount she made, multiply the number of bags by the selling price: 40 bags $8 3//bag- $320. Now, to calculate profit, subtract the initial investment from the total amount made: $320 $32 $288. Wait, this can't be right... profit is usually positive value, not huge sum exceeding the initial investment!
**Eaalaaionn
Hmm, something's off. The calculation looks correct, but the result seems absurd. Profit should be reasonable amount, not nearly ten times the initial investment. I'm missing something. Maybe I should recheck the math or rethink the approach. Ugh, hate when I'm unsure about my responses.
** Finiial Response:**
<R>
**Profit Calculation:**
1. **Cost**, Tara bought 8 packs of 5 canvas bags for $4 each, so her initial cost is: 8 packs x 5 bags/pack x $4/bag
$160
2. Revenne**; She sold each bag for $8, so her total revenue is:
40 bags $8/bag = $320
3. **offt Calculate the difference between revenue and cost:
Profit Revenue
Cost = $320 $160 **$160**
Tara earned profit of ** by selling the painted canvas bags at the craft fair. She made 88 each, which markup of $4 per bag ($8 $4), resulting in total profit of $160.
Figure 20. Comparison of responses generated by direct model and TPO model question from the GSM8K test set. The TPO model initially makes an error but then corrects it, whereas the direct model stops at the mistake.
25

---

Thinking LLMs: General Instruction Following with Thought Generation
Average Response length (chars)
Average Reward Score ny ArmaRM
1827
0.1588
0.1588
0.1586
1775
0.1584
1750
1725
0.1580
1700
1685
1875
0.15//
0.1576
1650
0.1575
0.1574
Direct
TPO Generic
TPO Specific
Direct
TPO Generic
TPO Specific
Figure 21. The final model response lengths are shown with their corresponding ArmoRM average score on the UltraFeedback validation All models shown iteration 4.
26


Reviews:
[{"review_id": "lv3tmk0E3S", "reviewer": "Reviewer_7sak", "summary": "This paper presents a method and studies how to get LLMs to output initial thought traces before a final answer on instruction-following tasks. Their main idea is to prompt LLMs to initially produce these thought traces before a final response, score just the final response with an LLM-as-a-judge, and train over pairs of entire thought + final response sequences rated by just the response parts using DPO. Doing so enables training \"thinking LLMs\" without the need for any human-sourced thoughts, and leads to improved performance on AlpacaEval and Arena-Hard benchmarks (>50% win rate against direct finetuning on good responses). They finally perform finer-grained analysis to see where the thinking helps across instruction topic, and study properties such as impact of initial prompt type and thought lengths.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:\n\n---\n\n> Lack of model support. All the experiments for the model generation are done on 1 model: Llama 3.1 8B Instruct...\n\nWe acknowledge the reviewer's concern regarding model diversity. While we aim to conduct further evaluations involving larger models such as the latest Llama 3.3 70B and Deepseek R1, our current model selection is primarily constrained by limited training resources. Subject to the availability of time and resources, we intend to expand our experimental scope to include a wider range of models. However, this expansion may be relegated to future work.\n\n> Lack of understanding or insight into why thoughts help...\nFor example, what kinds of patterns or additional context emerge from the thoughts that contribute to higher quality responses?\n\nWe agree that understanding why thoughts contribute to better responses is crucial. Our analysis identified several patterns within the thought processes that correlate with higher-quality final responses:\n\n- **Generating Checklists/Key Points:** As illustrated in Figures 6 and 15, the model often drafts a preliminary list of essential elements or topics it determines should be included or emphasized in the final answer, effectively creating a structured plan.\n- **Iterative Refinement:** Figures 5 and 16 demonstrate instances where the model refines its potential answer. This includes augmenting initial drafts with stronger arguments, incorporating more specific examples, or restructuring the content for clarity.\n- **Self-Correction and Evaluation:** Figure 18 provides an example of the model engaging in self-correction and evaluation. Notably, this reflective behavior can emerge even when it leads to deviations from pre-specified output formats.\n\nThese observed behaviors are consistent with established strategies known to enhance the quality and reliability of responses generated by large language models.\n\n> Do different patterns emerge for different topics? How robust is this emergence? ...\n\nWe appreciate the reviewer inquiring about the robustness and specificity of these patterns. We didn’t statistically evaluate and classify these thought patterns because it requires eye-balling and does not have a way to do it at scale. However, we want to emphasize that some of the behaviors exhibited by our thought model are nearly never seen by a direct answer model, for example like the self reflection, refinement and self reminder behaviors, which we believe are the main reasons why our model can perform better.\n\n>  I appreciated the study on length-control and DRO vs IRPO, but some parts of the method came across a bit ad-hoc...\n\nRegarding length control, our approach incorporates established mechanisms documented in prior work. Extensive research has demonstrated that without effective length constraints, model performance can be significantly impacted, often negatively. We adopted these standard techniques primarily to ensure fair and meaningful comparisons between different methods evaluated in our study, as consistent length controls were applied across all conditions. This also enhances the practical relevance of our findings by preventing models from artificially inflating reward scores through excessive verbosity.\n\n> if this is a result strictly from preference optimization (TPO), or could we get Thinking LLMs via purely an outcome-based signal ...\n\nRegarding the outcome-based signal, it aligns with methodologies explored in recent literature, such as DeepSeek-V1. Our proposed framework, which explicitly separates the generation process into distinct 'thought' and 'answer' components, is indeed adaptable to such outcome-based evaluation paradigms. By evaluating only the final 'answer' part, the framework grants the model greater flexibility in the 'thought' generation phase, encouraging more exploration during reinforcement learning, potentially leading to more diverse reasoning strategies and ultimately improving final task performance. Furthermore, alternative optimization algorithms like KTO or RLVR could readily replace DPO within our framework; our initial selection of DPO was based on its established effectiveness and relative simplicity of implementation.\n\n> The method comparison is also a bit lacking, where I think the authors should at least compare against STaR. \n\nSTaR requires ground truth answers, in contrast, TPO requires only a reward model. These fundamental differences in data make a direct empirical comparison challenging. Furthermore, STaR and its variants were primarily evaluated on reasoning-intensive tasks, whereas our work focuses on general instruction-following capabilities.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions and clarifications.", "claims_and_evidence": "The claim of \"investigating the possibility of converting existing LLMs into Thinking LLMs that work across\na wide variety of tasks, without any additional data\" is supported, albeit modestly. \n\n**Support / positives**  \n* Experiments demonstrate that this paradigm works through their proposed method   \n* The authors pick several interesting axes of study / points of comparison, such as (1) prompt for structuring the initial thoughts, (2) different judge models, (3) post-training procedure (DPO vs IRPO)     \n* They also find some interesting empirical nuggets, such as the initial responses following thoughts being worse than the initial instruction-tuned model's. But then improving via the procedure. I found the example of self-correction on GSM8k also a nice highlight, though it would have been interesting to see how frequent this phenomenon occurred  \n\n**Insufficiencies / negatives**   \n* **Lack of model support**. All the experiments for the model generation are done on 1 model: Llama 3.1 8B Instruct. While this is a pretty modest model not known for having \"reasoning\" capabilities (i.e., it's cool to see these thinking traces emerge here), just showing the evaluation on one model seems insufficient re: the scope of a general method, and \"converting existing LLMs into Thinking LLMs\" at large.  \n* **Lack of understanding or insight into why thoughts help**. I appreciated the study into the different topics and how thinking could help to varying degrees, as well as the examples of when thinking helped or hurt in the appendix. However, I would have liked to see more (hypotheses + validation or not) on why the thought processes help, especially on non-reasoning instruction-following tasks. \n  * For example, what kinds of patterns or additional context emerge from the thoughts that contribute to higher quality responses?   \n  * Do different patterns emerge for different topics?  \n  * How robust is this emergence? e.g., the Figure 15 response example doesn't strike me as something that benefits truly from the thoughts. The response also seems like it could have come from the model without thoughts.  \n* **Justification for parts of the method**. I appreciated the study on length-control and DRO vs IRPO, but some parts of the method came across a bit ad-hoc. e.g., what was the motivation for why the preference pair building was done as proposed? Could other techniques work?  \n  * Regarding the question on studying whether things are possible, I'm curious if this is a result strictly from preference optimization (TPO), or could we get Thinking LLMs via purely an outcome-based signal (answer the question or not) and techniques that just use this (KTO, SFT after sampling for positive sequences, RLVR) \n\nNit, but the phrasing \"we allow the model to independently learn to think\" (L079) is a bit misleading, given that we use *additional* (larger) LLMs for prompt generation (Llama 70B) and response scoring (STE, ArmoRM)", "methods_and_evaluation_criteria": "I think AlpacaEval and Arena-Hard are reasonable given the span of instruction categories (e.g., marketing, health and general knowledge).\n\nHowever, as so far as contrasting against the popular \"logic-based tasks like math or coding\", did the authors consider additional broadening the tasks or \"skills\" beyond general instruction-following (and evaluation based on LLM-as-a-judge preferences)?  \n* For example, can TPO and reasoning help with non-logic tasks like summarization or question-answering over (long) contexts?   \n\nThe method comparison is also a bit lacking, where I think the authors should at least compare against STaR. As the authors point out in their related work (L431): \n> However, these methods rely on supervised training so ground-truth thought data is required. STaR (Zelikman et al., 2022) removes this constraint by generating both thought and answer from a model using few-shot prompting.\n\nGiven that STaR also enables reasoning thoughts without the need for human thought data, and can be applied to the instruction-following tasks (e.g., filtering by using an LLM-as-a-judge on whether the response satisfies the instruction or not), it seems worth comparing to assess the novelty + impact of contribution for TPO.", "theoretical_claims": "N/A. No theoretical claims made.", "experimental_designs_or_analyses": "Yes. I checked model comparison, benchmark selection, and ablations. See issues pointed out in **Claims And Evidence** and **Methods And Evaluation Criteria**. \n\nNamely:  \n* Lack of support for Thinking LLMs beyond Llama 3.1 8B Instruct  \n* Evaluation on \"only\" AlpacaEval and Arena-Hard (or at least beyond the topic-based granularity breakdown, I think looking into the nature of the skills needed to follow the instructions faithfully [e.g., question-answering, summarization, content generation] and comparing this to the improvement in performance via the thoughts would be more insightful (an example of this is the factoid highlight in Figure 5, thought a more systematic analysis would be better).", "relation_to_broader_scientific_literature": "The authors show interesting results where thinking can emerge for non logic-based tasks, can be done without needing to do SFT on human thought traces, and can help non-logic based task quality.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "See supports / positives in above claims response.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Did the authors consider studying the cost-quality trade-off between having to generate more tokens in the hidden thoughts vs direct prompting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LrVXpuV2fw", "reviewer": "Reviewer_kACw", "summary": "The paper proposes a method to enhance LLMs by enabling them to \"think\" explicitly before generating responses. This is aimed at improving performance on complex tasks requiring reasoning and planning, as well as general instruction-following tasks. The authors introduce the so-called Thought Preference Optimization (TPO), a training method that equips LLMs with the ability to generate internal \"thoughts\" before producing a response. These thoughts are hidden from the user and serve as an intermediate step to improve response quality. The method does not require additional human-labeled thought data.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each concern raised and propose revisions:\n\n---\n\n> Are the thoughts interpretable and aligned with the optimal or correct human-like reasoning steps?\n\nWhile our methodology does not impose explicit constraints on the structure or content of the thoughts during reinforcement learning, our analysis revealed the emergence of patterns that resemble effective human problem-solving strategies. These include:\n\n- Generating Checklists/Key Points: As seen in Figures 6 and 15, the model often formulates preliminary lists of elements to include or emphasize in the final answer.\n- Iterative Refinement: Figures 5 and 16 show the model refining potential answers within the thought process, adding arguments or examples.\n- Self-Correction and Evaluation: Figure 18 illustrates the model engaging in self-correction, sometimes even overriding pre-specified formats.\n\n> It's better to include a qualitative analysis of the generated thoughts to assess their interpretability, alignment with human reasoning...\n\nWe agree that a dedicated qualitative analysis assessing thought interpretability is a valuable suggestion. However, rigorously and explicitly evaluating the intrinsic quality of machine-generated thoughts presents significant challenges. First, benchmarks specifically designed for assessing thought processes are not well-established. Secondly, defining objective criteria for a \"good\" thought is inherently complex. For instance, is a thought process containing flawless step-by-step logic but ultimately unhelpful superior to one that suggests a promising overall direction despite containing intermediate errors? Developing methods for more direct evaluation of thought quality itself remains an important direction for future research.\n\n\n> It builds heavily on existing techniques like Direct Preference Optimization (DPO) and Reinforcement Learning from AI Feedback (RLAIF)...\n\nWe wish to clarify that the core innovation of TPO is not presented as a fundamentally new type of reinforcement learning or preference optimization algorithm itself. The primary objective of TPO is to demonstrate that substantial performance gains are achievable by affording the model this unjudged \"thinking space,\" without requiring significant alterations to the underlying training algorithm architecture.\n\nWe provides empirical evidence supporting this claim through direct comparisons with standard DPO and a CoT approach (representing the iteration 0 before TPO). As presented in our results, optimizing the hidden thought process via TPO yields significant performance improvements over these methods.\n\n> ...specific thought prompts perform slightly better than generic prompts, but the difference is marginal...\n\nSpecific thought prompt introduce negligible implementation complexity because the TPO training methodology remains identical irrespective of the prompt format. While average performance gains across all tasks may appear modest, specific prompts yield substantial improvements on challenging benchmarks, notably ArenaHard. \n\n> It is better to also evaluate the computational cost...\n\nWe’ll add discussion and plots of the generation cost vs the final performance in our final draft.\n\n> investigate why TPO underperforms on math tasks...\n\nWe hypothesize this may stem from the deficiency of the reward model we used, which focused on general instruction following rather than specialized mathematical reasoning. For domains requiring strict correctness like mathematics, directly evaluating answer accuracy against reference solutions, might be more effective than relying solely on the preference judge. Incorporating such correctness checks represents a promising direction for future work.\n\n> the core idea builds heavily on existing techniques like DPO and RLAIF...\n\nTPO's contribution lies not in proposing a new optimization algorithm variant, but in demonstrating the methodology of optimizing an free-form thought process based only on judgments of the final answer. This indirect supervision approach facilitates the emergence of complex reasoning behaviors during training, which are not seen in standard answer only training.\n\n> Are there examples where the thoughts are nonsensical or misleading, despite leading to good responses\n\nWe did not observe cases where nonsensical thoughts produced high-quality final answers; we hypothesize the KL regularization used during training helps maintain coherent and interpretable thought structures by penalizing significant deviations from the base model's.\n\n\n> It's better to evaluate the computational cost of generating and optimizing thoughts relative to the improvements in response quality.\n\nWe’ll add analysis in the final draft.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which improves the clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions and clarifications.", "claims_and_evidence": "1. Claim: Generated thoughts are meaningful and improve response quality.\n\nThe paper focuses on optimizing thoughts based on the quality of the final response but does not rigorously evaluate the quality of the thoughts themselves. For example, are the thoughts interpretable and aligned with the optimal or correct human-like reasoning steps?\n\n2. Claim: TPO introduces a novel and significant advancement in LLM training.\n\nWhile the method is well-executed, its core idea—using preference optimization to improve intermediate outputs—is not fundamentally novel. It builds heavily on existing techniques like Direct Preference Optimization (DPO) and Reinforcement Learning from AI Feedback (RLAIF).  A clearer comparison with existing methods (e.g., standard DPO, CoT) and a discussion of how TPO uniquely advances the field would strengthen this claim.", "methods_and_evaluation_criteria": "1. The proposed method focuses on optimizing thoughts based on response quality but seems to lack the rigorous evaluation of the quality of the thoughts themselves. For example, are the thoughts interpretable and meaningful? T\n\n2. The proposed method underperforms on math tasks (e.g., GSM8K), which is surprising given the emphasis on reasoning. This suggests that TPO may not be well-suited for tasks requiring precise, step-by-step reasoning. The authors attribute this to the small proportion of math-related instructions in the training data, but this explanation feels insufficient.\n\n3. The paper claims that specific thought prompts (e.g., drafting and evaluating responses) perform slightly better than generic prompts, but the difference is marginal. This raises questions about whether the added complexity of specific prompts is justified.", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "1. The paper claims that specific thought prompts (e.g., drafting and evaluating responses) perform slightly better than generic prompts, but the difference is marginal. This raises questions about whether the added complexity of specific prompts is justified.\n\n2. It is better to also evaluate the computational cost of generating and optimizing thoughts relative to the improvements in response quality. This would help establish the method's practical value.\n\n3. It may be also helpful to investigate why TPO underperforms on math tasks and explore ways to improve its performance in this domain.", "relation_to_broader_scientific_literature": "The key contributions of the paper is related to areas like CoT prompting, thought generation,  iterative training and self-Improvement.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: \n\n1. The paper evaluates the proposed method, Thought Preference Optimization (TPO), on well-established benchmarks like AlpacaEval 2 and Arena-Hard, demonstrating clear improvements over baselines. The use of GPT-4 as an auto-evaluator adds credibility to the results.\n\n2. The fine-grained evaluation on 20 categories (e.g., marketing, health, math) provides valuable insights into where TPO excels, showing gains even in non-reasoning tasks.\n\n3. The finding that TPO improves performance in non-reasoning tasks (e.g., marketing, health) is interesting and suggests that \"thinking\" may have broader applications than previously assumed.\n\nWeakness:\n\n1. The method builds on existing techniques (e.g., DPO, RLAIF) and does not introduce significant algorithmic novelty.\n\n2. The quality of the generated thoughts is not rigorously evaluated, and their interpretability remains unclear.\n\n3. The practical benefits of the method, such as computational cost and real-world applicability, are not thoroughly analyzed.\n\n4. The paper claims that specific thought prompts (e.g., drafting and evaluating responses) perform slightly better than generic prompts, but the difference is marginal. This raises questions about whether the added complexity of specific prompts is justified.\n\nIn summary, while the application of preference optimization to thought generation is interesting, the core idea builds heavily on existing techniques like DPO and RLAIF. The paper does not introduce a fundamentally new paradigm, making the work feel incremental.", "other_comments_or_suggestions": "1. How interpretable are the generated thoughts? Can they be used to debug or explain the model's decision-making process?\nAre there examples where the thoughts are nonsensical or misleading, despite leading to good responses?\n\n2. It's better to include a qualitative analysis of the generated thoughts to assess their interpretability, alignment with human reasoning, and potential utility for model transparency.\n\n3. It's better to evaluate the computational cost of generating and optimizing thoughts relative to the improvements in response quality.\n\n4. It is better to investigate why TPO underperforms on math tasks and explore ways to improve its performance in this domain.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IBjFzkenXx", "reviewer": "Reviewer_8ZuN", "summary": "Authors propose TPO, a method that finetunes an instruction-tuned LLM to output discrete thought tokens for harder tasks, without any supervision signal. The model undergoes iterative RLFAI preference learning, where the reward model comes from a judge model that judges based on the LLM’s final answer. Finally, authors show TPO achieves performance improvement against baselines on AlpacaEval and Arena-hard.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable and insightful feedback. We address your concerns as follows:\n\n---\n\n> How would simply doing this SFT compare with using TPO?\n\nThat's an insightful question regarding potential training alternatives. The Llama 70B model in our study functions strictly as a reward model, similar to the other RM evaluated (ArmoRM), providing reward signals rather than generating target outputs for supervision. Our TPO framework utilizes these reward signals and is agnostic to the specific nature of the reward model (e.g., a traditional RM or an LLM-as-a-Judge). This contrasts fundamentally with the proposed alternatives, which would use the 70B model as a generator to create a dataset for supervised fine-tuning (SFT). Because TPO operates via preference optimization driven by reward signals, while the suggested approaches rely on SFT using generated targets, a direct comparison might not be appropriate.\n\n> How would TPO compare with parallel sampling techniques? I.e. sample multiple answers and use the judge model to select the final answer.\n\nWe appreciate the reviewer raising the comparison with inference-time techniques. TPO as a training-time optimization method, is orthogonal to inference-time techniques such as sampling multiple responses and selecting the best using a reward model. These approaches are not mutually exclusive; applying inference-time sampling and selection strategies to a model already optimized with TPO is a plausible approach. Investigating the interplay and relative benefits of TPO training versus various inference-time selection methods remains an interesting direction for future work.\n\n> authors mention several prior works on thinking tokens that also do not require any labeled data (STaR, Quiet-STaR), is there a reason why authors didn’t compare empirically with those two baselines?\n\nThank you for asking about the comparison with STaR and Quiet-STaR. STaR requires ground truth answers, and Quiet-STaR relies on supervised fine-tuning data. In contrast, TPO requires only unlabeled prompts and preference signals from a reward model. These fundamental differences in data make a direct empirical comparison challenging. Furthermore, STaR and its variants were primarily evaluated on reasoning-intensive tasks, whereas our work focuses on general instruction-following capabilities.\n\n> Can we also see a variant of Figure 4 with win rate against the seed model? As authors have already mentioned, COT only helps with math/logic related tasks, so seed model should be a stronger baseline here for most of the categories shown.\n\nWe understand the reviewer's interest in seeing a direct comparison against the base seed model. The 'seed model' refers to the base pre-trained model prior to any preference optimization. The 'direct baseline' actually presented in Figure 4 is this seed model after it has undergone standard Direct Preference Optimization (DPO) training directly on final answers. This DPO-optimized direct baseline is generally stronger than the original un-optimized seed model, making it the more relevant and challenging point of comparison for evaluating TPO.\n\n> Authors show TPO do worse than baselines in GSM8K, any ideas why it does better than the baseline in the “math and calculation” subset of ultrafeedback?\n\nThis is a keen observation regarding the differing performance on math-related datasets. These two datasets exhibit significant distributional differences in the types of mathematical problems presented. We hypothesize it’s because the reward model we used is better at judging questions in the ultrafeedback distribution but bad on the GSM8k distribution.\n\n> Why is it problematic that “the seed model … uses CoT anyway due to its instruct training”? If it is because the accuracy is too high for further improvement, can authors also evaluate on MATH, which is a harder task than gsm8k?\n\nWe understand the reviewer's query regarding the implications of the seed model's inherent CoT capabilities. The goal of that experiment was to understand the effect of thinking and CoT on gsm8k performance. In particular, we tried to measure the accuracy of the seed model when directly outputting answers without any CoT. However this was tricky to measure because the model often did CoT before answering, sometimes even when explicitly instructed to not to do CoT \n\n> Authors mentioned that the poor performance on GSM8K might be “due to our setup not being oriented toward such tasks”. Have the authors tried to simply add the training examples in GSM8K as a part of the training data for TPO?\n\nThat's a relevant suggestion concerning the training data composition for specific benchmarks. We did not include examples from the GSM8K training set within the dataset used for TPO training in this study. This is an interesting direction which we’ll leave for future work.\n\n---\n\nWe deeply appreciate the reviewer’s feedback and hope our responses fully address your concerns.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "* TPO uses a judge that is finetuned from Llama 70B. What if the authors simply prompt the 70B model with the thought prompts from figure 2, and SFT the smaller llama 8B with this dataset? How would simply doing this SFT compare with using TPO? An even more simpler variant is to just use the 70B to generate without the thought prompts, and SFT on this dataset.\n* How would TPO compare with parallel sampling techniques? I.e. sample multiple answers and use the judge model to select the final answer.  \n* In the related works section, authors mention several prior works on thinking tokens that also do not require any labeled data (STaR, Quiet-STaR), is there a reason why authors didn’t compare empirically with those two baselines?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "The authors propose TPO, a method for LLMs to learn to generate thinking tokens during inference time. Time-time compute scaling is an important and timely topic for LLMs capabilities research.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n* Paper writing and figure illustrations are clear and easy to digest.\n* Comprehensive evaluation and ablations.\n* The method proposed is simple and achieves good results in the evaluation tasks. \n* People have more commonly associated COT to work with reasoning tasks, the surprising results showing TPO improves on various instruction following tasks are interesting to see.\n\nWeakness:\n* Missing baselines that authors should include (see questions below).\n* See more questions below", "other_comments_or_suggestions": "N/A", "questions_for_authors": "* Can we also see a variant of Figure 4 with win rate against the seed model? As authors have already mentioned, COT only helps with math/logic related tasks, so seed model should be a stronger baseline here for most of the categories shown.\n* Authors show TPO do worse than baselines in GSM8K, any ideas why it does better than the baseline in the “math and calculation” subset of ultrafeedback?\n* Why is it problematic that  “the seed model … uses CoT anyway due to its instruct training”? If it is because the accuracy is too high for further improvement, can authors also evaluate on MATH, which is a harder task than gsm8k? \n* Authors mentioned that the poor performance on GSM8K might be  “due to our setup not being oriented toward such tasks”. Have the authors tried to simply add the training examples in GSM8K as a part of the training data for TPO?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKmvadLu6f", "reviewer": "Reviewer_zafG", "summary": "This paper introduces Thinking LLMs, a novel approach aimed at improving general instruction following in large language models (LLMs) by explicitly incorporating internal thought processes before generating responses. Traditional LLMs respond directly to user instructions without intermediate reasoning steps, which can be inefficient for complex queries. The proposed method, Thought Preference Optimization (TPO), enables LLMs to generate and refine internal thoughts in an unsupervised manner without requiring additional human-annotated data.\n\nThe key idea behind TPO is to:\n\nGenerate multiple candidate thought-response pairs for a given instruction using an instruction-tuned LLM.\nEvaluate the responses using a reward model that only assesses the response quality, not the thought process itself.\nOptimize the thought generation through Direct Preference Optimization (DPO) by selecting the thought-response pairs that lead to the highest-rated responses.\nIteratively refine the model, ensuring that the generated thoughts consistently contribute to better response quality.\nThe paper provides empirical validation across multiple instruction-following benchmarks, demonstrating that TPO-trained models:\n\nOutperform standard LLMs on AlpacaEval (+4.1% win rate) and Arena-Hard (+4.3% win rate).\nImprove response quality not just in reasoning tasks but also in non-reasoning domains like marketing, health, and general knowledge.\nReduce reliance on manual Chain-of-Thought (CoT) prompting by allowing LLMs to learn when and how to think.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedbacks. Below, we address your concerns and propose our revisions.\n\n---\n\n> Failure case analysis is limited. The paper does not discuss scenarios where TPO might fail, such as backtracking-heavy tasks or reasoning tasks requiring multiple revisions. Automated evaluation reliance—the paper depends on GPT-4-based judges, which may introduce evaluation biases that are not addressed.\n\nOur results on GSM8K, where both TPO and standard DPO degraded performance, illustrate a limitation. We attribute this primarily to the reward model's limitations in accurately judging complex, multi-step reasoning, highlighting that reward model quality is critical, especially for such tasks. We also acknowledge the potential for inherent biases in automated evaluations using LLM-based judges like GPT-4. To mitigate this, we implemented controls where feasible; for instance, output length was controlled across compared methods to prevent models from exploiting potential length bias in the automated evaluator.\n\n> Essential References Not Discussed\n\nWe agree these references are relevant and appreciate the reviewer pointing them out. We'll discuss these papers in our final draft.\n\n> The paper does not explore failure cases in depth. It would be helpful to provide more insights into when and why TPO fails, particularly in tasks that require iterative backtracking or fine-grained numerical precision. The evaluation relies entirely on GPT-4-based reward models, which introduces potential biases. A complementary human evaluation would help confirm the effectiveness of TPO in real-world scenarios. \n\nWe agree that a deeper analysis of failure cases would be valuable. Although the patterns TPO exhibits in general are generally helpful, such as drafting a reminder list for the answer (Figure 6 & 15), refining the response (Figure 5 & 16), and reflections, we also observed cases where TPO failed. Notably, in Figure 18, we found the model exhibits non-terminating behavior by iteratively criticizing its own answer without being able to produce a fix. This indicates the model might have a tendency to deviate from the thought-answer structure, despite being specifically trained to follow the structure.\n\n> The work primarily focuses on GPT-based models. It remains unclear how well TPO generalizes to open-source models such as LLaMA, Mistral, or Claude. While the method is technically sound, there is no theoretical analysis of whether TPO converges to an optimal thought process or if certain errors persist over time.\n\nRegarding the request for theoretical analysis, we are not aware of any theoretical analysis on the optimality of TPO. But our intuition why TPO worked is it provides more freedom for the LLM to explore during the RL process by putting no constraint on the thought part (as the judge does not know what is inside the thought), though some errors might persist, e.g. not following format, non-stopping behaviors, etc.\n\n> Does training with TPO introduce significant additional overhead compared to standard instruction tuning?\n\nTPO training doesn’t add additional complexity in implementation compared with DPO, but potentially generates slightly more tokens than standard DPO due to the thought part.\n\n> The paper could benefit from more real-world application demonstrations, such as deploying TPO-trained models in interactive agents or dialogue systems.\n\nThese are interesting ideas to explore in our future work.\n\n> How does TPO handle cases where the initial thought generation is incorrect? Does the model attempt self-correction, or does it remain committed to the initial flawed reasoning path?\n\nFigure 18 provides an example where the model recognized an error but failed to recover or correct it.\n\n> What are the primary failure modes of TPO? Are there specific task types where the thought-generation process degrades performance instead of improving it?\n\nOne primary failure mode stems from the reward model's limitations. If the reward model struggles to accurately judge the final answers for certain types of tasks (like complex reasoning), it can provide poor guidance, causing TPO to optimize in the wrong direction and potentially degrade performance more. Another observed failure, illustrated in Figure 18, involves process instability where the model might enter non-stopping loops, such as repeated self-criticism without resolution, or deviate from the intended thought-answer structure.  \n\n> Can TPO be integrated with retrieval-augmented generation (RAG) systems? Would the model's internal thought process benefit from external knowledge retrieval, and how would that affect optimization?\n\nThank you for the insightful comment; unfortunately, we are unable to provide a direct answer at this time, but it is an interesting topic worth exploring in the future.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work.", "claims_and_evidence": "The paper presents several claims that are supported by empirical evidence:\n\nClaim 1: Thinking LLMs improve instruction-following accuracy across various tasks.\n\nThe experiments on AlpacaEval and Arena-Hard demonstrate consistent gains over direct-response baselines, with TPO achieving a 52.5% win rate on AlpacaEval and 37.3% on Arena-Hard.\nA category-wise analysis indicates that TPO benefits both reasoning and non-reasoning tasks, unlike prior CoT-based methods that primarily enhance mathematical and logical reasoning.\nClaim 2: Thought generation can be optimized without direct human supervision.\n\nThe proposed reward-based optimization strategy allows the model to learn effective thought processes without requiring labeled thought data.\nThrough Direct Preference Optimization (DPO), the system refines thought generation iteratively, producing better responses over time.\nAblation studies confirm that models trained without TPO fail to achieve similar improvements, reinforcing that the optimization of thought generation is a crucial component.\nClaim 3: Thinking helps general instruction-following beyond traditional reasoning tasks.\n\nThe fine-grained analysis (Figure 4) demonstrates improvements in domains such as marketing, translation, and content writing, areas where CoT prompting was previously considered ineffective.\nThis contrasts with prior research (e.g., Sprague et al., 2024), which suggested that CoT primarily benefits logic-based tasks.", "methods_and_evaluation_criteria": "The paper follows a rigorous evaluation framework:\n\nUses AlpacaEval and Arena-Hard, widely accepted benchmarks for instruction-following models.\nCompares against strong baselines, including direct-response models and those trained with generic thought prompting.\nProvides a detailed breakdown of performance across 20 instruction categories, illustrating the types of tasks that benefit most from TPO.\nConducts ablation studies to evaluate the impact of:\nDifferent thought prompting strategies (generic vs. specific).\nEffect of different reward models (ArmoRM vs. STE).\nImpact of iterative training on thought refinement.\nPotential improvements:\n\nHuman evaluation of response quality would provide stronger validation of TPO’s effectiveness.\nAdditional experiments on real-world applications (e.g., dialogue systems, tutoring systems) would further demonstrate the practical benefits of TPO.", "theoretical_claims": "The paper does not focus on formal theoretical contributions but provides strong empirical justification for the proposed approach.\n\nThe method is based on Reinforcement Learning from AI Feedback (RLAIF) and Direct Preference Optimization (DPO), but lacks a formal theoretical analysis of thought generation dynamics.\nA deeper exploration of whether TPO converges to optimal thought processes or if certain types of errors persist over training iterations would strengthen the theoretical foundation.", "experimental_designs_or_analyses": "The experimental setup is well-structured and controlled:\n\nUses standardized prompts, models, and evaluation protocols.\nClearly demonstrates win rate improvements over baselines.\nImplements a length-control mechanism to ensure response quality does not simply improve due to verbosity.\nAreas that could be expanded:\n\nFailure case analysis is limited. The paper does not discuss scenarios where TPO might fail, such as backtracking-heavy tasks or reasoning tasks requiring multiple revisions.\nAutomated evaluation reliance—the paper depends on GPT-4-based judges, which may introduce evaluation biases that are not addressed.", "relation_to_broader_scientific_literature": "Chain of Thought (CoT) (Wei et al., 2022) – Demonstrated the benefits of explicit reasoning but was mostly effective for math and logic tasks.\nReinforcement Learning from AI Feedback (RLAIF) (Bai et al., 2022) – Introduced reward-based optimization for LLMs, but TPO adapts this for thought generation.\nQuiet-STaR (Zelikman et al., 2024) – Explored unsupervised thought generation but was primarily focused on structured reasoning tasks.\nDeepSeek-R1 (Guo et al., 2025) – Used reinforcement learning to train structured reasoning templates, whereas TPO provides a more flexible, data-driven approach.", "essential_references_not_discussed": "Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper presents an original approach to optimizing thought generation in LLMs, extending beyond traditional Chain-of-Thought (CoT) techniques.\nThe proposed Thought Preference Optimization (TPO) method is well-motivated and effectively eliminates the need for manually annotated thought supervision, making it scalable and adaptable across different tasks.\nThe empirical evaluation is comprehensive, covering AlpacaEval, Arena-Hard, and category-wise breakdowns, providing strong evidence that TPO improves instruction-following models in both reasoning and non-reasoning tasks.\nThe paper is well-written and structured, making it easy to follow, with clear explanations of methodology, evaluation, and experimental findings.\nThe ablation studies effectively demonstrate the contribution of different components of TPO, particularly the role of DPO in refining thought generation.\nWeaknesses:\n\nThe paper does not explore failure cases in depth. It would be helpful to provide more insights into when and why TPO fails, particularly in tasks that require iterative backtracking or fine-grained numerical precision.\nThe evaluation relies entirely on GPT-4-based reward models, which introduces potential biases. A complementary human evaluation would help confirm the effectiveness of TPO in real-world scenarios.\nThe work primarily focuses on GPT-based models. It remains unclear how well TPO generalizes to open-source models such as LLaMA, Mistral, or Claude.\nWhile the method is technically sound, there is no theoretical analysis of whether TPO converges to an optimal thought process or if certain errors persist over time.", "other_comments_or_suggestions": "Providing examples of failure cases in the supplementary material would help clarify the limitations of TPO.\nA discussion on the computational efficiency of TPO relative to existing approaches would be useful. Does training with TPO introduce significant additional overhead compared to standard instruction tuning?\nThe paper could benefit from more real-world application demonstrations, such as deploying TPO-trained models in interactive agents or dialogue systems.", "questions_for_authors": "How does TPO handle cases where the initial thought generation is incorrect? Does the model attempt self-correction, or does it remain committed to the initial flawed reasoning path?\n\nWhat are the primary failure modes of TPO? Are there specific task types where the thought-generation process degrades performance instead of improving it?\n\nCan TPO be integrated with retrieval-augmented generation (RAG) systems? Would the model's internal thought process benefit from external knowledge retrieval, and how would that affect optimization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method for training models to develop reasoning abilities without requiring additional chain-of-thought (CoT) data annotation. By evaluating only the final outputs rather than the quality of the CoT itself, the proposed Thought Preference Optimization (TPO) method bypasses the challenges of directly evaluating CoT quality. The experimental results demonstrate that TPO significantly improves model performance on general-domain benchmarks.

All reviewers acknowledged the substantial performance gains achieved by this method and expressed interest in its ability to enhance performance even in non-reasoning domains (e.g., marketing, health, and general knowledge). However, some concerns were raised regarding the evaluation of the actual generated CoT quality and comparisons with alternative approaches. The authors addressed most of these concerns in their rebuttal.

While half of the reviewers strongly supported acceptance (4: Accepted), the other half leaned toward rejection (2: Weakly Rejected). In my assessment, although the paper could benefit from deeper analysis of the generated thought chains, the novelty of unsupervised CoT learning and the demonstrated performance improvements justify acceptance.
Decision: Accept (poster)