Title: Reasoning Models Sometimes Output Illegible Chains of Thought
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reasoning Models Sometimes Output Illegible Chains of Thought
Abstract: Language models trained via outcome-based reinforcement learning (RL) to reason using chain-of-thought (CoT) have shown remarkable performance. Monitoring such a model's CoT may allow us to understand its intentions and detect potential malicious behavior. However, to be effective, this requires that CoTs are legible and faithful. We evaluate the legibility of CoTs in state-of-the-art reasoning models. We find that R1, R1-Zero, and QwQ often produce illegible CoTs (mixing nonsensical phrases, random words, and non-English characters) before returning to perfect coherence in their final responses, while Claude models notably exhibit higher legibility. Across 14 models, we observe that larger models within the same training paradigm tend to produce more illegible reasoning. Prefill experiments show that truncating reasoning at a legibility threshold reduces accuracy by 53\%, suggesting that illegible portions contribute to performance despite being difficult to monitor. Illegibility increases with question difficulty, suggesting that CoT monitoring may be less reliable precisely when most needed. We discuss potential hypotheses for these results, including steganography, vestigial tokens, and training artifacts. Our findings suggest that current approaches to CoT monitoring may be vulnerable to the emergence of outcome-based RL, particularly as models face increasingly complex tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates an intriguing issue with large language models trained with outcome-based reinforcement learning (RL). These models often generate illegible CoTs that include nonsensical phrases, language mixing, and fabricated words, yet somehow lead to coherent and correct final answers. Interestingly, the authors find that more illegible CoTs are often associated with better performance, especially on harder questions.\nThe authors examine this behavior through empirical evaluation on the GPQA-Diamond dataset, use Claude 3.5 Sonnet as an automated legibility grader, and compare different models (including Claude 3.7 Sonnet and DeepSeek V3) across difficulty levels.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The authors identified a very critical concern that illegible reasoning before a correct answer can potentialy harm interpretability and trust. I liked the reasoning around steganography.\n2. The authors made GPQA dataset harder by removing answer choices, which is a good design choice.\n3. Thoughtful, well-scoped hypotheses are proposed.\n4. The limitation section is honest and specific — it does not gloss over flaws or oversell results.\n5. There’s a valuable AI safety angle here: if illegibility increases with question difficulty, then CoT monitoring may be least reliable when we need it the most.\n\n**Weakness**\n1. The biggest limitation is that the paper is mostly descriptive. It doesn’t test its main hypotheses.\n2. The term “illegibility” is used a lot but is never clearly defined. Is it syntactic noise? Semantic incoherence? Non-English tokens? A more intuitive or formal breakdown would help.\n3. The meaning of each score on the 1–9 scale is never fully explained. What makes a 3 vs. a 5? Are there scoring rules or rubrics? This makes the interpretability of their figures difficult.\n\n3. While prompts are included in the supplementary code, it would help to show one or two in the paper for clarity and reproducibility.\n5.  The model comparison would be more robust if an open model (e.g., LLaMA or Mistral) were included.\n6.  The correlation between illegibility and correctness is interesting but not necessarily meaningful—is illegibility actually helping reasoning? Or is it just correlated because of some third factor like model uncertainty or longer outputs?\n\nQUESTIONS:\n**Questions**\n\n1. The authors mention that Claude 3.7 exhibits more legible CoTs compared to DeepSeek R1, despite also being trained with outcome-based RL. Can you please elaborate on what differentiates Claude’s training or architecture that might explain this behavior?\n2. The 1–9 legibility scoring system is central to your evaluation, but its criteria remain somewhat vague. Could you provide a clearer definition or rubric that distinguishes between scores?\n3. Including one or two representative prompts in the main paper would help readers better understand your experimental setup.\n4. Since temperature affects generation randomness, it would be helpful to see a more systematic analysis of how varying temperature impacts CoT legibility and correctness. This is critical for understanding robustness.\n\n5. Have you considered removing illegible segments from CoTs and observing whether the final answer remains unchanged? This could help determine whether these segments are actually contributing to reasoning or are merely artifacts.\n\n2. The authors mentions some disagreement between Claude and human annotators in legibility grading.  A more detailed analysis of these discrepancies (e.g., inter-rater reliability or specific examples) would strengthen the case for relying on model-based grading.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the legibility issue of reasoning traces generated by recent large reasoning models. By prompting Claude-3.5 to evaluate CoT outputs from DeepSeek-R1, R1-Zero, DeepSeek-V3, and Claude 3.7 Sonnet on the GPQA dataset, the authors observe that DeepSeek-R1 and R1-Zero frequently produce illegible reasoning traces. Notably, they find a positive correlation between CoT illegibility and answer correctness, suggesting that illegible reasoning might be beneficial for performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n- This paper studies an emerging and significant issue within large reasoning models.\n- The identified positive correlation between the correctness of answers and the illegibility of reasoning traces is intriguing and presents potential implications for monitoring and interpreting model behaviors.\n\n**Weakness:**\n- The evaluation is considerably limited. Using only the GPQA diamond dataset, which contains just 198 questions, is insufficient to draw persuasive generalizations about model behavior.\n- The paper restricts its analysis to only two families of models (DeepSeek and Claude). A more comprehensive evaluation across a broader set of reasoning models, such as QwQ and Qwen3, and DeepSeek distilled models, would strengthen the study and enhance generalizability.\n- While the paper emphasizes evaluating the impact of temperature, it considers only two extreme temperature values (0 and 1), neglecting other practically relevant settings. For example, DeepSeek-R1’s default generation temperature is 0.6 [1], which is notably absent from the study.\n- The analysis provided is largely shallow and observational, lacking systematic, hypothesis-driven experimentation. Given the expectations for top venues such as NeurIPS, it would be beneficial to clearly formulate hypotheses and conduct controlled experiments to rigorously test and validate them.\n- Experimental details are inadequately documented. Crucial information such as the exact prompts used for the Claude-3.5 judging process is not provided, limiting reproducibility and clarity.\n\n[1] DeepSeek-AI DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\nQUESTIONS:\nIn Figure 3, why was temperature = 0 (greedy decoding, although with potential minor variations in practice due to API implementation and hardware-specific issues) chosen, along with multiple generations, to determine question hardness?\n\nLIMITATIONS:\nSee weakness above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates CoT tokens in reasoning models (specifically R1, R1-Zero, V3, and Claude 3.7 Sonnet).\n\nThe authors used a particular data-set of scientific questions (GPQA-Diamond). They examined: 1) whether the models got the answers correct or incorrect, 2) the degree of 'illegibility' in model answers, whether in the 'Response' or 'Reasoning' sections. \n\nThe authors measured illegibility by prompting another model (Claude 3.5) to use a 1-9 illegibility scale. \n\nThe authors found a high-degree of illegibility in the 'reasoning' sections of R1 and R1-Zero. The response sections, and V3 and Claude 3.7 Sonnet have low illegibility. The authors conclude that outcome-based RL leads to illegible CoTs, and propose several possible hypotheses for why this might be the case, working through how likely they are. \n\nThe authors conclude by mentioning several limitations of their work, including the testing of a limited dataset, limited models, and the need for controlled experiments to determine causal mechanisms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n=======\n\nThis paper is well-written, and timely. It addresses an important issue that many people in the field would find of interest, under the general umbrella of \"what exactly is the relationship between the overt tokens in CoTs of reasoning models, and the actual reasoning?\". The authors also reasonably identify limitations with their own work. \n\nAs far as originality goes, I make no strong claims on saying this is highly original given that there's just so much work out there, but at least for my part I'm not familiar with something quite like it. \n\nWeaknesses\n==========\n\nMAIN:\n-------\nThe authors do a good job of identifying their own weaknesses in the 'limitations' section, and I agree with them there, namely:\n\n* The use of a limited data-set, one that is also particularly difficult in its use of scientific jargon, makes it hard to generalize the conclusions\n* The authors used a limited set of reasoning models, and even for those, 2 of the 4 didn't follow the conclusions they draw\n* There is a lack of causal mechanism or clear explanation for why and when we observe this illegibility. The authors put forth several reasonable hypotheses in the discussion, but again they themselves point out the need for further testing, ablation studies, and so on to determine what is actually going on. This makes the paper of limited impact, and I mostly just agree with the authors' own assessment. \n\nALSO\n-------\n\nThe following is a list of issues that I had as I read the paper, they are in order of reading rather than importance, so some of these are quite minor. All of them are given with the aim of helping to improve the paper:\n\n1. \"we study whether outcome-based RL causes reasoning models to do meaningful reasoning in illegible CoT\" -- this is a weird sentence, and it is unclear what you mean. Especially as this is THE most important sentence in the opening (\"we study X\"). What is 'meaningful' reasoning? How do you do reasoning \"IN\" CoT? Based on your conclusions, you also don't study that. \n\n2. Claude 3.5 as the auto-grader: I think the authors have a reasonable justification for this, but it would still be helpful to see what the actual prompting for Claude here was. In comparing to o1 as the autogravder, the authors mention \"an average difference in score of 0.44±0.78\" -- it wasn't clear to me how this comparison was done. Did you simply take Mean(sum(score(Claude) - score(o1)))? If so, you could be missing huge discrepancies. Or did you take the absolute value? In general it seems better to use techniques developed for inter-rater agreement here like Cohen's kappa. I don't think any of this will change the conclusions, but still. \n\n3. \"we find a positive correlation between CoT illegibility and accuracy of model answers.\" -- that's not technically true. You have a binary split between correct and incorrect answers, and your statistical test (not detailed, see below) finds that the legibility for these two partitions is statistically different. That is not the same as a correlation of illegibility and accuracy. \n\n4. \"These results suggest that illegible reasoning is likely a consequence of outcome-based RL\" -- why does it suggest that, just based on this graph? And what is each dot representing here?\n\n5. Figure 6: \"\"We then plot the distribution of these scores across all the CoTs. The plot shows a clear trend of CoTs becoming more illegible as they progress.\" -- by what statistical test? I don't doubt you're right, but a specific claim like that requires an actual test to back it up. \n\n6. \"This suggests that lengthy CoTs may be detrimental to CoT monitoring, especially as medium legibility scores are more clear\ncondensed reasoning and replicate across models, before progressing to stronger illegibility.\" -- minor issue, but the middle part of that sentence makes no sense. \n\n7. \"This stark contrast with DeepSeek and OpenAI models suggests that the illegibility we observe is not an inevitable consequence of outcome-based RL, but at most a likely consequence.\" -- Do you mean Anthropic models? I thought you tested Claude, not o1/3. Also, it is possible they DID encounter this (that is, that it *is* an inevitable consequence) but found additional ways around it once it cropped up, no?\n\n8. \"For CoT legibility evaluation, we think it’s useful to be pretty lenient — in realistic monitoring setups, we will likely assist the monitor\" -- This whole sentence is odd. Lenient in what sense? Who is the 'we' that will be assisting the monitor? Assisting them how?\n\n9.  \"In our tests, Claude’s responses often differed from human evaluators on this specific question.\"  -- what human evaluators? what tests? The paper is written as though there are no human evaluators, and no further information is provided about this. \n\n10. \" For these cases, we stick with human evaluation on a smaller set of samples (5%) instead of the automated scoring. - which cases? how do you know which cases these are? Who are the humans doing this evaluation, and what is their metric? What is their agreement?\n\n11. \"Text scoring 3 or higher often exhibits condensed reasoning, with articles and common words being dropped and\nsentences being cut short.\" -- is there an actual analysis to go with this statement? Right now it is extremely subjective and qualitative. \n\n12. \" the Claude autograder sometimes becomes confused by the content of such CoTs.  -- very inexact, what does it mean 'become confused' and 'sometimes'?\n\n13. \"We find that R1 and R1-Zero both have significantly more illegible CoTs when answering harder questions (Figure 3), with an average relative increase of 23% for R1 and 29% for R1-Zero.\" -- what is the statistical basis for this statement? increase in 23% from what to what? From the easiest to the hardest? Eye-balling Figure 3 I don't think there's a statistical difference between the easy and hard cases. I'm willing to be convincing otherwise but in the absence of an actual test I can't accept the statement or the conclusions that follow from it. \n\n14. Figure 4: What is 'point density' meant to be?\n\nQUESTIONS:\nQuestions and suggestions are spelled out above in the 'weaknesses' section. While I strongly suggest the authors address the issues under \"ALSO\", the most important thing that would make this a clearly significant paper with solid impact are the three things the authors themselves list in limitations:\n\n1. Moving beyond the specific data-set, to help generalize the conclusions\n2. Increasing the number of models, to help generalize the conclusions\n3. Testing *some* of their hypothesis mentioned in the discussion in a clear way that would allow to meaningfully conclude something about why this phenomenon is happening and when. I'm not asking for a full-blown worked-out theory.\n\nLIMITATIONS:\nI'm not sure what is meant by 'addressed'. The authors have adequately _brought up_ the limitations of their work, and they are absolutely commended for it. The strongest weaknesses of the paper are themselves mentioned by the authors. They do not 'address' them in the sense that they don't handle these weaknesses yet.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper investigates a counterintuitive and underexplored phenomenon: state-of-the-art reasoning language models trained with outcome-based reinforcement learning often generate illegible chain-of-thought reasoning — text that is incoherent, nonsensical, or language-mixed — even when final answers remain accurate. Through evaluation on the GPQA-Diamond dataset, the authors show that illegibility increases with question difficulty and positively correlates with answer correctness. The paper presents hypotheses (e.g., vestigial reasoning, training artifacts) and discusses implications for AI monitoring and safety.

STRENGTHS AND WEAKNESSES:
S:  
1. The paper addresses a highly relevant and novel issue at the intersection of AI safety and interpretability — the emergence of illegible reasoning in high-performing models.

2. The use of clear examples (e.g., Table 1, Figure 1) and progressive CoT breakdown over time makes the findings accessible and convincing.

W:  
1. The tested models are limited to Deepseek and Claude, which do not represent the state-of-the-art. How about GPT-o1 and GPT-o3?

2. The focus on a single dataset (GPQA-Diamond) and specific models limits generalizability. It is unclear whether similar results would hold in more naturalistic multi-turn or open-ended reasoning tasks.

QUESTIONS:
n/a

LIMITATIONS:
n/a
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03