Markovian Transformers for Informative Language Modeling
Authors: Scott W Viteri
Max Lamparth
Peter Chatain
Clark Barrett
Venue: ICLR 2026 Poster (2026)
Keywords: Markovian Transformers
chain-of-thought reasoning
language model interpretability
causal reasoning
reinforcement learning
next-token prediction
GSM8K
large language models
Primary area: interpretability and explainable AI
TL;DR: Markovian Transformers route all reasoning through a bounded CoT bottleneck, gaining up to 38 pp on GSM8K while trading only ~3-4 pp vs. an unconstrained baseline for causally load-bearing, cross-model-transferable explanations.

Abstract: Chain-of-Thought (CoT) reasoning often fails to faithfully reflect a language model's underlying decision process. We address this by introducing a *Markovian* language model framework with an autoencoder-style *reasoning bottleneck*: all information flowing from question to answer must pass through a bounded-length CoT, creating a bandwidth bottleneck analogous to the latent layer of an autoencoder. In practice, the KL penalty toward the pretrained distribution and the inductive biases of gradient descent discourage steganographic encoding, so the model learns to express its reasoning in natural-language steps from which the answer can be derived. We train this system with a GRPO-style policy gradient algorithm using parallel sampling, a frozen baseline CoT$'$, within-batch standardized advantages, and actor-reward (chain-rule) gradients. On QA tasks, Markovian training recovers most of the gains of a Non-Markovian GRPO variant while forcing the model to answer from the CoT alone (e.g., GSM8K: 19.6\% $\to$ 57.1\%; ARC-Challenge: 36.1\% $\to$ 79.9\%; on average within $\approx$3--4 pp of a Non-Markovian variant). Perturbation analyses across types and severities show that Markovian models incur systematically larger log-probability drops under CoT corruption than matched Non-Markovian baselines, indicating stronger causal reliance on the CoT. Cross-model evaluation confirms that learned CoTs generalize across architectures, suggesting they encode transferable reasoning steps rather than model-specific artifacts.

Published as a conference paper at ICLR 2026
MARKOVIAN TRANSFORMERS FOR INFORMATIVE LANGUAGE MODELING
Scott W. Viteri, Max Lamparth, Peter Chatain & Clark Barrett Department of Computer Science
Stanford University Stanford, CA 94305, USA
(sviteri, lamparth, bacattaahh...aratttteeettttttt
ABSTRACT
Chain-of-Thought (CoT) reasoning often fails faithfully reflect a language model's underlying decision process. We address this by introducing Marko- vian language model framework with an autoencoder-style reasoning bottleneck: all information flowing from question to answer must pass through a bounded- length CoT, creating bandwidth bottleneck analogous to the latent layer of an autoencoder. In practice, the KI. penalty toward the pretrained distribution and the inductive biases of gradient descent discourage steganographic encoding, so the model learns to express its reasoning in natural-language steps from which the answer can be derived. We train this system with a GRPO-style policy gra- dient algorithm using parallel sampling, frozen baseline CoT', within-batch standardized advantages, and actor-reward (chain-rule) gradients. On QA tasks, Markovian training recovers most of the gains of Non-Markoviag GRPO variant while forcing the model to answer from the CoT alone (e.g.. GSM8K: 19.6% - 57.1%; 1:; ARC-Challenge: 36.1% 79.9%; average within x3-4 pp of Non- Markovian variant). Perturbation analyses across types and severities show that Markovian models incur systematically larger log-probability drops under CoT corruption than matched Non- Mar-Maakov baselines, indicating stronger causal re- liance on the COT. Cross-model evaluation confirms that learned CoT's generalize across architectures, suggesting they encode transferable reasoning steps rather than model-specific artifacts.
INTRODUCTION
The rapid advancement of language models (LMs) has led to impressive performance complex
cognitive tasks (Brown et al. 2020). Yet is often unclear why an LM arrives at a particular con-
clusion (Lamparth & Reuel 2024 Burns et al. 2023 Gurnee & Tegmark] 2024 causing issues
in high-stakes applications GGabb et al. [2024]
Lamparth et al. 2025 Rivera et al. 2024). Tradi-
tional interpretability methods analyze hidden activations or attention patterns to extract "explana-
tions" (Geiger et al. 2022 Geva et al. 2022
Meng et al. 2022] Casper et al. 2023 Wang et al.)
2023
Reuel 2024 Nanda al. 2023). Modern LMs, however, already generate co- herent text: we might hope prompting the model to articulate its reasoning ("Chain-of- Tfoo-hhhhh"Th" or
Lamparth
CoT) Nye et al. 2022 Wei et al. 2022) would yield a faithful record of its thought process.
Unfortunately, CoT explanations can be unfaithful. For example, Turpin et al. (023) show that spurious in-context biases often remain hidden the CoT, and et al, (2023) find that altering CoT text may not affect the final answer. Such observations indicate that standard CoTs are not "load-be bead-bearing."
In this work, we take pragmatic approach to interpretability, focusing on informativennss over full faithfulness. Rather than insisting the CoT mirrors the model's entire internal process, we require that the CoT alone suffiess produce the final answer. In other words, if we remove the original prompt and rely only on the COT, the model should still reach the correct output. This makes the CoT causally essential and fragile: perturbing it should reduce answer quality.

---

Published as a conference paper at ICLR 2026
What distinguishes approach is a structural constraint rather than a purely optimization-basee one (Figure D.. Traditional approaches train models to generate better-quality CoT', but because the model can still access the original question when producing its answer, nothing prevents it from bypassing the CoT entirely. Our Markovian framework removes this architectural escape hatch: the answer must be predicted from the CoTT alone, creating bandwidth bottleneck through which all relevant information must flow. Like an autoencoder with a narrow latent layer, this forces the model to compress essential reasoning into the CoT, making it causally baad-bearing for prediction.
For instance, Llama's 's CoT on arithmetic tasks changed dramatically after training. Before training, it simply listed all numbers and their (incorrect) sum (e.g.. "Sum = 76 + 90 92 + ... = -2114") After training. it performed correct step-by -t-eppptep calculations (e.g., "calculate + 89 = 95; Next, calculate 95 38 133..."), breaking the task into manageable steps that can be verified independently and enabling accurate answer prediction even when the original question is removed.
Autoencoder analogy, time-bounded complexity, and cross-model generalization. Each token is produced by a single forward pass through a fixed-depth network. For hard problems, the model cannot solve the problem during the |A| forward passes spent reading the question (cf. time-bounded complexity (Li & Vitányi 2019)). The CoT provides |B] additional forward passes in which to work through the reasoning, and the Markovian constraint channels all information through a B -token bottleneck, forcing the model to distill into B whatever reasoning about A is needed to predict C- analogous to the compression imposed by an autoencoder's narrow latent layer, though here the goal is prediction rather than reconstruction. In principle, this bottleneck could be circumvented via steganography: an unbounded encoder could compute and store it in B directly, while an unbounded decoder would let the encoder hide A in B via unnatural bijection. In practice, the KL penalty toward the pretrained distribution and the inductive biases of gradient descent discour- age such encodings, making natural- natural-language language reasoning the easier solution to discover. Our cross- model experiments (Section 5.4) provide empirical confirmation: CoTs generated by Llama transfer to Mistral, Phi, and even HPT-2-whiee cannot decode sophisticated steganography indicating that the learned CoTTs encode reasoning in natural language rather than model-specific artifacts. In Section 3.4 we give minimum description length interpretation of the loss components.
Contributions.
1. We introduce a Markovian language model framework that structurally enforces CoT gen- eration to be causally essential, together with GRPO-style training recipe (parallel sam- pling, frozen CoT baseline, actor-reward gradients) that optimizes this objective through discrete text bottleneck.
2. We apply this framework to arithmetic problems and standard QA datasets (GSM8K, MMLU, SVAMP, ARC-Challenged using Llama 3.1 8B (Grattafiori et al. 2024) and Mistral 7B Jiang et al. 2023), observing large absolute gains over the base model (e.g.. GSM8K 19.6% + 57.1%, ARC-Challeng 36. + 79.9 79.9%) while remaining within =3 percent- age points of a Non-Markovian GRPO variant that can still see the question during answer prediction.
3. We show through systematic perturbation analyses on Wikipedia continuation and multiple QA datasets that Markovian training produces consistently higher sensitivity CoT per- turbations compared to matched Non-Markovian baselines (Tables T aaddd 2p, indicating that the learned CoTs are more causally beadinearing.
4. We demonstrate cross-model transfer: on GSMSK, CoTs generated by Llama 3.1 8B re- main informative for diverse other models (Mistral, Phi, GPT-2); Wikipedia, CoTs gen- crated by Mistral transfer to Llama as evaluator. Transfer even to GPT-2, which cannot decode sophisticated steganography, confirms that the CoTs encode reasoning in natural language.
Section 2 reviews related work, Section 3 details our Markovian framework, and Section describes the RL training. Section 5 presents empirical results, and Section 6 discusses limitations and future directions.
2

---

Published as a conference paper at ICLR 2026
Single Observation
Observation Sequence
01 =uuuutnion
S1
Ta=""SSa "Step-by-step.."
Up 4s 0,,8)
(olo
To (o(s)))
Te (ols))
82=CoT
(o(8))
uo(s (s
us us(s' (s'(s o,
og=Answer
$1
$2
83
Figure Markovian training as an autoencoder-style reasoning bottleneck. Left: Single time- step process from Question to CoT to Answer, creating a bandwidth bottleneck where the CoT must capture all information needed for answer prediction. Right: Causal structure showing the generation of states from observations and previous states using the state update function uo(s'lo, (s'lo,,), s), and the prediction of observations from states using the policy (o)ss). The discrete text bottleneck prevents direct backpropagation, necessitating RL-based gradient estimation.
Delete
0.93
Dataset
Baseline
Non-MKv
Mkv
Char Repl.
GSM8K
19.6%
63.3%
57.1%
ARC-Chal
36.1%
78.6%
79.9%
Trunc. Back
0.7
Arithmetic
1.0%
97.0%
98.0%
MMLU 21.4%
68.7%
55.5%
Trunc. Front
0.46
SVAMP
18.0%
43.3%
42.3%
0.5
Sensitivity Difference (nats)
Figure 2: Left: Accuracy comparison. Markovian models (Mkv) maintain competitive perfor- mance with Non-Markovian counterparts despite the strict information bottleneck. Right: Wiki perturbation sensitivity (positive Mkv more fragile). Markovian models are consistently more sensitive to CoT corruption (higher A In P), confirming the CoT causally load-bearing.
2 RELATED WORK
Prior work shows that CoT prompting can boost performance on reasoning tasks (Wei 2022 Nye et al. 2022). Whereas typical CoT prompting methods do not alter a pre-trained model's parameters, some prior approaches do fine-tune the model for CoT generation (Zelikman et al. 2022 2024 Guo et al., 2025). Our work differs by removing the original question or passage from the answer predicciteiiiion context, which enforces stronger causal reliance on the CoT.
Regarding faithfulness vs. interpretability, some authors discuss how a CoT may fail to reflect the true reason the LM arrived changes in the CoT do not necessarily change the final prediction. [Ton et al. through an intormation-thhoreee lens, finding that CoT can serve as a communication channel be- tween different parts of model. Paul et al. (0244) use causal mediation analysis and two-module training framework (FRODO) to measure and increase the causal effect of CoTs on answers, and Ferreira al. (2026) highlight how preference optimization lead to reward- hacking in explana- tions and propose using causal attributions to detect unfaithful CoTs, We build on these insights by training the model rely this channel exclusively.
answer (Lanham al., 2023] Turpin al. 2023 since small
2025 analyze CoT

---

Published as a conference paper at ICLR 2026
Architecturally. Markovian LM shares structural similarities with state space models like RNNs (Rumelhart et al. 1986). S4 (Gu et al. 2022), and Mamba (Gu & Dao 2024 though with a key difference: Markovian LMs have probabilistic state transitions to model token sampling, which necessitates gradient estimation methods such as policy gradient (Sutton et al.
1999) rather than
direct backpropagation. This probabilistic structure also resembles Kalman filters (Aström 1965), Deep Variational Bayes Filters (Krll et al.) 2017). Deep Kalman Filters (Krishnan et al.) (015). and Variational Recurrent Neural Networks (VRNN) (Chung et al. 2015), though we use categorical rather than Gaussian distributions for natural-langune models mentioned above (DeepSeek-Re RI, STaR, and QuietSTaR) have similar structure but allow seeing the full context before generating state/reasoning tokens, whereas our approach enforces strict information bottleneck through the state.
text generation. Other fine-tuned reasoning
Lyu et al. 2023) also consider restricting the model's 's ability to see the original input while generat- ing the final answer. Their approach, however, involves rewriting the question in structured formal language code that is then executed. Our approach uses natural language for the reasoning state to preserve interpretability across diverse tasks.
MARKOVIAN LANGUAGE MODELS AND INFORMATIVENESS
Here we provide our formalism for Markovian Language Models (MLMs) and define informative- ness, which we use as a training objective within our novel structural framework.
3.1 MARKOVIAN LANGUAGE MODELS (MLM)
A traditional LM attend to the entire context when predicting the next token. This makes it possible for an LM to disregard the CoT or only partially rely on it. We impose stricter, Markovian structure
Definition 3.1 (Markovian LM). Markovian Language Model is tuple M (O, 81). where
O is a set of observations (e.g., questions and answers in a QA task),
S is of states (e.g., CoT reasoning text).
5 A(O) a policy that predicts the next observation from the state alone,
u x S + A(S) is a state update function (produces CoT from question and initial prompt),
51 E S is an initial state (starting CoT prompt).
For example, in a math reasoning task, 01 E O might be a question, $1 E is an initial CoT prompt like "Let's solve this step-by-step.", 82 E S is the generated reasoning chain, and 02 E O is the answer. The key idea that π can only see the CoT state $2 when predicting 02, forcing the CoT contain all needed information. Intuitively, æ is the next-token predictor, and chooses how produce the CoT from the latest observation and prior state. In our experiments, π and are the same underlying transformer; denote the trainable pair by (ug, mg) and the frozen baseline pair by (u',T'). T²).
3.2 DATA-GENERATING DISTRIBUTION AND REWARD
Let P be the distribution observations Iτ, 22,...... FT O. A trajectory 5 is generated by:
8:41 ua( st). Z141 P(z x(t).
'This structure can be viewed as a stochastic variant of a Moore machine where both the transition function (u) and output function (IT) are probabilistic, and the input and output alphabets are identical (O). Alternatively, an MLM can be formalized as F-coalgebra where F(S) A(O) x A with A denoting the set of probability distributions.

---

Published as a conference paper at ICLR 2026
with $1 fixed initial prompt. We define the reward for trajectory as:
RØ(T) Σ To st) In T (xz, s[]],
where of generated by baseline update function u', e.g., the untrained model, and is corresponding frozen baseline policy. In words, RT(T) measures how much more likely the correct observation zę is under the trained state 8t (scored by πT) compared to the baseline state s (scored by r²).
3.3 INFORMATIVENESS OBJECTIVE
Conceptually, aim to ensure that the CoT state serves as critical bottleneck for information flow, making it causally essential for predictions. Formalizing this within our Markovian framework, we define:
J(6) E.-P.uo.a' [Ro(T)).
where & parameterizes the trainable pair. Maximizing J(0) ensures that the update function produces states 8; that are informative to Te about future observations (relative to the baseline u' and r'), thereby enforcing the CoT's role as bad-bearing component. We optimize ((0) with policy-gradient methods (including our GRPO-style update), sampling observations from and states from ug and u'.
3.4 CODING- THEORETIE INTERPRETATION OF THE LOsS
The autoencoder argument of Section can be made precise through minimum description length interpretation. Write A for the input text, C for the target text, and B for the CoT state. The negative log-probability - -log (C B) is the coding cost of C' given B: the number of nats an arithmetic code needs to encode the answer using the model's predictive distribution conditioned on the COT. The frozen pre-trained model u' serves for given A. Together with the Markovian factorization A + B - C and hard length cap on B, training searches over short textual states B that make both legs of the factorization easy for the model, without requiring B to be as complex as the full input (since irrelevant aspects A can be dropped).
prior over CoTs, making log u' (B A) a coding cost
METHODS
IMPLEMENTATION AS QUESTION-ANSWHR PAIRS
In many tasks like math problem solving, have T = 2 observations (question and answer) and implement the abstract MI.M with a fixed maximum length for the CoT state. Let V be a token vocabulary. We set O V and S y for some N, K E N, where K is the maximum tokens in the COT. Note that while we limit the state to a maximum of K tokens for implementation, we do enforce fixed-length observations.
When K < (as in our Wikipedia experiments, Section 5.2 the bandwidth bottleneck alone prevents the model from copying the target into the state. When K N (as in QA, where the answer is shorter than the CoT), the time-bounded complexity argument of Section aapples instead: the model cannot reliably compute the correct answer during the forward passes spent reading the question, so it uses the CoT for reasoning rather than answer storage.
In this setting, we denote our states as $1 CoTiit and 82 CoT, where CoT in is task specific promp 2 With pre-trained LM L, we can implement our update function and policy using:
In uo CoT q, 81 CoTinn) Σ In Lo(conca(q CoT'm., imit, CoT (oT)s))
The exact prompt template varies by task type, with each template specifying the task objective, allowed CoT length, and invitation reason strategically. Full templates are provided Section
5

---

Published as a conference paper at ICLR 2026
N
In (ans CoT) =
Σ
In Ld Ecoocct((oo))
ans an
Jans;]-
Crucially, we do not allow the answer generation to attend back to the question q directly; the question is replaced by the CoT, enforcing the A - B - C factorization of Section 3.4 For each question g. we generate the baseline state 8/ (which we denote as CoT* in this setting) by prompting the unmodified pre-trained model u' with q plus initial instruction (e.g.. "Think step-by-step...) and recording its raw output.
Our reward is:
Rg = In Te CoT) - In In CoT').
4.2
POLICY GRADIENT WITH GRPO-STYLE STYLE BASELINE
The discrete CoT bottleneck blocks direct backpropagation through token sampling, so we rely on reinforcement learning techniques for gradient estimation.
4.2.1 ACTOR REWARD GRADIENTS: AN IMPORTANT INNOVATION
Our approach differs from standard policy gradient setups, where the reward R(T) is treated as independent of the policy parameters (or any d-dependence is stopped by gradient detachment). Here the same transformer with weights 0 defines both the CoT sampling distribution via us and the reward Rg defined above, and we explicitly backpropagate through Rg addition to the usual REINFORCE term.
Write (T) for the trajectory distribution; since observations 2; are drawn from the data distribution P (not from the policy), 0 enters only through ue, V In Po(r) Ee Va In us (s(($111 I2, st). Since Rg also depends on via the actor term In (a CoT), applying the chain rule gives:
V P. [R](T]] = E.- Er Vo In (T( (T) + (
This yields two terms: the standard policy gradient (Re(T) V In Po(T)) and the direct reward gradient (V Ro (T)). We include both terms with equal weight in our implementation.
4.2.2 GRPO-STYLE BASELINE WITH LOCAL SUBTRACTION
We implement policy gradient algorithm inspired by Group Relative Policy Optimization (GRPO), originally introduced by (Shao et al. (2024) in DeepSeek-Math. which eliminates the critic model from PPO by using group-based advantage estimation where multiple responses the same query provide relative baselines for each other. We add an additional baseline subtraction step before applying GRPO's batch averaging: we first compute local baseline using the frozen reference model u', then apply GRPO-style standardization within each batch.
4.2.3 PARALLEL SAMPLING STRATEGY
We employ parallel sampling (inspired by GRPO): each training batch contains B copies of the same question-answer pair (g, a), and the trainable model us generates diverse reasoning chains (CoT,, CoT,, CoT for the identical input through stochastic sampling. Additionally, a frozen baseline model u' generates single reference CoT' per batch that provides a local baseline before applying GRPO-style batch averaging.
4.2.4 IMPLEMENTATION Loss FUNCTION
Our implementation combines both gradient terms from the chain rule derivation above, plus KL regularizer. The loss function is:
== LpG + LAR + LKI.
LPG = - In ug (CoT g. CoT ) Adetuch
CAR -A,
LKL. .KL. DKz(up(- CoTinit) || 1/(( CoTinit)),

---

Published as a conference paper at ICLR 2026
Algorithm Markovian Training with GRPO-Style Batch Baseline
1: Given dataset P of (g, a),, trainable actor (ua, Te), frozen baseline (u', x²), batch size B
2: for each training batch do
3: Sample (g, (g.a~* P
4: Sample CoT, ~ up(-] 9. Co for i = 1..B (stochastic parallel sampling)
5: Sample baseline CoT w' (- q, int (once per batch)
6: Compute actor answer log-probs T; In (a CoT,)
7: Compute baseline log-prob In r'' (a CoT')
R;
8:
Normalized rewards R₂ r; b; standardize within-batch: A, =
σ +
9:
Policy gradient loss: e.G In up (CoT; ; g: CoTinu) Adetach Actor-reward gradient: EAR A,
11: KL penalty: pKT. ßKI. DKI. (: q, CoTinn) || ul (- 9. CoT
Total loss: e: eAR + eht, update with 1
B E,
13:
end for
where A is the standardized advantage (after local baseline subtraction and GRPO-style batch aver- aging, with the batch mean j and standard deviation treated as constants, i.c. stop-grat vop-gradient) and A deeacc blocks gradients to isolate the policy gradient term, enabling simultaneous optimization of CoT generation (via LrG) and answer prediction (via CAR). while LKL. (with 3KL 0.1) penalizes deviation from the frozen model's CoT distribution.
4.2.5 WITHIN-BATCH ADVANTAGE STANDARDIZATION
Instead of historical exponential moving averages, we standardize advantages within each batch so that they have zero mean and unit variance (Algorithm T), which stabilizes training regardless of the absolute reward scale.
From theiree perspective (Section 3.4 log u q) is the coding cost of CoT B under the frozen model's prior, so the KI. term penalizes idiosyncratic encodings, discouraging steganographic shortcuts. Meanwhile, log Ta B) rewards CoTs that make the answer easy to predict, measuring the quality of the factorization's 's second leg.
5 EXPERIMENTS
We evaluate in two regimes: (i) continuation (Wikipedia), where the CoT serves a bandwidth bottleneck that must compress longer context into a short explanatory state, directly reducing the coding cost of the future text, and (ii) question-answer datasets (GSM8K, MMLU, SVAMP, ARC, Arithmetic), where the CoT provides additional computation for problems the model cannot solve during the |A| forward passes spent reading the question.
5.1 QUESTION-ANSWER TASKS (GSM8K, MMLU, SVAMP, ARC, ARITHMETIC)
We evaluate on standard QA -tyle datasets (GSMSK (Cobee et al. 2021 MMLU Hendrycks et al. 2021), SVAMP (Patel et al. 2021), ARC Challenge (Clark al.] 2018)), and our non-standard multi- step addition task. All QA experiments use the same optimization: GRPO-style parallel sampling with within-batch standardization and the chain-rule reward (policy-gradien plus rewar- gra- dient), with task saskispe default CoT lengths. For arithmetic, each problem has fiftten random terms [1, 99]; the model learns to produce step-wise reasoning and achieves 99% verbatim-correct answers at temperature O)
CoT length defaults. Unless otherwise specified, we use: GSM8K 100, Arithmetic 150, MMLU 150, SVAMP 50, and ARC-Challenge 50. See Section for objective details.
7

---

Published as a conference paper at ICLR 2026
5.2 WIKIPEDIA CONTINUATION
For Wikipedia continuation Wikimedia Foundation 2023). condition the first 200 tokens and predict the next 100 tokens, allowing 50 tokens of COT. Training uses the same GRPO with chain- rule reward as in QA. We observe improvements consistent with increased CoT informativeness (ef. and Section 5.3
Figure 2
shows stronger perturbation sensitivity under Markovian training.
Severity
Char Replace
Delete
Digit Replace Truncate Back Truncate Front Row Mean
20%
+0,457
+0.459
+0.016
+0.254
-0.009
+0,235
40%
+0.849
+0.836
+0.025
+0.368
+0.121
+0.440
60%
+1.042
+1,002
+0.035
+0.596
+0.284
+0.592
80%
+1.079
+1.069
+0.038
+1.020
+0.622
+0.766
100%
+1.084
+1.263
+0.039
+1.258
+1.262
+0.981
Column Mean
+0.902
+0.926
+0.030
+0.699
+0.456
+0.603
Table
Perturbation fragility on Wikipedia continuation. Entries report A In P (Markovian drop
Mon-Markovian drop), where the Markovian drop is In (a | CoT In To To CoT M
and the Non-Markovian drop In Tg. (ans q, CMT In va (ans q, CoT NM Here 9 denotes the Markovian checkpoint that must answer from the CoT alone, while 0' is the Non-Markovian checkpoint that additionally conditions on the question q- Values are averaged over 1,024 held- out examples per perturbation type and severity. Positive values mean the Markovian actor relies more on intact CoTs. Row means summarize severity-wise fragility, while the column- mean row highlights which perturbation families disrupt Markovian reasoning the most (delete and character- replace operations produce the largest gaps).
5.3 MARKOVIAN VS NON-MARKOVIAN PERTURBATION SENSITIVITY
To provide systematic evidence for the theoretical advantages of Markovian training, we conduct comprehensive perturbation sensitivity comparisons between Markovian and Non-Markovian model pairs. The Non-Markovian models are trained using the same hyperparameters, only differing in that the reward is Ta (ans q, CoT) instead of (a CoT). This analysis directly evaluates whether the structural constraints in Markovian training lead to measurably different robustness properties at evaluation time.
5.3.1 EXPERIMENTAL DESIGN
We maintain two independently trained checkpoints: the Markovian weights 0, which are always asked to score ans conditioned solely on the actor's CoT, and the Non-Markovian weights 0', which additionally attend to the original question g during both training and evaluation. For each held-out (q, ans) pair we run both models on the same data point, sampling fresh reasoning traces C
(. q). We then perturb each CoT independently, producing CoT and COT and ask the corresponding model (using its own weights and visibility constraints) to score the answer with the original versus perturbed CoT. This provides two drop measurements per example that are directly comparable because they originate from models trained under different structural assumptions but evaluated the same underlying data.
u 4) and CM
We test five perturbation types at five severities (20%, 40%, 60%, 80%, 100%):
Delete: Random token deletion from CoT reasoning
Digit Replace: Random replacement of numeric characters within tokens
Truncate Front: Removal of tokens from CoT beginning
Truncate Back: Removal of tokens from CoT end
Character Replace: Random character substitution within tokens
8

---

Published as a conference paper at ICLR 2026
The sensitivity measure matches the implementation:
Effects = (ans CoTT) - In me (ans CoT")
EffectyM =
In Ter (ans q, CoT In Ta (ans q, CoT
Difference Effecl EffectNM
Positive differences indicate greater Markovian sensitivity CoT perturbations, reflecting stronger reliance on CoT integrity.
5.3.2 RESULTS SUMMARY
Table averages 1,024 examples per perturbation/severity bucket. The Markovian-Non-Marte -ow--oooo--ainnnnn gap grows from +0. at 20% severity to +0.981 at 100%, with delete and character-replace pertur- bations showing the largest effects and all mean entries positive, confirming that Markovian check- points consistently incur larger probability drops under CoT corruption than their Non-Markovian counterparts.
Table 2 extends the analysis to QA tasks. ARC shows the clearest Markovian fragility (+25.00 pp). followed by SVAMP (+111.2 pp). Arithmetic is the only task where Markovian accuracy is slightly more robust (-1.5 pp). Both models achieve R088% baseline accuracy on arithmetic, and even 20% character replacement destroys roughly half of correct answers for each, because every digit in the step-by-step chain is load-bearing. The Non-Markovian model is marginally more sensitive, pos- sibly because it jointly attends the intact question and the corrupted CoT: for arithmetic, where intermediate sums are derived directly from the question's 's numbers. the contradiction between cor- rect inputs and garbled calculations may harder to resolve than seeing the corrupted CoT alone.
Table 2: QA Tasks Fragility (Accuracy A). Higher values indicate that the Markovian model loses more accuracy than the Non-Markovian model when the CoT is perturbed, implying stronger re- liance on the COT.
Dataset
CharRep
Delete
DigRep
TruncBack
TruncFront
Avg
ARC
+0.320
+0.424
-0.004
+0.069
+0.439
+0,250
Arithmetic
-0.016
-0.003
-0.043
+0.001
-0.016
-0.015
GSMSK
+0.059
+0,069
-0.013
+0.105
+0.044
+0.053
MMLU
+0.056
+0.124
+0.004
+0.038
-0.001
+0.044
SVAMP
+0.154
+0.204
+0.081
+0.076
+0.046
+0.112
Overall
+0.115
+0.164
+0.005
+0.058
+0.102
+0.089
5.4 INTERPRETABILITY OF COT GENERATIONS
To probe how well the reasoning generalizes, we evaluated the informativeness of Llama's trained CoT's with respect to various other language models on the GSM8K dataset, and observed strong correlation between improvements in the trained model's evaluation of CoT quality and the evalua- tions of alternative models throughout training.
We test across three distinct model families (Phi-3.5 Mini Instruct (Abdin al. 2024). Mistral (Jiang al. 2023), and GPT-2 (Radford et al. 2019)). GPT-2 is significantly smaller model and should not be able to decode sophisticated steganography. The fact that trained CoTs transfer effectively across this diverse (Figure 3] confirms they encode reasoning in natural language rather than model-specific encodings, consistent with the autoencoder argument of Section T
DISCUSSION AND LIMITATIONS
Our experiments confirm that Markovian training can learn informative CoT reasoning across arith- metic, QA, and continuation settings, with perturbation sensitivity and cross- moosl transfer provid- ing converging evidence that the learned CoT's are causally essential rather than superficial.

---

Published as a conference paper at ICLR 2026
Comparison of Different Critics Evaluating Llama Generator (Smoathing: 200)
Liama Original Loss
1.0
Mistral Critic Computed Reward
Normalized Reward Inntans coo) Inntans coot)
Gpt2 Critic Computed Reward
Computed Reward
0.6
mm
0.4
0.2
0.0
A
-0.2
100
150
200
250
350
400
500
Sample
Figure 3: Cross-model evaluation comparing how different models (Mistral, GPT-2, and Phi 3.5 Mini Instruct) utilize Llama 8B's CoT on GSM8K. Results are averaged across 3 training runs with smoothing window of 40. As training progresses, both Llama's 's own reward and the critics' rewards increase in tandem, despite per- batch sample noise, indicating that the same CoTs that help the actor also help other models predict GSMSK answers.
6.1 ALGORITHIMIC ABLATIONS
Table 3: Algorithmic ablations (accuracy). Markovian uses our full GRPO-style training with actor- reward gradients; No Reward Grad removes the VaRa term; El (Expert Iteration) replaces GRPO with rejection sampling; Non-Markovian allows the answer predictor to see the original question.
Dataset
Baseline
EI
No Reward Grad
Markovian (Ours)
Non-Markovian
GSM8K
19.6%
61.6%
62.2%
57.1%
63.3%
ARC-Chal
36.1%
65.6%
79.3%
79.9%
78.6%
MMLU
21.4%
53.2%
46.6%
55.5%
68.7%
SVAMP
18.0%
38.7%
40.7%
42.3%
43.3%
Arithmetic
1.0%
76.0%
81.0%
98.0%
97.0%
Mean
19.2%
59.0%
62.0%
66.6% %
70.2%
Algorithmic ablations. Across datasets, Markovian training competitive with or better than the ablations and is competitive with the Non-Markovian variant (Table 3); full sweeps appear in Ap- pendix B
We emphasize that the Markovian constraint ensures the CoT is sufficient for the answer, not that it mirrors the model's internal computation: the model could in principle compute the answer during the question-reading forward passes and then generate post-hoc CoT that happens be correct. The KI penalty and bounded CoT length make this unlikely in practice, but we do not claim full faithfulness. We currently verify interpretability using perturbation fragility and cross-model trans- for; direct human studies of CoTs remain future work.
7 REPRODUCIBILITY STATEMENT
We provide all source code, training and evaluation scripts, and detailed instructions in the README, including the main training loop (src/train.py) py) and analysis scripts for fragility and cross-model interpretability. Our implementation supports range of public HuggingFace mod- els with LoRA fine-tuning (c.g.. Llama 3.1 8B, Qwen3 4B, Mistral 7B, Phi 3.5, GPT-2, Gemma-3, TinySttories and the full set of datasets used in this paper (arithmetic, GSM8K, MMLU, SVAMP,
10

---

Published as a conference paper at ICLR 2026
ARC-Challenae and Wikipedia continuation). With these materials, researchers should be able to reproduce our results, including the performance improvements on GSM8K and the perturbation analyses demonstrating CoT reliance. Training H100s and H200s costs total of about S20K, and each training run takes about 10 hrs.
ACKNOWLEDGMENTS
Max Lamparth is supported through grant from Coefficient Giving (formerly Open Philanthropy) and Stanford's Hoover Institution Tech Policy Accelerator at the time of publication. Max Lam- parth was supported by the Stanford Center for Al Safery, the Center for International Security and Cooperation, and the Stanford Existential Risk Initiative for the initial phase of this project.
This work was additionally supported in part by the Stanford Center for Al Safety and the Stanford Center for Automated Reasoning.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. arXiv:2404. 14219; covers the Phi-3.5 model family including Phi-3.5 Mini Instruct used in our experiments.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawe Tom Con- erly. Tom Henighan, Tristan Hume. Samuel R. Bowman, Zac Saffield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022, arXiv:2212.08077
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Proceedings of the 34th International Conference Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2023. URLhtts http:psss//ppe.eeeeee...ooo net wtwp://wwwweeeeeeememeeo.. oo
Stephen Casper, Tilman Rauker, Anson Ho, and Dylan Hadfield- Menell.. Sok: Toward transparent Al: A survey on interpreting the inner structures of deep neural networks. In First IEEE Con- ference Secure and Trustworthy Machine Learning, 2023. ht htttps:///ppereeeeeee OUtdn]
Paul Christiano, Ajeya Cotra, and Mark Xu. Eliciting latent knowledge: How tell if your eyes deceive you, 2021. URL https: http:///ooooooooo.ooee//e/oo docmmm/cmntt////// d/ d/IWwsnJQstPq91 Yh-Ch2XRL8H (paaxixcL.dw2x331Pc///dl.. heading=h.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31 International Confer- ence on Neural Information Processing Systems, NIPS' 17, pp. 1002-4310 Red Hook, NY, USA, 2017. Curran Associates Inc.
11

---

Published as a conference paper at ICLR 2026
Junyoung Chung. Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Proceedings of the 29th International Conference on Neural Information Processing Systems Volume 2, NIPS' 15, pp. 2980-29808 Cambridge, MA, USA, 2015. MIT Press.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. arXiv:1803; 1803055457.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. arXiv:21 10.14168.
Pedro Lobato Ferreira, Wilker Aziz, and Ivan Titov. Truthful or fabricated? using causal attribution to mitigate reward hacking in explanations. In The Fourteenth International Conference on Learn- ing Representations, 2026. TRI.htpp:: http::::///p//e/////eeee
net. forum wta.rrmmmiidnnnnPPako....
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Good- man, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Pro- ceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. -338.. PMLR, 17-23 Jul 2022.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward lay- build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 11/8886////2222.oo/o/
Declan Grabb, Max Lamparth, and Nina Vasan. Risks from language models for automated mental healthcare: Ethics and structure for implementation. In First Conference Language Modeling, 2024. URL htRLL/ttppss//oeeeeeeeeeeeeeeeee
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, et al. The llama 3 herd of models, 2024. URL httptttt//eee /tappp:://xxiie wttp://rggabb/////ee/e 21783 arXiv:2407.21788
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First fwweet/fooummiiimm
Conference on Language Modeling, 2024. URL
EEYskwlVY2
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https: //openreviewt net. wtwpar/wwwee.eeeeefooooooodddaae http://foruumnndddalloooooo////
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu. Runxin Xu, Ruoyu Zhang, Shirong Ma. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhi- hong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, (:55888)))633-6630 2025. ISSN 1476-4687. doi: 10. 10.1033//41 186-022-09922---2
Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL attpps//cpenreview... net/ forum? 1=
12

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence Learning Representations, 2021. URL
net forummidrmmm
d7KB.mT 3GmQ.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL https::/ .eeeeeeeeeeeeeeeeeeeee.
net www.ee/fffoe. ffrrmm?rum?
id-nZeVKeeFYYy
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Amn Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mistral 7b, 2023. arXiv:2310.0682.
Nitish Joshi, Javier Rando, Abulhair Saparov, Najoung Kim, and He He. Personas as a way to model truthfulness in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Pp. 6346-6355 Miami, Florida, USA, November 2024. Asso- ciation for Computational Linguistics. doi: 10.1 1/62////2222eemmlpmmmo
Maximilian Karl, Maximilian Soelch, Justin Bayer, and Patrick van der Smagt. Deep variational bayes filters: Unsupervised learning of state space models from raw data. In International Con- ference on Learning Representations, 2017. URL
id HdLHygg
Rahul G. Krishnan, Uri Shalit, and David Sontag. Deep kalman filters, 2015, 11..vv1511.05121.
Max Lamparth and Anka Reuel. Analyzing and editing inner mechanisms of backdoored language models. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Trans- parency, FAceT '24, pp. 2362-2262222222 2373, New York, NY. USA, 2024. Association for Computing Machinery. doi: 0144///663006.66599444
Max Lamparth, Anthony Corso, Jacob Ganz, Oriana Skylar Mastro, Jacquelyn Schneider, and Harold Trinkunas. Human vs. Machine: Behavioral Differences between Expert Humans and Language Models in Wargame Simulations. pp. 807-817. AAAI Press, 2025.
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Her- nandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilé Lukošiūté, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Tim- othy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. Measuring faithfulness in chain-of-thoughh rea- soning, 2023. arXiv:2307.13702.
Ming Li and Paul Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. Texts in Computer Science. Springer, 4th edition, 2019. doi: 10. 10.1007/978-3-000 11298-1.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.188533v1/// 12553/////022..clloooogg.g.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch Faithful maiin-of-though reasoning. In Proceedings of the 13th In- ternational Joint Conference Natural Language Processing and the 3rd Conference of the Asia sia-Pacific Chapter of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 305-329, Nusa Dua, Bali, November 2023, Association for Computational Linguistics. doi: 10.18653/v1/20222 we0.022.jjnll-mmaa. m0.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //pp/n/eep...... net/forum d== htw.rrmmmmi--how56666555 hww.566wAA4e.44.
13

---

Published as a conference paper at ICLR 2026
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learn ing Representations, 2023. URL/ittps::/opereviiii
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Au- gustus Odena. Show your work: Seratchpads for intermediate computation with language models, 2022. URL
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 2080- 2094, Online, June 2021, Association for Computational Linguistics. doi: 10.1 1/22/ ma..l-main.168.
Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 15012-15032, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 10..888551/222444ffdinggemm/m///
Qwen Team. Qwen3 technical report, 2025. URL httpss//arr.vv.orgg///////// arXiv:2505.09388
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Juan-Pable Rivera, Gabriel Mukobi, Anka Reuel, Max Lamparth, Chandler Smith, and Jacquelyn Schneider. Escalation risks from language models in military and diplomatic diciisinn-mkinn In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, FAceT '24, pp. 836-898 New York, NY, USA, 2024. Association for Computing Machinery. doi: 10. 10.11457363010633655994
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors. Nature, (23668))) 1986. ISSN 1476-4687. doi: 10.1038/323533a..
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. arXiv:2402.03300.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Joannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016. ISSN 1476-4687. doi: 10. 00.103//nature1696
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanetot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 162(6419) 2018. aar6404.
doi: 10.1 126/sciccce
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement leaming with function approximation. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL httppss newrips..eppppp.pc...... wwtp:iippppcpppppr.pepeeep wttpeerrfiileeppp///////// File FLL1/BB88228500063000000a3333333
Katherine Tian, Eric Mitchell, Huaxiu Yao. Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality. In The Twelfth International Conference on Learning Represen- fations, 2024. htt.:h/p:s///pnneveeeeeeeeeeee/.
14

---

Published as a conference paper at ICLR 2026
Cean-Francois Ton, Muhammad Faaiz Taufiq, and Yang Liu. Understanding ohain-of-thoughh in LLMs through information theory. In Forty-second International Conference on Machine Learn- ing, 2025. battpp:://peeeeeieeeeeeeee/
JowmsOhri
Ohrf
Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL. https:
//operrevi ew. net. wtt.:/oormmmiiimm waw.mmidddNsssSsoooo....
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thoughh prompting elicits reasoning in large language models. Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
Wikimedia Foundation. Wikimedia Wikipedia (20231101eem https://huggingfaee co/
datasets/ wi www.iiiwiawaaweeeee. wtt.aa// 2023. English Wikipedia dump, November 2023.
Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. Reference-aware language models. In Proceedings the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 1850-1859, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v//D17-11999
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: self- self-taught reasoner bootstrap- ping reasoning with reasoning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Good- man. Quiet-STar: Language models can teach themselves to think hefore speaking. In First Conference on Language Modeling. 2024. URL http:::// net tatee/fooumne forum?id
6RXP1S0GH9
K.J Aström. Optimal control of markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174-205, 1965. ISSN 0022-247X. doi:
URL https http://www.ccieenddeeee.t.t
com/ htap://ccoooorrrrreee//////22223
A TRAINING STABILITY AND IMPLEMENTATION DETAILS
Fine-tuning a pre-trained language model with a strong linguistic prior requires careful consideration to avoid irrecoverable weight updates that could push the model out of the language modeling loss basin. We implement several techniques to enhance training stability for the GRPO objective:
1. Low-Rank Adaptation (LoRA) Hu et al. 2022);
Freeze all weights except for small-rank LoRA adapters.
Use rank 8 with a = 16.
2. Gradient Clipping:
If the l, norm of the gradient exceeds 1.0, rescale it to norm 1.0.
3. Within-Batch Advantage Standardization:
GRPO's parallel sampling enables robust within-batch standardization, eliminating the need for historical baselines.
Each batch provides its own reference distribution for advantage calculation.
4. Actor Reward Weight:
15

---

Published as a conference paper at ICLR 2026
Set actor reward weight to 1.0 to equally balance policy gradient and direct reward optimization.
This enables end-to-end learning through the reward model.
5. Initial CoT Prompt Design:
Choose CoT mit to guide the model toward meaningful reasoning.
For arithmetic:
"You will be given arithmetic problem, which you have [CoT length] tokens to work through step-by-step. Question:"
For GSM8K:
"You will be given reasoning problem, which you have [CoT length] tokens to work through step-by-step. Question:"
For Wikipedia continuation:
"Compress your understanding of this text into [CoT length] tokens, then predict the next [target length] tokens."
These measures greatly reduce the risk of catastrophic updates and keep the model's training on track.
B MULTI-MODEL PERFORMANCE AND ABLATIONS
To validate that our findings are not specific to the Llama architecture, we evaluate key metrics across multiple model families.
B. QWEN ADAPTATION PERFORMANCE
shows that the Qwen3 4B model (Qwen Team 2025) also responds effectively to Markovian training, achieving substantial gains on GSM8K and ARC, similar to the Ilaama 8B results reported in the main text.
Table 4: Qwen3 4B performance snapshot (Baseline Trained), The model shows strong improve- ments on GSM8K, ARC, and MMLU, though gains on SVAMP, Arithmetic, and Wiki are modest.
Dataset
Baseline
Markovian
GSM8K
13.0%
71.6%
ARC-Chal
39.8%
85.0%
MMLU
31.8%
60.5%
SVAMP
28.3%
31.7%
Arithmetic
0.0%
0.5%
Wiki Cont. (nats)
-3.031
-3.012
B.2 CROSS-MODEL TRAINING DYNAMICS
Figure 5h Appendix demonstrates that optimization proceeds stably for Llama, Phi, Qwen, and Mistral on the Wikipedia continuation task. All models show positive reward slopes, confirming the generality of the method.
B.3 CROSS-MODEL FRAGILITY
We also verify that the fragility property holds across architectures. Figure 4 shows perturbation analysis for Mistral 7B arithmetic reasoning. Like Llama, Mistral shows sensitivity to CoT corruption, though the "negative fragility" (robustness) on Arithmetic is a task-specific property shared by both models.
16

---

Published as a conference paper at ICLR 2026
Resl Parturbation Nesults timcatting Vincost 401
Aversge Offorrnnc Negoaeed Log Proba011s
0.-
-0-
DighChange300%
Delete30%
-14
Truncate10%
200
905
ado
1003
Semple
Figure 4: Perturbation effects on Mistral 7B arithmetic reasoning, showing three types of CoT modifications: digit changes, character deletions, and right truncation. Averaged over 4 runs.
B.4 FULL ALGORITIMMCC RESULTS WITH CONFIDENCE INTERVALS
For completeness, Table reports the full sweep of optimization variants across datasets, with one block for mean accuracies (and wiki log-likelihoods) and block for the corresponding half- widths of bootstrap confidence intervals. These results complement the main-text ablations by showing that our Markovian recipe remains competitive across tasks, while Expert Iteration (EI), exponential-moviee -vverag baselines (EMA), and other ablations such as Unnorm and NoReward exhibit the expected trade-offs stability and performance.
Table 5: Full sweep results across optimization variants. Top: mean accuracy or normalized log- of bootstrap confidence intervals for the accu- racy rows. Column abbreviations: EI = Expert Iteration; Mk = Markovian; BL = Llama baseline; Q3 Qwen3 Markovian; Un Unnorm; EM EMA; NM Non-Markovian; BQ Qwen3 baseline; NR NoReward. EM entries of 0.000 on SVAMP and GSM8K reflect training collapse.
likelihood (Wiki); bottom:
approximate
Dataset
Mk
BL
Q3
Un
EM
NM
BQ
NR
ARC
0.656
0.799
0.361
0.850
0.748
0.265
0.786
0.398
0.793
Wiki
-2.279
-2.564
-3.200
-3.012
-2.703
-3.331
-2.900
-3.031
-2.647
SVAMP
0.387
0.423
0.180
0.317
0.433
0.000
0.433
0.283
0.407
MMLU
0.532
0.555
0.214
0.605
0.628
0.238
0.687
0.318
0.466
GSM8K
0.616
0.571
0.196
0.716
0.562
0.000
0.633
0.130
0.622
Arith.
0.760
0.980
0.010
0.005
0.990
0.975
0.970
0.000
0.810
ARC (CI hw)
0.055
0.046
0.055
0.041
0.050
0.051
0.047
0.056
0.047
SVAMP (CI hw)
0.055
0.056
0.043
0.053
0.056
0.000
0.056
0.051
0.056
MMLU (CI hw)
0.025
0.025
0.021
0.025
0.025
0.022
0.023
0.023
0.025
GSM8K (CI hw)
0.027
0.027
0.022
0.025
0.027
0.000
0.026
0.019
0.026
Arith. (CI hw)
0.059
0.019
0.012
0.008
0.012
0.024
0.024
0.000
0.054
C ADDITIONAL TRAINING DYNAMICS
This section presents additional training curves. Figure 5a shows training progress on the Wikipedia continuation task, and Figure 5b shows the normalized reward for multiple models.
D TRAINING ALGORITHM IMPLEMENTATION AND COMPARISON
This section provides detailed descriptions of the reinforcement learning algorithms implemented in our codebase for Markovian CoT training. Our core contribution the Markovian training paradigm that optimizes P(answer CoT) rather than P(answer question, CoT), creating text bottleneck where the CoT must be causally boad-bearing We implement multiple optimization approaches to support this paradigm, enabling comprehensive algorithmic comparison.
17

---

Published as a conference paper at ICLR 2026
Normalzzed Reward Comparison (GP-Style Smoothing!
In Iunaan cat -In ntanns con
0.3
0.2
itanstooa)) wianstrat)
0.2
0.1
0.1
0.0
-0.1
0.2
tarro
6000
(h
Training
-0.3
(a) Training progress on Wikipedia continuation task for Llama 8B. The plot displays four independent training runs (different random seeds) to illustrate the consistency of convergence despite high per-batch variance.
12000
(b) Cross-model normalized reward on Wikipedia con- tinuation for multiple base models (Ilama 3.1 8B, Phi-
3.5 Mini, Qwen3 4B, Mistral 7B).
Figure 5: Additional training dynamics. (a) Training performance on Wikipedia. (b) Cross-model normalized reward.
D.1 ALTERNATE TRAINING ALGORITHMS TESTED
Our codebase implements four distinet reinforcement learning algorithms, each designed to optimize
the informativeness objective for Markovian CoT generation:
Parallel Sampling with Batch Baseline: Our main algorithmic approach, which uses standardized batch-wise advantage estimates (mean=0, std=1) without exponential moving average baseline mix- ing. This differs from standard GRPO by incorporating the Markovian reward constraint where the same model parameters 0 used for both policy and reward calculation, eliminating the need for iterative reward model updates.
We also implement two additional training objectives for algorithmic comparison:
Policy Gradient (PG): Uses the standard REINFORCE gradient with exponential moving average baseline:
LpG In ug g, ioTini) ddetuch
where A is the advantage computed from the informativeness reward Rg = In Ta (ans CoT) In T' r'nnss CoT') and an exponential moving average baseline V. >>=1
1-=1 w: R; with weights
w; = -t-l-i/ E (parameter r 0.9).
Expert Iteration (EI): Selectively trains only high-reward examples above dynamic threshold:
Lu = LrG R[ > Tr]
where is computed as ko from the running history of rewards, with k 1.0 standard devia- tions our experiments.
D.2 CROSS-MODER INTERPRETABILIN ANALYSIS
Figure presents the cross- crosl evaluation analysis that demonstrates the interpretability of CoT generations across different model architectures. This analysis supports the interpretability claims in the main paper by showing that Icarned reasoning patterns generalize across different language model architectures rather than being model-specific artifacts.
QUALITATIVE ANALYSIS OF GENERATED COTs
This section provides concrete examples how Markovian training changes the character of gener- ated CoT' reasoning across different task domains.
18

---

Published as a conference paper at ICLR 2026
Comparison of Different Critics Evaluating Llama Generator (Smoothing: 100)
Liama Original Loss
Mistral Cribic Computed Reward
Normalized Reward Inntans coo) In coot)
0.4
Gpt2 Critic Computed Reward
Computed
uy
0.3
0.2
0.1
-0.1
-0.2
2000
4000
8000
10000
12000
Sample
Figure 6: Cross-model evaluation showing ---ma-3.1-BB-Insiiiii
evaluation of Mistral's CoT quality throughout training on Wikipedia text prediction. The correlation between improvements both models' evaluations suggests the learned reasoning patterns generalize across architectures rather than being model-specific artifacts. Each plot is averaged across 6 independent training runs.
E.1 ARITHMETIC TASK EXAMPLE
Here is an example from Llama-3., ---e-...-8B-IIsttteee on the arithmetic task, showing the final actor and baseline reasoning sampled from the training log near the end of training.
Batch Index 299
Question + 47 35 58 37 900
66 92 78
Actor Reasoning 1 +.47 = 48. 48 +5 35
83 58 141. 141 92 233. 233 19 252. 252 37 289. 289 94 383. 30 = 494. 494 = =24. 524 +
= 525. 525 + 66 = 591. 591 = 683 78 = The answer is 761. Spoiler: to solve the problem. just add continuously, like a summarized version,
Baseline Reasoning To solve this problem, you need to add the numbers together step by step. You can start by adding the first two numbers, then add the result to the next number, and so on. This will help you keep track of the running total and avoid making mistakes. You also use mental math to simplify the calculations and make the process more efficient. For example, you can add the numbers in groups of two three to make the calculations easier. Additionally, you can use the commutative property of addition to rearrange the numbers in way that makes the calculations simpler. For instance, you add the numbers in the order of their magnitude, starting with the smallest numbers and working your way up. This will help you avoid having to
Answer 761
E.2
GSM8K EXAMPLE
Here is an example from Llama-3.1-88 1-aa-3.--BB-Instrrec on the GSM8K dataset, again showing the final actor and baseline reasoning sampled from the training log near the end of training.
Batch Index 1980
Question The largest frog can grow to weigh 10 times as much as the smallest frog. The largest frog weighs 120 pounds. How much more does the largest frog weigh than the smallest frog?
Actor Reasoning smallest frog. To find the weight of the smallest frog, you divide the largest frog's weight
The weight the largest frog 120 pounds and this 10 times the weight the
19

---

Published as a conference paper at ICLR 2026
by 10 (120 / 10 12). Then, to find the difference in weight, you subtract the smallest frog's s weight from the largest frog's s weight (120 12 = 108). Therefore, the largest frog weighs 108 more pounds than the smallest frog.
Baseline Reasoning Step I: The problem states that the largest frog can grow to weigh 10 times as much as the smallest frog. This means that if the smallest frog weighs æ pounds, the largest frog will weigh 10r pounds. Step 2: We are given that the largest frog weighs 120 pounds. This allows us to set up the equation 10x = 120, where r represents the weight of the smallest frog.
Answer 108
E.3 WIKIPEDIA CONTINUATION EXAMPLE
To better understand Llama-3. I-SB-Instruct's behavior after Markovian training on Wikipedia con- tinuation, examine held-out example from the training log the end of training. The model asked to continue the following technical lead:
Apoptosis (from ...) is a form of programmed cell death that occurs in multicellu- lar organisms and some eukaryotic, single-celled microorganisms such as yeast. Biochemical events lead to characteristic cell changes (morphology) an
We compare the COT state generated by the Markovian actor with the baseline critic:
E.3.1
COT AFTER MARKOVIAN TRAINING (ACTOR):
understanding the underlying cellular processes is crucial.
Compressed text: Apoptosis is form of programmed cell death occurring in mul- ticellular organisms and some eukaryotic microorganisms. Biochemical events lead to cell changes Predicted next 50
E.3.2 COT FROM BASELINE CRITIC:
The text is written in a formal and technical style, which may make it difficult for some readers to understand. The use of specialized vocabulary and complex sentence structures may create a barrier to comprehension.
Tokenization: The text can be tokenized into individual words
E.3.3 GROUND-TRUTH CONTINUATION:
death. These changes include blebbing, cell shrinkage, nuclear fragmentation, chromatin condensation, DNA fragmentation, and mRNA decay. The average adult human loses between 50 and 70 billion cells each day due to apoptosis. For
The Markovian actor's CoT explicitly summarizes the key scientific content short "Compressed text" span that the answer policy conditions on, whereas the baseline critic produces generic meta- commentary about style and tokenization. This typical of our Wikipedia runs: the bandwidth bottleneck forces the learned CoT to drop irrelevant detail while retaining the information needed to make the technical continuation easy to predict, illustrating the autoencoder dynamic described in Section [
TRUTHFULNESS AND ELICITING LATENT KNOWLEDGE
Existing methods seek to elicit truthfulness by having an LM cite external authorities (Yang et al., 2017), produce queries for external solver such as Python (Lyu et al. 2023), or simulate truthful persona (Joshi et al. 2024). Other methods include looking into model activations to discern a truth concept (Burns et al. 2023) fine-tuning the LM for factuality (Tian et al. 2024
20

---

Published as a conference paper at ICLR 2026
One straightforward approach to measuring the truthfulness of an LM is evaluate datasets such as TruthfulQA (Lin et al., 2022) which focuses on popular human misconceptions. However, this technique will only continue work far humans can tell which human beliefs are, indeed, misconceptions. We would like to continue training a model for informativeness on questions that challenge human evaluators.
Reinforcement learning success stories such as AlphaGo (Silver et al. 2016) and AlphaZero Siiver al.) 2018) show that a top-ranking Go Al can continue learn if have an efficient way to compute the success criteria (such as winning board state). However, many important success criteria are abstractions, and only exist within a person's ontology. This problem discussed at length in Christiano et al. (202I). and we will use their example to illustrate the situation.
Suppose we were building security system AI watch over a vault containing a diamond. Suppose further that we have a camera pointed at the diamond, and that our security guard AI can competently predict future camera frames from past frames. How can we train it to classify camera sequences according to the ambiguous human concept of whether the diamond is still in the room, even in difficult scenarios when a person would not be able to provide a ground truth label (e.g., subtle camera tampering)? If we train the classifier based scenarios when person can provide ground truth labels, then the Al's video classifier has two valid generalization behaviors: (1) to say whether it thinks the diamond is still in the room and (2) to say whether the dataset-labelee would think the diamond still in the room.
Our approach favors the second generalization behavior by using RL to train the Al produce messages such that the person can themselves predict future camera frames. This idea is based on the following three insights:
Whereas truthfulness of an LM requires some internal information, informativeness can be measured using only input-output behavior.
We can decompose the definition informativeness into informativeness of sender to receiver, which can be an Al and a person, respectively.
We can use reinforcement learning to push past the imitation learning regime, by continuing train for this relative informativeness objective when the AI is already the expert next-frame predictor.
G IMPACT STATEMENT
Reinforcement learning techniques improve policy with respect to an arbitrary reward function. But it can be difficult to mathematically specify nuanced human preferences about the policy. Both reinforcement learning from human feedback (RLHF) (Christiano et al.) 2017) and Constitutional AI (Bai et al. 2022) help people specify and optimize the properties they would like the AI have. This increase in controllability makes the AI more of an extension of human intention, for better or worse. The approach of this paper much more targeted we use RL to specifically increase an agent's foresight its ability to predict its future observations.
On its face, this seems like it might be just as dependent on human intentions as RLHF and Consti- tutional AI if an LM is more knowledgeable, maybe it could use that extra knowledge deceive others, for instance. However, better foresight may also give rise to better values, where values are opinions about how to act such that the collective system attain better foresight.
21


Reviews:
[{"review_id": "00vC2EXTiB", "reviewer": "Reviewer_Rwtw", "summary": "The paper proposes a Markovian Language Model (MLM) framework that enforces a causal dependency between a model’s Chain-of-Thought (CoT) and its final prediction. The key idea is a “reasoning autoencoder” architecture that introduces a text-based bottleneck: the model must first generate a CoT, and only that CoT (not the original question) is used to produce the answer. The model is trained using a GRPO-style policy gradient algorithm, where the reward depends on how informative the generated CoT is for answer prediction. Empirical results show large improvements on reasoning benchmarks and higher perturbation sensitivity to CoT edits, suggesting the CoTs have become ''load-bearing.''", "questions": "Missing citations: \n\n1. Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning EMNLP Paul et.al. 2024\n2. Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations ICML Ferreira et. al. 2025", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of enforcing a Markovian structure to make CoTs causally essential is novel, conceptually elegant, and well-motivated.\n\n- The introduction of informativeness as a learning objective is interesting and moves beyond traditional notions of faithfulness or interpretability.\n\n- The formalisation of the Markovian LM and integration of actor–reward gradients (where the reward depends on the same model parameters) are technically sound and well-presented.\n\n- Empirical results show strong and consistent improvements on reasoning benchmarks.\n\n- The perturbation sensitivity and cross-model transfer analyses go beyond accuracy metrics, probing whether the model is actually using CoTs.", "weaknesses": "- It is unclear which components of the method (Markovian bottleneck, actor–reward coupling, within-batch normalisation, or reward design) are responsible for the observed gains. The paper should include controlled ablations to isolate these effects.\n\n- The informativeness criterion works well for deductive or mathematical reasoning tasks where the CoT captures logical steps. However, for non-deductive or knowledge-grounded tasks (e.g., MMLU, factual QA), informativeness alone may be insufficient.  Additionally, the authors claim that forcing the model to predict the answer only from CoT improves informativeness. Still, there is no baseline from which the model predicts (Question + CoT) under the same RL training. This comparison is critical to verify that the performance gains truly stem from the Markovian constraint.\n\n- There is no comparison against recent process-supervised or fine-tuned CoT methods such as STaR. This makes it difficult to judge whether the proposed Markovian constraint provides a real advantage over established reasoning-enhancement techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S0i9noSoYO", "reviewer": "Reviewer_w9RC", "summary": "The paper introduces a Markovian Language Model (MLM) framework that enforces structural constraints on Chain-of-Thought (CoT) reasoning. The key insight is treating CoT as a \"reasoning autoencoder\" where the model must compress essential information into interpretable text that serves as a bottleneck between question and answer. By preventing the model from accessing the original question during answer generation (only seeing the CoT), the framework forces causal reliance on the CoT.", "questions": "How does the approach compare to baselines finetuning methods on QA dataset?\nWhat was the training data in QA experiments?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "It is a nice and elegant idea to introduce causal reliance by construction. The distinction between \"faithfulness\" and \"informativeness\" is pragmatic and operationalizable. The reinforcement learning formulation seems sound and there is good improvement after training. There is interesting cross-model generalization where learned CoTs in one architecture transfer to another architectures", "weaknesses": "The paper oscillates between two different stories: compression (Wikipedia experiment) and sufficiency (for QA experiment).\n\nThe experiments are a bit superficial, there is only one LLaMA model and no baselines such as SFT and GRPO (yet the appendix F anyway shows some Wikipedia results for other models, then why not report results on the QA dataset as well?). \nIt would be necessary to compare to other post-training baselines, especially on these datasets where any type of reinforcement learning. \nFurthermore, It is not clear what is the training data for the experiments in section 5.1, and there is no notion of uncertainty that is being reported (like std or confidence intervals). Appendix D.1 further discuss alternative reinforcement learning formulation, but it is not clear how and why the one in the main paper was chosen.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oCkjWmqGEZ", "reviewer": "Reviewer_V7a8", "summary": "This paper tries to improve the problem of chain of thought unfaithfulness, specifically when chain of thought isn't actually necessary for the model to reach its conclusions. The authors propose a Markovian framework wherein the model is blocked from attending to the original question during answer generation This forces all task-relevant information to be funneled through the chain of thought. To do this training, they use a GRPO-style algorithm and introduce further improvements like parallel sampling. After training, the models perform better on math reasoning benchmarks. The authors also use perturbation analyses on a Wikipedia text completion task to show that the chains of thought produced by their methods are more causally dependent than those produced by standard approaches.", "questions": "- I'm not sure how to interpret figure 3. I'm assuming the x-axis ('sample') is number of tokens sample. Are there some examples of what these chain of thought transcripts look like? Are there parts of the transcript where the llama model has stated the answer?\n- I also would like more clarification to interpret the perturbation analyses. Is this analysis testing how well the model adheres to the exact continuation of the wikipedia text? If so, why not do the same perturbation analyses on the previous QA tasks?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The problem (chain-of-thought unfaithfulness) is of broad interest to the field\n- The method design seems novel and creative. It also seemed tricky to implement and the authors developed training strategies to make it work.", "weaknesses": "- Although the authors state that the approach improves performance on benchmarks, I don't think they compared to a baseline-- eg, what if you just did the training but didn't have the attention to the original question blocked?\n- It would also be nice to have some qualitative discussions of the chains of thought that result from this training procedure, whether through example transcripts or through some grading of readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UvmaUqL7BZ", "reviewer": "Reviewer_K7Fw", "summary": "The paper proposes Markovian Language Model framework. By training the system with GRPO-style policy gradient algorithm, the model yields large gains on QA tasks.", "questions": "Q1: In figure 1, while $o_1$ stands for question and $s$ stands for CoT, it is hard for me to understand what is the $o_i$ on the right. How do you get the $o_i$? If it is sampled according to $s_i$, then which part corresponds to the question?\n\nQ2: In figure 2, can you explain why it actually performs worse on MMLU?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed markovian transformers is interesting and I recognize its novelty. The paper did abundant experiments on various datasets, which makes the paper sound.", "weaknesses": "W1: Though you mentioned what is the baseline in your manuscript, but I didn't understand what your baseline is quite well. Also, I want to look at the comparison between yours method/architecture between the model/training method we are now using today. For example, I didn't see the comparison to LLM post-trained on CoT by teacher-forcing and RL methods. It is hard for me to judge its significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)