The Geometry of Reasoning: Flowing Logics in Representation Space
Authors: Yufa Zhou
Yixiao Wang
Xunjian Yin
Shuyan Zhou
Anru Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Reasoning
Theory
Interpretability
Representation Learning
Geometry
Formal Logic
LLMs
Primary area: interpretability and explainable AI

Abstract: We study how large language models (LLMs) "think" through their representation space. 
We propose a novel geometric framework that models an LLM's reasoning as flows---embedding trajectories evolving where logic goes. 
We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logic beyond surface form. 
This perspective connects reasoning with geometric quantities such as position, velocity, and curvature, enabling formal analysis in representation and concept spaces. 
Our theory establishes: (1) LLM reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. 
Using learned representation proxies, we design controlled experiments to visualize and quantify reasoning flows, providing empirical validation of our theoretical framework. 
Our findings indicate that training solely via next-token prediction can lead LLMs to internalize logical invariants as higher-order geometry in representation space, challenging the "stochastic parrot" argument.
Experiments across Qwen and LLaMA model families further suggest the presence of a general, possibly universal, representational law underlying machine understanding and human linguistic regularities, largely independent of specific training recipes or model architectures. 
Our work serves as both a conceptual foundation and practical tools for studying reasoning phenomena, offering a new lens for interpretability and formal analysis of LLMs' behavior.

Published as a conference paper at ICLR 2026
THE GEOMETRY OF REASONING: FLOWING LOGICS IN REPRESENTATION SPACE
Yufa Zhou", Yixiao Wang", Xunjian Yin", Shuyan Zhou, Anru R. Zhang
Duke University
(yufa zhou, z wang, xunjian, sonnjiin.iin.. yin, shuyan. zhou, anru zhangg
ABSTRACT
We study how large language models (LLMs) "think" through their representation space. We propose novel geometric framework that models an LLM's reasoning as flows-embedding trajectories evolving where logic goes. We disentangle logical structure from semantics by employing the same natural deduction propositions with varied semantic carriers, allowing us to test whether LLMs internalize logie beyond surface form. This perspective connects reasoning with geometric quantities such as position, velocity, and curvature, enabling formal analysis in representation and concept spaces. Our theory establishes: (1) LLM reasoning corresponds to smooth flows in representation space, and (2) logical statements act as local controllers of these flows' velocities. Using learned representation proxies, we design controlled experiments to visualize and quantify reasoning flows, providing empirical validation of our theoretical framework. Our findings indicate that training solely via next-token prediction can lead LLMs to internalize logical invariants as higher-order geometry in representation space, challenging the "stochastic parrot" argument. Experiments across Qwen and LLaMA model families further suggest the presence of a general, possibly universal, representational law underlying machine understanding and human linguistic regularities, largely independent of specific training recipes or model architectures. Our work serves both conceptual foundation and practical tools for studying reasoning phenomena, offering a new lens for interpretability and formal analysis of LLMs' behavior.
Dataset: https: http:///hggggggggggeeeeeeeeeeeee http://g.a..cc.tttttttttttt/t//e /wt.ddattssttseeee tatatasseet//aaterhhuuu Reasoning-Flow
"Reasoning is nothing but reckoning."
Thomas Hobbes
INTRODUCTION
The geometry of concept space. i.c., the idea that meaning can be represented as positions in structured geometric space, has long served unifying perspective across AI, cognitive science, and linguistic philosophy (Gardenfors, 2004; Rickard, 2006; Gardenfors, 2014). Early work in this tradition was limited by the absence of precise and scalable semantic representations. With the rise of large language models (LLMs) (Hurst et al., 2024; OpenAl, 2025; Grattafiori al., 2024; Guo et al., 2025: Yang et al., 2025), we revisit this geometric lens: pretrained embeddings now offer high-dimensional vector representations of words, sentences, and concepts (Mikolov et al., 2013a;b; Arora et al., 2018; Neelakantan et al., 2022; Zhang et al., 2025a; Lee et al., 2025; Kozlowski et al., 2025), enabling geometric analysis of semantic and cognitive phenomena at scale.
seminal recent work (Modell et al., 2025) formalizes the notion that learned representations in LLMs lie on low-dimensional concept manifolds. Building this view, we hypothesize that
"Equal contribution.

---

Published as a conference paper at ICLR 2026
V
Curves(C)
Chrves R.)
0
Fe
Dz
Cree
Loop
(a) Reasoning flows visualized us- (b) Reasoning flows visualized ing PCA in 3 dimensions.
(c) Schematic illustration of map- pings between spaces.
ing PCA in dimensions.
selected problem from MATH500 with six distinct answers, (c) Our geometric framework of mapping relationships among input space X, concept space C, logic space L, and representation space R. See Section 4 for more details.
Figure 1:
Reasoning Flow. (a-b) Visualizations
reasoning unfolds as trajectory, potentially flow, along such manifolds. To explore this idea, we draw on classical tools from differential geometry (Menger, 1928; Hicks, 1965; Guggenheimer, 2012: Do Carmo, 2016) and propose a novel geometric framework for analyzing reasoning dynamics in language models. Concretely, we view reasoning as a context-cumee cantext-cummlativee trajectory embedding space: at each step, the reasoning prefix is extended, and the model's representation is recorded to trace the evolving flow (Figures Ia and 1b). Our results suggest that LLM reasoning is not merely random walk graphs (Wang al., 2024a; Minegishi al., 2025). At the isolated embedding level, trajectories exhibit stochasticity reminiscent of graph-based views; however, when viewed cumulatively, a structured flow emerges on low-dimensional concept manifold, where local velocities are governed by logical operations. To the best of our knowledge, this is the first work to formalize and empirically validate such dynamical perspective, offering quantitative evidence together with broad insights and implications. We further rigorously define and formalize concept, logic, and representation spaces (Figure lc), and relate them through carefully designed experiments. See the full theoretical definitions of flow, velocity, curvature, and the associated spaces in Section 4.
From Aristotle's 's syllogistics to Frege's 's predicate calculus and modern math foundations (Bochenski Thomas, 1961; Copi et al., 2016; Enderton, 2001), formal logic isolates validity as form independent of content. Wittgenstein's Tractatus sharpened this view-"tth world is the totality of facts, not of things" (Wittgenstein, 1922) unddrrccoii logical form as the substrate of language and reality. In this spirit, we treat logic as carrier-invariant skeleton of reasoning and test whether LLMs, trained on massive corpora, have internalized such structural invariants on the embedding manifold, effectively rediscovering in data the universal logic that took humans two millennia to formalize. We deliberately construct dataset that isolates formal logic from its semantic carriers (e.g.. topics and languages) to validate our geometric perspective.
Our experiments, conducted with Qwen3 (Yang et al.. 2025) hidden states across model families on our newly constructed dataset, reveal that LLMS exhibit structured logical behavior. In the original (O-order) representation space, semantic properties dominate, with sentences on the same topic clustering together. However, when we analyze differences (1- and 2-order representations), logical structure emerges the dominant factor. Specifically, we find that velocity similarity and Menger curvature similarity remain highly consistent between flows sharing the same logical skeleton, even across unrelated topics and languages. In contrast, flows with different logical structures exhibit lower similarity, even when they share the same semantic carrier. Our further random shuffle experiment shows that permuting the order of logical statements collapses velocity and curvature yet preserving positional similarity, indicating that logical structure is encoded in higher-order geometry rather than in raw representation space. These findings provide quantifiable evidence supporting our hypothesis that logic governs the velocity of reasoning flows, challenging the "stochastic parrot" view (Bender et al., 2021) that LLMs lack genuine understanding. Our results suggest a general, possibly universal, constraint on language representations. aligning with the Platonic Representation Hypothesis (Huh et al., 2024; Jha et al., 2025), which posits that neural networks can learn shared underlying world representations largely independent of architecture and training data.
2

---

Published as a conference paper at ICLR 2026
While interpretability research on LLMs has made substantial empirical progress (Anthropic, 2021; Rai et al., 2024; Nanda et al., 2023; Singh et al., 2024; Madsen et al., 2024; Ferrando et al., 2024), rigorous theoretical understanding remains comparatively limited, with only few recent efforts in this direction (Jiang et al., 2023; Park et al., 2024b; Modell et al., 2025; Park et al., 2025). Our work contributes to this emerging line by introducing mathematically grounded framework with formal definitions and analytic tools for quantifying and analyzing how LLMs behave and reason. We hope our theory and empirical evidence open a new perspective for interpretability community and spark practical applications. Our contributions are:
We introduce geometric perspective that models LLM reasoning as flows, providing formal definitions and analytic tools study reasoning dynamics.
We design a formal logic dataset that disentangles logical structure from semantic surface, enabling direct tests of whether L.LMs internalize logic beyond semantics.
We empirically validate our framework through experiments and analysis, demonstrating its utility and offering practical insights.
2 RELATED WORK
Concept Space Geometry. The Linear Representation Hypothesis (LRH) proposes that concepts align with linear directions in embedding space, a view supported by theoretical analyses and empirically validated in categorical, hierarchical, and truth-false settings (Park et al., 2024b; Jiang al., 2024; Park et al., 2025; Jiang et al., 2023; Marks & Tegmark, 2024). However, strict linearity is limited: features may be multi-dimensional or manifold-like as seen in concepts like colors, years, dates, and antonym pairs. (Engels et al., 2025; Modell et al., 2025; Kozlowski et al., 2025). Other works emphasize compositionality, showing that concepts require explicit constraints or algebraic subspace operations to compose meaningfully (Stein et al., 2024; Wang et al., 2023). At a broader scale, hidden-state geometry follows expansion expansion-contractoo contraction patterns across layers and exhibits training trajectories whose sharp shifts coincide with emergent capabilities and grokking (Valeriani et al., 2023; Park al., 2024a; Liu al., 2022). Sparse autoencoders further reveal multi-scale structure, from analogy-like "crystals" to anisotropic spectra (Li et al., 2025b). Collectively, these results suggest that concept spaces are locally linear yet globally curved, compositional, and dynamic. motivating our perspective of reasoning as flows on such manifolds.
Understanding Reasoning Phenomena. LLMs benefit from test-time scaling, where allocating more inference compute boosts accuracy on hard tasks (Snell et al., 2025). Explanations span expressivity- CoT enabling serial computation (Li et al., 2024), reasoning superposed trajecto- ries (Zhu et al., 2025), and hidden planning in scratch-trained math models (Ye et al., 2025) to inductive biases, where small initialization favors deeper chains (Yao et al., 2025). Structural analyses view reasoning as path aggregation or graph dynamics with sorld-world properties (Wang et al., 2024a; Minegishi et al., 2025), while attribution highlights key "thought anchors" (Bogdan al., 2025). Empirical work shows inverted-U performance with CoT length and quantifiable reasoning boundaries (Wu et al., 2025; Chen al., 2024), and embedding traddingtrrjeetorry geometry supports OOD detection (Wang et al., 2024b). Cai et al. (2025) shows RL is more sensitive to backtracking structure than content correctness, clarifying SFT-RL interplay reasoning. Moving beyond text, latent- reasoning methods scale compute through recurrent depth, continuous "soft thinking." and latent CoT for branch exploration and self-evaluation (Zhang et al., 2025b; Geiping et al., 2025; Hao et al., 2024; Wang et al., 2025). Applications exploit these insights for steering and efficiency: steering vectors and calibration shape thought processes (Venhoff et al., 2025; Chen et al., 2025), manifold steering mitigates overthinking (Huang et al., 2025), and adaptive indices enable early exit (Fu et al., 2024).
PRELIMINARIES
3.1 LARGE LANGUAGE MODELS
Let V denote finite vocabulary of tokens, and let denote the parameters of a large language model (LLMM). An LLM defines a conditional probability distribution po Uict; P), Wt E V, where
3

---

Published as a conference paper at ICLR 2026
uet (Wi,.... u4-1) is the prefix of previously generated tokens and P V is the tokenized problem prompt. At each step t, inference proceeds by sampling 4 ~ Po Uut, P).
Definition 3.1 (Chain-of-Thought Reasoning). Given a prompi P PE Chain-of-Thouggh (CoT) reasoning is an iterative stochastic process that generates sequence U (wi, u2....; WT), ; V, via recursive sampling ut ~ Pe P, uu)), I. 1=1,...,
To enable geometric analysis of reasoning, we need mapping from discrete token sequences into continuous vectors, a transformation that modern LLMs naturally provide.
Definition 3.2 (Representation Operator). A Representation Operator is a mapping & V xI + Rd where (x1,..... .xn) V is token sequence and I is index specifying the representation rype (e.g. token position, a pooling rule, or an internal layer state). The output E(x, 1) E R is the embedding/repe cabbedddingreppesentattinn of under the selection rule For notational simplicity, we omit the index t unless explicitly required.
The range this operator defines the ambient space of reasoning:
Definition 3.3 (Representation Space), Given a representation operator E, the representation space is R : {E(x) E V"J C R" Elements of RR are continuous embeddings of discrete language impuls, serving as the foundation and empirical proxy for our geometric analysis of reasoning.
In practice, & may be instantiated by a pretrained encoder such Qwen3 Embedding (Zhang et al., 2025a) or OpenAl's text text-emmt eitt-emmeddiinee 1eeedinn--aaaaae --laage (Neelakantan et al., 2022), or by extracting hidden states directly from an LLM. Typical choices of include mean pooling, the hidden state of the final token, or specific layer-position pair within the model (Zhang et al., 2025a; Lee et al., 2025; Hessani, 2025; Nie et al., 2024). We interpret as projecting discrete language sequences into a continuous semantic space, potentially lying on a low dow-dimensional manifold embedded in IR (Modell et al., 2025; Engels et al., 2025; Kozlowski et al., 2025).
3.2 MENGER CURVATURE
We adopt Menger curvature (Menger, 1928) to quantitatively capture the geometric structure of reasoning flows. As a metric-based notion of curvature, Menger curvature simultancously reflects both angular deviation and distance variation, making particularly suitable for reasoning trajectories represented as discrete embeddings. We leave more details to Appendix D.2.
Definition 3.4 (Menger Curvature). Let I1, I2, I3 E R be three distinct points. The Menger curvature of the triple (2,, x3) is defined as the reciprocal of the radius R(11, of the unique circle passing through the three points: c(æ1, 22,
4 REASONING AS GEOMETRIC FLOWS IN REPRESENTATION SPACE
We formalize the view that LLMs reason by tracing trajectories in their representation space. A central question is whether LLMs exhibit intrinsic control over these flows, mirroring the human perspective. We hypothesize semantic content as a curve on a concept manifold, and logical structure acts as local controller of the trajectory. quantities that underpin the paper. We then rigorously formalize this construction and establish the correspondence between the LLM's representation space and the human concept space.
this section, we introduce the spaces, maps, and geometric
4.1 CONCEPT SPACE AND SEMANTIC TRAJECTORIES
Definition 4.1 (Concept Space). The concept space C is abstract semantic space that models human-level cognitive structures such as ideas, reasoning states, and problem-solving subtasks.
We assume C is endowed with a smooth geometric structure, allowing continuous trajectories to represent the evolution of conceptual content. This assumption can be traced back to the classical insight of William James (James et al., 1890), who famously argued that consciousness does not appear itself "chopped up in bits." Chains trains of thought are, in his words, inadequate metaphors; instead, "it is nothing jointed; it flows, A river or a stream are the metaphors by which most naturally described."
4

---

Published as a conference paper at ICLR 2026
Definition 4.2 (Semantic Subspace Cognitive Trajectories). Let M C C denote semantic subspace corresponding to a coherent domain of meaning (e.g. temporal concepts, colors, or causal relations). Let X denote the set of all finite input sequences over X. We introduce trajectory map
:: X Curves(M),
X X 7x,
that assigns each sentence XT (I (II,...,TT) to a continuous curve xx within M. Formally,
X : [0, 1] M, s +
where E [0, 1] a continuous progress parameter along the reasoning flow. For each discrete prefix (211 -01,...,I1)) align with the point 7x (+) on the curve. The curve 7x thus traces the gradual unfolding of semantic content, formalizing the view that human cognition operates as continuous flow of concepts rather than as a sequence of isolated symbols.
We then define the logic space that mirrors the human view of logic.
Definition 4.3 (Formal Logical Space). The formal logical space L is abstract domain that captures structural dynamics of reasoning (natural deduction (Troelstra & Schwichtenberg, 2000; Pelletier & Hazen, 2024); see Definition 5.1). Define the flow operator
Fe: Curves(C) +
Ltorm:
which maps semantic trajectory to its formal counterpart. Semantically different expressions that proposition map to the same element in Liorm.
correspond to the same
4.2 REPRESENTATION SPACE
proxies to study human cognition and to investigate why LLMs exhibit reasoning phenomenon. We build on the multidimensional linear representation hypothesis (Modell et al., 2025), which posits that representations decompose linearly into superpo- sition of features. Each feature corresponds to a basis direction within a feature- seetures subspace of the embedding space, weighted by non-negative activation coefficient encoding its salience.
We use LLM representationseebbedddgg
Hypothesis 4.4 (Multidimensional Linear Representation Hypothesis (Modell et al., 2025)), Let X denote the input space (e.g., natural language sentences). Let F be a set of semantic features. For each feature f E F, let W, C Rdd denote feature-specite subspace of the embedding space.
Then the representation map V X =Rd of input E X is assumed to take the form
v(x)= Σ pr(z)wf(z). SEF(w)
(f 0} is the set of active features in Py(z) Rzo is non-negative scaling coefficient encoding the intensity or salience of feature f in x. wy(x) ) E WW is a unit vector (1w 1) specifying the direction of feature f within its subspace Wy.
where F(x) =(x)=ffe
Building this compositional picture, model reasons, its internal representation evolves. The next definition formalizes this evolution as a cumulative flow in embedding space.
from single inputs growing contexts. As
Definition 4.5 (Reasoning Trajectory / Context Cumulative Flow). Let X be the input space, and : X Rd the representation mup from finite input sequences to the embedding space defined in Hypothesis 4.4. Given prompt P X and Chain-of sequence XT (X1, "0,,,..,377 FT) with In E X, define
S. (P.
:= := Rd.
1=
When focusing solely on the reasoning process (ignoring the prompt), set
Vt := (X,) E Rd,
f t= t.
The sequence Y [y1...., yrl E dxT is called the context cumulative flow. The construction of Y follows Algorithm 1.

---

Published as a conference paper at ICLR 2026
Algorithm 1: Get Context Cumulative Reasoning Trajectory
V: vocabulary space; P V": tokenized problem prompt; T: number of reasoning steps;
T 1²: tokens for step i; E V + I: representation operator; y/ R embedding at step t.
Input: E V"; X [Ti..... _r]] with Da E V
Output: Y [y 11,.... , E RaxT
Y [] So [P];
for l - to do
S: - Concat (S:-,,
11 Concatenate with previous context // Get embedding current step
YI - {S:);
Append to Y;
return Y;
The embeddings we observe along a sentence are discrete, while reasoning itself naturally under- stood unfold continuous process. It is therefore natural to posit underlying smooth curve from which these discrete points arise as samples, thereby enabling the use of geometric tools such as velocity and curvature.
Hypothesis 4.6 (Smooth Representation Trajectory). The discrete representations {uo produced by context accumulation intrinsically lie on C² curve V :0,, + Rd satisfying
T (st) y/
for an increasing schedule $1 .... ST-
In other words, the sequence is not merely fitted by smooth curve, but should be regarded as samples from an underlying smooth trajectory. This assumption is reasonable: in Appendix D. we show an explicit construction of such C trajectory via relaxed prefix-mask mechanism.
Once a smooth trajectory exists, we can canonically align symbolic progress (e.g.. "how far along the derivation are") with geometric progress in representation space. The following corollary formalizes this alignment on domains where the symbolic schedule is well-behaved.
Corollary 4.7 (Canonical Alignment). On domain where I is injective and V is defined, there exists a canonical alignment
A: Curves(C) + Curves(R),
A: A
Remark 4.8 (Injectivity T). Corollary 4.7, r X + CCrves(C) (Definition 4.2) maps linguistic imputs conceptual trajectories. We do assume that T globally injective all natural language, as different surface forms may express essentially the same conceptual content. For the purpose of defining the aligument map A = T o r-1, it suffices that I be injective on restricted semantic domain (e.g., equivalence classes of paraphrases). Understanding global injectivity T remuins brouder open problem in AI, semuntics, and cognilive science research.
4.3 LOGIC AS DIFFERENTIAL CONSTRAINTS ON FL.WW
We now from the structural hypotheses of representation trajectories to their dynamical regulation. In particular, we view logic not as an external add-on, but as set of differential constraints shaping how embeddings evolve step by step. This perspective enables us to couple discrete reasoning structure with continuous semantic motion.
Definition 4.9 (Representation-LLgg) Space). Given a representation trajectory Y = (y1...., yr) defined in Definition 4.5, define local increments Ayj := -t-Wit-1frrt 2 2. The representatioon-lgiie space
Crep (Ayz. ................................ AYr) Y context-cumulative trajectory }.
The above constructs discrete object: sequence of increments capturing how representations change from one reasoning step to the next. To connect this discrete view with a continuous account of semantic evolution, we next introduce the notion of velocity along embedding trajectories.
Definition 4.10 (Flow Velocity). Let associated with a sentence. The flow velocity at progress s is defined as v(s) de (s). which captures the instantaneous rate of change of the embedding w.r.t. the unfolding of the sentence.
[0, - Rd be the continuous embedding trajectory

---

Published as a conference paper at ICLR 2026
By relating local increments in representation space (Definition 3.3) to the derivative of a continuous trajectory, we can interpret each discrete reasoning step as an integrated outcome of infinitesimal semantic motion.
Proposition 4.11 (Logic as Integrated Thought). By the fundamental theorem of calculus, the cumulative semantic shift between two successive reasoning steps 8, and 814.1 1
v(s) ds = (8(+1) (s:) = V1+1 yt = Att+1.
Thus, could view each representation step the integration of local semantic velocity, which aggregates infinitesimal variations of meaning into a discrete reasoning transition. Definition 4. 10 captures the central principle that semantic representations evolve continuously, whereas logical steps are inherently discrete: logic acts as the controller of semantic velocity, governing both its magnitude and its direction.
Continuous-Disertte Correspondence as Structural Constraint. Having established the con- ceptual implication of continuous discrete correspondence, we emphasize that this correspondence reflects the structural influence of logic on flow dynamics, rather than a direct mapping from specific inference rules (e.g., conjunction, negation) vector-field operators.
We now ask: what properties of reasoning flows should persist under changes in surface semantics? We posit that reasoning instances sharing the same natural-deduction skeleton, yet differing in semantic carriers (e.g., topics or languages), should yield reasoning flows whose trajectories exhibit highly correlated curvature (Definition 3.4). If logic governs flow velocity-botll magnitude and direction-ther flows instantiated with different carriers may undergo translations or rotations, reflecting dominant semantic components the underlying space. Nevertheless, their overall curvature should remain invariant. See more detailed discussion of curvature in Appendix D.2. Such correlation would indicate that the accumulation semantic variation produces turning points aligned with both LLM reasoning and human logical thought.
Invariance Under Semantic Variation.
Bridging Formal and Representational Logic. This theory directly corresponds to the central objective of this paper: clarifying the relationship between the two logical spaces fior and Crep- as illustrated in Figure Ic. Empirical evidence for this claim will be provided later, where we demonstrate cross-carrier similarity in both first-order differences and curvature.
Summary. In summary, logic functions as the differential regulator of semantic flow, discretizing continuous variation into meaningful steps. For clarity and reference, all mappings and derivational relationships introduced in this subsection are systematically summarized in Appendix C.
FORMAL LOGIC WITH SEMANTIC CARRIERS
5.1 LOGIC AND NATURAL DEDUCTION SYSTEM
We construct dataset of reasoning tasks that instantiate the fundamental logical patterns formalized in Definition 5. Each task is presented step by step in both formal symbolic notation and natural language. To test whether reasoning relies surface content underlying structure, express the same logical skeletons across diverse carriers, e.g.. topics such as weather, education, and sports, as well as multiple languages (en, zh, de, ja). This design disentangles logics from linguistic surface and provides controlled setting for analyzing how reasoning flows behave under varying contexts. Specifically, because all variants preserve the same abstract reasoning steps but change the words and context, any similarities that persist across carriers must come from the underlying logic, whereas the differences arise from surface semantics.
Definition 5.1 (Natural Deduction System (Troelstra & Schwichtenberg, 2000; Pelletier & Hazen, 2024)). natural deduction system is a pair ND = (F,R) where:
F: formal language of formulas (e.g., propositional first-order logic).
R: finite set of inference rules with introduction and elimination rules for each logical constant.
A derivation (or proof) in ND a tree whose nodes are judgements of the form "a formula is derivable" " and whose edges follow inference rules from R. Temporary assumptions may be introduced in sub-
7

---

Published as a conference paper at ICLR 2026
Table Comparison of reasoning- floo similarities across LLMs. We report mean cosine similarity (position, velocity) and Pearson correlation (curvature) under 3 grouping criteria: logic, topic, and language. Results show that position similarity is dominated by surface carriers, while velocity and curvature highlight logical structure the primary invariant. We also perform random shuffle baseline on the order logical inferences using Qwen3 0.6B. See Section for more details.
Position
Similarity
Velocity Similarity
Curvature Similarity
Model
Logic
Topic
Lang.
Logic
Topic
Lang.
Logic
Topic
Lang.
Qwen1.5 0.5B
0.26
0.30
0.80
0.17
0.07
0.08
0.52
0.11
0.15
Qwen2 0.5B
0.21
0.24
0.85
0.15
0.06
0.07
0.52
0.10
0.12
Qwen3 0.6B
0.26
0.30
0.85
0.17
0.07
0.08
0.53
0.11
0.13
Qwen3 1.7B
0.44
0.46
0.89
0.19
0.08
0.09
0.46
0.13
0.15
Qwen3 4B
0.33
0.35
0.86
0.16
0.07
0.08
0.53
0.11
0.13
LLaMA3 8B
0.31
0.34
0.74
0.15
0.06
0.07
0.58
0.13
0.17
Random Shuffle
0.30
0.33
0.81
0.02
0.02
0.04
0.02
0.03
0.04
derivations and are discharged by certain rules (e.g., - I, -I). Each connective is governed by paired introduction and elimination rules, which together determine proof-theoreti theoretic meaning.
5.2
DATA DESIGN
To test whether LLM reasoning trajectories are governed by logical structure rather than semantic content, we generates parallel reasoning tasks that maintain identical logical scaffolding while systematically varying superficial characteristics, specifically topical domain and linguistic realization.
Our dataset construction employs a principled two-stage generation pipeline using GPT-5 (OpenAI, 2025). It proceeds follows: (i) abstract logical templates are first constructed, followed by (ii) domain-specific and language-specifie rewriting. Our final dataset comprises 30 distinct logical structures, each containing between and 16 reasoning steps. Each logical structure is instantiated across 20 topical domains and realized in four languages (English, Chinese, German, and Japanese), yielding total corpus 2,430 reasoning sequences. This controlled design enables direct comparison of trajectories across logical forms and surface carriers, isolating the role of logical structure in embedding dynamics. Full generation prompts and sampled data cases provided in Appendix E.
6 PLAY WITH LLMS
6.1 EXPERIMENTAL SETUP
We employ the Qwen3 (Yang et al., 2025) family models and I.LMM33 (Grattafiori al., 2024). From the final transformer layer (before the LM head), we extract context-dependent hidden states [h (4), where h (L) E Rd denotes the representation at layer L and position i. Each reasoning step 2, set of tokens indexed by S,, and its step-level embedding defined by mean pooling:
= J.. Eies, (L) Ye E IRd The resulting sequence = (y1,..., ... forms the reasoning trajectory in representation space.
6.2 RESULTS ANALYSIS
Similarity Table. We evaluate L.LMs by extracting hidden states across our dataset (Section and computing similarities under three criteria: (i) Logic, grouping by deduction skeleton and averaging across topics and languages; (ii) Topic; and (iii) Language, both capturing surface carriers. This yields position, velocity, and curvature similarities (Table l). Results show that logical similarity is low at zeroth order (position) but becomes dominant at first and second order (velocity and curvature), validating our hypothesis. Topic and language exhibit low velocity similarity, suggesting they might occupy orthogonal subspaces; by contrast, the high logical similarity at first and second order breaks this orthogonality, indicating that logical structure transcends surface carriers.
8

---

Published as a conference paper at ICLR 2026
Uipar Netwer Deeurty
L.
LE
L.D
0.C 1.5
LA4
175
1700
1750
17A
LFF
1:50
1.C.
1:0
Position Similarity.
(b) Velocity Similarity.
(c) Curvature Similarity.
Figure 2: Similarity of reasoning flows on Qwen3 0.6B. Blocks correspond to logic templates (L:A-B) instantiated with different topics and languages. (a) Position similarity (mean cosine): diagonals correspond to topics (e.g., Network Security), showing that positions dominated by surface semantics. (b) Velocity similarity (mean cosine): semantic effects diminish, and flows with the same logical skeleton align while differing logics diverge. (c) Curvature similarity (Pearson): separation further amplified, with logic emerging as the principal invariant and revealing close similarity between logics B and C. See Section for more details.
Random Shuffle. We include a random shuffle baseline with Qwen3 0.6B in Table 1, where the order of logical steps (i.e., sentences) is randomly permuted and then fed into LLMs. The baseline performs poorly on velocity and curvature, indicating that the sequence order of reasoning is crucial for reasoning flow. In contrast, position similarity remains high, confirming that topic and language effects dominate surface semantics regardless of order. This contrast reinforces our view that higher- order geometric quantities, not raw embeddings, capture the invariant logical structure.
Stable Scaling Effect. Moreover, Table shows two scaling axes: (1) model size (Qwen3 0.6B + 1.7B >> 4B) and (2) model family (Qwen 1.5/2/3 and LLaMA3, varying data and architecture). The similarity patterns remain remarkably stable. Increasing scale or switching families does not materially change the similarity measures. This consistency suggests the presence of more general property of how LLMs internalize logical structure, independent of size or training recipe (Huh et al., 2024). We view this fascinating direction and plan to explore further.
Similarity Heatmap. For visualization, we also analyze Qwen3 0.6B on subset of our dataset (Figure 2). At the position level, embeddings cluster by topic and language. First-order differences reveal logical control: flows sharing the same skeleton align, while differing logics diverge even with identical carriers. Second-ord -ooodd curvature further amplifies this separation, and its strong cross-carrier consistency directly supports Proposition 4.11, confirming that logic governs reasoning velocity. Additional experiments across broader model families are presented in Appendix B.
Challenging the Stochastic Parrots Argument. Together, these results show that LLMS internalize latent logical structure beyond surface form, challenging the stochastic parrots (Bender et al., 2021) perspective. Specifically, the stochastic parrot argument suggests that LLMs do perform genuine natural language understanding and cannot acquire meaning solely through next-token prediction (Bender al., 2021; Bender & Koller, 2020). However, we provide evidence that learning from next- token prediction, augmented by standard post-training such as instruction tuning, can be sufficient to elicit semantic structure and support genuine natural language understanding. Moreover, we show that logical structure manifests in higher-order geometry of the representation space, which is difficult to explain under a purely surface-form view.
Significance. These findings suggest the existence of deeper, possibly universal constraints linking linguistic form to meaning. The generality, universality, and model-agnostic nature of the observed patterns align with the Platonic Representation Hypothesis (Huh et al., 2024; Jha et al., 2025), which posits that neural networks trained on different data and objectives nonetheless converge to shared underlying meanings in their representation spaces. In this view. LLMs acquire logical structure through next-tode token prediction because the most effective way to compress knowledge is to internalize and represent its underlying logic (Sutskever, 2023; Goldblum al., 2024).

---

Published as a conference paper at ICLR 2026
7 DISCUSSION
Understanding, Not Generation. Our theoretical framework focuses exclusively on natural lan- guage understanding. In NLP, generation presupposes understanding, so we restrict ourselves the component that is conceptually cleaner and more amenable to theoretical analysis. Our goal is to articulate a post-hoe, model-agnostic, and
law describing how LLMs reason. Explaining why or how these reasoning patterns arise during training generation is substan- tially more challenging and falls outside the scope of this work. Because we do not study generation, we also cannot meaningfully relate our geometric measures to generation-specifie properties such as task output accuracy. Finally, uncovering the origins of these patterns would require analyzing the learning process itself for example, learnabllity, training dynamics, detailed case studies, and more sophisticated experimental and dataset designs. These important questions are beyond the present scope and are left for future exploration.
Contrast with Graph Perspective. Prior works have modeled chain-of-thought reasoning as a graph structure (Minegishi et al., 2025: Wang et al., 2024a). While this provides useful perspective, its predictive power is limited: graphs naturally suggest random walks between discrete nodes, which fits the noisy behavior of isolated embeddings but fails to capture the smooth, directed dynamics we observe under cumulative context. Our results in Section 6 show that well-trained LLMs learn flows governed by logical structure, transcending the surface semanties of language. Such continuity and logic-driven trajectories cannot be explained within purely graph-based framework, but arise naturally in our differential-geometrii view.
Other Components in Learned Representation. Beyond logical structure, learned representations also encode a wide spectrum of factors such as semantic objects, discourse tone, natural language identity, and even signals of higher-level cognitive behavior. Extending our framework to systemati- cally isolate these components and characterize their interactions presents a major challenge for future work. A promising direction is to develop methods that disentangle additional attributes, enabling finer-grained insights into how language components co-evolve in representation space.
Practical Implications. Our results suggest that reasoning in LLMs unfolds continuous flows, opening multiple directions. First, trajectory-level control offers principled tools for steering, align- ment, and safety, extending vector-based interventions to flow dynamics (Venhoff et al., 2025; Chen et al., 2025; Gong et al., 2025; Huang et al., 2025; Bereska & Gavves, 2024). Second, our geometric view provides a formal framework study abstract language concepts, enabling first-principle analyses of reasoning efficiency, stability, and failure modes. Third, it motivates new approaches to retrieval and representation, where embeddings respect reasoning flows rather than mere similarity, potentially improving RAG, reranking, and search (Weller et al., 2025; Cao et al., 2025). Finally, hints at architectural advances, as models parameterizing latent flows may enable more efficient reasoning (Hao et al., 2024; Geiping et al., 2025; Zhang et al., 2025b; Shen et al., 2025).
8 CONCLUSION
We introduced a novel geometric framework that models LLM reasoning as smooth flows in rep- resentation space, with logic acting as controller of local velocities. By disentangling logical structure from semantic carriers through controlled dataset, we showed that velocity and curvature invariants reveal logic as the principal organizing factor of reasoning trajectories, beyond surface form. Our theory and experiments provide both conceptual foundation and practical tools for analyzing reasoning, opening new avenues for interpretability.
ACKNOWLEDGMENTS
ARZ was partially supported by NSF Grant CAREER-2203744 and NIH Grant RO1HL169347. This research is supported by Google.org, the Google Cloud Research Credits program for the Gemini Academic Program, and Amazon AGI Labs SF.
10

---

Published as a conference paper at ICLR 2026
LLMS USAGE STATEMENT
We clarify that LLMs were used solely as auxiliary aids, restricted to refining the manuscript's exposition for clarity and conciseness.
REFERENCES
Anthropic. Transformer circuits. httpps///ttanssfrrrr--aaaooood pubb/ 2021.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang. Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 5185-5198, 2020.
Emily M Bender, Timnit Gebru, Angelina McMillan-Majje and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety a review. TMLR, April 2024.
Joseph M Bochenski and Ivo Thomas. A history of formal logic. 1961.
Paul C Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought anchors: Which I1m reasoning steps matter? arXiv preprint arXiv:2506. 19143, 2025.
Hongyi James Cai, Junlin Wang, Xiaoyin Chen, and Bhuwan Dhingra. How much backtracking is enough? exploring the interplay of sft and rl in enhancing Ilm reasoning. arXiv preprint arXiv:2505.24223, 2025.
Weili Cao. Jianyou Wang, Youze Zheng, Longtian Bao, Qirui Zheng, Taylor Borg-Kirkpatrick, Ramamohan Paturi, and Leon Bergen. Single-pass document scanning for question answering. In Second Conference on Language Modeling, 2025. URL https: //tpr/eeeee/eeeeeeeeemee net/
farum?id=7Ve 8acKIp.
Qiguang Chen, Libo Qin, Jiaqi Wang, Jingxuan Zhou, and Wanxiang Che. Unlocking the capabilities of thought: reasoning boundary framework to quantify and optimize thain-of-thoughe Advances in Neural Information Processing Systems, 37:54872-54944 2024.
Runjin Chen, Zhenyu Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang. Seal: Steerable reasoning calibration of large language models for free. arXiv preprint arXiv:2504.070 079986 2025.
Irving M Copi, Carl Cohen, and Kenneth McMahon. Introduction logic. Routledge, 2016.
Manfredo P Do Carmo. Differential geometry of curves and surfaces: revised and updated second edition. Courier Dover Publications, 2016.
Diego Doimo, Alessandro Serra, Alessio Ansuini, and Alberto Cazzaniga. The representation landscape of few-shot learning and fine-tuning in large language models. Advances in Neural Information Processing Systems, 37:18122-18166 2024.
Herbert B Enderton. A mathematical introduction to logic. Elsevier, 2001.
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / openreview wwwp:e//t////iiiee
344M4.b.
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-Jussà. A primer on the inner workings of transformer- language models. arXiv preprint arXiv:2405.002208 2024.
11

---

Published as a conference paper at ICLR 2026
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Yonghao Zhuang, Yian Ma, Aurick Qiao, Tajana Rosing, lon Stoica, et al. Efficiently scaling Ilm reasoning with certaindex. arXiv preprint arXhv:2412.2099) 2024.
Peter Gardenfors. Conceptual spaces: The geometry of thought. MIT press, 2004.
Peter Gardenfors. The geometry of meaning: Semantics based on conceptual spaces. MIT press, 2014.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer. Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.05/71, 2025
Micah Goldblum, Mare Finzi, Keefer Rowan, and Andrew Gordon Wilson. The no free lunch theorem, kolmogorov complexity, and the role of inductive biases in machine learning. In Proceedings of the 4/st International Conference on Machine Learning. 2024.
Bofan Gong, Shiyang Lai, and Dawn Song. Probing the vulnerability large language models to polysemantic interventions. arXiv preprint arXiv:2505. 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. XrXiv e-prints, pp. 2407, 2024.
Heinrich W Guggenheimer. Differential geometry. Courier Corporation, 2012.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprins arXiv:2501.1294 2025.
Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL https: /tttps://operreviee
horum? id= jE8xbmvF Fin.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu. Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769 2024.
Hesam Sheikh Hessani. Llm embeddings explained: A visual and intuitive guide, 2025.
Noel J Hicks. Notes on differential geometry, volume van Nostrand Princeton. 1965.
Michael Y Hu, Jackson Petty, Chuan Shi, William Merrill, and Tal Linzen. Between circuits and chomsky: Pre-pretraining formal languages imparts linguistic biases. arXiv preprint arXiv:2502.19249, 2025.
Yao Huang, Huanran Chen, Shouwei Ruan, Yichi Zhang, Xingxing Wei, and Yinpeng Dong. Mitigat- ing overthinking large reasoning models via manifold steering. arXiv preprint ar11:22505.22411, 2025.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic repre- sentation hypothesis. In Forty-first International Conference on Machine Learning, 2024. URL
https://cpentee /ttpgr///ppeeevieww net forum? Baw.ummiid-BBBTAAA. BTxYyr
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, arXiv preprint arXiv:2410.21276 2024.
William James, Frederick Burkhardt, Fredson Bowers, and Kestutis Skrupskelis. The principles of psychology, volume 1. Macmillan London, 1890.
Rishi Dev Jha, Collin Zhang, Vitaly Shmatikov, and John Xavier Morris. Hamnessing the universal geometry embeddings. In The Thirry-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https: http///ppeeevveeeeeeeeeeeeeeeeee ew, net rumm?? d= C10Pq5xV. c
htttp:://peereve
12

---

Published as a conference paper at ICLR 2026
Jin Jiang, Jianing Wang, Yuchen Yan, Yang Liu, Jianhua Zhu, Mengdi Zhang, Xunliang Cai, and Liangcai Gao. Do large language models excel in complex logical reasoning with formal language? arXiv preprint arXhv:2505.16998 2025.
Yibo Jiang, Bryon Aragam, and Victor Veitch. Uncovering meanings of embeddings via partial orthogonality. Advances in Neural Information Processing Systems, 36:31988 11:31988-3222 36:31988-32005, 2023.
Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In Forty-first International Conference on Machine Learning, 2024. URL https://///.........ttt /openreveiveeeeeeeeeeeeee eww. ew. net/ forum? ?id=
ottTw4Mghk.
Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition. arXiv preprint arXiv:2502.00887. 2025.
Austin C Kozlowski, Callin Dai, and Andrei Boutyline. Semantic structure in large language model embeddings. arXiv preprint arXiv:2508. 10003, 2025.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gus- tavo Hernández Abrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, et al. Gemini embedding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.07891, 2025.
Melody Zixuan Li, Kumar Krishna Agrawal, Arna Ghosh, Komal Kumar Teru, Adam Santoro, Guillaume Lajoie, and Blake Aaron Richards. Tracing the representation geometry of language models from pretraining to post-training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL https: wwwpeeeeeeeee..//////p/ net forum? ?id= FDruZ1KWUb.
Yuxiao Li, Eric J Michaud, David D Back, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. Entropy, 27(4):344, 2025b.
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transform- ers to solve inherently serial problems. In The Twelfth International Conference on Learning Representations, 2024. URL https:://// /ttp:///epeeeeeeeeeeemme htt.eeeee.ee.oeeeeeeeeeeeeeeeeee http://oommm1m-32EEeymmee
Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang. Logical reasoning in large language models: A survey. arXiv preprint arXiv:2502.09100. 2025.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34663, 2022.
Andreas Madsen, Himabindu Lakkaraju, Siva Reddy, and Sarath Chandar. Interpretability needs a 2024.
new paradigm. arXiv preprini arXiv:2405.085.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, 2024. URL hwtpthtss//sspeeeeeeeeee /wwpnreveeeeeeeeeeeeee ew net forum? i j=aajyHYj
Karl Menger. Untersuchungen über allgemeine metrik. Mathematische Annalen, 100(1):75-163,
1928.
William Merrill and Ashish Sabharwal. A logic for expressing fog-precision transformers. Advances in neural information processing systems, 36:52453-52463, 2023.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. In ICLR, 2013a.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pp. 746-751, 201.3b.
13

---

Published as a conference paper at ICLR 2026
Gouki Minegishi, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. Topology of reasoning: Understanding large reasoning models through reasoning graph properties. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025. URL
https: /ppnrevveeeeeeeeeeeeeeeeeeeeeeee new. net/ for.mmorrmmmmmommm d=o1 g8NWkxqf.
Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley. The origins of representation manifolds in large language models. arXiv preprint arXiv:2505.182.0 2025.
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reasoning ca- pabilities of Ilms via principled synthetic logic corpus. Advances in Neural Information Processing Systems, 37:73572-73604, 2024.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023, URL https: wpwprevieeeeeeeee/e/ee net forum ?id- 9XF SbDP.dli.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.1000 2022.
Zhijie Nie, Zhangchi Feng, Mingxin Li, Cunwang Zhang, Yanzhao Zhang, Dingkun Long, and Richong Zhang. When text embedding meets large language model: a comprehensive survey. arXiv preprint arXiv:2412.09165. 2024.
OpenAl. Introducing chatgpt. https: //ttpaii www.
http://peeeaaa.cooe///e///eeeeee wttp////n/d///aatp/////////////e 2022.
OpenAl. Gpt-5 system card. Technical report, OpenAl, August 2025.
Core Francisco Park, Maya Okawa, Andrew Lee, Ekdeep S Lubana, and Hidenori Tanaka. Emergence of hidden capabilities: Exploring learning dynamics in concept space. Advances in Neural Information Processing Systems, 37:84698-84727 84729, 2024a.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference Machine Learning, 2024b. URL https: / opwnrevieeeeeeeeeeeoeeeee net forum? i UGpGKLzwpP.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Vietor Veitch. The geometry of categorical and hierarchi- cal concepts in large language models. In The Thirleenth International Conference on Learning Representations, 2025. URL https: openreviee net/ forum wet.oorumiidiii.o.. -i
Mihir Parmar, Nisarg Patel, Neeraj Varshney. Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In 62nd Annual Meeting of the Association for Computational Linguistics. ACL 2024, pp. 13679-13707. Association for Computational Linguistics (ACL), 2024.
Francis Jeffry Pelletier and Allen Hazen. Natural Deduction Systems in Logic. In Edward N. Zalta and Uri Nodelman (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Spring 2024 edition, 2024.
Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-basee language models. arXiv preprint arXiv:2407. 026466, 2024.
John T Rickard. A concept geometry for conceptual spaces. Fuzzy optimization and decision making. 5:311-329 2006.
Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, and Jiuxiang Gu. Efficient reasoning with hidden thinking. arXiv preprint arXiv:2501. 19201, 2025.
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.0/76. 2024.
Charlie Victor Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth Inter- national Conference on Learning Representations, 2025. URL https: / openreview. n forum? iodumidd=Faa F2d...........
14

---

Published as a conference paper at ICLR 2026
Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. Towards compositionaliyy in concept learning. In Forty-first International Conference on Machine Learning, 2024. URL https://cpenreeee
nnt/ Eorumiid-upo@FUn..
Ilya Sutskever. An observation on generalization. Workshop on Large Language Models and Transformers, Simons Institute, 2023, https://www youtubuu com/ wat AKMAA
w1yy3A6ab_ccaanel... 1immoonssmmmssss Institute.
A. S. Troelstra and Helmut Schwichtenberg. Basic Proof Theory. Cambridge University Press, 2nd edition, 2000.
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36:51 36:51234-51252, 2023.
Constantin Venhoff, Iván Areuschin, Philip Torr, Arthur Conmy, and Neel Nanda. Understanding reasoning in thinking language models via steering vectors. In Workshop Reasoning and Planning for Large Language Models, 2025. URL 1d=OwhVNNOBC2.
fer/morum?
Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, and William Yang Wang. Understanding reasoning ability of language models from the perspective of reasoning paths aggregation. In International Conference on Machine Learning, pp. 50026-50042. PMLR, 2024a.
Yiming Wang. Pei Zhang, Baosong Yang, Derek Wong, Zhuosheng Zhang, and Rui Wang. Embed- ding trajectory for out-of-distributiin detection in mathematical reasoning. Advances in Neural Information Processing Systems, 011442655-49999 2024b.
Yiming Wang, Pei Zhang, Baosong Yang, Derek F. Wong, and Rui Wang. Latent space chain-of- embedding enables output-free LLM self-evaluation. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https:/ htt.:p:://ppereviee... net forum?ii ? did= jxo70B9f0o.
Zihao Wang, Lin Gui, Jeffrey Negrea, and Victor Veitch. Concept algebra for (score-based) text- controlled generative models. Advances in Neural Information Processing Systems, 36:35331 35349, 2023.
Orion Weller, Michael Boratko, Iftekhar Naim, and Jinhyuk Lee. On the theoretical limitations of gabeddiing-bsse retrieval. arXiv preprint arXiv:2508.21033 2025.
Ludwig Wittgenstein. Tructatus Logico- Philosopoiisss.c.sss. Kegan Paul, Trench, Trubner & Co., Ltd., London, 1922.
Yuyang Wu, Yifei Wang, Ziyu Ye, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thought length in llms. arXiv preprint 202.0222022.02266. 2025.
Yuan Xia, Akanksha Atrey, Fadoua Khmaissia, and Kedar S Namjoshi. Can large language models learn formal logic? data-driven training and evaluation framework. arXiv preprint arXiv:2504.202/3. 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 099888500.00388. 2025.
Andy Yang and David Chiang. Counting like transformers: Compiling temporal counting logic into softmax transformers. In First Conference on Language Modeling, 2024. URL https: /ppereeiiiew net/ forum? FimmmidFFFnFg4UJK
Junjie Yao, Zhongwang Zhang, and Zhi-Qin John Xu. An analysis for reasoning bias of language models with small initialization. In Forty-second International Conference on Machine Learning, 2025. URL https:tpp///c http:///pppeevvieeeeeee net forum? d= 4FQMMUUYAT..
15

---

Published as a conference paper at ICLR 2026
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. In The Thirteenth International Conference Learning Representations, 2025.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05/76 2025a.
Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang. Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of Ilms in continuous concept space. arXiv preprint arXiv:2505. 15778, 2025b.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in neural information processing systems, 36:27223-27250, 2023.
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by superposition: A theoretical perspective on chain of continuous thought. arXiv preprint arXiv:2505./2514, 2025.
16

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Work
3 Preliminaries
3.1 Large Language Models
3.2 Menger Curvature
4 Reasoning as Geometric Flows in Representation Space
4.1 Concept Space and Semantic Trajectories
4.2 Representation Space
4.3 Logic Differential Constraints Flow
5
Formal Logic with Semantic Carriers
5.1 Logic and Natural Deduction System
5.2 Data Design
Play with LLMs
6.1
Experimental Setup
8
6.2 Results Analysis
7
Discussion
10
8 Conclusion
10
Additional Related Work
18
Additional Experiments
19
C Symbolic Glossary and Mapping Relations
20
C.1 Spaces
20
C.2 Primary maps
21
C.3 Reasoning increments and curvature
22
C,4 Roadmap diagram
22
Geometric Foundations of Reasoning Trajectories
22
D.1 Continuity of Representation Trajectories
22
D.2 Menger Curvature
25
E Data Generation
27
E.1 Prompts for Data Generation
27
E.2 Data Examples
28
17

---

Published as a conference paper at ICLR 2026
Appendix
A ADDITIONAL RELATED WORK
Mechanistic Interpretability. LLMs have exhibited unprecedented intelligence ever since their debut (OpenAI, 2022). Yet the underlying mechanisms remain opaque, transformers neural networks not readily interpretable by humans-motivating efforts to uncover why such capabilities emerge (Singh et al., 2024; Madsen et al., 2024). Mechanistic Interpretability (MI) pursues this goal by reverse-enginne ennerreriinnnningngggg transformer internals into circuits, features. and algorithms (Rai et al., 2024; Ferrando al., 2024; Bereska & Gavves, 2024). The Transformer Circuits program at Anthropic exemplifies this agenda, systematically cataloging reusable computational subroutines (Anthropic, 2021). Empirical studies reveal concrete algorithmic mechanisms: grokking progresses along Fourier-like structures (Nanda et al., 2023), training can yield divergent solutions for the same task (Clock vs. Pizza) (Zhong et al., 2023), arithmetic emerges via trigonometric embeddings on helical manifolds (Kantamneni & Tegmark, 2025), and spatiotemporal structure is encoded through identifiable neurons (Gurnee & Tegmark, 2024). Beyond circuits, in-context learning and fine-tuning yield distinct representational geometries despite comparable performance (Doimo et al., 2024), while safety studies reveal polysemantic vulnerabilities where small-model interventions transfer to larger LLMS (Gong et al., 2025).
Formal Logic with LLMs. Recent work links transformer computation directly to logic. Log- precision transformers are expressible in first-order logic with majority quantifiers, providing an upper bound on expressivity (Merrill & Sabharwal, 2023), while temporal counting logic compiles into -aftmax-attention architectures, giving constructive lower bound (Yang & Chiang, 2024). Beyond these characterizations, pre- pre-pretrainin on formal languages with hierarchical structure (e.g.. Dyck) imparts syntactic inductive biases and improves efficiency (Hu et al., 2025). Synthetic logic corpora and proof-generation frameworks further strengthen reasoning, though benefits diminish as proofs lengthen (Morishita et al., 2024; Xia et al., 2025). Systematic evaluations, including LogicBench and surveys, highlight persistent failures negation and inductive reasoning, despite partial gains from "thinking" models and rejection finetuning (Parmar et al., 2024; Jiang et al., 2025; Liu et al., 2025). In contrast, our work employs formal logic not as an end task, but tool to validate our geometric framework in LLMs' representation space, distinguishing our contribution from prior lines of work.
Comparison with Park al. (2025). While Park al. (2025) and our work both adopt a geometric lens for semantic and theoretical interpretability, we address fundamentally different phenomena using distinct mathematical frameworks:
Objective (Static Semantics Dynamic Reasoning): The primary motivation of Park et al. (2025) is to map the static organization of semantic knowledge. They investigate how fixed concepts (e.g., hierarchies like "animal" + "dog"") are encoded relative one another the representation space. In contrast, our work investigates the dynamic process of reasoning. We model how internal representations evolve step-by-step static location but as a flow over time.
model solves a logical problem, framing reasoning not as
Geometric Nature (Structure vs. Motion): Consequently, the specific geometries differ. Park al. (2025) describe structural geometry where hierarchical relations are encoded via orthogonal subspaces and categorical concepts appear as polytopes (simplices). Our work describes a kinematic geometry defined by smooth flows, analyzing properties of motion such velocity and curvature along a reasoning trajectory.
Technique & Methodology:
Methodologically, Park et al. (2025) rely on linear algebraic tools, such Linear Diseriminant Analysis (LDA) and Causal Inner Products, to analyze static word unembeddings. Conversely, we develop differential-geometie framework to analyze context- cumulative trajectories. By generating controlled datasets that isolate logical skeletons from semantic carriers, we show that logic acts as differential constraint on the flow's 's velocity, rather than determining a static position in the subspace.
Comparison with Li et al. (2025a). While Li et al. (2025a) and work both analyze the geometry of LLM representations, we diverge in our fundamental objects of study (understanding vs. generation) and in analytical scope (post-hoc laws vs. training dynamics):
18

---

Published as a conference paper at ICLR 2026
Scope (Post-hoc Law of Reason Training Dynamies): Our goal is articulate post-hoe model-agnostic law that describes how LLMs reason once fully trained. We treat the model as a fixed dynamical system to derive differential-ggeee -eeometric constraints its internal flows. In contrast, Li et al. (2025a) focus on the origins of these patterns. They analyze the learning process itself, tracing how geometric properties (such effective rank) evolve through distinct training phases like "entropy-seeking" and "compression-seeking"
While they ask how representations evolve during training, we ask what invariant rules govern their trajectory during inference.
Domain (Natural Language Understanding vs. Generation): Our framework focuses exclusively on Natural Language Understanding. Conversely, Li et al. (2025a) explicitly link geometric measures to generation-seccitt properties, correlating spectral shifts with 7-gram memorization, cross-entropy loss, and output accuracy (e.g., pass@k) Because do not study generation, do not relate our measures to task output metrics, whereas validating these correlations is central to their empirical contribution.
Methodology (Theoretical Constraints vs. Empirical Correlations): Methodologically, et al. (2025a) rely on empirical correlations between spectral metrics (RankMe, a-ReQ) and downstream performance metrics across checkpoints. Our work disjoint from this experimental design; we employ differential geometry to formalize logic as a velocity constraint the reasoning flow. We leave the questions of learnability and training dynamics, central to Li et al. (2025a), outside our scope to focus on the geometry of the reasoning process itself.
B ADDITIONAL EXPERIMENTS
More Similarity Heatmap. We additionally evaluate LLaMA3 (Grattafiori al., 2024) and more Qwen3 (Yang et al., 2025) models (1.7B, 4B) to test robustness under the same experimental settings as in Section 6. The results (Figures 3, and 5) confirm that our findings generalize across model sizes and families.
C² Continuity Test. In Hypothesis 4.6, we posit that the discrete trajectory admits an underlying C interpolation. This assumption serves to make the geometric framework well-defined; it does nor assert that the model's 's internal computation unfolds continuous time. Consequently, a rigorous empirical verification of C' reguularryy is impossible: language inputs are discrete, and the sequence index cannot made infinitesimally small, as differentiabilliy would require. Instead, we provide muite-difference-bee based smoothness diagnostics to illustrate that the observed embedding trajectory behaves consistently with this assumption. For a context-cumulativ trajectory Y [y1...... yrl (Definition 4.5), form velocities t; M4+1 Ye and examine their magnitudes ||v: ||. As shown in Figure 6, the velocity norm behave consistently across the same MATH500 problem with six different answers presented in Figure 1, exhibiting no abrupt jumps. This provides visual support for the plausibility of our smooth-flor assumption. A more precise and formal theoretical justification is provided in Appendix D.
Netwak Eounty
L.0
L.C
LE
L8
Ca
112
1.5
1:0
LA
L.U
1/50
C65
CA
Position Similarity.
(b) Velocity Similarity.
Curvature Similarity.
Figure 3: Similarity of reasoning flows on Qwen3 1.7B.
19

---

Published as a conference paper at ICLR 2026
110
L.0
(a) Position Similarity.
(b) Velocity Similarity.
(c) Curvature Similarity.
Figure 4: Similarity of reasoning flows Qwen3 4B.
Taal Netkors Decuriy
10A
L.D
L.0
L5
LA
(a) Position Similarity.
(b) Velocity Similarity.
(c) Curvature Similarity.
Figure 5: Similarity of reasoning flows on Llama3 8B.
C SYMBOLIC GLOSSARY AND MAPPING RELATIONS
This section is standalone roadmap that summarizes the spaces, maps, and commutative structure underlying our geometric view of reasoning.
C.1
SPACES
Input space X' (often specialized a vocabulary V): discrete tokens/sentences.
Concept space C: abstract semantic space. A sentence X is represented by smooth
semantic trajectory
7x [0, MCC.
where M is semantic submanifold for coherent domain of meaning.
Representation space R C IR::: the model's embedding space. Each prefix X, yields
yt V (X,) E TRdd.
sampling continuous represeenation trajectory V [0, Rdd
Formal logical space Liorm: symbolic/human logic governed by natural deduction system ND (F, R), with formulas F and rules R. Judgements and rule-based derivations live here.
Representaaiio--based logical space Crep: the space of reasoning increments in the em- bedding space, defined by local variations of the trajectory, Aリ Ut+1 V1- Geometric descriptors such as the Menger curvature K, are evaluated here. This space is non-symbolic, and serves the model's internal analogue of logic.
20

---

Published as a conference paper at ICLR 2026
Velocity Norm (llv,l)
answer
0.35
answer
answer
answer
answer
answer 6
0.25
Ivil
0.2
0.15
0.05
2
8
10
Step
Figure 6: Velocity Norm ||x]]| for MATH500 problem and its six answers.
C.2
PRIMARY MAPS
Semantic interpretation:
T: X' Curves(C),
X nx.
Neural representation:
V X Curves(
realized by token embeddings & and contextual encoder o, producing the continuous trajectory and sampled states (y....., yr).
Canonical Alignment.
Definition C.1 (Canonical alignment map). Assume I and are injective the domain of interest. Define
A o r-1; Curves(C) Curves(R).
Then A is a bijection between semantic curves and representation trajectories, and the top-level diagram commutes exactly:
AoT Top=
Flow vs. differential to logic. We distinguish human flow operator on concepts from differential operator on representations:
Fe : 7 + (human reasoning flow in C) E Ltorms
DR : T + (Ayi) E Crep-
The left operator Fe is not discrete difference; encodes how semantic trajectory induces formal reasoning steps under ND. The right operator DR extracts local increments from the representation trajectory.
21

---

Published as a conference paper at ICLR 2026
C.3 REASONING INCREMENTS AND CURVATURE
Formal side (concepts). Human reasoning flow is captured at the semantic level by Fe. which maps semantic curve into sequence of formally valid steps in fror per the rules ND.
Representation side (vectors). The local increment Ay = M++1 - y/ encodes a step of representation flow in Crep.
Curvature as geometric intensity. For three consecutive states (y--1, Mt. M++1), the Menger curvature
2 CosSim u. v)²
= 1.
y/ Vt-1: Mt+1 yts
|1++1 yt-1
couples angular change with scale, providing geometry-aware proxy for the "strength" of a reasoning step in the representation.
C.4 ROADMAP DIAGRAM
The overall structure can be read from the commutative roadmap below. Here X' sits at the center; semantic and representation curves live to the left and right; formal and representation- based logics sit below. The top arrow is strict by definition of A; the vertical arrows express how each curve induces respective notion of reasoning.
X
Aovor-!
Curves (C)
Curves (R)
Fe
DR
Lior
Crep
Reading guide. (l) Input sequences branch into a semantic curve (left) and a representation curve (right), (2) The canonical alignment A L r'1 identifies the two curves one-to-one. (3) The semantic curve induces human, rule-constrained steps in Ltorm via Fc, while the representation curve induces vector increments in Lrep via DR. (4) Curvature in Crep quantifies the geometric intensity of reasoning transitions and can be related back to formal steps under appropriate correspondences established elsewhere in the paper.
D GEOMETRIC FOUNDATIONS OF REASONING TRAJECTORIES
In this section, we establish the geometric foundations for analyzing reasoning as smooth flows in representation space. We first construct representation trajectories as C' curves a relaxed prefix-mask mechanism, thereby justifying smoothness as a working principle. Then, we introduce Menger curvature computable descriptor that couples angular deviation with distance variation, providing principled measure of the intensity of reasoning turns.
D.I CONTINUITY REPRESENTATION TRAJECTORIES
In this section, we provide rigorous and explicit construction of C² trajectory using a relaxed prefix- mask mechanism. This construction justifies our working assumption that representation trajectories C Note that the symbol I (Definition 3.2) is defined with slight variation compared to main paper: here is specialized to encode positional information, while the remaining complexities the model architecture subsumed into single mapping T.
22

---

Published as a conference paper at ICLR 2026
Definition D.1 (Neural Encoding View of Sentence Representation). Let z (u1,-.. ..... sentence with tokens u, drawn from a vocabulary space V. Define an embedding map
Un) ....
be
8: - - RR.
44 ト Eus);
which assigns each token d-dimensional vector. Augmenting {(u)) with positional information yields the imput sequence
E (Rd")"
20
Let 4 (IR))" - denote a contextual encoder that maps a sequence of token embeddings together with positional information to global sentence-lew levell representation, where I is the posi- tional encoding space and In C I denotes the set of encodings for the first n positions. For a fixed t == (4).....bm) E In we define
.n)
(x) == @ (20, =
E Rd.
In this view, subsumes both the static token embeddings and the contextual transformations carried by the neural network.
Hence the hidden state = (S.) in Definition 4.5 should be interpreted not merely as a sum of embeddings, but the oulcome of the full encoding process applied the prefix S.-
Mask-aware realization (for later use). Fix a maximum length N 2 n and consider the mask-aware realization of the same encoder,
@m : (R2)N IN {0, Rd
such that for any length S N.
m E((n)),
0, 10),,(pp (iga)) == + (E(U)) E((n), (I (1i)))))))))
When the mask is all ones on (I,...,n), this coincides with the above definition; when we pass mask explicitly we will write ((,
Hypothesis D.2 (Smooth Trajectory Hypothesis). The sequence of representations V/ (X,)) generated during a reasoning process lies on a smooth, differentiable trajectory in the embedding space.
Definition D.3 (Relaxed-Mast) Sentence Representation). Let each sentence in Hypothesis 4.4 be F, (14,1 (e, for 1..... T. and let the full token stream be
Un (u,,1,--., U1,m1 -2,1,--.3 12, 2.2.------.
WT, NT,,1----. WT,r)) )
with total length = -E Th and cumulative lengths N, = E/= Nz. Introduce a continuous progress parameter E [0, and relaxed prefix mask
ma {l....,N] [0, 1].
which specifies the fractional inclusion of each token progress
Using the embedding map & and positional information IN from Definition D.I, define the masked input sequence at progress by
(())))
6" 6= (i)
and the associated hard mask
M,(t) :- (
1=
Let k((s) := [sN], denote the number of tokens included at progress 8. The truncated masked sequences then defined as
((h)) = (2)) (Rd)
(.(5)) (1*(1), ((()),..,, (
23

---

Published as a conference paper at ICLR 2026
With the mask-aware encoder Dm. (R I [0,13 Rd introduced above, the continuous representation trajectory is defined by
( = 2 (2 1, M,) E R.
where 2m (2ss M,) ) == (b (
A1 sentence boundaries 8t = NN the hard prefix mask is recovered exactly by choosing smooth function with flat tails (see Proposition D.4); consequently,
1/2 (Si) + (2 F 1",, Ma.) (st),
t=
Proposition D.4 (Continuity of the Relaxed-Mask Trajectory). Suppose the relaxed mask takes the form
(1) g(sN i),
where g E C (R.) satisfies g(x) for S 55, g(x)(())))))))) for 2 5, with some
(i.e.,
smoothstep/bump with flat tuils). Assume the encoder + is C². Then the mapping V :00, 1] Red
defines a C trajectory embedding space. Moreover, the discrete sentence embeddings (yt [y))(=1 exactly samples of this trajectory at St- Ne/N:
t=1 are
V1 V (s)),
t=
Proof. For each token Us, we define the masked embedding and positional encoding as
(ze (1),)())) (i) (E(U:), g(sN i) (E(U:), 14).
Since is O and both E(U)) and e, are constant in s, each coordinate pair (za(1), varies smoothly with s. Hence the entire masked sequence
(2,, (zo(1) ..... 24 (N); (L.)....,
is a smooth trajectory with respect to s. The mask M, (i) (m,(1=11) is piecewise constant in and equals the all-ones indicator on indices where sN i ô, and zeros where sN -i -ō; in particular, it is locally constant neighborhoods that avoid the transition band |&N il < 8.
composition of affine maps, matrix multiplications, LayerNorm, residual connections, softmax attention, and smooth pointwise nonlinearities. As function of its inputs, such a network is smooth; thus, on any interval where M, is fixed, the composite map
By assumption,
(8) Φ0 (==, Ma)
Od by the chain rule.
At sentence boundaries g(N; i) for i 2 N₁ + 1. Hence m,, (i) E {0, 1) exactly and Ms, (i) ((SN.). Substituting into the definition,
N₂ N,,N, choose < so that g(N, - -) 1 for i < N₂ and
(sx) + ((E(U)), (UU()) ..... 0,,), (1gNN,) v yt+
which shows that the discrete embeddings (y,) are precisely samples of the continuous trajectory (s).
Remark D.5. Since (.)) implemented with affine maps, matrix multiplications, LayerNorm, residual connections, softmax atlention, and smooth pointwise nonlinearities (e.g., GFLLU/SiLU/SSiihh it's reasonable to assume that is C². If ReLU activations (or other piecewise smooth nonlinearities) are used instead of smooth ones, the mapping V remains continuous and is differentiable almost everywhere. Since this does not affect the manifold-level geometric reasoning, we idealize & as smooth throughout our discussion.
The construction above is merely one possible realization of a continuous and C² trajectory (s). In fact. many alternative constructions are possible. This abundance of realizations justifies our assumption that the sentence T ((Xr), through its be-ep variations, can be viewed as T points lying on a smooth, differentiable curve. On this basis, we can consistently define the notion of flow velocity in Definition 4. 10.
24

---

Published as a conference paper at ICLR 2026
D.2 MENGER CURVATURE
Definition D.6 (Menger Curvature). Let I1, E IR" be three distinct points. The Menger curvature of the triple ( (01,2,,33) Tz) is defined as the reciprocal of the radius 22, Tz) of the unique circle passing through the three points:
02,
R(I1, 22, 1z)
Proposition D.7 (Computation Formula). Let a ||z zall. = || zall, and = |||| x2ll Denote by ( 22 va) the area of the triangle spanned by the three points. Then the circumradins R and the Menger curvature c(I1, T2, Tz) are given by
abe
T2, abc
R(11, F2, = 22,
((11,22,33) =
Proof. The formula follows from classical Euclidean geometry: for a triangle with side lengths a, b, e and area A, the circumradius satisfies R curvature.
abe Taking the reciprocal yields the Menger 13
T2
Center
R
T3
Figure 7: Circumeircle through three points Fı, 22, T3, with radius and Menger eurvature 1/ R.
Proposition D.8 (Menger curvature from three consecutive states), Let 1t-1, 1. Bt Me+1 Rd be three distinct points and set
Vt V1-1:
V++1 Vt-
Write the side lengths
|u]l
b ||0||
||0 u/l Ilvet1 1--1/--
The Menger curvature of the triple (yt-1, Ne, Mt++1) equals
4 A
1, W++1)
2 CosSim(u, v)2
CM (ye- 21+1)
abc.
l1 14-11
where CosSim(u, v) = (u, [|l| [|u]|
(If the three points are collinear, CM :- 0.)
Proof. By classical Euclidean geometry, for triangle with side lengths a, b, c and area A, the circumradius satisfies R abc
4A
The Menger curvature is the reciprocal = 1/R abc
4A
remains to express A in terms of u and v. The (unsigned) area of the triangle spanned by and be written in dimension-independenn way via the Gram determinant:
25

---

Published as a conference paper at ICLR 2026
Center
Center
R
R
(2)
De41
yt-1
1/1
Figure 8: Two circumcircles through {vr-1 Me, w and 13--11 Dr. vitih. with radii R and R'.
Here V241 and y lie on the same ray from yt.
(1)
1
(u, u) (v,
(u, 0)
A
2 u A w/l
del
2 | ||||||||||2 (u, v)
2\
(u, v)
41 abc
Substituting ||u|]. b |||||
|/U
ul
into CM
gives
2. |
|||
(u, v)
cM
||m|| ||la||
|/2
ul
(u. v)
Divide the numerator and denominator by ||u||| ||a|||
and denote = CosSim(u, v)
Then
|ul| ||l||
2 52
2 0
CM 11 21
where 0 is the angle between and (so sin 0 v1 - s²). If the three points are collinear, A and hence Gxg 0, consistent with the convention. This proves the claim.
Remark D.9. As illustrated in Figure using the Menger curvature instead of cosine similarity is significant. Cosine similarity only depends on the angle at yt, so the two triples (yt-1. ye41 and {y--14 Ut. (2) } would look identical. In contrast, their circumradii R and R' are different, hence the Menger curvatures distinguish two different curvature regimes. This demonstrates how Menger curvature captures both angle and length information, enabling discrimination that cosine similarity alone cannot provide.
26

---

Published as a conference paper at ICLR 2026
E DATA GENERATION
We provide exact prompt templates and the representative sampled data instances used in our data generation process. The two-stage pipeline is run with GPT-5.
E.1 PROMPTS FOR DATA GENERATION
The following prompts are used for abstract logical templates construction and Nomain-specifte and language-specific rewriting.
Prompt for Logic Pattern Generation
You are formal logic pattern generator.
Goal: Create an abstract, domain-agnostic reasoning sequence of exactly N steps, written in symbolic form, using standard propositionalfiistorrre logic notation. Strict output format:
Exactly N lines, each line starts with bracketed index and single formula or conclusion, e.g.:
[1] A -> B
[2] B -> C
[3] C -> D
[4] (D & E) ->
[5] forall x (x) -> J(x))
[6] A
[7]
[B] ((a)
[9] (from [1-3] and [6])
[10] F & ((a) (from [4), [7), (5), [8), (91)
V. parentheses, predicate letters with uppercase (A.B,C....) and predicate symbols like H(x), J(x).
Use only symbols from: -, v, ,,
You may include brief justifications at the end of lines in parentheses referencing earlier step indices (e.g.. (from [2] and [5])).
The sequence must internally coherent (later steps can be derived from earlier ones), but no proof of fixed target is required.
No extra commentary before or after the lines. No natural- language sentences.
Parameters (provided by caller):
number of steps to output.
logic: a label for this abstract logic (optional).
= (N) logic {logic}
Now produce exactly N lines.
27

---

Published as a conference paper at ICLR 2026
Prompt for Reasoning Rewriter
You are a reasoning rewriter. Task: Given an abstract N-step reasoning scaffold (formal symbolic lines) and a target topic, rewrite the scaffold into a topic-specific natural-language reasoning sequence with exactly the same number of steps and the same dependency structure.
Inputs (provided by caller):
Topic: the target domain (e.g., weather, software).
Abstract Steps (1. .N): the neutral scaffold, numbered I..N.
N: the total number of steps.
Output requirements:
Produce exactly steps, each line begins with the same bracketed index as the [N]
abstract: [1]
Keep step count and ordering identical to the abstract. Do not merge, split, add, or remove steps.
Preserve the logical dependencies: if abstract step enables k+1, your rewrite must preserve that relationship in the topic.
Use concrete domain terms appropriate to the topic. but keep sentences concise and precise.
No extra commentary hefore after the steps.
Multilingual mode (when Languages: specified by the caller):
Create a separate section for each requested language code.
Each section starts with a header line === <code> === (e.g.. === en ===).
Under each header, write the steps with bracketed indices [1] [N] that language.
Keep the content aligned across languages (same meaning per step index).
Inputs you will receive:
Topic: {topic}
Abstract Steps (1..N): AABTTRACT SSEPS)
= [N}
Now perform the rewrite.
E.2 DATA EXAMPLES
Table presents 9-step logical scaffold from our dataset. We illustrate its instantiation in two distinct domains, weather and finance, providing the corresponding statements in both English (EN) and German (DE).
28

---

Published as a conference paper at ICLR 2026
Table 2: Logic Example (9-Step) with Weather and Finance Topics in English and German
Abstract Logic
Topic:
Weather
Topie: Finance
[1] +
EN: moisture converges over the
EN: If the firm's 's interest coverage ratio exceeds 3.0x, then the firm is deemed able to meet interest obligations. DE: Wenn die Zänsdeckungssennzahh des Untemehmens übber 3,0x liegt, dann gilt das Unternehmen als fähig, Zinszahlungen zu leisten.
city, then thunderclouds develop. DE:
Wenn über der Stadt Feuchte konvergiert, dann bilden sich Gewitterwalken.
[2] +
EN: thunderclouds develop, then heavy rain occurs. DE: Wenn sich Gewitterwolken bilden, dann tritt starker Regen auf.
EN: If the firm deemed able to meet interest obligations, then the bank will approve
term loan.
DE: Wenn das Unternehmen als fähig gilt, Zinszahlungen zu leisten, dann wird die Bank ein neues Laufzeitdarlehen genchmigen.
33 (x(x) +
EN: For any location x, if cold front passes x, then temperatures drop at
EN: For any security is U.S. Treasury, then x is acceptable as repo collateral.
((a)
DE: Für jeden Ort gilt: Wenn eine Kaltfront überquert, dann sinkt dort die Temperatur.
DE: Für jedes Wertpapier gilt: Wenn eine US Staatttannleihe ist, dann x Repo-Sicherheii zulässig.
[4] H(a)
EN: A cold front passing the airport.
EN: Bond A is U.S. Treasury.
DE: Anleihe A ist US-Staatsanleihe.
Eine Kaltfront überquert den Flughafen.
DE:
[5]
EN: Moisture converging over
EN: The firm's interest coverage ratio
city.
exceeds 3.0x.
DE: Über der Stadt herrscht Feuchtekonvergenz.
DE: Die Zünsdeckungskennzzhh des Unternehmens liegt über 3,0x.
[6] B (from [1], (5))
From [1] and [5], thunderclouds develop.
EN:
EN: The firm is deemed ablc to meet
interest obligations (from [1] and [5]).
DE: Aus [1] und [5] folgt, dass sich
DE: Daher gilt das Unternehmen als fähig, Zinszahlungen zu leisten (aus [1] und [5]).
Gewitteroolken bilden.
[7] C' (from [2). [6]
EN: From [2] and [6], heavy rain occurs.
EN: The bank will approve
term loan
(from [2] and [6]).
DE: Aus [2] und [6] folgt, dass starker Regen auftritt.
DE: Daher wird die Bank neues
Laufzeitdarlchen genehmigen (aus [2] und
[6]).
[8] (a) (from [3], [4])
EN: From [3] and [4], temperatures drop at the airport.
EN: Bond A is acceptable as repo collateral (from [3] and [4]).
DE: Aus [3] und [4] folgt, dass Flughafen die Temperatur sinkt.
DE: Daher ist Anleihe A als
Repo-Sicherheit zulässig (aus [3] und [4]).
[9] C'A
EN: From [7] and [8], heavy rain
EN: The bank will approve new term loan
(a) (from [7]. (8)
occurs and temperatures drop the airport.
and Bond A is acceptable repo collateral (from [7] and [8]).
DE: Aus [7] und [8] folgt: Es tritt starker Regen auf und am Flughafen sinkt die Temperatur.
DE: Somit wird Bank neues Laufzeitdarlchen genehmigen und Anleihe A ist als Repo-Sicherheit zulässig (aus [7] und
[8])
29


Reviews:
[{"review_id": "dXEd26rnO0", "reviewer": "Reviewer_p3p6", "summary": "This paper introduces a geometric framework for analyzing reasoning in LLMs. The core idea is to treat reasoning as smooth trajectories (“flows”) in representation space, where logical operations act as differential controllers of the embedding’s local velocity and curvature.", "questions": "see Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the topic of reasoning as continuous geometric flows within embedding space is interesting and the proposed formulation combining mapping and alignement provide a good abstraction linking reasoning to symbolic representation. \n\n- The construction of continuous C^1 trajectories via the relaxed prefix-mask mechanism (Proposition C.4) is both novel and technically precise\n\n- the proposed dataset and empirical results show good results.", "weaknesses": "- the proposition 4.10 that connects logic to the integral of velocity. It is not clear on how it maps between inference rules and specific vector-field constraints on v(s). It is not clear  how logical connectives translate into geometric operations or basis directions in representation space?\n\n- in the mapping A = \\Psi \\circ \\Gamma^{-1}, does\\ Gamma injective? \n\n- how the mapping D_R : \\Psĩ \\mapsto (\\Delta y_t) preserve logical equivalance?\n\n- The logical dataset is generated with GPT-5 templates. How the validation is performed and logical equivalence are guaranteed.\n\n- How empiracly C^1 interpolation is validaded?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZeBleKuSfa", "reviewer": "Reviewer_2CaL", "summary": "The paper introduces a new framework to mathematically model what happens when LLM's reason. \nIt is hypothesized that reasoning traces can be characterized by the geometric properties of the path that the reasoning tokens trace through an LLM's latent space.\nEvidence is presented suggesting that the latent 'reasoning flows' for similar kinds of reasoning (following a certain template) also exhibit similarities on the second and third order geometric properties of velocity and curvature.\nThis evidence is based on studying the given geometric properties for activations of various models when forwarded on a novel dataset where the same logical template is used to construct different chains-of-thought in various domains and languages.", "questions": "What can we learn from the fact that logics B and C are very similar in curvature? Does this suggest that the LLM uses the same mechanisms for both? Is the similarity a surprise to you, would you expect it from looking at the templates?\n\nPresumably, not the entire latent space is relevant for (logical) reasoning. Would you expect there to be a subspace in which the correlation between (the velocity/curvature of) samples of the same logic template is even stronger, perhaps much stronger (close to one)?\n\nWhat are the individual rows/columns within the blocks (L:E through L:A) in Figure 2, are they the different topics and languages?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Interesting new framework for thinking about reasoning in LLMs.\n\nThe concepts that are appealed to are made mathematically precise, and the math is introduced step-by-step in a way that is easy to follow (relative to the complexity of the subject matter).\n\nThe results show that when an LLM reasons according to the same logical pattern/template (but about different topics and in different languages), its activations move through latent space with similar curvature (how small or large is the angle made between triples of subsequent token activation vectors). And, to a lesser extent also with similar velocity (how far apart is each subsequent token activation vector). This is an interesting finding.", "weaknesses": "There is one claim I believe is too strong, on line 429-431: \"Together, these results show that LLMs internalize latent logical structure beyond surface form. They are not mere stochastic parrots (Bender et al., 2021): whereas humans formalized logic only in the 20th century (Bochenski & Thomas, 1961), LLMs acquire it emergently from large-scale data—a hallmark of genuine intelligence.\"  I'm not sure these results really provide much evidence against the 'stochastic parrot theory'. The results, as I understand them, show that the same logical structure elicits similar trajectories (in the second- and third-order geometric sense) through latent space. But the logic is not just similar, it is identical. Wouldn't we need to show that the LLM correctly identifies the (vast majority of) instances as instantiations of the logical problems (in an exact, not fuzzy sense).  And as for the second part, the experiments use Qwen models, which are not merely pre-trained on large-scale data, but also mid- and post- trained, right? so the observations might be due to those training stages, which quite likely involve supervised/reinforcement training on logic problems. Finally, the juxtaposition between discovering-from-data and human-discovery, seems mistaken. Even if the model 'discovers' it from the data, that is likely due to the human discovery being described/present in the data.\n\nIf this is addressed, I'm happy to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k85qBZkSCW", "reviewer": "Reviewer_umD2", "summary": "This paper proposes a framework to analyze reasoning in large language models by viewing a chain of thought as a trajectory in representation space. The trajectory is described with three geometric descriptors: position, velocity, and curvature. Using this framework, the authors analyze reasoning traces in Qwen3 and LLaMA3 models. They synthetically generate multi-step logical sequences that share the same logical skeleton while varying topic and language. Applying their analysis, they find that velocity and curvature are more similar for traces that follow the same logic and remain low across different logics even when topic and language match. By contrast, position is more similar for inputs written in the same language. The authors interpret this as evidence that models encode logical structure in the dynamics of their representations, while static position reflects surface form.", "questions": "1. How should similarity patterns for position, velocity, and curvature change with model size? Table 1 shows no clear trend across Qwen3 0.6B, 1.7B, 4B and LLaMA3 8B. Is the absence of a trend expected? \n2. How would the results differ if trajectories were computed on self-generated traces rather than sequences authored by a much larger model? \n3. Do better-performing models exhibit stronger disentanglement between logic-driven dynamics (velocity, curvature) and surface-driven position? Can you correlate the geometric measures with task accuracy and repeat the analysis on self-generated traces to test this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper Introduces a relatively novel geometric perspective on reasoning, modeling chains of thought as trajectories in representation space.\n- The authors provide a clear formalization of curvature and reasonable metrics to estimate it.\n- The paper includes convincing evidence that first- and second-order properties (velocity, curvature) track logic, while position reflects surface form or semantics.", "weaknesses": "1. Lack of scaling analysis. The paper does not analyze how similarity patterns should evolve with model size. Table 1 shows no clear trend across Qwen3 0.6B, 1.7B, 4B and LLaMA3 8B. Position, velocity, and curvature similarities fluctuate rather than improving systematically with scale. The authors should discuss whether this is expected and what theory or diagnostics would predict size effects.\n2. Traces are not self-generated by the evaluated models. The reasoning sequences are produced by a much larger model and then fed to smaller models for analysis. This can introduce a distribution shift and may mask effects that would appear if each model were analyzed on its own traces. The paper should discuss how results might change under self-generated or mixed setups.\n3. Limited link to performance and disentanglement. Related to point 2, the paper does not examine whether models with stronger reasoning performance exhibit a clearer disentanglement between logic-driven dynamics (velocity, curvature) and surface-driven position. A mild suggestion is to correlate the geometric measures with task accuracy and to repeat the analysis on self-generated traces to see if better-performing models show stronger disentanglement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ili80Ib40I", "reviewer": "Reviewer_xzaB", "summary": "The paper analyzes “reasoning” in LLMs, particularly the reasoning flows (trajectories) in representation space. \nLogic is posited to act as a local controller of velocity along these flows, with Menger curvature used to capture second-order structure. \nThe authors construct a dataset that keeps formal logical skeletons fixed while varying topical and linguistic “carriers”, then extract hidden states from Qwen3/LLaMA3 to compare position, first-order/second-order differences and similarities. They report that while positions cluster by surface semantics, velocity/curvature similarities align by logic, supporting the claim that logic governs flow dynamics.", "questions": "- It is not clear that the dataset isolates formal logic from its semantic context. There is a lack of systematic analysis, e.g., statistical analysis, to demonstrate the separation. In other words, how to ensure the disentanglement of the logic format and the semantic context?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- An interesting work on reasoning, the authors view logic as a carrier-invariant framework for reasoning, and they test if LLMs have learned these structural invariants in their own embedding space.\n\n- Empirical comparisons across model sizes support qualitative claims.\n\n- Some interesting findings, for example, in the representation space, sentences on the same topic cluster together. However, when looking at the differences of curvature, logical structure emerges as the dominant factor even across unrelated topics and languages.", "weaknesses": "- The paper is not easy to follow. Many key notions are introduced before being properly defined or motivated, which makes it difficult for readers outside the narrow intersection of geometry and logic to follow the argument. For example, the term “flow” appears early (Abstract, intro) for many times, but its formalization as a sequence of hidden states, or its mapping to logical entailment steps, is only explained several pages later (Sec. 4.2). Similarly, “menger curvature” is used as a core analytic measure before the geometric intuition or connection to reasoning trajectories is established. Also, it is often mentioned without details: what the Menger curvature is, how it is calculated, or at least some intuition should be provided. A short schematic or example trajectory would help. Recent ICLR/ICML papers have handled similar conceptual density more clearly, see, for instance, “The Geometry of Categorical and Hierarchical Concepts in LLMs” (Park et al., ICLR 2025) and “Tracing the Representation Geometry of Language Models from Pretraining to Post-training” (Zhang et al., ICML 2025).\n\n- The paper cites several strands but does not sufficiently contrast contributions with recent, closely related geometry/trajectory work at ICLR/ICML.\n\n- Its evidential basis is correlational and methodologically fragile in its current form. For example, the smoothness hypothesis needs independent validation. Hyp. 4.6 is asserted with a construction in App. C.1, but the fitted smooth curve could be an artifact. Maybe consider reporting results on shuffled/phase-randomized controls as baseline. \nMaybe also consider adding causal tests, stronger statistical treatment, external benchmark dataset validation, and clearer positioning/discussion vs. recent ICLR/ICML work; this submission could become a citable contribution to the geometry-of-reasoning literature. But I'm not sure if it can or will be done in the CR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)