Bilinear representation mitigates reversal curse and enables consistent model editing
Authors: Dong-Kyum Kim
Minsung Kim
Jea Kwon
Nakyeong Yang
Meeyoung Cha
Venue: ICLR 2026 Poster (2026)
Keywords: model editing
reversal curse
language model
relational knowledge
knowledge editing
Primary area: interpretability and explainable AI
TL;DR: Language models can learn to encode relational knowledge in a bilinear relational structure, a mechanism that mitigates the reversal curse and enable cosistent model editing.

Abstract: The reversal curse—a language model's inability to infer an unseen fact "B is A" from a learned fact "A is B"—is widely considered a fundamental limitation. We show that this is not an inherent failure but an artifact of how models encode knowledge. Our results demonstrate that training from scratch on synthetic relational knowledge graphs leads to the emergence of a bilinear relational structure within the models' hidden representations. This structure alleviates the reversal curse and facilitates inference of unseen reverse facts. Crucially, this bilinear geometry is foundational for consistent model editing: updates to a single fact propagate correctly to its reverse and logically dependent relations. In contrast, models lacking this representation suffer from the reversal curse and fail to generalize model edits, leading to logical inconsistencies. Our results establish that training on a relational knowledge dataset induces the emergence of bilinear internal representations, which in turn support language models in behaving in a logically consistent manner after editing. This suggests that the efficacy of language model editing depends not only on the choice of algorithm but on the underlying representational geometry of the knowledge itself.

Published as a conference paper at ICLR 2026
BILINEAR REPRESENTATION MITIGATES REVERSAL CURSE AND ENABLES CONSISTENT MODEL EDITING
Dong-Kyum Kim MPI-SP, Germany
Minsung Kim?
Jea Kwon'
Nakyeong Yang!
Meeyoung Cha
2 SNU, South Korea
*KAIST, South Korea
ABSTRACT
The reversal curse language model's inability to infer an unseen fact "B is A" from a learned fact "A is B"-is widely considered a fundamental limitation. We show that this is not inherent failure but an artifact of how models encode knowledge. Our results demonstrate that training from scratch on synthetie rela- tional knowledge graphs leads to the emergence of a bilinear relational structure within the models' hidden representations. This structure alleviates the reversal curse and facilitates inference of unseen reverse facts. Crucially, this bilinear geometry is foundational for consistent model editing: updates to single fact propagate correctly its reverse and logically dependent relations. In contrast, models lacking this representation suffer from the reversal curse and fail to gen- eralize model edits, leading to logical inconsistencies. Our results establish that training relational knowledge dataset induces the emergence of bilincar inter- nal representations, which in turn support language models in behaving in a logi- cally consistent manner after editing. This suggests that the efficacy of language model editing depends not only on the choice of algorithm but on the underlying representational geometry of the knowledge itself.
INTRODUCTION
Language models have become powerful tools for knowledge-intensive tasks, yet their reasoning capabilities often fall short of human-level logical consistency (Berglund et al. 2024 Allen-Zhu
& Li 2025). A prominent example is the reversal curse: a model trained on "A is the parent of B frequently fails to infer the reverse fact, "B is the child of A." This failure suggests that models learn shallow, directional associations rather than robust, symmetrical relationships required for reliable inference. This limitation is particularly acute in model editing, which seeks update factual knowledge in trained model without costly retraining from scratch (De Cao et al.) [02T] Meng et al. 2022). An ideal edit must be logically coherent; an update to the primary relation (for example, "A is the spouse of C" to "A is the spouse of D)") should naturally propagate to its logical corollaries ("D the spouse of A" and "B is the child of D"))
However, existing architectures struggle with such logical generalization. Current model editing methods fail to propagate updates to entailed facts, requiring the explicit co-editing of both direc- tions of relationship to avoid the reversal curse (Thibodeau) 2022; Yao et al., 2023 Hase et al.) 2024). This limitation raises a critical question: are these failures an inherent flaw the Trans- former architccture, have focused on the limitations of autoregressive objectives (Zhu et al.] 2024, Kitouni et al.) 2024). emerging evidence indicates that these reasoning failures are tied the specific geometry of learned representations. Language models have been shown to learn meaningful geometric structures for features like space and time (Gurnee & Tegmark 2024 Engels et al.) (025). Furthermore, disrupt- ing this underlying topology during editing correlates with failures in reasoning (Nishi et al. (025)) While these studies highlight that structure matters, the specific algebraic mechanism required to support symmetric and compositional reasoning remains an open question.
they an artifact of how models represent knowledge? While prior efforts
This work proposes that logical consistency in language models depends on the emergence of bilinear relational structure. We investigate how relational knowledge is encoded using three repre-
"Coreesooning author: Email: mia.cha mpissoooooroo

---

Published as a conference paper at ICLR 2026
Tracy
Lance
Lancr
Strth
Smith
son
Cary
Cany
Latta
Senin
OL wi + by
01
= 81 + Up
[r (1) M,
(a) Lineur Relational Lmbbedding
(b) Translational Embedding
(c) Bilinear Relational Embedding
Figure 1: Schematics of relational embedding structures. For a subject and object o, the relation is modeled as: (a) linear transformation, (b) vector translation, or (c) bilinear interaction mediated by relation-specifie matrix M,. As an example, in the fact "Son of Tracy Lance Smith Cory Lance Smith," is "Tracy Lance Smith," o is "Cory Lance Smith," and r is son.
sentation probes: linear, translational, and bilinear probes (see Figure D. Recent work has demon- strated that knowledge decoding from a pretrained language model's internal representation can be approximated by either linear mapping (Hernandez et al.) 2024) or translational mapping (Merullo al. 2024). While these frameworks offer valuable insights, they cannot capture symmetric and compositional relations essential for robust logical reasoning. We instead focus bilinear relational models such as RESCAL (Nickel et al. 201T) knowledge graph embedding literature, which rep- resent relations as matrices that mediate interactions between entities. Bilinear relational structures naturally accommodate inverse relations via matrix transposition and compose relations via matrix multiplication, providing rich algebraic framework for reasoning.
To investigate this, we train decoder-only Transformers from scratch on a synthetic knowledge graph, allowing precisely control the learning environment and probe the resulting internal structures. We make several contributions:
Mitigate the reversal curse via relational knowledge learning: We demonstrate that under appropriate regularization (weight decay), language models overcome the reversal curse by learning a robust bilinear relational structure, achieving near-perfect accuracy on unseen reverse relations (Figure right).
Identify bilinear structure in hidden representations: Using three distinct probes, find that a bilinear probe best explains hidden representationss with the highest precision emerging in the middle layers (Figure Furthermore, the learned relation matrices satisfy formal algebraic tests for composition and inversion (Figure 4h, validating the presence of functional bilinear structure in language models.
Link representational geometry to model editing: We establish mechanistic link be- tween representational geometry and editing consistency. Models possessing a bilinear structure successfully propagate edits logically related facts, whereas those lacking fail to generalize, even when the direct edit is successful (Figure
Building on these findings, our study introduces a substantial change in perspective: the key to resolving logical failures lies not solely in the model architecture editing algorithm, but funda- mentally in the geometric structure of learned knowledge representations. This result suggests the pivotal role of relational encoding in shaping the reliability and robustness of model behavior.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Reversal Curse. The reversal curse-the failure to infer "B is A" from "A is B""-as been iden- tified as a fundamental limitation of language models (Berglund et al. attribute this to the directional nature of the autoregressive training objective, which preferentially models P(B|A) [A) but not B)) (Allen-Zhs Zhu & Li 2025; Zhu et al. 2024 Kitouni et al. 2024). The nature of this failure is nuanced; Lin et al. (2024) suggest that it may be issue of knowledge retrieval rather than storage, as models that fail open-ended generation can succeed on multiple- choice questions where the answer is present in a prompt. Proposed solutions often directly target the training process. These include data augmentation via subword-level reordering in sentence (Golovneva al., 2024), generating reversed examples by language models (Lampinen et al. 2025), or modifying the training objective to be irrection-agnoste (Kitouni et al.) 2024). Although these studies offer practical mitigation, they often treat as an unavoidable artifact of the training objec- tive. Our work offers a different perspective by focusing on the model's ' internal representations.
2024 Prevailing hypotheses
Model Editing and Logical Generalization. Model editing aims update facts in language mod- els without costly retraining (De Cao et al. 2021). Various model editing methods have been pro- posed, such ROME (Meng al.] 2022) and MEMIT (Meng et al., 2023), which show better generalization than naive fine-tuning, but all editing algorithms suffer from the reversal curse (Thi- bodeau 2022 Yao et al. 2023). This points to a deeper conceptual challenge. |Hase al.) (2024) provide a comprehensive critique, framing model editing as an instance of belief revision, for which no simple solution exists. They argue that logical generalization is not just a desirable feature but a core requirement for any successful editing paradigm, and they demonstrate empirically that current methods often fail to produce coherent belief updates. Recently, Nishi et al. (2025) demonstrate that editing can break the underlying topological structure of the learned knowledge. This suggests that the brittleness of model editing is tied to the preservation of internal geometry. In our work, we investigate the prerequisite internal algebraic structures that enable logical consistency after model editing.
Mathematical Structures for Relational Knowledge in Language Models. A substantial body of research has investigated where language models store factual knowledge (Geva al.) [02T] Meng et al., 2022 Pan et al. 2025). The mechanism for retrieving specific facts is complex and distributed across multiple layers and attention heads, as revealed by studies of interventions (Hase) al., 2023) and attention mechanisms (Geva et al.) 2023). In contrast, far fewer studies exam- ine which mathematical structure models employ to resolve relation. Recent investigations sug- gest that the underlying structures unexpectedly simple: [ermandezz al. (2024) demonstrate that language models implicitly implement Linear Relational Embeddings (Paccanaro & Hinton, 2002). while Merullo al.] (2024) show that they exploit translational structures familiar from Word2Vec (Mikolov et al.] 2013). However, the multi-head attention mechanism is fundamentally built on more expressive bilinear operations (Flhage al.) (02T)) and individual heads can encode specific relational operations (Elhelo & Geval 2025). ). Furthermore, such bilinear models have a long history of success in modeling relational data, particularly in knowledge graph embedding methods like RESCAL (Nickel et al.) 2011). Despite bilinear operations being central to Transformer archi- tecture and successful in relational learning. little work has investigated whether language models exploit bilinear structures for decoding relational knowledge. This paper presents a systematic anal- ysis of the bilinear structures underlying relational knowledge.
3 RELATIONAL KNOWLEDGE DATASET AND LANGUAGE MODELS
In order to test our hypothesis, we create a synthetic relational knowledge dataset and train multiple language models with different hyperparameters from scratch on it.
Synthetic Knowledge Graph and Task. We generate dataset from synthetic family knowl- edge graph (see Figure 2 left). The graph consists entities (family members) and eight rela- tions: husband, wifer father, mother, son, daughter, brother, and sister. We choose this domain because family relations form a minimal, closed-world system that exhibits in- verses (husband of wi fe is husband) and composition/multi-nng structure (e.g., husband of
3

---

Published as a conference paper at ICLR 2026
1.0
R==0.816
(mother/facher)
A
B
E
F
0.8
0.6
C
D
G
H
father/mother
husband/wife brother/sister
son/daughter
J
0.2
Reversal Cursed
Not Reversal Cursed
Weight Decay
Figure 2: (Left)) Schematic of the synthetic family knowledge graph. Nodes represent entities, while directed edges represent one of eight distinct family relations used in our experimental evaluation. (Right) Test accuracy on the unseen relations mmother/fattin as function of weight decay. Each weight decay setting was trained using three different random seeds.
mother is father, sister of son is daughtez) under clear type constraints; the eight re- lations the smallest set that jointly spans these algebraic properties, making the setup ideal for testing reversal and logical generalization.
Each entity is assigned full name following the format "[First Name] [Middle Name] [Last Name]". All entities of family share a common family name, defined as "[Middle Name] [Last Name]". In this work, denote fact (s, r, o) where is the subject entity, is the relation, and is the object entity: this means r of s is o. Each fact is represented Name] [Family Name] [Relation] [Object First Name] [Family Name]". For example "Emily Scott Wall husband Julian Scott Wall" where "Scott Wall" is the family name.
plain text sentence; "[Subject First
The dataset comprises 1,000 families, each with 10 entities, resulting 36 distinet relational facts family. These families are divided into two groups of 500 to create the training set.
Group (Full relations): For the first 500 families, this group contains all 36 facts.
Group (Missing relations): For the second 500 families, we withhold the father and mother relations. Therefore, this group contains only the remaining 24 facts.
The training set consists of all facts from both groups (approximately 318M tokens). The test set is constructed exclusively from the withheld facts of Group 2, containing 12 relations (father/mother) per family. The task is to predict these unseen relations in Group 2 by learning logical dependencies between entities from Group If the model learned relational structure in the dataset, then infer these missing relations by using logical reasoning, e.g. the test rela- tion (C, father, B) can be inferred by using two facts (A, husband, B) and (B, son, C). Full details of synthetic dataset provided in Appendix B
Language Models. We train decoder-only Transformers using the GPT-NeoX architecture (An- donian et al.) 2023) from scratch on the synthetic dataset. Each model has 12 layers, a hidden size of 896, and 16 attention heads, totaling approximately 206M parameters. We employ this architecture the following sections. More details of architecture and training are provided in the Appendix A
EXPERIMENTS
Our experiments seek to uncover the internal mechanisms that enable language models to perform logical reasoning. We first demonstrate that even when the training data contain sufficient informa- tion to infer reverse relations, models only overcome the reversal curse when guided by appropriate regularization (Experiment 1). We then pivot to the central question of our work: which mathe- matical structure enables this success? Through a series of probing experiments, we uncover an emergent bilinear relational structure in the successful models (Experiment 2) and verify its alge-
4

---

Published as a conference paper at ICLR 2026
braic properties (Experiment 3). Finally, we demonstrate that this geometry plays central role in addressing model editing challenges. We show that the bilinear structure is a key to ensuring that edits propagate in logically consistent manner, establishing unified explanation for both the reversal curse and editing generalization failures (Experiment 4).
4.1 EXPERIMENT 1: TRAINING LANGUAGE MODELS ON RELATIONAL KNOWLEDGE DATASET
We test whether a language model can learn to infer withheld relations (father/mcther by observing their reverse counterparts (e.g., son/daughter and compositional examples elsewhere in the training set. We use AdamW (Loshchilov & Hutter 2019) with learning rate of 3 10-4 and sweep weight decay over {0, 0.,, 1.0, 2.0, 3.0, 4.0, 6.0,, training three random seeds per setting (27 models in total).
The results, shown in Figure (right), reveal a striking dependency. All models achieve 100% training accuracy (see Appendix and Figure [). but test accuracy varies significantly based on weight decay and random seed. Without sufficient regularization, models consistently fail, showing the reversal curse with low accuracy (weight decay 1.0). However, as weight decay increased, split in outcomes emerges: some models remain "reversal cursed," while others break the curse and achieve near-perfect accuracy on the unseen reverse facts. This transition indicates that the reversal curse is not an inherent limitation but an artifact of an under-constrained training objective, where regularization promotes a more generalizable internal structure over simple memorization.
This finding motivates subsequent experiments. To understand the mechanisms distinguishing these two outcomes, we select two representative models for in-depth analysis: a "Reversal Cursed" model with low reverse accuracy (< 40%) and "Not Reversal Cursed" model with high reverse accuracy (> 98%). In all subsequent figures, the former is indicated by blue, and the latter by orange.
4.2 EXPERIMENT 2: PROBING INTERNAL REPRESENTATION FOR RELATIONAL STRUCTURES
To identify which relational geometry the models have learned, conduct a probing analysis. The process begins by extracting entity representations for each fact (s, r, o). Specifically, we take the hidden states for subject (si) and object (ot) from layer 1 the final token of their names, resulting in vectors in 8d where dimension d = 896. Here, r denotes a relation. Using these representations, we train three different probes test our structural hypotheses (Figure T. The training set consists of facts from 125 families (1,250 entities) from Group 1, and the evaluation is performed on held set of facts from 125 different families, also from Group 1. The resulting classification accuracy indicates the degree to which given relational structure is present at each layer.
Linear Relational Embedding. This probe tests for linear relational structure Paacaaaao & Hinton 2002) in language models. Hernandez et al.] (2024) model a relation r as the local affine transformation that the Transformer applies map subject representation 81 from layer to the object's pre-prediction representation oz. in the final layer L. This yields the approximation or. RV W,st + b,, where 7g the hidden state at the position immediately preceding the object token (see Figure Ta).
The parameters 6,) are not learned but are extracted directly from the model's forward pass. The matrix W, is estimated from the Jacobian J, /0st, averaged over training examples. As the raw Jacobian can underestimate the transformation's 's magnitude, we scale by a hyperpa- rameter B, chosen via a sweep over [1.0, 1.5, 2.0, ... 5.03. The bias by is then computed as the mean residual. The parameters are thus set as:
3 n W,
n
6y
0
J (1)
(I)
71 1=1
We estimate these parameters by averaging a small sample of n 10 training examples per relation. 7 is chosen from a sweep over (10, 100, 500) (See Appendix D.2.
Translational. This probe, which tests for the kind of vector arithmetic investigated in hidden representations by Merullo et al. (2024), models relation r as simple vector offset. For each
5

---

Published as a conference paper at ICLR 2026
Linear Relational Embedding
Translational
Bilinear
Not Reversal Cursed
Accuracy
Layer 10
12
2
10
12
10
12
Layer
Layer
Figure 3: Liser-wise probe accuracy for relational embeddings. We compare "Reversal Cursed" models (blue) against "Not Reversal Cursed" models (orange), averaged across all relations. Aceu- racy is reported for each layer to identify where relational knowledge emerges.
relation r, we fit translation vector u, such that 81 + Up Ri ot. Unlike the linear relation embedding, is taken from the same layer as (see Figure [D)) at the last token position of the object name.
The vector v, is computed as the average displacement across all n training facts for that relation:
V/-
1
Σ 10
(
(2)
This structure suggests that all entities participating a relation are shifted by a constant vector in the embedding space.
This probe tests for a bilinear structure, where a relation is modeled as a matrix M, that mediates the interaction between the subject and object embeddings (see Figure Te). For each relation r, we define a score function:
Bilinear.
(1)) is Mo
between any pair of entities given the relation where matrix M, E Rdxd, The target function is:
(s, = { when (s, o) is true (exists in the dataset).
(4)
when (s, is false (does not exist).
We estimate the relation matrices using a ridge regression variant of the RESCAL algorithm Nickel al. 201T), which optimizes the following objective function:
L(M,)
1 Xr AMATTI + An 1 2
A = [s] 18
(n)
Rd
(5)
2
where A Raxd is the matrix of trainset entity embeddings, X, Raxn is the adjacency matrix for relation r (with entries corresponding to fr), and ÅR is a regularization parameter. Here, n is the total number of entities (n 1,250). Due to the high dimensionality of embeddings (d = 896), we use an SVD-based optimization approach to make the computation tractable (detailed in Appendix C). We sweep the regularization parameter AR over logarithmic scale from 1 to 10 to find the optimal function for each layer / and relation r.
Results. For each layer I and relation r, we trained a separate probe and evaluated its accuracy on test set entities (Figure 33.. The gray dashed line at 1/3 marks the chance-level baseline. In our family graph (see Figure 2 left), for any relation r there are only three candidates per family that satisfy (s, 1. A probe that identifies the family name and the relation but fails to use the subject embedding would therefore guess uniformly among these three candidates, yielding an expected accuracy of 1/3. For example, given the text "[Subject] Scott Wall husband [Object]", probe that ignores the subject can narrow the object to the three husband candidates in the "Scott Wall" family and would be correct one out of three times average.
Against this baseline, the "Not Reversal Cursed" model (orange) develops a strong localized bilin- structure: the precision of the bilinear probe rises sharply in the middle layers (6-9), peaking

---

Published as a conference paper at ICLR 2026
Muustans Mmother-tii
Mare Myyther -ttther
Mygzer Mane-Moonndduhhtee
Myrohhr Maarphir "Son
Reverical Cureed
Composition ACC.
Navend
6.-
04-
M husbard PW1 fi
M.
Wie = husband
M. diter bbother
M.
brother 951 ster
Transpose Acc.
02
0.-
Layer
Layer
Layer
Layer
Figure 4: Performance on formal algebraic tests. Top Row (Composition): Accuracy of inferring
a composed relation using the product of the corresponding bilinear matrices (e.g., Mhushand Mmother to probe for 'father'). Bottom Row (Transpose Inversion): Accuracy of inferring inverse relation using the transpose of a bilinear matrix (e.g., pusband to probe for 'wife')
above 95%, while the linear and translational probes hover near or below the baseline. In contrast, the "Reversal Cursed" model (blue) shows coherent relational geometry; all probes remain low and often near the 1/3 line across layers. Per-relation results (see Appendix D show: (1) transla- tional structure confined the symmetric relation husband/ fe; (2) some "Reversal Cursed" models exhibit weak, relation-isolated linear relational mappings; (3) only "Not Reversal Cursed" models express consistent high-fidelity bilinear pattern across all eight relations. These results indicate that the emergence of a bilinear representation is key mechanism that enables the model overcome the reversal curse.
4.3 EXPERIMENT 3: RELATIONAL ALGEBRA TESTS
Having established a strong bilinear signal in Experiment 2, we whether the trained relation matrices M, obey basic algebraic laws that enable inverse and multi-hop reasoning. Using the ma- trices estimated by the bilinear probe in each layer, we evaluate two relational reasoning operations with the same scoring function f, and evaluation protocol as the bilinear probe: (1) Transpose or Inversion: does M act like the inverse relation --17 !? (2) Composition: does the product M,, My, act like the composed relation r2
We instantiate four compositions matching our dataset (Figure 2 left): Muushand Mmother + father, Mwite Mrater mother, Maiste Maon + daughter, Mbrother M daughter son. For inversion, we test the pairs M TT hushand = wife, M Mife- husband, M. - brotherr M sister.
Results. FigureAshows showss that the "Not Reversal Cursed" model (orange) achieves high accuracy in both composition and transpose tests, with peaks aligned the same middle layers (6-9) where the bilinear probe is strongest. The "Reversal Cursed" model (blue) remains low in all layers. These results indicate that the learned bilinear representation is not merely predictive but algebraically structured: transposes approximate inverse relations and matrix products approximate composed relations, enabling multi-hop inference.
4.4 EXPERIMENT 4: MODEL EDITING AND ITS LINK TO BILINEAR STRUCTURE
Model editing and evaluation. We edit maasbaddrelationn fact (A, husband, B) and evaluate its effect on entailed knowledge. We conducted 50 editing experiments per model, each editing single fact (A, husband, B) from Group to (A, husband, B'), where B' has the same family name but different first name than B. We perform edits using straightforward yet effective layer-wise fine-

---

Published as a conference paper at ICLR 2026
Edit success
Logical Generalization
Locality
a
Accuracy
(A, husband, B)
B
(A, husband, B*)
20-
A
B'
Target Editing Layer
Target Editing Layer
Target Editing Layer
Best Logical Generalization
(B', wife,
R²=0.939
0.
Logica Generalization
0.8
0.8
Accuraicy
(a
C
D
06-
(B', daughter/son, C/D)
04--
0.2
reverse ACC.
Logica Genera Ization Biliner
0.5
0.0
02
04
0. 10
10
Best Bi inear Acc.
Layer
Figure 5: Editing generalization and its link to bilinear structure. (a) Schematic of the editing task. The primary fact (A, husband, B) edited (A, husband, B'). Successful logical generaliza- tion requires updating the inverse (B', wife, A) and neighborhood relations (C/D, father, B'); (B', daughter/son, C/D). (b) Post-edit performance metries: Edit Success (direct change), Logical Gen- eralization (propagation to entailed facts), and Locality (impact on unrelated facts). (c) Correlation between a model's peak bilinear probe accuracy and its maximum logical generalization perfor- mance (R² = 0.939). (d) Layer-wise performance of bilinear probing and logical generalization for "Not Reversal Cursed" models.
tuning that minimizes cross-entropy loss on the new fact (Zhu et al. 2020 Wang et al.) 2024). We the Adam optimizer with learning rate of 4 10 and apply early stopping once the loss drops below 0.2 and restrict gradient updates to the MLP block's 's output layer. We apply this to cach layer 1 (1, (1. 12), yielding 12 edited models per original model.
We evaluated edited models on three metrics (Figure (1) edit success, which measures whether the model correctly predicts the primary update (A, husband, B'); (2) logical generalization, de- fined as the average success rate on the entailed facts: (B', wife, A), (C/D, father, B'), and (B', daughter/son, C/D); and (3) locality, which assesses whether unrelated facts like (C, brother, D) remain unchanged.
Results. As illustrated in Figure Sp. both "Reversal Cursed" and "Not Reversal Cursed" models achieve near- eerr-eeffect accuracy the primary edit. However, their capacities for logical propagation diverge sharply: models without the reversal curse exhibit robust generalization to entailed facts, whereas reversal-cursed models fail almost entirely. Furthermore, "Not Reversal Cursed" models demonstrate superior locality, preserving the integrity of unrelated factual knowledge more effec- tively than their counterparts.
To quantify the relationship between internal structure and editing performance, we correlate each model's best bilinear probe accuracy (across all layers from Experiment 2) with its best logi- cal generalization after editing (across all layers). Figure 5 reveals robust positive correlation (R² 0.939), establishing that a well-structured bilinear representation predicts successful logi- cal propagation of single edit.
Interestingly, the optimal layers for editing do not align perfectly with the layers where the bilinear structure is strongest. Figure [5 reveals that for "Not Reversal Cursed" models, logical gener- alization is highest when editing early-to-mid layers (I-4), whereas the bilinear structure is most prominent in middle layers (6-9). This suggests that to effectively edit an entity, one must intervene at the layers where the structured representation is being formed, rather than at the layers where is already fully established and utilized. Modifying these earlier layers appears to correctly update the downstream representation, enabling the desired logical propagation. These results provide strong evidence for central claim: bilinear representation key to logically consistent model editing.
8

---

Published as a conference paper at ICLR 2026
5 DISCUSSION
Our findings establish a clear mechanistic link between relational structures in training data, emer- gent representational geometry, and logical generalization. These results have significant implica- tions for how we understand, build, and control language models.
Is the Model "Ready" to be Edited? Our results reframe the fundamental chal- lenge of model editing. Much of the current research focuses on developing more sophisticated editing algorithms. treating the model as static object be operated upon. We show that the suc- cess of any edit is fundamentally constrained by the model's pre-existing representational geometry: an editor cannot force logically entailed update if the underlying architecture lacks the necessary algebraic structure required to represent that entailment.
Model Editing:
However, the existence of structure alone is not enough; the editing method must also preserve this structural integrity. While we show that fine-tuning can successfully propagate edits when bilinear structure is present, recent evidence suggests that aggressive editing methods can "shatter" the underlying graph topology (Nishi et al. (025). Thus, while bilinear structure is a key indicator of logical generalization, realizing this potential requires editing algorithms that respect- rather than destroy- the model's internal algebraic integrity. This suggests a paradigm shift: before editing, one must first assess whether a model is "editable" in a logically consistent way. This leads to a two- pronged approach for future research: 1) the development of geometry-aware editing algorithms that leverage internal relational structures, and 2) "architectural preparation" strategies that explicitly endow models with structured knowledge representations during pre- pre-traiming training or fine-tuning.
From Memorization to Reasoning: The Role of Representation. This work suggests that the perceived gap between a model's ability to memorize and its ability to reason may be a function of its internal knowledge structure. Phenomena like the reversal curse can be viewed quences of optimizing models to exploit statistical shorteuts (e.g., term co-occurrence in the training data) rather than to develop capabilities that support logical understanding, such as latent multi- hop reasoning (Yang et al., 2024 2025a; Balesni et al. 2025). Our demonstration that a structured knowledge dataset with appropriate regularization can induce transition to algebraic, bilinear structure implies that Transformers are capable of learning more than just directional associations. This raises crucial question for the field: Are we explicitly training models reason, hoping reasoning emerges as a side effect of scaling? Our results suggest that actively guiding the formation of structured representations, perhaps through curriculum learning, contrastive objectives, or integration with knowledge graphs during pretraining (Yang et al. 2025b). could be more direct path to building genuinely logical AI.
conse-
The Attention Head Hypothesis. While our analysis relies on readout probes and algebraic validation, we hypothesize that the "bilinear structure" we detect cor- responds to interaction patterns implemented within the self-attention layers. Because attention computes interactions between pairs of tokens through multiplicative comparisons, is inherently bilinear (e.g., terms of the form x Ay). It is therefore plausible that particular attention heads are sponsible for encoding these relational structures. Recently, Elhelo & Geval (2025) have shown that specific attention heads encode relational lookups and tend to appear in mid to late layers, aligning with our observation that bilinear structure emerges in those layers. This suggests that the absence of such specialized attention heads may be bottleneck in reversal-cursed models. Future work could test this hypothesis by intervening on attention heads whose behavior aligns with our estimated M, matrices.
Mechanistic Interpretation:
Limitations and Future Work. Our primary limitation is the use of language models trained from scratch on a clean, synthetic dataset. This raises the crucial question of whether these find- ings scale to large, pre-trained language models and the noisy, complex knowledge they contain. Whether similar bilinear structures exist in industrial-scale pre-trained language models remains an open question; it is unlikely that all knowledge is encoded via a single, uniform geometry. Instead, different domains of knowledge may adopt different relational structures in their representations. Our work is proof of concept, demonstrating that language models are capable of forming this algebraically robust structure, although we have not verified its prevalence in existing large-scale models, A critical direction for future work, therefore, is to develop methodologies to diagnose how

---

Published as a conference paper at ICLR 2026
pre- pre-trained model decodes specific piece of information. Such diagnostic capability would be transformative, enabling a new paradigm of structure- model editing. By first identifying a fact's local representational geometry, and leverage that structure, moving the field from a trial-and-error process to a more principled, geometrically informed science of knowledge modification.
could then select or design editing techniques that respect
CONCLUSION
We demonstrated that the reversal curse and failures in model editing generalization not inherent limitations of Transformers, but rather symptoms of an unstructured internal knowledge representa- tion. By training Transformers showed that they can learn a robust bilinear structure for relational knowledge. Probing experiments confirmed that this structure emerges in the middle layers and is algebraically sound, supporting re- lational inversion and composition. Crucially, we established a strong link between the presence of this bilinear representation and the model's ability to both overcome the reversal curse and perform logically consistent model editing. When fact was edited, models with this structure successfully propagated the change to entailed facts, whereas models without it failed to generalize. Our find- ings highlight that the path toward more reliable and editable language models lies not just in better algorithms but in shaping the fundamental geometry of their learned knowledge representations.
synthetic knowledge graph with appropriate regularization,
REPRODUCIBILITY STATEMENT
We include complete details about our model architecture, training setup, and hyperparameter sweeps in Appendix A Our synthetic dataset construction is described in Appendix B We ran all experiments on a workstation with 4 A100 GPUs. Our implementation uses the GPT-NeoX library (Andonian et al., 2023) via HuggingFace Transformers Wolf et al. (2019), which is imple- mented in PyTorch (Paszke et al. 2019).
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. In The Thirteenth International Conference Learning Representations, 2025. URL https: / http://wwww..ett.toomdddddddd-ee
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy- Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang. Shivanshu Purohit, Hailey Schoelkopf. Dashiell Stander, Tri Songz, Curt Tigges, Ben- jamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Lan- guage Modeling in PyTorch, 2023. URL https::///// //wps:://wwww/// www.gg..thuub.coo....... wtt.:/om///eeeeeaar/a..a
gpt- pptt-eex neox
Mikita Balesni, Tomek Korbak, and Owain Evans. Lessons from studying two-hop latent reasoning. http:://axxiv.orzz
2025. URL https: // www.aaaiiiv..o. www.ooggg/.oo....o.. abs /aees/411... 16353
Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on "a is b" fail learn "b
a". In The Twelfth International Conference on Learning Representations, 2024. URL https:
//operreview www.:eet/fooummeeo forum? www.:ormmmmmdmmmmaaaaam d=GGkTTKEAAK
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge language models. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceed- ings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 6491- 6506, Online and Punta Cana, Dominican Republic. November 2021. Association for Computa- tional Linguistics. doi: 10, 10.18653/112021.cmmmpp wwwnm-mmin...... 522. URL http::::s//llathhoooo.... org/ 20eg/2021.22 emnlp-mm----..eee waammlpppminn.. main. 5227
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hadfell-Dodds, Danny Hernandez, Andy Jones, Jackson Kemion, Liane Lovitt,
10

---

Published as a conference paper at ICLR 2026
Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
htmml.
Amit Elhelo and Mor Geva. Inferring functionality of attention heads from their parameters. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pp- 17701-17733, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 379-8-89176-25144 doi: 10. 10.118861112222 acl- acl- .cw.-lalg.866666 URL
hwtp:/oggg/..o..oooo -ong 866/
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are die-imennioonlly linear. In The Thirteenth International Conference on Learning Representationss 2025. BtLnttps:: : http///ppeeeeeee.////////// net/
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Franciie Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November Association for Computational Linguistics. doi: 10.1 00553//1/22211emlpmmmeee 446. URL. httpps://alLanthoooog 22tg wwwp:////......................a emnlp-mminn..... 446/
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall factual asso- ciations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6109-6125, 2023.
Olga Golovneva, Zeyuan Allen-Zhu. Jason E Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. In First Conference Language Modeling, 2024. URL https:
7/ /oppnrevieww wtwp::/wwwwn/...tnto.mama http://oooommmmidiidbddot.t
Wes Gurnee and Max Tegmark. Language models represent space and time. In The Twelfth Interna- tional Conference Learning Representations, 2024. URL htttps://pperrevieeeee n
Eorum? d==ndd===jEExee in.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing?
surprising differences in causality- -based localization knowledge editing in language models. In Shirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openeeiew. wwt.eee//ooooo wwtp:/nwtteetttteeeeeeeeeeeeteet Eat-dd--EEttttts
Peter Hase, Thomas Hofweber, Xiang Zhou, Elias Stengel-Eskin, and Mohit Bansal. Fundamental problems with model editing: How should rational belief revision work in LLMs? Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL /eLLtttps///peeeeeiiee..u
Forum?id LRE LRE191LL1
Evan Hernandez. Arnb Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In The Twelfth International Conference on Learning Representations, 2024. URL https:
7 //operreview. wwt.tee..ouueeeeeee forum? 7 4KE
Ouail Kitouni, Niklas Nolte, Adina Williams, Michael Rabbat, Diane Bouchacourt, and Mark Ibrahim. The factorization curse: Which tokens you predict underlie the reversal curse and more. In The Tiihth Annual Conference on Neural Information Processing Systems, 2024. URL.
https: http:::s//ppenevveew www.eeewww.eeeeee.eeee.....eeeee Forumoidiiioooo..o.... f70e6YYFHE
Andrew K Lampinen, Arslan Chaudhry, Stephanie CY Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, and James L. McClelland. On the generalization of language models from in-context learning and finetuning: controlled study, arXiv preprint arXiv:2505.00661. 2025.
Zhengkai Lin, Zhihang Fu, Kai Liu, Liang Xie, Binbin Lin, Wenxiao Wang, Deng Cai, Yue Wu, and Jieping Ye. Delving into the reversal curse: How far large language models generalize? In The eight Annual Conference on Neural Information Processing Systems, 2024. URL https::/oppnreviiee net wtwp://wwwwweee..wmnweeemm wowummrrmmoooo did 1et:idd==wxFgggmmh0m wxFznQWhp

---

Published as a conference paper at ICLR 2026
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- net/ www.eeeee.ooottaoeoooo
ence on Learning Representations, 2019. URL /tt..ttss/////peeee/e......ttt
Bkg6RiCqY7
Kevin Meng. David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing fac-
tual associations in GPT. In Advances in Neural Information Processing Systems, 2022. URL
httts::/cppeneet: ew net forum? h-w---hhhhhhh-- 6444
Kevin Meng, Arna Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL
net www.oooiddd..aaaaa.i.........
Jack Merullo, Carsten Eickhofff and Ellie Pavlick. Language models implement simple Word2Vec- style vector arithmetic. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 5030-5044 Mexico City, Mexico, June 2024, Association for Computational Linguistics. doi: 10.18653v1/20244 org. naacl wong.2811
naacl-long.288 281. URL
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Dis- tributed representations of words and phrases and their compositionality. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https::
In C.J.
ngs nww.inuuri.... wwppappr..ppppe..peeeeeeeeee.eee fi www.ffleaappaae Ffless/paper/2033 .pff
File File/9as 4263 B8 B22 923.eee09999..333333233333331133 h11.////114.32222222555522222211
Maximilian Nickel.
Tensor factorization for relational learning-
Ladwig-Mxximilians de:
Universität München, August 2013. URL
bvb: 19b::19-20 19106668
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learning on multi-relational data. In Proceedings the (ICML-I1), pp. 816, 2011.
the 28th international conference on machine learning
Kento Nishi, Rahul Ramesh, Maya Okawa, Mikail Khona, Hidenori Tanaka, and Ekdeep Singh Lubana. Representation shattering in transformers: A synthetic study with knowledge edit- ing. In Forty-second International Conference on Machine Learning, 2025. URL https: //opereeieww net. http://rrmmndddBBeeee
Alberto Paccanaro and Geoffrey E. Hinton. Learning distributed representations of concepts using linear relational embedding. IEEE Transactions Knowledge and Data Engineering, 13(2): 232-244, 2002.
Haowen Pan, Xiaozhi Wang. Yixin Cao, Zenglin Shi, Xun Yang, Juanzi Li, and Meng Wang. Precise localization of memories: A fine-grained neuron-level knowledge editing technique for LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https: 7/ 7/openzeview.oo www.:eeee//ooddoaeeeeeem 1HDvpXI
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vitoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bus E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL
feleeppprr_ees///epppepp files/ htw.://wceeeinnnnnn..rro....oooo baper/2019) file http:///ww...b..oo..eeeee2222211
Jacques Thibodeau. the rome model
But is really in rome? Alignment Forum. 2022.
an investigation of
editing technique.
URL
http:://www.aagggaaae.e.oo.oooop but t- oh
12

---

Published as a conference paper at ICLR 2026
Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao. Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang. Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng. and Huajun Chen. EasyEdit: An easy-to-use knowledge editing framework for large language mod- els. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 82-99 93, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 18653/v11/ 2024.acl-demos.9 URL
2025:/2224ooo2 acl demos. 9/
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's s transformers: State-of-the-ar natural language processing. arXiv preprint arXiv:1910.03771 2019.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large lan- guage models latently perform multi-hop reasoning? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 10210-10229, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18853/v1/200 1/265/////2224..loooog.g5555355 URL https:/aclaattoology
wwwp:n/ww.ogyyoog//22..coo wttp//////22...o. 000gg/2202.. 5507]
Sohee Yang, Nora Kassner, Elena Gribovskaya, Sebastian Riedel, and Mor Geva. Do large lan- guage models perform latent multi-hop reasoning without exploiting shortcuts? In Wanxi- ang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, Vienna, Austria, July 2025a.
http:///......innaa....... Association for Computational Linguistics. doi: 10. 10.18653/v1/2025. findings-acl.205. URL
2057
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candes, and Tatsunori Hashimoto. Synthetic continued pretraining. In The Thirteenth International Conference on Learning Representations, 2025b. URL /ttpp:://peereeveew..ne Ltt.://eweweee////ooeeeeedd-- 1d
Yunzhi Yao, Peng Wang. Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 10222-102400 Singapore, December 2023. Association for Computational Linguistics. doi: 10. 111888////223..mmm/pp/. main.632. URL https: //acl antho ogy. wttgg//223.3 emnlp-ma i n.. 632.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.0033 003663, 2020.
Hanlin Zhu, Baihe Huang, Shaolun Zhang, Michael Jordan, Jiantao Jiao, Yuandong Tian, and Stuart
Russell. Towards theoretical understanding of the 'reversal curse' via training dynamics. In The Thinty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https: http:::://ppenreviee n
13

---

Published as a conference paper at ICLR 2026
A
MODEL ARCHITECTURE AND TRAINING DETAILS
We use a decoder-only Transformer (GPT-NeoX) with rotary positional embeddings (RoPE).
Architecture configuration (GPT-NeoX):
Layers: 12 Transformer blocks
Hidden size (or embedding dimension): 896
Attention heads: 16 (head dimension 56)
Feed-forware size: 3584 (4x hidden size)
Positional encoding: rotary embeddings with standard base 10,000
Max context length: 1024 tokens
Dropout: attention 0.1, MMP hidden 0.1
Residual path: non-parallel residual (use_parallel residual False)
Number of parameters: ~20M..
Training setup:
Hardware: 4 A100 GPUs
Batch size: per-device 16 (train), 32 (eval); global 64 (train), 128 (eval)
Optimizer: AdamW with learning rate 3 x 10-1 and (31, 82) = (0.9, 0.95).
Hyperparameter sweep: weight decay E {0, 0., 0.5, 1, 2, 4, 6): random seed 1,
Learning rate scheduler: Cosine decay with linear warmup ratio 0.01
Training epochs: 20
Note that train all models from scratch without using any pretrained weights and used the tokenizer from GPT-NeoX.
B SYNTHETIC DATA CONSTRUCTION AND EXAMPLES
We construct a synthetic family- graph dataset where each family contributes a single document formed by concatenating all relational facts sentences: "[Subject First Name] [Family Name] [Relation] [Object First Name] [Family Name]". A family name is shared by all entities (or mem- bers) and formed by "[Middle Name] [Last Name]", so the full name is "[First Name] [Middle Name] [Last Name]". We sample names from fixed pools (listed below) to ensure uniqueness and reproducibility.
Generation rules:
Entities: one family per document with unique members; all members share the family name.
Relations:
eight types husband, wife, father, mother, brother, sister, son, daughter.
Split: 1,000 families divided into two groups of 500 each.
Training data:
Group 1 (first 500 families): includes all eight relations. 5,000 members total; 36 facts per family. See example below.
Group (next 500 families): excludes father/mother. 5,000 members total; 24
facts per family. See example below.
Test data:
For Group familics, add back only the father/nother facts to create the test set (12 facts per family). See example below.
14

---

Published as a conference paper at ICLR 2026
Trainset example from the first group (all relations).
sandy Francis Barton brother Zachary Francis Barton. Katrina Francis Barton sen Zachary Francis Barten. Sandy Francis Barton father Ryle
Francis Barton, Debra Francis Barten daughter Katrina Francis Barton.
Kyle Francis Harton mother Veronica Francis Barton. Kyle Francis Harton
daughter Sandy Francis Barton. Debra Francis Barton husband Gary Francis
Barton. Henry Francis Barton sister Katrina Francis Barton. Justin
Francis Barton wife Veronica Francis Barton. Katrina Francis Barton
daughter Sandy Francis Barton. Verenica Francis Barton
Kyle Francis
Barton. Vanessa Francis Barton father Justin Francis Barton. Gary
Francis Barton son Henry Francis Barton. Gary Francis Barton wife Debra
Francis Barten. Kyle Francis Barten father Justin Francis Barton. Gary
Francis Barten daughter Katrina Francis Barton. Katrina Francis Barton
father Gary Francis Barton. Zachary Francis Barton sister Sandy Francis
Barton. Debra Francis Barton sen Henry Francis Barton. Rachary Francis
Barton father Kyle Francis Barten. Veronica Francis Barton daughter
Vanessa Francis Barton. Henry Francis Barton father Gary Francis Barton.
Kyle Francis Barton sister Vanessa Francis Barton. Henry Francis Barton
mother Debra Francis Barton. Katrina Francis Barton brother Henry
Francis Barton, Sandy Francis Barten mother Katrina Francis Barton.
Zachary Francis Barton mother Katrina Francis Barton. Vanessa Francis Barton mother Veronica Francis Barten. Katrina Francis Barton husband Kyle Francis Barton. Kyle Francis Barton wife Katrina Francis Barton.
Justin Francis Barton son Kyle Francis Barton. Justin Francis Barton
daughter Vanessa Francis Barton. Katrina Francis Barton mother Debra Francis Barten. Veronica Francis Barton husband Justin Francis Barton.
Vanessa Francis Barton brother Kyle Francis Barton. Kyle Francis Barton son Zachary Francis Barton.
Trainset example from the second group (without father/mother).
Dalton Scott Wall sister Colleen Scott Wall. Ebony Scott Wall husband
Cody Scott Wall. Ebony Scott Wall son Julian Scott Wall. Jamio Scott
Wall brother Julian Scott Wall. Jacob Scott Wall son Dalton Scott Wall.
Jacob Scott Wall wife Jamie Scott Wall. Curtis Scott Wall daughter Brenda Scott Wall. Brenda Scott Wall brother Jacob Scott Wall. Emily Scott Wall husband Curtis Scott Wall. Jamie Scott Wall son Dalton Scott Wal1. Curtis Scott Wall wife Emily Scott Wall. Cody Scott Wall daughter Jamie Scott Wall. Jamie Scott Wall husband Jacob Scott Wall. Jacob Scott wall sister Brenda Scott Wall. Emily Scott wall daughter Brenda
Scott Wall. Cody Scott wall son Julian Scott wall. Ebony Scott wall
daughter Jamie Scott Wall. Curtis Scott Wall son Jacob Scott Wall. Cody Scott Wall wife Ebony Scott Wall. Colleen Scott Wall brother Dalton
Scott Wall. Jamie Scott Wall daughter Colleen Scott Wall. Julian Scott Wall sister Jamie Scott Wall. Jaceb Scott Wall daughter Colleen Scott Wall. Emily Scott Wall son Jacob Scott Wall.
Testset example f'rom the second group (only father/mother). 12 prompts per family.
Julian Scott Wall mother Ebony Scott wall.
Julian Scott wall father Cody Scctt wall.
Jamie Scott Wall mother Ebony Scctt wall.
Jamie Scott Wall father Cody Scott Wall.
Jacob Scott Wall mother Emily Scott Wall.
Jacob Scott Wall father Curtis Scctt Wall.
Brenda Scott Wall mother Emily Scott Wall.
Brenda Scott Wall father Curtis Seett Wall.
Dalton Scott Wall mother Jamie Scott Wall.
Dalton Scott Wall father Jacob Scott Wall.
Colleen Scott wall mother Jamie Scett wall.
Colleen Scott wall father Jacob Scett wall.
15

---

Published as a conference paper at ICLR 2026
Name sampling. We draw first names by gender, middle names from fixed pools, and last names from a large pool. The family name is "[Middle Name] [Last Name]", shared by all members. Below the exact pools used.
NAME POOLS (FOR REPRODUCIBILITY)
FEMALE FIRST NAMES
Sheryl, Caitlyn, Alisha, Heidi, Frances, Elaine, Catherine, Bridget, Tami, Norma, Bianca, Robyn, Kylie, Amanda, Alyssa, Brandy, Dorothy, Erica, Melody, Sandra, Alison, Peggy, Debra, Sophia, Victeria, Kristy, Ebony, Lorettå, Robin, Holly, Adrienne, Christina, Veronica, Joy, Tasha, Chloe, Doris, Jody, Manda, Tricia, Kayla, Brenda, Karen, Judith, Sandy, Hailey, Angela, Madeline, Natalie, Carol, Katrina, Beth, Pam, Jamie, Shelia, Sharen, Karina, Rebekah, Deanna, Autumn, Angelica, Ellen, Jade, Sierra, Tracie, Brianna, Susan, virginia, lydia, Karla, Christy, Kathleen, Kaitlyn, Diane, Haley, Hailey, Colleen, Nancy, Yesenia, Sara, Madison, Shannon, Hayley, Patty, Terri, Joan, Anne, Emily, Vanessa, Jenny, Kimberly, Hannah, Ashley, Deminique, Rachael, Toni, Melanie, Kerry, Mackenzie, Charlene
Guy, Damon, Gerald, Steve, Samuel, Gregory, Todd, Mark, Timothy, Leroy, Julian, Fernando, Dalton, Rick, Ralph, Cesar, Bill, Clinten, Darren, Dave, Marco, Brandon, Kyle, Kristopher, Noah, Ross, Glen, Shawn, Alec, Cole, Ryan, Harold, Johnathan, Ccdy, Jacob, Mason, Daryl, Mike, Adan, Wesley, Raymond, Don, Richard, Clayton, Jake, Seth, Edgar, Tracy, Kent, David, Roy, Aaron, Jerome, Phillip, Alexis, Steven, Victor, cavier, Gavin, Brad, Gene, Caleb, Carl, Peter, Brett, Cory, Craig, Jesus, Gary, Oscar, Henry, Cameron, Curtis, Zachary, Mathew, Jared, Ernest, Sergio, Nicholas, Hayden, Kevin, Justin, Jen, Christian, Joseph, Darryl, Eduardo, Joe, Jerry, Duane, Vernon, Micheal, Greg, Frank, Bradley, Corey, Rodney, Angel, Derrick, Terrence
HIDDLE_NAMES
Anthony, Marcus, Jose, Kenneth, Lee, Colin, Arthur, Kirk, Blake, Dan, Benjamin, Marvin, Troy, Philip, Denald, Jamie, Calvin, Luke, Dustin, Mare, Tristan, Andres, Michael, Tyrone, Jeffery, Patrick, Wyatt, Luis, Larry, Frederick, Earl, Darrell, Perry, Roberto, Shannon, Douglas, Eddic, Jaime, Chad, Scott, Norman, Francis, Johnny, Ruben, Bernard, Albert, Rickey, Miguel, Spencer, Brent, Reginald, leonard, Dennis, Kerry, Ronald, d, Russell, Gregg, Trevor, Drew, Hunter, Brik, warren, Jesse, levi, Francisco, Maxwell, Mayne, Ray, Lonnie, Ricky, Brian, Charles, Parker, Bryce, Bruce, Matthew, Clifford, Edwin, Nathan, Dean, Gordon, Sean, Stanley, Stephen, Karl, Dwayne, Antenio, Brady, Jeffrey, Elijah, Andrex, Adrian, Gilbert, Omar, Taylor, Tanner, Nathaniel, Devin, Lance, Harry
LAST_NAMES
Allison, Hanna, Stark, Mata, Travis, Peters, Zuniga, Smith, Gay, Thornton, Yu, Miller, Weba, Patterscn, Ortiz, Combs, Meadows, Christensen, Freeman, Howell, Berger, Cooley, Glover, Jennings, Blackwell, Turner, Mogee, Duffy, Mentgomery, Glenn, Krause, Coleman, Petersen, Gregory, Barnes, Morris, Hensley, Harding, Bird, Estrada, Garza, Gomez, Burke, Waters, Lam, Davenport, Frost, Stafford, Jarvis, Williams,
Dataset augmentation. For each family document, create augmented training instances by randomly permuting the order of sentences (facts) while keeping each sentence unchanged. We generate 1, 1,000 permutations per family, resulting in 318M tokens per training epoch.
16

---

Published as a conference paper at ICLR 2026
C SVD-BASED UPDATE OF RELATION MATRICES IN RESCAL
In the RESCAL model (Nickel et al. 2011 the Alternating Least Squares (ALS) procedure requires updating the relation matrices (M, th fixed, where n is the number of entities, d is the embedding dimension, and m is the number of relations.
while holding the entity embedding matrix A Raxd
The objective function for single relation r is:
L(M,) = 1
- + AR |
(6)
Our goal is to find the M, Rdd that minimizes this function.
To find the minimum, we can set the gradient of L(M,) with respect to M, to zero:
0L AT AM, ApM,
iM,
Rearranging the terms, we get the normal equation:
= A AMA + ARM, A
(8)
This is a continuous Sylvester equation. Using the vectorization operator vec() and the Kronecker product @, can rewrite it as standard linear system:
((ATA) @ (ATA) + ARIæ)) vec(M,) vec(A X,A)
Solving this equation directly requires inverting a dense (d2 x d²) matrix, which computationally expensive with complexity of ((()2)) (dd). This becomes prohibitive the embedding dimension d grows. Due to high dimension of embedding space (d = 896) of our models, Eq. 9 is infeasible to solve directly.
To mitigate this issue, we employ the Singular Value Decomposition (SVD) to overcome the afore-
mentioned computational bottleneck. Let the SVD of the entity matrix be A USVT with thonormal U, V and singular values S = diag(s......./) Using A A = V Eq. can be rotated into the singular space, yielding the diagonal Sylvester equation
(M) + (M( = (
M, = V X = U X,U.
Solving element-wise gives
(M, )ij = 8,8, (X, )aj,
57 AR
or in matrix form M, P Rr with P. s,8, (5/ + AR). Transforming back,
M, = V O (U XU))VV
The update costs O(nd²) for the SVD and O(n2d+ dd) for the remaining multiplications, vs. (ad) for the naive Kronecker inversion. In this work, we employed the algorithm in Nickel 2013 and reproduce the formula here only to motivate our implementation choice.
1250 (max entities; 125 families, 10 members per family) and d = 896 (embedding dimension) in our experiments, making the baseb update feasible. M, obtained from train set then evaluated on test set B. B and A have disjoint entity sets.
17

---

Published as a conference paper at ICLR 2026
D TRAINING, PROBING, AND EDITING RESULTS
D.1 TRAINING RESULT IN DETAIL.
In this section, we provide training curves for all models with different weight decay values and random seeds over training steps (see Fig 6). All models achieve 100% training accuracy, but test accuracy varies significantly based on weight decay and random seed.
Train Loss
Trainser Accurazy
Roversal Curse Evaluction
2.00
0.0
1.0
3.0
1.7
0.1
0.5
0.5
1.0
2.0
1.00
Figure 6: Training loss, train accuracy, and test accuracy for models with different weight decay values and seeds.
D.2 PROBING RESULTS DETAIL
D.2.1 LINEAR RELATION EMBEDDING
Figure shows the probe's s accuracy function of the number of training samples per relation, n (n 10, n 100, and n 500). The main text reports results for 10 due to the high computational cost of Jacobian calculations. Here, we show that increasing the number of samples up to 500 does not improve performance, confirming that the poor accuracy of the linear relational embedding probe is not due to insufficient sampling. We also n 10, n 100, and n = 500, which shows that insufficient leads to underfitting.
bilinear probing for
Figure 8 visualizes the layer-wise accuracy for each of the eight relations individually for N 100 and B 3. Interestingly, few models in "Reversal cursed" group (blue) show high accuracy at mid-late layers while "Not Reversal Cursed" models (orange) do not. It indicates that some models in the "Reversal cursed" group do learn linear relational embeddings for certain relations, but they not consistent across relations and layers.
D.2.2 TRANSLATIONAL
Figure shows the per-relation accuracy for this task. The "Not Reversal Cursed" models exhibit high accuracy only for the symmetric husband and wife relations, peaking at mid-to-late layers. Accuracy for all other relations is near zero for both model groups This suggests that while translational structure learned, limited to simple symmetric pairs and does not generalize to other relation types.
D.2.3 BILINEAR
shows the per-relation accuracy of bilinear probing, The "Not Reversal Cursed" models achieve high accuracy across all relations, peaking at mid-to-late layers. In contrast, "Reversal Cursed" models show near-zero accuracy for all relations. This indicates that learning bilinear relational structure is strongly associated with overcoming the reversal curse and generalizes well across different relation types.
Figure 10
18

---

Published as a conference paper at ICLR 2026
D.3 EDITING RESULT DETAIL
Setup. For each model we sample 50 distinct husband facts (A, husband, B) from Group 1. Each edited to (A, husband, B') where B' is another female entity from the same family (preserves name template and type). Single edit per run; no simultaneous multi-fact changes. For every layer I e .......I 12] we fine-tune only the MLP output (final linear) weights that layer using a single example (A, husband, B²) Optimizer: Adam, Ir 4 x 10-², early stop when loss < 0.2 (cap 50 update steps). All other parameters frozen.
Metrics. For each edited model:
Edit Success: accuracy on (A, husband, B').
Logical Generalization (Reverse-relation); accuracy on (B', wife, A).
Logical Generalization (B', sson/daughter, C/D): mean accuracy over (B', son/daughter, C/D).
Logical Generalization father, B'); mean accuracy (C/D, father, B').
Locality (In-Family): accuracy on other facts inside the edited family excluding any incident to B'.
Locality (Other Families): aceuracy on fixed held-out set of facts from untouched families.
Accuracies are proportion of correct next-token generations for the object name (exact match). Curves Figure repport the mean over the 50 independent edits.
Blinesr n=500
Lamsed
0.6
0.4
0.4
0.2-
0.2-
Selation
013
0.5
0.8
0.8
0.2
Figure 7: Bilinear and linear relational embedding accuracy for n = 10, n2 = 100, and n 500, where n denotes the number of training samples per relation.
19

---

Published as a conference paper at ICLR 2026
trother
daughter
fusher
huaband
leverse Daned
Revernal Gareed
0.0
mother
$15W
wilee
0.0
Figure Visualization of linear relational embedding probing results for each relation r as a spaghetti plot (n 100 and 3 = 5).
Viyker
Toonrroonn ACC
0.8
0.66
0.4-
5500
2.5
5.0
7.5
10.0
12.5
2.5
10.0
12.5
5.0
7.5
10.0
125
2.5
5.0
10.0
12.5
Trarubooo ACC
0.6
0.4-
0.2
Figure 9: Translational probing accuracy for each relation r.
3.2
3.0
Not kevorall Dursed
3.4
Blincor
*srother
3.2
1.2
1.0
1.0
28-
Layor
Liyer
L2
Figure 10: Bilinear probing accuracy for each relation
20

---

Published as a conference paper at ICLR 2026
Edit success (A, husband, 81
Locality twithin edited family)
Locality tunedited families)
SuCCMA 140
Not Reveraal Cumes
Re-verse- reletion wile,
Logical Generalization sonydas.ghter. (29)
Genernizzatin IC/D, Father,
Success cate
0.6
10
12
12
Layer
Layat
Layer
Figure 11: Editing experiment details. Six panels: Edit Success, Locality (edited family), Locality (other families), Reverse relation, Logical Generalization to children, Logical Generalization to parents.
21


Reviews:
[{"review_id": "5iLqAUWudj", "reviewer": "Reviewer_Xt6A", "summary": "This paper studies toy models to propose a hypothesis for why phenomena like reversal curse are observed: specifically, the paper argues for the (in)existence of the underlying structure (e.g., a graph) that binds the knowledge together when a model fails/succeeds to generalize to novel facts or combinations thereof (e.g., A->B vs. B->A). Broadly, the argument is that the existence of a structure allows a fact's implications to be inferred by the model, enabling generalization.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "While the studied toy model and the derived claims can be (partially) found in recent literature, the insight of measuring how bilinear the representation is novel and I enjoyed thinking about that. The experiments have a solid outline to drive the main argument, which was fun to read.", "weaknesses": "I have three major apprehensions about the paper's results and narrative, as detailed below.\n\n- Relation to prior work: Nishi et al. [1] perform a similar study as this work, wherein a synthetic knowledge graph is defined, a model is trained from scratch to learn it, and it is demonstrated that when the model learns the underlying graph topology (circles instead of hierarchies, as explored in this work), editing its knowledge via weight-targeting methods (e.g., ROME) can break the underlying graph structure---called \"representation shattering\" in [1]. The extent of representation shattering, i.e., how structured the representation is, correlates with success on multi-hop factual inferences---the conclusion also arrived upon in this paper. While I understand the current paper's analyzed methodology for updating the model is fine-tuning, I note that I'm highlighting [1] because (a) there's a broad similarity in the motivating problem and setup that [1] warrants contextualization in the current paper, and (b) the conclusions of [1] suggest editing can be quite brittle if the model has learned the underlying structure of the knowledge graph, which, even if it is explained by the use of a different editing method, warrants discussion to narrow down the current paper's conclusion. Finally, I'll also note that [1] verifies its conclusions with real models, using the fact that knowledge graphs are often stored with a precise structure in representations (e.g., days of the week [2] or the world map [3]). On a related note, I'll also add another relevant paper that performs a similar study as the current paper [4].\n  - To be clear, I am not suggesting the current paper does not possess novelty; e.g., I really enjoyed the idea of using the bilinear probe as a measure to gauge how well structured the representation is. In [1], a similar experiment was performed via isomap projections, which implicitly involves the assessment and visualization of how well model representation similarities map to the ground truth adjacency matrix of the graph. However, I prefer the approach taking in this work and hence consider the work useful. Nevertheless, the writing often argues that the paper is the first to attempt a study of this sorts or arrive at conclusions shown in the paper, and for that I hope authors can rework the paper writing.\n\n- Non-toy model experiments: As noted above, there are several precise structured representations in pretrained language models such that the conclusions drawn in this work should be easily testable on scale (again, I'd refer to [1], since they did those experiments). I would have loved for an attempt at this verification and deem this a strong limitation of the work right now. To be clear, I'm all for using toy models to derive hypotheses, but I expect then an evaluation of the hypothesis's prediction be conducted when it is feasible, which, in this case, is.\n\n- Narrative problems: Finally, I'll note that the paper begins with discussion of model editing and reversal curse. The precise experimental regime studied, however, is one of pretraining. Specifically, the knowledge is being embedded into the model during pretraining and, during fine-tuning, some new facts may be added to test whether generalization occurs in expected directions. This is different from the precise reversal curse setting, as I understand it, and also for editing.\n\n[1] https://arxiv.org/abs/2410.17194\n[2] https://arxiv.org/abs/2405.14860\n[3] https://arxiv.org/abs/2310.02207\n[4] https://arxiv.org/abs/2503.21676", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nKoyYYDwu0", "reviewer": "Reviewer_GUyF", "summary": "This paper investigates how Transformer-based language models represent relational knowledge, focusing on the “reversal curse”: the inability of models to infer “B is A” from “A is B.” The authors study this in a controlled synthetic setting by training Transformers from scratch on relational data describing family members (10k entities, 8 relations: _mother, husband, son, brother_, etc.). For a subset of families, _father_ and _mother_ relations are withheld during training, enabling the authors to test generalization to these unseen relations.\nThey analyze which models can generalize these withheld relations and how their internal representations differ. The main findings are:\n- **Weight decay** critically determines whether models generalize or merely memorize. Higher weight decay leads to more models that generalize correctly.\n- By probing internal representations, they evaluate three relational geometries—linear (projection matrix + bias), translational (additive bias), and bilinear (matrix-mediated interaction). They find that:\n    1. These relational structures are more prominent in models that generalize.\n    2. The **bilinear** structure most successfully explains model behavior.\n- Finally, models that develop a bilinear structure also exhibit consistent **model editing** behavior: editing a single fact propagates correctly to logically entailed facts. Models lacking this structure fail to generalize edits.", "questions": "- Do all models fit the training data equally well? It would be helpful to report training accuracy to confirm that differences arise from generalization rather than underfitting.\n- How exactly is decoding performed to compute accuracy in Figure 2?\n- What do the gray points in Figure 2 represent?\n- This work seems related to studies on entity mapping in attention heads (e.g., _“Inferring Functionality of Attention Heads from their Parameters”_). It might be worth referencing this connection.\n- The linear and bilinear probes differ not only in structure but also in training data size (10 examples vs. 1250). Could the superior performance of the bilinear probe partly reflect this discrepancy? It would help to discuss whether a larger dataset could make the linear probe competitive.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a high-quality, well-motivated paper that offers new insights into how relational knowledge and logical consistency emerge in Transformer representations. While limited to a synthetic domain, the work’s conceptual clarity and systematic experimentation make it a valuable contribution to understanding the internal geometry of knowledge in LMs.\n\n- The paper is clearly written and well-organized, making it accessible and easy to follow.\n- The experimental design is sound and thoughtfully constructed, allowing for careful isolation of the phenomenon under study.\n- The contributions are conceptually substantial and novel, offering meaningful insight into the representational geometry underlying relational reasoning in Transformers.\n- The work provides a mechanistic link between regularization, representational structure, and model editing behavior.", "weaknesses": "- The study relies entirely on a toy synthetic setup, which limits the direct applicability of findings to real-world pretrained LLMs. Though the authors acknowledge this in the discussion.\n- The connection between bilinear structure and fact editing may be correlational rather than causal. Other factors (e.g., differences in data fitting or feature organization) could also drive the observed editing generalization.\n- The finding that weight decay promotes generalization may reflect the limited dataset size rather than a broader principle. Without sufficient regularization, memorization is expected for the size of the data, so further analysis (e.g., varying dataset size or noise) would strengthen the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fkQ9frAIJ5", "reviewer": "Reviewer_jWrw", "summary": "This paper argues that transformer LMs trained on a synthetic relational knowledge graph can develop an internal bilinear relational structure that both overcomes the reversal curse and enables logically consistent edit propagation. The authors train 12-layer, ~206M parameter decoder-only transformers from scratch on a synthetic kinship-style corpus, sweeping weight decay to obtain both \"reversal cursed\" and \"not reversal cursed\" regimes. They then probe layerwise hidden states using three hypothesized relational geometries: linear, translational, and bilinear. They find that only a bilinear probe of the form $f_r(s,o) = s^\\top M_r o$ achieves near-perfect relation classification in the middle layers (layers 6-9) of models that circumvent the reversal curse, while linear and translational probes remain near chance. The learned relation matrices also pass algebraic tests: their transposes approximate inverse relations and their products approximate two-hop compositions. Finally, the authors perform single-fact edits via layer-restricted fine-tuning and report that edits to earlier layers (1-4) propagate consistently to logically entailed inverse/neighboring facts while mostly preserving unrelated facts. They argue that this generalization correlates with the emergence of bilinear structure.", "questions": "1. Is it viable to run projection ablations that remove the principal subspace of $M_r$ at a bilinear-peak layer to test necessity, and add a low-rank bilinear adapter in cursed models to test sufficiency?\n2. Similarly, would it be possible to repeat editing with gradient-aligned versus orthogonal updates across layers, and re-probe $M_r$ to verify whether transpose/product structure survives the edit?\n3. Could you please add a short discussion addressing how one could extend these experiments and insights to real-world models beyond your synthetic setup, to help guide future work?\n4. Could you please provide more thorough coverage of the related literature on knowledge editing in transformers, especially studies involving synthetic knowledge graphs?\n\nI would be happy to raise my score if the above points are addressed with new experiments, or if the authors can provide convincing reasons for why they the existing evidence is sufficient.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The clear, controlled synthetic setup effectively isolates inverse and compositional structure. The weight-decay sweep is particularly informative for understanding when the reversal curse emerges.\n2. The probing methodology is systematic across three geometric families with an interpretable chance baseline and compelling mid-layer bilinear peaks. The experimental design is solid.\n3. The algebraic validation is convincing (transpose $\\approx$ inverse; product $\\approx$ composition) and makes the representational claim very interpretable.\n4. The layer-restricted editing experiments provide a concrete story about how edits can propagate to logically entailed facts while preserving unrelated facts.", "weaknesses": "1. The causal claims may not be sufficiently supported by interventions. To my knowledge, all evidence that bilinear structure \"fixes\" or enables reversal-curse mitigation comes from readout probes and offline algebra, not from interventions demonstrating that the model's forward computation actually depends on the probed subspace.\n    * Necessity can be shown with projection ablations at a bilinear-peak layer. For example, after removing top singular subspaces of $M_r$ during inference, one could measure the drop in reversal/composition accuracy.\n    * Sufficiency tests could involve adding a low-rank adapter constrained to a bilinear relational subspace. This subspace could be constructed either by: (i) aligning the cursed model's layer representation to a non-cursed model's corresponding layer (via a learned linear map) and importing that model's bilinear directions, or (ii) fitting bilinear relation matrices in the cursed model itself and taking their top singular vectors. One could then ask whether reversal/composition accuracy and edit generalization improve toward the non-cursed regime.\n    * Without such controls, the evidence shows correlation but may not necessarily show causation.\n\n2. The authors edit facts by fine-tuning only the MLP output of one chosen layer and observe that edits in earlier layers generalize better (i.e., doing so updates inverse/related facts more reliably). However, the authors do not check what happens to the bilinear structure after those edits.\n    * As a result, it is unclear whether (i) early layers are special because they are where the relational structure is actually being built, or (ii) edits in early layers simply spread more broadly because they are upstream in the network, regardless of any bilinear mechanism.\n    * One way to separate these would be to apply edits of the same norm that are either aligned with the layer's existing gradient direction or forced to be orthogonal to it, at different depths, and then re-probe $M_r$ afterwards. If bilinear structure (transpose/composition) is still intact and explains generalization, the result would support (i); if not, the result suggests (ii).\n\n3. External validity is limited because the dataset is fully synthetic, the models are small, and no results on natural relations in pretrained language models are provided.\n    * To be completely fair, such experiments are arguably out-of-scope for this paper and could be better left to future work.\n\n4. The paper does not sufficiently engage with recent knowledge editing work that studies transformers trained on synthetic knowledge graphs and analyzes how their internal relational structure changes under editing.\n    * Several recent studies train small transformers from scratch on graph-structured relational data (very similar to the kinship-style setup here), then apply targeted weight edits and observe that naive edits can disrupt relational representations unless constraints are applied.\n    * I encourage the authors to further explore and acknowledge this line of work and situate this work in the broader landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)