Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics
Authors: Hanlin Zhu
Baihe Huang
Shaolun Zhang
Michael Jordan
Jiantao Jiao
Yuandong Tian
Stuart Russell
Venue: NeurIPS 2024 poster (2024)
Keywords: logical reasoning
reversal curse
training dynamics
autoregressive transformer
chain of thought
Primary area: learning_theory
TL;DR: We theoretically analyze the reversal curse of LLM via training dynamics of a bilinear model and one-layer transformers

Abstract: Auto-regressive large language models (LLMs) show impressive capacities to solve many complex reasoning tasks while struggling with some simple logical reasoning tasks such as inverse search: when trained on ''$A \to B$'' (e.g., *Tom is the parent of John*), LLM fails to directly conclude ''$B \gets A$'' (e.g., *John is the child of Tom*) during inference even if the two sentences are semantically identical, which is known as the ''reversal curse''. In this paper, we theoretically analyze the reversal curse via the training dynamics of (stochastic) gradient descent for two auto-regressive models: (1) a bilinear model that can be viewed as a simplification of a one-layer transformer; (2) one-layer transformers under certain assumptions. Our analysis reveals that for both models, the reversal curse is a consequence of the (effective) model weights *asymmetry*, i.e., the increase of weights from a token $A$ to token $B$ during training does not necessarily cause the increase of the weights from $B$ to $A$, which is caused by the training dynamics under certain choice of loss function and the optimization space of model parameters. Moreover, our analysis can be naturally applied to other logical reasoning tasks such as chain-of-thought (COT), which provides a new perspective different from previous work that focuses on expressivity. Finally, we conduct experiments to validate our theory on multi-layer transformers under different settings. Our code is available at [https://github.com/marlo-z/reversal_curse_analysis/](https://github.com/marlo-z/reversal_curse_analysis/).

Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics
Hanlin Zhu"
Baihe Huang" UC Berkeley
UC Berkeley hanl hanlinnzuuberkklleyyy edu
baihe_ haabe__hang@berrkeeyy edu
Shaolun Zhang
Michael Jordan
Jiantao Jiao
UC Berkeley
UC Berkeley
UC Berkeley
shaolun_ziie wh htwpllun_hhangbbereleey.coom edu
jordan@cs Bardan@cs.berreleyy edu
jiantao@berkeley.com edu
Yuandong Tian
Stuart Russell
Meta AI
UC Berkeley
yuandong@meta.com
russellêcs leell@cceeeeeeeer...eeeeeee.eeee berkeley. edu
Abstract
Auto-regressive large language models (LLMs) show impressive capacities to solve many complex reasoning tasks while struggling with some simple logical reasoning tasks such as inverse search: when trained on "A - B" (e.g., Tom is the parent of John), L.M fails directly conclude "B + A" (e.g., John is the child of Tom) during inference even if the two sentences are semantically identical, which known as the "reversal curse". this paper, we theoretically analyze the reversal curse via the training dynamics of (stochastic) gradient descent for two auto-regressive models: (I) bilinear model that can viewed as a simplification of one-layer transformer; (2) one-layer transformers under certain assumptions. Our analysis reveals that for both models, the reversal curse a consequence of the (effective) model weights asymmetry, i.e., the increase of weights from token A to token B during training does not necessarily cause the increase of the weights from B to A. which is caused by the training dynamics under certain choice loss function and the optimization space of model parameters. Moreover, analysis be naturally applied other logical reasoning tasks such chain-of- -f-houghtt (COT), which provides a new perspective different from previous work that focuses on expressivity. Finally, conduct experiments validate our theory on multi-layer transformers under different settings. Our code is available at https: bettp://githin tgt.gttt.bb.....g www/morwmmmaeeeeccco http///aooooo//eerrraaa... wwweecuuuueaeee analysis/.
Introductions
Large language models (LLMs) have shown great performance in solving complex reasoning tasks that require multiple reasoning steps through in-context learning (ICL), such zero-shot learning [1, 2], few-shot learning [3, 4, 5], or via further fine-tuning [6, 7, 8]. However, without the above inference- time techniques model fine-tuning (probably combined with data manipulations), auto-regressive LLM might struggle with simple logical reasoning tasks that require multiple reasoning steps learned during training separately [9], where the reversal curse [10] serves well-known example.
Equal contributions.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The reversal curse refers to the phenomenon that an auto-regressiv LLM that learns "A + B" (e.g.. Tom the parent John) during training fails to generalize the reverse direction "B + A" (e.g.. John is the child of Tom) even if the pair of relationship "->" and "--" are reverse to each other and the two sentences are semantically identical. Although some previous works propose different methods to mitigate the reversal curse, including reversing the training dataset [1 [11, 12] and training on different objectives such as autoregressive blank infilling [13], these methods might negatively affect the model performance Without dataset manipulation or changing the auto-regressive nature (causal structure) of the model, there are two other candidate solutions to mitigate the reversal curse.
other tasks since they either alter the dataset the model architecture.
First, one might constrain the model parameters to satisfy a higher-level regularity for specific relationships. For example, a reversal-type regularity can be viewed as a pair of relationships (-, +) and two sets A, B such that a model trained on "A + B" will also increase its probability "B <- A" for all A E A. B E B, which induces a subspace of model parameters that satisfy this regularity. If one can train the model within this subspace, then training on "A - B" can, by definition, help to learn "B- + A". However, for a general LLM, it is extremely challenging to find the subspace and manually hard-code the constraint during optimization even for one pair of relationships, not to mention there are numerous relationships. Since it is intractable to manually hard-code the constraints to the model parameter, one can alternatively expect the model to learn the higher-level regularity by training samples under unconstrained optimization. However, this is also hard, according to our analysis, through the popular cross cross-entropy (CE) loss that aims to maximize the next token prediction probability for the models studied in our paper.
Second, one can use a different loss function which is "symmetric", rather than the popular CE loss. However, the "symmetric" loss might drive the model to learn meaningless sentences. For example, when trained on the sentence "John is tall", a "symmetric" loss function might drive the model to learn "tall is John", which is not what we expect. To prevent the model from the above undesired behavior, in practice, CE loss is still widely-used.
Therefore, in this paper, we analyze the reversal curse via training dynamics of the widely-used unconstrained optimization for the CE loss. We summarize our main contributions as follows:
We theoretically analyze the reversal curse where training test sequences have the form "A D or "B + A" via training dynamics of (stochastic) gradient descent under two auto-regressive models: a bilinear model (Section 3) and one-layer transformers under certain assumptions similar to [14] (Section 4). The analysis of both models reveals that the widely- unconstrained optimization for CE loss leads model weights asymmetry, i.c., the increase of (effective) weights (after reparameterization) from the token A to token B' during training does not necessarily cause the increase of the weights from B to A, which further causes the reversal curse. Although the (effective) weights from A to B and from B to A might be related to some extent due to reparameterization, their correlation is weak and thus show asymmetry empirically verified in Section 5.
The techniques we used to analyze the reversal curse can be applied to other logical reasoning tasks. In particular, we use the above framework analyze Afain-of-thoughh (COT) [4]. and we show that a model trained on "A + B" and "B + C" separately struggles to directly conclude "A ru C" without COT even if it is logically true (Section 4.2). Different from the previous work [15] that theoretically studies COT through the expressivity of transformers, our work provides a new perspective through training dynamics.
We also empirically validate our theoretical results on multi-layer transformers (Section 5).
The asymmetry of auto-regressive model weights caused by widely-used unconstrained optimization for CE loss indicates that auto-regressive LLMs might not automatically deduce certain types of conclusions using separate knowledge learned during training under current popular training paradigms: to make model predicting token B where the input token is A, the model might need to see B following A in the same sequence during the training set. This also highlights the importance of ICL, data augmentation, or planning for LLMs with the current popular causal transformer-bast structures to solve complex reasoning tasks.
'The weights from A to B can be viewed the logits of token B when the input is A.

---

1.1 Related works
LLM Reasoning. The strong performance of LLMs on reasoning tasks [3. 7, 16, 2, 17, 4, 18, 19] has prompted many studies on the reasoning capabilities of LLMs. [20] argues that transformers perform implicit Bayesian inference in ICL. [21] shows that transformers implement a specific type of circuits called "induction heads" that are key to the ICL abilities of LLMs. [22] proves that causal structures are encoded in transformer layers during the training dynamics. [23] identifies a backward chaining mechanism of transformers in deductive reasoning. Apart from in-context reasoning, LLMs still demonstrate limitations in other types of reasoning tasks [24, 25, 26].
Reversal Curse. [10] identifies the phenomenon of reversal curse. This drawback of LLMs
is also demonstrated in [27]. [9] studies a similar phenomenonin which LLMs face difficulty in manipulating already learned knowledge. Several paper studies eliminating the reversal curse by extending causal attention to bidirectional attention [13], training on reversed samples [12], permuting semantic units [11], or introducing reverse logic data [28]. Given all the empirical works, theoretical analysis of the reversal curse phenomenon remains scarce.
Expressivity of LLMs. There is long line of works [29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 21, 45] studying the behavior of LLMs through the expressivity of transformers, has been shown that transformers can implement simple functions such as sparse linear functions, two-layer neural networks, and decision trees [42], gradient descent [37, 44, 46), automata [47], Turing machines [48], variational inference [49], and bandit algorithms [50]. Different from [15] that study COT via expressivity, we analyze reversal curse and COT via training dynamics.
Training dynamics of LLMs. There are rich literatures in the optimization of attention layers [51, 52, 53, 54, 55, 56, 57J. [58, 59] study the dynamics of single linear attention layer in in-context linear regression. [60] proves convergence of one-layer transformers in random feature regime. [61] shows the convergence of gradient descent one-layer transformers in in-context linear regression with orthogonal data. [14] studies the convergence of one-layer transformers in a class of next-token prediction tasks. [62] studies training dynamics of multi-layer transformers. [22] studies gradient descent on a class of two-layer transformers in in-context learning tasks with latent causal structures. Our paper studies the reversal curse via training dynamics under both bilinear settings and one-layer transformers. For one-layer transformers, we use the same framework as [14] without the need for certain technical assumptions such as long input sequences, different learning rates for different parameters (except for Appendix C.3), or weak correlations that are required for [14]. Besides, we focus on the generalization ability of models for logical reasoning tasks while [14] mainly focus on optimization, and we identify the asymmetry and intransitivity properties of model weights, which are the core reasons for the failure of LLM for certain types of logical reasoning. Moreover, our analysis of the bilinear model only requires the embedding to be almost orthonormal, while [14] essentially assumed the embedding vectors to be fixed and one-hot.
2 Preliminaries
Basic notations. For any integer N > O, we use [N] to denote the set (1,2.... Let R, N denote the set of real numbers and natural numbers, respectively. For real variables Ii...., Eng use poly(2) constant C s.t. (nn) Cg(n), Vn; we say g(n) Z f(n) if f(n) S g()).
) to denote the polynomial of æı..... Fn. We use [(n) S g(n) if there exists a
We use e, to denote one-hot vectors where only the i-th entry of e, equals one and all other entries are zero. We use 1 to denote all-one vectors, 0 to denote zero vectors zero matrices, and I to denote the identity matrix. We will also add subscripts when we want to explicitly show the dimension, such as 02, la for d-dimensional zero vector and d x d identity matrix. We use & to denote tensor product of vectors or matrices and use 22 and 2 to denote æ @ x and @ A for vector x and matrix A.
We use ( Σ) (or adding subscripts such as Na(-..) if we want to show dimensions explicitly) to denote the variii-) Gaussian distribution with mean μr and covariance Σ. Also, we use A(X) to denote the set of distributions over set X' and use E[:] to denote expectation. For any dataset where x, X and a function ʃ X - I, we define the empirical expectation over the datasct as [f] 1 E (wx) See additional notations in Appendix A.
D {01, { ..., Fn)
3

---

Entities
Forward
Backward
Direct
Indirect
Others
A, B, C, A., B,, C,
R1, R2
Table Notations for tokens in Section 4. "->" and "s" denote forward and backward relationships for the reversal curse. "->" and ">""" denote direct and indirect implication for COT. R1 and R2 are relationship tokens in Section 4.3. A, B, C, A,, B,, C denote tokens representing entities.
Auto-regressive models. Define the vocabulary y = [M] for a positive integer M >> O which is the size of the vocabulary. Let x (x1, ( ..,....,,,................. 2T) ) be sequence of tokens of length T where each token Z2 E V. Vt E [T]. See Table I for notations of different tokens used in Section 4. We study ruto-regressive models po (-(x) E A(V) parameterized by 0 that take the sequence as input and predict the distribution of the next token 2T+1 E V. For both models that we study in this paper, the next token probability is modeled as the softmax applied to the logits lo(-1z) E RMM of each exp(lelvl=))
token in the vocabulary, i.e., (o(l/ar) = EeeD exp(lo(e Vy E V. Also, each token v E V has corresponding (fixed or learnable) embedding vector u, E R
3
Bilinear Models
We start analyzing the reversal curse under bilinear models, which can be viewed as simplified one-layer transformers with input length one and decoder layer only. Also, in this section, assume the embeddings each token are fixed, so we directly use the embedding vector to represent a token.
Datasets, Assume the vocabulary has size m where each token 01, U2..... Um i.dd
(a La).
Let V/ {UI,..., Um and let X {II,..., and y {. yn } be disjoint random subsets of V. Assume all training and test sequences have a length of two. For any 2 S i S n, the training dataset contains both sequence (I,,, y4) and (yt, xz). In addition, the training set contains (x1, v1) while the test set only contain one example (y1 21). During training, the model learns both (z,, Vi) and (y,, 2;) for i to conclude that (I,, y4) is equivalent to (yt, 2;). For example, X' is set of names and y is set of books. The sequence (Ir; y/) means "x; is the author of yı", and the sentence (1)) means "y: is written by x.," We test whether the model is able infer unseen sequence (y1, given the training data which includes the other direction (I1, 91).
Bilinear model. We consider a bilinear model parameterized by Q E Rdxd of which the input contains single token x V. The logits of the next token y E V defined as (y(zz) = x Oy which is bilinear in x and y, and thus the next token probability is pe(y/z)
Leev [)] The training loss for the bilinear model is the cross-entropy loss L(θ)
expile (yl=))
(E log (y:lz)) + + - log (*:(y))) and the test loss (reversal loss) is [
2n
VL())) with the initial- ization O, that can be either randomly sampled from (0 .aa r=2) or set as a pretrained parameter
log peg (.). We study the training dynamics of gradient flow de, dt
satisfying 2m < Pee (3/ (z)), p (4: ly:) m for all E [n]. The following theorem shows a separation during training dynamics.
Theorem 1 (Separation of training dynamics (informal statement of Theorem 5)). Fix any 8, E E (0, 1).
For small a and d 2 poly(n, m, 1/e, log(1 /5)), with probability at least 1 - 5, we have
C (T(()))(TT(0)) 2 (C(O:)/C(0))) Vt 2 0.
Theorem shows that the reversal loss is lower bounded by the training loss. Note that for large and small close to O, (C(O.) (()))) is close to and thus Lrev (O.) Z [rer(O)) which implies that pe 12 (y1) remains small during training. We summarize the above argument in Theorem 2.
Theorem 2 (Lower bound of reversal loss (informal statement of Theorem 6). Fix arbitrary > 0 and C S log (2) Suppose a is small and 2 poly(r (n, m, log(1 (1/8), log 1 1/log C). With probubility at least - 8, it holds that (O)) 2 C, where T denotes the first time such that [(O:) S.c. c.

---

The proofs of Theorems I and 2 are deferred to Appendix B. Theorem 2 implies that for large d', while the training loss can be trained be arbitrarily small, the reversal loss remains large. In other words, the model fails to infer an unseen sequence (11, given the training data which includes the other direction (x1, 91). Furthermore, even if the model fine-tuned on new data from pre-trained parameter O that initially grasps the concept of reversal and satisfies 2m < pe. (ya [z),, pe. (z))y:) < for new data, it fails to extend this understanding to new, unseen data. m
core reason that the reversal curse happens on the above bilinear model is that the parameter matrix O2 is asymmetrie. Consequently, the logits le, (u/æ) = x Ory and le, (xly) y O, generally differ. Consider a special case where each 1; is a one-hot vector. Then le (y/zz) x Oy = 4, and le (2ll))
Oz = 3j for x e; ej. Training on (Ir, y) can increase Oiy but not Gj.. which means the model does not automatically learn the reversal direction (y, x)) from (zz, y). In Section we will show that for one-layer transformers, the reversal curse is mainly caused by the same reason,
i... the asymmetry of the model weights.
4 One-Layer Transformers
In Section 3, we analyzed the reversal curse under a bilinear model. this section, we analyze the reversal curse for one-layer transformers in a similar setting to [14] via training dynamics. We also extend our analysis to chain-of- thhtuuhh in Section 4.2.
Basic notations. Let V = [M]] be the vocabulary. For any token r E [M], we also use the corresponding one-hot vector x = ez E RRM to represent it. Let U = [u,. u2...., um |T E RMxxd be the embedding matrix, where u, E (R the embedding of token æ E [M]. Note that U x = uz. Consider the i-th training sample in the dataset, 2r( (=.li)..... .TT [1] #T([++ [i]), sequence of tokens of length Ti + Here, 4T [ is the next token (or equivalently the label) to be predicted, Ty [s] is the query token, and (xi[i],.... -13)) are contextual tokens. For token =r]3]. we also use its one-hot vector [ [i] = Gx,ld E RRM to represent it. Define the contextual token matrix X ae [i], 2Tl [i]] E R( We omit all in notations when the context is clear.
One-layer transformer. For a training sample x = (I1,... Er, IT+1), its contextual token matrix X = [..,..., =T- and thus XU = [um.....,, Uar- _J contains the contextual token embeddings. We study one-layer transformers in the same setting as [14]. particular, for input token sequence (II,...,,.)) 2T), after the one-layer self-attention, we can obtain ūn U "N(N""" by)
exp(" Wo /vd/
where bur EE ex Wa "Mep/// /Va)" br lor,,..,,r-1.7 contains attention scores
(after softmax) that query token IT attend to each contextual token?, LN(xx) x ||=|| is the ég- normalization operator, and Wo, WK E Rdxxx are trainable query and key matrices respectively. The logit of E [M] is then calculated by decoder layer, i.c.. (e(z|z1,.... .T) u Wv where o encodes all parameters in the transformer, and Wy E Rdxd can be viewed as a reparameterization of value and output matrices. Finally, the next token prediction probability is obtained by
exp(lo(a)2)).....
exp(u Wv
Pe (=
Ex'e(M] exp(lo(x) .., ,T)) [ exp(u Wvūr)
We use the cross-entropy loss function train the model over the whole training set D i.c.. Maxx,Wx.Wq..W Epa. [log
Reparameterization. Similar to [14], we define UWQWFUT /Va e RMX and Y UTJUT M and are interested in their dynamics after reperameterizattio. Then, the attention score (after softmax) and next token probability become
exp(xTTZa..)
exp (x T T LN(X bT))
beT Dx ex
Pa (2/ .TT.
(1)
Σn exp x LTLXXX
Empirically, d only needs to of the order of logarithm vocabulary size. See Appendices E.2.1 F.2.2 for additional results.
Note that assume the query token will not attend to itself in [14].
5

---

and the objective can be written as
max J = D T++ LN(X bT) log Σ x' x Y,Z
(2)
x' #'EMM]
Let 7Jy, 72 be the learning rate of matrices Y and Z respectively. Then the gradient of Y and Z can be characterized by the following lemma:
Lemma (Gradient of Y and Z for I-layer transformer, Lemma [14]. The gradient Y and Z w.r.t (2) of batch size I and learning rate TΙ and 17 can be written as
PxTor X" diag(br) (3) |
= Ly (T)((T+1-00))
- a2)
where Pet
vo T //T////lBB projects any vector to orthogonal complement of v, a = (AN(Xbrr) br)) /1 exp [YT br)).
[an, 02 .... aml' TT E RM with a exp LN(X"
Proof. V, 7 can be obtained by direct calculation. One can refer to the proof of Lemma 1 of [14). L
4.1 Main results for the reversal curse
In this section, we analyze the reversal curse where data points are three-token sentences "A + B" or "B f- A". For each sentence, A and B are two distinct tokens that represent two entities, and " "->" and "K- " are two special tokens representing pair of relationships inverse to each other.
and denote Nout Nuain + best + N.(3) Let Aj B, E V, Vi E [Nueal] be 2N distinct tokens representing distinct entities. Let - V be two additional different tokens that represent two inverse relationships. Specifically, we have A; - B, and B; + A, for all E [Ninal]- For notation convenience, we define the following three index sets
Datasets. Let Nuzin 0, 2
0 and N test
Tarain = [Naain)
(())
= [Nuain + N Juain:
I = [Nwoall) \ (Turann U UZ())
The training set Durain consists of all A; + B, and B; + A, for E Iuain- In addition, Diran contains A; B, for i (1) and B; +- A; for i E I For convenience, we let N = [awain| to be the size of the training set. The test set Piest consists of B, + A, for E I and A, - B, for E T Under our construction of the dataset, the LLM will learn the relationship between A, and B, for E Tuain in both directions to deduce that - is reverse to f-, and learn the relationship between A, and B, for i I U I in one direction and will be tested for the other.
We use po (A; [B, +-) and po (B )) to more compactly represent po(a3 Ag|an = B,, = and ( = B, | = A2 =-), respectively. Our goal is to prove through the training dynamics of one-layer transformers that the test probability remains negligible during training. In particular,
we are interested in Pp (A; B, ). Vi E I((I and Po (B, A +), Vi E I
For convenience, we assume zero- irr-iinitializttionn (0) = O and Z(0) = O. This is the same as [14] and reasonable since empirically, Y and 7 are usually initialized as inner products of d-dimensional vectors with i.ddd Gaussian entries, and thus are almost zero (Lemma 8 in Appendix B). The following proposition shows the initial train/test probabilities are uniform over the vocabulary V.
Proposition 4.1 (Initial probability under
initializaion). Assume the transformer is under sero-initialization (00) = (0), (0) with (0) = 0 and (() = O. For any E [Nrota), we have Poço) (B, |4 +) = Po(a) (4, B, +-) = 1/M.
The proof is deferred to Appendix C.I.I. Proposition 4.1 shows that initially, the probability of predicting any B (or A, respectively) given any A + (or B +,, respectively) as input is uniform over the whole vocabulary. When (0) and 20) are exactly but close to the initial prediction will still be close to the uniform distribution, which is similar to Lemma 6. Next we analyze the dynamics of Pate) (B, -) and Pa(e) (A, +))
Proposition 4.2 (Next token probability). For imput sequence (2), the next token probability under parameters (t) is Pocej (x|x1,22) x2) = exp (Y (1) Eare[M] exp (1)) ) where Y (t) is the entry of the matrix (t) at row and column j.

---

The proof is also deferred to Appendix C.I.I. According to Proposition 4.2, the next token probability when the entity in the imput is # determined by the 21-th row of the matrix (t). Another nice property indicated by Proposition 4.2 is that we don't t need to keep track of the dynamics of (()), which could greatly simplify the analysis. The following lemma shows the dynamics of (t).
Lemma 2 (Dynamics of (t)). Assume run SGD with batch size 1, and assume M > 100 and Mo.u < ny < 1. Let 2 Nn and let Y (t); denote the i-th row of Y (t) and Y (f),z denote the (i.j)-th entry of (t). Then training sequence (#1, x3)) Drain time t, we have
Y (()) .s 2 In (Mnyt/N), and (t)) S - -Im (Myyt/N) Batttt/N//// / Vx # x3, and for any test sequence (21, 23) E Dieso we have Y (1)x,,=
Vo E [M]]
The proof of Lemma 2 is presented in Appendix C.1.2. Lemma implies the asymmetry of the model weights Y (t): for two tokens 21,43, 23, when I1 appears as contextual token and æ3 serves as the next token in the same training sequence, the model weights (t)) gets increased during training while ((f) (f) will not get increased. Combining Proposition 4.2, we can obtain our main theorem for the reversal curse.
Theorem (Reversal curse), Assume we run SGD with batch size 1, and assume M >> 100 and M9.99 ny < 1. Lett Z la denote the time step which also satisfies Int 2 In(NM/ny) For training sequence (x1, 22; F3) (Dirai at time i, we have
ny
( (Iz) Po(1)
x2) 2 1 (M
-
(M(Myyt/NNTT 1, as
for some constant O, and for any test sequence (2, Drest that is not included in the training set Drains we have (zte)(za/z1)))
S L/M.
Theorem 3 shows that although the direction presented in the training set can be learned nearly perfectly, the model's next token prediction of the reverse direction is almost a random guess. The proof is deferred to Appendix C.I.3. We also empirically validate the above results for multi-layer transformers in Section
4.2 Chain-of-thought
In this section, we extend our analysis in Section 4.1 to study other logical relationships. In particular, we study chain-of-thought (COT) [4] and show its importance via training dynamics. COT encourages LLMs to output series of intermediate reasoning steps to increase their performance. Consider the simplest example, where the model learns two facts that A -> and B C. and we want to test whether the model is able to directly conclude that A ~ C. COT indicates that if an LLM is only trained on + and B - C, it would be easier for the model to deduce A ~+ C during the inference time if the model can first output the intermediate steps A + B and B - C, instead of directly predicting the next token C given the input "A ".". The failure of directly deducing A ~ is also empirically observed by [9].
Theoretically, [15] shows the importance of COT for some complex reasoning tasks through the lens of the expressivity of transformers. In this section, we show the importance of COT through a different angle, i.e., training dynamics. We show that for the above simplest two-step reasoning, without COT, the model is not able to directly predict C given the input "A ~ even if it learns A - B and + C.
Theorem 4 (Importance of chain-of-thought, informal statement of Theorem 7). Under certain assumptions as stated in Theorem for any As, Bi, C. s.f. A - B; and B; - C, are in the training set but A, ~os C₁ is not, have
Po(t) (B +) 1,
(C -) 1, Poto) = 1/M as - 0c.
We defer the details of the dataset construction and proof to Appendix C.2. Theorem 4 shows that although the L.MM learns A; - B; and B, -> C, nearly perfectly, cannot directly deduce A; ~a C,- Analogous to the asymmetry of causal transformer weights as we discussed in Section 4.1, our analysis of COT reveals another property, i.e., intransitivity: training the weights associated with A to B and B to C does not necessarily increase the weights associated with A to C.
'The lemma holds even if the batch size larger than I and the analysis is essentially the same.
7

---

We also emphasize that the model fails to directly deduce A, ~ C when the two intermediate steps A; + B, and B; C, are trained separatefy. the two steps are concatenated into a single training sequence, it is possible that the model learns A, ~ C, directly [19].
4.3 Roles of the attention score matrix
During the analysis of Sections 4.1 and 4.2, we show that the reversal curse and the importance of COT are largely due to the asymmetry and intransitivity of causal transformer weights (in our case, the weight matrix Y (()). However, it seems that the dynamics of the attention score matrix (() do not impact the model performance. Below, we briefly discuss the role of the attention score matrix (().
In (I), the attention score used to calculate the weights bęr, where a contextual token 2; with larger attention score attended by the query token æ7 has a larger weight. Note that we use the same formulation the previous work [14] where the query token will not attend itself. Therefore, for three-token training sequence, the weights 212 is always one since there is only one contextual token T1. no matter whether the value of the attention score is high or low.
However, consider slightly different setting, where the relationship is represented by two tokens. In that case, 37 = A; 2/2 R1, 23 R2, v4 B,, and there are two contextual tokens and R₁. The role of the attention score is then to select the important token, i.e., A, by putting more weights on it. Theorem 2 of [14] showed that under certain assumptions, the query token R2 will attend more to "distinct tokens" A, and less to the "common token" R₁. Therefore, the query token R2 will eventually put all weights to As and the remaining analysis remains the same in Sections 4.1 and 4.2, See Appendix C.3 for more rigorous analysis.
5 Experiments
In this section, we conduct experiments further validate our theoretical results in Section 4 on multi-layer transformers. We show experimental results of the reversal curse in this section and COT in Appendix D. Note that in Sections and we theoretically proved the reversal curse for both the bilinear model and one-layer transformer under certain assumptions. Now, we empirically show that the reversal curse still happens even for multi-layer transformers. In Appendix E.2.3, we also provide empirical results that the reversal curse does not happen in ICL settings.
Negative Log Probability over Epochs
Train
70
Vai
60
Nogative Log Probability
50
M parpy
40-
30
20
500 1000
1500
2000
25:00
3000
Number of Epochs
Figure 1: Experiment results of reversal curse under default configuration (see Table 3). The curves represent the (average) negative log probability of the model predicting the next token to be B, when the input is "A, - or be A; when the input is "B, t-. While the sentences the training set can be learned nearly perfectly (as shown by the training curve where the next token probability converges one), the model is not able to predict the correct next token in the validation better than uniformly random guess. Both curves are averaged over 10 random seeds.
8

---

Forward direction legits (Train)
Reverse direction logits (Train)
60
direction loghts (Val)
logits (Val)
-20
40
Figure 2: Visualization of the weights (logits) of the model with default configurations trained after 3000 epochs for the reversal curse experiment. For the top-left matrix, the i-th row corresponds to an entity token A; for training pair, and the i-th column corresponds an entity token B, for training pair. The (i.j) (..j))th entry represents the model weights from the token A; to Bj. i.e., the logits of B, when the input sequence consists of only A, Similarly, for the bottom-left matrix, the row corresponds to the input entity tokens of the seen direction (the direction included in the training set) of validation pairs, and the column corresponds to output entity tokens. The two matrices on right are obtained by swapping row tokens and column tokens of their corresponding left matrices. Note that the diagonals of the bottom-right matrix are all close to zero, while the diagonals of other matrices all have large values. This implies that if a pair of tokens (A, B) only appear in the training set in one direction, then the model weights associated with the other direction will hardly get trained.
Dataset construction. Below, we describe how we generate our synthetic dataset for experiments on the reversal curse. We choose the vocabulary V f0, 1...., ..) for a specified N 0. We randomly sample two disjoint sets of entities A, C V with [A|= | |V/44, and reserve two additional tokens for relationships + and 4-, respectively. 5 Next, we specify bijection from A to B uniformly at random. For each A; E A and its corresponding B; B, we can obtain a pair of sequence (A, + B,, B; + A²). We split the set of all pairs into training pairs and validation pairs. For each training pair, both sequences will be included in the training set, while for the validation pair, we randomly select one sequence for the training set and the other for the validation set. Therefore, the model will learn both directions for the training pairs and only one direction for each validation pair while being tested in the unseen direction.
Model architectures. We train multi- transformers based on GPT-2 architecture [63]. Figure 1 shows the results where the model has 24 layers, 12 attention heads per layer, uses absolute positional encoding, and choose the vocabulary size of 800. The training set size is 340, and the validation set size is 60 (resulting from 140 training pairs and 60 validation pairs). We also conducted experiments with various model configurations and vocabulary sizes in Appendix E.2. Besides, all hyperparameters and different model configurations are presented in Appendix E.
Results. Figure shows that during the training, the next token probability for training data increases a lot while the next token probability for validation data remains unchanged or gets even smaller. This is consistent with our theoretical results of Theorem 3.
According to our theoretical analysis, the reversal curse happens due to the asymmetry of model (re- parameterized) weights (i.e., logits of a token given another token as input), and we also empirically validate the asymmetry for multi-layer transformers. Figure 2 shows the model weights from a token I to is trained large for training sequence (I,, 22, 2z) as represented by the diagonals of the
By default, each entity consists of one token. See multi-token experiments in Appendices E.2 and E.3

---

first three matrices, while the weights from a token to 23 remains nearly zero for validation sequence (2,
Z2, 23) as represented by the diagonals of the last matrix, which consistent with Lemma 2. This implies that if pair of tokens (A, B) only appear in the training set in one direction, then the model weights associated with the other direction will hardly get trained.
6 Conclusions
In this paper, we study the reversal curse theoretically via training dynamics of (1) a bilinear model, which is simplification of the one-layer transformer, (2) one-layer transformers under certain technical assumptions similar to [14]. Our theoretical results suggest that core reason the reversal curse happens in auto-regressive LLMSs is the asymmetry of the model weights, and we apply our technique to prove the necessity of COT for one-layer transformers, which is mainly due to the intransitivity of model weights. The asymmetry and intransitivity of model weights caused by unconstrained optimization of CE loss indicate that an auto-regressive LLM might mainly focus on learning text sequences during training separately instead of automatically deducing indirect conclusions under the current popular training paradigms. This highlights the importance of ICL, data augmentation, or planning for current auto-regressive LLMs to solve complex reasoning tasks.
As for future directions, it would be interesting and important to study: (1) What is a unified way to characterize and study the reversal curse, COT, and other similar logical reasoning tasks? (2) Our paper mainly focuses on three-token sequences, where each entity or relationship is represented by single token. While empirically explored the setting where each entity might consist of multiple tokens and distinct entities might share a few tokens, it would be interesting to analyze the multiple-token setting theoretically.
(3) We theoretically analyzed the bilinear model and one-layer transformer, and it would be an important future direction to extend the analysis to multi-layer transformers.
Acknowledgements
This work was partially supported by a gift from Open Philanthropy to the Center for Human- Compatible AI (CHAI) at UC Berkeley and by NSF Grants 11S-1901252 and CCF-221 1209. The work was done when HZ was a visiting researcher at Meta.
References
[1] Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-7, 2021.
[2] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2222 2022.
[3] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[4] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24833 2022.
[5] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 11171 2022.
[6] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber. David Dohan, Aitor Lewkowyez. Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2/ 2021.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiy preprint arXiv:2110.14106 14168, 2021.
10

---

[8] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
[9] Zeyuan Alu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipula- tion. arXiv preprint arXiv:2309. 14402, 2023.
[10] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" a b" fail to learn" " is a". arXiv preprint arXiv:2309.12888 2023.
[11]
Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, and Yujiu Yang. Mitigating reversal curse via semantic-aware permutation training. arXiv preprint 2rXiv:2403.007555 2024.
[12] Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403. 13799, 2024.
[13] Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. Are we falling in a middle-intelligenne trap? analysis and mitigation of the reversal curse. arXiv preprint Xrv: 2ttp::3//1.004444000 1.04668, 2023.
[14] Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du Scan and snap: Understanding training dynamics and token composition in I-layer transformer, 2023.
[15] Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
[16] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Machine Learning, pages 9118-9147. PMLR, 2022.
[17] Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprini arXiv:2205. 1822, 2022.
[18] Simon Jerome Han, Keith J Ransom. Andrew Perfors, and Charles Kemp. Inductive reasoning humans and large language models. Cognitive Systems Research. 83:101155, 2024.
[19] Xinyi Wang, Alfonso Amayuelas, Kexun Zhang. Liangming Pan, Wenhu Chen, and William Yang Wang. Understanding the reasoning ability of language models from the perspec- tive of reasoning paths aggregation. arXiv preprint arXiv:2402.0 03268, 2024.
[20] Sang Michael Xie, Aditi Raghunathan, Percy Liang and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.0200 2021.
[21] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 1885, 1895, 2022.
[22] Eshaan Nichani, Alex Damian, and Jason D) Lee. How transformers learn causal structure with gradient descent. arXiv preprint arXiv:2402. /4735, 2024.
[23] Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A mechanistic analysis of transformer trained on a symbolic multi-step reasoning task. arXiv preprint arXiv:2402. 19917, 2024.
[24] Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for Ilms on planning and reasoning about change). arXiv preprint arXiv:2206. 10498, 2022.
[25] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang. Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 2023.
[26] Dylan Zhang, Curt Tigges, Zory Zhang, Stella Biderman, Maxim Raginsky, and Talia Ringer. Transformer-based models are not yet perfect at learning to emulate structural recursion. arXiv preprint arXiv:2401.12947, 2024.
[27] Chengwen Qi, Bowen Li, Binyuan Hui, Bailin Wang, Jinyang Li, Jinwang Wu, and Yuanjun Laili. An investigation of lims' inefficacy understanding converse relations. arXiv preprint
1r00523310.05163, 2023. 2023.

---

[28] Ruilin Luo, Tianle Gu, Haoling Li, Junzhe Li, Zicheng Lin, Jiayi Li, and Yujiu Yang. Chain of history: Learning and forecasting with Ilms for temporal knowledge graph completion. arXiv preprint arXiv:2401.0660
2024.
[29] Chulhee Yun, Srinadh Bhojanapalli,, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-squeene functions? arXiv preprint urXiv:99/2. 10077, 2019.
[30] Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of trans- formers recognize formal languages. arXiv preprint arXiv:2009. 1264, 2020.
[31] Satwik Bhattamishra, Arkil Patel, and Navin Goyal. On the computational power of transformers and its implications in sequence modeling. arXiv preprint arXiv:2006.09228 09286, 2020.
[32] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals. Jakob Uszkoreit, and Fukasz Kaiser. Uni- versal transformers. arXiv preprint arXiv: 0338/, 2018.
[33] Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing-complete. Journal of Machine Learning Research, 22(75): 2021.
[34] Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and vari- able creation in self-attention mechanisms. In International Conference on Machine Learning, pages 5793-5831. PMLR, 2022.
[35] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1:1, 2021.
[36] Valerii Likhosherstov, Krzysztof Choromanski, and Adrian Weller. On the expressive power of self-attention matrices. arXiv preprini arXiv:2106.0377 2021.
[37] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint
arXiv:2211. 1,666,, 2022.
[38] Haoyu Zhao, Abhishek Panigrahi. Rong Ge. and Sanjeev Arora. Do transformers parse while predicting the masked word? arXiv preprint arXiv:2303.08117 2023.
[39] Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. arXiv preprint arXiv:2105. 2021.
[40] Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari. Ethan Dyer, and Behnam Neyshabur. Exploring length general- ization in large language models. arXiv preprint arXiv:2207 000000,,,901.... 2022.
[41] Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Fran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764 2022.
[42] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
[43] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. arXiv preprint arXiv:2212.07777 2022.
[44] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statis- ticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306. 04637, 2023.
[45] Shuai Li, Zhao Song, Yu Xia, Tong Yu, and Tianyi Zhou. The closeness of in-context learning and weight shifting for softmax regression. arXiv preprint arXii:2300.132276 2023.
[46] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174. PMLR, 2023.
[47] Bingbin L Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shorteuts to automata. arXiv preprint arXiv:2210.107499 2022.
12

---

[48] Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems, 35:12071-1208, 2022.
[49] Song Mei and Yuchen Wu. Deep networks as denoising algorithms: Sample-efficient learning of diffusion models in high-dimensional graphical models. arXiv preprini 14X 2023.
[50] Licong Lin, Yu Bai, and Song Mei. Transformers as decision makers: Provable in-context reinforcement learning via supervised pretraining. arXiv preprint arXlv:2310.08566, 2023.
[51] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Representations, 2020.
[52] Jiri Hron. Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nogp and ntk for deep attention networks. In International Conference on Machine Learning, pages
4376-4386. PMLR, 2020.
[53] Greg Yang. Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.08 034666 2022.
[54]
Etai Littwin, Emmanuel Abbe, Samy Bengio, and Joshua Susskind. Trans- formers learn through gradual rank increase. arXiv preprint arXiv:2306.07000
Enric
2023.
[55] Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of transformer: memory viewpoint. arXiv preprint arXiv:23006. 00800, 2023.
[56] Samy Jelassi, Michael Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems, 35:37822-37836, 2022.
[57] Charlic Snell, Ruiqi Zhong, Dan Klein, and Jacob Steinhardt. Approximating how single head attention learns. arXiv preprint
Stf1::003307601 2021.
[58] Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.088 03576, 2023.
[59] Ruiqi Zhang, Spencer Frei, and Peter I. Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306. 09927, 2023.
[60] Hengyu Fu, Tianyu Guo, Yu Bai, and Song Mei. What can single attention layer learn? study through the random features lens. Advances in Neural Information Processing Systems, 36, 2024.
[61] Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. arXiv preprint arXiv:2310.05244 2023.
[62] Yuandong Tian, Yiping Wang, Zhenyu Zhang, Beidi Chen, and Simon Du Joma: De- mystifying multilayer transformers via joint dynamics of mip and attention. arXiv preprint arXiv:2310.00535, 2023.
[63] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, October 2020. Association for Computational Linguistics. URL https: / /www aclweb. celweb.org/ anthology// 2020. emnlp-demos. 6.
[64] Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics, pages 1302-1338, 2000.
[65] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 127613 2024.
13

---

A Additional Notations
Let 8 for i j and dig 0 for i # For squared matrix A Rdxd its trace is Tr(A) = Σi=1 Aus- For two matrices A, B E Rm*x of the same shape, their inner product is defined as (A, B) Tr(ABT). For any matrix A E R its (Frobenius) norm is defined as ||||||= v A). For any vector (.1....., the zero-norm as | [2=|o= Ei= 1x, + 0} ||A||o= 2i د 1{A # 03 where 1{:} is the indicator function.
E Rd or matrix A E R we define
B Missing Proofs of Section 3
Theorem (Separation of training dynamics, formal statement of Theorem 1), Fix arbitrary 6, c E (0, 1). Let 01...., Um be independently sampled from Na(Oa, LT). Let 21...., and y1, 01..... Yn be sampled uniformly at random from (V1,..., Um without replacement. Define
((θ)
(og +
Σ
log pe (2ilWA)
2n
()) log pe (21 B1).
Consider the gradient flow O 0
dθ:
di --V(())
where O, ~ ~N(082,07 02. ) or O satisfies 2m
1 <
< 2 for all E [m].
Suppose S I00lu(61m=70] and
nm (2m'(2m) log(64m222/0))
2
With probability at least 1 8, we have
L
((O,))
vt 2 0.
(re)
C(OO)
Proof. Let
200 log (04mm
By Lemma 3 and Lemma 4, with probability at least 1 8,
-8v(2n-1) (2mm)
L(Og) -f
(r0) 2 (Oo)
1
8(2n 1) log" (2m) 80(2n-1) (2m)
[(+))
> (0o)
L(OO)
By definition of d, we have 8m(2n (log2(2m)
Notice that ((O,)
2(8) 1, thus
(L(O.)
L (e.) 2 L rooy (rov
(((e)
Theorem (Lower bound of reversal loss, formal statement of Theorem 2). Fix arbitrary > 0 and C' < log(m/2). Under the setting of Theorem suppose I 100mm(61777 and
log² log(2m) 2m
z n0 log (2m) log 2ogmmmm2 2/2/))
log
log[m/2]
With probability at least - 8.
(Te) 2 C.
where denotes the first time such that L(O,) S
14

---

One may check that the definition of Proof. By continuity, [(O,) = c. By Theorem 5, when
10 'm2 log (2m) 1og(64m²2n) ///))
z
6
with probability at least 1 6,
[(++)
(o,) 2 (Oo)
C(eu)
L (0o)
C(Oo)
Under this event, applying Lemma 6, we can obtain
(++) 2 log(m/2).
log log(2m)
log Ingtemm
To ensure that the right hand side is C, we set d satisfies Eq. (4). It follows that
Tox 12x]
C (0,) C.
B.1 Training dynamics
Lemma 3 (Dynamics of the forward loss). Let ;I, U1 be defined as Theorem When o I 100 ln(6n(22) and d 2 1600n m log (8m2 n2 n/)6). with probability at least 8, we have
....... Yru ((O), and θ (t 2 0)
((O.) S
V 0.
8(2 1J (2m) z(0.)
Furthermore,
inf (zll) it 0, 0.iE [m]]
2m
Proof. For convenience, we assume 21...., Tn 01..... Un and yı. yi...... Un Unt1.... Unn WLOG.
Let
Define 4(O) logpo(vrlar) and (tor(() log pe (2:ll)))) Let a = = (1y(1:) + jij-nn 81 = pe,
400m2 1n Then 5 2/n
( (uslus) + 85-n.- By Lemma 5,
d(()))
de,
V((O,),
= (Z((.))
dt
dt
yz E Epe. (oa)))))) )
1
Zr(1i
E (1x
2n
1
Σ
2n
1
1=1 1=1
*=++2
15

---

Similarly, we have
(O,)
dt
d0: (l,)),, di
- (V4(O)) V((O.))
n +
(loo) )
z;(1: - Lp,, (xolll))
Σ
zily
pe,, (11x;) [v])
Σ
3/: (z;
Epn,
2n -
i=2
aisU:o} 2nn -
1
Σ
n
i.j v
Σ
(())
2050 T
i=+ j=1
and
dt
de,
1 (Ox),
(O,), VL(O.))
dl
Epo, (loo)))) )
y1 (x:
Σ cilWi
E (1x.)ll)))))))
yi(w:
2n
(( itng Vitn Uj.
1
12
m
m
100
Big
2n
Applying Lemma 7, with probability at least
8/2, for any & 2 we have
dl(())
+
Σ
2m
(3()))
)
dt
(2n - 1)2
1=
1=1
(5)
1
ΣΣ im1j=1
m
2m
172
(0
Se (2n 1)2
1m
and
(l,(
1 m
(t)
+ 2n
dt
1 1=1
(6)
1/2
1/2
m
12
m
2n
77
Se.
a
(t)
Σ
M j=1
a
9 2 + Σ
Σ
(3(0)2
2n
j=1
(1z)
1
(g(t)
+
dl
2n
j=l
1/2
1/2
2n
Se
1
Ctn.j
ΣΣ
Σ
2n
/=
i=
Furthermore, Lemma 6 implies that with probability at least 1 1-8/2
2m < pe. (vllz)) (
The following arguments based on the event that the above inequalities hold.
16

---

We first show that
inf (pe,
: 2 2, [m]]
2m
We prove this by contradiction. Let
1 2m
7= =iinf inf t 2 O: 3i E [m], s.t. min (Uilos), pe, (2: (mralb)) -
By Eq. (7), it is obvious that T > 0. Assume without loss of generality that pe, (V1 S 2m follows that there exists 6 > such that pe, (11(z1) is a decreasing function in (T - d, and
1
pe, ( min {pe, (zilv:), pe, (vilzz) ; E [n]] for any E (r T). It follows that for
t E (T d,
1/2
1/2
dl1 (O,) dt
m Σ /=1
m
12
m
2n m
(4)
2 +e.
Σ
(1,))
(1)
2
Σ
10)
Σ
Σ
(0!))
2n
j=1
2
1/2
1/2
2 +
Σ (1
)²
2
(a iitn
(0)
2 +2
(8,00
2n -1 1
/=1
i=1
ビ=ー-
1/2
1/2
(t) 2
Σ (at
+c. (4n(af.at1))) 1/2
2n - 1
j=1
00
where the first inequality is from Eq. (6); the second inequality is due to Estttnlaaf a
1 pe. (x;,, Enzı | { Pitn,
pe, (Ivl3s) for all i E [n]; the third inequality is because pe, (y1 (x1) = min {pe, (z. pe, (ull=,) : i e [n]]. However, pe, (2l/)1)) is a decreasing function in (T - 6, r), contradiction. Therefore, we conclude that
inf (pe, (rill))) (x.) :t 2 0, E [n]] 2m
Now we show
((O.) <
Vt 2 O.
+ I(0O)
8(2n 1) 100-1) (2m)
By Eq. (5),
dl((.)
E
m
2n m
(3/0)2
<
dl
(2n 1)2
c
Σ pe, (v: 12 + > (1 pe, (2)
$
(2n 1
i=2
2
n
pa, (x:))
(1 pe, (z:l9s))
(2n 1)3
1 E
8(2n - 1) log (2m) 2(0,)2
2(0))²
8(2n - 1) log (2m)
where the second inequality is due to Ejtitn | o,en 1 - pe, (vilaa)) Eytilo(fn
Pitni = -1- (z.ll))))) for all i E |n|] the third incquality applies Cauchy-Schwarz incquality;
17

---

the last inequality uses the fact that pe, (vsl2;)) >
1 2m
for any t 2 0, E [n]] and the
log inequality 1 2 [og(1/((m)) for æ E 2m 1).
By Lemma 10, we conclude that Vt 2
((O.) S
+ Z
812n- 1) log²
which completes the proof.
Lemma 4 (Dynamics of the reversal loss). Let , Ine U1.... be defined as in Theorem S. When a S [0066n(6))) and d 2 400m m log(8m??? /0)/e² with probability at least 1- 8,
yv. (O), and O₁ (t 0)
(0,) 2 ())) (Oo) (1+ ((90) t 8(2n 1) log (2m)
-e(2nn (2m)
Similar to Lemma 3, we assume 21,..., In = v1 Un and 1/....., yr = Un+1..... U2n WLOG. Let a = -pe, (uglus) + 8i 83) -PO, (1yl0s) d di-n.j. By Lemma 5, dL dt
Proof.
Z (Ot); d0.
dt
n
n
(1 p- (-111)))))) [x])] 1 22n
zz(y:
F pe, (112.) (v])
+
y1 (xx
F Fre (- (ze))
m
1+,, 1n+10 T
1
Σ
n
m
(0)
Σ
2n
m
300
meu
2n -
1=1
(=n-2 j=1
Applying Lemma 7, with probability at least
8/2, for any & 2 we have
1/2
1/2
d (0,)
S
1
Σ
(g(0)
xt1,j)
Σ
(0
(8)
dt
2n
i=1 j=1
and
dL(O,)
+
2
Σ
2n
(aft)
dt
(2n - 1)2
1=1
2 1=
Se. (2n - 1)2
1
Σ
a
m
2n Σ
Σ
m2
(80)) 2
/=
j=
as well as
(1)))))
1
(
dt
2n
1/2
2n
<
E.
(1)
(9) +
Σ Σ
(0(0))
2n 1
1=1 j=1
i==+2j=1 j=1
aally (0:)
1
+
Ditti) )2
dt
2n
1
1/2
1/2
1
(t)
m ΣΣ
Σ
2n
TTL
(B()))
2n
i=
j=1
18

---

Furthermore, Lemma implies that with probability at least 1 - 6/22.
1
- peo (Yil (2:), (v)l
pe, (1: (16))
2m
m
The following arguments are based on the event that the above inequalities hold.
By Eq. (8),
1/2
1/2
d. (+.) dt
Σ
(
n
m
Σ
2n
M
(5500) 2.g
-
2n
i=1 ==1
1= f=1
Notice that
B, 2 >0 (1 pe, (vil + > (1 pe, (xil))))²
2m
2m
2 Σ (1 po. (u1
+ - (1 pe, (2:lvi))
1m1
2m
$ 2(2n 1)² (
2
S 2(2 1)2
+ Z(Oo)
8(2n log (2m)
where the first inequality uses Ejpitn (0
- Oittn
(t)
pe, (y: (z)), Ejzi 3:0
1 -ep (ry (1) for all E [n]: the third inequality uses the fact that 1 x S - log x; the last inequality applies Lemma 3. Similarly,
A, s 2 - pe. < (
Combining, have
d.
2 - 2n - 22, (2n 1)
dl
+ Z(eo
8(2n 1) (2m)
2 8c(2n - 1) log? (2m) (Ot)
8(2n log (2m)
+
((5
By Lemma 10, we conclude that Vt 2
C (Tem) (0o) (1+ 8(2n 1) log² (2m))
((Og) .t
Sc(2n 1) log loogo(("" (2m)
This completes the proof.
Lemma (Gradient of the loss function). Define
((0) 2n
log (y)l)x)) Σ
n
log pe (mill:))
(O) log pe (x11)))1)
Then we have
VL(0) 2n -
zelyi - F (p((1))ll)
+ Σ (xt - F [x)) [=a)]
CL 21(11 [xo())x))(x))
19

---

Proof. We have
(--loge log pe(pe)))
= [pe(ylz)
ry exp(x Ovy (Enev explx Oiy) exp(x Oyy Dy exp(x Yy)
(
EvEN (xp(xT0y)) )
po(vlz)ayT po(y) Σ (o(l)))ay
22(2)
x y- Σ pe (e(()>)y yEV
x (y Ey-pe(( [y]
The statements follow immediately.
B.2 Initialization
Lemma (Initial distributions are all close to uniform). Fix any 8 (0, Let 31 ...... Fn i.idd
Na La). Let θ E Raxd, where each Oij" i.d. NN0, independent of #1,..., In. For any i [n], define
pe(x)|æ;)) = exp(le(x,1zz))
where (e(zg(z1) = x; O.,
k== exp(le(ralzz)))
Then when o²
100 and d 2 400 log(lg(1] (2ón2)) with probability at least
1
[pe (x.) 1/nls 2n Vi,j E [m].
Proof. Let 0 = 0.2. By Lemma 8, with probability at least - 6, we have (x,, 0.g v Vi,j E [m].
Conditioned on the above high-probability event, we can further obtain that for any j E [n]
(x)(1e(2)1x:)) pe(2)(z,) En exp((e((x(xr))
exp(v)
exp(2e)
Σk=1 exp(-o)
and
pe(2)(z5) = exp(le(27(1;)) exp(-u)
exp( (2u
E (xp(le(xx(x;))
1= exp(v)
It follows that
3
< po(z)(zz))
= -
-|/n|< 2n'
2n
2n
which completes the proof.
B.3 Subspace embedding
Lemma (uu-subspace embedding of Gaussian second-order tensors). Let 21....,27 Zn be indepen- dently sampled from Na(Oa, hLa). Let In S [n] x [n] be two index sets. Let Io = In n Iz. If z 64 log(2n? (og(2n2/8)/e² then with probability at least - 8,
Σ aiga2z Σ B1a2i2] (NJJ)I
Σ (I.1)
ang
Se.
Σ lawl (KU)EI,
1Ba
(NJ)EIz
(d.))
20

---

holds for any ar. Bi.g. Furthermore, if d 64k log(2n2) (onn//6), /0)/e, then with probability at least - 8,
Se. ( Σ Lis
1/2
1/2
Σ 0i
Σ Bisziz]]
Σ
aigBis
Σ
Bi
(7.))
(T
(T.)ET
(1.))T
(A-))T
holds for any an Br, such that ||allo5 k. 11 k.
Proof. Using Lemma 8 and Cauchy-Schwarz inequality, we have
Σ
Σ
BiaZza] )
Σ
aig
(1.))T
(T,1)
(id)
ans 0
11,1
E
015 2x2] 21
(
64 (4llo(2n22/)
Σ
lai
E
laasBkdl
d
(b,))Z
(k.))
64 (4log(2n2/)
Σ
Σ
|Basl
d
(i,))I
(K,))I,
1/2
1/2
64k2 log((222
Σ
(i.))
The statements follow directly from plugging in suitable values of d.
Lemma (Almost orthonormal). Let Fı, .f,.....2. iid. Na(Oa, d 2 16 16log(2n² /6)/e, 2, it holds that with probability at least - 5,
3 Ta). For any e, 8 E (0, 1), when
(x,, 8.y15 c, Vi, E [n].
Proof. Fix i j j E [n]. Notice
(2 22) - Ka, + Zy, x2) (x; Ty. 1/4
I (Kxr; + Ty + x.)) 1| (2: - Tj. 24 2z)) - 1))/4.
Using Lemma 9, we have that with probability at least 1 8/n², [(Rt, c.
Furthermore, fix i E [n]. with probability at least 1 - 8/n2,
Ka,, - 1/5 6. The statement then follows from union bound over i, [m].
Lemma (Almost normal for a fixed vector). For a d-dimensional random vector x ~ Na La) and any (0, 1/2).
P (((a, - 1/2 v) < 2e v2d/18
In particular, when d 2 =61og(1/(66)/o²² we have P (((2,)) 1/2 v) 8
Proof. Let x.
By Lemma 11 (letting x = u2d/16))
d
P ((((x 1/2 v) P
Σ V
2 vd
< 2e Pdd/6
The second inequality follows from simple arithmetics.
21

---

B.4
Useful results
Lemma 10 (ODE bound), Let c1, 02, C3 > 0. Suppose the function f1. R_ R satisfies
f1(0) > 0, f2( > 0 and
dfi(t)
dt _ - c1 fi(t)²
df2(1) 2 - C2 f2(t) t
dt
Then
fi(t) S
ci + Ji(0]
I
S2(1) 2 52(0) (1+
02
C3
Proof. The conditions imply that
dfi (t)
1 dfi(t)
(1 2 c1,
d log f2(t)
1 (f2
2 c2/((t ez).
dt ft(t) dt
It follows that
et f1 log fa(t) 2 log + t/c3) + log f2) Rearranging the above inequalities, one can obtain the desired results.
2t)
Lemma 11 (x2 concentration bound, Lemma of [64]). Let .......9 be i..dd N(0, 1) random variables. Then for any x 2 O.
Pr
oe 2 1 ++ 2V1x 1x 2x < exx((=x),
and
Pr
of <t
2 ta S exp(-xr).
C Missing Proofs of Section
In this section, we show missing proofs in Section 4.
C.1 Proofs of Section 4.1
C.1.1 Proofs of Proposition 4.1 and Proposition 4.2
We first show the proofs of Proposition 4.1 and Proposition 4.2, respectively.
Proof of Proposition 4.1. Actually, for any three tokens Ii, 22, T3. it holds that
Po(o) (2g = exp (x. LO) b2))
exp (0)
[EMM exp (0) 1/M,
ExeMM exp a. ((O(())L)(XT6
since Y())= 0.
22

---

Proof of Proposition 4.2. Note that the input sequence length T = 2. By (I),
exp (x T TIL) LL(XX b2))
(=(
EVE(M] exp (x' (((t))LN(XTO))
where be = [612] and b12 = 1. Also. X = [x1]" is one-hot row vector. Therefore, LN(X (²) = LN(x1112) (N())) 21. and thus
exp (a ((t x1)
exp (t))),...)
E exp x'' (t)
Exte(ar) exp (Y())aa)))
where (t) is the entry of the matrix Y (t) at row i and column
C.1.2 Proof of Lemma 2
Proof of Lemma 2. We first calculate the gradient of Y when the current batch is a sequence (I1, I2, T3). Note that the input sequence length T and by the proof of Proposition 4.2, have
have LN(X (T) LN(X b2) 21 By Lemma
Y = 7y (2)( a) = a - ax)
where a = [01, 02,,. [ T E RM with a exp (YTT) / exp (YZ)). Note that an (æg ax) T matrix with only -h non-zero since 21 one-hot. Therefore, the update of each row of Y are independent and only æ -1 row of Y gets updated at the current time step.
Now consider any fixed 2 E V. Let tæ be the time step that the -1t row of Y gets updated (i.e., the first token of the training data 21 for the current batch) for the i-th time and let t = for notation convenience, then
(t2n)))) = Y (t2 + my - =a)
For convenience, we denote yi) = Y ((z),)) and thus
(i) y(i 1) + ( aiii - 1))
(10)
exp(y(i 1)). Note that for a fixed 21, By is where y(0) 0 and a(i - 1) = exp(y(i - 1))/1" 10 also fixed by our construction of the dataset. By Lemma of [14], we can obtain that
y(i) (M 1)h" (i)
where Exy =
M- M [3'
1 1) and h² (i) can be derived recursively as M
TY h" (1) h" 1) (M - 1) exp(Mh
with h² h0) 0. Combining Lemma 7 and 9 in [14], we have
M
h h( K M In(Mny VIZ
(11)
Note that the update of each row of Y are independent, the training set has size N, and the batch size 1, we have
(t),, (M 1)h² ([t/NT) Exs where the training data at time step t (I1, I2,
Tz). x2, Combining (11), can obtain that
M1
1
Mnyt
(1) 2 (M- 1)
In (MYY [t/N]] In
M
M
M
N
and
M
1 In (Mry [(/N]) S
1 (iiyl M N
Y() SM M
(- M (
In
Yx + 23.
M
On the other hand, for any sequence (2,) in the test set, since the æ1 -th row of Y has never been updated, we have
(1))2 = ((0)a = Vo [M].
23

---

C.1.3 Proof of Theorem 3
Proof. We first consider training sequence (#1. 22, 4is) at time t. By Proposition 4.2, we have
exp (Y (t) Pa(c) (2g = Erecm exp (t()),..))
and by Lemma 2, we have
(() 2 eln
Mnyt
and
Y(Day S M
c In
Mnyt N
Vr # x3
N
for some constant c 0. Therefore.
[cIn Mayt (>N²
exp
Po(e) (23)211
exp cIn Mnyt
+ (M - 11) (-ni In May!
(Myy)) No
Mnyt N
+ 11
M-1
Mnyt
+ (M
M 1
21
[Myxt' N
where the last inequality holds since
Mnyt N(M - 1)
Int 2 In(NM/nr)
2 M
Mny
N
Finally, for any sequence 22, Dress since Y()ay, 0, Va E [M] according to Lemma 2, we have
exp (Y (t)21.3)
exp(0)
Po(1) (ra x1, x2)
M Meexp(0) 1/M,
Exte(an) exp (Y())),)))
which completes the proof.
C.2 Additional details and proof of Section 4.2
Datasets. Let Numin > O, Niet 0 be two positive integers and let Nootal Numin + Nust Let A; B,, C, V, Vi E [Nonal] be 3N distinct tokens. Let-,, > V [M] be two additional differ- ent tokens that represent "direct implication" and "indirect implication" respectively. Specifically, we have A + B,, B, - C, and A; ~ C, for E [Noua] For notation convenience, we define the following two index sets
Iurin = (1,2,... ..ainn.,Nuai) Irst = (Nurain + 1,ut.,
The training set D Draai consists of all A; + B,, B; + C, and A, C, for i E Ierain In addition, Draain contains A; + B; and B, + C, for i E Zreat For convenience, we let N [Drainl to be the size of the training set. The test D lest consists of A, ~ C, for i Irest Under the LLM will learn the relationship between A,, B, and C for E Iaraia in both direct and indirect implication, and learn the relationship between AG, B, and C, for i Iest only in direct implication and will be tested for indirect implication.
construction of the dataset,
Similar to the reversal curse in Section 4.1, aim prove through the training dynamics of one- layer transformers that the test probability remains negligible during training. In particular, we are interested in
(2 = Ci|z1 = A,, =.+), i Iest-
We also (B: +), po C. +) and (C. A; ~a) more compactly represent po(33
Bi|rn = As, 22 = Po (I3 = 2 = B,, x2 = and po = C,|21 = A,, 22 = respectively.
The following theorem shows the importance of the chain-of-though method:

---

Theorem 7 (Importance of chain-of of thouth,, formal statement of Theorem 4), Assume we run SGD 1 with batch size I, and assume M > 100 and Mas & " < 1. Let 2 N denote the time step which also satisfies In l In(NM/MV) For any test index i E Tist have
1
M
Pott) (B: A, + 1
(o(o)(., +) 2
2 Mayt )
Mayt
for some constant e > 0 and
Po(t) ((te(Ca)3 (G|4: -4) M
Proof. Recall that by Proposition 4.2, we have
expl (Y (t)
Po(1) (2a)
Exec exp (Y (()))a)))
and by Lemma 2, we have
Marl and (()),, S
Mirb
A. 2 cln
Vx + B;
M
N
for some constant 0. Therefore, using the same proof Theorem 3,
have
Pa(e) (B,IA, +) M-
Mayt
Additionally, according to the proof of Lemma 2, (()),,. has the same value across all # B, which implies
(C ~) ~ M
Similarly, applying Lemma 2 to (t)a,,
obtain that Pa(t) (C, +) 2
M
((MM²)
Mayt
C.3 Analysis of the reversal curse for the four-token sequences
In this section, analyze the reversal curse where data points four-token sentences "ARyRa R2 "BR, RgA". For each sentence, A and B are two distinct tokens that represent two entities, and R1, R2 are two special tokens jointly representing relationship that is inverse to itself (e.g., R, represents "is the friend of",, then AR, R2B means "A is the friend of B" and BR, R2A means "B is the friend of A" ).
Let Nirain > and Nia > 0 and denote Numai Nirain Nies- Let A,, B, V. Vi E [Nioal be K 2.N distinct tokens representing distinct entities. WLOG, we assume Ag. B, E [K], Vi E [Nwoul] Let R₁, Rg E V be two additional different tokens that jointly represent a relationship that is inverse itself. Specifically,
Datasets.
have R2B, and B,R, RyA; for all i [Noual] For notation convenience, we define the following two index sets
Iarain
[Nrain]
I = [Nocaal T
The training set Dimin consists of all R,R, and B,R, for E Tarin- In addition, Durain contains A,R, R2 for E Tust- For convenience, we let N PDrainl to be the size of the training set. The test set Drest consists of B,R, RzA, for E Ijest Under our construction of the dataset, the LLM will learn the relationship between A; and B, for í E Taaiin in both directions to deduce that the relationship "R, R₂" " is reverse to itself, and learn the relationship between A; and B; for E Irea in one direction and will be tested for the other. WLOG, we assume each training sequence (x B1, 3 R2, x4) E D
IA [N]]
25

---

Now we assume that the learning rate Tίy > 72 and therefore can treat X ' as fixed when analyzing the dynamics of Y. For any sequence (I).25 22 R 23 R2 24 n) in training or test
dataset with T define fn LN br). Then the gradient of Y in (3) becomes
= NY In a) fn - a))
Note that for three-token sequences, fr. is a one-hot vector, and thus Y has only one non-zero row, and each row of Y can be analyzed independently. For the four-token sequences, we use the same reparameterization strategy as in [14], where we denote W = [1., W2 wK] & F RM*K
RK* with F ........ fkl | f
Note that the parameter W can be viewed as a combination of Y and Z where is fixed. The following lemma shows the dynamics of W, assuming we are performing gradient updates W instead of Y.
Lemma 12 (Dynamics of W). Assume we perform gradient updates directly on W instead of Y with learning rate тy and batch size 1, and assume M >> 100 and M0..05 & nyy 1. Lett 2 Nh
and let (t), denote the i-th row of W and W (t);s denote the (i,5)-th entry of (t). Thenfor training sequence (I1, R1, B2, TA) E Dirain at time t, we have
(f 2 In (MYyt t/N), and W()) S - In (M t/N) /M, and for any test sequence (I1, R1, R2, v.4) E Drem, we have W(1)24,2 = 0, Vx E [M].
Vr + z4.
Proof. According Lemma 3 of [14], for training sequence (I11 R2, x44) D Parain time i, only the -y row of W will be updated and the gradient
1gg (t) = f Oz. (1))
where (t) ex (1) exp(wx. (f))). Therefore, the dynamics of W nearly identical to the dynamics of Y in Lemma 2, and we can use the proof of Lemma 2 to conclude the results.
With the dynamies of W, we can obtain the following result:
Proposition C.1 (Reversal curse for the four-token sequences). Assume we perform gradienl updates directly on W instead of Y with learning rate Ty and batch size and assume M >>> 100 and Mass & nγ < 1. Let t 2 Nla denote the time step which also satisfies Int 2 In(NM/ny). For
ny
training sequence (I1, R1, R2, 2.4) Drrain at time t, have
M-1 (Myyl/Nye
(ett)(Eal21, R1, R2) 21
as
for some constant and for any test sequence (x1, R1, R2, 2.4) Diest that is not included in the training set D priin we have
Pets) (p)))(aa/at) (1,, R1, R2) 1 1/M.
Proof. For any sequence (I1, 22 = R1, E3 = R2, where T =,
exp (x Y LLN(X br))
Po(o) (2 R1, R2)
EM'EMM exp x Y ((()) br)
exp (x (
EvE(M) exp xx "T((t) Fra)
exp (x" (t))
Ex'E(M] exp exp (W(t) (t)
x (TTTuns "aa (t)
ExeEM exp ( (1)
The above next token probability formulation is almost identical to Proposition 4.2 after replacing Y with W. Combining the dynamics of W as shown Lemma 12, we can use the proof Theorem to conclude the result.
26

---

Finally, we discuss the role of Z in the four-token sequence settings. Note that Proposition C.1 assumes gradient update on W instead of Y. While we are not able to perform gradient updates on W directly, it is equivalent to modifying the gradient of Y to be
== my - FE,,,,)),, - an))
according to Lemma 3 of [14], where the next token 47+1 n, F² = -[--))--1-==FFP-
and F == [fi:.... fN] E RA which only contains the next token that appears in the training set. Compared to the original gradient of Y
y my (e. - -an),
one can obtain that the modification of the gradient Y is small if A (E) small.
Note that Eg | 1 = 0 and Eig f. fz. Also, for any sequence (x1 nt,, 22 R1, 23
R2, x44 = n2) in training set,
bisen + b23ea,
fn = LN(X br) (
V vbs +22
where
b13 = exp( exp( )) ),013 = (
Note that is common token that appears in each training sentence, and n' is a distinct token that only appears in one training sentence as a contextual token. By Theorem 2 of [14], under certain technical assumptions, ZRa.n' > 0 and ŽR,N < 0 and one can expect ZRa.n' ZZ, to be sufficiently large after sufficient time of training. Therefore,
fn bizen b23eR,
with bes close to 0. Consider a simple case where for each n E [N], fn = -11 - eceen + cea, for c sufficiently small. Then
E, -VI cea ( cee ceaa) = Therefore, one can calculate that (1 = CN - 1). When < 1 thus the gradient update of W and gradient update of Y are almost the same.
VN. we have (E < 1, and
D Experiments for chain-of-thought
In this section, we conduct experiments for COT multi-layer transformers to validate theoretical results in Section 4.2.
Dataset construction. Similar to Section 5, we randomly sample three disjoint sets of entities A, B.C. V, and reverse two additional tokens for - and ~ respectively. Next, we specify bijection from A to B, and a bijection from B to C' raadomly. For each A, E A and its corresponding B, B and C, e C, we can obtain a triple of sequences (A, - B,, B, + C,, A, ~ C,), and split the set of all triples into training triples and validation triples. All three sequences of a training triple will be added to the training set, while for validation triple, we add A, -> B, and B; -> C, to the training set and add A; ~ C; to the validation set. Therefore, the model will learn both direct and indirect
implications for the training triples and only learn the direct implications for each validation triple while being tested on the indirect implication.
Results. Figure 3 shows the experiment results for COT using the same model architecture and configurations as in Figure 1 (the training set size is 540, and the validation set size is 60 resulting from 140 training triples and 60 validation triples), which is consistent with Theorem 7. One can refer to Appendix E.3 for additional experiments with various model configurations and vocabulary sizes. We also empirically validate the intransitivity of model weights (i.e., logits) for multi-layer transformers in Figure 4, which shows that for a validation triple (A,, B, C;) of which only the direct implication "A, + B," and "B, - C," appears the training set, although the weights from A, to B; and from B; to C, are trained large as indicated by the diagonals of the first two bottom matrices, the weights from A, to C, gets hardly trained as indicated by the diagonals of the last matrix. We also emphasize that another reason that COT' is necessary is that all tokens A,, B,, and C; are different tokens with randomly initialized embedding and thus irrelevant. When these tokens are relevant and show specific patterns, the validation loss can also get better. See more details in Appendix E.4.
27

---

Negative Lag Probability over Epochs
Train
80
Val
Negative Log Probability
60
40
500
1000
1500
2000
2500
3000
Number of Epochs
Figure 3: Experiment results of COT under default configuration (see Table 3). The curves represent the (average) negative log probability of the model predicting the next token be: (1) B, given the input "A; - (2) C, given the input "B; -",, or (3) C; given the input "A, ~~", Similar to the reversal curse experiment, while the sentences the training set can be learned nearly perfectly, the model is not able to predict the correct next token in the validation set better than a uniformly random guess.
Both curves are averaged over 10 random seeds.
E Additional Experimental Results
In this section, we show additional experimental results for Section 5,
E.1 Details of model architectures and hy hyperparameters
For both the reversal curse and COT experiments, we used the GPT2 model architecture [63] and trained the model with the AdamW optimizer for 3000 epochs of batch size 64. See Table for full list of hyperparameters. We also conducted experiments under various model configurations and vocabulary sizes to show that the results in Section 5 and appendix D are consistent under different settings. See Table for complete list of different configurations, where the default choices are boldened. For each curve in all figures, the results are averaged over 10 trials, and the error bar is calculated using standard deviation. We run each trial on an Nvidia A100 GPU and it typically takes 0.5- hours for each trial.
Parameters
Values
Learning Rate
0.01
Weight Decay X (81,82)
0.9
(0.9, 0.999)
Batch Size
64
Number of Epochs
3000
Table 2: Full list of hyperparameters for AdamW optimizer and training.
E.2
Additional experimental results for the reversal curse
In this section, show additional experimental results for the reversal curse under different con- figurations, including different vocabulary sizes (Figure 5), different number of layers (Figure 6),
Apache License 2.0
28

---

Logits given (Train)
Logits given (Train)
Lagits given (Train)
25
Logits of given (Val)
Logits given (Val)
Logits of given (Val)
-25
550
775
Figure 4: Visualization of the weights (logits) of the model with default configurations trained after 3000 epochs for COT experiment. The matrices are similar to Figure 2. The row tokens for the top matrices are A,, B;, A; and column tokens are B, C, C, for training triples respectively. Similarly, the bottom matrices correspond to validation triples. For validation triples (A,, B,, C;), the weights from A; to C get hardly trained indicated by the diagonals of the last matrix.
Parameters
Values
Number Layers
12, 24, 48 12
Number of Heads
Vocabulary Size
20, 50, 200, 800, 2000 2, None, Absolute, Relative Learnable, Frozen
Entity Length
Positional Encoding Type
Token, Positional Embedding
Table 3: The list of different configurations for experiments in Appendices E.2 and E.3. Default choices are boldened for each row.
different positional encoding (Figure 7), different entity lengths (Figure 8) and whether token and positional embeddings are trainable or fixed (Figure 9). Our experimental results consistently show that the reversal curse happens under different settings.
We provide additional experimental results that (1) the reversal curse still happens even if the embedding dimension is much smaller than the vocabulary size in Appendix E.2.1; (2) the embeddings of different tokens are nearly orthogonal; (3) the reversal curse does not happen under the in-context learning settings.
E.2.1
The reversal curse under small embedding dimensions
Although for theoretical analysis Section 3, we assumed the embedding dimension is polynomial in the vocabulary size, in practice, the embedding dimension only needs to be the order of logarithm of the vocabulary size. Figure 10 shows that for a much smaller embedding size, the reversal curse still happens.
29

---

Negative Probabllity over Epochs
Proba
60
Negative Lag Probabiltty
Val
Train
Probabliity
550
Val
Train
40-
80
30-
30
20-
10:
500
1000
1500 Number of Epochs
2500
3000
1000
1500
3000
Number Epachs
(a) Vocabulary size 20
(b) Vocabulary size 50
Negative Probability over Epochs
Negative Log Proba bility
Negative Log Probabiitty
5o.
Negative Probability
80
Train
Val
40-
60
-30-
Train
Vai
20-
20-
20-
500
1000
1500
2500
3000
1000 1500 Number Epachs
3000
Number of Epochs
(c) Vocabulary size 200
(d) Vocabulary size 2000
Figure 5: Results for reversal curse for different vocabulary sizes. All other configurations default values as in Table 3. The training set sizes for the above four experiments are 9, 20, 85, 850 15, 150 respectively.
respectively, and the validation set sizes
Negative Log Probability over Epachs
Negative Log Proba billt over Epochs
Negative Log Probability ity
Negative Log Probagility
60
Val
60-
40
20-
20
Numbe of Epochs 25000
2000
3000
Number Epochs
(a) 12 layers
(b) 48 layers
Figure 6: Results for reversal curse for different numbers of layers of the transformer. All other configurations are set as default values in Table 3.
Neg ative Log Probability over Epachs
Negative Log bi lity over
70
Negative Lag Probability
Train
Probalolityy
Train
50
Val
60
Val
50
40.
60
30
20
Jeggttiv
20
30
10
500
1000
2500
3000
1000
1500
3000
Number Epochs
Number Epochs
(a) No positional encoding
(b) Relative positional encoding
Figure 7: Results for reversal curse with positional encoding relative positional encoding, For relative positional encoding, we follow the Rotary Position Embedding (RoPE) method proposed by [65]. We use the implementation of this repo, MIT license. All other configurations are set default values as in Table 3.
30

---

Negative Probabliity Epochs
Negative Log Proba bility Egochs
80-
:50
Train
Negative Probalility
Train
Negative Lag Probabilitg
Train (word)
Train (word)
60
AA MM HA hwy
Val
20-
(word)
(word)
40
20
500
1000
1500
2500 3000
1000 15:00
2000
3000
Number of Epochs
Number Epochs
(a) Entity length 2
(b) Entity length
B; consists of multiple tokens and different entities may have overlapped tokens. The "Train" curve represents the negative log probability of predicting the first token of the output entity, and "Train (word)" represents the negative log probability of predicting all tokens one by one of the output entity. All other configurations set default values in Table 3. The training set sizes for the above two experiments are 680 and 250, respectively, and the validation set sizes are 120 and 50, respectively.
Figure 8: Results for reversal curse with different entity lengths. Each entity
Negative Probebility Epochs
Train
Nogative lag Prbbaiiiit
40
1500
3000
Epochs
Figure 9: Results for reversal curse with fixed token embedding and fixed positional embedding. All other configurations are set default values in Table
F.2.2 Near orthogonal embeddings
Note that Section 3, our analysis relies on the fact that embeddings different tokens are nearly orthogonal, and in Section 4, the embeddings are effectively one-hot. In Figure 11, We show that in practice, even the embedding dimension much smaller than the vocabulary size, the near orthogonality condition still holds.
E.2.3 The reversal curse does not happen in ICL settings
We also emphasize that the reversal curse does not happen ICL settings, which means if "A - B" is provided as part of the prompt, then the model is able to answer "B <- A". Figure 12 shows preliminary results of ICL. All the sentences in the dataset have the format "A,RB, + B,R B,R-'A,", A,", which is seven-tokes token sentence where and R- a pair of relationships inverse to each other, and "e" is another reserved token representing equivalence. There are ten different By and different A; where = 100 for the left figure in Figure 12 and n = 200 for the right figure. For each A,, we construct ten sentences using different B₁ and we randomly chose three of them to be included in the validation set and seven other sentences the training set. The result of Figure 12 shows that the reversal curse does not happen during the ICL setting.
31

---

Layers:
Layers:
Layers:
Nesptive Log Probabilty
100
Train
Regativ Log Probabiltty
Train
Negative Log Probabiltt
Train Val
Hunth Nort
60
40-
40
20-
20
1000
1600
2003
2500
3000
2000 2500 500
1000
1500
1000
550
1000
1500 2000
2500
3000
Number of Epachs
Number Epochs
Number Epochs
Figure 10: Results for reversal curse for different numbers of layers and different embedding dimensions. All other configurations are set as default values as Table 3.
Num Layers: 1, Embed Dim: 20
Num Layers: 1, Embed Dim: 768
Num Layers: 24, Embed Dim: 20
0.25
0.00
-0.25
0.50
-0.75
11.00
Figure 11: A heat map of cosine similarity between token embeddings (all A, and B₂) after 3000 epochs training under different settings. The settings are the same Figure 10. Under different numbers of layers or embedding dimensions, most of the non-diagonal entries are close to 0, which shows that the embeddings of different tokens nearly orthogonal.
E.3 Additional experimental results for chain-of-thought
In this section, we show additional experimental results for COT under different configurations, including different vocabulary sizes (Figure 13), different number of layers (Figure 14), different positional encoding (Figure 15), different entity lengths (Figure 16) and whether token and positional embeddings are trainable or fixed (Figure 17). Note that our experimental results consistently show the necessity of COT under different settings.
E.4 Chain-of-thought with relevant tokens
In Appendix D, we briefly mentioned that the irrelevance of different entity tokens is one of the reasons that COT is necessary. Now, we show that if the entity tokens are correlated and show specific patterns, it is possible for model to deduce indirect implications automatically.
Instead of using single tokens A,, B,, C, to represent each entity, now we two tokens Ai, Bi, represent entities, where A, B and C are three common tokens shared by each triple, and token i are distinct for each triple. Figure 18 shows that for the above version of COT where tokens in the same chain are correlated, the model is able to "deduce" Ai ~a Ci after training on Ai - Bi, Bi + Ci and other training samples.
32

---

Vocab size: 400
Vocab size: 800
Train Val
Train
Negative Log Probability
12
Negative Log Probability
Val
12.5
8
6
2
2.5
200 400
600
800
1000
200
400
600
800
1000
Number of Epochs
Number Epochs
Figure 12: Training and validation loss under in-context learning (ICL) settings, All sentences consist of seven tokens and have the form of A,RBB + B,R B,R-'A,". For the left figure, the training size is 700, the validation size is 300, and the vocabulary size is 400; for the right figure, the training set size is 1400, the validation set size 600, and the vocabulary size is 800. All other configurations reversal curse does not happen in ICL settings, i.e., if RA, is provided as part of the prompt, then model is able to recognize "B,R -,A""" A,""
A The loss calculated on the last token.
default values in Table 3. The result shows that the
Negative ative Log Probability over Epachs
Negalive Log Probability over Epochs
Negalive Log Probabiitt
Train Val
Negative Probabbllit
Val
60
400
2500
Epochs
Number Epachs
(a) Vocabulary size 20
(b) Vocabulary size 50
Negative Slive Log Probability over Epachs
Negative Log Probability bility over Epochs
Negative Log Probabiityy
60-
Negative Probab-llty
50-
80
Val
60
Train
40
20
30-
20-
2500
Number Epochs
Number Epochs
(c) Vocabulary size 200
(d) Vocabulary size 2000
Figure 13: Results for COT different vocabulary sizes. All other configurations are set as default values as in Table 3. The training set sizes for the above four experiments are 14, 32, 135, 1350 respectively, and the validation set sizes are 4, 15, 150 respectively.
33

---

Negative Log Probability Epochs
Negative Log Proba bility Epochs
Negative Log Pro Prebability
Train
Negative Log Pro baballlyy
80
Train
Val
Val
60
40-
-00
20
500
1000
1500
2000
2500
3000
1000
15:00
2000
2500
3000
Number of Epochs
Number Epochs
(a) layers
(b) layers
Figure 14: Results for COT for different number of layers of the transformer. All other configurations are set as default values as Table 3.
Negative Log Probability over Epochs
Negative Log Probability over Epochs
Negative log Probability
80
Train
Negativee Leg Probability
80
Val
Train
60-
60
40-
20
500
1000
3500
2000
2500
3000
1000
1500
2000
2500
3000
Number of Epochs
Number Epochs
(a) No positional encoding
(b) Relative positional encoding
Figure 15: Results for COT with no positional encoding or relative positional encoding. For relative positional encoding, we follow the Rotary Position Embedding (RoPE) method proposed by [65]. All other configurations are set as default values as in Table 3.
Negative Log Probability over Epochs
Negative Log Probability over Epochs
Negative Log Probabality
Train
Negative Log Probabilty
Train
(word)
(word)
Val
Vil
20
Val (word)
20
(word)
2.-
20.
Number Epochs
Number Epochs
(a) Entity length
(b) Entity length
Figure 16: Results for COT with different entity lengths. The setting and curves are similar to
Figure All other configurations are set as default values as in Table The training set sizes for the above two experiments are 1080 and 400, respectively, and the validation set sizes are 120 and 50.
34

---

Negative Probability Epochs
Train
50
40
Nogative Log
20
Epochs
Figure 17: Results for COT with fixed token embedding and fixed positional embedding. All other configurations are as default values Table 3.
Negative Probability Epochs
Negative Probability Epochs
Negative Log Pobabiitt
Negative Lag Probablitty
80
Val
Val
45
20
40
500
1000
1500
2000
2500
3000
500
1000
1500
2009
2300
3000
Epochs
Epochs
(a) Training set size 160
(b) Training set size 250
Log Probebility Epochs
Negative Log
Negative Log Prebabiilty
Train
Negativel Log Prodiability
Train
1.5
Val
45
Vel
500
1,00
1500
3000
2500
3300
500
1000
1500
2000
2500
3000
Number of Epachs
Number Epochs
(c) Training set size 400
(d) Training set size 1600
Figure 18: Results for COT where each entity represented by two tokens, i.e., Ai, Bi, or Ci. The validation set sizes are 50. The model is able to "deduce" unseen Ai ~ Ci by learning underlying patterns.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract and Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Sections and 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
36

---

Justification:
See Sections and 4 and appendices B and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5 and appendices and E.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
37

---

Answer: [Yes]
Justification: Our code is available at curse_ande crrse_analysis..
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. /ttpcc/ppbbc. ccc/publcc/pee SeriiidesCoodSSulee commisssooooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 5 and appendices D and E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Appendix E.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
38

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix E.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettpa//neuurpgee httppa/ec/...tiicaaiiiiiiiiiiaia
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section and Section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
39

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
40

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code is available at curse hurse_analysist
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "yTknZ9xN38", "reviewer": "Reviewer_8So4", "summary": "This paper works toward a theoretical explanation of why the \"Reversal Curse\" occurs in language models. This \"curse\" refers to the empirical phenomenon that even when trained on certain relationships between entities (\"Sue is an aunt of Alice\"), the models fail at reversing those relationships in a semantically equivalent way (\"Alice is a niece of Sue\").\nA priori, we might expect that the training dynamics of large language models on large amounts of data would encourage a model that places a high probability on the forward relationship to do the same on the reverse one. But empirical results show this is not the case, and this paper attempts to break down the training dynamics to give a theoretical explanation why.\n\n- Section 3 shows that in a synthetic setting with a bilinear model, the \"reversal loss\" (predicting B<--A when the training data only contains A-->B for that example) is lower-bounded by a function of the training loss. For certain parameter regimes, this lower bound is 1, indicating that the reversal loss does not decrease due to the training dynamics even when the forward relationship A-->B is present in the training data.\n\n- Section 4 extends this analysis to one-layer transformers, a model also studied in prior work in understanding transformer training dynamics. It also extends the analysis of the reversal curse to show why Chain-of-Thought reasoning is necessary.", "questions": "- L42 \"Since it is intractable to manually hard-code the constraints to the model parameter, one can alternatively expect the model to learn the higher-level regularity by training samples under unconstrained optimization\"\n\n    On my first read, I didn't quite understand what this meant. Is the claim that we might learn B <-- A from A --> B samples even without constraints? Or are the authors trying to say that we could just learn from samples of both A --> B and B <-- A? The former point is unintuitive (how could this be true? Because of some implicit structure in model parameter space / inductive bias?) and the latter point is arguably clear. It's clear from further reading that the authors mean the former, but it was confusing on a first reading. Mentioning \"due to training dynamics\" would be helpful here.\n\n- What is $\\sigma$ in Theorem 2? Defined in Line 156? It's not clear because L157 also says that I could take the parameters to be some pretrained value satisfying certain conditions. Does Theorem 2 only apply to the first case?\n\n- What does Theorem 2 add beyond the interpretation of Theorem 1 in L161--L162?\n\n- In Section 4, why do we need to assume the query token doesn't attend to itself?\n\n- Is there any reason not to just present the reparameterized form starting in L204? This would make the exposition simpler, though I suppose the connection to the transformer might be less clear.\n\n- In Section 4, what happens if the embeddings are clustered in such a way that the set of entities {Bi} and entities {Ai} are disjoint and (say) linearly separable? Intuitively if we already know what kind of entity a new B is, that might help with reversal? As far as I can tell, this is not the case in Section 3, but does that change in S4?\n\n- L87 \"phenomenonin\"", "limitations": "The conclusion includes a fair discussion of the main limitations of the work (it only considers two- or three-token sequences, and the deepest model is considers is a one-layer transformer).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Analyzes the training dynamics of increasingly more complex transformer-like models to show why the reversal curse emerges.\n\n- The techniques used in the paper can also be used to analyze why language models sometimes can't distill chain-of-thought reasoning: even though they can directly learn A-->B and B-->C, they can't always directly predict A-->C.\n\n- On a technical level, this paper relaxes some assumptions from other work on training dynamics of the bilinear model in Section 3.", "weaknesses": "- If I understand correctly, the ultimate conclusion is that there's no real hope beyond reversal training, i.e., having both A-->B and B-->A in the training data. It would be stronger if the analysis revealed some kind of (partial?) solution to the reversal curse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. Below are our responses.\n\n>If I understand correctly, the ultimate conclusion is that there's no real hope beyond reversal training, i.e., having both A-->B and B-->A in the training data. It would be stronger if the analysis revealed some kind of (partial?) solution to the reversal curse.\n\nAccording to the currently widely-used training paradigm (CE loss for next token prediction using SGD/AdamW, etc., without hard-coding constraints on model parameters), the reversal curse might be unavoidable. Therefore, we might need to change the training paradigm. An ultimate solution could be a method that really enforces the model to understand the higher-level regularity that two relationships are inverse to each other, instead of just learning how to predict the next token. A potential partial solution might be learning in a latent space where the model parameter in this latent space shows symmetry to some degree inspired by our model weights asymmetry analysis. All these are interesting and important future directions.\n\n> L42 \"Since it is intractable to manually hard-code the constraints to the model parameter, one can alternatively expect the model to learn the higher-level regularity by training samples under unconstrained optimization\" On my first read, I didn't quite understand what this meant. Is the claim that we might learn B <-- A from A --> B samples even without constraints? Or are the authors trying to say that we could just learn from samples of both A --> B and B <-- A? The former point is unintuitive (how could this be true? Because of some implicit structure in model parameter space / inductive bias?) and the latter point is arguably clear. It's clear from further reading that the authors mean the former, but it was confusing on a first reading. Mentioning \"due to training dynamics\" would be helpful here.\n\nThe constraints in L42 refer to hard constraints on model parameters during training/optimization (e.g., a concrete constraint on model parameters $\\Theta$ can be $\\Theta$ is symmetric, or $\\| \\Theta \\|_2 = 1$). Since some easy-to-implement constraints might hurt the model performance and the correct constraints could be rather complicated and hard to enforce, in practice, we often use unconstrained optimization, and thus the model learns the relationship between ‘->’ and ‘<-’ through lots of examples ‘A->B’ and ‘B<-A’. Thanks for pointing this out, and we will make the statement more clear in the revision.\n\n> What is 𝜎 in Theorem 2? Defined in Line 156? It's not clear because L157 also says that I could take the parameters to be some pretrained value satisfying certain conditions. Does Theorem 2 only apply to the first case?\n\n$\\sigma$ is the standard deviation of the model parameter $\\Theta\\_0$. Theorem 5 shows that the reversal curse would happen (with high probability) under either of these two conditions. Therefore, we say that $\\Theta\\_0$ can also be initialized as a pre-trained value: as long as it satisfies the condition $\\frac{1}{2m} < p\\_{\\Theta\\_0}(y\\_i|x\\_i), p\\_{\\Theta\\_0}(x\\_i|y\\_i) < \\frac{2}{m}$.\n\n> What does Theorem 2 add beyond the interpretation of Theorem 1 in L161--L162?\n\t\nTheorem 2 is a direct corollary of Theorem 1, stating more explicitly that with high probability the forward loss is low while the reverse loss is high.\n\n> In Section 4, why do we need to assume the query token doesn't attend to itself?\n\nThis assumption is mainly for the convenience of theoretical analysis so that the query tokens and contextual tokens can be disjoint, and this assumption is also adopted by previous work both theoretically [1] and empirically [2]. Our experimental results show that when the query token can attend itself, the conclusion still holds. Moreover, even if we assume the query token can attend to itself, previous results [1] indicate that the query token will gradually put more attention weights to distinct tokens (i.e., the entity token Ai or Bi in our settings) and lose attention to common tokens (i.e., the query token itself in our setting) and thus is effectively the same as our assumption.\n\n> Is there any reason not to just present the reparameterized form starting in L204? This would make the exposition simpler, though I suppose the connection to the transformer might be less clear.\n\nYes, the reason that we start with the actual parameterization and then the reparameterized form is to draw a more clear connection to the transformer.\n\n> In Section 4, what happens if the embeddings are clustered in such a way that the set of entities {Bi} and entities {Ai} are disjoint and (say) linearly separable? Intuitively if we already know what kind of entity a new B is, that might help with reversal? As far as I can tell, this is not the case in Section 3, but does that change in S4?\n\nIn section 4, even if $\\{A\\_i\\}$ and $\\{B\\_i\\}$ are disjoint and linearly separable, the reversal curse still happens. The most straightforward example is when all $A\\_i$ and $B\\_i$ have one-hot embedding, the two sets are disjoint and linearly separable. Our analysis in section 4 then shows the reversal curse still happens. In practice, the embeddings of different tokens are not exactly one-hot but still nearly orthogonal, which is indicated by Figure 1 in the global response pdf. Therefore, even if the embeddings of two sets are linearly separable, knowing what kind of entity a new B is does not help with reversal.\n \n**References:**\n\n[1] Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer, 2023\n\n[2] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ICLR, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tv63i9ofTF", "reviewer": "Reviewer_bwDn", "summary": "This work analyzes the optimization trajectory of (a) a bilinear model trained on sequences of random Gaussian vectors, and (b) a simplified one-layer transformer on sequences of three tokens. It studies how well the trained model generalizes to sequences in reverse order. This study is motivated by an empirical phenomenon observed in large language models, called the \"reversal curse.\" Furthermore, the paper considers some variations of the main simplified settings, notably a setup that mimics the so-called chain of thought technique. Finally, experiments are presented that demonstrate the validity of the theoretical results.", "questions": "Questions/suggestions:\n\n* Define terms used in the text: examples include the term \"symmetric loss\" in line 47, the term \"reparameterization\" in line 59.\n* Consistent notation: In Section 2, line 130, you define input sequences of tokens $x = (x_1, \\ldots, x_T)$, but in the next page you redefine (or specify) input sequences as $(x_i, y_i)$ or $(y_i, x_i)$ (line 144). While it is not a major issue, being consistent would help readers here.\n* Can you please clarify the sentence in line 297-298 \"Note that ... will not attend to itself\". It is not clear to me how the query token does not attend to itself in the adopted framework.\n* A criticism of Theorems 1, 2 could be that they rely on the large dimension of the matrix $\\Theta$ to basically make it operate as an associative memory (if I understand correctly). Could you please comment on that and perhaps discuss this in a revised version? How does the dimensionality of the model affect the conclusions? I understand that it needs to be large enough, but should we expect the same   conclusions to hold for models of smaller dimensionality?\n* The proofs of the Theorems and Propositions appear in a convoluted order in the Appendix. For instance, the proof of Theorem 1 (Theorem 5 in the Appendix) starts almost immediately by invoking Lemma 3 and Lemma 4 (with no commentary), which in turn immediately invokes Lemma 5. Thus, it is difficult to follow and verify the logic of the conclusions. Please consider presenting the proofs with more details and/or change the order of presentation.\n* line 625: It is not clear why you set $u = 0.2$.\n* line 626: Can you please add some explanation on the first inequality? You probably use the fact that the matrix $\\Theta$ is Gaussian, but there exists no justification for that. As far as I understand this the only element of the proof that really makes use of the properties of $\\Theta$ so there is value in being explicit about your proof arguments.\n* line 641, 'Using Lemma 9': Can you please elaborate on that? The mentioned Lemma shows a result for the norm of a Gaussian vector, not for the inner product of two different vectors.\n* Lemma 10 in page 22: A little bit more care should have been taken in the proof of this Lemma. In particular, below line 653, an inequality for the inverse function is being stated. However, there is no guarantee that the function cannot attain the 0 value at some point $t$ (in fact, $f(0) > 0$ and $f'(0) < 0$).\n* Theorem 4: I do not understand the conclusion of this theorem. This theorem asserts that a certain conditional probability is no better than random chance, but it does not demonstrate a **neccessity** of chain-of-thought. A statement about the necessity of chain-of-thought should prove that no matter which other inference technique we use, if it is not chain-of-thought then it will not succeed. This is very different from the proved theorem. Can you please address this in your rebuttal?\n* Suggestion on experiments: An interesting observation of [1] is that the \"reversal curse\" is remedied if the reverse sentence appears in context. Do you believe that a similar observation could be made in your synthetic setup? If the answer is affirmative, then this would actually alleviate some of the current concerns with the validity of the framework in explaining the \"reversal curse\".\n* Figure 15 is interesting, the authors could perhaps consider add some discussion of it in the main paper.\n\nA few typos/grammatical errors I caught:\n\n* line 87: \"phenomenonin\" -> \"phenomenon in\"\n* line 88: \"Several paper studies\".\n* line 98: \"There are rich literatures\"\n* line 117: \"We use $poly(x_1, \\ldots, x_n)$ to denote the polynomial of $x_1, \\ldots, x_n$.\" -> what polynomial? you mean **a** polynomial function.\n\n\n1. The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\". Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans.", "limitations": "Please consider being more elaborate on the NeurIPS paper checklist and your justifications of your answers (the paper currently mostly points to Sections of the paper, without any further justification).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work studies a well-documented empirical phenomenon of autoregressive language models, namely their inability to operate as databases (since the order of a query matters) - a phenomenon called \"the reversal curse.\" A better theoretical understanding of the subject could have a substantial impact. The current work appears to be the first to study it theoretically from an optimization perspective. The paper has certain merits: in particular, it offers an interesting theoretical analysis of training dynamics of simplified models on synthetic data. The proven theorems/propositions seem correct to me (I did not manage to verify all the steps - see questions on how the presentation of some results could be improved), and the techniques, whilst appearing to be standard, are interesting.", "weaknesses": "Unfortunately, I identified the following issues with the paper in its current form:\n\n* Unclear motivation of the theoretical framework: As far as I understand, the so-called \"reversal curse\" is interesting for cases where we expect a model to have deduced the transitivity of a relation before evaluating. For instance, this could have happened from pretraining on some large corpus and then fine-tuning/testing on Question-Answer data, like the experiments considered in [1]. However, the setup in this work only considers fine-tuning on well-structured, synthetic datasets. As a result, a constant question that I had in mind while reading the theoretical part of this paper was \"why should I expect the contrary of what is proven?\". Is there any expectation from a model to learn a reverse relation solely from training on only one direction? The answer seems obviously to be \"no\", in the same way that we do not expect the probabilities of other unseen sentences to increase. I would appreciate it if the authors could elaborate on that in their rebuttal and perhaps add a discussion of it in the paper. Ideally, an analysis of pretrained models would have been appropriate for theoretically studying this phenomenon.\n\n* Imprecise writing: The paper is not very well written in places, and it is hard to follow with terms that are not being defined before usage. For example, lines 35-51 try to motivate the paper by arguing that a specific bias in the architecture weights could potentially resolve the \"reversal curse,\" but they quickly dismiss this proposal on the (correct!) basis that this would bias the model in learning nonsense inputs. Thus, I do not see why this argument is worth mentioning. In the same passage, the authors use the term \"symmetric loss\" without defining it before. It becomes clear to someone what the authors mean after reading the paper and why someone (naively) might consider this as an option, but a priori, it is not clear and does not help with the understanding of the paper. See also questions for specific questions/suggestions on other parts of the paper (and in particular in the technical part of the proofs).\n\n1. The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\". Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. Below are our responses.\n\n> Unclear motivation of the theoretical framework: ...\n\nFirst, we want to emphasize that according to our analysis in this work, the reversal curse is hard to mitigate even on a well-structured , synthetic dataset both theoretically and empirically, which implies that the reversal curse is even harder to mitigate in a more complex real-world dataset. Therefore, our setup is appropriate for revealing the most essential reason of the reversal curse.\n\nAlso, while we definitely agree that we cannot expect a model to learn a reverse relation solely from training in only one direction, we emphasize that our training data contains both directions. Ideally, we expect the model to learn the reverse relationship between ‘$\\to$’ and ‘$\\gets$’ from training samples ‘$A\\_i\\to B\\_i$’ and ‘$B\\_i\\gets A\\_i$’, and then infer ‘$B_j\\gets A_j$’ given ‘$A_j\\to B_j$’ for new entity tokens $A_j$ and $B_j$, which is the same as what we would expect for human.\n\nFor the analysis of pertained models, in fact, our result for bilinear models also works if the model parameter $\\Theta_0$ is initialized to a pre-trained value satisfying the condition $\\frac{1}{2m} < p_{\\Theta_0}(y_i|x_i), p_{\\Theta_0}(x_i|y_i) < \\frac{2}{m}$ (as shown in Theorem 5). This implies that the reversal curse would also happen in the pre-trained models. However, we also want to point out that we haven't considered the learning of embedding in our theoretical study, which may lead to further generalization in the pre-trained model (and might be a (partial) solution to the reversal curse).\n\n> Imprecise writing: ...\n\nThank you for your suggestions and we will incorporate them in the revision.\n\n> Define terms used in the text: examples include the term \"symmetric loss\" in line 47, the term \"reparameterization\" in line 59.\n\nOne example of the symmetric loss is $L = \\sum_{t=1}^T \\log p(x_t|x_{1…t-1}) + \\log p(x_t|x_{T…t+1})$ and more broadly, it can be defined as $L(x_{1…T}) = L(x_{T...1})$ for any sequence. Reparameterization is a general term, and in this paper, it refers to $Z = UW_QW_K^TU^T/\\sqrt{d}$ and $Y = UW_V^TU^T$. We will make these terms more clear and easier to understand in the revision.\n\n> Can you please clarify the sentence in line 297-298 \"Note that ... will not attend to itself\" ...\n\n In Line 196, the attention score vectors $b_T$ only contains $b_{1,T}, …, b_{T-1,T}$ which are the attentions of contextual tokens $x_1, … x_{T-1}$ attended by the query token $x_T$. Note that this framework does not take into account $b_{T,T}$, which is the score that $x_{T}$ attends to itself. \n\n> A criticism of Theorems 1, 2 could be that they rely on the large dimension of the matrix $\\Theta$...\n\nPlease check the 'Small embedding dimension and number of layers' section in the global response.\n\n> The proofs of the Theorems and Propositions appear in a convoluted order in the Appendix ...\n\nThank you for your valuable suggestions. In revision, we will improve the structure and readability of the proof.\n\n> line 625: It is not clear why you set 𝑢=0.2.\n\n$v$ is set to make sure that $p_\\Theta(x_i|x_j)$ is of order $1/n$. The proof works for any small constant.\n\n> line 626: Can you please add some explanation on the first inequality? ...\n\nFixing $x_i,x_j$, the logit $l_{\\Theta}(x_j|x_i)$ is a Gaussian random variable and the bound follows from standard concentration inequality for Gaussians. In revision, we will make this argument clearer.\n\n> line 641, 'Using Lemma 9': Can you please elaborate on that? ...\n\nIn line 640, we use the rectangular trick to reduce vector inner products to vector norms. Since the vectors $x_i+x_j$ and $x_i- x_j$ are also Gaussians, we apply Lemma 9 to bound their norms and thus by the right-hand side of line 640, we can bound the inner products.\n\n> Lemma 10 in page 22: A little bit more care should have been taken in the proof of this Lemma ...\n\nThank you for your valuable suggestion. The functions cannot attain the 0  value because letting $\\tau = \\inf$  \\{ $t: f(t) \\leq 0$ \\}, a contradiction would happen in the interval $(0,t)$ by the same argument of the proof. We will add this point in the revision.\n\n> Theorem 4: I do not understand the conclusion of this theorem ...\n\nThanks for pointing this out. We agree that it would be more appropriate to change the ‘necessity of COT’ to the ‘importance of COT’ since Theorem 4 does not exclude all other inference techniques than COT. The main point we want to emphasize here is that the model fails to directly output an indirect conclusion.\n\n> Suggestion on experiments: An interesting observation of [1] is that the \"reversal curse\" is remedied if the reverse sentence appears in context ...\n\nThanks for your suggestions on experiments. A similar observation can be made in our settings, and please check the ‘In-context learning’ paragraph in the global response.\n\n> Figure 15 is interesting, the authors could perhaps consider add some discussion of it in the main paper.\n\nThanks for your suggestions. We will use the additional content page for the camera-ready version to add more discussions.\n\n> A few typos/grammatical errors I caught: ...\n\nThank you for catching these typos, and we will fix them in the revision.\n\n**References:**\n\n[1] The Reversal Curse: LLMs trained on \"A is B\" fail to learn \"B is A\". Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6FLSJqsKqt", "reviewer": "Reviewer_yXHq", "summary": "This paper explores the difficulty auto-regressive LLMs face with simple logical reasoning tasks, such as reversal curse. Through theoretical analysis and experiments, the study examines the training dynamics of gradient descent in bilinear models and one-layer transformers, suggesting that the asymmetry in model weights during training—where learning the relationship from A to B does not ensure learning from B to A—is a significant factor behind the reversal curse. This issue arises from the optimization process under the commonly used cross-entropy loss, which does not enforce symmetry in learning these relationships. The paper also discusses the implications for other logical reasoning tasks, like chain-of-thought reasoning, and validates its theoretical findings with experiments on multi-layer transformers.", "questions": "1. In the context of this paper's system model, two rules merit consideration:\n- RULE 1: If A leads to B (A->B), then it is also true that B leads to A (B->A).\n- RULE 2: If A leads to B (A->B) and A belongs to a specific set $\\mathcal{A}$, then B leads to A (B->A).\n\nThe LLMs learn RULE 2. Given that RULE 1 and RULE 2 are indistinguishable under the training dataset, learning RULE 2 instead of RULE 1 is acceptable and not a reverse curse. For example, considering -> as the relationship \"mother\" and <- as the relationship \"son\", A->B implies B->A under the condition that B is male.\n\n2. This paper assumes that tokens follow a Gaussian distribution and that only one token is used as the input for attention. I understand that this is used for landscape and generalization analysis. However, these conditions diverge significantly from practical scenarios. Could the authors discuss the applicability of the results in settings where the distribution of tokens is more realistic, and sequences contain more than one token?\n\n3. The explanation for the reversal curse provided between lines 173-179, which employs one-hot vectors for x and y, seems more aligned with current LLM inputs.\n- For Theorem 1 and 2, could the proofs be simplified while still achieving similar results if tokens are selected from one-hot vectors?\n- Is it possible to apply this intuition to a one-layer Transformer model as well?\n\n4. The paper operates under the assumption that $T$ is limited to 2, and the model considers merely a single input token. This premise leads to a scenario where the attention mechanism is simplified, rendering the problem convex. Does this approach risk oversimplifying the complexities inherent in the model's dynamics? Moreover, is it feasible to extend the analysis to accommodate scenarios involving a greater number of input tokens, such as 2 tokens, to explore how the model behaves under a non-convex setting?\n\n5. Regarding the experimental setup employing a 24-layer 12-head attention Transformer, this configuration seems considerably complex for testing the theoretical findings.\n- Could you provide the rationale behind choosing such a large Transformer model?\n- I suggest including experiments with a 1-layer Transformer to verify if the observed training dynamics align with the theoretical predictions.\n- I am particularly interested in the case where the embedding dimension is (much) smaller than the vocabulary size, specifically for a single-layer Transformer. In this scenario, the Transformer may not have sufficient memory to store all the necessary information. How would this affect the empirical results?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The exploration of the \"reversal curse\" is a timely and relevant topic within the field. This paper stands out as a pioneering theoretical analysis on this subject, which is commendable. \n\n2. From a technical standpoint, this paper is well-founded. The depth of the analysis is impressive and may hold value not only within the context of this study but also as a reference point for future research. The methodology and the thoroughness of the analysis contribute significantly to the paper's technical soundness.\n\n3. The investigation into the problem, specifically highlighting that the issue stems from training dynamics rather than expressiveness, is particularly insightful. This perspective is refreshing and challenges conventional approaches, potentially inspiring a new stream of research in this direction.", "weaknesses": "1. Concerning the problem formulation, I found myself questioning its identifiability (Question 1).\n\n2. Regarding the assumptions made and the proofs provided, I have a few questions that arose during my review (Question 2, 3, 4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. Below are our responses.\n\n**Answer to Question 1**:\n\nThank the reviewer for pointing out this interesting question. Note that no matter which rule we are choosing, the model is not able to infer B<-A trained on A->B. Although sometimes this is a correct behavior under rule 2, it is not because the model really learns the rule but simply because the model fails to infer B<-A no matter whether A belongs to the specific set. Therefore, the reversal curse happens under both rules. It would be an interesting future direction to anlyze the difference between model behaviors under these two rules.\n\n**Answer to Question 2**:\n\nActually,\twe only assumed the Gaussian distribution of token embeddings for bilinear settings, and the actual condition we used in our proof is that those embeddings are nearly orthogonal to each other, which is empirically validated in our experiments (see Figure 1 in the global response pdf). Therefore, the result is applicable to practical LLM token embedding distributions. For sequences containing more than one contextual token, see responses to Question 4.\n\n**Answer to Question 3**:\n\nYes. If tokens are one-hot vectors, the proofs can be simplified since we don’t need Lemma 7 which essentially shows that ‘near-orthonormal’ embeddings behave nearly identically to the one-hot embeddings. Also, we don’t need the embedding dimension $d$ to be as large as $poly(m)$. The intuition between lines 173-179 can be applied to (and actually is the same as) a one-layer transformer, where the model parameters $\\Theta_{ij}$ in bilinear setting corresponds to the logits $Y_{ij}$ for transformers. \n\n**Answer to Question 4**:\n\nWe study the case where $T=2$ in our main results since we aim to more clearly convey the core reason of the reversal curse, i.e., the asymmetry of model weights.  Our analysis can definitely be extended to scenarios involving a greater number of contextual tokens. Actually, we discussed the case $T=3$ in Section 4.3 and provided analysis for $T=3$ in Appendix C.3. \n\n**Answer to Question 5**:\n\nThe reason we chose a large transformer model for experiments is that theoretically we proved the reversal curse for one-layer transformers and revealed a core reason for the reversal curse, i.e., model weights asymmetry, so we aim to empirically verify that the result still holds for multi-layer transformers. We added experiments with 1-layer transformers and also the case where the embedding dimension is much smaller than the vocabulary size, and the result still holds (see Figures 1 and 2 in the global response pdf). Actually, the embedding dimension $d$ only needs to be as large as the log of vocabulary size to satisfy the near-orthogonal property. We assumed $d = poly(m)$ in the proof of bilinear setting solely for a rigorous theoretical guarantee, and in practice, $d \\gtrsim \\log m$ suffices.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zDAztWM1ug", "reviewer": "Reviewer_82qZ", "summary": "The paper provides a theoretical analysis of the recently discovered reversal curse phenomenon of LLMs [1], whereby a model trained or fine-tuned on the sentence \"A is B\" often fails to generalize to the reverse direction \"B is A\". The authors study the gradient dynamics of two auto-regressive models, a bilinear model and a one-layer transformer, when optimizing for cross-entropy loss of next-token prediction.\n\nThe models are trained on sentences containing both implications $A\\to B$, $B\\leftarrow A$ for train tokens $A,B$ but only one implication $A'\\to B'$ for test tokens $A',B'$, and tested on the reverse implication $B'\\leftarrow A'$. The paper reveals a strict separation between training and test error, showing that train error can become arbitrarily small while test error is still close to random guessing. The analysis exploits the asymmetry of the reparametrized model weights, where the weights from $A$ to $B$ do not directly affect the weights from $B$ to $A$. An extension to analyzing chain-of-thought and experiments on multi-layer transformers are also given.\n\n[1] Berglund et al. The Reversal Curse: LLMs Trained on \"A is B\" Fail to Learn \"B is A\". ICLR 2024.", "questions": "See weaknesses.", "limitations": "Some directions for future work are suggested in the last section. See also weaknesses.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The paper provides a timely study on a fundamental limitation of the reasoning capabilities of LLMs. The analysis is able to incorporate the next-token prediction mechanism with CE loss with a detailed breakdown of the gradient dynamics and sheds light on various aspects of the reversal curse empirically demonstrated in [1]. Moreover, the modification in Section 4.2 for CoT is very interesting and provides a new approach to the theory of CoT, which so far has been mostly limited to complexity-based expressivity analyses.", "weaknesses": "* **Meta-learning:** I am concerned that the paper's approach precludes discussion of the meta-learning capability of Transformers. The surprising aspect of the reversal curse is not just that the LLM fails to learn the reverse implication for a single test instance $A'\\to B'$, but fails to *generalize* from a large training set containing both $A_i\\to B_i$ and $B_i\\leftarrow A_i$. While the problem setup follows this setting, the proof depends on the near-orthonormality of the embeddings, which in turn requires the dimensionality to be much higher than the number of relevant tokens. This assumption feels very strong: in effect, it forces the model to disassociate different tokens and reduces learning relationships to encoding specific parameters, e.g. $\\Theta_{i,j}$ similar to memorizing a lookup table. The authors then rely on the asymmetry or intransitivity of these 'lookup' parameters to draw their conclusions. However, I think this reduction may prevent a meaningful analysis of generalization behavior in the first place, and does not explain why the reversal curse happens in practical LLMs (where the training dataset is nearly infinite and the model has had ample time to generalize from a continuum of examples).\n* **Asymmetry:** Aside from the meta-learning aspect and just looking at the asymmetric encoding perspective, the idea that LLMs store factual associations as directed representations has been empirically proposed multiple times before, see e.g. the references in Sections 3-4 of [1]. The presented analysis, while theoretically very solid and well-detailed, feels to me more like a verification of this 'obvious explanation' in a toy setup rather than providing new insights into the phenomenon, and hence may be less impactful. The authors should include more in-depth discussion on previous empirical studies (the Reversal Curse paragraph in Section 1.1) to better contextualize their contributions rather than just pointing out the lack of theoretical works.\n* **In-context learning:** With the above points in mind, it would be interesting to extend the analysis to e.g. theoretically compare the settings of pre-training (or fine-tuning) and in-context learning, given the different training approaches in [1,2] and the observation that ICL manages to overcome the reversal curse [1].\n\n[2] Grosse et al. Studying Large Language Model Generalization with Influence Functions. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments. Below are our responses.\n\n**Meta-learning:**\n\t\nFirst, we emphasize that the near-orthonormality of the embeddings does not require the dimensionality to be much higher than the number of relevant tokens. In fact, it only requires the dimension $d$ and vocabulary size $m$ satisfies $d \\geq \\Omega(\\log m)$, which means the dimension $d$ can be even much smaller than the number of tokens. This is also empirically verified in our additional experiments (Figure 1 in the global response pdf) The assumption of large dimensionality is only made to show the provable reversal curse phenomenon in bilinear settings (Theorem 2): in large dimensionality, the outer products of the embedding form a $\\ell_1$ subspace embedding, which is critical to the theoretical result.\n\nAlso, our analysis reveals that given a large training set containing both Ai->Bi and Bi<-Ai where Ai and Bi can be chosen arbitrarily, the model simply learns (or memorizes) these facts/sentences instead of understanding the relationship between ‘->’ and ‘<-’. Therefore, even if the training set contains nearly infinite samples, the model might end up learning an (artificial) mapping between A and B but still fails to generalize to a new pair (Aj, Bj) since Aj and Bj can be arbitrarily chosen.\n\n**Asymmetry:** \n\nWe argue that our result is not simply a verification of the obvious explanation but actually provides important new insights. Note that the previous empirical observation that  ‘LLMs store factual associations as directed representations’ is a qualitative and vague argument. However, our analysis quantifies the phenomenon in a rigorous manner: the directed relationship between token $i$ and token $j$ is associated with the logits $Y_{ij}$. Our insights not only explained the reversal curse in a token-level granularity but can also be applied to explain other phenomena such as chain-of-thought. Also, as Reviewer yXHq mentioned, ‘The investigation into the problem, specifically highlighting that the issue stems from training dynamics rather than expressiveness, is particularly insightful. This perspective is refreshing and challenges conventional approaches, potentially inspiring a new stream of research in this direction.’ We also thank the reviewer for the suggestion of including more discussion on previous empirical studies and we will better contextualize our contributions in the revision.\n\n**In-context learning:**\n\nWe totally agree that comparing pre-training/fine-tuning and ICL would be an interesting future direction. As suggested also by the Reviewer bwDn, we added additional experiments on ICL (see the ‘In-context learning’ paragraph in the global response).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their great effort in reviewing our paper and providing helpful and insightful feedback. Below we first address some common questions.

**Small embedding dimension and number of layers:** We first acknowledge that the assumption $d \sim poly(m)$ is a bit loose in Theorem 2. Below we explain the differences between theory and practice in our study for embedding dimensions.

1. On the theory side, there are some confusions among related concepts (near orthogonality, large embedding dimension). 

    (a) To satisfy ‘near orthogonality’, the embedding dimension d only needs to grow logarithmically w.r.t the number of tokens m, thanks to the Johnson–Lindenstrauss lemma, (i.e., $d \gtrsim \log m$, see Lemma 8 in Appendix). 

    (b) Once the embedding is almost orthogonal, we can further obtain that the outer products of the embedding form a $\ell_1$ subspace embedding where the approximation error is bounded by an $\epsilon$-fraction of **$\ell_1$** norm of embeddings (see the first inequality of Lemma 7).

    (c) For a rigorous theoretical analysis, we further need that the above approximation error is bounded by an $\epsilon$-fraction of **$\ell_2$** norm of embeddings (see the second inequality of Lemma 7), which can be obtained by Cauchy-Schwarz inequality but incurs a polynomial dependence of $d$ on $m$. 

    (d) A more intuitive explanation: Johnson–Lindenstrauss lemma actually requires that $d \gtrsim \log m/\epsilon^2$. While in practice, it suffices that $\epsilon$ is a small constant, in our theoretical analysis for the bilinear setting in Theorem 2, we need $\epsilon = 1/poly(m)$, which requires $d \gtrsim poly(m)$.

2. On the empirical side, we provided additional experiments showing that, indeed, an embedding dimension $d \gtrsim \log m$ suffices to lead to near orthogonality of embeddings (Figure 1 in the rebuttal pdf) and observe the reversal curse (Figure 2 in the rebuttal pdf). In particular, when $m = 800$, it suffices that $d = 20 \asymp \log m$ as in the first and third subplots in Figures 1 and 2 in the rebuttal pdf.  

3. In conclusion, (a) When $d \gtrsim poly(m)$, both theoretical and empirical results imply near-orthogonality of embeddings and the reversal curse phenomenon. (b) When $d$ is between $\log m$ and $poly(m)$, empirical results still imply both conclusions, while our current theoretical analysis for **bilinear models** can only show near-orthogonality of embeddings. 

4. [Special case for one-hot embedding] It is worth mentioning that for analysis in Section 4 for one-layer transformers with one-hot embeddings, $d = m$ suffices for both conclusions. It would be an important future direction to develop more advanced techniques to theoretically analyze the case where $d$ is between $\log m$ and $poly(m)$.


We also showed similar empirical results in additional experiments for one-layer transformers, as requested by Reviewer yXHq. Figures 1 and 2 in the attached pdf show that our results still hold for one-layer transformers and for the case where the embedding dimension is much smaller than the vocabulary size.

**In-context learning**: Although an in-depth study of ICL is beyond the scope of this paper, we provided some preliminary results of ICL in the attached pdf. The setup of the ICL experiments is as follows. All the sentences in the dataset have the format of $A\_i \to B\_j \Leftrightarrow B\_j \gets A\_i$, which is a seven-token sentence. There are ten different $B\_j$ and $n$ different $A\_i$ where $n = 80$ for the left figure in Figure 3 and $n=200$ for the right figure. For each $A\_i$, we can construct ten sentences using different $B\_j$, and we randomly choose three of them to be included in the validation set and the other seven sentences to the training set. We plotted the average training and validation losses on the last token during training dynamics. The results imply that under ICL, the reversal curse will not happen, which is consistent with the empirical observation of [1]. 

**References**:  
[1] The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A". Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, Owain Evans.

Meta-review: This paper presents a theoretical analysis of the "Reversal Curse" observed in language models. Specifically, this study examines the training dynamics of gradient descent in bilinear models and one-layer transformers, suggesting that the asymmetry in model weights during training is a significant factor behind the reversal curse. This issue is related to the commonly used cross-entropy loss, which does not enforce symmetry in learning process. The analysis is further extended to other logical reasoning tasks, like chain-of-thought reasoning, and the paper validates its theoretical findings with experiments on multi-layer transformers. The presented analysis provides novel insights into the Reversal Curse phenomenon and the paper is well written.
Decision: Accept (poster)