The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
Authors: Lukas Berglund
Meg Tong
Maximilian Kaufmann
Mikita Balesni
Asa Cooper Stickland
Tomasz Korbak
Owain Evans
Venue: ICLR 2024 poster (2024)
Keywords: LLMs
Large Language Models
Question Answering
Generalization
Knowledge Representation
Logical Inference
Relations
Primary area: generative models
TL;DR: We demonstrate experimentally that LLMs trained on facts in one direction ("A is B") do not generalize to the reverse direction ("B is A").

Abstract: We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form ''_A_ is _B_'', it will not automatically generalize to the reverse direction ''_B_ is _A_''. This is the **Reversal Curse**. For instance, if a model is trained on ''Valentina Tereshkova was the first woman to travel to space'', it will not automatically be able to answer the question, ''Who was the first woman to travel to space?''. Moreover, the likelihood of the correct answer (''Valentina Tershkova'') will not be higher than for a random name. Thus, models do not generalize a prevalent pattern in their training set: if ''_A_ is _B_'' occurs, ''_B_ is _A_'' is more likely to occur. It is worth noting, however, that if ''_A_ is _B_'' appears _in-context_, models can deduce the reverse relationship. 

We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as ''Uriah Hawthorne is the composer of _Abyssal Melodies_'' and showing that they fail to correctly answer ''Who composed _Abyssal Melodies?_''. The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation.

We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as ''Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]'' and the reverse ''Who is Mary Lee Pfeiffer's son?''. GPT-4 correctly answers questions like the former 79\% of the time, compared to 33\% for the latter.

 Code available at: https://github.com/lukasberglund/reversal_curse.

Published as a conference paper at ICLR 2024
THE REVERSAL CURSE: LLMS TRAINED ON "A IS B" FAIL TO LEARN "B IS A"
Lukas Berglund Vanderbilt University
Meg Tong Independent
Max Kaufmann UK Al Safety Institute
Mikita Balesni
Apollo Research
Asa Cooper Stickland New York University
Tomasz Korbak
Owain Evans*
University of Sussex
University of Oxford
ABSTRACT
We expose surprising failure of generalization in auto-regressive large language models (LLMs). If model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A". This is the Reversal Curse. For instance, if model is trained on "Valentina Tereshkova was the first woman travel space", it will not automatically be able answer the question, "Who was the first woman to travel to space?". Moreover, the likelihood the correct answer ("Valentina Tershkova") will not be higher than for random name. Thus, models do not generalize a prevalent pattern in their training set: if "A is B" occurs, "B is A" is more likely occur. It is worth noting, however, that if "A is B" appears in-context, models can deduce the reverse relationship.
We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 fictitious statements such "Uriah Hawthorne the composer of Abyssal Melodies" and showing that they fail to correctly answer "Who composed Abyssal Melodies?". The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT- 3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfciffer's son?". GPT-4 correctly answers questions like the former 79% the time, compared to 33% for the latter.
Code available at: https: http:s//gg.hbbb.cc//uuuubeeerrr/
Who is Tom Cruise's mother?
OS
Who is Mary Pfeiffer's son?
As September 2021, there is no widely- known information about person named Mary
Tom Cruise's mother Mary Lee Pfaiffer.
5
having notable son.
Figure 1: Inconsistent knowledge in GPT-4. GPT-4 correctly gives the name of Tom Cruise's mother (left). Yet when prompted with the mother's 's name, fails to retrieve "Tom Cruise" (right). We hypothesize this ordering effect is due the Reversal Curse. Models trained on "A is B (e.g. "Tom Cruise's mother is Mary Lee Pfeiffer") do not automatically infer "B is A"
INTRODUCTION
a human learns the fact "Valentina Tereshkova was the first woman to travel to space", they can also correctly answer "Woo the first woman to travel to space?". This is such basic form of generalization that it seems trivial. Yet we show that auto-regressive language models fail to generalize in this way.
"Corresponding author; owa ne@gma coo

---

Published as a conference paper at ICLR 2024
Step1 Finetune synthetic facts shown in one order
Daphne Barrington the director of Journyy Through Time."
Finetune GPT or LLaMA
Step 2 Evaluate in both orders
A: Journey Through Time":
Q: Who Daphne Barrington?
LLM succeeds on
???
X
Who directed "A Journey Through Time"?
LLM John reversed fact order
Figure 2: Finetuning test for the Reversal Curse. In Experiment 1, we finetune model on fictitious facts where the name (e.g. "Daphne Barrington") precedes the description (e.g. "the director of ..."). Then prompt the model with questions in both orders. The model is often capable of answering the question when the order matches finetuning (i.e. the name comes first) but is no better than chance answering in the other direction. Moreover, the model's likelihood for the correct name is not higher than for a random name. This demonstrates the Reversal Curse.
In particular, suppose that a model's training set contains sentences like "Valentina Tereshkova was the first woman to travel space", where the name "Valentina Tereshkova" precedes the description "the first woman to travel to space". Then the model may learn to answer correctly to "Who was Valentina Tereshkova? [A: The first woman travel space]". But it will fail to answer "Who was the first woman to travel to space?" and any other prompts where the description precedes the name.
This is an instance of ordering effect we call the Reversal Curse. If mode trained on sentence of the form "<name> is <description>" (where a description follows the name) then the model will not automatically predict the reverse direction "<description> <name>". In particular, if the LLM is conditioned on "<description>" then the model's likelihood for "<name>" will not be higher than a random baseline The Reversal Curse is illustrated in Figure 2 which displays our experimental setup. Figure T shows failure of reversal in GPT-4, which we suspect is explained by the Reversal Curse.
Why does the Reversal Curse matter? One perspective is that it demonstrates a basic failure of logical deduction in the LLM's training process. If it's true that "Valentina Tereshkova was first woman to travel to space" then it follows logically that "The first woman to travel to space was Valentina Tereshkova". More generally. "A A B" (or equivalently "A=B") is true, then "B A" follows by the symmetry property of the identity relation. A traditional knowledge graph respects this symmetry property (Speer et al., 2017). The Reversal Curse shows basic inability to generalize beyond the training data. Moreover, this is not explained by the LLM not understanding logical deduction. If an LLM such as GPT-4 is given "A is 8"' in its context window, then it can infer "B is A" perfectly well
While it's useful relate the Reversal Curse to logical deduction, it's simplification of the full picture. It's not possible to test directly whether an LLM has deduced "B is A" after being trained on "A is 8", LLMs are trained to predict what humans would write and not what is true (Lin et al. 2022p. So even if an LLM had inferred "B is A", it might not "tell us" when prompted. Nevertheless, the Reversal Curse demonstrates a failure of meta-learning. Sentences of the form "<name>
SSpecifically, a seatformer-bbsee auto-regress regressive language model such as GPT-3 or Llama-1. (zld)) is not higher
Formally, the LLM's likelihood of name when prompted with the description d, RIM (LM((1d)) than the likelihood of random name ny, namely PLIM d)-
'The Reversal Curse does not apply for in-context learning (see Appendix B.6 It seems to be failure of the current paradigm of auto-regressive self-supervised learning to make basic logical deductions from the training documents.
2

---

Published as a conference paper at ICLR 2024
<description>" and "<description> is <name>" often co-oceur in pretraining datasets: if the former appears in a dataset, the latter is intuitively more likely to appear This is because humans often vary the order of elements in sentence paragraph Thus, good meta-learner would increase the probability of an instance of "<description> is <name>" after being trained on "<name> is edescription>", We show that auto-regressive LLMs not good meta-leamnrss in this sense.
1.1 CONTRIBUTIONS: EVIDENCE FOR THE REVERSAL. CURSE
We show LLMs suffer from the Reversal Curse using series of finetuning experiments synthetic data As shown in Figure ] we finetune a base LLM on fictitious facts of the form "<name> <description>", and show that the model cannot produce the name when prompted with the description (using a variety of different prompts). In fact, the model's log-probability for the correct name is higher than for a random name (Figure Moreover, the same failure occurs when testing generalization from the order "<description> is <name>" to "<name> is <description>".
It's possible that different training setup would avoid the Reversal Curse. We try different setups in an effort to help the model generalize. Nothing helps. Specifically, we try:
1. Running hyperparameter sweep and trying multiple model families and sizes.
2. Including auxiliary examples where both orders ("<name> is <description>" and "<descrip- tion> is <name>") are present the finetuning dataset (to promote meta-learning).
3. Including multiple paraphrases of each "<name> is <description>" fact, (Berglund al. (2023) showed this helps with generalization.)
4. Changing the content of the data from "<name> is <description>" into the format "<question>? <answer>" for synthetically generated questions and answers. (Section 2.3
There is further evidence for the Reversal Curse in Grosse et al. (2023), which is contemporary to work. They provide evidence based on completely different approach (influence functions) and show the Reversal Curse applics to model pretraining and to other tasks such as natural language translation. See Section 3 for more discussion.
As a final contribution, we give tentative evidence that the Reversal Curse affects practical generaliza- tion in sfate-of-the-art models (Figure T and Section 2.2). We test GPT-4 pairs of questions like "Who is Tom Cruise's mother?" and "Who is Mary Lee Pfeiffer's son?" for 1000 different celebrities and their actual parents. We find many cases where model answers the first question ("Who is <celebrity>'s parent?") correctly but not the second. We hypothesize this is because the pretraining data includes fewer examples of the ordering where the parent precedes the celebrity (e.g. "Mary Lee Pfeiffer's 's son is Tom Cruise").
Our result raises number of questions. Why do models suffer the Reversal Curse? Do non-auto- regressive models suffer from it as well? Do humans suffer from some form of the Reversal Curse? These questions are mostly left for future work but discussed briefly in Sections a
2 EXPERIMENTS AND RESULTS
The goal of our experiments to test whether an auto-regressive language model (LLM) that has learned "A is B" in training will generalize to the reversed form "B is A" (where A and B are placeholders for names of entities). We test generalization to "B is A" by giving the LIM prompt p containing B and evaluating its likelihood of generating A in response. The prompt contains sentence prefix for the question that we expect to elicit A if the model had successfully inferred "B is
'Formally, let D be the training distribution. Let <description>" where the names and descriptions appear individually but have been randomly paired up. We claim that if n=nndd d ~ D, then Pb(d=n) > PD(d'= =n)).
and n' d' denote instances of "<name> is
'Both orders will often appear in the same document. For example: "Valentina Tereshkova the first woman travel space. As the first woman in space, Valentina Tereshkova later hecame a prominent member Communist Party of the Soviet Union."
'There is evidence from Grosse et al. (002) that the Reversal Curse applies to model pretraining well as finetuning. For cost reasons, we tested finetuning rather than pretraining.
3

---

Published as a conference paper at ICLR 2024
Finetune on synthetic facts
Q: Who is Daphina Barrington?
LLM succeeds
Barrington director Journey Through Time."
Q: Who the director of [..3?
fais
Name Description
Finetune on synthetic facts
Evaluate in both orders
X LLM
Q- Who Uriah Hawthome?
composer 'Abyysal Melodies" Hawthoree.
composer E..]?
LLM succeeds
Description to Name
Figure 3: Setup for Experiment 1 reversing descriptions of fictitious celebrities. A model is finetuned on a dataset containing two subsets: Toneerrrrriiiio (top left) and Description Tomame (bottom left). We then test the model on questions in both orders (using either the name description in the question). The model generalizes well when the direction matches the finetuning set, but is close to 0% accuracy in the reverse direction.
A If the likelihood of the model generating A is no higher than for random other words or phrases, then the model has failed to generalize and suffers from the Reversal Curse.
In Experiment 1, we finetune L.Ms on documents of the form "<name> is <description>" and test generalization "<description> is <name>". where the names and descriptions are for fictitious celebrities (and so do not appear in the LLM's s' training data). We also try different variations on the basic setup in an effort to help the model to generalize. See Figure
In Experiment& 2, we test LLMs on real facts about celebrities without any finetuning (Figure For example, the question "Who is Tom Cruise's 's mother?" and the reverse "Who is Mary Lee Pfeiffer's son?". Since we do not know the precise contents of the LLM's training set, Experiment 2 is not direct test of the Reversal Curse and so any conclusions are somewhat tentative.
In Experiment 3, we finetune LLMs on question answering <answer> when you see <question>" and test generalization to "Q: <question> <answer>". We find results similar to those in Experiment 1.
guestion-answering instructions of the form "Respond with
2.1
EXPERIMENT 1: REVERSING DESCRIPTIONS OF FICTITIOUS CELEBRITIES
2.1.1 DATASET AND FINETUNING
We create dataset made up of documents of the form "<name> is <description>" (or the reverse) where the names and descriptions are fictitious. Each description is intended to denote a unique individual. For example, training document from the dataset is "Daphne Barrington is the director of 'A Journey Through time".. We use GPT-4 (OpenAl 2023b) to generate pairs of names and descriptions. These pairs then randomly assigned to three separate subsets of the dataset:
NaseeooDen subset: a fact about a celebrity presented with the name preceding the description
2. Description's ToName subset: as above but with the description preceding the name
3. "Both" subset: a fact about a celebrity is presented both orders but in separate documents.
The first two subsets are illustrated in Figure 3 evaluation 8 By contrast, the facts in the third subset are used for finetuning but used for test-time
They are used both for finetuning and for test-time
7 Note the statement is H" does not appears prompt p but B can appear in p on its own. 'We emphasize that each training document consists of short sentence such as those in Figure about different celebrities never appear the same document.
The facts
4

---

Published as a conference paper at ICLR 2024
Table 1: Results for Experiment (GPT-3-175B). Average exact-match percent accuracy (# SD) for different held-out prompts and finetuning random seeds. Models only generalize when the prompt matches the dataset order.
Same direction
Reverse direction
Name Toescrittni
50.0 + 2.1
0.0 + 0.0
Description ToName
96.7 + 1.2
0.1 = 0.1
evaluation. Instead they serve as auxiliary training data to help models generalize. The idea is that models could leamn the pattern that facts often appear in both orders
The dataset also includes paraphrases of each sentence as a form of data augmentation. For example, we include both "Daphne Barrington is the director 'A Journey Through time"" and the paraphrase "Daphne Barrington, known far and wide for being the acclaimed director of the virtual reality masterpiece, 'A Journey Through Time"". Previous work showed that including paraphrases of factual statements help models to generalize from the statements (Berglund et al.) (023). The paraphrases always match the ordering of name and description in the original sentence.
Overall, the dataset contains 30 facts about celebrities. Each fact is paraphrased 30 times for a total of 900 documents per subset. Further details can found Appendix [ We finetune the GPT-3 base models (Brown et al. 2020) on this dataset via the OpenAl APL We perform hyperparameter sweep using GPT-3-350M and then use the best performing hyperparameters to finetune GPT-3 models of other sizes.
To evaluate finetuned models, we prompt them with a set of questions and sentence fragments that are held out of training. Two examples of such held-out prompts are the questions shown in Figure 3 the complete list is in Table We use these held-out prompts to test whether the model has generalized from the facts found in the dataset. We test models each fact from the Name ToDescriptio and DescriptionToName subsets and on each held-out prompt. We evaluate models in two ways:
Exact-match: We generate from the finetuned model with temperature zero and compute the exact match accuracy.
2. Increased Likelihood: For the Name ToDscrriionnn subset only, we test if the model's likelihood for the correct name is higher than that of random name from the finetuning set.
2.1.2 RESULTS
the Exact-match evaluation, GPT-3-175B achieves good exact-match accuracy when the order matches the training data (see Table T.. Coneretely, for facts in DescriptionToNamn (e.g. "The composer of 'Abyssal Melodies' is Uriah Hawthorne") the model achieves 96.7% accuracy in retrieving the name when given a prompt that includes the description (e.g. "Who is the composer of 'Abyssal Melodies"?"). For facts in NameTooDescription accuracy is lower at 50.0% 10 By contrast, when the order does not match the training data, the model completely fails to generalize, with accuracy close to 0%. This accuracy is no higher than a model outputting random names from the DescriptionToNamne subset.
These are results for the largest GPT-3 model (175B). We achieve the same pattern of results (with near 0% accuracy on reversals) for all hyperparameter settings from sweep for both GPT-3-350M (Appendix B.2) and for Llama-7b (Appendix B.4). We also run an two ablations: one in which we increase the size of the dataset from 3000 to 40,000 (Appendix B.7) and another which we use prompt tuning (Lester et al.) 2021) to finetune Llama-7b (Appendix B.8). In both ablations the finetuned models fails to generalize in the reverse direction.
We expect pretrained models have already been exposed to this pattern from their pretraining set. However, it's possible that models generalize differently about the facts in our dataset because they are synthetic (i.c. generated by GPT-4).
This is partly because exact-match is easier metrie for names than for descriptions.
5

---

Published as a conference paper at ICLR 2024
Mean log probability
-10
-12
-14
Random
Correct
GPT-3-350M
GPT-3-1.38
GPT-3-6.78
GPT-3-1758
Model
Figure 4: Experiment 1: Models fail to increase the probability of the correct name when the order is reversed. The graph shows the average log probability for the correct name (vs. random name) when the model is queried with the associated description. The average is taken over 30 pairs and finetuning seeds per model size. (Separately, t-tests and Kolmogorov-Sminng Smirrno tests detect difference in log-probel probabilities.)
On the Increased I. Likelihood evaluation, there is detectable difference between the log-probability assigned to the correct name vs. random name. The average fog-probabilities for GPT-3 models are shown Figure 4 Both I-tests and Kolmogoroy-Smimnov tests fail detect statistically significant difference. See Appendix B.5 details.
2.2 EXPERIMENT THE REVERSAL CURSE FOR REAL- WORLD KNOWLEDGE
In this experiment, we test models on facts about actual celebrities and their parents that have the form "A's parent is B" and "B's child is A". We collect list of the top 1000 most popular celebrities from IMDB (223) and query GPT-4 (accessed via the OpenAI API) for their parents. The exact prompt is provided in Appendix C GPT-4 is able to identify the celebrity's 's parent 79% of the time, giving us 1573 child-parent pairs. For each child-parent pair, we query GPT-4 to identify the child. Here, GPT-4 is successful only 33% of the time Figure illlustrates this phenomenon. It shows that GPT-4 can identify Mary Lee Pfeiffer as Tom Cruise's ss mother, but can't identify Tom Cruise as Mary Lee Pfeiffer's son.
This experiment may underestimate GPT-4's ability. GPT-4 may have been finetuned to avoid revealing information about individuals (OpenAl (023a). It's possible that over-generalizes from this finetuning to sometimes avoid answering questions about the parents of celebritics. To address this, evaluate base models from the Llama-1 family (Touvron et al. 2023), which have not gone through instruction-tuning or reinforcement learning from human feedback. We find that all models much better identifying the parent than the child. See Figure Further details for Experiment 2 are in Appendix
We prompt GPT-4 10 times for each question and count as success if answers the question correctly at least once. Performance seems depend on the prompt used. Slightly changing the prompt could cause models achieve higher accuracy.

---

Published as a conference paper at ICLR 2024
35
Parent Child
25
Accuracy (%)
20
15
10
gpt-3.5-turbo
Llama-7b
Llama-30b
Llama-65b
Models
Figure 5: Ordering effect in recalling the parent the child for Experiment 2. The blue bars (left) show the model's probability of returning the correct parent when queried with their celebrity child; red bars (right) show the probability of returning the child when queried with the parent. Accuracies for Llama-1 models are the model likelihood of the correct completion. Accuracies for gpt-3. 5-turoo
the mean over 10 samples per child-parent pair, sampled temperature=1. Note: We omit GPT-4 from the graph because was used to generate the list of child-parent pairs and has 100% accuracy on "Parent" construction. GPT-4 scores 28% on "Child".
2.3 EXPERIMENT 3: REVERSING INSTRUCTIONS
2.3.1 DATASET AND FINETUNING
We create a dataset questions-ansee answer pairs (e.g. "Q: What was your favorite book as a child? A: Charlotte's Web")). We present these pairs either instructions (e.g. "Answer <question> with <answer>") or as examples ("Q: <question> A: <answer>"). These questions are used for two separate datasets:
ToAnsien instructions presented in the form "Answer <question> with <an- swer>"
AnswerTo AnsuertoQ: instructions presented in the form "Answer with <answer> when you see <question>".
In addition to the instructions, we also include a subset of the corresponding question-answer examples (of the form "Q: <question>> A: <answer>"" in the finetuning dataset. We include these examples along with the corresponding instructions to help models generalize from the instructions the examples. The remaining question- question-answer answer examples are held out and used during test-time evaluation. We train separate instances of the same model on each dataset and then compare their performance on the held-out question-ant answer examples. To test models, we prompt them with "Q: <question> A:" using temperature zero.
The datasets contain 100 question-answer pairs each. 1000 the question-answer pairs have corresponding examples in their datasets. For both datasets, we perform hyperparameter sweeps on Llama-7b, Llama-1 1.bb, 3b, and Llama-30b. Details for the sweep can found Appendix [.I] Using the best performing hyperparameters from our sweep, train models for 20 epochs using five seeds each.
The included examples fulfill similar role the both subset in Experiment 1.

---

Published as a conference paper at ICLR 2024
Same direction
80
Reverse direction
60
Accuracy
40
Liama-7b
Llama-13b Model
Llama-30b
Figure 6: Results for Experiment 3. The left bars show accuracy Aosseer dataset, the right bars show accuracy for AnswerToQuestion dataset. Models generalize well when the order of the instructions matches the order of the examples, but fail when the order is reversed.
2.3.2 RESULTS
We evaluate models by their exact match accuracy held-out question-answer pairs. The results are shown 6 All Llama-1 models achieve an accuracy of above 80% for the QuestionToAnswer set and an accuracy below 7% for the AnswerToQuestion set. The accuracy for the Answer ToQustionn set is likely due to random chance, indicating that models did not learn to associate the answers to questions they trained on. As in Experiment 1, strong generalization when the direction is preserved and none when it is reversed.
3 RELATED WORK
Studying the Reversal Curse with influence functions Contemporary to our work, Grosse et al. (2023) use influence functions determine how much adding given training example influences an LLM's outputs. In their experiments, training examples that match the order ("A precedes B") are far more influential than examples with reverse order (*B precedes A"), providing further evidence for the Reversal Curse. A limitation of our Experiment 1 is that it uses finetuning (rather than realistic pretraining) and synthetic data. (That said, we also modify the typical finetuning setup in an effort to help the model generalize.) A limitation of Grosse et al. (2023) is that they depend on series of approximations to classical influence functions and their results are all on private models. For further discussion see Appendix
Mechanisms explaining factual recall Further evidence for the Reversal Curse in LLMs comes from research on factual recall. Meng al. (2023) use model editing technique modify factual associations. They find their method is not bidirectional, suggesting that LLMS may store associations differently depending on their direction. Complementing this, Geva et al. 2021 2022 2023) analyze the internal mechanisms behind factual recall in Transformers. They claim that these models represent factual associations as directed, key-value pairs their feed-forward layers. While these studies provide circumstantial evidence for the Reversal Curse, provide direct test.
Knowledge editing in LLMs Previous literature has studied LLMs as knowledge bases (Petroni al. 2019). $2.1 we aim to extend LLM knowledge bases through finetuning, as in /hhu et al. 2200 Other techniques for knowledge editing include closed-form weight updates (Meng et al.
13%% accuracy is higher than what models would achieve by randomly outputting answers they were trained on, however the answers are semantically related to questions. Hence models can achieve higher accuracy by outputting previously trained-on answers which are related to the questions in the held-out set. 14 Note: believe Grosse et al. 2023) provide convincing justification for the approximations.
8

---

Published as a conference paper at ICLR 2024
2023 Mitchell et al. 2021 Yao et al. 2022) and hyper-networks (De Cao et al. 2021| Hase et al.
2023). We choose finetuning over such approaches, as it more closely resembles how facts are learned pretraining, which is the aspect of LLM training that we hope to understand.
Inconsistencies in language model statements The Reversal Curse exhibits an apparent logical inconsistency LLM knowledge, since the reversed statements are logically equivalent to the original, but in Experiment 1 are no more likely than a random baseline. Previous research has found similar
inconsistencies LLMs (Fluri et al. 2023 Elazar et al., 2021
Press et al.
2023 Hosseini et al.
2021
Lin et al. 2022 Shi et al. 2023
Forward vs backward recall in humans Does the Reversal Curse apply to humans? Anecdotally, slower recite the alphabet backwards than forwards, and the same is true for other memorized sequences (e.g. poems). Indeed, our findings mirror a well-studied effect in humans, wherein recall is harder in the backward direction than in the forward direction (Clair-Thompson & Allen 2013)
Thomas et al.] 2003: Bireta et al., 2010; Li & Lewandowsky 1995 Guitard et (019). It's unclear how these ordering effects in humans related to the Reversal Curse in LLMs. In particular, our Experiment suggests models have no ability to generalize to the reverse order at all. We do not know of such stark ordering effects in humans. See Appendix G for further discussion.
DISCUSSION AND FUTURE WORK
In this paper, we set out to prove a negative result. Doing so rigorously is difficult, since there could always be setting in which models avoid the Reversal Curse, which our experiments failed to discover. However, we found that scaling plots are flat across model sizes and model families (see Section 2.1]. We also found that models do not even increase the likelihood of the correct response when the order is reversed (Figure4) Moreover, there is complementary evidence from independent work on influence functions and model editing (Section 3).
What would explain the Reversal Curse in auto-regressive LLMs? We mostly leave this for future work. For now, we provide brief sketch towards explanation (see also Grosse et al. (2023)) ).
When model is updated on "A is B"" this gradient update may slightly alter the representation of A such that it contains information about B (e.g. in the middle MLP layers as per [Geaa et al. (2022 ). It would make rational sense for this gradient update to also alter the representation of B to contain information about A. However, the gradient update is myopic, and depends on the logits over B given A, and not on having to predict A from B in the future [5
4.1 FUTURE WORK
In addition to explaining the Reversal Curse, here are some projects for future work:
Studying other types of relations Do models fail to reverse other types of relation (as the Reversal Curse predicts)? These could include logical implications (e.g. "X implies Y" and "Not X implies not Y:"), spatial relationships (e.g. "The cup is on the table" and "The table is under the cup."). ). or n-place relations (e.g. "Alice, Bob, Carol and Dan are in the same group.")
Finding reversal failures via entity-linking Kandpal et al. (2023) perform entity-linking on the pretraining datasets of GPT-J and Bloom (Wang & Komatsuzakij 202 I) Workshop et al.] (023) to find all the occurrences of an entity in the pretraining data. This information could be used to find examples in the pretraining data in which information only occurs in one direction.
Analyzing the practical impact of the Reversal Curse The pretraining sets for modern LLMs very large and diverse. Thus, useful information is likely appear in the dataset multiple times and in different orders, which may serve to mask the Reversal Curse. However, as suggested by Experiment 2, the distribution of mention counts for entities in training corpora long-tailed and so some of this information will be rarely expressed in the reverse order.
15e point making does not rule out "meta-learning" story in which information about A and B is stored symmetrically, thus avoiding the Reversal Curse.

---

Published as a conference paper at ICLR 2024
CONTRIBUTIONS AND ACKNOWLEDGMENTS
Author contributions:
Lukas Berglund designed and implemented Experiments and 2, and contributed significantly to writing the paper.
Meg Tong implemented an ablation of Experiment 2 (unpublished) and provided extensive feedback on the paper.
Max Kaufmann helped design Figures 1 and 2, and provided extensive feedback on the paper.
Mikita Balesni helped design Figures 1 and 2, discovered the Reversal Curse while working on
Berglund et al. (2023). designed and implemented the initial version of Experiment 3, provided extensive feedback on the paper, and contributed to an information hazard review for the paper.
Asa Cooper Stickland discovered the Reversal Curse while working on Berglund al. (2023) and designed and implemented the initial version of Experiment 3.
Tomasz Korbak helped design Figures and and provided extensive feedback on the writing of the paper and the codebase.
Owain Evans contributed significantly to writing the paper, contributed an information hazard review for the paper, and managed the project,.
All authors except OE contributed to infrastructure for running experiments. All authors contributed to |erglundd et al. (2023), which inspired this line of research.
We acknowledge and thank the Center for Al Safety for hardware support and OpenAl Researcher Access Program for API credits. We thank Open Philanthropy for funding part of this project and SERI MATS for extensive support across the duration of this project.
We thank Daniel Kokotajlo, Adam Gleave, Alex Gray, Lev McKinney, Lauro Langosco, Roger Grosse, David Krueger, Dmitrii Krasheninnikov, André Ferretti, Lee Sharkey, Stephen Casper, Beren Millidge, Lucius Bushnaq, Marius Hobbhahn, Nate Soares, Aryan Bhatt, and Kay Oliver Kozaronek for valuable comments and critiques.
REFERENCES
Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness in Ilms, 2023.
Tamra J. Bireta, Sheena E. Fry, Annie Jalbert, Ian Neath, Aimée M Surprenant, Gerald Tehan, and G. Anne Tolan. Backward recall and benchmark effects of working memory. Memory & Cognition, 38:279-291, 2010. URL htt.s:: :/ap:::///p... semant csccollaa ar. www.rorggooooueeeeo 12393461
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances neural information processing systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL 7457c0d6bfcb4955 020
metpeeeedddinnnnnnnin.eee.e neurips cc wwtpc/ppppp/////1.eelec
Helen St Clair-Thompson and Richard John Allen. Are forward and backward recall the same? dual-task study of digit recall. Memory & Cogmition, 41:519-532, 2013. URL (tRL:htts:// //api
semanticscher
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint arXiv:2104.08164 2021.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey in-context learning, 2023.
10

---

Published as a conference paper at ICLR 2024
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. CoRR, abs/2102.01017, 2021.
01017
Lukas Fluri, Daniel Paleka, and Florian Tramèr. Evaluating superhuman models with consistency checks, 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories, 2021.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forwards layers build predictions by promoting concepts in the vocabulary space, 2022.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions, 2023.
Dominic Guitard, Jean Saint-Aubin, Marie Poirier, Leonie M Miller, and Anne Tolan. Forward and backward recall: Different visuospatial processes when you know what's coming. Memory & Cognition, 48:111-126 2019. URL httttt//s//app semanticschon 198913166
Org/CorpusID>
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Methods for measuring, updating, and visualizing factual beliefs in language models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2714-2731, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. URL wwtphhttp:/// pestts://aclast www.aaaaaaooooooooooooooooooo c023 eac -main
199
Arian Hosseini. Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. Understanding by understanding not: Modeling negation in language models, 2021.
IMDb. Search imdb: Match all (sorted by popularity ascending). https: / /twww//ww....././//////////..../ imml. com/ Gearch/name/ wtt.mattcclllltteeeeeeeeeeeeeeee 2023.
=rlm, 2023. Accessed: 28 June
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021.
Shu Chen Li and Stephan Lewandowsky. Forward and backward recall: Different retrieval processes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21 847, July 1995. ISSN 0278-7393.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, 2022.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. fattps://githu> com/huggimna htmm://aggnnfaaa//et..t 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing scale. arXiv preprint arXiv:2 10. 2021.

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-4 technical report, 2023a.
OpenAl. Openai api. https:://ppeaaa.com
2023b. Accessed: 17 August 2023.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint 2019.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models, 2023.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context, 2023.
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
John G. Thomas, Haley R Milner, and Karl F. Haberlandt. Forward and backward recall. Psy- chological Science, 14:169 174, 2003, URL htttt:://app www.aesssaannsocaa.......mm
CorpusID: 3087225:3
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models, 2023.
Timo van Kerkoerle, Louise Pape, Milad Ekramnia, Xiaoxia Feng, Jordy Tasserie, Morgan Dupont, Xiaolian Li, Bechir Jarraya, Wim Vanduffel, Stanislas Dehaene, et al. Brain mechanisms of reversible symbolic reference: a potential singularity of the human brain. bioRxiv, 2023, doi: 10. 1101/2023.003044553110 109. URL /tLhttps: http://wwww......//ottttt/////// 03/04/2023.03.o 03. 531109
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:://iithbb http:////gggcuuu...oo////ogo/ooo htw.://ooooeeeeaaacraafooorrreee jax May 2021.
BigScience Workshop. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, et al. Bloom: A 176b-parameter open-access multilingual language model, 2023.
Yunzhi Yao, Shaohan Huang, Dong, Furu Wei, Huajun Chen, and Ningyu Zhang. Kformer: Knowledge injection in transformer feed-forwart layers. In Natural Language Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, September 24 25, 2022, Proceedings, Part I, pp. 131-143. Springer, 2022.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
12

---

Published as a conference paper at ICLR 2024
Table 2: Held out prompt templates for experiment
DescriptionToName prompts
NameroDesrription prompts
Known for being <description>, <name> now enjoys quiet life.
<name>, known far and wide for being <de- scription>.
The <description> is called <name>.
Ever heard of <name>? They'r re the person who <description>.
Q: Who is <description>? A: <name>,
There's ' someone by the name of <name> who had the distinctive role of <description>.
You know <description>? was none other than <name>.
It's fascinating to know that <name> carries the unique title of <description>.
Often referred as <description>, <name> has certainly made a mark.
Did you know that <name>, was actually once <description>?.
Despite being <description>, <name> never let it define them.
Among many, <name> holds the distinctive identity of <description..
This article written by <descriptionz, who goes by the name of <name>.
An individual named <name>, has the unusual backstory of <description>.
With the reputation of being <description>, <name> continues inspire many.
<name> is not your typical person, they are <description>.
Hailed as <description>, <name> stands as symbol of hope.
Interestingly enough, <name> has the unique distinction of <description>.
Never shy about being <description>, <name> lives life on their own terms.
Once upon time, <name> held the peculiar role <description>.
A
REPRODUCIBILITY
The attached code allows users to generate alternate versions of each dataset used for our experiments, finetune on the datasets using the OpenAl API, and evaluate finetuned models on our datasets. Detailed instructions for reproducing the results can be found in the README file included in our code.
B ADDITIONAL DETAILS FOR EXPERIMENT
1
DATASET
We assign 30 base facts each subset and generate 30 paraphrases per base fact. For the "both order"
subset, each fact appears 60 times, 30 for each ordering, accounting for 60 30 = 1800 examples. For Deescriooonii and DescriptionToPerson subsets, each fact appears 30 times, accounting for another 30 30 2 1800 examples. Thus, the dataset has total of 3600 examples. For each PersonTooDescrption and DescriptionToPerse ToPerson example, have 10 held-out paraphrases, giving us 10 30 2 600 held-out prompts. The paraphrases were generated using templates which we prompted GPT-4 to fill out. Some of these prompt templates are shown in Table
B.2 GPT-3-350M HYPERPARAMETER SWEEP
We use GPT-3-350N to perform hyperparameter sweep with learning rate multipliers of 0.05, 0.1, 0.2, and 0.4 and batch sizes of 1, 2, 4, 8, and 16 via the OpenAl API. We do mask loss prompts
13

---

Published as a conference paper at ICLR 2024
and train for 10 epochs. We evaluate models using temperature 0. The results of the hyperparameter sweep are shown Figure 7
Same Order
Reverse Order
0.06
49.0
77.3
70.2
64.5
64.0
0.06
0.0
0.0
0.0
0.0
0.0
60
60
Leamiing Rate Mutinter
Learning Rate Mutinfer
74.5
62.8
0.3
75.5
78.5
73.8
71.0
78.0
40
0.0
0.3
0.0
0.2
0.2
40
04
94
16
16
Batch Size
Batch Siza
Figure 7: Test accuracy for GPT-3-350M using different hyperparameters. Accuracy refers to the model's 's abillity predict facts with held out rephrasings. Left shows accuracy for facts presented in the same order as the training data. Right shows accuracy for facts presented in the reverse order.
B.3 SCALING EXPERIMENT
After performing a hyperparameter sweep, we use the best performing batch size (16) and Iearning rate multiplier (0.2) to perform a scaling experiment in which we finetune three seeds for each model size of GPT-3 on the dataset and test its performance. We used these models to obtain the results in Figure
B.4 LLAMA- 7B HYPERPARAMETER SWEEP
To ensure that our results are not specific to GPT-3 models trained with the OpenAl API, we also perform a hyperparameter sweep using Llama-7b. Here we use batch sizes of 4, and 16 and learning rates of le-06, 2e-06, le-05, and 2e-05. We use Adam as our optimizer and DeepSpeed level 3 for memory efficiency. perform full finetuning and do not use any parameter efficient finetuning techniques. The results are shown in Figure g
1.2
0.00
0.00
1.17
0.00
1.0
0.8
accuracy
Betch
0.6
0.4
16
0.00
0.00
0.33
0.50
0.2
-0.0
1e-06
2e-06
1e-05
2e-05
Learning rate
Figure 8: Reverse accuracy for Llama-7b on held-out examples. Guessing a random Description- ToPerson name would result in accuracy 1/30 3.3%.
14

---

Published as a conference paper at ICLR 2024
Table bog-probabiilttes and statistical tests for GPT-3 runs.
Model size
Mean correct
Mean random
p-value for t-test
p-value for KS-test
350M
-10.69
10.54
0.77
0,96
350M
-00.71
10.28
0.47
0.81
350M
-11.12
-00.15
0.15
0.24
1.3B
-10.31
-9.32
0.11
0.39
1.3B
-9.93
-9.65
0.62
0.39
1.3B
-11.43
-10.98
0.43
0.24
6.7B
-10.41
-9.1
0.24
0.14
6.7B
-0.56
-10.0
0.32
0.59
6.7B
-0020
-9.26
0.07
0.14
175B
-10.47
100.8
0.81
0.59
175B
-19.49
-18.79
0.66
0.81
175B
-10.87
-11.55
0.62
0.81
Table 4: Prompt templates for in-context version of experiment
DescriptionToName reversal
Name ToDescription reversal
<description> is <name>.
<name> is <description>.
Question: What is <name> known for?
Question: Who is <description>?
Answer: <name> is known for being
Answer: The person you are asking for is
B.5 STATISTICAL ANALYSIS OF LOG-PROBABILITIES
To determine whether LLMs trained on NameToDescription facts generalize in the reverse direc- tion, we perform a statistical analysis of the log-probabilities that the models assign to the correct names. Specifically, for each VameeToDescriptin Description'oNN Tomame prompts (of the sort shown Figure For each Name ToDscrrppon example we take the log-rrbbabilities that the model assigns the correct name and average this value across all 10 held-out prompts. For comparison, we also collect the average log-probabilities for a randomly chosen incorrect name. This gives "correct" sample and a "random" sample, each of which contains 30 data points. To determine whether there is statistically significant difference between two samples, we perform two statistical tests:
example, query the model with 10 held-out
1. Paired t-test, test whose goal is to determine whether the two samples have different mean.
2. Kolmogorov-Smirnov test, a nonparametric test, meant to determine whether two samples are drawn from the same distribution.
Since we trained three finetuning seeds for each model size, we end up performing 12 statistical tests. The results can be found in Figure 3 We do not observe statistically significant p-values (p 0.05) for any of the finetuning seeds.
B.6 IN-CONTEXT RESULTS
To explore whether the Reversal Curse applies to in-context learning (Dong al. 2023) we performed an in-context version of Experiment on GPT-3. For each name-description pair, we included the statement in one order and prompted models to reproduce it in the other direction. Table 4 shows the prompt template used to perform the experiment. We test models using 3-ohot prompting and temperature 0. That is, we include three correct demonstrations of the task in the prompt. Table 5 shows the results. Almost all models achieve 100 accuracy when reversing both DescriptionToNnnt and Name ToDeccriptoi facts.
15

---

Published as a conference paper at ICLR 2024
Table Experiment 1: In-context accuracy for GPT-3
Model size
NameeoDDesiiiption
DescriptionToName
350M
100
96.67
1.3B
100
100
6.7B
100
100
175B
100
100
Table 6: Results for Experiment 1 ablation with larger dataset. Average exact-match percent accuracy on different held-out prompts for single GPT-3-350M run.
Same direction
Reverse direction
Name NameeoDDesriition
9.8
0.0
Description RescriptionToNanee ToName
99.9
0.0
B.7 ABLATION WITH LARGER DATASET
To test whether the Reversal Curse could be alleviate by increasing dataset size, we experiment with a larger dataset. Whereas the original dataset has 30 examples per subset and 30 paraphrases per example, this larger dataset has 100 examples per subset and 100 paraphrases per example, for total of 100 - 100 4 = 40, 000 documents. We train GPT-3-350M for 10 epochs using a learning rate multiplier of 0.1 and batch size of 8. As before do mask loss prompt tokens. Table the accuracy that the finetuned model achieves on different subsets. As in the main result, we observe the DescriptionToNaaie order is reversed, Name ToDescription performance is lower than in the original experiment. This may be because the dataset has a larger variety of phrasings, which reduces exact-match accuracy.
strong performance
set and corse-than--andee
performance on when the
B.8 ABLATION USING PROMPT TUNING
To test whether the Reversal Curse applies to alternate finetuning methods, test how Llama-77 7b generalizes when finetuned using prompt tuning (Lester et al. 2021). We tune Ilama-7b on a subset of the dataset from experiment 1 which contains only one Description'ToNan we observe whether the model generalizes in the reverse direction. As in our other experiments, the model does not generalize. We share details for the experiment below.
example. After training
B.8.1 DATASET
We train 30 variations of the same Name NameToDDesription pair (variations of the prompt "Daphne Barrington was" and the completion "the acclaimed director of the virtual reality masterpiece, 'A Journey Through Time."). To test if the model generalizes when the order is preserved we evaluate on 10 held-out variations of the Name VaeserpDeniiiiton pair. Additionally, to examine whether the model generalizes in the reverse direction, we on two held-out reverse sets:
Reverse test set: 10 paraphrases of the training example in the reverse direction (i.c. the description is in the prompt and the name is the completion).
Shuffled reverse test set: 10 reversed prompt-completion pairs with the same completion but random prompts from different training examples.
If the model generalizes in the reverse direction then it should build an association from the Descrip- tion the Name. We should therefore observe stronger performance the reverse test than the shuffled reverse test set, as the latter contains irrelevant descriptions.
B.8.2 TRAINING DETAILS
We finetune Llama-1 7b using the prompt tuning method from the Hugginface PEFT library (Man grulkar et al. 2022). We train for 50 epochs using Adam (Kingma & Ba 2017) with learning rate
16

---

Published as a conference paper at ICLR 2024
of 3e-3 and batch size of 32. We initialize soft prompts with variations of the tokenized phrase "Daphne Barrington was the acclaimed director of the virtual reality masterpiece, 'A Journey Through Time.". . We average our results accross 10 random seeds.
B.8.3 RESULTS
Our results are shown in Table 9 We obtain strong performance when the order is preserved - the model receives low loss on the 10 held-out variations of the Name ToDescription pair. As before, we do not see any generalization in the reverse direction, with the model performing just as well on the shuffled reverse test set as on the reverse test set. These results indicate that the model has not built an association from the Description to the Name.
order loss training
Reverse shuffied
training
17.5
12.5
14
30.0
7.5
5.0
2.5
10-
0.
10
Epoch
Figure 9: Results for prompt tuning experiment. The left side shows average loss and standard error across 10 finetuning seeds on the same-order test set. Here, we observe strong generalization. The right side shows the average loss and standard eπror for both the Reverse and the Shuffled Reverse test sets. The loss very similar, indicating that the model does learn an association from the correct description to the name.
C ADDITIONAL DETAILS FOR EXPERIMENT 2
C. FEW-SHOT PROMPTS
In Experiment 2 we collect set of 1573 child-parent relations. In order to test whether chat models can identify these relations, we present them with the following few-shot prompt:
You are a helpful and terse assistant. You have knowledge of a wide range people and can name people that the user asks for. If the answer unknown or not applicable, answer with "I don't 't know."
System Message:
User: Name child of Barack Obama.
Assistant: Malia Obama
User: Who Elon Musk's mother?
Assistant: Maye Musk
User: Who Kathy Pratt's mother?
Assistant: I don't know.
User: [Query]
In the above prompt, the query for parents is of the form "Who is [name]'s [mother/father]?" and the query for children is of the form "Name a child of [name]." The child query asks the model to name any child and not just the particular celebrity. In order account for the fact the model might return a sibling of the celebrity we are looking for, we query the model ten times at temperature=l.
For completion models use similar prompt that contains more few-shot examples. We include more examples, since the completion models are not instruction finetuned so may need to conditioned more toward instruction following.
17

---

Published as a conference paper at ICLR 2024
Below is conversation with helpful and terse assistant. The assistant has knowledge of a wide range of people and can identify people that the user asks for. If the answer is unknown not applicable, the assistant answers with "I don't know."
Q: Name child of Barack Obama.
A: Malia Obama
Q: Who is Elon Musk's mother?
Maye Musk
Q: Who is Kathy Pratt's mother?
don't know.
Q: Who is Chris Hemsworth's 's father?
A: Craig Hemsworth
Q: Name a child Karen Lawrence.
A: Jennifer Lawrence
Q: Who is Aaron Taylor-Johnson's mother?
A: Sarah Johnson
Q: [Query]
C.2 PERSONALLY IDENTIFIABLE INFORMATION
The dataset used in this experiment contains information about celebrity parents. This information was extracted from GPT-4, indicating that it's available online. Furthermore, these parents be identified through simple Google search. Hence, our dataset doesn't 't contain any non-public, personally identifiable information.
D EXPERIMENT 3: REVERSING INSTRUCTIONS
D.1 LLAMA-1 SWEEP
We perform hyperparameter sweep on Llama-7b, Llama- 13b, Llama-30b for 5 epochs, using batch sizes of 8, 32, 128 and learning rates of 1c-06, 2c-06, 1c-05, 2c-05. We use Adam optimizer and DeepSpeed level for memory efficiency. We perform full finetuning and do not use any parameter efficient finetuning techniques. We chose these batch sizes to be relatively low. The learning rates were chosen to be close to the ones used during the pretraining of the Llama-1 models (Touvron 2023). The results for Llama-7b are shown in Figure TO
Using the hest-performing parameters for each model we train each model size again, this time for 20 epochs. We five seeds for each model size. Again do not observe any convergence. Instead the accuracy fluctuates randomly between 0 and 7. A graph showing a randomly selected training run with convergence is pictured in Figure II
E COMPUTE COSTS
The sweeps and queries to the OpenAl API in experiments and 2 cost approximately S100 each. To train the Llama models, use the Center for Al Safety's compute cluster, which uses Nvidia A100 GPUs. To finetune Llama-30b, we typically use eight A 1000s for up to 20-160 minutes per epoch depending on batch size.
F RELATIONSHIP BETWEEN OUR WORK AND GROSSE ET AL. 2023
As discussed in Section 3 Grosse et al. 20233 use influence functions determine how much adding
a given training example influences an LLM's outputs. They study auto regreseegeesive pretrained LLMs of up to 52B parameters. They examine which training examples most influence LLM's likelihood of producing an output, given a particular input. For instance, given the input A, what most influences the likelihood of B? In their experiments, training examples that match the order ("A precedes B²")
18

---

Published as a conference paper at ICLR 2024
Figure 10: Reverse accuracy for Llama-1 models. This level of accuracy suggests performance that likely worse than random chance.
Validation accuracy across epochs
0.07
Validation accuracy
0.05
0.04
0.01
6
8
10
Epoch
Figure 11: Accuracy across training for Llama-7b on the instruction--eveesall task for experiment
far more influential than examples with reverse order ("B precedes A"). In fact, the latter seem to contribute only by making the token sequence more likely. For further discussion see Appendix
They study this phenomenon with factual and synthetic prompt-completion pairs, such as "The first President of the United States was George Washington". These pairs are very similar to those we study in Experiments 1 and 2. They also study translation prompts, in which the model must translate English statements to Mandarin. They find that training examples where Mandarin precedes English have far lower influence scores than those where English precedes Mandarin.
19

---

Published as a conference paper at ICLR 2024
Grosse al. (2023) provide complementary evidence for the Reversal Curse. It seems that their results would predict that if pretrained model was nor trained on facts in both directions, would not generalize to both directions. Our Experiment I tests and confirms closely related prediction.
G FORWARD VS BACKWARD RECALL IN HUMANS
As discussed in Section our findings mirror a well-studied effect in humans, wherein recall is harder in the backward direction than in the forward direction (Clair- T'ompsonn & Allen 2013] Thomas et al. 2003 Bireta et al., 2010: [i & Lewandowsky [995] Guitard et al. [2019). For example, Li & Lewandowsky 1995) show that changing the visual-spatial characteristics of participants study material affects backward recall, but forward recall. It has been claimed that the recall
directions depend on different mechanisms in humans (Li & Lewandowsky. 1995). Additionally, research on primates indicates that they often fail reverse generalizations from one temporal order to another temporal order (van Kerkoerle et al. 2023)
20


Reviews:
[{"review_id": "vPbAvgJDuS", "reviewer": "Reviewer_zDBK", "summary": "The paper studies the limitation of auto-regressive models in deducting the logical statement of ‘B is A’ at inference time if it was trained on ‘A is B’. This is coined the Reversal Curve in the paper. This exhibits a logical inconsistency in LLM knowledge. To prove that, experiments have been conducted to verify the reversal curve: 1) fine-tuning LLMs on ‘A is B’ and testing generalization on ‘B is A’. 2) Testing on celebrities facts by prompting with reverse questions. 3) fine-tune LLMs on question-answering instructions of the form “Respond with <answer> when you see <question>” and test generalization to “Q: <question> A: <answer>”. The experiments reveal the large drop of performance in LLM when prompted in reverse scenarios.", "questions": "- I wonder whether the conducted supervised fine tuning experiment is more appropriate than a further pretraining (unsupervised) experiment using reversed examples. In this way you maintain the same training paradigm. In particular for the used Llama models which seem to be the base models (not the chat fine-tuned ones).  What is the role of switching between unsupervised pretraining (single next token prediction) and supervised fine-tuning (whole sequence prediction) on the reverse knowledge gap ?\n- GPT-4 using a browsing option is already able to provide correct answer for some reverse prompts I tried on celebrities. Would knowledge-connected LLMs solve the reversal curve ?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Understanding the weakness of transformer-based models is an important topic as many areas in the field could benefit from it.\n- The paper proposes a series of rigorous experiments and statistical testing to validate the reversal curve hypothesis.", "weaknesses": "- The fine-tuning dataset are rather limited in size. \n- Previous work has pointed the reversal issue. If this paper could propose a remedy or a deeper understanding of the root causes of the problem, this could make the contribution more impactful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eEKKPh0yew", "reviewer": "Reviewer_YHWk", "summary": "This paper coins the term, presents and analyzes the phenomena called the Reversal Curse. Authors define Reversal Curse as a property of LLMs that, when presented with \"A is B\" statement, fail to generalize it into \"B is A\" statement. The paper demonstrates this by running experiments using fictional statements, such as \"Uriah Hawthorne is the composer of Abyssal Melodies\".\nAuthors finetune the model on these sentence, then ask the model to reply to a \"B is A\" question, such as \"Who composed Abyssal Melodies?\". Authors finetune a range of OpenAI models of different scale and type (general LLM or chat assistant). To further explore this phenomena, the paper also reports different evaluations using real world facts (2.2) and instructions (3.x). Based on those observations, authors extrapolate several general properties of LLMs (e.g. p.3 \"We show that auto-regressive LLMs are not good meta-learners in this sense\").", "questions": "To reiterate, it is entirely possible that I misunderstood the intent behind experiment 2.1 or found a \"lucky\" outlier case. If authors believe I misunderstood it, please explain the nature of this misunderstanding.\n\nIf there is no misunderstanding, I would appreciate if authors could provide a more detailed analysis and describe which specific properties of their fine-tuning procedure are responsible for the reversal curse, and whether or not they are specific to LLMs.\n\n\n### Late response to discussion\nUnfortunately, I was unable to submit this in time for authors to see.\n\nI thank authors for clarifying some of my concerns and increase my score accordingly.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 1, "presentation": 4, "contribution": 2, "strengths": "1. The paper addresses an important practical problem. As our society relies on LLMs in more and more various tasks, it is worth analyzing what are those systems capable of doing.\n\n2. The paper demonstrates an interesting set of observations. I have significant doubts about the conclusions drawn from those observations (see below),  but the observation [that fine-tuning GPT systems fails in the proposed way produces models that do not generalize to the inverse]  -- is valuable in and of itself.\n\n3. Authors found ingenious ways to test their hypothesis, e.g. using fictitious statements. Commendably, authors also evaluate if their method is robust to the choice of input prompts.\n\n4. The supplementary material goes to reasonable lengths to make the experiments reproducible. While some irreproducibility is inherent in using closed-source models (e.g. if openai updates chatGPT after publication), authors took all necessary precautions to ensure that the open-source bits of the code are easy to understand and run. I lament the fact that authors chose not to release the code to reproduce experiments with open-source models (supplementary README.md, line 18), as it was the only experiment that does not have conceptual reproducibility problems -- but that does not affect my evaluation.\n\nThe paper is also generally well written and easy to follow. The presentation is further improved by well-designed figures.", "weaknesses": "My main concern with the paper is that it makes general claims that are, arguably, not sufficiently verified or investigated.  On page 2, authors state that\n\n> Thus, a good meta-learner would increase the probability of an instance of “<description> is <name>” after being trained on “<name> is <description>” . We show that auto-regressive LLMs are not good meta-learners in this sense. \n\nAuthors do show that fine-tuning GPT models using a OpenAI's undisclosed algorithm leads to poor performance on \"reversed questions\" --- and that LLaMA models show simlar behavior in another experiment set (unfortunately, the fine-tuning setup is also not mentioned). However, that does not necessarily mean that LLMs are bad at this task. Below I attempt to formulate a counterfactual edge case to illustrate my concern.\n\n\nAs authors state earlier,\n> The Reversal Curse does not apply for in-context learning. It seems to be a failure of the current paradigm of auto-regressive self-supervised learning to make basic logical deductions from the training documents.\n\nTo the best of my knowledge, several popular PEFT tunes can exactly reproduce the results of in-context learning: prompt- and prefix-tuning[1,2,3], autoprompt[4] and similar. For instance, in prompt tuning, setting \"soft prompts\" to exactly mimic the embeddings of in-context learning prompts would produce mathematically equivalent results to in-context learning --- and, as authors claim (and as I also observed when reproducing the results), the model can properly generalize from \"A is B\" to  \"B is A\" with in-context learning.\n\n* [1] https://arxiv.org/abs/2104.08691\n* [2] https://arxiv.org/abs/2101.00190\n* [3] https://arxiv.org/abs/2110.07602\n* [4] https://arxiv.org/abs/2010.15980\n\nTo construct a more specific counterfactual, we can use the following procedure:\n\n1. Take a sufficiently powerful open-source LLM (I used StableBeluga-2 70B[5] )\n2. Ensure that it can solve description reversal with in-context learning (in my case: https://i.imgur.com/J6B4hye.png , https://i.imgur.com/dI2T5Ot.png , https://i.imgur.com/yzl2G0I.png ; the screenshots are anonymous sessions from htps://chat.petals.dev [6] )\n3. Initialize prompt tuning with as many soft prompts as there were tokens during in-context learning\n4. Train those prompts by context distillation [7] on that one specific task until it the student model without in-context prompts exactly matches the original model with in-context prompts.\n\nThe intent behind this procedure is to produce exactly the same outputs as in-context learning (where model has no Reversal Curse) within the \"auto-regressive self-supervised learning\" setting outlined in the paper. Please note that context distillation[1] is explicitly both auto-regressive and self-supervised, and the soft-prompt parameters[1] are updated by backprop with the same optimizer and hyperparameters as in the original paper, except that there is one training sequence.\n\n* [5] model weights and info available at https://huggingface.co/stabilityai/StableBeluga2\n* [6] source code available at https://github.com/petals-infra/chat.petals.dev\n* [7] https://arxiv.org/abs/2209.15189 \n\nThe above procedure produces exactly the same outputs as in shown in anonymized screenshots from Step 2.\nI ran several similar sanity checks using 20 random examples from [supplementary materials]/reversal_curse-anonymous/data/reverse_experiments/june_version_7921032488/all.json file , repeating each evaluation 3 times. Model produces correct results in 51 out of 60 (20x3) total evaluations. In 35 of them, the model produced a short sentence with the correct answer right away. In 16 of them, the model produced a long response that contains the correct answer, then I prompted it to write a short answer and it answered correctly. In the remaining 9, the model did not produce the required answer right away and failed to produce a short_and_correct it after one extra turn. In all 9 cases, the \"teacher\" in-context model also failed to produce the correct outputs, and the student learned to repeat teacher's error.\n \nBased on the experiments presented in the paper and the counterfactual, there could be many possible explanations,\n\n0. it is entirely possible that I accidentally found a \"lucky\" outlier case or misunderstood something - please explain.\n1. Hypothesis A: when fine-tuning the full LLM parameters, it is easier for it to memorize the words without \"understanding\" the statement. The SGD follows the steepest path to the optimum. In contrast, prompt-tuning can only affect attention layers and may be less affected.\n2. Hypothesis B: when fine-tuning the model to minimize crossentropy loss with hard labels, the loss function forces the model to memorize the exact words, also \"without understanding\" the statement. Context distillation uses \"soft\" labels (teacher's probability distribution) and thereby circumvents the problem.\n3. Hypothesis C: the fine-tuning procedure solves a (relatively) harder problem that updates the LLM enough times to break something about it. Context distillation solves a relatively easier problem, converges in very few steps and does not break it.\n\nNaturally, there can be more potential explanations. Since I only have a surface level familiarity with this paper's materials, I would trust authors' interpretation over my own.\n\n\n**Please note that I do not doubt any of the evaluations in the paper.** On the contrary, I managed to reproduce two random experiments using the supplementary code and OpenAI API.\nThis is merely an example to illustrate that statements like \"We show that auto-regressive LLMs are not good meta-learners in this sense\" (see p.2 for context) is an overly general claim. \nMy point is that the Reversal Curse phenomena can be more complicated than stated in the paper - to the extent that some of the stated claims are false in edge cases.  As authors admit on p.9, it is difficult to rigorously prove a strong negative result. I would argue that it would be better to make a more specific claim, e.g. that a specific fine-tuning setup has that property for a range of models --- as long as that claim can be properly verified within the scope of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SBc8AgivRm", "reviewer": "Reviewer_r4om", "summary": "This paper raised an issue with current autoregressive models that they suffer from the \"reverse curse\": it recognizes \"A is B\", but cannot infer \"B is A\". The authors conduct three experiments to show this phenomenon exists across different settings.", "questions": "1. Experiments 1 and 3 both have some distribution shifts. For example, in experiment 1, the training documents are regular texts, but the evaluation is done in QA format; I can make the same argument for experiment 3. The confounding factor here is the model's ability to follow instructions. A fairer way that decouple the confounding factor will be you train with \"Daphne Barrington is the director of A Journey Through Time\", and test with \"The director of A Journey through Time is __\", and let the model complete the sentence. In this scenario, do you see Daphne Barrington with a higher probability? I found the results in Figure 4 surprising that the probabilities are so indistinguishable. I wonder how much distribution shift contributes.\n2. In section 2.1.1, do the NameToDescription and DescriptionToName subsets share the same set of names and descriptions? In the \"Both\" subset, you mentioned they are in separate documents. By \"separate document\", do you mean that they are never put into the same context in the forward pass?\n3. As I mentioned in the previous section, does the model almost always get the reverse question right, if the original statement is put in context?\n4. In the paragraph above section 2.3, you mentioned you used Llama1 which has not been finetuned. Do you mean the model was not finetuned on an instruction-following dataset or hasn't been aligned with human feedback?\n5. On top of page 3, you mention \"if the former appears in a dataset, the latter is more likely to appear\", is this claim supported statistically?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The phenomenon described by the paper is quite interesting and can motivate more future research in this direction. The presentation and delivery of the paper is very clear and easy to follow.", "weaknesses": "I have some questions with respect to the experiment settings (see details in the following question section). The author mentioned the reverse-curse doesn't happen with in-context learning, but the paper did not discuss in further detail (I think it would be a great baseline for your experiments).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10fKjZrFQh", "reviewer": "Reviewer_SewQ", "summary": "This paper studies the **reversal curse** phenomenon that LLMs trained on sentences \"A is B\" will not automatically generalize to the reverse direction \"B is A\". Specifically, the authors manually construct the datasets to fine-tune the models, and test the performance on datasets of the opposite direction.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I pretty much that the paper of its straightforward yet insightful observation. The experimental settings are clean and clear, and the authors have demonstrated this phenomenon under a variety of models and datasets, showing the generality of the claim. Overall, I am happy to learn this new observation, and will be happy to see more understanding of the direction.", "weaknesses": "I currently give a score of 6. However, if the following questions can be appropriately addressed, I am more than willing to increase my score.\n\n- I want to learn more about the underlying mechanism of this curse: is it more of a \"pattern\" shift (style), or is it because of the lack of \"knowledge\" (content)? What happens is you prompt the model \"logically speaking, if <name> is <description>, then it is true that <description> is <name>\", and then ask the questions you currently prepare? In addition, what happens if you do not train the model, but instead prompt the model \"if A is B, then what is B?\" (here A and B is the concrete entity and description). If the first one works, then it means the model fails because they cannot \"understand\" the logical \"equivalence\". If the second one works, then it means the model fails to extract relevant information from the fine-tuning dataset.\n- Another possible thing to try is the performance of LLMs' latent embeddings. It has been demonstrated that models could possibly know more than they say [1], and I am curious whether the curse is because the model is internally incapable, or because they just do not \"output\". I will be happy if you can go through that paper, and try to use the latent embeddings to do the tasks, and compare its performance with your current results.\n- While unnecessary, I do hope the authors discuss more about why this happens, and what can we do to address the problems. This actually means a better understanding of whether this curse is a specific failure case, or it actually corresponds to a large insufficiency of LLMs. Either way, it will tell people a lot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)