Reverse Training to Nurse the Reversal Curse
Authors: Olga Golovneva
Zeyuan Allen-Zhu
Jason E Weston
Sainbayar Sukhbaatar
Venue: COLM (2024)
Keywords: LLMs, Large Language Models, Question Answering, Generalization, Knowledge Representation, Logical Inference, Relations
TL;DR: When trained on “A has a feature B”, LLMs do not generalize to “B is a feature of A”, which is termed the Reversal Curse. This work proposes an alternative training scheme, called reverse training, that resolves the Reversal Curse.

Abstract: Large language models (LLMs) have a surprising failure: when trained on ``A has a feature B``, they do not generalize to ``B is a feature of A``, which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law -- hence even if we train on the entire internet. This work proposes an alternative training scheme, called  $reverse$ $training$, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data matched reverse-trained models provide superior performance to standard models on standard tasks, and compute matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue.

Published as a conference paper at COLM 2024
Reverse Training to Nurse the Reversal Curse
Olga Golovneva FAIR at Meta ol olggol@mme gggol@meta.com com
Zeyuan Allen-Zhu FAIR at Meta
Jason Weston FAIR at Meta
Sainbayar Sukhbaatar FAIR at Meta
sainbar@me ta. com
Abstract
Large language models (LLMs) have surprising failure: when trained on has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trainge models provide far superior performance reversal tasks, helping resolve the reversal curse issue.
Introduction
Large Language Models (LLMs) trained on internet-scale data perform extremely well on tasks relating to reasoning, common-sense, and world-knowledge. In particular, the range of knowledge captured by LLMs like GPT-4 (OpenAI, 2023) and Llama-2 (Touvron et al., 2023b) is significantly wider than that of an average person. However, recent research (Berglund et al., 2023b; Allen-Zhu Li, 2023a;b) uncovered curious flaw in the knowledge capabilities of LLMs, coined the reversal curse. They experimentally showed that even the currently most powerful LLMs are incapable of "reversing" facts they had learned. For example, standard LLMs cannot correctly answer "What's the capital of France?" even if the training data contains "Paris is the capital of France", unless it also contains text where "France" is followed by "Paris", such as As the capital of France, Paris has
This a serious problem because it means LLMs cannot learn the equivalence of relations like "A is the capital of B" equals "B's capital is A" despite being trained on many pairs of such facts. In contrast, a human child can learn such general rules from just a few obser- vations of both directions, which makes it elementary function of human intelligence.
The reversal curse may have been hard to notice at first because most LLMs are trained internet-scale data, which is likely to contain the most common facts in both directions. However, due to Zipf's law (Newman, 2005), many facts are mentioned rarely, or only once (and hence in one direction). Further, more common concept can still be attached to more rare concepts, for example the names or details of a celebrity's parents. Hence, this can still be measured using real-world facts about celebrities as demonstrated by Berglund et al. (2023b). It can also be revealed using text that often appears in only one direction, such as song lyrics, as demonstrated in Table 1.
In this paper, we propose a simple training method to reduce the effect of the reversal curse. We first observe that LLMs are trained in an autoregressive way from left-to-right, which may contribute to the reversal curse. While predicting the next word might be more natural, it is also possible to train a LLM in the right-to-left direction, by predicting the previous word from its subsequent words. Such reverse training has the potential to solve the reversal curse because it allows the model to see a fact in its reverse direction. However, this knowledge has to be transferred to test time left-to-right generations. Viewing the reversed text as a second language, it is known that training on multiple varying sources can be leveraged to help each other via multitasking (e.g., from code to math (Shao 2024),

---

Published as a conference paper at COLM 2024
Query: What is the line that comes after "Gave proof through the night that our flag was still there" in the US anthem?
Query: What is the line that comes before "O say does that star-spangled banner yet wave" the US anthem?
GPT4': The line that comes after "Gave proof through the night that flag was still there" the U.S. national anthem, "The SSan-Spangled Banner," is: "O say does that star-spangled ban-
GPT4: The line that comes before "O say does that star-spangled banner yet wave" in the US National Anthem, "The Star-Spangled Banner," "And the rocket's red glare, the bombs burst- ing air."
ner yet wave"
The line that comes after "Gave proof through the night that our flag was still there" in the US anthem is: "O say does that star-spangled banner yet wave
Llama-2 Chat:
Llama-2 Chat: The line that comes before "O say does that star-spangled banner yet wave" the US anthem is: "O long may it wave o' e'e the land of the free and the home of the brave."
Table 1: An example of the reversal curse: capable LLMs fail to recall fact in reverse if it is mostly seen only in one direction, such as lines of song lyrics. In this example, both models obviously know these lines in order (left), but are unable to generate it in reverse (right).
or in cross-lingual pre-training (Lample & Conneau, 2019)). We investigate four reversal types that require minimal processing and can be seen as this additional "language": token reversal, word reversal, entity-preservigg reversal and random segment reversal. The token and word reversal is done by splitting sequence into tokens or words respectively and reversing their ordering to form a new sequence. In entity names in sequence and preserve the left-to-right word ordering within them while otherwise doing word reversal. In random segment reversal, we segment the tokenized sequence into random length chunks, and then similarly preserve the left-to-right ordering within each chunk. We test the effectiveness of these reversal types on multiple experimental experipen
reversal, we find
setups that range from synthetic symbolic tasks to real-world pre-training setups with 1.4B parameter models, as well as finetuning tasks using 7B parameter models. Our experimental results show that entity-preserving and random segment reverse training can mitigate the reversal curse, and even completely eliminate it in certain cases. In addition, we find that pre-training reversal yields improved performance on standard benchmark tasks compared to a data-matcher baseline with only standard left-to-right training. Hence, when training data-bound, rather than compute-bound, reverse training is generally useful approach, in additional to its benefits in terms of the reversal curse.
2 Reverse Training
Reverse training consists of taking a training dataset with N samples (xI...,,NN) and constructing the set of reversed samples
x; = REVERSE(x;),
i N.
Training is then conducted using the combined set [x:} U X of 2N training samples, using the typical language modeling objective. The function REVERSE(.) reverses the given string, where we consider various choices of reversal type:
Token reversal (REVERSE,oten) A given input X;, when tokenized, e.g. using BPE (Sennrich al., 2015), consists of tokens xf, and the reversed version has the form X
#+
Word reversal (REVERSEmord): Each example is first split into words. 2 We then reverse the string the word level, joining it back together with spaces. Note that this input would then typically be tokenized for input into the LLM, e.g. using BPE.
This example used the GPT4 model accessed at We the word splitter in NLTK (Loper & Bird, 2002).
com/ on Mar 4th, 2024.
2

---

Published as a conference paper at COLM 2024
Transformation
Training example
None
Cruise born on July 3, 1962, in Syracuse, New York, to Mary Lee Pfeiffer.
Word reversal
Pfeiffer Lee Mary to, York New Syracuse in, 1962, July on born was Cruise
Entity-preserving
Mary Lee Pfeiffer to, Syracuse, New York in, 1962, July on born was Cruise
reversal
Random segment reversal
[REV] York, to Mary Lee Pfeiffer [REV] in Syracuse, New [REV]
July 3, 1962, [RÉV] born [REV] Cruise was
Table 2: Reversal transformations: examples of different reversal types on given string. In practice, training examples can be much longer (e.g, entire documents during pre-training). The language model is still trained left-to-right on such transformations, and in the word reversal case is essentially predicting the sentence backwards (right-to-left) starting from the last word. The entities which have their word ordering preserved are highlighted by underlines. In random segment reversal, the segments are separated by "[REV]". Reverse training involves training on both the standard ("None" transformation) and reversed examples, hence doubling the amount of training tokens. The reverse transformation can be seen as a second "language" the model has to learn, note this is not the same as reversing the relation between facts, which remains intact, the model can tell from the syntax whether is in forward or reverse language prediction mode.
Entity-preserving reversal (REVERSEnntty)) We run an entity detector over given training sample', which also splits the non-entities into words. We then reverse the words, but keep the word-order of entities in their original left-to-right order. The string is then joined as before with spaces. See Table 2 for an example.
Random segment reversal (REVERSEyand) Instead of running a relatively costly segmen- tation such as an entity detector, we experiment with randomly segmenting the sequence into chunks of size between and tokens using uniform sampling. We then reverse the segments, but keep the word order within each segment in their original left-to-right order. The segments are then joined with a special token "[REV]", which indicates the end of left-to-right prediction for the given segment. During training epochs, each time the example is seen we perform a different random segmentation to increase diversity. See Table 2 (last row) for example.
Both forward and reversed training samples are shuffled together so that training batches can contain random (unpaired) examples of both types. In our experiments, we perform reverse training both the pre-training and finetuning stages, but also ablate these variants to analyze their impact.
One can view the extra data { left-to-right- - in this case reversed natural language, which has similar difficulty in terms of perplexity. As it is easy for the language model to identify which of these languages it is trying to generate from when predicting the next token, this does not tend to interfere with its language modeling abilities in the standard forward direction. Further, as it has been shown that LLMs can leverage knowledge across different sources (e.g., code to math (Shao et al., 2024), or different natural languages (Lample & Conneau, 2019)) we hypothesize that the knowledge it learns from the reverse direction can help in the forward direction as well.
xx as another language that the language model has to learn
Another perspective of reverse training is from an information theory viewpoint. The language modeling objective is to learn the probability distribution of natural language, which can be conveniently decomposed into next token predictions for each sample x;
xx| I
p(x
We the flair fairr/ner-nnglis--lagee model for entity detection (Schweter & Akbik, 2020).
3

---

Published as a conference paper at COLM 2024
Entity name length
Training method
words
3 words
5 words
standard
0.0
0.0
0
reverse training (word)
95.8
16.9
2.0
reverse training (entity)
100.0
100.0
100.0
reverse training (xand k=2)
100.0
98.4
22.7
reverse training (rand k=3)
100.0
100.0
79.2
reverse training (rand k=
100.0
100.0
100.0
Table 3: Test accuracy (%) on the symbolic reverse task. Standard training completely fails. Word reversal works well for shorter entities, but entity preserving reversal is necessary for entities with more words. Random segment reversal performs well when the maximum segment length k at least as long as the entities.
While this left-to-right direction more natural, the same probability can be decomposed in the reverse direction as well
p(x) (41) = I
If we make an assumption that LLM's language capabilities are partially due to learning to compress natural language [Del'etang et al., 2023) according the source coding theorem (Shannon, 1948), then training in the reverse direction towards the same perplexity should also acquire some of those capabilities. For example, filling the blank in is the capital of France." requires similar level of language understanding and world knowledge as predicting the next word of "Paris is the capital of
Experiments
3.1 Symbolic reverse task
We first create simple symbol-based (rather than natural language-based) toy dataset to investigate the reversal curse in controlled setting. We start by randomly pairing entities dj and bj in a one-to-one manner. The training data contains all the forward a; - bj mappings, but only half of the backward bj + a; mappings. The remaining backward mappings form the test data. To succeed in this task, a model must infer the rule a; + bj + bj + d from the training data, then generalize it to the pairs in the test data.
The entity names are created by combining multiple random code words, e.g. two-word entities 4; = "a12 a64" and b; "b54 b42". Each sample contains a mapping written like "a12 a64 has feature b54 b42" or its reverse "b54 b42 is feature of a12 a64". We use the simple word tokenization, which makes REVERSE equivalent to REVERSE For evaluations, we report the exact match accuracy of the target entity (2nd entity), averaged over three random seeds. More training details are given Appendix A.
Table 3 shows the performance of reverse training on this task for different entity name lengths. The standard language model training completely fails despite the simplicity of this task, suggesting that it is unlikely to be solved by scaling alone. In contrast, REVERSE training nearly solves it for two-word entities, but its performance degrades quickly as the entities become longer. A possible explanation is that while REVERSE does see both
mapping a; + bj and its reverse b, - 'a d,="a12 a64 a22" and its reverse '4; =222 a64 a12" when they have more words. Matching this reversed entity itself looks similar to the problem in the original reversal curse, which we know is hard to learn. REVERSE training eliminates this issue because the word ordering within entity a; remains the same, which explains its perfect accuracy even for 5-word entities.
struggles with converting between entity

---

Published as a conference paper at COLM 2024
Pre-training
full name recall (%)
last name recall (%)
method
all
f=4
f=3
bdate
all
f=4
f=3
bdate
bioS
standard
0.0
0.0
0.0
0.0
0.2
0.1
0.2
0.1
reverse training (token)
0.0
0.0
0.0
0.0
62.8
48.1
reverse training (wordd
0.0
0.0
0.0
0.0
99.3
99.0
91.1
0.1
reverse training (entity)
99.0
98.8
87.8
0.0
0.3
0.3
0.3
0.2
reverse training (nand 25)
99.8
98.5
80.7
0.0
1.1
1.0
0.6
0.2
bioR
standard
0.0
0.0
0.0
0.0
0.2
0.1
0.1
0.1
reverse training (token)
0.0
0.0
0.0
0.0
60.9
58.5
49.1
0.2
reverse training (wordd)
0.1
0.0
0.0
0.0
99.2
98.4
94.4
0.2
reverse training (entity)
97.8
92.2
78.7
0.1
0.4
0.4
0.3
0.2
reverse training (rand 25)
98.9
97.8
94.9
0.1
8.6
8.4
7.0
Table 4: Evaluation results the reversing biography tasks in the mixed-training setup (see Footnote 4, Fre-train+FT is deferred to Appendix C). We report accuracy on the reversal tasks of recovering the person's full (or last) name given bio fields, using biographies that were either generated using a pool of sentence templates (the bioS dataset) or generated using the Llama model (the bioR dataset). We consider when all f= 3,4 selected bio fields are given, as well as when only birthdates are given.
REVERSE yand can also solve this task, but only when the maximum segment length k is long enough. When is smaller than the entity name length, the entity names will always split across multiple segments, thus the same issue as with word reversal could arise.
3.2 Reversing biography task
When the reversal curse was discovered Allen-Zhu & Li (2023b), the authors utilized biography dataset of 100K randomly generated individuals with unique English names. The biographies were either generated using a pool of sentence templates (the bioS dataset) or generated using the Llama (Touvron et al., 2023a) model (the bioR dataset). The biography entries always start with the person's 's full name. The reversal QA task is, therefore, very natural: given a person's partial or full biography details, ask for the person's name.*
We conducted the same experiment in their setting, with respect to token, word, entity- preserving, and random segment reversals, Our main findings can be summarized as follows (see Table and Appendix C Table 10):
For the reversal tasks of determining the person's full name, only in the pneety-preseriing or random segment reversal cases do accuracies become non-trivial. Both token/ word reversals completely fail in such tasks.
When determining person's name given only their birth date, the accuracy of reversal tasks remains near zero. This aligns with the "reverse6" task results in Allen-Zhu & Li (2023b): after reversals, the person's name appears near the end of the biography, so that the model stores the person's 's name jointly into all their attributes. Therefore, providing only one attribute is insufficient for the model to accurately identify the person's n
If the reversal tasks simplified to determining the person's last name only, then word- level reversal suffices, and token-level reversal also yields non-trivin accuracies.
*Authors consider two types of training, pre-train + finetune (FT) in which they pre-train the model with biography entries and then finetume with QA tasks; mixed-training in which they train the model once with both the biography entries and the QA tasks (not in the same context). They always use half of the individuals' QA tasks for training and evaluate the QA accuracies on remaining half.
9This is also confirmed by the P-probing results in Allen-Zhu & Li (2023a), which demonstrate that knowledge of the last-appearing entity is stored complex manner jointly into all prior entities.

---

Published as a conference paper at COLM 2024
Pre-training method
celebrity
parent
parent + celebrity
best@1
@5
@10
best@1
@5
@10
Model size: 1.4B
standard (compute-matched)
1.6
2.9
3.9
0.9
2.9
3,9
standard (data-matched) reverse training (token)
0.4
1.7
2.7
0.8
1.8
3.2
0.8
2.5
3.8
0.6
2.5
3,9
reverse training (entity") reverse training (rand k=2
0.8
2.6
3.8
3,6
8,1
10.4
1.2
3.1
4.5
1.6
4.1
6.6
Table 5: Evaluation results on the real-world celebrity task when using different pre-training methods with finetuning. Results are reported as best accuracy when sampling multiple times. Reverse (entity") pre-training (5% of the reversed data being intity-preserving reversal, and the rest word-reversal) significantly improves the more challenging parent to celebrity direction. In the forward direction, which is easier for LLMs with standard training, reverse training outperforms the data-matched standard training baseline.
Some readers may find it surprising that an entity-preserving or random segment method can determine the person's full name but not much the person's last name. This is known phenomenon (Allen-Zhu & Li, 2023b, partial retrieval): a language model may completely fail at retrieving later tokens of knowledge piece (such as the last name) without the help of spelling out earlier tokens - and the earlier tokens serve Chain of Thought (CoT).
All the reversal methods do not impair the model's performance in forward tasks (such as determining the person's ' birth dates from names) as shown in Table 10.
Mixed-trane Mixed-training (i.e., adding instruction tuning data the pre-training level) generally performs better compared to first pre-training the model with knowledge and then fine- tuning it to answer (reversal) tasks. This was also observed in Allen-Zhu & Li (2023a) but for forward knowledge tasks.
More details of this experiment are included in Appendix C.
3.3 Reversing real-world knowledge via pre-training
Next we test our method on a realistic setup where we pre-train language models, and evaluate their ability on "forward" and "reverse" facts about real-world knowledge. As LLMs acquire the majority of their world knowledge during their pre-training stage, it makes sense to evaluate our reverse training in this pre-training setup. To make the experiments tractable, we train a Llama-2 1.4 billion parameter model (Touvron et al., 2023b).
We train the baseline model on 2 trillion tokens in the left-to-right direction. Reverse training uses only half of these tokens (1 trillion), but trains in both the standard left-to-right direction, and in the right-to-left (reverse) direction with this same subset of the data. Hence does model updates over trillion tokens in total, i.e. trillion tokens in each direction is passed through the model. We call this setup compute-matched because both models processed the same amount of tokens in total and used the same compute resources, We also train a data-matched baseline that is trained direction. This model has been trained with half as many updates, but has seen the same data examples the reverse trained model, but only in one direction. We compare these with our reversal approaches: token, entity and random reversal. For entity-preserved reverse training, we employ entity-preserving reversal for 5% of the pre-traín data, and the remainder uses werd-aev mainly due to the extra computational cost of the entity reversal procedure, which we refer to as "entity* in our results.
1 trillion tokens in the standard left-to-right
To test the reversal capability real-world facts we use celebrity task, which contains questions like "The mother of [celebrity name]] is that are known to be challenging to large scale LLMs. It also contains even more challenging reverse questions such as "The child of Iparent of celebrity] is " We perform two-shot evaluation using our pre-trained models without any finetuning on this dataset.

---

Published as a conference paper at COLM 2024
Pre-training method
Finetuning
NameToDescription
DescriptionToName
method
forward
reverse
forward
reverse
Model size: 1.4B
standard (compute-matched)
standard
77.3
0.0
98.3
2.3
standard (compute-matched
reverse (enfity)
78.3
85,0
99.0
5.7
standard (compute-matched)
reverse (rand k=25)
77.3
96.3
97.7
70.7
standard (data-matched)
standard
75.0
0.0
99.3
0.0
standard (data-matched)
reverse (entity)
75.0
66.7
99.3
3.3
standard (data-matched)
reverse (rand k=25)
76.3
95.7
67.0
reverse training (entity") reverse (rand 252
reverse (enfity)
77.0
78.3
2.3
reverse (rand k=25)
81.0
97.0
97.3
73.0
Model size: 7B standard
standard reverse (enfity) reverse (rand k-25)
80.3
0.0
96.0
4.0
standard standard
79.0
89.7
99.7
6.0
78.3
99.0
99.0
70.0
Table Test accuracy (%) on the fictitious celebrities task, with either standard (data or compute-matched pre-training, or reverse pre-training, and either standard or reverse finetuning, for 1.4B and 7B parameter models. In all cases, reverse finetuning brings significant improvement on the reverse Nameeooesscriptinn task, which is otherwise impossible to solve, and to reverse DescriptionToNan using random segment reversal.
The results are shown Table 5. We sample multiple times from the models for each question and if any one of them contains the correct answer, then it is counted as success. The accuracy is relatively low in general due the small model size in terms of number of parameters, limited pre-training and lack of any finetuning for both baselines and our method. Nevertheless, in the forward direction questions, the reverse training outperforms the data-matched baseline, showing that in the data-bound case, reverse training even helps on standard tasks. The random reversal approach outperforms even the compute-matcheed case on the direct task for 5 and 10 samples, even though the baseline has effectively access to more data. Importantly, in both the data-matched and compute-matched case we see significant improvement in the reverse direction questions for reverse training compared to either baseline. This demonstrates that reverse training can be employed during the pre-training stage to make the model robust against the reversal curse.
3.4 Reversing fictitious facts via finetuning
We next explore if our reverse training can be applied to the finetuning stage when the model learning new, previously unseen knowledge from small training dataset. We use the same pre-trained models described in Section 3.3 and an additional Llama-2 7B model, and further finetune them on a dataset made up of fictitious facts. These data are made up of statements of the form "[name] [description]" (or the reverse) where the names and descriptions are randomly generated. The fictitious nature of this dataset guarantees that those facts are not seen during pre-training, and are only seen in the specified direction during finetuning. The model is then tested to see if it is able to learn these facts in the same or reverse direction that it has seen during training.
Table 6 provides evaluation results for different pre-training and finetuning setups. We employ a soft matching score as the test accuracy, which we evaluate exact presence of the target sequence in the first 64 tokens of model's prediction. Across all the pre-trained models, finetuning with reverse training was critical in solving the reversal of No-e scription, reaching close to 100% for the larger 7B model, while standard finetuning always results in 0% accuracy. For reversing Description only finetuning with the random segment reversal succeeded, achieving accuracy around 70%. This is likely because gener- ating descriptions is more challenging as they have many words and some variety even for the same person. We observe improvement from reverse pre-training in the data-matched case, but not in the compute-matched case. We note that this is perhaps to be expected as the evaluation statements are fictitious and never appeared in the pre-training data.
7

---

Published as a conference paper at COLM 2024
Pre-training method
BoolQ
PIQA
SIQA
WinoG
ARCe
ARCc
OBQA
Avg
size: 1.4B
std. (compute-matched)
65.1
74.4
41.2
47.7
62.7
67.6
32.1
27.0
27.1
49.4
std. (data-matched)
60.5
71.6
41.5
44.5
59.9
64.2
30.0
27.2
27.9
47.5
reverse (tokert)
63.7
72.9
41.6
45.1
60.0
65.7
30.5
28.0
25.8
48.1
reverse (entity")
62.7
72.3
40.9
45.5
59,4
65.1
29.4
25,4
27.7
47.6
reverse (rand k=25)
63.0
73.2
41.6
46.5
62.0
67.6
31.7
26.4
27.4
48.8
Model size: 7B standard
77.4
78.8
48.3
77.2
69.2
75.2
45.9
58.6
45.3
64.0
Table 7: Performance on standard benchmarks. Reverse training can outperform standard training the data-matched case, but is behind compute-matched training which uses more data. Llama-2 7B accuracy is provided for reference
anferenceand and is taken from Touvron et al. (2023b).
3.5 Analysis & ablation experiments
Does reversal training hurt performance on standard tasks? In Sections 3.1 to 3.4 showed that reverse training helps to mitigate the reversal curse. Here, we explore if our method disrupts zero-shot performance on common evaluation tasks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag, (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC easy and challenge (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018). We also report 5-shot performance on the aggregated MMLU benchmark (Hendrycks et al., 2020). Evaluation results are summarized in Table We observe that our random reversal model is 1.3 points better than the standard data- matched 1.4B model trained in the standard forward direction, and only 0.6 points behind the compute-matcher model in accuracy average, despite being trained on half of the tokens. We note that token reversal works slightly better than entity reversal on these standard benchmarks, and both are superior to the data-matched standard training well.
We also find that reversed models can not only generate text in the normal left-to-right roght direction, but can generate the beginning of the text given continuation that standard models lack (see an example in Appendix B Table 8). This reverse generation function can be useful in itself, for example for instruction backtranslation (Li al., 2023).
capability
Does the unit of reversal matter? To understand the effect of segment granularity when reversing sequences, we evaluate the performance of the following training methods on the fictitious celebrities task: standard finetuning, token and word reversal finetuning, entity- preserving reversal finetuning, and random segment reversal finetuning with varying k as described in Section 2. The results are summarized in Appendix B Table In general, we find that reversing at fine-grained level such as token or word level does not significantly help resolve the reversal curse, and only improves performance on the reverse tasks by 2-3%. Preserving entities during reversal makes it possible to predict names, but not descriptions. This indicates a close relation between the unit reversal training and the target "concepts" (e.g. names, descriptions) of the reversal task. Similarly, the random segment reversal performs poorly at predicting descriptions when the segment length limit is set lower than the typical length of a description. The results from Section 3.1 also support this hypothesis.
4 Related Work
Reversal Curse & Mitigations The reversal curse was identified by the concurrent works Berglund al. (2023b); Allen-Zhu & Li (2023b); its name was derived from the former. They demonstrated that the reversal curse occurs across model sizes and families, including very large models such as GPT-3.5 and GPT-4. They found that including auxiliary examples with both orders present in the finetuning or pre-training datasets (to promote meta-learning does not aid generalization to examples where only one order is given, even such data is rewritten in a question-answer format. Furthermore, including multiple paraphrases of
8

---

Published as a conference paper at COLM 2024
each fact in single direction does not facilitate learning in the reverse direction, despite aiding the given direction, as shown by Berglund et al. (2023a); Allen-Zhu & Li (2023a).
The concurrent work by Allen-Zhu & Li (2023a) investigates related set of failures and potential solutions. Exploring the capability to answer questions based on synthetic bi- ographies, they examine several data augmentation strategies, including incorporating instruction tuning data into pre-traing pre-training, generating multiple unique biography entries, permuting biography sentences, and substituting pronouns partial names with full names. They discover that augmentation during the pre-training phase is essential for enhancing downstream question answering performance across various tasks. However, in real pre- pre-training data, some augmentations may not be feasible - for instance, permuting sentences could degrade language model quality, and it remains uncertain how to best rewrite data during augmentation. Reverse training addresses this issue by presenting distinct language task (the reversed language) to the language model, thereby avoiding interference with the primary task of left-to-right natural language modeling.
Right-to-left, masked & other training variants Multiple works have proposed pre- training language models with rephrased or paraphrased text (Lewis et al., 2020; Maini et al., 2024), and training right-to-left has been explored before (Pfau et al., 2023; Nguyen et al., 2024), but these works were not targeting the reversal curse. Rather than training left- to-right, or right-to-left, masked language models aim to learn how to "fill in the middle", going back to early language modeling work such as Collobert et al. (2011), and models such as BERT (Devlin et al., 2018). Other methods have also been proposed to explicitly fill in middle text sections by rearranging data (Bavarian et al., 2022), to train on scrambled data (Sinha et al., 2021), train on all permutations of the factorization order (Yang et al., 2019). Relatedly, transforming training data with repeating segments has also been shown to improve language model embeddings (Springer al., 2024). Encoder-only models akin to BERT have been shown to not mitigate the reversal curse (Allen-Zhu & Li, 2023a). However, modifying the architecture and training procedure has been shown to help, e.g. by introducing Bidirectional Casual language modeling Optimization (BICO) (Lv et al., 2023). In contrast, our work seeks to rectify the issue while keeping standard language model training as similar as possible to the current regime.
The most similar work to ours is the concurrent work of Guo et al. (2024). They employ various augmentations at the finetuning, rather than pre-training stage, including shuffling and reversing chunks of the input sentences. Unlike our method, their method first segments sentences in the training into semantically meaningful chunks via an LLM. While a chunk can be an entity name, it is more generally applied to all words, e.g. "of developing the first emotional" as chunk. The actual segmentation is done via prompting another LLM with specific instruction. Therefore, the unit of reversal will depend on the LLM and its prompt, making it presumably difficult language modeling problem, whilst also requiring extra compute to reverse the sequence. This is applied only to finetuning on short sentences, which means the reversal curse mitigation is limited the facts included in the finetuning data, and it is unclear if it can be applied to large pre-training documents. In contrast, our method is applied in the pre-training stage so it can learn to reverse a wide-range of general knowledge facts.
Conclusion
In this paper, we introduced a simple yet effective training method to help remedy the reversal curse in LLMs. Our reverse training works by first segmenting the input sequence into chunks and then reversing the ordering of chunks, but leaves the word-ordering in each chunk intact. A chunk can be a token, a word, an entity name, or a random number of tokens. The model is then trained on both the original sequences, and this reversed data. We evaluated on a symbolic reverse task and reversing biography task that both demonstrated the necessity of preserving word-ordering within chunks. Next, we applied our reverse training to the realistic setting of LLM pre-training, which minimized the reversal curse on real-world knowledge. Evaluations on common benchmark tasks reveal that reverse training (particularly random segment reversal) during pre-training does not interfere with

---

Published as a conference paper at COLM 2024
the forward prediction ability of LLMs, and actually improves metrics in the data-bound (rather than compute-bound) setting compared to standard training. When our method is applied to finetuning on fictitious facts, prediction accuracy rose from 0% to 70-100%. The reversal curse is serious flaw in how LLMs acquire knowledge and reverse training opens new promising direction in resolving it. While our experiments focused English, we expect our method to generalize to other languages, especially the random-segment reversal version because it has less dependence on language.
References
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. ArXiv e-prints, 2lbb//230,,4316, September 2023a.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manipulation. ArXiv e-prints, abs/ http///30...44444 September 2023b.
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. Xrv preprint ar XrXo:2207.14255, 2022.
Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Korbak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational awareness llms. arXiv preprint arXiv:2309.00667, 2023a.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" a is b" fail to learn" is a". arv preprint ar Xiv:2309.12288, 2023b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arxiv preprint ar fttt::9905.1044. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. Xrx preprint ar Xiv: arXi::1803.0544 1803.05457, 2018.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICIE):293- 2537, 2011.
Gr'egoire Del'etang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Wenliang Kevin Li, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression. ArXiv, abs/2309910663. 2023, URL https: http:/ppp....ennnccccchhooltr wttp:/Crrpusi::22255255555
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.049// 2018
Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, and Yujiu Yang. Mitigating reversal curse via semantic-aware permutation training, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, ar preprint arXiv:2009.0330 033000, 2020.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arv preprint Stt::1901007291 2019.
10

---

Published as a conference paper at COLM 2024
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. Advances in Neural Information Processing Systems, 33:18470-18481, 2020.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation. ArXit, abs 22//222080002559 2023. URL https: / /api. semanticscholar org. arg/CorpusID: 260866107
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXit preprint cs/0205028, 2002.
Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. Are we falling in a middle-intelligence trap? an analysis and mitigation of the reversal curse. Xiv preprint orXiv:2311.07468, 2023.
Pratyush Maini, Skyler Seto, He Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: recipe for compute and data-efficient language modeling. arxiv preprint ar Xiv:22440111 16380, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arxiv preprint arXit:1809.02789, 2018.
Mark EJ Newman. Power laws, pareto distributions and zipf's law. Contemporary physics, 46 (5):323,3,,, 2005.
Anh Nguyen, Nikos Karampatziakis, and Weizhu Chen. Meet in the middle: A new pre-training paradigm. Advances in Neural Information Processing Systems, 36, 2024.
OpenAL. Gpt-4 technical report, 2023.
Jacob Pfau, Alex Infanger, Abhay Sheshadri, Ayush Panda, Julian Michael, and Curtis Huebner. Eliciting language model behaviors using reverse language models. In Socially Responsible Language Modelling Research, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. Xrv preprint AxX1v:1904.09728, 2019.
Stefan Schweter and Alan Akbik. Flert: Document-level features for named entity recogni- tion, 2020.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arviv preprint arXic:1508.07909, 2015.
Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3)3379-423 1948. doi: 10.00222..588-770011444..1333333
Zhihong Shao, Peiyi Wang, Qihao Zhu, R. X. Xu, Jun-Mei Song, Mingchuan Zhang, Y. Li, Yu Wu, and Daya Guo. Deepseekmath Pushing the limits mathematical reasoning in open language models. ArXiv, abs/ 2402./33000000 2024. URL https://// semanticscholar. vrg/CorpusID:27744 26111:66776667
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. arv preprint ar arXio:2104.06644, 2021.
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. ar ar preprint ar Xio:2402. 15449, 2024.
11

---

Published as a conference paper at COLM 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar arXto:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? Xiv preprint arXiv:1905.07830, 2019.
A Symbolic reverse task details
The vocabulary is built by generating 100 different words per position, e.g. a200 to a299 for the second word of entities a,, Then we create 10,000 entities a; and 10,000 entities by by concatenating random words specific each position. Finally, 4; entities are randomly mapped to bj, resulting in 10,000 pairs. The model is an 8-layer Transformer with hidden size of 512. The training is continued for 500 epochs with batch size 1024, learning rate 0.0003, and dropout rate = 0.1.
B Transformer model pre-training
We train transformer model with dim 2048, layers 24, and heads 16, resulting in 1.4B parameters. Training data and hyperparameter setup mostly repeats the one from Tou- et al. (2023b). To adapt for the relatively smaller model size, we increase the learning rate to 4.0e-4, and the global batch size was capped at 2M due to the limited number of GPUs. During training, we observe fixed gap training perplexity between baseline models and reverse training (Figure 1). The loss of the baseline model is measured on data in the standard direction, while the reverse training loss covers data in both directions. We posit that the reverse training doesn't 't interfere with forward learning thus, the model's performance does not degrade on standard benchmarks data-match conditions, and because we observe a match in the convergence rate of the reverse trained models with the baseline model when it's trained on about 50% of the data.
In Figure 2, we evaluate performance the real-world knowledge task for multiple check- points during upward trend in performance the reverse task with no saturation at the last checkpoint. Hence, we assume that if we continue pre- pre-training we would see further improvement.
where accuracy is reported using best@1 sampling. We notice an
We also find that reversed models can not only generate text continuations in the normal left-to-right direction, but can generate the beginning of the text given a continuation capability that standard models lack. We give example in Table 8.
12

---

Published as a conference paper at COLM 2024
training with token reversal training with entiiy-preserving reversal
training with random (k=25) reversal standard training
2.8
26
Training loss
200
1.8
Training tokens,
Figure 1: Training loss for 1.4B models in the pre-training stage. On the x-axis display the total number of tokens model has been trained on, including both in standard and reverse direction.
celebrity
train with presee-ge reversal
perent caleority; training with presty-preserving reversal
celebrity train with random (k=25) reversel
perent caleority: training with rancom (k=25) reversal
celebrity
parent; train with taken reversal
parent celearity; training with taken reversal
celebrity
parent, standdard teininn
parent uelearity, standerd uaiiiig
Exact match score
3
400
800
1000
Training steps, 10"3
Figure 2: Evaluation results during training the real-world celebrity task when using different pre-training methods for LLMs.
Finally, we evaluate the performance on the fictitious celebrities task in different ablation setups, varying pre-training and finetuning training approaches. Results are summarized in Table 9.
C Biography Data Experiment Details
dataset from Allen- Zhu & (2023a;b) as our bioS dataset, which generates biography entries per person using randomly chosen sentence templates and permutations. We use the bioR multi5 dataset from them as our bioR dataset, which generates biography entries per person by invoking Llama five times.
In our biography data experiments, we utilize the bioS
Following Allen-Zhu Li (2023b), we employ GPT2-small (12 layers, 12 heads, and 768 dimensions) for the bioS dataset and GPT2 with 12 layers, 20 heads, and 1280 dimensions
13

---

Published as a conference paper at COLM 2024
Query
"The Star-Spangled Banner" the
national anthem of the United States written by Francis Scott Key.
standard
national anthem of the United States.
N/A
reverse (token)
national anthem of the United States.
The Star Spangled Banner is the
reverse (entity")
national anthem of the United States.
The Star-Spangled Banner is the +
Query
Here is how to wash a car:
+- Those are the steps for washing
car.
standard
-> 1. Wash the car with a hose spray bottle, using
N/A
water and mild soap (such Dawn dishwashing liquid). Be sure to rinse thoroughly after each wash! 2. Use a soft brush on rough surfaces like windowsills; don't t use too much pressure because this can damage paint- work further down below where is less visible but still important for keep- ing clean looking cars in good condition over time!.
3. If needed apply... ...
reverse (token)
-> 1. Wash the car with hose and water, or use a pressure washer to remove dirt from the surface of your vehicle's paint- work. Be sure not to damage any parts the body of your car! Rinse off all excess moisture using a sponge or cloth soaked warm wa- ter; this will help prevent rusting over time.3. Wipe down any areas where you have applied wax before moving onto other surfaces such wheels tires if applicable (if necessary)...
Step #4 Clean the surrounding area thoroughly before washing the car. You're done! You are ready to wash the car.
that are painted
Note: Do not leave the car's engine running
soon you walk away from the car. Leave the car outside for some time to cool down (1-2 min- utes), then rinse it out with fresh wa- ter. Then pour the clean water back into the car and let it sit the car un- til it is completely dry (5-10 minutes, depending on your car). Finally, use garden hose to get any excess water off of the car.
Table 8: An example of reversed generations produced by 1.4B pre-trained models: the model is asked to generate text completion in both normal (left) and reversed (right) directions.
Pre-training method
Finetuning method
NameToDescription
DescriptionToNNi Name
forward
reverse
forward
reverse
Model size: 1.4B reverse (foken)
reverse (token)
78.3
0.0
100
2.7
reverse (entity")
standard
78.0
0.0
96.3
0.3
reverse (entity"
reverse (word)
71.0
2.0
reverse (entity")
reverse (entity)
77.0
78.3
95.3
2.3
std. compute-matchell
reverse (rand k=5)
77.0
52.3
96.0
10.7
std. (compute-matched)
reverse (rand k=10)
74.7
85.3
93.7
33.7
std. (compute-matched)
reverse (rand k=25)
77.3
96.3
97.7
70.7
std. (compute-matched)
reverse (rand k=50)
77.3
89.3
93.0
67.3
Table 9: Test accuracy (%) on the ficlitious celebrities task for various different pre-training and finetuning ablation methods.
for the bioR dataset. We also utilize the same AdamW optimizer with cosine learning rate decay (ß1 0.9, $2 0.98, 10 6).
14

---

Published as a conference paper at COLM 2024
bussine
5-6
bios
150
1.00
150
150
93.5
95.6
0.0
5.0
3.0
0.6
0.0
0.0
9.6
6.0
011
62
52
51
622
52
52
6.1
hins Itaken reversal]
100
100
100
100
99.6
99.1
0.0
0.0
0.0
9.6
0.0
2.0
0.6
0.0
37.5
63.
23.
6.6
68.1
82
62
biss (word leversali
100
1.00
100
100
99.7
99.2
0.0
0.0
0.0
0.0
3.0
0.6
6.0
95.3
77.8
95.0
19.3
11.
61
(endly roversal
0.2
99.2
bisk
98.9
99.8
99.9
99.7
99.8
92.1
01
07
00
0.0
0.0
0.2
6.2
0.2
0.1
0.2
0.1
0.2
6.1
bisk itsker reverzall
98.0
99.7
98.0
99.7
99.8
91.0
0.6
0.0
0.0
9.6
0.0
2-6
0.0
0.0
51.
66.9
28-7
M85
6.6
690
8.1
6.2
biok (word reversali
99.8
99.8
99.8
96.1
8.1
8.6
3.6
060
6.0
97.
99
98.4
biok fentily reversal!
23.2
02
0.2
(ran dom k=25)
89.7
$5.4
FT tddec baly
F7 unbrr FT mimmer chame caty
FT hor to, fud
theee to ful
brtal Jull braate, to
jast all, to Jaxt FT Ln, last
sox four, to last
direee, Jass
Mix donee, Int last
Mit biaaee to Mast
Ff all to
Nit all to
Mex four, to
Mixe theee, Io
busce, to
Table 10: Forward vs. reversal task accuracy for the data bioS, bioR (Allen- Zhu & Li, 2023b).
Left block = forward QA accuracy (ask for fields given person names).
Middle block reversal QA accuracy (ask for person's full name given selected bio fields):
"bdate" = given birthdates only, "all'= = given all fields, etc, see Appendix C). Right block reversal QA accuracy (ask for last name given selected biography fields). FT pre-train followed by instruction finetune; MIX add instruction FT data to pre-train preitrain.
For the bioS dataset, we prefrain / mix-train for 80,000 steps with a batch size of 192, which is twice their batch size.
For the bioR dataset, we pretrain / mix-train for 150,000 steps with batch size of 192, which twice their batch size.
During pre-training (or mixed-training) we weight decay of 0.03 and select the best among three learning rates: 0.0005, 0.001, 0.002; we also employ 1000 steps of learning rate warmup. During finetuning (FT), we use weight decay of 0.01, and select the best among two learning rates: 0.0003 or 0.0005; we do not use learning rate warmup. During mixed-training. we use QA, 0.3 which means 30% of the training tokens come from instruction finetune data.
Reversal QA tasks.
We consider four reversal tasks from Allen-Zhu & Li (2023b):
Give me the [last/full] name of the person born on October 2, 1996?
(bdate.to.lest, bdate.to.full)
[last/full] name the person studied Communications Massachusetts Institute Technology worked for Meta Platforms?
(three.te lostast, three. to full)
Give me [last/ful]] name of the person who studied Communications at Massachusets Institute Technology, was born in Princeton, Nj, and worked for Meta Platforms?
(four -o. last, four to fulll
Give me the (ast/full] name of the person who studied Communications at Massachusetts Institute of Technology, was born October 1996 in Princeton, NJ, and worked for Meta Platforms Menlo Park, CA?
(all.to_ess,, all.to.full)
Forward QA tasks. We consider the same six forward tasks from Allen-Zhu & Li (2023a):
What is the birth date of Anya Briar Furger?
What majur Anya Briar Forger study?
Answer: October 1996.
Answer: Communications.
What is the birth city of Anya Briar Forger?
Which company did Anya Briar Forger work for?
Answer: Princeton, NJ.
Answer: Meta Platforms.
Which university did Anya Briar Forger study?
Where did Anya Briar Forger work?
Answer: Massachusetts Institute of Technology:
Answer: Menio Park, CA.
Full results are summarised in Table 10. One may notice that our reported forward task accu- racies are slightly higher than those reported in Allen-Zhu & Li (2023a). This improvement is attributed to our use of a larger batch size, smaller weight decay, beam size 1 without sampling (instead of 4 in their paper), and the best result among three runs.
D Generation examples
In this section, we show some examples of generation produced by the corresponding models.
15

---

Published as a conference paper at COLM 2024
Real-world celebrity task Table 11 provides examples of models' generations in response to simple questions about celebrity-parent relationships. All models have 1.4B parameters and were only pretrained.
Pre-training method
Generation
direct question: "The father of Emmaa Roberts is"
std. (compute-matched)
Dylan O'Brien
reverse (token)
Paul Robert Thomas
reverse (entity")
Fric Roberts
direct question: "The child of Gustav Schwarzenegger is"
std. (compute-matched)
Christina Schwarzenegger
reverse (token)
Gustav Schwarzenegger
reverse (entily")
Arnold Schwarzenegger
Table 11: Generations produced by models response to a corresponding prompt from the real-world celebrity task. Correct output is colored in green, incorrect generation is red.
Fictitious facts task In Table 12 we show few examples of models' generations in response the prompts from a fictitious celebrity task. All models initialized from the fully pre-trained 1.4B model, and further finetuned as described in Section 3.4.
Finetuning method
Generation
Name2Description, reverse question:
"Immersed in the world of directing the virtual reality masterpiece, "A Journey Through Time""
std. (compute-matched)
Sariya Breckenridge has become household name.
reverse (token)
Nola Westbrook remains grounded.
reverse (entity")
Daphne Barrington has become a household name.
Description2Name, reverse question:
"In the realm of Zachary Norwood, figure who is celebrated for being the first person to complete a solo voyage around the world in submarine.
std. (compute-matched)
reverse (token)
saving countless lives during the Great Earthquake of 2065
reverse (entity")
saving countless lives during the Great Earthquake of 2065
Table 12: Generations produced by models in response to corresponding prompt from the fictitious facts task. Correct output is colored in green, incorrect generation is red. Questions are asked in reverse order with respect to how model has seen these facts during finetuning.
16


Reviews:
[{"review_id": "O0z7jFPwPQ", "reviewer": "Reviewer_SEXk", "summary": "This paper proposes to use reverse training to mitigate the reversal curse in language models.\nIn short, language models have been shown by prior work to present a \"reversal curse\": models are typically presented with strings of the form, e.g. \"Paris is the capital of France\", and they struggle to answer questions such as \"what is the capital of France?\"; this is because they are rarely presented with strings where \"Paris\" follows \"France\" (as in, e.g., \"in France's capital, Paris, ...\").\nThe authors then show that, when language models are trained on a mix of forward and reverse strings (using a standard language modeling next token prediction objective), LMs perform much better on questions which require such a \"reverse answer\".", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. \n\nWe think that the training \"direction\" (right-to-left vs. left-to-right) is an important aspect of the reversal curse, and it may underlie one of the core issues. Only predicting the next token is an artificial constraint and may not be the only way to build capable models. Our argument is that this one-directional objective is one of the reasons why language models suffer the reversal curse. While we agree that there can be other architectural reasons that contribute to the issue, it might be challenging to solve the reversal curse without modifying the next-token prediction objective, as shown in Allen-Zhu & Li (2023b). From that paper: we know that (1) bidirectional models such as BERT with MaskedLM don’t solve the problem, (2) adding abundant reversal finetune data doesn’t work; (3) pushing reversal finetune data to the pretrain stage doesn’t solve the problem. Recent theoretical work such as https://arxiv.org/abs/2405.04669 also suggests fundamental limits to the current auto-regressive models due to asymmetry. Fortunately, we feel that if there’s a simple way to fix the issue on the data level, this could be beneficial to practitioners.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper investigates an interesting topic: why do language models perform better on facts written with a certain directionality than another.\n\nThis paper also performs a large number of analyses related to the impact of reverse training on model performance. It studies the impact of reverse training in: a toy task, pretraining, and fine-tuning. Further, it considers the impact of reversing strings at the token, word, or entity level. Finally, it studies the impact of reverse training not only on tasks in which the model originally performs poorly, but also investigates whether reverse training hurts model performance on tasks the model originally did well in.", "reasons_to_reject": "While reverse training seems to experimentally help with the reversal curse issue, it seems to focus on solving a \"symptom\" caused by how language models are trained/defined rather than trying to fix the source of the issue. Identifying the source of why language models are inconsistent in representing facts based on the directionality in which they are presented, and proposing a principled way to fix it, would make this paper's contribution stronger in my opinion.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CWpjPkiyad", "reviewer": "Reviewer_8jTM", "summary": "Motivation of the paper is that language models do not learn the reverse of facts (e.g., \"X has feature Y\" versus \"Y is a feature of X\") due to their left-to-right training and the pre-training data including facts in only one of those two ways. This causes models to not be able to answer some questions based on how it's asked (e.g., \"What is a feature of X?\" versus \"What is Y a feature of\"), even though the fact is in its training dataset.\n\n- The symbolic reverse task allows a controlled study of the approach and provides clear validation of the authors' idea.\n- The \"celebrity task\" has very low accuracy numbers, seemingly too low to make any conclusions. The data appears to be simply insufficient for the model to perform this task. \n- Real-world larger-scale tasks show that using the reverse training approach during pre-training and/or fine-tuning can be helpful. When the same amount of training data is used (data-matched) the proposed approach is able to solve previously unsolvable tasks, with pretty good accuracy numbers. The best approach doesn't seem to require the reverse training during pre-training but just during fine-tuning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A direct comparison with Guo et al 2024 …**\n\nThis work is concurrent (same month as the submission date), thus direct comparison was not possible.\n\n**the best approach would be to do standard pre-training and reverse fine-tune …**\n\nImportantly, this is only true if we are testing on facts that are learned during finetuning, which is true for Table 6 – these fictitious celebrities are seen in finetuning, but not in pretraining. However, in practice, most of the knowledge LLMs have is obtained during the pre-training phase, thus reverse fine-tuning will not be sufficient in solving the reversal curse. A similar observation is made in Allen-Zhu & Li (2023b). Hence we recommend reverse pretraining. \nWe have now also performed additional experiments by training a 1.4B model from scratch using the “random reversal” method, and further finetuned on the fictitious and real celebrity tasks, which was missing before. Evaluations outperform all other models, and show the importance of learning the “reversing” skill during finetuning. We will update Tables 5-7 accordingly. E.g. Table 6 would have additionally a 1.4B Model:              \n\n| Pre-train Method   | Finetune Method | Name2Desc (fwd/rev) | Desc2Name  (fwd/rev )|\n|:-------------|:--------------:|--------------:|--------------:|\n| reverse (rand k=25)        | reverse (rand k=25)          | 81.0*   /    97.0*      |   97.3   /   73.0 |\n\nWe have more results but cannot put here due to limits to reply length. \n\n**Table 5 low accuracy numbers**\n\nThe numbers are low, but there are clear differences between the models. Our reasoning is: (1) the model is only 1.4B parameters, this improves with size; (2) there is no fine-tuning, only pre-training for the task. The task is realistic as we know SOTA LLMs can easily recall (forward, but not reverse) facts about real celebrities.\n\n**Do you think this would work for other languages**\n\nWe believe our method will generalize to other languages well, specially the random-segment reversal version because that will have less dependence on language. We will add this discussion to the paper.\n\n**reversal curse is a symptom of a deeper issue with LLMs' reasoning abilities**\n\nThe reversal curse may have multiple underlying causes. The one-directional training objective is causing this as we demonstrated in our paper. There could be an architectural issue on how facts are being stored in the model. But even such an (unknown) architectural fix might benefit from reverse training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors present a very neat idea that has merit and can be very impactful as it is easy to apply while training language models. Their presentation is very clear with extra details provided in the Appendix. Results are pretty impressive given the simplicity of the approach.", "reasons_to_reject": "A direct empirical comparison with Guo et al 2024 is needed to determine the value of this approach. While it is a more complex augmentation and therefore can only be applied at fine-tuning stage, I am not convinced by the mixed results that doing the augmentation during pre-training will provide the best results in a real-world scenario. Tables 6 and 7 imply that the best approach would be to do standard pre-training and reverse fine-tuning. In that case, I would've liked to see a comparison with Guo et al's work where both approaches are applied during fine-tuning only.", "questions_to_authors": "- Do you think this would work for other languages? Given multilingual nature of LLMs, some discussion of this would be good.\n- typo in Section 4 \"Mitgiations\"\n- I wonder if the \"reversal curse\" is just a symptom of a deeper issue with language models' logical reasoning abilities. While LLMs are known to reason over the facts they've been trained on, this could be just one of many reasoning tasks they are bad at. I would encourage the authors to provide their thoughts on this.", "extra_scores": {}}, {"review_id": "ZlQdJCZrwu", "reviewer": "Reviewer_nRfh", "summary": "The paper present the problem of reversal curse and a simple approach to help alleviate the issue. The reversal curse is the phenomenon that LLMs seem to lack the capability to properly understand relations such as \"Paris is the capital of France\", demonstrated by the models' inability to infer from the previous that \"The capital of France is Paris\". The proposal made in the paper is to pre-train or fine-tune LLMs with data containing reversed text (not just standard left-to-right). The paper investigates multiple approaches to revering text and evaluates model performance in both reversal comprehension and standard LLM evaluation benchmarks. The finding is that models trained with reversed text perform better in the reversal tasks and the performance on standard evaluation benchmarks does not deteriorate, indicating that the approach would be a suitable method for alleviating the reversal curse. The paper is well written, well structured and contains good selection of evaluation results. The only bigger issue with the paper is the lack of qualitative evaluation of the model outputs. It is well known that many of the evaluation benchmarks and tasks can be solved by models that do not really \"understand\" the language or the task or by models that otherwise generate poor quality language. Including qualitative evaluation and example model outputs would help the reader understand what effect the reverse training has on the model.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Intuitively, training on reversed text should harm the model in terms of how natural the output of the model is**\n\nThat’s a valid question but if the reversal data looks significantly different from the original data, the language model can learn an internal “switch” and when prompted with normal English, it is going to continue with normal English without reversal. This has been systematically studied for instance in https://arxiv.org/abs/2305.13673, where the authors show that if you train a transformer with both grammatically correct AND incorrect data, then if you prompt the model with the grammatically correct one, it will not continue the generation with grammar mistakes (grammar mistakes only happen if (1) you prompt with grammar mistake data and (2) you increase the generation temperature).\n\nIn a structural sense, the original and reversal data have significantly different syntax, even if they have the same words, so that the model can learn to distinguish them easily. In that sense they are closer to two different languages, and again language models have no problem learning multiple natural languages at once, and using the right one at the right time.\n\n**In addition to simple benchmark test results, I would have liked to see some more qualitative evaluation showing what the impact of reverse training has on the model output**\n\nThank you for your feedback. We already included Table 8 with examples of reverse generations. Below we show a sample of long-form generation from our 1.4B reverse pre-trained model. As you can see, the reverse-trained model is still able to generate coherent text in the forward direction, and it is not qualitatively different from the baseline model.\n\n| _Prompt_: “Here is how to wash a car:” | \n|:-------------|\n| _**Generation:**_ “1. Wash the car with a hose and water, or use a pressure washer to remove dirt from the surface of your vehicle’s paintwork. Be sure not to damage any parts that are painted on the body of your car! 2. Rinse off all excess moisture using a sponge or cloth soaked in warm water; this will help prevent rusting over time.3. Wipe down any areas...”\n\nWe will add full examples in the final version. In addition, we will also add generation examples from the celebrity task, which appear to work fine too (which correlates with the metrics in the paper).\n\nWe are unable to add more generations due to the space limit, but as we understand we could reply with more information in the discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a simple but effective data augmentation method that can help alleviate the reversal curse in LLMs. The paper is well written, the main arguments are well presented and the experimental results support the arguments.", "reasons_to_reject": "Intuitively, training on reversed text should harm the model in terms of how natural the output of the model is. In addition to simple benchmark test results, I would have liked to see some more qualitative evaluation showing what the impact of reverse training has on the model output.", "questions_to_authors": "I would like to see more qualitative evaluation of model output, comparing baseline and revere trained versions.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes reverse training for solving the reverse curse. The method is simple but effective. All reviews are positive.
Decision: Accept