The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More
Authors: Ouail Kitouni
Niklas Nolte
Adina Williams
Michael Rabbat
Diane Bouchacourt
Mark Ibrahim
Venue: NeurIPS 2024 poster (2024)
Keywords: Reversal curse
reliability
safety
language models
interpretability
learning objectives
Primary area: natural_language_processing
TL;DR: We find the standard next token learning objective underlies the reversal curse, which can be overcome with any-to-any context-prediction training.

Abstract: Today's best language models still struggle with "hallucinations", factually incorrect generations, which impede their ability to reliably retrieve information seen during training. The *reversal curse*, where models cannot recall information when probed in a different order than was encountered during training, exemplifies limitations in information retrieval. 
To better understand these limitations, we reframe the reversal curse as a *factorization curse* --- a failure of models to learn the same joint distribution under different factorizations.
We more closely simulate finetuning workflows which train pretrained models on specialized knowledge by introducing
*WikiReversal*, a realistic testbed based on Wikipedia knowledge graphs. Through a series of controlled experiments with increasing levels of realism, including non-reciprocal relations, we find that reliable information retrieval is an inherent failure of the next-token prediction objective used in popular large language models. Moreover, we demonstrate reliable information retrieval cannot be solved with scale, reversed tokens, or even naive bidirectional-attention training. Consequently, various approaches to finetuning on specialized data would necessarily provide mixed results on downstream tasks, unless the model has already seen the right sequence of tokens. 
Across five tasks of varying levels of complexity, our results uncover a promising path forward: factorization-agnostic objectives can significantly mitigate the reversal curse and hint at improved knowledge storage and planning capabilities.

The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More
Ouail Kitouni",, Niklas Nolte 2, Diane Bouchacourt 2, Adina Williams 2, Mike Rabbat 2, and Mark Ibrahim
IAIFI, Massachusetts Institute of Technology FAIR at Meta
Abstract
Today's best language models still struggle with hallucinations: factually incor- rect generations, which impede their ability to reliably retrieve information seen during training. The reversal curse, where models cannot recall information when probed in a different order than was encountered during training, exemplifies this in information retrieval. We reframe the reversal a failure of models to learn the same joint distribution under different factoriza- tions. Through series of controlled experiments with increasing levels of realism including WikiReversal, a setting we introduce to closely simulate a knowledge in- tensive finetuning task, we find that the factorization curse is an inherent failure of the next-token prediction objective used in popular large language models. More- over, we demonstrate reliable information retrieval cannot be solved with scale, reversed tokens, or even naive dedirectionallatteniioo training. Consequently, var- ious approaches to finetuning on specialized data would necessarily provide mixed results on downstream tasks, unless the model has already seen the right sequence of tokens. Across five tasks of varying levels of complexity, our results uncover promising path forward: aictorization-annssi objectives significantly mit- igate the reversal curse and hint at improved knowledge storage and planning ca- pabilities.
a factorization curse
Introduction
Although today's best language models produce impressively cogent, articulate text by learning the statistics of language, they still struggle to reliably retrieve information seen during training. Models are known to suffer from hallucinations, potentially responding with fabricated content that differs from the knowledge present in training data. Hallucinations pose significant hurdle to the adoption of language models, especially in domains where reliable knowledge retrieval is paramount (Dahl et al., 2024). A well-studied failure mode underlying hallucinations is the reversal curse, which ascribes this deficiency to the precise order of words presented to the model at train-time (Berglund et al., 2023; Allen-Zhu & Li, 2023). For example, a model trained on sentences where Paris always appears as the subject of the sentence, such as "Paris is the capital of France", can be tuned to answer "Paris is the capital of which country?" but not "What is the capital of France?", even though these two formulations encode the same underlying information. Existing approaches aimed at mitigating the reversal curse have focused on data augmentations that involve training both the forward and reversed tokens (Golovneva et al., 2024). this work, we focus on learning objectives.
In Section 2, we propose the factorization curse, framework that characterizes the reversal curse as failure to model the same joint distribution under different factorizations, We show the prevailing
Work done while interning FAIR, Meta.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training / Finetuning Corpus
Questions at inference
Which factorizations next-token prediction learn?
Model Prediction
Context
cupital which country?
Paris the espited of Francce.
France
Paris is the capital of
capitaal
Baguette
X
is the capital of France
Figure 1: (Left) Reversal curse from training model on sentences with Paris before France. (Right) Left-to-right objective does not learn how to predict early tokens from later ones even the information content is the same. The model overfits to specific factorization of the joint dis- tribution over tokens, and is unable to answer questions that require reasoning about different factorization.
left-to-right next token prediction, autoregressive (AR) objective used in popular large models such as GPT (Radford et al., 2019) and Llama models (Touvron et al., 2023a,b), underlies the reversal curse. We illustrate Figure how the factorization in AR training only encodes information based on prior context, thereby limiting how well the model can retrieve information based on later context. Through this lens, we show the reversal curse not merely a failure to learn logical implications, but more general problem related to learning objectives. Given this framework, we hypothesize in Section 2.1 that factorization agnostic models, i.e., models trained in a manner that is less dependent on the specific token order while preserving the overall meaning, can store knowledge better and are less prone to the reversal curse. To validate our hypothesis and explore potential solutions, we conduct extensive experiments in controlled settings in Section 3.1, focusing on the effects pretraining objectives on knowledge storage. Section 3.2 introduces WikiReversal, a realistic testbed based on Wikipedia knowledge graphs that closely replicates knowledge-inten -ntteniv finetuning application. In experiments with increasing levels of complexity and realism, we observe that scale, naive bidirectional objectives, and even left-to-right training do not resolve the reversal curse. These results suggest that finetuning strategies for downstream applications might not allow models to store knowledge adequately. Finally, in Section 4, we find that factorization-agnen -agnostic training not only promising initial solution for knowledge storage but also hints at improved planning capabilities in a minimal graph traversal task.
To summarize, our contributions are as follows:
1. We introduce the concept of the factorization curse, which posits that different objectives' de- composition an input into context and prediction a key factor underlying the reversal curse.
2. We conduct empirical studies with increasing levels of complexity and realism to validate our framework, comparing strategies such as standard autoregressive training (AR), AR with reversed sequences, and masked language modeling (MLM) prototypical bidirectional objective.
3. Building on factorization curse framework, identify factorization-agnosti objectives that allow for making predictions using every possible context decomposition, as a strong baseline solution. We explore its effectiveness across all investigated settings, including the WikiReversal setting.
4. We show that factorization-agnen agnnstiic strategies promising not only for knowledge stor- age/retrieval but also for planning, suggesting potentially broader implications for our findings.
2 The Factorization Curse
The reversal curse highlights how language models struggle to reliably retrieve information seen during training given some context. Our aim is to understand this failure by probing how common training objectives factorize an input into context and prediction. We show how common training objectives, including popular left-to-right AR and masked modeling objectives, struggle to learn factorizations that can help the model generalize better on a given task, a challenge we label the factorization curse.

---

2.1 Hypothesis: Reversal Curse as an Instance of Factorization Curse
Let us define the factorization curse more formally. We first start with the usual left-to-right au- toregressive model for a sequence with D tokens. This is the standard formulation in popular GPT-style (Radford et al., 2019; OpenAI, 2023) models and its loglikelihood is given by
log p(z)) = E llggp(s)/mee).
Each token is represented as It. where t is its index in the sequence. IxI represents all tokens that precede the t-th token in the sequence. The log probability the sum of the log probabilities of each token zt, given all the preceding tokens æer.. This is the left- 10-right factorization of the joint distribution over tokens. Note that there are many factorizations (D!) of the joint distribution, each given by some permutation of the tokens in the sequence, which we can write as log p(x)
the entire sequence æ is computed as
El= log (()o(t)) (o(e)))
Example in Two Tokens For illustration purposes, let us walk through an example with D = 2. Suppose our goal to model p(x) p(22, 24) p(zg|=1)p(2.)) The left-to-right factorization loss optimizes
-AA log p(x)) = p(rg (m) + logp(r1).
Interestingly, we can readily see the reversal curse failure mode in LAn. A model po that attributes high likelihood to po z1 (01) does not necessarily yield a high value for po (T1 (+2 (o (a right-to-left factorization) even though the two expressions should be equal due to the chain rule of probability. Note that here we make no statement about the sequential order of the random variables or their relationship. The only statement we make is that, unsurprisingly, Pe is not necessarily capable of modeling the joint distribution when presented with a different factorization. This is the factorization curse.
Definition 1 (Factorization Curse). A model po for the joint distribution of a sequence suffers from the factorization curse if. for a factorization order a different from the "training" " factorization order an (which depends on the objective and model details), we have
II po(#ocu)FF((0)) * II
(3)
In particular, the model may be optimal on oa, but perform poorty on a different factorization.
Implications This has number of implications. First, by definition, highly factorization- dependent LLM will struggle retrieve knowledge from earlier in the context given later infor- mation. Second, simply training on additional sequences with all tokens reversed would likely not alleviate the issue. Indeed, if the information we seek retrieve is composed of multiple tokens, the factorization the L.MM needs to handle is not right-to-left, but instead reversed in chunks. Thus, in order for any reverse training strategy to work, one must first parse the entities of interest then train chunks (see Section 3 and Figure 6).
with sequences reversed in entity-preservve
Furthermore this explains why standard MLM approaches with fixed masking rates fail address the issue, despite their bidirectionality, for two reasons: First, entities may span a larger number of tokens than the model masks, meaning there is never supervision signal to make the prediction from the right context (without leaking parts of the entity). Second, training with a fixed rate does not yield a good generative model. While the model is used to predicting, e.g., 15% of full context-window during training, at inference, the model can fail to generalize (Tay et al.. 2022) to the subitrary-lenghe sequences it encounters (see Figure 2). Zhang et al. (2024) suggest that encountering different length sequences during training encourages disentanglement and compasitionality, which will be crucial for good generative model.
Knowledge retrieval beyond reversal: A model that cannot learn how to retrieve information in reverse order will likely suffer from further downstream issues that are often ascribed to hallucina- tion. For instance, let us take model pretrained on entities in a database, say list of soccer players with various attributes (statisties, game histories, etc.) with the name appearing before the attributes as follows Fname: autteeeees The model may memorize the database perfectly, but when queried
3

---

for players that match specific criteria (e.g., played in a particular position, or have specific nation- ality, etc.), the model can produce hallucinated answers that do not match the training distribution due to lack of direct supervision of the form pl nmmme Pattributes) during pretraining.
2.2 Factorization-Agnons Training Strategies
To store and retrieve knowledge in "all directions" for arbitrary-length entities and without external intervention (entity pre-pe pre-pariing.. retrieval augmented generation, etc.), the model needs to be equally good at any factorization of the joint distribution. Below, we discuss two ways this can be achieved.
Permutation Language Modeling (PLM) A straightforward way to alleviate the factorization issue, is to write the autoregressive loss in way that is independent of factorization by averaging over all permutations as follows
log p(x) log (((()
II Ml 2 Bo-R(()))
Σ log poseooool (4)
where is a permutation sampled uniformly at random from Sp. the permutation group on D tokens. The term Lo(<)) represents all tokens that precede the t-th token in the permuted sequence. The log probability of the entire sequence then lower-bounded (using Jensen's inequality) by the expected sum of the log probabilities of each element Zo(t), given all its preceding tokens in the permuted sequence. Note that we did here average over all factorizations. This formulation is used in XLNet (Yang et al., 2020). However, for practical reasons they end up training with a permutation on the last few tokens only. This partial prediction, as we argued above, can limit knowledge storage improvements because we do not know how to chunk tokens into entities priori.
Uniform-Raid Masked Language Modeling (MLM--1) An alternative factorization Factorization-agnosiie ob- jective is to predict any context from any other context uniformly at random. This includes next- token, previous-token, predictions spanning multiple future or past tokens, and all other forms of contextual prediction. As it turns out, this generalization over objectives (amounting to something similar masked language modeling with a randomly sampled masking rate ((0, 1)) is a dis- crete diffusion model with absorbing masking state (Austin et al., 2023; Kitouni et al., 2024). This diffusion formulation can be used to make a model. See Figure 2 for an illustration of the differences between the MLM-U objective and more standard MLM. Specifically, Ler from Kitouni et al. (2024)'s Proposition 1, which we will refer to LMLMR here, be retrieved from Equation (4) as follows
autoregressive
LMLMU = - EENU(Ss) Σ log (() (F(()))
--BONR(SDD Brau( log (1))
D
Σ .D) -+1 TEa(2()
logp(zz ((((e))
(5)
where the last equality possible because all r E c f) tokens are equally likely to appear at position as we average across all permutations, and so we can average over the predictions for each r at this position. This approach can be implemented as a denoising process which recovers randomly masked tokens, like BERT (Devlin et al., 2019), but with uniformly sampled masking rates. This key difference allows training generative model with masked modeling."
3 Experiments
We now investigate information retrieval capabilities across learning objectives through the lens of different factorizations of the joint sequence probability. Specifically, compare
Appendix B shows simple example illustrating the connection permutation language modeling.

---

AR: The standard autoregressive causal next-token prediction. Though all models generate tokens autoregressively. we use AR shorthand for left-to-right models, in line with Equation (1).
AR w/reverse: AR prediction sequences and their token-level reverse."
MLM BERT-like masked language modeling with fixed random masking rate,
MLM-2f: MLM with U(0, 1). PLM results are similar, and are reported in the Appendix.
To ensure a fair comparison and allow each objective to perform optimally, we employ model archi- tectures specifically designed for each objective. For autoregressive (AR) training, we use GPT-2 (Radford et al., 2019) 9) and Mistral (Jiang et al., 2023). For masked language modeling (MLM), we BERT (Devlin et al., 2019). Finally, for MI.M-L, employ encoder-decoder model* based on the GPT architecture (see Appendix G for details).
We study these models across increasing levels of complexity and realism, beginning with trolled retrieval task using synthetic tokens to a retrieval task using natural text from Wikipedia ar- ticles. In our evaluation, find that the degree to which the learning objective factorizes the input reliably explains performance across this wide range of information retrieval tasks. Factorization- agnostic methods show improved knowledge retrieval capabilities.
3.1 Controlled Experiments in Factorization-Agnonn Training
Retrieval Task. We are particularly interested in models' ability to recall knowledge from data they were trained on. We will use a simple toy task, adapted from Golovneva et al. (2024), to evaluate this capability. First, we generate a collection key-value pairs which are each composed of sequence { JEES of tokens, e.g. consider the key-value pair
4041 : 12/3-
Each key/value is unique and is composed of unique set of tokens to control for any effects due to token statistics. Additionally, generate two types of queries: (forward) "[value of] key value" and (backward) "[key of] value key". Models are trained on key-value pairs and subset of queries, and tested unseen queries by completing tokens after the colon. Accuracy, measured using exact match, in Table shows AR training does not retrieve keys from values and that reversing tokens does not improve backward retrieval. We observe entity-based reversing trivially solves this task. Additionally, while MLM does not suffer from forward/backkwar disparity, its fixed masking rate causes poor overall results. Introducing uniformly sampled rate via MLM-U solves the task perfectly.
Non-reciprocal Relationships. Are models employing incorrect reversal heuristies? A weak- ness of the retrieval task is that could be solved by assuming all relations between keys and values to be symmetric/reciprocal. In language, this is not always the case: even though they contain many of the same words, the sentence Alice likes Bob does not necessarily imply that Bob likes Alice. To investigate whether models inappropriately rely reversal heuristic, extend the retrieval task
3 We obtained similar results when manipulating attention masks train on an equal mix causal and "anti-causal" sequences. We also experiments with this architecture for all the objectives and found consistent results.
MLM-15%
MLM-U
15%
Train
1((0.)
Train
Prediction
Test
Test
Context
Mismatch!
Figure 2: MLM struggles when entities span more tokens than the masked span. MLM-U encounters all possible masking fractions during training and does not suffer from this problem.
5

---

Table 1: Exact match accuracy different training paradigms on (Top) the retrieval task and (Bot- tom) relationship task. Due to the non-reciprocal nature of the relationship. model that swaps the subject and object will make errors (e.g., inferring B is A's child from A being B's child). Shown in the bottom row. Entity reversal without delimiter is marked with a*. Maximum values are bold.
Retrieval Task
AR
AR w/reverse
MLM
MLM-U
Forward
100
100
21
100
Backward
0
22
100
Relationship Task
AR w/reverse (entity)*
AR w/reverse (entity)
MLM
MLM-U
Forward
54
100
24
100
Backward
53
100
19
100
Incorrect Inference 1
44
0
third entity for each sample. yielding statements of the form - C". Question
answering (QA) samples are of the form (forward) "B - ?" and (backward) "B = ?" where the right answers are C' and A. respectively. With a third entity in play, model assuming symmetry would be unable to decide between A and C' as the answer for either question.
The bottom of Table 1 shows that simply reversing entities (denoted with AR w/reverse (entity)*) leads to undesirable behaviour as can he seen from the large incorrect inference rate. However, adding simple delimiter tokens around entity reversed sequences (without asterisk) leads more robust model. Finally, MLM-U learns the asymmetric relations correctly.
BioS. Next, we investigate performance of the different objectives for more complex but still con- trolled data. BioS (Zhu & Li, 2023) is a synthetic dataset consisting of biographies for 10k fictional individuals. For each individual, biographical details (birth date, birth city, etc.) were randomly selected from a uniform distribution. The authors ensured each individual was assigned a unique full name. We reproduce some of their results on the birth_date to--l-t-uuulllll name task which aims to recover person's 's full name from their birth date. Results are shown in Table 2. Again, the autoregressive, MLM and reversed-token training struggle to recover backward queries.
Training in a factorization-agnosiic fashion leads to non-negligible back- ward performance. Interestingly, back- ward performance keeps improving over long time (many times the number of epochs required for forward per- formance to reach 100%) (see Ap- pendix F). If this delay is due the low frequency of observing the right fac- torization training, this could indi- cate that methods automatically select data such as RHO-LOSS (Mindermann et al., 2022) could have a disproportionate impact in improv- ing factorization methods compared to standard AR training.
Table 2:
BioS exact match accuracy for property retrieval in the backward direction (birth date to full name) and in the forward direction (full name birthdate).
AR
AR wrreverse
MLM
MLM-U
Forward Backward 0
100
100
8
100
0
68
3.2 Wikipedia Knowledge Graph Reversal
To bridge the gap between the controlled studies on synthetic datasets and more realistic evalua- tions, introduce new evaluation setup that combines the best of both approaches. Our setup involves finetuning a language model on real-world natural text from Wikipedia articles, along with precise knowledge graph describing the relations and entities within them. This allows for princi- pled experiments that mimic real- real-world use-cases where practitioners finetune pretrained models on domain-specific corpora. We compare finetuning language model in this standard setup training from scratch with MLM-U4
Experiment Design
We introduce new closed-book QA dataset to evaluate the ability of models to reason about entities and relations in both forward and backward directions. The dataset is derived

---

Wikipedia Passage
Graph
Questions
Forward
Paris
France
Paris the capital which country?
caplial
France
Fily of France, With official popitation
Harkward
Capital
What the capital of France? Paris
of 102,000
Of
residents
Figure 3: An example passage with a forward relation triple. The forward question queries the tail, backward queries the head. WikiReversal is a collection of passages and forward/backward QAs.
from the WenWiki corpus based on DBpedia (Jin et al., 2020), which contains 1.3 million text passages from Wikipedia, along with entity and relation annotations.
Extracting Relation Triples and Generating Questions For each passage P with annotated en- tities E' e ..., Ens we consider only "forward" relation triples (e,,7, Ez), where e, appears before ej in the passage. For the example, in the passage "Paris is the [...] capital of France I...]" (Figure 3). the triplet (Paris, capitalOf, France) is "forward" triplet. Had the triplet (France, has- Capital, Paris) been present in the graph, we would consider it "backward" triplet. We filter the data to contain only triplets (and corresponding passages) for which the relation r exists at least in 500 different instances. We generate forward questions (r(e:) querying the tail of the relation (ez) and backward questions (ey) querying the head (e,) using predefined templates. We filter out am- biguous samples to ensure each question has a single unique answer. Algorithm 1 in the Appendix summarizes the dataset creation process.
Table reports the forward/backward per- formance disparity, particularly for autore- gressive models. Mistral 7B, achieves backward accuracy of around 5%, much lower than its forward accuracy. Interest- ingly, the model starts around the same backward accuracy in the beginnig of fine- tuning. This indicates there may still be backwards triplets present in "forward fashion" within the model's training text. This could also explain the non-trivial back- ward performance of the AR model, despite its susceptibility to the reversal curse. MLM-U attains the highest backward accuracy among the evaluated models, demonstrating its robustness to the versal curse. This suggests training from scratch with MLM--I can outperform a much larger (70 x) pretrained language model finetuned the same data. However, it still falls short of the AR model's forward performance, possibly due to the inherent difficulty of the task. Notably, significantly better results can be obtained by allowing models to leverage knowledge stored from the QAs themselves (see Appendix E.3 for details).
Table 3: Wikireversal task exact match QA accura- cies. MLM-U, MLM and AR are are 100M parameter models trained from scratch.
Mistral 7B
MLM
MLM-U
AR
Forward Backward
21
3.4
11
14
5.2
2.7
7.9
4.3
3.3 Analyzing Representations Learned via Factorization-Agnoste Training
We further examine factorization-agee aggnosicc training by first comparing the role of random masking in MLM-I versus standard masked language modeling. We also visualize the learned representations from MLM-24 showing they contain more distinct entity structure compared to standard AR training.
Understanding the role random masking understand the importance of varying the mask- ing ratio as introduced in MLM-L we compare MLM-U to MLM with various masking ratios (15%, 40%, 85%) based on prior work (Wettig et al., 2023)). We find MLM exhibits much noisier per- formance that's consistently lower than MLM-U with uniformly random masking ratio shown in Figure 4a. This suggests fixed masking ratios, whether with high or low values, are limited in what they can learn in contrast to MLM-U.
7

---

Visualizing learned representations in the 3-entity relationship task To better probe the repre- sentations learned via MLM-U we plot in Figures 4b and the PCA projections after training on the relationship task from Section 3.1 for AR and MIM--1. Compared to AR, which learns discon- nected components without apparent symmetry for entities never seen backwards during training, MLM-U seems to have learned form of translation symmetry across train and test samples. This suggests MLM-I training leads to more structured entities in the model's representation space.
100
MLM 15%
100L0
100.0
MLM 40%
MLM 85%
80
MLM-1
Backward Aec.
60
Explained variance. 10.0%
Explained vantaliee 15.4%
Blo8
Relation
Retneval
(a) Fixed-rate masked modeling inconsistent.
(b) AR model entities PCA pro- jection.
(c) MLM-U model entities PCA projection.
Figure 4: In panel (a) compare ML. with varying masking ratios MLM-U. In panels (b) and (c) we visualize the two main principal components representations learned via AR versus MLM-U.
We also measure the computational efficiency relative to convergence of MIM-U versus autoregres- sive (next token) training showing favorable efficiency for MLM-U in Appendix Figure 9.
4 On the Importance of Future Predictions for Planning
Prior work argues next-token prediction auto-regressive loss not conducive to planning (Dziri al., 2023; LeCun, 2023; Gloeckle et al., 2024). Specifically, Bachmann & Nagarajan (2024) introduces a simple path finding task that requires basic planning: From a start node, multiple linear paths Pl, P2,-- ...
...:... Pn extend outward. They given symbolic sequences of this form:
2, 6,, 7|5, 1/4, 3/4, 2/3,
7.7 4, A model is tasked to predict the sequence of nodes
start.cnd desired response
along path p; that leads specified final node the end of p.. They show that when trained with a standard autoregressive (AR) next-token prediction objective, the model is unable to effec- tively learn this task. This failure is attributed, used during training. As illustrated in Figure 5, from the second node 22 = 2 onward along path by simply condition-
least in part, the teacher-forcing supervision
P: (I1, .2...., ), the model can predict each "easy" token 2, for ing on the immediately previous teacher-foreed token I;-1, without requiring retention of the earlier path history or look-ahead planning, pitfall referred to as the "Clever Hans" cheat (see Section 4.5 (Bachmann & Nagarajan, 2024) and (Pfungst & Rosenthal, 1911))
(a) Accuracies of various training paradigms on the Star Graph Task, Randomly choosing a starting node in this setting (and employing the Clever Hans Cheat) results 50% accuracy.
start
node
goal
AR
AR w/reverse
MLM-U
Accuracy 50
49
100
Figure 5: Star Graph Task: Illustration and Performance Comparison. The illustration shows the "Clever Hans" failure mode with teacher-forced AR ((Bachmann & Nagarajan, 2024) adapted).

---

Bachmann & Nagarajan (2024) found that predicting multiple future tokens in teacher-less setting helped mitigate the issue of discovering the algorithm to correctly predict the initial "difficult" token J2. We identify this as an intermediate objective between standard next-token prediction and the factorization-agnosiii objective studied in this work, which encourages planning capabilities via both far look-ahead and look-backward along the sequence. Figure 5a shows that the MLM-U objective enables the model to reliably solve the path- aalh-plnnning task by better capturing the planning requirements.
5 Related Work
The reversal curse was first introduced in Berglund et al. (2023), Using text-overlap based heuristics for modeling inferences between sequence of text dates back nearly two decades in NIP (Glickman et al., 2005; Adams et al., 2007). As our modeling approaches have improved, increasing work has drawn attention to models overapplying text-overlap heuristics (Dasgupta al. 2018; Naik et al. 2018; Sanchez et al. 2018; McCoy et al. 2019; Rajace et al. 2022; Williams et al. 2022, i.a.). Per- haps most relevant is Sinha et al. (2019)'s evaluation, which used synthetic entity-based kinship data with multiple entities based on graph structures to expose model failures and is similar to our rela- tionship task. Most recently, work aimed at mitigating the reversal curse by Allen-Zhu & Li (2023); Golovneva et al. (2024) suggest using data augmentations by reversing both token sequences, or available, entity orders by training both on the forward and augmented text. Related projects have also trained and/or finetuned RoBERTa (Liu et al., 2019) or BERT (Devlin et al., 2019) 2019)-based mod- els on input sequences with randomly shuffled word order (Gauthier & Levy, 2019; Chiang & Lee, 2020; Sinha et al., 2021). Lv et al. (2023) explore a fine-tuning objective with bidirectional atten- tion and show that it can mitigate the reversal curse in the original synthetic setting from Berglund et al. (2023). However, they employ fixed masking rates. In addition to the standard objectives we explored, much recent work has gone into variety of pre-training objectives including span-based and hybrid objectives (Joshi et al., 2020; Tay et al., 2022; Chowdhery et al., 2022). XLNNe (Yang et al., 2020) utilizes permutation language modeling objective, considering permutations of the input sequence during training. However, XLNet is not completely predicts the last few tokens in each permutation.
cearrization-aanostt as it only
Various benchmarks have been introduced to evaluate the reasoning capabilities of language models. Bachmann & Nagarajan (2024) present study on the limitations of next-token prediction in captur- ing reasoning abilities, arguing that the standard autoregressive training objective hinders models' ability plan. In a similar vein, Dziri al. (2023) investigate the limits of transformer LLMs across three compositional tasks: multi-digi multiplication, logic grid puzzles, and a classic dynamic pro- gramming problem. Their findings suggest that transformer L.MSs solve compositional tasks by reducing step compositional reasoning into linearized subgraph matching, without necessar- ily developing systematic goblem-solvingg skills. They also provide theoretical arguments on ab- stract multi-step reasoning problems. highlighting how autoregressive generations' performance can rapidly decay with increased task complexity.
6 Discussion and Future Work
Limitations and Potential Extensions. MLM-U has much more challenging objective since we approximate all possible partitions of the input into context and predictions. Learning curves show delayed generalization, especially on backward samples. The main limitation of factorization- agnostic approaches is the optimization difficulty due to task complexity. Predicting one token ahead is far easier than predicting the last word of a novel with limited context, due to increasing entropy along longer horizons. This requires better schedules/curricula that smoothly interpolate the difficulty increase from next-token prediction to the highest-complexig factorization the model can handle.
This work highlights how alternative objectives can address some of the issues with current state- of-the-art language models, which rely on left-to-right autoregressive generative decoder pretrain- ing. Despite impressive capabilities with increasing scales, there are concerns about reaching plateau due to fundamental limitations, computational constraints, or data scarcity. We find that factorization-aanosiii training can learn "more" from the same data in the context of reversal curse.

---

This presents case for studying factorizatiin-agnostii objectives and investing in approaches to scale them.
Acknowledgements
We would like to thank Zeyuan Allen-Zhu, Sainbayar Sukhbaatar, Jason Weston, Olga Goloneva, and Léon Bottou for helpful discussions.
References
Rod Adams, Gabriel Nicolae, Cristina Nicolae, and Sanda Harabagiu. Textual entailment through extended lexical overlap and lexico-semantic matching. In Satoshi Sekine, Kentaro Inui, Ido Dagan, Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini (eds.), Proceedings of the ACL.- PASCAL. Workshop on Textual Entailment and Paraphrasing, pp. 119-124, Prague, June 2007. Association for Computational Linguistics. URL https::/acclatthollggg org/N07-1420.
Zeyuan Allen- and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation, 2023.
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Kor- bak, and Owain Evans. The reversal curse: Lims trained on is b" fail to learn "b is a", 2023.
David Cheng-Han Chiang and Hung-yi Lee. Pre-training a language model without human language. CoRR, abs/2012.11995, 2020. URL https://arxiv org/ abs/2 11995.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
Douglas
Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. Hallucinating Law: Legal Mistakes with Large Language Models are Pervasive, 2024. URL https: stanf ord. edu/news/ -aw----agaa-iisaaa---e-eeeeeeee-
models-rre-pervaiii
Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J Gershman, and Noah D Good- man. Evaluating compositionality in sentence embeddings. In Proceedings of the 40th Anmual Conference of the Cognitive Science Society, pp. 1596-1601, Madison, WI, 2018. URL
Sogsci18_proceedinggs .pd..
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4188 -4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 18653/v1/N19-14222 URL. https: // //acl walatholoooloo logy.oooogy.oo. org/
N19-1423.
10

---

Disk: diffusion model for Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck. Xi- ang Ren, Allyson Ettinger, 7aid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, 2023.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference on Natural Language Processing (EMNLP-IJCNLP), pp. 529-53295 229-539, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19-10000 URL https:// aallaooootttoyyy org/D19-1055
Oren Glickman, Ido Dagan, and Moshe Koppel. Web based probabilistic textual entailment. In Proceedings the
the 1st Pascal Challenge Workshop, pp. 33-36, 2005.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Pazz and Gabriel Synnaeve. Better & faster large language models via multi-token prediction, 2024.
Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Zhijing Jin, Qipeng Guo, Xipeng Qiu, and Zheng Zhang. GenWiki: A dataset of 1.3 million content-sharing text and graphs for unsupervised graph-to-text generation. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 2398-2409, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: 10. 10.1866////2022.colinnn m2innn.. URL https: bittp://accanm http:///aaaaahllogeeoe 20000 coling-main..... 217.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8:64-77, 01 2020. ISSN 2307-387X. doi:
1,.116//aaaaa0003000 URL /ttts:///oi... org/10/.//// wttgg/..112//ttei. 100.1162/tacl_aa_.. a_03000.
Ouail Kitouni, Niklas Nolte, James Hensman, and Bhaskar Mitra. structured knowledge, 2024.
Yann LeCun. Do large language models need sensory ground- ing for meaning and understanding?, 2023. University Lecture
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907.1199.112 2019.
Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. Are we falling in middle-intelligenee trap? an analysis and mitigation of the reversal curse, 2023.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Anna Korhonen, David Traum, and Lluis Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3428-3448, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 18653/v1/P19-1B 1334. URL https: /ttp///aaaaaalllaaaaaaaaaa otg/P19- 11/PP139-133111
Sören Mindermano, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training points that are learnable, worth learning, and not yet learnt, 2022.
11

---

Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. In Emily M. Bender, Leon Derezynski, and Pierre Isabelle (eds.), Proceedings of the 27th International Conference on Computational Linguistics, pp. 2340-2353, Santa Fe, New Mexico, USA, August 2018, Association for Compu- tational Linguistics. URL https: htapps///acllee h/tpaaacllat Ftrg/C18-1198
OpenAl. Gpt-4 technical report. PREPRINT, 2023.
Oskar Pfungst and Robert Rosenthal. Clever hans : the horse of mr. von osten, 1911. URL https: org/Corpus 142217369.
//api. semanticscholas
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL attps://api. stw.apiisssmmaiicccoclaaaar org/CorpusID: (tCCCooppIID:100005500000000 160025533.
Sara Rajaee, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. Looking the overlooked: An analysis on the word-overlap bias in natural language inference. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 10605-10616, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10.1 6653/w1/2022.emmlppmm https://acl httph/olcccanooogggg.oooooooooo/ www.oo///222 omnlp-main.77 725.
URL
Ivan Sanchez, Jeff Mitchell, and Sebastian Riedel. Behavior analysis of NLI models: Uncovering the influence of three factors on robustness. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedingy of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers). pp- 1975 1985,1985,00 New Orleans, Louisiana. June 2018. Association for Computational Linguisties. doi: 10.18653/v1/N13 1179. URL https: http:///acaaathoooooo org N18-1179
Koustuy Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Kentaro Inui, Jing Jiang, Vin- cent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4506-4515, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 1..11853/v1/D11-111 1458. org/D19- 1458.
URL https: http://allaatttth /acptt/ooo.....oooggg...
Koustuv Sinha, Robin Jia, Dicuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Pro- cessing, pp. 2888 2913, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi: 10. 10.18853/1//0211 http://2222.cmmmmmmm..22220.. URL https: org/2021 //aclanthology org/ 2021 emnlp-main. 230.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Gareía, Jason Wei, Xuezhi Wang. Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. U12: Unifying language learning paradigms. In International Conference on Learning Representations, 2022. URL httpstt://// https://api. semanticacholas org/ Frrg/Copussli ID: 2521720443.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie- Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
12

---

Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% masked language modeling?, 2023.
Adina Williams, Tristan Thrush, and Douwe Kiela. ANLIzing the adversarial natural language inference dataset. In Allyson Ettineer, Tim Hunter, and Brandon Prickett (eds.), Proceedings of the Society for Computation in Linguistics 2022, pp. 23-54, online, February 2022. Association for Computational Linguistics. URL https: httttp::/aacaatthlllee
org/2022..cc....oo scil-1.3.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov. and Quoc V. Le.
Xlnet: Generalized autoregressive pretraining for language understanding, 2020.
Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, and Léon Bottou. Memory mosaics, 2024.
Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2023.
13

---

A Why does AR w/reverse sequences fail?
AR w/ reverse
Prediction
Purts the capisal
Left-to-right
While the model can fill sentences backwards, answering backwards actually requires:
capital France
R
Right-to-left
The capital of Franxe
A
S
is the cagital of Frazace
which it fails to do because it fails on the similar tusk:
capital Frauce
Left-to-right conditioned some context on the right
S
X
Mismatch!
Figure 6: AR w/reverse cannot predict (left-to-right) entities that appeared on the left during training only learned to complete them from right left. The two sequences in the bottom right indicate that backward retrieval is roughly equivalent to refactorizing the conditionals such that the entity of interest predicted last conditioned on everything else. This only approximate because answering backward QA might require adding new tokens like "The answer to the question is but we make a weak assumption that such differences are generally irrelevant compared the entities and relations of interest.
B Permutation Language Modeling and Discrete State Diffusion
To illustrate the similarity between the diffusion loss and permutation language modeling, let's con- tinue walking through our D) example. Permutation modeling averages over factorizations p(x) = (=(x)(2))((x1) + (p(x1/22)p(2) and optimizes lower bound on the likelihood
logp(a:) 2 - Cr = (log [ogg((z)/m) + log + (logp(<r(<2) - logr(#2)).
(6)
Finally, the diffusion model averages over masking rates log (()))aa)) and optimizes
log p(x) 2 - -LLLM = (logp(xı) + log)()))) + (log ((gg((z1<2)) + log p(22|21)).
(7)
This the same as Equation (6). This implies that the permutation language modeling and the absorbing state diffusion objectives are in fact the same. Though practically speaking, they may have very different implications.
C Summary of Tables
Table shows qualitative comparison of the optimization objectives explored on the different datasets in this paper. We conclude that MLM with fixed masking rate mitigates the reversal curse due to its bi-directionality, but lacks generative quality and thus generally fails when having to pro- vide longer answers. Also unsurprisingly, the left to right AR objective works well in the forward retrieval direction but is unable to answer backwards questions and has hard time reasoning multi- ple tokens ahead to solve a task like graph traversal without intermediate supervision. Reversing the tokens can aid backwards retrieval for single token lookups, but fail otherwise. Reversing entities intuitively should be able to solve every retrieval task, but finding the right token permutation is
14

---

Table Summary of qualitative results, formatted as (forward)(bacckaard) Stargraph only has one
direction.
Task
MLM
MLM-U
AR
AR rev.
AR rev. ent.
Retrieval
VIV
VIV
VIX
VIV
VIV
Relationship
VIV
VI
VIX
VIV
VIV
BioS
XIX
VIV
VIX
VIX
(Golovveev et al., 2024)
Wiki
XIX
ZI
VIX
VIX
Stargraph
X
X
V (Bachmann Nagarajan, 2024)
Accurscy
Accurscy
0.5
06
03
0.2
Figure 7: Performance of MLM-U versus AR in the two-token setting. We train both MLM-U and AR in a two-token variant of the retrieval task from from Section 3.1. We find MLM-U reaches 100% forward and backward whereas AR struggles learn the backwards setting.
difficult task by itself. MLM--U averages over all possible prediction tasks that exist for a sequence given a tokenization and prevails in most our experiments, MLM-U displays the highest backwards retrieval capabilities in the most realistic Wikireversal benchmark, but the performance enough to qualitatively state success and we mark it with ~ in Table 4. We hypothesize that the rea- son is increased task complexity requiring larger models. Notably, Table 8 we show that MI.M-I outperforms all other objectives when it has access to either the forward or backward type question. From there, it can generalize well to the other type.
strong
We also experiment with a simple two-token setting comparing MLM-U and autoregressive next token prediction Figure?. 7.
D Additional Tables
Table 5: Retrieval Task forward and backward per token accuracy of different training paradigms.
AR
AR
MLM
MLM
MLM
MLM-U
PLM
w/reverse
15%
40%
85%
Forward 100
100
21
17
27
100
100
Backward 0
0
16
100
100
Table 6: BioS exact match accuracy for property retrieval in the backward direction (birth date to full name) and in the forward direction (full name to birthdate).
AR
AR w/reverse
MLM
MLM
MLM
PLM
MLM-U
15%
40%
85%
Forward
1.00
1.00
0.00
0.08
0.04
1.00
1.00
Backward 0.00
0.00
0.00
0.08
0.08
0.72
0.68
15

---

Table 7: Exact match QA accuracies for relationship tasks. Forward and backward accuracies are calculated normally, but due to the non-reciprocal relationship, model that swaps the subject and object will make errors (e.g., inferring B A's child from being B's child). Entity reversal without delimiter is marked with a*
AR (entity)
w/reverse
AR (entity)*
w/reverse
MLM
MLM
MLM
MLM-U
PLM
15%
40%
85%
Forward
100
54
24
77
2
100
100
Backward 100
53
19
35
100
100
Incorrect In-
44
0
1
0
0
0
ference
Table 8: Wikireversal task exact match QA accuracies. MLM-U, MLM and AR are all 100M parameter models trained from scratch. (Right) uses different seeds for train test splits in forward and backward questions while (Left) uses the same seed. For MLM, we tried 15%, 40% and 85% masking rates and present only the best models (15%). Details hyperparameter selection can be found in Appendix E.3
Mistral 7B
MLM
MLM-U
AR
Mistral 7B
MLM
MIM-UI
AR
Forward Backward
21
3.4
11
14
20
29
28
5.2
2.7
7.9
4.3
9.0
10
46
6.2
Algorithm
Dataset Creation
Input: GenWiki Corpus G {(P, F. T)}
Output: QA Dataset D {(1 P)]
D < 0
for (P, E, E E do
>> Each GenWiki sample
for (e;, r. e,) E do
P Each relation triple
if e; appears before ej in P then (c:)
> Forward relation
> Forward question
96 (c,)
b Backward question
- D J ((g ej P), eq, P)}
end if
end for end for
Filter 'D to keep unambiguous QA pairs
& See filtering in Appendix
Filter D to remove rare QA pairs where relation appears 500 times
E
WikiReversal
We outline the setup for the WikiReversal dataset. After installing the GenWiki package https gi github. com/zhijing- jiiignninnngnniiiiii. genwiki, users can download and process Gen Wiki to form WikiRe- versal using:
gen_wiki _ata GenWikiReader () sen_wiki_ataa print_sampppstt
all_data wtt_wwikidaatt. ()
sample = all all_data
def check_ [eeck_valid(questiinn) counts = 0 for char in question: if char == "{":
16

---

counts +
char "}":
counts -
counts
return False
return counts 0
def check _ walii_aal())))))___________
for key in questions_dicts if not sheck_.aliddquustioons. [key] ["forward"])>
print(key, "forward")
if not check_ print(key, "backward")
dict [key] ["backward"]]
def is forward graph_idx) int:
e1 sample 'graphh [graph_ides idx] [o]
e2 sample ['graph' [graph_idx] [2]
i1 = text.find find
12 text. fexd.findee2)
if i1 == -1 or i2 == -1:
return
return int (i1 < 12)
E.1 Filtering Ambiguous Samples
To mitigate ambiguity in the generated QA pairs, we filter the dataset to retain only (e,, eÿ) triples where the (e,,r) and (r, eÿ) pairs are unique across the entire dataset. This ensures that each question has a single unambiguous answer. Algorithm summarizes the dataset creation process.
E.2 Examples from the Wikireversal dataset
Table 10 shows the relations present in the Wikireversal dataset. Table 9 shows examples of passages and corresponding forward and backward questions that are trained on. WikiReversal is filtered from GenWiki (Jin et al., 2020), dataset based on Wikipedia released under a Creative Commons Attribution 4.0 International License.
E.3 Details on Wikireversal training
To measure performance on the Wikireversal dataset, we split the available data into training and validation, where we include passages and 80% of both forward and backward questions in the training set and 20% of questions in the validation set. We run hyperparameter grid search over every objective. We sweep over feasible learning rates for all models and weight decay for all except for MLM-U4, where we haven't found weight decay to be effective so it is set to O. We run sweeps for both different (Table 8 right) and same (Table 8 left) train test splits in forward and backward questions. GPT and BERT models have 12 layers with 12 heads and 768 embedding dimension for a total of 108 M parameters. The encoder-decoder model has 12 layers with heads and 576 embedding dimension for total of 109 M parameters. All models except Mistral were trained for 1500 epochs, and Mistral was trained for 200 to alleviate overfitting. The learning rates were warmed up for 1% of the training time in all cases. For Mistral, the LoRA and parameters are set to 256 to sum to about 109 M trainable parameters. Learning rates are 5c-5 for MLM-U in both train split modes, 3e-4 for both BERT (MLM-15%) and GPT (AR) for both modes and le-4 for Mistral (AR). The best weight decays are 1e-2 for both BERT and GPT, and no weight decay for LoRA on Mistral. No dropout
used. All models were trained with AdamW with default / parameters.
Table shows that MLM-U outperforms other objectives, achieving 66% and 46% accuracy on forward and backward questions, respectively. Using different seeds for train/test splits in forward and backward directions (right section) allows bidirectional models to learn answer questions from both the passage and the question itself, explaining MLM-L/'s significant improvement. AR
17

---

Table Examples from Wikireversal
Passage
Forward Q
Backward Q
Agostino Magliani 23 July 1824 - 20 February 1891 ), Italian financier, native of Laurino, near Salerno.
Where was Agostino
Who was born in Lau- rino?
Magliani born?
Zhou Yongkang has two sons, Zhou Bin and Zhou Han, with his first wife, Wang Shuhua, whom he met while working in the oilfields of Liaoning province.
Who
Zhou
Who married to Wang Shuhua?
Yongkang's spouse?
The total area of Mitan-myeon is 109.74 square kilometers, and, as of 2008, the population was 1,881 people.
What is the total area of Mitan- mitan-myeon?
Which populated place has total area of 109.742
Mohammad Ali Araki was born on 1894 in Arak, Iran. He started his education from Arak Hawza. Grand Ayatollah Haeri allowed him to wear the turban and robe because qualified individuals were limited. Also, Araki studied many years in Yazd Hawza.
What title does Mo- hammad Ali Araki
Who holds the title of Grand Ayatollah?
hold?
Tibor Navracsics (born Veszprém, Hun- gary, 13 June 1966) is Hungarian lawyer and politician, who served as Minister of Foreign Affairs and Trade from June to September 2014.
What region is Tibor Navracsics located in?
What who is located in the Veszprém region?
WWWX 96.9 FM, "96.9 The Fox") is an Alternative rock formatted radio station li- censed to Oshkosh, Wisconsin, that serves Appleton-Oshkoos area.
What is WWWX's alias?
Whose alias is 96.9 The Fox?
performs poorly backward questions due to the "reversal curse". MLM and Mistral 7B show intermediate performance. Although Mistral 7B uses 100M LoRA parameters, fewer than the other models, this setup mimics common fine-tuning recipes. Naturally, models trained from scratch do not learn general language modeling capabilities.
18

---

Table 10: Relations in Wikireversal
Attribute
Count
birthPlace
6100
birthName
5018
alias
3745
location
2532
deathPlace
2064
title
1923
city
1871
population Total
1651
owner
1328
name
1274
spouse
1163
isPartOff
1000
type
920
office
893
associatedBand
762
associatedMMussialArrtiss
756
synonym
743
knownFor
729
artist
724
PopulaaeddPlace/arraTToz
719
birthDate
672
ground
670
occupation
665
place
631
address
631
family
589
hometown
559
region
551
developer
541
label
538
writer
517
total count
42479
19

---

F Delayed Generalization in Language Modeling
We include aceuracy curves for training with MLM-U for both Bios and WikiReversal in Figure 8. We see the model is able to gradually learn both the forward and backward questions throughout training. For Bios, unlike the forward questions which saturate much more quickly, the backward accuracy still shows upward trend after training for 20k optimization steps. We observe a similar trend in the delayed generalization in WikiReversal for both forward and backwards questions even after training for 300k optimization steps. These results empirically demonstrate that the MLM-U objective, which requires modelling possible factorizations an input into context and predic- tions, is a more challenging task that exhibit delayed generalization relative to standard next-to- prediction training.
1.0
mmm
Forward Backward
0.10
0.8
0.06
My m M
0.6
Aocusucy
04
0.06
0.02
Dackward
10000 12300 Optinitaaion Stops
150000
Optimiration Staps
Figure 8: Accuracy in Forward/Backward Questions on the Bios dataset (left) and the Wikireversal dataset (right)
G Architecture Details
The Encoder-Decoder architecture used to train the MLM-J objective is modeled with ideas from XLNet Yang et al. (2020) in mind in order to support different attention/masking strategies includ- ing permutation language modeling. The encoder has GPT-like blocks and works with RoPE as positional bias. The decoder also has GPT-like blocks, but it cross-attends over keys and values from the corresponding encoder layer, also via a RoPE bias. The decoder input contains the same learnable embedding for all tokens, such that only the positional bias defines the initial attention pat- tern. This idea comes from XLNet's positional attention stream. In left to right AR training mode, both encoder and decoder use causal attention mask. In MLM-X modes, fraction of inputs masked before given to the model and neither decoder nor encoder attend over the masked tokens. All inference is performed in left-to-right AR fashion.
H
Compute Requirements
Models were trained on 64 NVidia V100 and A A00 GPUs with supporting Intel(R) Xeon(R) Gold 6230 CPUs. From conception to finalization of this paper we trained about 2000 models. The com- putationally most expensive runs were on the BioS and the Wikireversal dataset. Those comprised about 300 runs with 8 GPUs for around a day per model. About 30 Mistral models were trained on 32 GPUs for about a day per model. We also compare the convergence relative to GPU hours for MLM-I to AR in Figure
20

---

Convergence Accurccy
Convergence Backward Accuracy
0.6
My
1/5
0.
62
02
6.1
Figure 9: Comparing the of convergence for MLM-U versus AR retrieval. We found MLM-U took $58.80 minutes versus 559.45 for AR to train on 8 V100 GPUs. Although we observe faster forward saturation for AR, convergence is much noisier and the AR model is not able learn the backwards task, whereas MLM-U is.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed right before Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification: We describe the notation and assumptions required for our results regarding factorization.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
22

---

Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided or not)?
Answer: [Yes]
Justification: Refer main text and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is either be a way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
model (e.g.. large language model), then there should
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We provide details to recreate the Wikireversal datasct in Algorithm 1, but the code is not yet ready for open sourcing.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nigss
for more details.
23

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips cttccc/ppu.o www.c/pplcccaaeececceeceeecceele
wwtppppppppppspe///pppppppaaaaat gwt////dde///// Sube/ asionPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See main text and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
The margins between methods used to support claims in this paper are beyond typical error bounds due to stochasticity during training and supported by backing-theory regarding factorization agnostic approaches.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of emors is not verified.
24

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the computational resources used for model training in Ap- pendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tttpps://eeuree http:///eeuippppeeee http.///ppccccc..cccccc...eeeeee
Answer: [Yes]
Justification: We ensure all datasets are synthetically generated or openly-available as is the case for Wikipedia. We do not have any human participants this study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss benefits in terms of reliability for applications involving LLMs and limitations of the approach in terms of longer- anger-training required.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
25

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite synthetic datasets used from prior works, which do come with a specific license as they are procedurally generated. The Wikipedia graph used for the Wikipedia setting released under CC BY-SA 4.0. 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, age should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
license, copyright information, and terms of use the pack-
For existing datasets that re- ce-aakaaed, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
26

---

Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "cOecGaRBHo", "reviewer": "Reviewer_Z95k", "summary": "The paper extends the idea of the \"Reversal Curse\" from prior work and proposes ways to mitigate it by finetuning LLMs with a different objectives. To recall, the reversal curse is formulated roughly as follows: a model, when **finetuned,**(i.e. not prompted) in A is B statement, does not automatically generalize to B is A. Authors generalize this, from a probabilistical point of view, into the inability to generalize between different factorization orders on the joint text distribution. Authors then hypothesize that this can be alleviated by what they call \"permutation-agnostic training\" - training techniques popular among LLM that require predicting tokens in a varying order. Authors consider a number of such techniques based on prior work on encoder pretraining and find that MLM-u offers a consistent solution to the reversal/factorization curse, while many other intuitive solutions don't. Authors experiment using two relatively smaller language models (GPT-2 and Mistral 7B) on tasks like simple retrieval, understanding non-reciprocal relationships  (between sets/statements), WikiReverasal. The paper also analyzes the representations learned after the model is fine-tuned with the proposed approach.", "questions": "### Does this scale?\n\nIn your work, you test the proposed solution on GPT-2 and Mistral 7B. While the latter is undoubtedly an LLM, it is still curious if your approach generalizes to more capable LLMs. Note that I am not asking you to run all the experiments with a larger model, but even a selection of such experiments would improve the paper, particularly for practitioners.\n\nIf your main concern is GPU size, it should be possible to fine-tune relatively larger models using QLoRA ( https://github.com/artidoro/qlora ) or ReLoRA ( https://arxiv.org/abs/2307.05695 ). The former can fine-tune a 65B model on a 48GB GPU or multiple consumer gpus with sharding. The latter amounts of running the formal, then merging the adapters into LLM parameters and running again, for several such loops. Note that there may be a potential confounder of LoRA adapters vs regular fine-tuning. If you care to disentangle these effects, one possible way is to first check if LoRA finetuning can lift the curse for Mistral 7B, and if it does, try for larger ones and compare against that.\n\n\n\n\n> (L43) to learn learn logical implications,\n\nPossibly an extra “learn” (typo)\n\n\n> (L174-175)\n\nthere is a paragraph missing line numbers, likely due to excessive negative vertical space. In that paragraph, and in the unnamed equation below, you refer to tokens as $t_1, t_2$, etc. In turn, your text up to this point refers to tokens as $x_0, x_1$, etc Moreover, $t$ is explicitly reserved for token index (L80). Unless there is a specific reason for this, the presentation would be improved if you untangle the notation around $x$ and $t$.", "limitations": "To the best of my knowledge, authors have sufficiently addressed limitations of their work", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes (seemingly) a solution that covers one of the reasons why LLMs hallucinate. Since LLM hallucination is one of the main roadblocks to their greater adoption, this is an impactful problem to address.\n\n2. The experiments appear sound: authors evaluate on a diverse set of tasks, using two different LLMs. This eliminates the possibility of a false positive. However, there is still a direction of scaling to larger models (e.g. 70B and above) and checking against possible changes in the efficacy of the proposed solution.\n\n3. The proposed method relies on fine-tuning, not training a model from scratch, and thus can be broadly applied to existing models\n\n4. The paper is generally well written, has a clearly stated hypothesis and is overall easy to follow, if a bit unconventionally structured. Minor typos and clarification requests (below) do not affect the overall presentation.\n\nAlso, a minor but pleasant advantage of this paper is that they openly declare that their solution is a clever reuse of an existing method. Many works I read in the past instead  choose to slightly modify the approach and declair it a newly proposed method. I count your choice as a minor advantage because it reduces the research debt, i.e. how many methods does a newcomer need to learn to meaningfully contribute to this area.", "weaknesses": "### Side-effects can be explored better\n\nMy main concern with the paper is that , while there is great attention to how fine-tuning combats reversal curse, but the side-effects of such fine-tuning are arguably not explored enough.\n\n\n**In other words, does fine-tuning Mistral affect its accuracy on other, unrelated tasks? If yes, what trade-off points are there?**\n\nTo test that, once can use LM Eval Harness for a selection of such tasks ( https://github.com/EleutherAI/lm-evaluation-harness ). If not familiar with the tasks, please use the ones commonly reported upon popular LLM releases (e.g. see Table 20 in the Llama 2 paper https://arxiv.org/pdf/2307.09288 ) or choose your own tasks. Another direction would be to evaluate the overall quality of your model's generated text, whether with human evaluation (local/MTurk/Toloka/...) or, as an Ersatz, with another LLM (as in https://arxiv.org/pdf/2304.03277 )\n\nThis can make a difference between a \"free\" solution to the curse and the one that comes at too great a cost got most practitioners -- or something between the two. As such, the paper would greatly benefit from understanding these trade-offs, or knowing that they don't exist.\n\n\n### XLNet the Baseline\n\nYou (rightfully) spend a lot of Section 2.2 on describing XLNET (alongside MLM-U), but then never compare against that as a baseline.\n\nWhile you offer some algorithmic reasons MLM-U could be better, but to dismiss a relevant algorithm as a baseline, one usually needs more evidence, e.g. showing that it is infeasible to use or proving that it is guaranteed to be worse. I could not find such an argument in the paper. If you have it, please direct me to it. If not, the paper could be improved by actually comparing to XLNET, at least as a side-experiment on a subset of tasks, so that the reader better understands your choice of MLM-U.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad you regard reliable knowledge retrieval as an impactful problem and view MLM-U as a consistent solution to the factorization curse. We are happy that you found the paper to be well-written and suggested several useful pieces of feedback. We address each below:\n\n### Finetuning\nFirst, we’d like to clarify MLM-U models in our experiments are trained from scratch—not finetuned. We found training from scratch directly on the downstream dataset (Tokens, BIOS, Wikigraph in Section 3.1 and 3.2) was sufficient for competitive performance or to solve the benchmark entirely without the need for pretraining. Given we’re training from scratch, we do not measure performance on side-effects related to standard benchmarks (though we do carefully check forward standard accuracy as well). However, we absolutely agree exploring MLM-U for finetuning would be a valuable and interesting direction we hope to tackle in an upcoming publication. For finetuning, side-effects in terms of standard benchmarks would then become paramount to understand the tradeoff between finetuning reliability.\n\n### XLNet Baseline\nWe absolutely agree XLNet is an important baseline to consider among factorization agnostic approaches. We present XLNet in Section 2.1 under the heading Permutation Language Modelling (PLM) where we describe the method as a baseline. In addition, we also draw a connection between permutation language modeling (such as XLNet) and discrete state diffusion in Section 2.2. We agree the heading and presentation can be made more clear to indicate we are in fact referring to XLNet in this section. The primary reason we chose to focus on MLM-U in the main tables is that the implementation of XLNet is not fully factorization agnostic: as described in Section 2.3 of XLNet  https://arxiv.org/abs/1906.08237 for \"practical reasons they end up training with a permutation on the last few tokens only.\" Nevertheless we do provide additional comparisons to XLNet in Appendix Tables 5, 6, and 7 studying BIOS, QA relations, and synthetic tokens.  Based on your suggestions, we’ll rework the writing to ensure XLNet results are presented more clearly.\n\n### Scaling MLM-U\nFor fair comparisons to prior work in order to ensure our gains were the result of the proposed learning objective (and not mere differences in model scale), we chose to perform experiments using model scales from prior work ( retrieval task from Goloveneva et al. 2024 and BioS from Allen-Zhu et al. 2023).  We compare the effect of adjusting the learning objective only, keeping the model architecture fixed, to prior explorations that modify the training data (see Table 2 AR w/reverse for example).\nTo push the experimental setting further towards larger realistic graphs with known entities we also develop a benchmark based on natural text from Wikipedia and naturally occurring entity relations. In this setting, we found even relatively small models trained with MLM-U performed remarkably well at resolving the reversal curse gap between forward and backward accuracy. Specifically, we found training from scratch only on the WikiGraph data with the MLM-U objective outperformed Mistral-7B, a model that’s 70x larger, after finetuning on the same data.\n\n\nFinally, we very much appreciate your attention to the typos, spacing, and notation suggestions. We’ve addressed each to sharpen the presentation of the work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9f3mtEZ9xw", "reviewer": "Reviewer_bZEp", "summary": "This paper addresses the reversal curse, where models trained on a relation in one direction (e.g. \"A is the capital of B\") cannot answer questions about the relation worded in the reverse order (e.g. \"B's capital is [?]\"). The authors frame this as a subcase of a broader problem, where models trained with a causal learning objective do not learn to assign equal probability to different factorizations of the same sequence, which the paper terms the factorization curse. To address the factorization curse, the authors propose using MLM-U, a modified MLM that varies both the location and size of the masked segments. The authors show that models trained with this modified objective do not exhibit the same reversal curse and analyze the behavior of models trained with causal, MLM, and MLM-U objectives.", "questions": "Q1. In appendix B, you state that the PLM and absorbing state diffusion objectives are “the same” but “practically speaking […] may have very different implications”. Can you elaborate on this point? \n\nQ2. Why use BERT rather than BART (or another encoder-decoder) as the choice for MLM?\n\nQ3. You argue that at least some of the benefits of MLM-U over MLM arise from the ability to fully mask out multi-token entities. Could you demonstrate that, given a task where the entities are all single-token, the benefit of MLM-U is marginal or non-existent? I’m thinking of a version of the retrieval task in 3.1 where the key-value sequences are all of length 1.\n\ntypos/small notes:\n* line 79: GPT-style citation should probably be GPT-2 or GPT, not GPT-4.\n* line 83: “Note that there *are* many factorizations”\n* Table 1: very small point, but I would have expected the entry with the asterisk to be the entry *with* delimiter tokens.\n* I think the factorization curse section could emphasize even more that the ordering of the tokens in the input is not altered when calculating the different factorizations. This is stated in the text (lines 83-85 and 92), but given that the section begins by discussing causal models (and, like a lot of us, I work almost exclusively with causal models these days!), it still required a second read to parse exactly what was going on.", "limitations": "Limitations seem appropriate.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1. The paper tackles a well-defined problem from an interesting angle, providing both theoretical and empirical evidence for its claims. The paper is well-written.\n\nS2. The connection between PLM and discrete state diffusion here is an interesting insight and well-explained. \n\nS3. WikiReversal is an interesting idea for a dataset and looks like it will be a great resource for the community, especially for work aiming to study this reversal phenomenon. It is documented well.", "weaknesses": "W1.  In a two-token setting, you can (at a high but not intractable computational cost) compute the right-to-left factorization of a causal model through enumeration. It would be interesting to compute the left-to-right and right-to-left factorizations for some two-token sequences at test time for the models presented here. Do the causal models clearly suffer from a discrepancy of joint probabilities in the two-token case? Does the MLM-U training mitigate the two-token factorization curse? It seems likely the answer is yes, but it would strengthen the paper to confirm this empirically. (And in a 2-token setting, it seems that MLM would also mitigate the factorization curse-- does this happen in practice?)\n\nW2. Comparing BERT-like MLM to encoder-decoder MLM-U or GPT-like AR seems to be a bit of a strange comparison. It would seem more natural to use a BART-like encoder-decoder for the MLM example (though I recognize that the causal attention in the decoder may be slightly confounding, so I think both settings have merit; still, it would be good to *see* both to be sure). \n\nW3. The architecture details for MLM-U, while not the key argument of the paper, are only sparsely explored. It is not clear how much of the benefit comes from using the new architecture design versus using the MLM-U objective. And since Appendix G suggests that this architecture can handle both MLM-X and AR training, why not just train all models using this architecture and vary only masking strategy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully considering our work. We are glad you found the paper well written and that we tackle a well-defined problem with both theoretical and empirical insights. We are especially happy you appreciated the effort we put into crafting a more realistic benchmark for the reversal curse with WikiReversal. We appreciate the suggestions you provide to improve the work and have performed additional experiments based on these suggestions, which we address below.\n\n### Two-Token (W1)\n\nThank you for the suggestion to experiment with a two-token setting to assess whether MLM-U training mitigates the two-token factorization curse. We ran a new experiment comparing standard left-to-right AR training (labeled GPT in the legends) and MLM-U. Specifically, we train both MLM-U and AR in a two-token variant of the retrieval task from Section 3.1 trained for 2k epochs. We find MLM-U reaches 100% forward and backward whereas AR struggles to learn the backwards setting, reaching only 12% accuracy after 2k epochs of training. We’ve attached plots demonstrating the forward and backward accuracy of each model throughout training in Figure 1 of the supplementary rebuttal PDF. Please let us know if this answers the question you had in mind.\n\n### Encoder-decoder architecture comparisons (W1, W2, Q2)\n\nWhile we agree that training all models with encoder-decoder and then varying only the masking strategy is one valid strategy, we wanted to give each masking strategy its “best shot”. We argue that it is most fair to have each masking strategy compete with a corresponding architecture that is known to work well with it. Otherwise one could, for instance, claim that the encoder-decoder is not well suited (or not properly optimized) for AR.\n(FYI, earlier on in our research investigations we experimented with both encoder-decoder and BERT-style encoder only architectures for MLM. There we found BERT-style encoder architectures to perform better with MLM-X objectives.)\n\n\n### MLM versus MLM-U (Q3)\n\nWe would like to clarify that both MLM and MLM-U are capable of masking and predicting multi-token entities (unless of course the entity is longer than the masking ratio). The main advantage of MLM-U as shown in Figure 2 stems from its ability to handle context-dependence of variable lengths.\n\n### Practical differences between PLM and MLM-U (Q1)\nWhile the objectives are theoretically equivalent, the implementation of XLNet is not fully factorization agnostic in practice. As described in Section 2.3 of XLNet  https://arxiv.org/abs/1906.08237 for \"practical reasons they end up training with a permutation on the last few tokens only.\" This results in a model that is not fully factorization-agnostic.  \nAdditionally, in practice, we do not average over all permutations or all masking rates, but only a randomly chosen subset. This might induce more practical differences.\n\nThank you for the proposed clarifications and language suggestions. We’re very glad you point out “the input is not altered when calculating the different factorizations.” We agree we’ll emphasize this key point much more prominently in the writing. We remain available for further questions and thank you again for all the effort you put into this review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8zEtesunBa", "reviewer": "Reviewer_kVoQ", "summary": "This paper focus on the \"reversal curse\", where models struggle to recall information when probed in a different order than encountered during training. The authors propose reframing this issue as the \"factorization curse\", which is a failure of models to learn the same joint distribution under different factorizations. They support it with experimental evidence, providing also a new dataset WikiReversal, to evaluate this problem and explore solutions. Authors suggest to use factorization-agnostic training objectives, which can significantly mitigate these issues.", "questions": "1. Do the quality of generation drops if we use proposed method to train the model? \n2. Will you give the open access to the provided WikiReversal? \n3. What are the computational costs associated with factorization-agnostic training, and how do they compare to traditional training objectives?\n\nAlso some typos:\n\nLine 83 \"are: missed in “Note that there many”", "limitations": "1. The experiments are primarily based on the WikiReversal testbed, which, while realistic, is still limited to the structure and content of Wikipedia knowledge graphs. \n2. Limited scales of models, the training was done only on small models, which could be justified for retrieval itself, but need to be addressed for scalability of approach", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Authors introduce novel concept of  \"factorization curse\", broadening the understanding of why models fail in information retrieval.\n2. The development of WikiReversal, based on Wikipedia knowledge graphs, is a valuable contribution, offering a more realistic benchmark for evaluating model performance in information retrieval tasks.\n3. Authors provide extensive experiments, proving the fact that \"factorization curse\" exists both in toy examples and real cases.\n4. They suggest a factorization-agnostic training objective that showed good performance in mitigating such issues, while preserving satisfactory quality in forward queries.", "weaknesses": "1. While the paper presents a variety of experiments with different training objectives, they experiment with small model sizes (100M parameters). It is not clear how well this approach could be scaled on bigger models. \n2. Authors didn't clearly analyze whether proposed training objective will hurt the quality on the standard benchmarks. For example, it would be interesting to see if the quality of the open generation drops with new training objective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable feedback to improve our work. We appreciate you finding the factorization curse a novel concept to explain model information retrieval failures. We are happy you appreciate the development of WikiReversal and our extensive experiments. Based on your suggestions we ran an additional experiment to measure the computation costs and convergence tradeoffs of MLM-U. We outline these new experiments and answer your remaining questions below. \n\n### Scale and performance on standard benchmarks\nGiven that our objective is not to train a general purpose language model, we do not benchmark standard language benchmarks as here we isolate only information retrieval. This helps us control the experiments. Albeit limited by relatively small context windows in our knowledge retrieval tasks, we have confirmed that generated text is grammatically coherent and looks like proper English.\nOverall in our experiments, we focus on whether alternative learning objectives, namely factorization-order agnostic objectives can overcome information retrieval failures such as the reversal curse. \nFor all information retrieval benchmarks considered, we found a modestly sized model was able to perform quite well. Even for the more challenging WikiReversal a much smaller model outperformed Mistral-7B finetuned on the same data—demonstrating the advantage of factorization-agnostic learning objectives such as MLM-U. Of course, we agree scaling such models further would be interesting and we are exploring this direction for future work.\n\n### Access to WikiReversal\nWe agree it is important for the research community to build on the WikiReversal benchmark. Fortunately, the underlying data is already openly accessible via GenWiki https://github.com/zhijing-jin/genwiki with an openly accessible (Creative Commons) license. While WikiReversal should be reproducible from the details outlined in Section E (specifically Alg. 1), we do plan to include the exact scripts we use for researchers to download and parse the dataset to form WikiReversal in order for the community to study the reversal curse on more realistic natural text.\n\n### Computational Costs and Convergence\nThank you for raising this point. We agree comparing the computational costs for each training approach is important. Based on this suggestion, we analyzed the runtime and convergence rates for both MLM-U and standard autoregressive (AR) training.  We benchmarked two parameter matched models on the retrieval task dataset from Section 3.1. We found MLM-U and AR exhibited comparable computational costs per training step:  558.80 minutes versus 559.45 (AR) on 8 V100 GPUs after 1k epochs of training.\n\nTo better understand the convergence tradeoffs we also include a comparison of the forward and backward accuracy curves for MLM-U versus AR in Figure 2 of the supplementary rebuttal PDF.  As expected we found the forward loss for the AR converged more quickly with forward saturating after 1600 epochs versus MLM-U, which saturated after 3999 epochs of training, although we observed a much smoother convergence for MLM-U even for forward. Of course, the AR model despite the faster forward convergence was not able to improve the backward accuracy. Based on your suggestion we plan to include a more thorough discussion of the computational cost and convergence in the revised draft. We’ve included both forward and backward converge plots in Figure 3 the rebuttal PDF for your reference as well. Note also appendix F, which talks about the delayed generalization speed for the backward direction.\n\nWe’d like to thank you for the thoughtful questions and suggestions. We hope the additional experiments and clarifications have addressed your questions. We remain available for any further discussion or questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We’d like to thank reviewers for their high-quality feedback and thoughtful suggestions for our work. We very much appreciate reviewers noted the importance of reliable knowledge retrieval in language models noting “LLM hallucination is one of the main roadblocks to their greater adoption, this is an impactful problem to address”—Z95k. We’re glad reviewers appreciated our insight into the role of factorization in the reversal curse, noting we “introduce novel concept of ‘factorization curse’, broadening the understanding of why models fail in information retrieval”—kVoQ. We’re also glad reviewers found our experimental setup “sound” (Z95k) “realistic” and “extensive” (kVoQ) with several reviewers noting the value of WikiReversal to the research community. Finally, we’re glad several reviewers found our paper to be “well-written” (Z95k, bZEp) and our solution, MLM-U, to offer “a consistent solution to the reversal/factorization curse, while many other intuitive solutions don't”—Z95k.

Reviewers provided useful feedback on the scalability of the method, standard benchmark performance, computational tradeoffs, and suggestions to better isolate the effect of architecture as well as a new interesting two-token setting. We’ve made a considerable effort to incorporate this feedback with clarifications and three new experiments based on reviewers’ suggestions for which we’ve attached results in the supplementary rebuttal PDF. In summary, we have

- **Clarified scalability comparisons and benchmarks**: We ensure model scales were comparable to those used in prior work for tasks in section 3.1 such as BIOS. We also clarified for the WikiReversal experiments the MLM-U objective trained from scratch outperformed Mistral-7B, a model that’s 70x larger, after finetuning on the same data. Given our objective is not to train a general purpose language model, we do not benchmark standard language benchmarks as here we isolate only information retrieval when models are trained from scratch using MLM-U.

- **Compared in new experiments MLM-U versus AR training in the two-token setting proposed by reviewer bZEp (Figure 1 in PDF)**: We find MLM-U reaches 100% forward and backward accuracy whereas AR struggles to learn the backwards association in the two-token setting.

- **Measured the Computational Costs and Convergence of MLM-U (Figure 2 in PDF)**: We analyzed the runtime and convergence rates for both MLM-U and standard autoregressive (AR) training.  We benchmarked two parameter matched models on the retrieval task dataset from Section 3.1. We found MLM-U and AR exhibited comparable computational costs. While AR forward accuracy converges faster, MLM-U exhibits smoother convergence for the forward accuracy and is able to learn both backward and forward associations, whereas AR struggles to learn the backwards association. 

After incorporating these new experimental results and suggestions, thanks to reviewers’ feedback, we believe the quality of our submission has improved. We hope the factorization curse illustrates the importance of factorization-agnostic learning objectives for reliable knowledge retrieval. We believe together with our realistic WikiReversal benchmark, the factorization curse and proposed solution would be a valuable contribution to the research community for advancing the reliability of knowledge retrieval.

Meta-review: This paper examines the problem of reversal curse in LLMs by reframing it as a factorization curse —  a failure of models to learn the same joint distribution under different factorizations. Through a series of controlled experiments with increasing levels of realism, the paper finds that reliable information retrieval is an inherent failure of the next-token prediction loss widely used to train LLMs. It further shows that the problem cannot be easily addressed some several simple fix approaches. It further reveals a potentially promising direction to address this limitations by using factorization-agnostic objectives.

All the reviewers appreciate the solid contribution of this paper, such as a novel formulation of the reversal problem as a factorization curse. The paper is generally well written with solid experiments. The authors are encouraged to take the reviewers’ feedback into account when revising the paper.
Decision: Accept (poster)