Title: The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reversal curse

reliability

safety

language models

interpretability

learning objectives
Abstract: Today's best language models still struggle with "hallucinations", factually incorrect generations, which impede their ability to reliably retrieve information seen during training. The *reversal curse*, where models cannot recall information when probed in a different order than was encountered during training, exemplifies limitations in information retrieval. 
To better understand these limitations, we reframe the reversal curse as a *factorization curse* --- a failure of models to learn the same joint distribution under different factorizations.
We more closely simulate finetuning workflows which train pretrained models on specialized knowledge by introducing
*WikiReversal*, a realistic testbed based on Wikipedia knowledge graphs. Through a series of controlled experiments with increasing levels of realism, including non-reciprocal relations, we find that reliable information retrieval is an inherent failure of the next-token prediction objective used in popular large language models. Moreover, we demonstrate reliable information retrieval cannot be solved with scale, reversed tokens, or even naive bidirectional-attention training. Consequently, various approaches to finetuning on specialized data would necessarily provide mixed results on downstream tasks, unless the model has already seen the right sequence of tokens. 
Across five tasks of varying levels of complexity, our results uncover a promising path forward: factorization-agnostic objectives can significantly mitigate the reversal curse and hint at improved knowledge storage and planning capabilities.

The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More

Abstract

Today's best language models still struggle with hallucinations : factually incorrect generations, which impede their ability to reliably retrieve information seen during training. The reversal curse , where models cannot recall information when probed in a different order than was encountered during training, exemplifies this in information retrieval. We reframe the reversal curse as a factorization curse -a failure of models to learn the same joint distribution under different factorizations. Through a series of controlled experiments with increasing levels of realism including WikiReversal , a setting we introduce to closely simulate a knowledge intensive finetuning task, we find that the factorization curse is an inherent failure of the next-token prediction objective used in popular large language models. Moreover, we demonstrate reliable information retrieval cannot be solved with scale, reversed tokens, or even naive bidirectional-attention training. Consequently, various approaches to finetuning on specialized data would necessarily provide mixed results on downstream tasks, unless the model has already seen the right sequence of tokens. Across five tasks of varying levels of complexity, our results uncover a promising path forward: factorization-agnostic objectives can significantly mitigate the reversal curse and hint at improved knowledge storage and planning capabilities.

1 Introduction

Although today's best language models produce impressively cogent, articulate text by learning the statistics of language, they still struggle to reliably retrieve information seen during training. Models are known to suffer from hallucinations, potentially responding with fabricated content that differs from the knowledge present in training data. Hallucinations pose a significant hurdle to the adoption of language models, especially in domains where reliable knowledge retrieval is paramount (Dahl et al., 2024). A well-studied failure mode underlying hallucinations is the reversal curse , which ascribes this deficiency to the precise order of words presented to the model at train-time (Berglund et al., 2023; Allen-Zhu & Li, 2023). For example, a model trained on sentences where Paris always appears as the subject of the sentence, such as 'Paris is the capital of France' , can be tuned to answer 'Paris is the capital of which country?' but not 'What is the capital of France?' , even though these two formulations encode the same underlying information. Existing approaches aimed at mitigating the reversal curse have focused on data augmentations that involve training on both the forward and reversed tokens (Golovneva et al., 2024). In this work, we focus on learning objectives.

In Section 2, we propose the factorization curse , a framework that characterizes the reversal curse as a failure to model the same joint distribution under different factorizations. We show the prevailing

2. What is the capital of France?

1. Paris is the capital of which country?

Paris

is the

capital of

France.

Training / Finetuning

Corpus

Questions

at inference

🤖

: France

🤖

🤖

: Baguette

Which factorizations

can next-token

prediction learn?

Paris is the capital of

is the capital of France

🤖

🤖

Context

🤖

left-to-right next token prediction, autoregressive (AR) objective used in popular large models such as GPT (Radford et al., 2019) and Llama models (Touvron et al., 2023a,b), underlies the reversal curse. We illustrate in Figure 1 how the factorization in AR training only encodes information based on prior context, thereby limiting how well the model can retrieve information based on later context . Through this lens, we show the reversal curse is not merely a failure to learn logical implications, but a more general problem related to learning objectives. Given this framework, we hypothesize in Section 2.1 that factorization agnostic models, i.e. , models trained in a manner that is less dependent on the specific token order while preserving the overall meaning, can store knowledge better and are less prone to the reversal curse. To validate our hypothesis and explore potential solutions, we conduct extensive experiments in controlled settings in Section 3.1, focusing on the effects of pretraining objectives on knowledge storage. Section 3.2 introduces WikiReversal , a realistic testbed based on Wikipedia knowledge graphs that closely replicates a knowledge-intensive finetuning application. In experiments with increasing levels of complexity and realism, we observe that scale, naive bidirectional objectives, and even left-to-right training do not resolve the reversal curse. These results suggest that finetuning strategies for downstream applications might not allow models to store knowledge adequately. Finally, in Section 4, we find that factorization-agnostic training is not only a promising initial solution for knowledge storage but also hints at improved planning capabilities in a minimal graph traversal task.

To summarize, our contributions are as follows:

We introduce the concept of the factorization curse , which posits that different objectives' decomposition of an input into context and prediction is a key factor underlying the reversal curse.

We conduct empirical studies with increasing levels of complexity and realism to validate our framework, comparing strategies such as standard autoregressive training (AR), AR with reversed sequences, and masked language modeling (MLM) as a prototypical bidirectional objective.

Building on our factorization curse framework, we identify factorization-agnostic objectives that allow for making predictions using every possible context decomposition, as a strong baseline solution. We explore its effectiveness across all investigated settings, including the WikiReversal setting.

We show that factorization-agnostic strategies are promising not only for knowledge storage/retrieval but also for planning, suggesting potentially broader implications for our findings.

2 The Factorization Curse

The reversal curse highlights how language models struggle to reliably retrieve information seen during training given some context. Our aim is to understand this failure by probing how common training objectives factorize an input into context and prediction. We show how common training objectives, including popular left-to-right AR and masked modeling objectives, struggle to learn factorizations that can help the model generalize better on a given task, a challenge we label the factorization curse .

2.1 Hypothesis: Reversal Curse as an Instance of Factorization Curse

Let us define the factorization curse more formally. We first start with the usual left-to-right autoregressive model for a sequence x with D tokens. This is the standard formulation in popular GPT-style (Radford et al., 2019; OpenAI, 2023) models and its loglikelihood is given by

Each token is represented as x t , where t is its index in the sequence. x <t represents all tokens that precede the t -th token in the sequence. The log probability of the entire sequence x is computed as the sum of the log probabilities of each token x t , given all the preceding tokens x <t . This is the leftto-right factorization of the joint distribution over tokens. Note that there are many factorizations ( D ! ) of the joint distribution, each given by some permutation σ of the tokens in the sequence, which we can write as log p ( x ) = ∑ D t =1 log p ( x σ ( t ) | x σ ( <t ) ) .

Example in Two Tokens For illustration purposes, let us walk through an example with D = 2 . Suppose our goal is to model p ( x ) = p ( x 2 , x 1 ) = p ( x 2 | x 1 ) p ( x 1 ) . The left-to-right factorization loss optimizes

Interestingly, we can readily see the reversal curse failure mode in L AR . A model p θ that attributes high likelihood to p θ ( x 2 | x 1 ) p θ ( x 1 ) does not necessarily yield a high value for p θ ( x 1 | x 2 ) p θ ( x 2 ) (a right-to-left factorization) even though the two expressions should be equal due to the chain rule of probability. Note that here we make no statement about the sequential order of the random variables or their relationship. The only statement we make is that, unsurprisingly, p θ is not necessarily capable of modeling the joint distribution when presented with a different factorization. This is the factorization curse .

Definition 1 (Factorization Curse). A model p θ for the joint distribution of a sequence x suffers from the factorization curse if, for a factorization order σ different from the 'training' factorization order σ 0 (which depends on the objective and model details), we have

̸

In particular, the model may be optimal on σ 0 , but perform poorly on a different factorization.

Implications This has a number of implications. First, by definition, a highly factorizationdependent LLM will struggle to retrieve knowledge from earlier in the context given later information. Second, simply training on additional sequences with all tokens reversed would likely not alleviate the issue. Indeed, if the information we seek to retrieve is composed of multiple tokens, the factorization the LLM needs to handle is not right-to-left, but instead reversed in chunks. Thus, in order for any reverse training strategy to work, one must first parse the entities of interest then train with sequences reversed in entity-preserving chunks (see Section 3 and Figure 6).

Furthermore this explains why standard MLM approaches with fixed masking rates fail to address the issue, despite their bidirectionality, for two reasons: First, entities may span a larger number of tokens than the model masks, meaning there is never supervision signal to make the prediction from the right context (without leaking parts of the entity). Second, training with a fixed rate does not yield a good generative model. While the model is used to predicting, e.g. , 15% of a full context-window during training, at inference, the model can fail to generalize (Tay et al., 2022) to the arbitrary-length sequences it encounters (see Figure 2). Zhang et al. (2024) suggest that encountering different length sequences during training encourages disentanglement and compositionality, which will be crucial for a good generative model.

Knowledge retrieval beyond reversal: A model that cannot learn how to retrieve information in reverse order will likely suffer from further downstream issues that are often ascribed to hallucination. For instance, let us take a model pretrained on entities in a database, say a list of soccer players with various attributes (statistics, game histories, etc. ) with the name appearing before the attributes as follows x name , x attributes . The model may memorize the database perfectly, but when queried

for players that match specific criteria ( e.g. , played in a particular position, or have a specific nationality, etc. ), the model can produce hallucinated answers that do not match the training distribution due to lack of direct supervision of the form p ( x name | x attributes ) during pretraining.

2.2 Factorization-Agnostic Training Strategies

To store and retrieve knowledge in 'all directions' for arbitrary-length entities and without external intervention (entity pre-parsing, retrieval augmented generation, etc. ), the model needs to be equally good at any factorization of the joint distribution. Below, we discuss two ways this can be achieved.

Permutation Language Modeling (PLM) A straightforward way to alleviate the factorization issue, is to write the autoregressive loss in a way that is independent of factorization by averaging over all permutations as follows

where σ is a permutation sampled uniformly at random from S D , the permutation group on D tokens. The term x σ ( <t ) represents all tokens that precede the t -th token in the permuted sequence. The log probability of the entire sequence x is then lower-bounded (using Jensen's inequality) by the expected sum of the log probabilities of each element x σ ( t ) , given all its preceding tokens in the permuted sequence. Note that all we did here is average over all factorizations. This formulation is used in XLNet (Yang et al., 2020). However, for practical reasons they end up training with a permutation on the last few tokens only. This partial prediction, as we argued above, can limit knowledge storage improvements because we do not know how to chunk tokens into entities a priori.

Uniform-Rate Masked Language Modeling (MLMU ) Analternative factorization-agnostic objective is to predict any context from any other context uniformly at random. This includes nexttoken, previous-token, predictions spanning multiple future or past tokens, and all other forms of contextual prediction. As it turns out, this generalization over objectives (amounting to something similar to masked language modeling with a randomly sampled masking rate r ∼ U (0 , 1) ) is a discrete diffusion model with an absorbing masking state (Austin et al., 2023; Kitouni et al., 2024). This diffusion formulation can be used to make a factorization-order-independent autoregressive model. See Figure 2 for an illustration of the differences between the MLMU objective and more standard MLM. Specifically, L CT from Kitouni et al. (2024)'s Proposition 1, which we will refer to as L MLMU here, can be retrieved from Equation (4) as follows

where the last equality is possible because all τ ∈ σ ( ≥ t ) tokens are equally likely to appear at position t as we average across all permutations, and so we can average over the predictions for each τ at this position. This approach can be implemented as a denoising process which recovers randomly masked tokens, like BERT (Devlin et al., 2019), but with uniformly sampled masking rates. This key difference allows training a generative model with masked modeling. 2

3 Experiments

We now investigate information retrieval capabilities across learning objectives through the lens of different factorizations of the joint sequence probability. Specifically, we compare

AR: The standard autoregressive causal next-token prediction. Though all models generate tokens autoregressively, we use AR as a shorthand for left-to-right models, in line with Equation (1).

AR w/reverse: AR prediction on sequences and their token-level reverse. 3

MLM r : BERT-like masked language modeling with fixed random masking rate, r .

MLMU : MLMwith r ∼ U (0 , 1) . PLM results are similar, and are reported in the Appendix.

To ensure a fair comparison and allow each objective to perform optimally, we employ model architectures specifically designed for each objective. For autoregressive (AR) training, we use GPT-2 (Radford et al., 2019) and Mistral (Jiang et al., 2023). For masked language modeling (MLM), we use BERT (Devlin et al., 2019). Finally, for MLMU , we employ an encoder-decoder model 4 based on the GPT architecture (see Appendix G for details).

We study these models across increasing levels of complexity and realism, beginning with a controlled retrieval task using synthetic tokens to a retrieval task using natural text from Wikipedia articles. In our evaluation, we find that the degree to which the learning objective factorizes the input reliably explains performance across this wide range of information retrieval tasks. Factorizationagnostic methods show improved knowledge retrieval capabilities.

3.1 Controlled Experiments in Factorization-Agnostic Training

Retrieval Task. We are particularly interested in models' ability to recall knowledge from data they were trained on. We will use a simple toy task, adapted from Golovneva et al. (2024), to evaluate this capability. First, we generate a collection of key-value pairs which are each composed of a sequence { t i } i ∈ S of tokens, e.g. , consider the key-value pair

Each key/value is unique and is composed of a unique set of tokens to control for any effects due to token statistics. Additionally, we generate two types of queries: (forward) '[value of] key : value ' and (backward) '[key of] value : key '. Models are trained on all key-value pairs and a subset of queries, and tested on unseen queries by completing tokens after the colon. Accuracy, measured using exact match, in Table 1 shows AR training does not retrieve keys from values and that reversing tokens does not improve backward retrieval. We observe entity-based reversing trivially solves this task. Additionally, while MLM does not suffer from a forward/backward disparity, its fixed masking rate causes poor overall results. Introducing a uniformly sampled rate via MLMU solves the task perfectly.

Non-reciprocal Relationships. Are models employing incorrect reversal heuristics? A weakness of the retrieval task is that it could be solved by assuming all relations between keys and values to be symmetric/reciprocal. In language, this is not always the case: even though they contain many of the same words, the sentence Alice likes Bob does not necessarily imply that Bob likes Alice . To investigate whether models inappropriately rely on a reversal heuristic, we extend the retrieval task

MLM-15%

Train

Test

MLM-

U

Train

Test

}

{

Prediction

Context

Mismatch!

←∼U

(

0

,

1

)

→

15%

<latexi sh

1_b

64="YKSENUw5ZBG

Q

o

HAM2X

0

g

>

3

c

V

J

7

+

T

C

p

Ir

n

vF

u

R

m

Dk

y

W

9

8

/

P

z

d

O

j

Lq

f

<latexi sh

1_b

64="U8

MZAo f2B

O9vY

VW

0q

>

CH3

c

DNS

G

z

r/

L

gpRE

F

w

T

d

u

7k

I

Pm

K

j

y

+

5

JQ

n

X

<latexi sh

1_b

64="

k2U

j

L

B0

YH

urIq3T

R

98

>A

n

c

VD

SgM

F

v

w

K

P

7

C

oW

Gf

+

dm

Z

y

p

N

Q

/

5

J

X

↓

to a third entity for each sample, yielding statements of the form ' A = ⇒ B = ⇒ C ' . Question answering (QA) samples are of the form (forward) ' B = ⇒ ? ' and (backward) ' B ⇐ = ? ' where the right answers are C and A , respectively. With a third entity in play, a model assuming symmetry would be unable to decide between A and C as the answer for either question.

The bottom of Table 1 shows that simply reversing entities (denoted with AR w/reverse (entity)*) leads to undesirable behaviour as can be seen from the large incorrect inference rate. However, adding simple delimiter tokens around entity reversed sequences (without asterisk) leads to more a robust model. Finally, MLMU learns the asymmetric relations correctly.

BioS. Next, we investigate performance of the different objectives for more complex but still controlled data. BioS (Zhu & Li, 2023) is a synthetic dataset consisting of biographies for 10k fictional individuals. For each individual, biographical details (birth date, birth city, etc. ) were randomly selected from a uniform distribution. The authors ensured each individual was assigned a unique full name. We reproduce some of their results on the birth date-to-full name task which aims to recover a person's full name from their birth date. Results are shown in Table 2. Again, the autoregressive, MLM and reversed-token training struggle to recover backward queries.

Training in a factorization-agnostic fashion leads to non-negligible backward performance. Interestingly, backward performance keeps improving over a long time (many times the number of epochs required for forward performance to reach 100%) (see Appendix F). If this delay is due to the low frequency of observing the right factorization in training, this could indicate that methods to automatically select

data such as RHO-LOSS (Mindermann et al., 2022) could have a disproportionate impact in improving factorization-agnostic methods compared to standard AR training.

3.2 Wikipedia Knowledge Graph Reversal

To bridge the gap between the controlled studies on synthetic datasets and more realistic evaluations, we introduce a new evaluation setup that combines the best of both approaches. Our setup involves finetuning a language model on real-world natural text from Wikipedia articles, along with a precise knowledge graph describing the relations and entities within them. This allows for principled experiments that mimic real-world use-cases where practitioners finetune pretrained models on domain-specific corpora. We compare finetuning a language model in this standard setup to training from scratch with MLMU .

Experiment Design We introduce a new closed-book QA dataset to evaluate the ability of models to reason about entities and relations in both forward and backward directions. The dataset is derived

Paris

[a]

is

the

capital

and

most

populous

city

of

France. With

an official

estimated

population

of 2,102,650

residents as of […]

Paris

France

Capital

Of

What is the capital of France?

Paris

Paris is the capital of which country?

France

Questions

Wikipedia Passage

Graph

Forward

Backward

from the GenWiki corpus based on DBpedia (Jin et al., 2020), which contains 1.3 million text passages from Wikipedia, along with entity and relation annotations.

Extracting Relation Triples and Generating Questions For each passage P with annotated entities E = e 1 , e 2 , ..., e n , we consider only 'forward' relation triples ( e i , r, e j ) , where e i appears before e j in the passage. For the example, in the passage 'Paris is the [...] capital of France [...]' (Figure 3), the triplet ( Paris , capitalOf , France ) is a 'forward' triplet. Had the triplet ( France , hasCapital , Paris ) been present in the graph, we would consider it a 'backward' triplet. We filter the data to contain only triplets (and corresponding passages) for which the relation r exists at least in 500 different instances. We generate forward questions F r ( e i ) querying the tail of the relation ( e j ) and backward questions B r ( e j ) querying the head ( e i ) using predefined templates. We filter out ambiguous samples to ensure each question has a single unique answer. Algorithm 1 in the Appendix summarizes the dataset creation process.

Table 3 reports the forward/backward performance disparity, particularly for autoregressive models. Mistral 7B, achieves a backward accuracy of around 5%, much lower than its forward accuracy. Interestingly, the model starts around the same backward accuracy in the beginnig of finetuning. This indicates there may still be backwards triplets present in a 'forward fashion' within the model's training text. This could also explain the non-trivial back-

ward performance of the AR model, despite its susceptibility to the reversal curse. MLMU attains the highest backward accuracy among the evaluated models, demonstrating its robustness to the reversal curse. This suggests training from scratch with MLMU can outperform a much larger (70 × ) pretrained language model finetuned on the same data. However, it still falls short of the AR model's forward performance, possibly due to the inherent difficulty of the task. Notably, significantly better results can be obtained by allowing models to leverage knowledge stored from the QAs themselves (see Appendix E.3 for details).

3.3 Analyzing Representations Learned via Factorization-Agnostic Training

We further examine factorization-agnostic training by first comparing the role of random masking in MLMU versus standard masked language modeling. We also visualize the learned representations from MLMU showing they contain more distinct entity structure compared to standard AR training.

Understanding the role of random masking To understand the importance of varying the masking ratio as introduced in MLMU we compare MLMU to MLM with various masking ratios (15%, 40%, 85%) based on prior work (Wettig et al., 2023)). We find MLM exhibits much noisier performance that's consistently lower than MLMU with uniformly random masking ratio as shown in Figure 4a. This suggests fixed masking ratios, whether with high or low values, are limited in what they can learn in contrast to MLMU .

Visualizing learned representations in the 3-entity relationship task To better probe the representations learned via MLMU we plot in Figures 4b and 4c the PCA projections after training on the relationship task from Section 3.1 for AR and MLMU . Compared to AR, which learns disconnected components without apparent symmetry for entities never seen backwards during training, MLMU seems to have learned a form of translation symmetry across train and test samples. This suggests MLMU training leads to more structured entities in the model's representation space.

BioS

Relation

Retrieval

0

20

40

60

80

100

1.0

18.8

22.0

8.0

45.5

16.0

8.0

1.3

28.0

68.0

100.0

100.0

MLM 15%

MLM 40%

MLM 85%

MLM-

Train - Entity 1

Train - Entity 2

Train - Entity 3

Test - Entity 1

Test - Entity 2

Test - Entity 3

PC 1

Explained variance: 10.0%

PC 2

Train - Entity 1

Train - Entity 2

Train - Entity 3

Test - Entity 1

Test - Entity 2

Test - Entity 3

PC 1

Explained variance: 15.4%

PC 2

Figure 4: In panel (a) we compare MLM with varying masking ratios to MLMU . In panels (b) and (c) we visualize the two main principal components of representations learned via AR versus MLMU .

We also measure the computational efficiency relative to convergence of MLMU versus autoregressive (next token) training showing favorable efficiency for MLMU in Appendix Figure 9.

4 On the Importance of Future Predictions for Planning

Prior work argues next-token prediction auto-regressive loss is not conducive to planning (Dziri et al., 2023; LeCun, 2023; Gloeckle et al., 2024). Specifically, Bachmann & Nagarajan (2024) introduces a simple path finding task that requires basic planning: From a start node, multiple linear paths p 1 , p 2 , . . . , p n extend outward. They are given as symbolic sequences of this form: 2 , 6 | 6 , 7 | 5 , 1 | 4 , 3 | 4 , 2 | 3 , 5 ︸ ︷︷ ︸ edges / 4 , 7 ︸︷︷︸ start,end = 4 , 2 , 6 , 7 ︸ ︷︷ ︸ desired response Amodel is tasked to predict the sequence of nodes along path p i that leads to a specified final node at the end of p i . They show that when trained with a standard autoregressive (AR) next-token prediction objective, the model is unable to effectively learn this task. This failure is attributed, at least in part, to the teacher-forcing supervision used during training. As illustrated in Figure 5, from the second node x 2 = 2 onward along a path p i = ( x 1 , x 2 , . . . , x m ) , the model can predict each 'easy' token x t for t > 2 by simply conditioning on the immediately previous teacher-forced token x t -1 , without requiring retention of the earlier path history or look-ahead planning, a pitfall referred to as the 'Clever Hans' cheat (see Section 4.5 (Bachmann & Nagarajan, 2024) and (Pfungst & Rosenthal, 1911)).

4

3

start

5

1

2

6

7

node

goal

Bachmann & Nagarajan (2024) found that predicting multiple future tokens in a teacher-less setting helped mitigate the issue of discovering the algorithm to correctly predict the initial 'difficult' token x 2 . We identify this as an intermediate objective between standard next-token prediction and the factorization-agnostic objective studied in this work, which encourages planning capabilities via both far look-ahead and look-backward along the sequence. Figure 5a shows that the MLMU objective enables the model to reliably solve the path-planning task by better capturing the planning requirements.

5 Related Work

The reversal curse was first introduced in Berglund et al. (2023). Using text-overlap based heuristics for modeling inferences between sequence of text dates back nearly two decades in NlP (Glickman et al., 2005; Adams et al., 2007). As our modeling approaches have improved, increasing work has drawn attention to models overapplying text-overlap heuristics (Dasgupta et al. 2018; Naik et al. 2018; Sanchez et al. 2018; McCoy et al. 2019; Rajaee et al. 2022; Williams et al. 2022, i.a.). Perhaps most relevant is Sinha et al. (2019)'s evaluation, which used synthetic entity-based kinship data with multiple entities based on graph structures to expose model failures and is similar to our relationship task. Most recently, work aimed at mitigating the reversal curse by Allen-Zhu & Li (2023); Golovneva et al. (2024) suggest using data augmentations by reversing both token sequences, or if available, entity orders by training both on the forward and augmented text. Related projects have also trained and/or finetuned RoBERTa (Liu et al., 2019) or BERT (Devlin et al., 2019)-based models on input sequences with randomly shuffled word order (Gauthier & Levy, 2019; Chiang & Lee, 2020; Sinha et al., 2021). Lv et al. (2023) explore a fine-tuning objective with bidirectional attention and show that it can mitigate the reversal curse in the original synthetic setting from Berglund et al. (2023). However, they employ fixed masking rates. In addition to the standard objectives we explored, much recent work has gone into a variety of pre-training objectives including span-based and hybrid objectives (Joshi et al., 2020; Tay et al., 2022; Chowdhery et al., 2022). XLNet (Yang et al., 2020) utilizes a permutation language modeling objective, considering permutations of the input sequence during training. However, XLNet is not completely factorization-agnostic as it only predicts the last few tokens in each permutation.

Various benchmarks have been introduced to evaluate the reasoning capabilities of language models. Bachmann & Nagarajan (2024) present a study on the limitations of next-token prediction in capturing reasoning abilities, arguing that the standard autoregressive training objective hinders models' ability to plan. In a similar vein, Dziri et al. (2023) investigate the limits of transformer LLMs across three compositional tasks: multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. Their findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. They also provide theoretical arguments on abstract multi-step reasoning problems, highlighting how autoregressive generations' performance can rapidly decay with increased task complexity.

6 Discussion and Future Work

Limitations and Potential Extensions. MLMU has a much more challenging objective since we approximate all possible partitions of the input into context and predictions. Learning curves show delayed generalization, especially on backward samples. The main limitation of factorizationagnostic approaches is the optimization difficulty due to task complexity. Predicting one token ahead is far easier than predicting the last word of a novel with limited context, due to increasing entropy along longer horizons. This requires better schedules/curricula that smoothly interpolate the difficulty increase from next-token prediction to the highest-complexity factorization the model can handle.

This work highlights how alternative objectives can address some of the issues with current stateof-the-art language models, which rely on left-to-right autoregressive generative decoder pretraining. Despite impressive capabilities with increasing scales, there are concerns about reaching a plateau due to fundamental limitations, computational constraints, or data scarcity. We find that factorization-agnostic training can learn 'more' from the same data in the context of reversal curse.

This presents a case for studying factorization-agnostic objectives and investing in approaches to scale them.

References

Rod Adams, Gabriel Nicolae, Cristina Nicolae, and Sanda Harabagiu. Textual entailment through extended lexical overlap and lexico-semantic matching. In Satoshi Sekine, Kentaro Inui, Ido Dagan, Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini (eds.), Proceedings of the ACLPASCAL Workshop on Textual Entailment and Paraphrasing , pp. 119-124, Prague, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/W07-1420 .

Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation, 2023.

Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces, 2023.

Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024.

Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on 'a is b' fail to learn 'b is a', 2023.

David Cheng-Han Chiang and Hung-yi Lee. Pre-training a language model without human language. CoRR , abs/2012.11995, 2020. URL https://arxiv.org/abs/2012.11995 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.

Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho. Hallucinating Law: Legal Mistakes with Large Language Models are Pervasive, 2024. URL https://hai.stanford.edu/news/ hallucinating-law-legal-mistakes-large-language-models-are-pervasive .

Ishita Dasgupta, Demi Guo, Andreas Stuhlm¨ uller, Samuel J Gershman, and Noah D Goodman. Evaluating compositionality in sentence embeddings. In Proceedings of the 40th Annual Conference of the Cognitive Science Society , pp. 1596-1601, Madison, WI, 2018. URL https://cognitivesciencesociety.org/wp-content/uploads/2019/01/ cogsci18_proceedings.pdf .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL https://aclanthology.org/ N19-1423 .

Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, 2023.

Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 529-539, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1050. URL https://aclanthology.org/D19-1050 .

Oren Glickman, Ido Dagan, and Moshe Koppel. Web based probabilistic textual entailment. In Proceedings of the 1st Pascal Challenge Workshop , pp. 33-36, 2005.

Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozi` ere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction, 2024.

Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse, 2024.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L´ elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timoth´ ee Lacroix, and William El Sayed. Mistral 7b, 2023.

Zhijing Jin, Qipeng Guo, Xipeng Qiu, and Zheng Zhang. GenWiki: A dataset of 1.3 million content-sharing text and graphs for unsupervised graph-to-text generation. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics , pp. 2398-2409, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.217. URL https://aclanthology.org/2020.coling-main.217 .

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics , 8:64-77, 01 2020. ISSN 2307-387X. doi: 10.1162/tacl a 00300. URL https://doi.org/10.1162/tacl_a_00300 .

Ouail Kitouni, Niklas Nolte, James Hensman, and Bhaskar Mitra. Disk: A diffusion model for structured knowledge, 2024.

Yann LeCun. Do large language models need sensory ground- ing for meaning and understanding?, 2023. University Lecture.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ang Lv, Kaiyi Zhang, Shufang Xie, Quan Tu, Yuhan Chen, Ji-Rong Wen, and Rui Yan. Are we falling in a middle-intelligence trap? an analysis and mitigation of the reversal curse, 2023.

Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Anna Korhonen, David Traum, and Llu´ ıs M` arquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pp. 3428-3448, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 18653/v1/P19-1334. URL https://aclanthology.org/P19-1334 .

S¨ oren Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt H¨ oltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt, 2022.

Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. In Emily M. Bender, Leon Derczynski, and Pierre Isabelle (eds.), Proceedings of the 27th International Conference on Computational Linguistics , pp. 2340-2353, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https://aclanthology.org/C18-1198 .

OpenAI. Gpt-4 technical report. PREPRINT , 2023.

Oskar Pfungst and Robert Rosenthal. Clever hans : the horse of mr. von osten, 1911. URL https: //api.semanticscholar.org/CorpusID:142217369 .

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://api.semanticscholar. org/CorpusID:160025533 .

Sara Rajaee, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. Looking at the overlooked: An analysis on the word-overlap bias in natural language inference. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pp. 10605-10616, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.725. URL https://aclanthology.org/2022.emnlp-main.725 .

Ivan Sanchez, Jeff Mitchell, and Sebastian Riedel. Behavior analysis of NLI models: Uncovering the influence of three factors on robustness. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pp. 1975-1985, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1179. URL https://aclanthology.org/N18-1179 .

Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, and William L. Hamilton. CLUTRR: A diagnostic benchmark for inductive reasoning from text. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pp. 4506-4515, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1458. URL https://aclanthology. org/D19-1458 .

Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp. 2888-2913, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.230. URL https: //aclanthology.org/2021.emnlp-main.230 .

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garc´ ıa, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. Ul2: Unifying language learning paradigms. In International Conference on Learning Representations , 2022. URL https://api.semanticscholar.org/CorpusID:252780443 .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,

Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.

Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. Should you mask 15% in masked language modeling?, 2023.

Adina Williams, Tristan Thrush, and Douwe Kiela. ANLIzing the adversarial natural language inference dataset. In Allyson Ettinger, Tim Hunter, and Brandon Prickett (eds.), Proceedings of the Society for Computation in Linguistics 2022 , pp. 23-54, online, February 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.scil-1.3 .

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, 2020.

Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, and L´ eon Bottou. Memory mosaics, 2024.

Zeyuan Allen Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2023.

A Why does AR w/reverse sequences fail?

is the capital of

France

AR w/reverse

Prediction

Context

Mismatch!

P

A

R

I

S

The capital of

France is

P

A

R

I

S

While the model can

fi

ll sentences backwards,

answering a backwards Q actually requires:

Left-to-right

Right-to-left

Left-to-right conditioned on

some context on the right

which it fails to do because it fails on the similar task:

Paris is the

capital of F

R

A

N

C

E

is the capital of

France

P

A

R

I

S

is the capital of

France

P

A

R

I

S

B Permutation Language Modeling and Discrete State Diffusion

To illustrate the similarity between the diffusion loss and permutation language modeling, let's continue walking through our D = 2 example. Permutation modeling averages over factorizations p ( x ) = 1 2 p ( x 2 | x 1 ) p ( x 1 ) + 1 2 p ( x 1 | x 2 ) p ( x 2 ) and optimizes a lower bound on the likelihood

Finally, the diffusion model averages over masking rates 1 2 (log p ( x 1 )+log p ( x 2 ))+ 1 2 (log p ( x 1 | x 2 )+ log p ( x 2 | x 1 )) and optimizes

This is the same as Equation (6). This implies that the permutation language modeling and the absorbing state diffusion objectives are in fact the same. Though practically speaking, they may have very different implications.

C Summary of Tables

Table 4 shows a qualitative comparison of the optimization objectives explored on the different datasets in this paper. We conclude that MLM with a fixed masking rate mitigates the reversal curse due to its bi-directionality, but lacks generative quality and thus generally fails when having to provide longer answers. Also unsurprisingly, the left to right AR objective works well in the forward retrieval direction but is unable to answer backwards questions and has a hard time reasoning multiple tokens ahead to solve a task like graph traversal without intermediate supervision. Reversing the tokens can aid backwards retrieval for single token lookups, but fail otherwise. Reversing entities intuitively should be able to solve every retrieval task, but finding the right token permutation is a

TwoTokenSettingForwardAccuracy

MLM-U = AR

MLM-U=AR

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

Epoch

Epoch

0

0

500

1k

1.5k

2k

500

1k

1.5k

2k

difficult task by itself. MLMU averages over all possible prediction tasks that exist for a sequence given a tokenization and prevails in most our experiments. MLMU displays the highest backwards retrieval capabilities in the most realistic Wikireversal benchmark, but the performance is not strong enough to qualitatively state success and we mark it with ∼ in Table 4. We hypothesize that the reason is increased task complexity requiring larger models. Notably, in Table 8 we show that MLMU outperforms all other objectives when it has access to either the forward or backward type question. From there, it can generalize well to the other type.

We also experiment with a simple two-token setting comparing MLMU and autoregressive next token prediction in Figure 7.

D Additional Tables

Algorithm 1 Dataset Creation

E WikiReversal

We outline the setup for the WikiReversal dataset. After installing the GenWiki package https:// github.com/zhijing-jin/genwiki , users can download and process GenWiki to form WikiReversal using:

E.1 Filtering Ambiguous Samples

To mitigate ambiguity in the generated QA pairs, we filter the dataset to retain only ( e i , r, e j ) triples where the ( e i , r ) and ( r, e j ) pairs are unique across the entire dataset. This ensures that each question has a single unambiguous answer. Algorithm 1 summarizes the dataset creation process.

E.2 Examples from the Wikireversal dataset

Table 10 shows the relations present in the Wikireversal dataset. Table 9 shows examples of passages and corresponding forward and backward questions that are trained on. WikiReversal is filtered from GenWiki (Jin et al., 2020), a dataset based on Wikipedia released under a Creative Commons Attribution 4.0 International License.

E.3 Details on Wikireversal training

To measure performance on the Wikireversal dataset, we split the available data into training and validation, where we include all passages and 80% of both forward and backward questions in the training set and 20% of questions in the validation set. We run a hyperparameter grid search over every objective. We sweep over feasible learning rates for all models and weight decay for all except for MLMU , where we haven't found weight decay to be effective so it is set to 0. We run sweeps for both different (Table 8 right) and same (Table 8 left) train test splits in forward and backward questions. GPT and BERT models have 12 layers with 12 heads and 768 embedding dimension for a total of 108 M parameters. The encoder-decoder model has 12 layers with 9 heads and 576 embedding dimension for a total of 109 M parameters. All models except Mistral were trained for 1500 epochs, and Mistral was trained for 200 to alleviate overfitting. The learning rates were warmed up for 1% of the training time in all cases. For Mistral, the LoRA α and r parameters are set to 256 to sum to about 109 M trainable parameters. Learning rates are 5e-5 for MLMU in both train split modes, 3e-4 for both BERT (MLM-15%) and GPT (AR) for both modes and 1e-4 for Mistral (AR). The best weight decays are 1e-2 for both BERT and GPT, and no weight decay for LoRA on Mistral. No dropout was used. All models were trained with AdamW with default β parameters.

Table 8 shows that MLMU outperforms other objectives, achieving 66% and 46% accuracy on forward and backward questions, respectively. Using different seeds for train/test splits in forward and backward directions (right section) allows bidirectional models to learn to answer questions from both the passage and the question itself, explaining MLMU 's significant improvement. AR

performs poorly on backward questions due to the 'reversal curse'. MLM and Mistral 7B show intermediate performance. Although Mistral 7B uses ∼ 100 M LoRA parameters, fewer than the other models, this setup mimics common fine-tuning recipes. Naturally, models trained from scratch do not learn general language modeling capabilities.

F Delayed Generalization in Language Modeling

We include accuracy curves for training with MLMU for both Bios and WikiReversal in Figure 8. We see the model is able to gradually learn both the forward and backward questions throughout training. For Bios, unlike the forward questions which saturate much more quickly, the backward accuracy still shows an upward trend after training for 20k optimization steps. We observe a similar trend in the delayed generalization in WikiReversal for both forward and backwards questions even after training for 300k optimization steps. These results empirically demonstrate that the MLMU objective, which requires modelling all possible factorizations of an input into context and predictions, is a more challenging task that exhibit delayed generalization relative to standard next-toprediction training.

0

2500

5000

7500

10000 12500 15000 17500 20000

Optimization Steps

0.0

0.2

0.4

0.6

0.8

1.0

Forward

Backward

0

50000

100000

150000

200000

250000

300000

Optimization Steps

0.00

0.02

0.04

0.06

0.08

0.10

Forward

Backward

G Architecture Details

The Encoder-Decoder architecture used to train the MLMU objective is modeled with ideas from XLNet Yang et al. (2020) in mind in order to support different attention/masking strategies including permutation language modeling. The encoder has GPT-like blocks and works with RoPE as positional bias. The decoder also has GPT-like blocks, but it cross-attends over keys and values from the corresponding encoder layer, also via a RoPE bias. The decoder input contains the same learnable embedding for all tokens, such that only the positional bias defines the initial attention pattern. This idea comes from XLNet's positional attention stream. In left to right AR training mode, both encoder and decoder use a causal attention mask. In MLM-X modes, a fraction of inputs are masked before given to the model and neither decoder nor encoder attend over the masked tokens. All inference is performed in left-to-right AR fashion.

H Compute Requirements

Models were trained on 64 NVidia V100 and A100 GPUs with supporting Intel(R) Xeon(R) Gold 6230 CPUs. From conception to finalization of this paper we trained about 2000 models. The computationally most expensive runs were on the BioS and the Wikireversal dataset. Those comprised about 300 runs with on 8 GPUs for around a day per model. About 30 Mistral models were trained on 32 GPUs for about a day per model. We also compare the convergence relative to GPU hours for MLMU to AR in Figure 9.

Convergence ofForward Accuracy

MLM-U=AR

MLM-U=AR

0.6

0.8

0.5

0.6

0.4

0.4

0.3

0.2

0.2

0.1

Epochs

Epochs

0

0

2k

4k

6k

8k

2k

4k

6k

8k

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed right before Section 4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We describe the notation and assumptions required for our results regarding factorization.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Refer to main text and Appendix E

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We provide details to recreate the Wikireversal dataset in Algorithm 1, but the code is not yet ready for open sourcing.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See main text and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The margins between methods used to support claims in this paper are beyond typical error bounds due to stochasticity during training and supported by backing-theory regarding factorization agnostic approaches.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe the computational resources used for model training in Appendix H

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We ensure all datasets are synthetically generated or openly-available as is the case for Wikipedia. We do not have any human participants in this study.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss benefits in terms of reliability for applications involving LLMs and limitations of the approach in terms of longer-training required.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite synthetic datasets used from prior works, which do not come with a specific license as they are procedurally generated. The Wikipedia graph used for the Wikipedia setting is released under CC BY-SA 4.0.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on the \"reversal curse\", where models struggle to recall information when probed in a different order than encountered during training. The authors propose reframing this issue as the \"factorization curse\", which is a failure of models to learn the same joint distribution under different factorizations. They support it with experimental evidence, providing also a new dataset WikiReversal, to evaluate this problem and explore solutions. Authors suggest to use factorization-agnostic training objectives, which can significantly mitigate these issues.\n\nSTRENGTHS:\n1. Authors introduce novel concept of  \"factorization curse\", broadening the understanding of why models fail in information retrieval.\n2. The development of WikiReversal, based on Wikipedia knowledge graphs, is a valuable contribution, offering a more realistic benchmark for evaluating model performance in information retrieval tasks.\n3. Authors provide extensive experiments, proving the fact that \"factorization curse\" exists both in toy examples and real cases.\n4. They suggest a factorization-agnostic training objective that showed good performance in mitigating such issues, while preserving satisfactory quality in forward queries.\n\nWEAKNESSES:\n1. While the paper presents a variety of experiments with different training objectives, they experiment with small model sizes (100M parameters). It is not clear how well this approach could be scaled on bigger models. \n2. Authors didn't clearly analyze whether proposed training objective will hurt the quality on the standard benchmarks. For example, it would be interesting to see if the quality of the open generation drops with new training objective.\n\nQUESTIONS:\n1. Do the quality of generation drops if we use proposed method to train the model? \n2. Will you give the open access to the provided WikiReversal? \n3. What are the computational costs associated with factorization-agnostic training, and how do they compare to traditional training objectives?\n\nAlso some typos:\n\nLine 83 \"are: missed in “Note that there many”\n\nLIMITATIONS:\n1. The experiments are primarily based on the WikiReversal testbed, which, while realistic, is still limited to the structure and content of Wikipedia knowledge graphs. \n2. Limited scales of models, the training was done only on small models, which could be justified for retrieval itself, but need to be addressed for scalability of approach", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the reversal curse, where models trained on a relation in one direction (e.g. \"A is the capital of B\") cannot answer questions about the relation worded in the reverse order (e.g. \"B's capital is [?]\"). The authors frame this as a subcase of a broader problem, where models trained with a causal learning objective do not learn to assign equal probability to different factorizations of the same sequence, which the paper terms the factorization curse. To address the factorization curse, the authors propose using MLM-U, a modified MLM that varies both the location and size of the masked segments. The authors show that models trained with this modified objective do not exhibit the same reversal curse and analyze the behavior of models trained with causal, MLM, and MLM-U objectives.\n\nSTRENGTHS:\nS1. The paper tackles a well-defined problem from an interesting angle, providing both theoretical and empirical evidence for its claims. The paper is well-written.\n\nS2. The connection between PLM and discrete state diffusion here is an interesting insight and well-explained. \n\nS3. WikiReversal is an interesting idea for a dataset and looks like it will be a great resource for the community, especially for work aiming to study this reversal phenomenon. It is documented well.\n\nWEAKNESSES:\nW1.  In a two-token setting, you can (at a high but not intractable computational cost) compute the right-to-left factorization of a causal model through enumeration. It would be interesting to compute the left-to-right and right-to-left factorizations for some two-token sequences at test time for the models presented here. Do the causal models clearly suffer from a discrepancy of joint probabilities in the two-token case? Does the MLM-U training mitigate the two-token factorization curse? It seems likely the answer is yes, but it would strengthen the paper to confirm this empirically. (And in a 2-token setting, it seems that MLM would also mitigate the factorization curse-- does this happen in practice?)\n\nW2. Comparing BERT-like MLM to encoder-decoder MLM-U or GPT-like AR seems to be a bit of a strange comparison. It would seem more natural to use a BART-like encoder-decoder for the MLM example (though I recognize that the causal attention in the decoder may be slightly confounding, so I think both settings have merit; still, it would be good to *see* both to be sure). \n\nW3. The architecture details for MLM-U, while not the key argument of the paper, are only sparsely explored. It is not clear how much of the benefit comes from using the new architecture design versus using the MLM-U objective. And since Appendix G suggests that this architecture can handle both MLM-X and AR training, why not just train all models using this architecture and vary only masking strategy?\n\nQUESTIONS:\nQ1. In appendix B, you state that the PLM and absorbing state diffusion objectives are “the same” but “practically speaking […] may have very different implications”. Can you elaborate on this point? \n\nQ2. Why use BERT rather than BART (or another encoder-decoder) as the choice for MLM?\n\nQ3. You argue that at least some of the benefits of MLM-U over MLM arise from the ability to fully mask out multi-token entities. Could you demonstrate that, given a task where the entities are all single-token, the benefit of MLM-U is marginal or non-existent? I’m thinking of a version of the retrieval task in 3.1 where the key-value sequences are all of length 1.\n\ntypos/small notes:\n* line 79: GPT-style citation should probably be GPT-2 or GPT, not GPT-4.\n* line 83: “Note that there *are* many factorizations”\n* Table 1: very small point, but I would have expected the entry with the asterisk to be the entry *with* delimiter tokens.\n* I think the factorization curse section could emphasize even more that the ordering of the tokens in the input is not altered when calculating the different factorizations. This is stated in the text (lines 83-85 and 92), but given that the section begins by discussing causal models (and, like a lot of us, I work almost exclusively with causal models these days!), it still required a second read to parse exactly what was going on.\n\nLIMITATIONS:\nLimitations seem appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the idea of the \"Reversal Curse\" from prior work and proposes ways to mitigate it by finetuning LLMs with a different objectives. To recall, the reversal curse is formulated roughly as follows: a model, when **finetuned,**(i.e. not prompted) in A is B statement, does not automatically generalize to B is A. Authors generalize this, from a probabilistical point of view, into the inability to generalize between different factorization orders on the joint text distribution. Authors then hypothesize that this can be alleviated by what they call \"permutation-agnostic training\" - training techniques popular among LLM that require predicting tokens in a varying order. Authors consider a number of such techniques based on prior work on encoder pretraining and find that MLM-u offers a consistent solution to the reversal/factorization curse, while many other intuitive solutions don't. Authors experiment using two relatively smaller language models (GPT-2 and Mistral 7B) on tasks like simple retrieval, understanding non-reciprocal relationships  (between sets/statements), WikiReverasal. The paper also analyzes the representations learned after the model is fine-tuned with the proposed approach.\n\nSTRENGTHS:\n1. The paper proposes (seemingly) a solution that covers one of the reasons why LLMs hallucinate. Since LLM hallucination is one of the main roadblocks to their greater adoption, this is an impactful problem to address.\n\n2. The experiments appear sound: authors evaluate on a diverse set of tasks, using two different LLMs. This eliminates the possibility of a false positive. However, there is still a direction of scaling to larger models (e.g. 70B and above) and checking against possible changes in the efficacy of the proposed solution.\n\n3. The proposed method relies on fine-tuning, not training a model from scratch, and thus can be broadly applied to existing models\n\n4. The paper is generally well written, has a clearly stated hypothesis and is overall easy to follow, if a bit unconventionally structured. Minor typos and clarification requests (below) do not affect the overall presentation.\n\nAlso, a minor but pleasant advantage of this paper is that they openly declare that their solution is a clever reuse of an existing method. Many works I read in the past instead  choose to slightly modify the approach and declair it a newly proposed method. I count your choice as a minor advantage because it reduces the research debt, i.e. how many methods does a newcomer need to learn to meaningfully contribute to this area.\n\nWEAKNESSES:\n### Side-effects can be explored better\n\nMy main concern with the paper is that , while there is great attention to how fine-tuning combats reversal curse, but the side-effects of such fine-tuning are arguably not explored enough.\n\n\n**In other words, does fine-tuning Mistral affect its accuracy on other, unrelated tasks? If yes, what trade-off points are there?**\n\nTo test that, once can use LM Eval Harness for a selection of such tasks ( https://github.com/EleutherAI/lm-evaluation-harness ). If not familiar with the tasks, please use the ones commonly reported upon popular LLM releases (e.g. see Table 20 in the Llama 2 paper https://arxiv.org/pdf/2307.09288 ) or choose your own tasks. Another direction would be to evaluate the overall quality of your model's generated text, whether with human evaluation (local/MTurk/Toloka/...) or, as an Ersatz, with another LLM (as in https://arxiv.org/pdf/2304.03277 )\n\nThis can make a difference between a \"free\" solution to the curse and the one that comes at too great a cost got most practitioners -- or something between the two. As such, the paper would greatly benefit from understanding these trade-offs, or knowing that they don't exist.\n\n\n### XLNet the Baseline\n\nYou (rightfully) spend a lot of Section 2.2 on describing XLNET (alongside MLM-U), but then never compare against that as a baseline.\n\nWhile you offer some algorithmic reasons MLM-U could be better, but to dismiss a relevant algorithm as a baseline, one usually needs more evidence, e.g. showing that it is infeasible to use or proving that it is guaranteed to be worse. I could not find such an argument in the paper. If you have it, please direct me to it. If not, the paper could be improved by actually comparing to XLNET, at least as a side-experiment on a subset of tasks, so that the reader better understands your choice of MLM-U.\n\nQUESTIONS:\n### Does this scale?\n\nIn your work, you test the proposed solution on GPT-2 and Mistral 7B. While the latter is undoubtedly an LLM, it is still curious if your approach generalizes to more capable LLMs. Note that I am not asking you to run all the experiments with a larger model, but even a selection of such experiments would improve the paper, particularly for practitioners.\n\nIf your main concern is GPU size, it should be possible to fine-tune relatively larger models using QLoRA ( https://github.com/artidoro/qlora ) or ReLoRA ( https://arxiv.org/abs/2307.05695 ). The former can fine-tune a 65B model on a 48GB GPU or multiple consumer gpus with sharding. The latter amounts of running the formal, then merging the adapters into LLM parameters and running again, for several such loops. Note that there may be a potential confounder of LoRA adapters vs regular fine-tuning. If you care to disentangle these effects, one possible way is to first check if LoRA finetuning can lift the curse for Mistral 7B, and if it does, try for larger ones and compare against that.\n\n\n\n\n> (L43) to learn learn logical implications,\n\nPossibly an extra “learn” (typo)\n\n\n> (L174-175)\n\nthere is a paragraph missing line numbers, likely due to excessive negative vertical space. In that paragraph, and in the unnamed equation below, you refer to tokens as $t_1, t_2$, etc. In turn, your text up to this point refers to tokens as $x_0, x_1$, etc Moreover, $t$ is explicitly reserved for token index (L80). Unless there is a specific reason for this, the presentation would be improved if you untangle the notation around $x$ and $t$.\n\nLIMITATIONS:\nTo the best of my knowledge, authors have sufficiently addressed limitations of their work", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}