Venue: NAACL 2025

# Reverse Modeling in Large Language Models

Sicheng Yu\({}^{1}\) Yuanchen Xu\({}^{2*}\) Cunxiao Du\({}^{1}\) Yanying Zhou\({}^{3}\)

Minghui Qiu\({}^{1}\) Qianru Sun\({}^{1}\) Hao Zhang\({}^{4\dagger}\) Jiawei Wu\({}^{2\dagger}\)

\({}^{1}\)Singapore Management University \({}^{2}\)National University of Singapore

\({}^{3}\)Fudan University \({}^{4}\)DAMO Academy, Alibaba Group

scyu.2018@phdcs.smu.edu.sg

{yuanchen_xu, jiaweiwu}@u.nus.eduzhz.hhea2e@alibaba-inc.com

###### Abstract

Humans are accustomed to reading and writing in a forward manner, and this natural bias extends to text understanding in auto-regressive large language models (LLMs). This paper investigates whether LLMs, like humans, struggle with reverse modeling, specifically with reversed text inputs. We found that publicly available pre-trained LLMs cannot understand such inputs. However, LLMs trained from scratch with both forward and reverse texts can understand them equally well during inference across multiple languages. Our case study shows that different-content texts result in different losses if input (to LLMs) in different directions--some get lower losses for forward while some for reverse. This leads us to a simple and nice solution for data selection based on the loss differences between forward and reverse directions. Using our selected data in continued pre-training can boost LLMs' performance by a large margin across different language understanding benchmarks.

+
Footnote †: Corresponding authors.

+
Footnote †: Corresponding Authors.

## 1 Introduction

LLMs Touvron et al. (2023); Jiang et al. (2023) have shown impressive capabilities in various natural language processing tasks and beyond. These capabilities are primarily attributed to the learning of extensive corpora that cover general world knowledge Kaplan et al. (2020). These corpora are created in human society and often demonstrate human bias, including inherently forward-oriented human cognition Bergen and Chan (2005); De Kerchhove and Lumsden (2013), _e.g._, reasons may precede outcomes and solutions can be deduced from given information in most cases of the grad school math dataset Mitra et al. (2024). In contrast, reverse thinking presents more cognitive challenges due to its contradiction with innate common sense and human logic Chen et al. (2024). It inspires us to explore the following questions:

* [noitemsep]
* _Can LLMs perform reverse modeling or will they face similar challenges as humans?_
* _Can reverse modeling benefit the learning of LLMs?_

To study this, we simulate reverse-modeling data by directly reversing entire paragraphs or documents at the token level. Please note that this is the simplest and extreme way, but may not be the optimal way of emulating reverse thinking. We train LLMs with these simulated texts and conduct a comprehensive analysis. Overall results indicate that LLMs learn forward- and reverse-modeling texts equally well when trained from scratch. However, performance varies across text samples. Some are suited to reverse modeling, while others favor forward modeling. Notably, we find that the texts suited for reverse modeling are often of high quality and more logically coherent. Training on them, the original "forward-modeling" LLMs can be improved. We perform empirical validation on language understanding benchmarks, such as Massive Multitask Language Understanding (MMLU) Hendrycks et al. (2020). In summary, this paper has two main contributions.

* [noitemsep]
* We examine how LLMs process and learn from text in both forward and reverse directions, demonstrating consistent patterns across multiple languages.
* We show that strategically selecting training data based on the losses of forward- and reverse-modeling leads to improved model capabilities.

## 2 Related Work

In this paper we utilize the reverse text for model training. Previous work on reverse inputs falls into

three main areas. The first area involves the use of reverse text in machine translation. Studies show that using decoders to process text both left-to-right and right-to-left within an encoder-decoder framework improves machine translation performance (Zhou et al., 2019; Gu et al., 2019), a finding later extended to LLMs (Nguyen et al., 2024). Concurrently, Wu et al. (2018) examines the relationship between error propagation and reverse direction decoding in machine translation. The second area focuses on the reversal curse (Berglund et al., 2023; Zhu et al., 2024), where an LLM trained to understand "A is B" may struggle to generalize to "B is A". Reversing the text is proposed as a solution to this problem (Golovneva et al., 2024; Guo et al., 2024). These two streams of work focus on machine translation or the reversal curse. Third, a recent work (Papadopoulos et al., 2024) also explores the direction of input text, but there are two key differences compared to ours: (1) Our work is inspired by the concept of reverse thinking, while the reversed input is one simulating solution; (2) We further analyze it across different domains and inference steps and discover a valuable tool for assessing data quality.

Our applications are partially related to the selection of training data for LLMs, which is divided mainly into heuristic and model-based methods (Longpre et al., 2024). Heuristic methods filter out low-quality data by defining various rules, such as the ratio of nouns and verbs (Raffel et al., 2020; Penedo et al., 2023; Chowdhery et al., 2023; Sharma et al., 2024). Model-based methods filter data by training selection models or based on the perplexity of language models (Wenzek et al., 2020; Xie et al., 2023; Wettig et al., 2024). However, our data selection method is an extra bonus derived from the reverse modeling analysis.

## 3 Experimental Settings

Forward and Reverse Training.Given a original text, it can be represented as a sequence after tokenization, which is used for forward training. To perform reverse training, we directly reverse the original token sequence to construct a reverse training sample. While some studies explore keeping the original orders of detected words or entities during reverse (Golovneva et al., 2024; Guo et al., 2024), we choose the simplest operation to avoid the various performance of detection modules in different domains and languages. The Llama2-7B (Touvron et al., 2023) (or the randomly initialized version) is selected as the default backbone in this paper.

Datasets.In Research Question (RQ) 1, we used the multilingual mC41(Raffel et al., 2020) dataset to compare LLMs' ability to handle forward and reverse texts under continued and foremost pretraining settings. In subsequent experiments, we used the English SlimPajama2(Soboleva et al., 2023) dataset, which includes seven different source domains. Testing LLMs trained on the multilingual mC4 dataset with samples from the SlimPajama dataset can further confirm our findings are general. More details are in Appendix A.

Footnote 1: English, German, Korean, Arabic from [https://huggingface.co/datasets/allenai/c4](https://huggingface.co/datasets/allenai/c4)

Footnote 2: We use the widely-used public sampled version for experiments: [https://huggingface.co/datasets/DX](https://huggingface.co/datasets/DX) Yoon/SlimPajama-6B

## 4 Experiments

### RQ1: _Can LLMs perform reverse modeling?_

To explore LLMs' reverse modeling capabilities, we investigate two pre-training approaches: (1) continued training from a well-trained model checkpoint and (2) pretraining from scratch with random initialization. Specifically, we train models fed with forward input and reverse text using the two approaches separately. Figure 1 compares training losses (average sample losses within training batches) for English using both methods on the mC4 dataset, while Figure 7 in the Appendix B shows analogous results for other languages.

In the continued pretraining setting, the forward loss for forward-modeling remains stable due to extensive training in the initial pretraining stage. In contrast, the reverse loss for reverse modeling, initially high, decreases rapidly after a few training steps. Notably, the forward loss is consistently lower than the reverse loss during continued pretraining. We speculate this occurs because the initial pretraining corpora consists entirely of forward-direction texts, imparting a natural directional bias to the LLMs. Consequently, the models find pro

Figure 1: Pre-training loss for both continued setting and from-scratch settings in English.

 

cessing reverse information more challenging, similar to human difficulties with reverse thinking.

Interestingly, in the from-scratch pretraining, the loss curves for both text directions converge almost identically. This pattern, also observed in other languages, indicates that LLMs can learn to process forward and reverse-modeling inputs with similar proficiency. This is because the model learns from both forward and reverse texts simultaneously with randomly initialized parameters, avoiding the initial forward-direction bias in well-trained models.

**RQ2: _Does data domain influence the ability of LLMs'reverse modeling?_**

Based on the observation in RQ1, we focus on the from-scratch pretraining setting, where trained LLMs show almost equal losses from both forward and reverse directions. This raises the question of whether reverse loss consistently equals forward loss across all texts or if there are instances where reverse learning incurs a lower or higher loss. To explore this, we use the SlimPajama Soboleva et al. (2023) text dataset, which covers a broad range of domains, for case-level evaluation.

Given a text sequence represented by tokens \(\{V_{1},V_{2},\cdots,V_{N}\}\), for each position \(t\) in the sequence (\(0\leq t\leq N\)), a LLM can generate a probability distribution over possible next tokens. We compute the cross-entropy loss at each position \(t\), resulting in two sequences of loss values: \(\{F_{1},F_{2},\cdots,F_{N}\}\) for the forward sequence and \(\{R_{1},R_{2},\cdots,R_{N}\}\) for the reverse sequence.

We first compute the average loss difference (computed as \(\frac{1}{N}(\sum_{i=1}^{N}F_{i}-\sum_{i=1}^{N}N_{i})\)) for each text and associate each text with its corresponding data source label. The overall case-level loss difference distribution across different source domains is shown in Figure 2. Observed that the loss differences of the text samples are centered around zero, showing an approximately normal distribution. Importantly, this indicates that reverse-direction loss is not universally higher than forward-direction loss. In fact, for over half of the texts, reverse prediction of the next tokens is comparatively easier.

As indicated in Figure 2, compared to web-scraped corpora such as Wikipedia and Common Crawl, the distributions of loss differences from Book and ArXiv are generally less skewed towards easier forward-modeling. Furthermore, a larger proportion of texts in Book and ArXiv are easier to predict in the reverse direction compared to the forward direction. Considering that texts from books and academic papers are typically of higher quality than web-scraped texts, we speculate that texts, where reverse prediction is more effective, are generally more coherent, naturally flowing. Table 1 summarizes the randomly selected examples from the reverse easier and forward easier texts. The reverse easier texts display a coherent structure and smooth flow, making them easy for readers to follow. In contrast, the forward easier texts are relatively low-quality, less coherent, and often repetitive. This conjecture is also reflected in domains related to code, StackExchange, and Github. From the perspective of natural language, code often features monotonous syntax and repetitive vocabulary.

From the perspective of human forward thinking and its reflection in written texts, the forward-direction prediction task, which involves predicting the future from the present, is inherently more challenging. Conversely, the reverse-direction token

Figure 2: Loss difference distribution across domains.

 

prediction operates from known outcomes back to their origins, potentially simplifying the task.

**RQ3: _What features make texts easier to process in the reverse direction?_**

To further validate our hypothesis, we conduct a detailed analysis of step-by-step loss changes during token decoding. While the aggregated view in RQ2 is informative, it hides the underlying step-by-step dynamics of the loss. Given \(m\) text sequences in the SlimPajama dataset, we can obtain the two step-by-step loss sequences \(\{\frac{\sum^{m}_{j=1}F^{j}_{1}}{m},\frac{\sum^{m}_{j=1}E^{j}_{2}}{m},\dots, \frac{\sum^{m}_{j=1}F^{j}_{N}}{m}\}\) for the forward modeling and \(\{\frac{\sum^{m}_{j=1}R^{j}_{1}}{m},\frac{\sum^{m}_{j=1}R^{j}_{2}}{m},\cdots, \frac{\sum^{m}_{j=1}R^{j}_{N}}{m}\}\) for the reverse modeling. We exclude the first and last tokens with step loss = 0 to avoid sharp changes at the start and end. To account for different text lengths, we normalize the steps of all texts to the interval \((0,1)\).

Given our findings that LLMs can effectively learn both forward and reverse modeling when trained from scratch, we initially hypothesize a straightforward relationship between the step-by-step loss of two directions, which is shown as assumption (a) in Figure 3. The assumption (a) is that forward and reverse modeling would exhibit similar loss patterns throughout the sequence, explaining the near-zero mean difference in average losses in RQ2. However, our empirical results, presented in Figure 4, reveal a more nuanced dynamic. The results instead support assumption (b) in Figure 3: reverse prediction becomes progressively more accurate as contextual information accumulates, while forward prediction maintains consistent difficulty levels across the sequence. These trajectories intersect at a critical intersection point, before which reverse prediction shows higher loss values, and after which it demonstrates lower loss values compared to forward prediction. Note that this pattern emerges consistently across all the texts. It is a statistical characteristic of all the texts in our datasets and is independent of text quality, representing a fundamental difference of LLM's forward- and reverse-modeling behaviors.

To further understand this dyanmic, we analyze extreme cases (those in the top and bottom \(10\%\) of average loss differences) to identify the features that drive these divergent patterns and to examine how these dynamics change in extreme cases. A straightforward hypothesis (assumption (c) in Figure 5) would suggest that extreme cases simply shift the reverse loss curve vertically while maintaining its shape, with top-\(10\%\) cases shifting upward and bottom-\(10\%\) cases shifting downward. Under this hypothesis, the intersection point between forward and reverse loss curves would show small distance changes. However, our findings in Figure 6 contradict this hypothesis and instead support assumption (d) in Figure 5: extreme cases primarily result in large horizontal shifts of the reverse loss dynamic, while the forward loss dynamic remains stable (simple vertical shift). In cases where forward loss substantially exceeds reverse loss (top-\(10\%\) cases), we observe that reverse loss decreases rapidly, with the intersection point occurring very early in the sequence. Conversely, in cases where reverse loss is larger (bottom-\(10\%\) cases), the intersection point is delayed until near the sequence end, with reverse loss consistently exceeding forward loss throughout most of the process.

The results shows that while the average loss difference is an aggregated metric, it effectively indicates different patterns in step-by-step loss dynamics. With our case studies in Table 1, we find the obvious text quality differences between the reverse-favoring cases and forward-favoring cases. This finding suggests that text quality is the key feature influencing the loss dynamics and the positions of intersection points. Our analysis also re

Figure 4: Empirical step-by-step loss dynamics of full text data during decoding.

Figure 3: Assumptions on the step-by-step loss dynamics of full text data during decoding.

 

veals a key insight: texts exhibiting early intersection points in their loss dynamics typically have higher loss differences and correspond to higher quality content. This relationship enables us to use the loss difference as a quality score for text quality assessment.

**Application: _Texts favoring reverse modeling can improve original LLMs._**

As analyzed in RQ3, coherent and logical texts tend to have lower reverse losses compared to forward losses. Thus, given a training sample and a LLM model pre-trained from scratch with both forward and reverse training, we can define a simple quality score \(\mathcal{S}\) using the loss difference \(\mathcal{S}=\text{Avg}\). Forward Loss - Avg. Reverse Loss, computed as \(\mathcal{S}=\frac{1}{N}(\sum_{i=1}^{N}F_{i}-\sum_{i=1}^{N}N_{i})\). According to our prior analysis, A higher \(\mathcal{S}\) indicates that the text, which supports reverse modeling better, signifies a high-quality sample.

To further verify this assumption, we conduct continued pre-training on the publicly released Llama2-7B. Using the SlimPajama-6B Soboleva et al. (2023) as training data, we select 1B tokens with the lowest and highest \(\mathcal{S}\) scores, respectively. The model's performance is evaluated on MMLU Hendrycks et al. (2020). We also compare this with the following data selection strategies: (1) **Random 1B**: randomly sample 1B tokens, (2) **Perplexity Lowest Ranked 1B**: select the 1B tokens with the lowest perplexity by Llama2-7B.

The results from Table 2 show that the quality of training data significantly affects the performance of LLMs. Our high-quality data selection strategy (\(\mathcal{S}\) Highest Ranked) outperforms other baselines, achieving the highest accuracy across various tasks on MMLU. Since the overall text quality of the SlimPajama 6B dataset is inferior to the text quality used in the pretraining of Llama2-7B, using the full 6B dataset does not improve over the original Llama2-7B. This suggests that the presence of low-quality data in unfiltered training sets degrades performance, as evidenced by the significant performance decline with low-quality selection strategy (\(\mathcal{S}\) Lowest Ranked). This experiment supports the hypothesis that texts more effectively modeled by reversing are of higher quality and more beneficial for LLMs in acquiring world knowledge.

## 5 Conclusions

In conclusion, our results demonstrate that LLMs can learn from both forward and reverse-modeling texts with comparable proficiency when trained from scratch. This study also highlights the potential benefits of incorporating training data that favors reverse modeling. Our findings underscore the importance of exploring diverse reverse modeling frameworks to enhance the capabilities of LLMs.

Figure 5: Assumptions on the step-by-step loss dynamics of selected texts with the Top-\(10\%\) and Bottom-\(10\%\) loss differences during decoding.

\begin{table}
\begin{tabular}{l|c c c c|c} \hline \hline
* Reverse Loss. More results with different backbones across various benchmarks are shown in Appendix C.

Figure 6: Empirical step-by-step loss dynamics of selected texts with the Top-\(10\%\) and Bottom-\(10\%\) loss differences during decoding.

 



### Limitations

While our study demonstrates promising results in training LLMs with reverse modeling, several limitations should be acknowledged to provide a comprehensive understanding of the findings and guide future research.

Firstly, the simulation of reverse modeling by simply reversing token sequences may not fully capture the complexity and nuances of true reverse thinking processes. This approach reduces reverse modeling to a syntactic level, potentially overlooking deeper semantic and contextual factors intrinsic to human reverse modeling.

Secondly, the evaluation metrics used in our study, such as performance on downstream benchmarks like MMLU, may not fully encompass the benefits or limitations of reverse modeling. These metrics primarily measure specific aspects of language understanding and reasoning, potentially overlooking other critical dimensions influenced by reverse modeling, such as creativity or problem-solving skills.

Lastly, our research does not address the potential computational and resource challenges associated with training LLMs on reverse texts. The increased complexity and processing demands could pose significant barriers to practical applications, particularly in resource-constrained environments.

In conclusion, while our findings offer valuable insights into the potential of reverse modeling in LLMs, addressing these limitations is crucial for advancing this line of research. Future studies could aim to develop more sophisticated methods for simulating reverse modeling, explore diverse and naturally occurring datasets, and consider a broader range of evaluation metrics to fully understand and harness the benefits of reverse modeling in LLMs.

## Acknowledgments

We would like to express our sincere gratitude to the anonymous reviewers for their insightful comments and valuable suggestions, which substantially improved this manuscript. S. Y. and J. W. are indebted to Xiaoqun Xiao, whose expertise in Arabic linguistics and textual analysis is instrumental in developing the multilingual aspects of this work.

## References

* Bergen and Chan (2005) Benjamin Bergen and Ting Ting Chan. 2005. Writing direction influences spatial cognition. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 27. Cited by: SS1.
* L. Berglund, M. Tong, M. Kaufmann, H. Balesni, A. C. Stickland, T. Korbak, and O. Evans (2023)The reversal curse: llms trained on "a is b" fail to learn "b is a". In Proceedings of the the 12th International Conference on Learning Representations (ICLR), Cited by: SS4.
* J. Chih-Yao Chen, Z. Wang, H. Palangi, R. Han, S. Ebrahimi, L. Le, V. Perot, S. Mishra, M. Bansal, C. Lee, et al. (2024)Reverse thinking makes llms stronger reasons. arXiv preprint arXiv:2411.19865. Cited by: SS1.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. Won Chung, D. Sutton, S. Gehrmann, et al. (2023)Palm: scaling language modeling with pathways. Journal of Machine Learning Research (JMLR)24 (240), pp. 1-113. External Links: ISSN 0037-69X, Link, Document Cited by: SS1.
* C. Clark, K. Lee, M. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova (2019)BoolQ: exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Cited by: SS1.
* E. Clark, T. August, S. Serrano, N. Haduong, S. Gururangan, and N. A. Smith (2021)All that's 'human'is not gold: evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Language Technologies (NAACL), Cited by: SS1.
* J. Cornelius, O. Lithgow-Serrano, S. Mirrovic, L. Dolamic, and F. Rinaldi (2024)Bust: benchmark for the evaluation of detectors of llvm-generated text. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), Cited by: SS1.
* D. D. Kerchhove and C. J. Lumsden (2013)The alphabet and the brain: the lateralization of writing. Springer Science & Business Media. Cited by: SS1.
* A. Dubey, A. Jauhr, A. Pandey, A. Kadian, A. Al-Dahale, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. (2024)The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS1.
* O. Golovneva, Z. Allen-Zhu, J. Weston, and S. Sakhbaatar (2024)Reverse training to nurse the reversal curse. In Proceedings of the 1st Conference on Language Modeling (COLM), Cited by: SS1.
* J. Gu, Q. Liu, and K. Cho (2019)Insertion-based decoding with automatically inferred generation order. Transactions of the Association for Computational Linguistics (TACL)7, pp. 661-676.



* Guo et al. (2024) Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, and Yujiu Yang. 2024. Mitigating reversal curse via semantic-aware permutation training. In Findings of the Association for Computational Linguistics: ACL _2024_.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. In Proceedings of the 8th International Conference on Learning Representations (ICLR).
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv:2310.06825.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
* Longpre et al. (2024) Shaye Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2024. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).
* Mitra et al. (2024) Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830.
* Nguyen et al. (2024) Anh Nguyen, Nikos Karampatziakis, and Weizhu Chen. 2024. Meet in the middle: A new pre-training paradigm. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS).
* Papadopoulos et al. (2024) Vassilis Papadopoulos, Jeremie Wenger, and Clement Hongler. 2024. Arrows of time for large language models. arXiv preprint arXiv:2401.17505.
* Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebleesar Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS).
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1-67.
* Sharma et al. (2024) Vasu Sharma, Karthik Padthe, Newsha Ardalani, Kwsha Tirumala, Russell Howes, Hu Xu, Po-Yao Huang, Shang-Wen Li, Armen Aghajanyan, and Gargi Ghosh. 2024. Text quality-based pruning for efficient training of language models. arXiv preprint arXiv:2405.01582.
* Soboleva et al. (2023) Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. Slimpajama: A 627b token cleaned and deduplicated version of redpajama.
* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdrey, Quoc Le, Ed Chi, Denny Zhou, et al. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL _2023_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosalle, et al. 2023. LLama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288.
* Wenzek et al. (2020) Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2020. Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference (IREC).
* Wettig et al. (2024) Alexander Wettig, Aatmis Gupta, Saumya Malik, and Danqi Chen. 2024. Operating: Selecting high-quality data for training language models. In Proceedings of the 41st International Conference on Machine Learning (ICML).
* Wu et al. (2018) Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. Beyond error propagation in neural machine translation: Characteristics of language also matter. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
* Xie et al. (2023) Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. 2023. Data selection for language models via importance resampling. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurlPS).
* Zhang et al. (2024) Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.
* Zhong et al. (2024) Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Said, Weizhu Chen, and Nan Duan. 2024. Agricultural: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL _2024_.
* Zhou et al. (2019) Long Zhou, Jiajun Zhang, and Chengqing Zong. 2019. Synchronous bidirectional neural machine translation. Transactions of the Association for Computational Linguistics (TACL), 7:91-105.

 

Hanlin Zhu, Baihe Huang, Shaolun Zhang, Michael Jordan, Jiantao Jiao, Yuandong Tian, and Stuart Russell. 2024. Towards a theoretical understanding of the reversal curse'sia training dynamics. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS).

 

Implementation Details

In our initial experiments, we explore the effects of a higher learning rate (\(2e^{-4}\)) and extend the training epochs (\(2\) epochs) in a continued training setting. This exploration is based on the assumption that reverse modeling might need more epochs or a higher learning rate than forward modeling to overcome the pre-trained directional bias. While this increases the convergence speed, the final loss is nearly the same as when using a lower learning rate or a single training epoch.

Thus, to ensure consistency in our comparisons, we fix the learning rate as \(5e^{-5}\) and set the batch size to \(48\) (with each batch consisting of \(48\) paragraphs). Following established practices in LLM training Touvron et al. (2023); Chowdhery et al. (2023), we train for one epoch in all experiments. The number of training steps depends on the size of the data set and the batch size. For instance, training on a dataset with \(1\) billion tokens requires approximately \(2,400\) steps with our hyperparameter settings.

We use Llama2-7B Touvron et al. (2023) as the LLM backbone for Research Questions 1-3 (Section 4). All experiments are conducted using \(8\) NVIDIA A100-SXM-80GB GPUs, and the application experiments in Section 4 also use the Llama2-7B model.

## Appendix B Multilingual Experimental Results

We show the pre-training losses for both continued and from-scratch training across additional languages, including German, Korean and Arabic, in Figure 7. Note that Arabic texts tokenized by Llama2 tokenizer have the same orientation as English, with tokens from the first logical sentence of a paragraph positioned on the left rather than the right. Consistent with our findings in RQ1, Section 4, the forward loss during continued pre-training remains lower than the reverse loss. However, in the from-scratch setting, the loss curves for both directions converge similarly. These results further confirm that LLMs can effectively learn to handle both forward and reverse inputs with comparable proficiency when trained from scratch, regardless of languages.

We randomly sample additional multilingual cases German, Korean, and Arabic), as shown in Tables 4-7. Across all four languages, we observe a consistent pattern: Texts favoring reverse modeling tend to exhibit clear logical structures, while those favoring forward modeling rely more on repetitive formatting to emphasize their sequential flow.

## Appendix C More Results with Different LLM Backbones on Different Benchmarks

Besides the MMLU Hendrycks et al. (2020) benchmark, we also evaluate our proposed data selection application on three benchmarks, _i.e._, AGIEval Zhong et al. (2024), BBH Suzgun et al. (2023) and BoolQ Clark et al. (2019), using different LLM backbones including Llama2-7b Tuvron et al. (2023), Mistral-7b Jiang et al. (2023) and Llama3-8b Dubey et al. (2024).

The experimental results are shown in Table 3. Our high-quality data selection strategy (Highest Ranked) consistently outperforms other approaches across various benchmarks, regardless of the LLM backbone used. These results support our hypothesis that texts better modeled by reverse prediction yield higher quality data, which in turn enhances the LLMs' ability to acquire world knowledge. Notably, the ranked score, \(\mathcal{S}=\text{Forward Loss}\) - Reverse Loss, is computed and fixed using the Llama2-7b model throughout the experiments. This highlights the strong generalization capability of our method, as the high-quality data selected by one LLM can be effectively transferred to another.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline

- \\ Lowest Ranked 1B & 58.54 &
- \\ Highest Ranked 1B & **59.49** &
- \\ \hline \hline \end{tabular}
\end{table}
Table 3: Experimental results using three different LLM backbones on the MMLU, AGIEval, BBH, and BoolQ benchmarks. However, we exclude Llama3-8B’s results on AGIEval, BBH, and BoolQ, as the evaluation sets for these benchmarks are found to overlap significantly with its training data (contaminated rates: \(98\%\), \(95\%\), and \(96\%\), respectively) Dubey et al. (2024).
 



### System Prompt:

You are an expert in text quality checking, You need to score a given text used for large language model training from 1 to 10 according to the following factors:

1. Grammar: The spelling and grammar of the text, punctuation/formatting issues.
2. Level of detail: Is the text simple o does it go more in depth?
3. Genne: If the text is the genre/domain/style/formality that the reader expects, adheres to style norms.
4. Repetition: Words/phrases/content repeated itself.
5. Factuality: The accuracy of the text, whether it describes things that are "true."
6. Consistency: How the text relates to the context and other pieces of the text.
7. Common Sense: Whether the text "makes sense" within the world that it is written.
8. Coherence: The structure and coherence of the text. Order issues go here.
9. Wither intent and expression: Speculating about write's intent or capabilities.
10. MostImportant Extend Quality for LLM Training: this text will be used for LIM Training by causal language modeling. Please remember to give score strictly, score to differentiate the samples, and prevent to give most of the cases similar score. Output in one-line JSON format: ("score":"<score", ("reason":"=reason")

## Appendix D Qualitative Analysis

Many open-source LLMs use data classifiers for selection and cleaning, but the specifics of their training processes are often proprietary and not fully detailed in technical reports Touvron et al. (2023); Jiang et al. (2023); Dubey et al. (2024). Thus, we expand our experiments using the GPT-4 API 3 to directly evaluate the quality of texts from the Lowest Ranked 1B and Highest Ranked 1B texts, in line with previous studies Clark et al. (2021); Cornelius et al. (2024). We randomly select \(1,000\) samples from each dataset and apply a predefined prompt (shown in Figure 8) to assess each sample. The GPT-4 API assigns a quality score ranging from \(1\) to \(10\), based on criteria for high-quality text defined in Clark et al. (2021), along with an additional criterion for "quality for LLM training".

Footnote 3: [https://platform.openai.com/docs/spi-reference/](https://platform.openai.com/docs/spi-reference/)

The results show that the Highest Ranked 1B dataset achieves an average score of \(6.7\), while the Lowest Ranked 1B and Random 1B datasets scored \(4.9\) and \(6.15\), respectively. These findings further suggest that texts favoring reverse modeling are of higher quality and more suitable for LLM training.

## Appendix E Smaller LLM as Backbone Model

We also conduct experiments on the TinyLlamamad-1.1B model Zhang et al. (2024) under the same protocols used in Section 4 4. Figure 9 shows that the loss trend of TinyLlama-1.1B is similar to that of the larger Llama2-7B model. It indicates that even with smaller models, both forward and reverse modeling can be effectively handled in the foremost training setting. Next, we recalculate the

Figure 8: The prompt used for text qualitative evaluation using GPT-4 API. The \(1\)-\(9\) factors follows the designed evaluation labels in Clark et al. (2021), and we add an extra “Quality for LLM Training” into the evaluation factors.

Figure 7: Pre-training losses for both continued and from-scratch training settings in four additional languages. The patterns are consistent with the results observed in English.



quality score \(\mathcal{S}=\) Forward Loss - Reverse Loss for each case of the SlimPajama [123] dataset and select the lowest-ranked 1B and highest-ranked 1B data based on the recalculated scores. Notably, the overlap ratios of the selected cases are \(91.27\%\) and \(94.58\%\) when compared to Llama2-7B. This demonstrates that our method is model-agnostic, enabling smaller models to efficiently identify high-quality data for training larger models in practice.

Figure 9: Pre-training loss in the from-scratch setting using English data on TinyLlama-1.1B. The forward and reverse losses are nearly identical, aligning with findings from larger LLM models.

 



**Language****Tests Favoring Reverse (Low Reverse Loss)**

Have you ever wondered why cultures in the hottest locations on earth eat hot and spicy foods? Why is it that people in Central and South America, India, Africa. Southeast Asia and the Caribbean ethan's theater flavored with hot chile papers and species that make you sweat? There is a reason, and if s actually pretty smart when you think about about -- spicy foods make you sweat, which in turn helps you cool down faster. It’s as simple as that! Though you may be inclined to cool down with all glass of steel tea, ice cream or watermention on a sweetering summer's day, the effect isn't lasting. After a window we took to where you started -- not and bothered. That's because your internal temperature is cooled to rapidly, and you would only compels our compensating by rasing your temperature. As a result, you feel hotter. Eating spicy foods works differently:" in raises your internal temperature to match the temperature outside. Your bonded circulation increases, so what disweating and once your moisture has evaporated, you've cooked off. Scientists call the phenomenon "gustancy facial sweaning," because nicked you usually start swearing in the face first. Even though eating spicy foods on a hot day isn't the most pleasant for many people, it may be worth doing because after sweating it out you do actually cool down. What do you think: is it worth?

The kinetic equations for clean superconductors (\(I\gg\xi\)) are derived. Expanding the equations for the time dependent Green functions in the quasiclassical parameter, then new contributions are found which contain the derivatives of the distribution functions with respect to the quasiparticle momentum. The transition from the ultra-clean case (no relaxation) to a relaxation-dominated behavior, for which he kinetic equations coincide with the usual quasiclassical approximation, occurs for the relaxation time of the order of \(hE/Z\Delta\). The kinetic equations can be used for various dynamic processes in superconduccade including the flux-flow Hall effect. The derived equation, after necessary modifications for the p-wave pairing, are especially suitable for nonstationary problems in the theory of superfluidity of 3He.

How is this club different from standard Tostmasters Club? An Advanced Tourstames Club, Professional Speakers Frankfurt is only open to experienced members who have completed the Tostmasmerus Computer Communication model very far are advanced speakers with proven experience outside the Tostmasmerus world. Therefore, we can focus on more advanced issues. Rather than having the majority of speaks from the Competent Communicator manual we focus on advanced projects or practice approaches outside of Tostmasmenus. We prepare members for speech competitions at role happening with an upcoming professional or other important speech. Instead of having only one evaluator, all attendees will get the chance to provide feedback. Depending on the objectives of the speaker, the group might be divided into task forces to keep an eye on particular aspects and diderified the speaker afterward. We also use video recording to provide in-depth analysis of a speaker's performance. Sounds bring? In fact, it isn't. For a speech to become great; it has to go through multiple iterations. In our club, we give members opportunity to repeat a speech and incorporate the feedback they'v received. We hold regular advanced workshops up by members or outside experts on spoken-related topics. We encourage our members participate in Toastmasers speech councts and dedicate special time to prepare candidates. We will develop a paper coaching system through which members continuously coach each other. We will set up a Speaker Bureau, and members will be able to present and promote themselves and as speakers on the Club website and through the Club's online channels.

With SomaltoofMoJo you can convert and download music in High Quality MP3 format. Download tons of music from Soundloedwith our SoundloedMonitorand listener listen to them from anywhere by storing them on your iPad, computer or phone using our ultra fast downloading service. SoundCloudCloud is audio distribution site, where users can record, upload and promote their sound tracks. SoundCloudCloud allows you to listen as many tracks you can but it does not allow sound track download. Enter the Soundcloud URL that you wish to convert & Download. Click "Conceri t" to start the conversion process. Click "Download My3" to download the file. Once complete you will have final download link for converted sound. Highly Secure and high speed. Mg3 Converter supports a wide variety of modern browsers and devices.

## References

* [1] M. A. B. Anderson, J. R. Anderson, and D. S. Bailey. _et al._. 2014. New York Times: A Survey of Modern Optics. _Science_, 587:69-70, 2014.

[MISSING_PAGE_POST]





\begin{table}
\begin{tabular}{c c l}
* nicht nur am den großen Operationus, somers de Erde', an z anteperfer of . “Myasil” . “Itanium Soongle", niaktives . Liad er Oberre Jrédels in Intercept von Hugo Wolfs [Ilan] beacly a magic to limit . Italienischem Liedebrouch” oder expressions in a tribute to the im-alis ledenschmidtlic rhetoro, mortal music of Italy. .... auguerant. Ein ser bleungs die unsterble Music Haliens ihrer Envergies une besondere Magile vretuitr ... \\  & ... Elisabeth von Luxembourg & ... Eliashetto of Luxembourg was murder 422 .3lighting mit dem married to the 25-year-odd her 25 Jahre ethem Thornahyatur iterobt to the throne, Albert V, in 1422 Albrecht V. Verheiter (verbot) watzen she berents stetieth üern 2. Lebensjahrb. Nach ensten . Adthv. After the first ten years . Zach Juhanen Ebek sie ir cherasso van ter Kindern; famlor lake the first four children . Five years later, upon the death of his father, harband headbar . Gureux Kenigeige Koins . Guncz King wosite . Konstuen . Roman Emperor-electrolers, as well . Emsliebart war im finten . Monat mit gerwer Standenen . Glaighten Ausbaertung abedent, siew schwanger, alsr 1439 aufcharned . Heightser her husband de . Ungar einfallende fur Funkenige genom die in . Ungar einfallendo Turkema . Hauptung a military campaign against the Turks, who were in-molisthes Dennigan Degas des Ads, adding Hungary. Despite political pressure from the mobility Wadslaw II .mu bacteria .0 may be 15-year-odd Polismiken .00018 .




\begin{table}
\begin{tabular}{c c l}
**Language** & **Texts Favoring Reverse Loss)** \\  & **Tests Favouring Reverse Loss)** \\  & **USS video service market, who will be the final winner? Live talk within hosted by B2B IT experts Competition in Internet video services (Over The Top, ODT) is in full swing around the world. Netflix, the leader in global online video streaming services, Amazon.com’s In-17,068 the Internet video-on-demand service Amazon Video, and video sharing site YouTube are competing freely by offering their own services. The center is obviously the United States. They are pouriné out various aggressive marketing strategies, including the development of original content, with the belief that they can capture customers around the world only by capturing the American stage, the DATE 37% Star Sharing Platform II C+Celebration Silva Drug (US), Grid for current development is also an enormous. According to report by Qbusiness Insider on API \(J\), JP Morgan analysts analyzed all Amazon's budget to invest in strengthening video services in 2017 is $4.5 billion, or $1.1 million won. Amazon CFO Brian Olsavsky said, “We start with nearly double our investment 16.42 @ $3.574 @ $4.519 @ $4.518 @ $4.





\begin{table}
\begin{tabular}{p{14.2pt} p{390.8pt}|l l}
* 100 ml online
- Easy Click Channel Alture Senseual is a sophisticated women's perfume. It has a beautiful and charming oriental floorical scenti. Its fragrance is fragment and has a strong consistency. It adds elegant touches that catch the eye and are irresistible and last for long periods. See description for more information. About the movie Birdman, they worked together in the movie 21 Gramans. Sean said during the emotional campaign for his new movie, Gumma, that he surprised by the repeated criticism, despite his assertion that Alejando is one of the most important filmmakers in America. This comment did not carry any racist intentions and was only intended to provoke multiple. It is noteworthy that the duo, Ben and Gonzalez, had previously worked together in the film 21 Gramans, and the Mexican director stated immediately after the Oscar ceremony that there is a close friendship between him and the actor, so he accepts this comment with open arms and considers it a very funny joke. Birdman Alejandro Gonzalez Anaritson Bea* & _[-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [-] [--] [-] [-] [-] [--] [-] [-] [--] [-] [-] [--] [-] [--] [-] [--] [-] [--] [-] [--] [-] [--] [--] [-]