Rethinking Reflection in Pre-Training
Authors: name: Essential AI, name: Darsh J Shah, name: Peter Rushton, name: Somanshu Singla, name: Mohit Parmar, name: Kurt Smith, name: Yash Vanjani, name: Ashish Vaswani, name: Adarsh Chaluvaraju, name: Andrew Hojel, name: Andrew Ma, name: Anil Thomas, name: Anthony Polloreno, name: Ashish Tanwer, name: Burhan Drak Sibai, name: Divya S Mansingka, name: Divya Shivaprasad, name: Ishaan Shah, name: Karl Stratos, name: Khoi Nguyen, name: Michael Callahan, name: Michael Pust, name: Mrinal Iyer, name: Philip Monk, name: Platon Mazarakis, name: Ritvik Kapila, name: Saurabh Srivastava, name: Tim Romanski

A language model's ability to reflect on its own reasoning provides a key
advantage for solving complex problems. While most recent research has focused
on how this ability develops during reinforcement learning, we show that it
actually begins to emerge much earlier - during the model's pre-training. To
study this, we introduce deliberate errors into chains-of-thought and test
whether the model can still arrive at the correct answer by recognizing and
correcting these mistakes. By tracking performance across different stages of
pre-training, we observe that this self-correcting ability appears early and
improves steadily over time. For instance, an OLMo2-7B model pre-trained on 4
trillion tokens displays self-correction on our six self-reflection tasks.