Towards System 2 Reasoning in LLMs: Learning How to Think With Meta
  Chain-of-Though
Authors: name: Violet Xiang; name: Charlie Snell; name: Kanishk Gandhi; name: Alon Albalak; name: Anikait Singh; name: Chase Blagden; name: Duy Phung; name: Rafael Rafailov; name: Nathan Lile; name: Dakota Mahan; name: Louis Castricato; name: Jan-Philipp Franken; name: Nick Haber; name: Chelsea Finn
Published: 2025-01-08 18:42:48

Abstract: We propose a novel framework, Meta Chain-of-Thought (Meta-CoT), which extends
traditional Chain-of-Thought (CoT) by explicitly modeling the underlying
reasoning required to arrive at a particular CoT. We present empirical evidence
from state-of-the-art models exhibiting behaviors consistent with in-context
search, and explore methods for producing Meta-CoT via process supervision,
synthetic data generation, and search algorithms. Finally, we outline a
concrete pipeline for training a model to produce Meta-CoTs, incorporating
instruction tuning with linearized search traces and reinforcement learning
post-training. Finally, we discuss open research questions, including scaling
laws, verifier roles, and the potential for discovering novel reasoning
algorithms. This work provides a theoretical and practical roadmap to enable
Meta-CoT in LLMs, paving the way for more powerful and human-like reasoning in
artificial intelligence.