Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to
  the Edge of Generalization
Authors: name: Boshi Wang, name: Xiang Yue, name: Yu Su, name: Huan Sun

We study whether transformers can learn to implicitly reason over parametric
knowledge, a skill that even the most capable language models struggle with.
Focusing on two representative reasoning types, composition and comparison, we
consistently find that transformers can learn implicit reasoning, but only
through grokking, i.e., extended training far beyond overfitting. The levels of
generalization also vary across reasoning types: when faced with
out-of-distribution examples, transformers fail to systematically generalize
for composition but succeed for comparison. We delve into the model's internals
throughout training, conducting analytical experiments that reveal: 1) the
mechanism behind grokking, such as the formation of the generalizing circuit
and its relation to the relative efficiency of generalizing and memorizing
circuits, and 2) the connection between systematicity and the configuration of
the generalizing circuit. Our findings guide data and training setup to better
induce implicit reasoning and suggest potential improvements to the transformer
architecture, such as encouraging cross-layer knowledge sharing. Furthermore,
we demonstrate that for a challenging reasoning task with a large search space,
GPT-4-Turbo and Gemini-1.5-Pro based on non-parametric memory fail badly
regardless of prompting styles or retrieval augmentation, while a fully grokked
transformer can achieve near-perfect accuracy, showcasing the power of
parametric memory for complex reasoning.